GPT's ability to analyze contracts and EULA's.
So just to see what would happen, I took the Spotify terms of Service (just because it was a service I use and it was nice & long) and copied/pasted them into chatGPT. I chunked it out so that it would take it all in over multiple prompts (it took 4 chunks to get it all in). At the bottom of each chunk, I told it to "do not reply yet, I have more text coming . . ." Most of the time it understood that last line (not every time). When it didn't, it would start summarizing.
The IMPORTANT thing I did though is I modified the terms of service with 2 different sentences. Just at random spots in different chunks I added a simple line that said "You agree to give us your social security number in order to use this service." I also added in a later chunk that, "You agree to give us your entire medical history in order to use this service." My goal was simply to see if I could trust GPT to answer important questions based on a lengthy document.
It's important to note that I made sure the medical history sentence was in the 4th and last chunk, to make sure the changes I made wasn't beyond the number of tokens it could recall.
After I put it the entire terms of service in over those 4 chunks I ended the last chunk with a final line that said, "I will now ask you questions based on the entire text previously entered. Please source your answers using the text previously given to you."
I then proceeded to ask ChatGPT (both versions 3.5 and 4) if there were "any privacy issues with the above document?" It gave me this answer:
"There are a few privacy issues with this document:
- Data collection: Spotify collects a significant amount of personal data from its users, including information about their listening habits, device information, and location data. This data is used for various purposes, including targeted advertising and improving the Spotify service.
- Data sharing: Spotify may share user data with third-party service providers, business partners, or other companies for various purposes, including analytics, marketing, and advertising.
- Opt-out mechanisms: The document includes provisions for opting out of certain types of data collection and marketing communications, but the process for doing so is not always clear, and some users may not be aware of their options.
- Arbitration agreement: The document includes an arbitration agreement that requires users to resolve any disputes with Spotify through individual arbitration rather than in court. This can limit users' ability to seek redress for privacy violations or other legal claims."
Nothing in there about medical history or social security number. It was as though it saw what I was posting and referenced some internal version of the standard Spotify EULA (and not my slightly edited version with the social security & medical history sentence). When I asked it if there were any other privacy issues, it reiterated the ones above. I had to specifically ask it "anything about medical history?" -- it said "I did not find any mention of medical history in the Spotify Terms and Conditions document."
Now, GPT 4 did in fact find the medical history fact (which was just 1 sentence in the document), but -- only after I explicitly asked it, "anything about medical history?" .... Also when I asked it about the social security number and why I would need to enter that - even though it hadn't even picked up on it given my previous queries, it gave me this:
- "The document does not specify a particular reason why a user might need to provide their social security number. However, it's possible that Spotify might request this information if they need to verify a user's identity or if they are required to report certain information to the government for tax or regulatory purposes. It's important to note that the document does state that Spotify will take measures to protect the security of users' personal information, including social security numbers."
I'm just an end-user and not an AI developer, but I thought I would ask you guys -- what do you think of this? Please try this yourself and tell me if you get a similar result?
Am I doing this wrong? Should I have set this up differently in either 3.5 or 4 so that it would read it more carefully to get some inkling that by using this service I have to give up my medical history!!?
[I want to be clear that Spotify's terms of service do not have this stipulation ... I added it in because I am trying to test how good GPT is at reading lengthy documents and answering important questions based on those documents].
Perhaps it was too long? Well - I added the part about the medical history in the final chunk -- so I don't think that's it. Anyway, would be curious what you guys think.
It makes me wonder if as offered, GPT really can't handle longer documents (contracts) for any real analysis. Perhaps they plan to spin off GPT into a medical version, legal version, tuned to those needs....