Can generative AI models like ChatGPT be “reverse engineered” in order to develop competing models? If so, will this activity be deemed legal reverse engineering or illegal trade secret misappropriation?
I have now written a few articles exploring this question, including Trade Secrecy Meets Generative AI and Keeping ChatGPT a Trade Secret While Selling It Too. But when I first asked this question a year and a half ago, I was getting responses purely in the negative. I asked a panel at a trade secret conference at Georgetown in 2023, “Can ChatGPT be reverse engineered?” Several members of the panel laughed. I would talk to AI experts, and the answer I got was along the lines of: “it’s not going to happen.”
Now it seems clear that at least partial reverse engineering of generative AI models is indeed possible, and of increasing concern to AI developers.
A few weeks ago, OpenAI momentously alleged that DeepSeek (a competing Chinese AI model) was “inappropriately” developed from ChatGPT using “knowledge distillation.” In comparison to “model extraction attacks,” “knowledge distillation” is less frowned-upon and is typically motivated by efficiency and cost reduction, rather than exact replication. Knowledge distillation typically involves training a smaller ‘student’ model to mimic a larger ‘teacher’ model’s outputs on specific tasks for efficiency, while model extraction attacks are more aggressive attempts to replicate a model’s entire functionality through systematic querying designed to extract the underlying architecture and parameters. That said, the difference between these two is nuanced, and some sources depict knowledge distillation as a form of “model extraction” or “model stealing.” Apparently OpenAI sees neither as an “appropriate” means of copying.
Assume it’s true that DeepSeek used the outputs of ChatGPT to develop a competing model. A major question is whether this is legal “reverse engineering,” or instead a violation of trade secret law.
This legal question is about to be tested. A new trade secret lawsuit was filed on Wednesday, Feb. 26, alleging that extracting data from a generative AI is misappropriation of trade secrets and breach of contract, among other things.
In OpenEvidence, Inc. v. Pathway Medical, Inc., OpenEvidence Inc., alleges that a Canadian company, Pathway Medical Inc., used a so-called “prompt injection attack” to extract “trade secrets” from OpenEvidence’s generative AI model with the goal of developing a competing system.
OpenEvidence’s claims include: (1) acquisition of trade secrets in violation of the Defend Trade Secrets Act, (2) breach of contract due to violation a terms of use, (3) unauthorized access to a computer system in violation of the Computer Fraud and Abuse Act, and (4) circumvention of access control measures to obtain copyrighted content in violation of the Digital Millennium Copyright Act.
OpenEvidence (which is not affiliated with OpenAI) distributes a popular generative AI tool for use by medical professionals and patients. OpenEvidence’s large language model, similar to ChatGPT, appears to users as a chatbot which can be used to ask natural language questions about medical issues, like diagnoses, treatments, and medication side effects. (Complaint, 14-15). OpenEvidence is open to the general public for free, but general public users only get two questions per week. (Complaint, 16). Meanwhile, licensed medical professionals can get unlimited access, upon proving their license number and attesting, through a terms of use, to be a licensed medical professional. (Complaint, 16-17).
OpenEvidence’s trade secret misappropriation case is, at least initially, going to look very strong to the U.S. district court in Massachusetts where it was filed. Judge Myong J. Joun has been assigned to case.
This case raises some big picture questions about protecting generative AI models as trade secrets.
First: how hard is it, in fact, to reverse engineer generative AI’s? A few years ago, people apparently thought reverse engineering generative AIs would be hard if not impossible. But now it’s not so clear.
Second: how will courts view data extraction through strategic prompting in order to learn about how a particular model was developed? Will they see this as akin to buying a product on the open market and picking it apart, i.e., traditional legal reverse engineering? Or will they view this as acquisition by improper means, like hacking into a computer or flying a plane over an unfinished plant to see what’s inside?
Third: how much deference will courts give to contracts? Can attaching a terms of use that prohibits reverse engineering turn otherwise-lawful reverse engineering into acquisition by improper means? When construing the scope of liability under the Computer Fraud and Abuse Act (CFAA), the Supreme Court has indicated merely violating a contract isn’t necessarily a CFAA violation. But courts in some trade secret law cases have held that accessing information in knowing violation of a terms of use is by definition “improper.”