A Lawyer Just Got Fined for Letting ChatGPT Invent His Witnesses
AI is having a rough week in the courtroom — and a surprisingly interesting one in Washington. Today’s stories cover a legal disaster that’s become a cautionary tale for professionals everywhere, a political push to reshape how American AI gets built, and a quiet software update that affects anyone building AI tools. The thread running through all of it: who’s responsible when AI gets something badly wrong?
A New Mexico Lawyer Was Fined $5,000 for Submitting ChatGPT’s Invented Evidence
This one is hard to read, but important. A New Mexico lawyer used ChatGPT to help draft legal documents for what appears to have been a murder case, then submitted those documents to the court without checking whether the details were real. They weren’t. The AI had fabricated case citations and invented witnesses who don’t exist. The court sanctioned the lawyer and handed down a $5,000 fine, according to Ars Technica and The Verge.
This happens because AI models like ChatGPT don’t look things up the way you’d search Google. They generate text by predicting what sounds plausible, based on patterns learned from vast amounts of writing. When asked about a specific legal case, the model can produce something that reads exactly like a real citation — correct formatting, plausible names, convincing details — while being entirely made up. Researchers call this “hallucination,” meaning the model confidently states something false.
For anyone using AI at work, the lesson here is direct: AI is a drafting tool, not a research tool. Treat anything it tells you about specific facts, names, or sources the way you’d treat a tip from a friend — worth checking, not worth staking your reputation on. Lawyers have professional obligations that make this especially high-stakes, but the same risk applies to anyone submitting AI-written content in a professional setting.
Why this matters: Courts are seeing more of these cases, and judges are running out of patience. If you use AI to help with any serious document, verify every specific claim before it leaves your desk.
“Lawyer submitted fake case details and witnesses invented by ChatGPT to court”
Y Combinator’s Garry Tan Wants American AI Labs to Copy a Chinese Playbook
Distillation — the process of training a smaller, cheaper AI model to mimic a larger, more powerful one — became famous earlier this year when Chinese lab DeepSeek used it to build a capable model at a fraction of the usual cost. Now, according to TechCrunch, Y Combinator president Garry Tan is arguing that American startups should do the same thing with leading US models.
Open-weight models are AI systems where the underlying code is made publicly available, so anyone can download, modify, and build on top of them. Tan’s argument is that if smaller US companies distilled from powerful American frontier models — the large, cutting-edge systems built by companies like OpenAI and Anthropic — you’d end up with a richer ecosystem of capable, American-made open-source tools. Right now, developers who want a free, open alternative sometimes reach for Chinese-built options instead.
For everyday people, this debate might seem abstract, but it shapes what AI tools end up in the apps and services you use. A stronger domestic open-source ecosystem means more competition, more variety, and less dependence on a handful of companies for the AI that powers everything from customer service bots to writing assistants.
Why this matters: The race to build open AI isn’t just about technology — it’s about which country’s values and rules get baked into the tools the world relies on.
“Distill frontier models to build stronger American open-source AI ecosystem”
Also Happening in AI
LangChain — a popular toolkit that developers use to connect AI models to outside data sources and tools — released version 1.6.3 of its core library, adding a feature that helps AI models better track what they’re doing mid-task. It’s a small update, but LangChain sits inside a lot of AI-powered products you may already use. Separately, MIT Technology Review is digging into a genuine rift inside the AI industry: how seriously should companies take warnings about catastrophic AI risk, and who gets to decide when those warnings cross from responsible caution into public alarm? It’s a quieter story than the courtroom drama above, but it may be the more consequential one long-term. And if you’re building a startup, TechCrunch’s Disrupt 2026 conference has one week left to book exhibit tables, with the deadline falling on September 18.
What to Watch
The lawyer story isn’t isolated — it’s part of a pattern of courts, regulators, and employers figuring out exactly where AI responsibility lands. Watch for formal guidelines from bar associations and professional licensing bodies over the next few months, because informal warnings are giving way to binding rules. Meanwhile, Garry Tan’s distillation push will be worth tracking as a signal of where Y Combinator’s investment appetite is heading — the startups that get funded in the next cycle may look very different because of arguments like this one.