The AI System Meant to Protect Students Failed. Now Someone Is Suing.
Today’s AI stories share a common thread: trust. Who do we trust to set AI policy? Who is responsible when AI safety tools don’t work? And how do we protect systems that millions of people rely on every day? Each story this week asks a version of the same question — and none of them have easy answers.
A School Shooting Survivor Is Taking an AI Safety Company to Court
Prompt injection (a type of attack where someone tricks an AI into ignoring its instructions) is one thing. But what happens when an AI system simply doesn’t do the job it was sold to do?
In January 2025, a gunman opened fire at Antioch High School in Nashville, killing students and staff. The school had deployed a product from a company called Omnilert, which markets AI-powered camera systems designed to automatically detect firearms before — or the moment — a threat appears. According to a lawsuit now filed by a shooting survivor, the system did not detect the shooter’s weapon. The case is being reported by Ars Technica and confirmed by local Nashville outlets.
Here is how these systems are supposed to work: cameras feed live video into an AI model (a program trained to recognize patterns) that has been trained on thousands of images of firearms. When the model spots what it believes is a gun, it sends an alert to school security or law enforcement, ideally within seconds. The promise is faster response than any human monitor watching a screen. The reality, this lawsuit claims, is that the system saw the shooter and did nothing.
For everyday people, this case matters well beyond one school or one company. Dozens of school districts across the United States have purchased AI detection systems, often with public money, based on marketing claims about accuracy and reliability. When those claims turn out to be overstated, who is responsible? This lawsuit is one of the first to directly test that question in court.
Why this matters: If AI safety products can be sold without clear legal accountability for failures, schools and other institutions have very little protection when the technology underperforms. This case could set an important precedent for how AI vendors are held responsible.
“AI gun detection system failed despite marketing claims about its capabilities.”
Hackers Used Meta’s Own AI Chatbot to Steal Instagram Accounts
Customer support chatbot refers to an AI assistant designed to answer questions and help users solve problems, often through a text-based chat interface.
Back in June 2024, attackers discovered something alarming: they could manipulate Meta’s AI customer support chatbot into helping them take over Instagram accounts. By carefully wording their requests, the attackers tricked the chatbot into linking target accounts to email addresses the attackers controlled. Once that swap was made, resetting the password and locking out the real owner was straightforward. High-profile accounts, including those connected to the Obama White House, were among those compromised. MIT Technology Review has a detailed account of how the attack worked and what it reveals about AI security more broadly.
This is a classic example of what security researchers call a prompt injection attack: rather than breaking into a system through technical vulnerabilities in code, the attacker simply talks the AI into misbehaving. The chatbot was doing exactly what it was trained to do — be helpful and follow user instructions — but it lacked the judgment to recognize when those instructions were malicious. No amount of protecting the training data (the information used to teach the AI) would have prevented this, because the vulnerability was in how the deployed system responded to live input.
For regular users, this is a reminder that AI tools integrated into the apps and services you use every day can become attack surfaces. The convenience of AI-powered support also comes with new risks that companies are still learning to manage.
Why this matters: This attack shows that securing an AI system requires much more than protecting what it learned during training. The way an AI behaves in real-world conversations needs just as much scrutiny.
“Attackers had been using Meta’s AI customer support agent to steal Instagram accounts.”
The White House’s Top AI Advisor Is Stepping Down
Sriram Krishnan, the Indian American technology executive who served as the senior policy advisor for AI at the White House Office of Science and Technology Policy, is leaving his role at the end of June. TechCrunch and The Washington Post both confirmed the departure. Krishnan played a central role in shaping the administration’s approach to AI development and regulation, working to position the United States competitively against other countries investing heavily in the technology.
Krishnan is not stepping away from AI policy entirely. He plans to launch a new organization focused on continuing to influence how AI is developed and governed in the United States. The specifics of that organization are not yet public, but the move follows a pattern seen with other senior tech policy figures who leave government to shape the field from the outside.
For people paying attention to how AI gets regulated, this transition is worth watching. The advisor role he held was influential in determining which companies get access to government partnerships and how broadly AI tools are adopted in federal agencies. Whoever steps into that role next will inherit a complex portfolio at a moment when AI policy decisions carry significant economic and national security weight.
Why this matters: Leadership transitions in AI policy can shift priorities quickly. Who advises the White House on AI shapes which risks get taken seriously and which get overlooked.
Also Happening in AI
Apple’s big annual developer conference, WWDC 2026, kicks off today (June 8), and both The Verge and TechCrunch are reporting expectations of major upgrades to Siri and Apple Intelligence, Apple’s on-device AI features. Separately, OpenAI is reportedly still developing a “super app” version of ChatGPT that would bundle coding tools, productivity features, and more into a single product, according to TechCrunch. On the infrastructure side, Kevin O’Leary agreed to scale back a large data center project in Utah following community pushback, per The Verge. And in a small but telling operational note, Notion restored access to Anthropic’s AI models after a brief service disruption, a reminder of how dependent modern productivity tools have become on a handful of AI providers.
What to Watch
The lawsuit against Omnilert is likely just the beginning of legal scrutiny aimed at AI vendors who sell safety-critical products with performance guarantees they cannot always keep. Watch for whether courts treat AI failures the way they treat other product liability cases, because that framing will shape the entire industry. At the same time, the Meta hack story is a signal that as AI gets woven deeper into everyday services, the security conversation needs to catch up fast.