OpenAI has released a set of tools aimed at stopping the creation and spread of child sexual abuse material (CSAM) on its platform. The move comes as the company faces growing legal and public pressure over the safety of its products, particularly for younger users. TechCrunch AI has the full story.
A new classifier, new warnings, and new rules for developers
The centrepiece of this release is an image classifier, a type of AI tool that scans images and assesses whether they contain abusive content. OpenAI built this classifier alongside the National Center for Missing and Exploited Children (NCMEC) and is making it available to select companies through its API at no charge. The company is also working with the Tech Coalition and an organisation called ROOST to share access more broadly.
In early testing with partners, the classifier flagged 48% fewer images while still catching the same number of abusive ones. Images it does flag are twice as likely to actually contain abusive content. That means human reviewers spend less time looking at material that turns out to be harmless, and more of their attention goes to real cases.
Alongside the classifier, OpenAI is adding warning screens inside ChatGPT for users who try to prompt the system toward content that sexualises children. Users who do this will see an alert, be pointed toward support resources designed to help people stop harmful behaviour, and have their account suspended for 24 hours. Repeated offences can lead to permanent removal. OpenAI developed the wording of these warnings with several organisations, including the Lucy Faithfull Foundation, Protect Children, and researcher Joel Scanlan at the University of Tasmania, who independently tested the deterrent messages.
The company is also publishing new guidelines for developers who build products using OpenAI tools, to help them prevent, detect, and allow users to report CSAM. Early testing of these guidelines has led some developers to register with NCMEC to report such content. OpenAI is additionally providing law enforcement with AI-powered investigative tools, dedicated plug-ins, and data analysis features, along with training materials and a field guide.
Why the classifier matters
Think of a human content reviewer as someone sorting an enormous pile of mail, looking for a small number of dangerous letters hidden inside. Without help, they have to open almost everything. The classifier acts like a pre-sorter that removes most of the clearly safe envelopes first, so reviewers spend their time on the pile most likely to contain something harmful. The result is faster, more accurate work with less exposure to distressing material.
The classifier can also detect AI-generated CSAM, not just photographs, which matters as AI image tools become more capable and accessible.
What this means if you use ChatGPT
For the vast majority of users, none of this will be noticeable. ChatGPT already blocked CSAM-related requests before this update. What is new is the formal warning system and the 24-hour suspension, which create a more structured consequence for anyone who tries to push past those blocks. The guidelines for developers also mean that apps built on top of OpenAI tools should, in theory, be held to clearer standards on child safety.
This comes at a moment when OpenAI has faced criticism on multiple fronts. Safety-focused nonprofit Common Sense Media labelled the company’s technology as “unacceptable risk” for teenagers, and OpenAI has faced several lawsuits relating to harm caused by its products.
What to watch
Watch whether the classifier gets adopted widely beyond OpenAI’s own platform, particularly by smaller companies that may lack resources to build their own detection tools. It is also worth seeing whether the developer guidelines become an enforceable condition of API access, or remain voluntary recommendations. NCMEC reporting numbers from developers would be a concrete signal of whether the guidelines are having a real effect.