Your Detection Model Is a Training Signal
Why detection systems become curricula for adversaries, and what enforcement reports still fail to measure
I spoke to Business Standard about platform responsibility for CSEAM detection; the piece is here.
In 2025, the National Center for Missing and Exploited Children received more than 1.5 million CyberTipline reports with a generative AI nexus. Two years earlier that number was 4,700. It reads as a catastrophe or as a detection breakthrough, depending on what you already believed.
It is mostly neither. Of those 1.5 million, 1.1 million came from a single source, Amazon AI Services, and NCMEC classified all of them as containing no actionable information. When Bloomberg investigated the subset filed in the first half of 2025, it found that every one of those reports flagged known material, discovered by hash-scanning Amazon's own AI model training data. None of it was AI-generated. Amazon has since said it deliberately set an over-inclusive scanning threshold that yields a high proportion of false positives, and NCMEC says the reports arrived without the file attached, or even the hash value.
Strip that block out and NCMEC counts roughly 182,000 reports involving an offender who possessed, generated or attempted to generate the material. A further 133,000 had some AI connection that could not be characterized at all.
So the headline figure is largely a measurement of how hard one company ran one scanner, at a sensitivity it chose for itself. The next largest slice is a measurement of what could not be determined. Neither is a measurement of the adversary. That gap is the whole problem with how we think about detection, and it took six months and an investigative newsroom to surface it.
The line I gave, and the part that didn't fit
I was asked about platform responsibility for child sexual exploitation material in Business Standard recently, and the point I made was that malicious actors evolve their tactics faster than detection models can adapt. Networks operate deliberately in the grey areas, using coded language, edited visuals, screenshots, memes, private groups, ephemeral content and multilingual communication.
That is the short version. The longer version is that this is not a child safety problem that happens to involve AI. It is a structural property of every detection system deployed against an opponent who is paying attention, and the field already has a name for it.
Adversarial drift
Ordinary concept drift is the world changing underneath your model. Spending habits shift, language moves, last year's training distribution stops describing this year's data. You retrain on a schedule and you are fine.
Adversarial drift is a different animal, and the literature separates it on three properties. It originates only from the malicious class. It is always aimed at subverting the deployed classifier. And, most importantly, it is a function of that classifier, because the adversary reverse-engineers what you shipped and moves specifically around it.
Read that third property slowly, because it inverts the usual mental model. Your detector is not a wall that erodes. It is a curriculum. The moment you deploy it, you have published a specification of what gets through, and the other side gets to read it, run experiments against it, and keep whatever works. Every block you issue is a labeled training example delivered free to your opponent.
Retraining does not escape this. It just starts the next round.
The same shape, three industries
Hash matching finds material that has already been reported, validated and added to a database. Genuinely new content is invisible to it by definition, and perceptual hashes, which exist precisely to survive cropping and re-encoding, can still be defeated by edits targeted at small regions of the image. Synthetic material never had a hash to begin with.
In digital advertising, Fraudlogix measured 105.7 billion impressions across 2025 and put invalid traffic at just over 20 percent. Take the number with the appropriate salt, since it comes from a company that sells fraud detection. The interesting part is not the volume anyway, it is the behavior: current bots simulate mouse movement, scroll cadence, realistic dwell time, and hesitation before form submission. They pause before submitting because we taught them to. Instant submission used to be the tell, we shipped detectors that keyed on it, and the population that survived was the population that waited.
In AI security, a 2025 collaboration between researchers at OpenAI, Anthropic, Google DeepMind and ETH Zürich took 12 published defenses against prompt injection and jailbreaking and broke essentially all of them, with success rates above 90% in most cases. The majority of those defenses had originally reported near-zero attack success rates. The difference was not a cleverer attack. It was allowing the attacker to adapt to the specific defense in front of it. Their paper is called The Attacker Moves Second, which is the entire lesson in four words.
Three unrelated industries, three different data types, one failure. That is not a modeling problem you fix with a better model.
What would actually be different
Score sequences, not items. An image that is innocuous in isolation can be one frame of a grooming sequence when you read the conversation it sits in. No single click in a fraud campaign is fraudulent either. When the harmful unit is a pattern over time and your detector's unit is a single artifact, you are not being outsmarted, you are answering a different question than the one that matters.
Measure the adversary, not just yourself. Precision and recall describe your model. They say nothing about how fast the other side is turning over its techniques. Meta's adversarial threat reports come closest, documenting how specific operations shifted tactics under enforcement pressure, down to breaking up words with zero-width unicode characters so text matching would not catch them. But that is narrative threat intelligence about a handful of named networks, not a measurement. The enforcement reports, the ones that actually cover detection at scale, publish removal volumes, proactive rates, zero-view rates and automation accuracy. Every one of those figures describes the detector's own performance. How fast the adversary is adapting is the number that tells you whether you are ahead, and it is not in any of them.
Constrain the surface instead of filtering it. The most useful development in AI security right now is the shift from better classifiers toward architectural limits on what a compromised system can do. Content moderation deserves the same move. That is what prevention by design means, and it is the point the child safety experts have been making for years while platforms kept treating this as a content moderation scoring exercise.
The number I would rather see
Meta says it removed 160,000 accounts in India, more than four million suspicious accounts overall, and 36 million pieces of exploitative content. Every one of those figures describes the detector.
I would trade all of them for one honest disclosure of how quickly the evasion patterns changed in response. That is the number that tells you who is winning, and it is the one no enforcement report contains.
Sources
- NCMEC, The Work Never Stops: A First Look at NCMEC's 2025 Data (31 March 2026): missingkids.org
- NCMEC, CyberTipline Data: missingkids.org
- NCMEC, Generative AI: missingkids.org
- Riana Pfefferkorn, Stanford Center for Internet and Society, Letter to NCMEC about AI-CSAM Report Statistics (29 January 2026): cyberlaw.stanford.edu
- Bloomberg, Amazon Found Child Sex Abuse in AI Training Data (29 January 2026): bloomberg.com
- Sethi and Kantardzic, Handling adversarial concept drift in streaming data, Expert Systems with Applications (2018): sciencedirect.com
- Nasr, Carlini, Sitawarin, Schulhoff et al., The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections, arXiv:2510.09023 (October 2025): arxiv.org
- Meta, Quarterly Adversarial Threat Reports (Transparency Center): transparency.meta.com
- TikTok, Community Guidelines Enforcement Report and DSA transparency reporting: newsroom.tiktok.com
- Fraudlogix, Ad Fraud Statistics 2026 (2025 annual dataset, 105.7 billion impressions): fraudlogix.com
About the author
