Substack's AI Detector Has an Off Switch, and Readers Will Notice
Substack turned on AI detection for every post and note published after July 21, 2026, powered by Pangram, a classifier that scored between 99.8% and 100% accuracy in a University of Chicago Booth evaluation of roughly 2,000 human-written passages. Readers run the scan themselves from the three-dot menu. Writers can switch it off post by post, and that opt-out, not the detector, is what will change how newsletters get read.
The off switch is the real product decision
Substack shipped the detector and the escape hatch in the same release. Open the three dots on a post, hit Disable detection, and readers no longer see a Pangram result on that piece. It works per post rather than per publication, according to Substack Writers at Work, which means the choice comes up every single time you publish.
On paper that sounds like a sensible privacy default. And for a while it will be. The problem is what happens to optional labels once enough people stop treating them as optional. Nutrition panels are the obvious comparison. Nobody reads them closely until they notice a product that hid one, and then the absence carries more weight than any number on the box would have.
I don't think this bites in week one. Most readers have no idea the gesture exists yet. But give it a few months of the scan becoming a normal thing people do when a post feels off, and a disabled scan starts reading as an answer to the question rather than a refusal to take it. Writers who turn it off for perfectly good reasons (they got a bad short-form result, they object to detection on principle, they don't want their comment replies graded) will be sorted into the same bucket as writers hiding a fully generated draft.
My guess, and it is a guess: by early 2027 a meaningful share of paid newsletters, call it north of 10%, will put a standing provenance line in their welcome email rather than field the question one reader at a time. The ones who wait will be answering it in DMs.
The accuracy panic is aimed at the wrong number
Jane Friedman reported that most writers are deeply unhappy about the rollout, for two reasons: detection tools are not 100% accurate, and they get used punitively without human judgment. The second objection is completely fair. The first is weaker than the volume of complaint suggests.
In the Chicago Booth study, run by Brian Jabarian and Alex Imas across roughly 2,000 human-written passages in six categories (blogs, reviews, news, novels, restaurant reviews, and résumés), Pangram never dropped below 99.8% accuracy and held a false positive rate near zero across most decision thresholds. Its false negative rate ran 2% to 4%, meaning the likelier failure is AI text slipping through, not a human getting accused. GPTZero held 96% even on short passages. Originality.ai kept false positives under 1% but missed AI text between 10% and 40% of the time. Pangram's own published figure is roughly one false positive per ten thousand essays, about 0.004%.
So the witch hunt scenario is real but it is mostly a governance problem, not a math problem. Someone screenshotting a 70% score and running a pile-on does damage whether or not the classifier was right.
There is one technical caveat that deserves more attention than it's getting, though. Every detector in that study lost accuracy on passages under 50 words. Notes, comments, and replies are almost entirely under 50 words. Substack made scanning most casual in exactly the place the classifier is weakest. Sources also disagree on the floor: TechCrunch says the tool needs 100 characters, Friedman's write-up says 100 words. Those are wildly different thresholds and the gap matters, because one of them sits comfortably inside the unreliable zone.
The study also didn't test non-native English writers, which is historically where detectors have embarrassed themselves worst. That's the gap I'd want closed before anyone treats a score as evidence of anything.
A percentage is a bad unit for a mixed workflow
The scan returns a ratio: how much of this looks human, how much looks AI-assisted. Which sounds precise and isn't, because two very different processes land in similar territory. A writer who brainstorms structure with a model and then writes every sentence from scratch can score close to a writer who generated a draft and rewrote the headers. Same number, opposite amount of work.
Substack CEO Chris Best framed the goal to TechCrunch as writers handling the hard part, the original ideas, while software handles the rest, calling it a good use of AI. As a philosophy I don't hate it. It just isn't a thing a percentage can measure. Ideas don't have a detectable prose signature.
Which is why the quieter half of the release is the more useful half. Substack also added an optional "How I make this" statement, a short note where writers explain their process, and it surfaces alongside the scan. That's the piece worth caring about, and it got maybe a tenth of the coverage the detector did.
This does not stay on Substack
Content provenance has been drifting from policy page to interface element for about a year now. Cloudflare flipped its crawler default to blocked and made AI access a negotiated thing rather than an assumed one. Now a consumer newsletter platform has handed readers a button that grades the writer. The direction is consistent even if the specific implementations are messy.
If you run your list on Beehiiv, Kit, or Mailchimp, none of this touches your product today. There's no scan button on your archive and there probably won't be one soon. And to be fair, that's a real reprieve. But reader behavior doesn't respect platform boundaries. Someone who learns to scan on Substack carries the suspicion into every inbox they open, including yours, and the only difference is they'll have to judge by feel instead of by number. Feel is worse for you, not better, because you can't dispute it.
The provenance audit worth 20 minutes this week
Four things, roughly in order of how much they'd change my week if I ran a newsletter of any size.
Run your last 10 published pieces through Pangram before a reader does. It's the same classifier Substack licensed, so you're seeing the number they'd see. The benchmark I'd use: if more than 2 of 10 come back majority AI-assisted and you'd describe your own process as human-written with tooling, your editing pass isn't doing what you think it's doing. That's a useful thing to learn privately.
Write the process statement now, at 40 to 60 words, whether or not you're on Substack. Name the tools, and more importantly name where they stop. "Research and outlining with an assistant, every sentence written and fact-checked by me" is a real answer. "We use AI responsibly" is not, and readers have been trained to hear it as a dodge.
Decide your disable policy in writing before the first uncomfortable score arrives, because deciding in the moment always looks like a reaction. And keep short-form human where you reasonably can. Not for ethics, for statistics: a 30-word comment is the worst possible input for any classifier tested, and a bad score there is the one you'd have the hardest time arguing with.
One more, if you send email in volume. The same discipline that fixed measurement after Apple's privacy changes applies here. We wrote about which email metrics still tell the truth once the obvious one broke, and the pattern repeats: when a public signal gets noisy, the teams who already had their own accounting were fine and everyone else spent a quarter arguing about the noise.
Where I land on it
I keep going back and forth on whether this is good. A near-perfect detector attached to a voluntary display is a strange object. It's rigorous and gameable at the same time, and the people it will hurt most are probably not the people it was built to catch.
What I'm fairly sure about is smaller than that. Readers are going to start asking how newsletters get made, and they'll ask it whether or not there's a button. Having a real answer ready costs you an hour. Not having one costs you the benefit of the doubt, and that has never been an easy thing to earn back.
Notice Me Senpai Editorial
