Testing Pangram AI Detection with Scanned Forgotten Books
Scanning for Pangram Errors
I tested Pangram's AI detection claims by scanning forty-five old, non-digitized books to rule out memorization of training data. While the tool flagged a few segments, these errors stemmed from Mistral OCR hallucinations rather than false positives on human writing. The results suggest Pangram is not simply memorizing its dataset, though human review remains essential for reliable moderation.
If an author wrote as many words as Stephen King, they would expect to have three segments come up as AI in their entire career.
- pinnate
I’m a teacher that requires students to write, so I’m always on the lookout for reliable AI detection. Haven’t anything close to reliable yet, but Pangram definitely seemed to be better than anything else in my (very limited) tests.
That said, it did incorrectly flag something that I personally wrote as AI, which baffled me. I spent a while playing around with the text and feeding it back into Pangram, trying to figure out what the issue was. Turns out, I had a section in the text with three bullet points. I removed the bullets (literally just the bullets themselves, no actual text) and it passed as human.
While it seems to perform better than anything else, I’m a bit skeptical of their claimed success rate.
- jcalx
These are all genre fiction books from the 1960s-80s and I imagine they are very stylistically different from the majority of AI-generated text today, in an easily distinguishable way. I'm curious if Pangram would flag LLMs explicitly emulating, or even trained on, genre fiction from that era, given that its primary uses are detecting contemporary AI-generated content (essays, prose, code, assignments).
- Ukv
> Pangram boasts a false positive rate of 1 in a 10,000. That is, if Pangram says a block of text is AI there is only a one in ten thousand chance that it was written by a human.
That'd be if they had a false discovery rate of 1/10,000.
If for instance:
* 100,000 samples are tested
* 100 of which are AI-generated, the rest human-written
* Pangram flags 50 of the AI-generated samples (true positives)
* Pangram also flags 10 human-written samples (false positives)
Then the FPR is 1 in 10,000, but the chance that a flagged sample isn't actually AI (FDR) is 1 in 6.
- Aurornis
Interesting methodology. The results for Pangram are surprisingly good.
These tools are definitely not 100% perfect, which is the primary complaint used to dismiss them. However the error rate is also getting impressively low.
In most cases I see socially and online, there is a high suspicion that the content is AI generated before someone thinks to submit it to Pangram. It’s used on-demand as a tool to confirm suspicions. I have seen several cases where Pangram had some false negatives where the content was judged to be likely human written but the author later admitted it was written by an LLM.
Pangram is very interesting in the context of Substack because the platform was a target for lazy AI newsletters. People realized they could start 10 (or maybe many more) newsletters and spend only a few minutes getting ChatGPT to write posts for them. Starting an AI generated substack and trying to get paid subscribers for it was becoming one of the popular ways to use AI to try to get a little cash. Having a tool that makes it a little bit harder, at least until the LLMs get good enough to evade it, was important for the platform.
- gjm11
On the other hand, see https://freddiedeboer.substack.com/p/i-wouldnt-say-pangram-i... where Freddie deBoer shows some instances where Pangram is definitely giving wrong or misleading results.
(Somewhat-plausible-to-me explanation: It's looking for various stylistic features; older writing very rarely has the most AI-like features, or perhaps almost always has some non-AI-like features that outweigh whatever signs of AI-ness might be there. Present-day writers are more likely to get wrongly flagged as AI.)