99 percent accuracy. That sounds like a reliable AI detector. Wanting a tool that says “this was AI” at the push of a button is understandable, and vendors advertise with numbers like this. Let’s do the maths anyway.
One percent sounds small until you scale it
A school runs 1000 essays through a detector. Every one of them was written honestly. A one percent false alarm rate means ten students fall under suspicion, even though they did nothing wrong.
False alarms grow with volume. The more texts you check, the more innocent students get flagged, even if the percentage stays the same. Each case is a difficult conversation with trust at stake.
And 99 percent is a deliberately optimistic figure.
In practice the tools are worse
In 2023, a research team led by Weber-Wulff tested fourteen detection tools. Their verdict: the tools are neither accurate nor reliable.
A second study, by Liang and colleagues, points to a fairness problem. English texts by people writing in a language that isn’t their first are flagged as AI especially often. The study looked at English essays, including TOEFL essays. It says nothing about other languages.
The researchers suggest that detectors may unintentionally penalise writers with more constrained linguistic expression. They explicitly caution against using these tools in evaluative or educational settings.
OpenAI, the company behind ChatGPT, shut down its own detector in July 2023. The reason: its low rate of accuracy.
A reason to look closer, not proof
At most, a detector result is a reason to look more closely. Other things do more in everyday teaching. In the video I recommend three:
- Make the writing process visible. Drafts, notes and intermediate versions become part of the submission. Students who developed a text themselves can show how they got there.
- Talk with the student about the text. Not an interrogation, but part of the assessment: Why this structure? What was hard?
- Design tasks that require their own thinking. For example, by connecting them to your own lessons, personal experience or a decision that has to be justified.
The maths for your own staff room
The calculation above works in any staff meeting. How many written assignments are handed in at your school each semester? Take one percent of that, the optimistic figure. The result is the number of conversations with honest students you would have to have.
After that, it is worth asking which tasks in your own teaching would make a detector unnecessary.
Trust doesn’t come from software. It comes from good tasks.
Frequently asked questions
How reliable are AI detectors?
Not very. In 2023, a research team led by Weber-Wulff tested fourteen detection tools and concluded that they are neither accurate nor reliable. OpenAI shut down its own detector in July 2023, citing its low rate of accuracy.
Is an AI detector result proof?
No, at most it is a reason to look more closely. Even at an optimistic 99 percent accuracy, ten out of 1000 honestly written essays would be flagged wrongly.
Are AI detectors biased against non-native writers?
For English, yes: according to a study by Liang and colleagues (2023), English texts by people writing in a language that isn't their first are flagged as AI especially often. The study did not look at other languages. The researchers caution against using these tools in evaluative or educational settings.
