Building an AI Text Detector From Scratch

An AI-text detector's score reflects patterns learned from its training examples, not a general probability that AI wrote a passage. Sebastian Raschka's tutorial introduction makes that distinction while proposing a detector built by further training a DistilBERT text classifier.

The classifier would assign text a score for the AI-generated category. Raschka explains why such systems face a moving target: a new generator can produce different patterns, while human writing can resemble patterns the detector associates with AI. The educational value is understanding how to build and interpret a learned scoring system. Its score alone cannot establish authorship.

“This is a small educational project for studying the limitations of AI detectors and exploring a verifier-based LLM application beyond regular reasoning models trained on math and code.”

Sebastian Raschka

“Disclaimer: AI checkers are essentially a cat-and-mouse game. AI checkers may learn to detect a certain pattern that is indicative of AI-generated content. Then, the next LLM may incidentally or deliberately not exhibit that pattern and avoid detection.”

Sebastian Raschka

“Here, we are going to develop a method similar to Pangram models, which, as far as I know, are behind Substack AI detection feature.”

Sebastian Raschka

“In essence, there are different ways to detect AI-written text, from supervised classifiers and perturbation-based probability tests to perplexity measures and watermarking.”

Sebastian Raschka