Reverse Engineering GPTZero: Verdicts are Document-Level, Non-Additive, and Order-Dependent

Over the past month while developing Penned, a free AI humanizer and detector, we discovered intriguing, counterintuitive results when testing our algorithm against GPTZero v3 API. While most people assume AI text detectors function as sentence-level statistical classifiers, our empirical testing revealed that GPTZero operates at the document level, with non-additive and order-dependent classifications. Here are seven key findings from our evaluation:

1. Document-level verdicts with sharp thresholds: Near the 0.5 decision threshold, GPTZero exhibits a sudden non-linear change. The addition of even one extra paragraph or a single table caption can cause the entire document to flip its classification from human to AI-generated. Once this threshold is crossed, the API marks every sentence as AI-generated, including sections that would otherwise score 0% AI in isolation.

2. Non-additive and order-dependent classifications: The most striking aspect is the behavior when combining sections. Section 5.2 alone gets a 0% AI score, and Section 5.3 (paragraph 1) alone also earns a 0% AI score. However, when these two sections are combined, the classification flips to 100% AI. Additionally, the order of sections matters greatly.

In one permutation of the results section from an academic paper, the text scored 39% AI when reordered, but the same text scored 100% AI when placed in its actual narrative order. Swapping just one paragraph for a Penned-generated alternative can flip the entire document's classification.

3. Segmented sentence probabilities: When examining GPTZero's generated probabilities at the sentence level, the scores are not smooth but rather segmented into structural blocks. In documents classified as human (e.g., 39% AI), individual AI segments are apparent. For example, sections 4.1–4.3, sections 6, 7, and 8 all hover near 100% generated probability, while sections 4.4 and 5.1 hover near 10%.

The global document score appears to weigh these segmented blocks against each other rather than simply averaging raw sentence scores.

4. Precision and numbers as strong human indicators: Detailed numerical information strongly influences detection. In our test paper, replacing specific empirical numbers with generic placeholders (e.g., substituting 42.8% with N%) transformed a 39% AI human document into a 100% AI document. AI detectors associate dense, concrete domain measurements and precise numerical observations with human-written scientific work.

5. Formatting artifacts affecting detection: How text is copied or extracted plays a significant role. Taking the same 39% AI text, copied directly from a PDF with hard line breaks, increased the AI detection score to 60% AI. However, clean semantic paragraph breaks did not trigger this jump – it was the irregular hard line breaks from PDF copy-pasting that caused the detector to flag the text as AI-generated.

6. Vulnerability of discussion/conclusion sections: Summarizing findings is particularly challenging with modern detectors. Discussion and conclusion sections, which synthesize findings, restates implications, and provide broad takeaways, are often flagged as 100% AI using various rewriting techniques, such as multiple candidate passes, intentionally broken grammar, and fact-only sentences.

On the other hand, raw measurement-based reporting (step-by-step procedural descriptions and direct observations) tends to fare better under these detectors, as LLMs are trained to summarize and synthesize, making synthetic summaries appear overly machine-like.

7. Open-source detectors' limitations: We benchmarked several open-source detection models against the same test suite: DeBERTa, ModernBERT, Binoculars/Fast-DetectGPT (with Qwen2.5-0.5B), and ZeroGPT. The results showed that these local detectors fail to detect AI-generated text that GPTZero flags as 100% AI. This indicates that proprietary v3 detectors possess unique capabilities that open-source alternatives lack.

These findings have significant implications for building AI humanizers. Our experience with Penned demonstrated that a simple paragraph-by-paragraph approach using local detectors and logistic regression features is insufficient. The detector's non-additive and order-dependent decision boundary requires a document-level coordinate descent signal, where candidate combinations are tested across the entire document structure to find the global optimum.

Fine-tuning local pickers necessitates calibration on large batches of paired detector verdict labels. We are currently creating open benchmarks and testing calibration pipelines publicly, and Penned is available for free testing at penned-ai.vercel.app.