Who Checks the Drafter? A Behavioural Audit of Reference and Claim Errors in LLM-Assisted Education Research

Abstract

Large language models are now routinely used to draft, summarise and reference research in education, a field where the resulting claims reach teachers and policymakers. We report a small behavioural audit of one such case: twelve published papers on AI in education, drafted with substantial help from language models (the records name Claude and, for some planning material, Gemini), and the planning documents around them. We combine four kinds of evidence: the human orchestrator's own error records, a Crossref check of 21 DOIs cited across the papers, an independent database lookup of the 14 references in one Claude-written draft, and an audit of 35 items supplied by a second assistant. Existence checks and citation-accuracy checks gave different answers. In the 14-reference draft, 13 references resolved to a real work, but only 9 were correct in every field we checked; the others had paraphrased or truncated titles, a wrong author initial, a stale publication status, or no match. Errors clustered in fine-grained details (titles, initials, dates, percentages, attributions) while the overall register stayed confident. The checker we used also produced a false alarm. We propose a taxonomy of nine error types and a simple independence test for verification. This is a small, retrospective case study with no access to model internals, and we state its limits.

Keywords