A reviewer asked for a citation. I could not tell whether it was real. That was the problem.

A reviewer questioned the authenticity of a citation in a draft they were reviewing. The draft contained 41 references, with 38 of them verified. The problematic reference, number 22, appeared to be legitimate but, upon investigation, was found to be nonexistent. The citation had been generated by an AI language model and appeared structurally accurate, yet it did not correspond to any actual paper.

This posed a significant problem for the reviewer, as they could not definitively prove that the citation was fake. The issue stemmed from the fact that the absence of evidence does not equate to evidence of absence. In their review process, which spanned two years, the reviewer had developed a checklist that encompassed formatting, suitability, and alignment with the claimed contribution.

However, the checklist included only one item related to references, which was whether the citation format appeared correct. This check proved insufficient, as it could not definitively verify the existence of the cited work. To address this, the reviewer rebuilt their checks around three reliable methods. Firstly, they performed DOI resolution, which, if successful, confirmed that the work existed and allowed for a comparison of the title and authors.

Secondly, they conducted an author-and-title search within open venues such as arXiv, ACL Anthology, and CVF-style platforms. Finally, when neither method yielded results, the reference was marked as unverified. The author then had the opportunity to independently verify the reference. The rubric outlined a scoring system of 100 points, with the unverified references constituting a crucial portion of the assessment.

By tracking these unverified citations, the reviewer aimed to instill in authors the understanding that the model's knowledge had its limits. This approach aimed to prevent authors from blindly trusting the model's output. The reviewer emphasized that tools that silently remove references risk fostering an environment where authors become overly reliant on automated systems, while tools that merely report issues without providing any insight hinder the learning process.

They advocated for tools that surface problems and allow authors to make informed decisions based on the remaining information. The reviewer also highlighted that the check did not predict acceptance or generate citations; its sole purpose was to ensure compliance with the specific requirements of the venue being reviewed. It was crucial to maintain the distinction between a tool that surfaces problems and one that resolves them invisibly, as the latter can inadvertently teach authors to trust the system blindly.