Tag: academic-writing

Automatically Verify and Correct the Reference Citations in a Paper

Peer reviewers and journal editors have a new tool to automatically check the references in manuscripts, developed by Jennifer Kleiman at the University of Georgia. You copy the list of references from the manuscript, paste it into the tool’s web page, and it searches the internet for each of the cited works. If it finds the work, then it corrects the reference format as needed to conform to APA style. If it cannot find the work, then it informs you so you can resolve whether the reference is genuine.

I tried out the tool on a manuscript that was recently assigned to me as Associate Editor. The tool is at https://jenkleiman.com/reference-checker/  I copied the list of references page by page from the .pdf of the manuscript and pasted them into the Paste References box on the tool’s web page. In a few cases the tool mistook the last line of a citation for the start of a new one. In all the other cases it found the work the citation referred to. It corrected the format of the citations to comply with APA style, and in several cases it inserted information such as DOI number that had been left out of the reference in the manuscript.

As a further test, I gave the tool some fake citations that I made up. I based the fake citations on correct citations to real works, but changed some of the information such as title or author. In many cases, the tool found the real work that I had based the fake citation on, and showed me a corrected citation that reversed the errors I had inserted. 

But if I made the fake citation different enough from a real one, such as by changing the author and title and leaving out the DOI number, then instead of returning a corrected version the tool labeled the citation as “hallucinated”; that is, the kind of reference to a nonexistent work that an Artificial Intelligence program might make up. Kleiman notes that there is a grey area between a citation that is close enough to matching a real work that is should be corrected, and one that is different enough from any real work that it should be flagged as a hallucination. https://jenkleiman.com/reference-checker/ 

The tool uses an Artificial Intelligence program, which is called Gemini and operated by Google. As Kleiman explains, the tool:

“…uses Gemini 3.0 Pro with search grounding. Instead of just asking ‘is this reference real?’, it performs web searches to verify each citation exists. This mimics how humans do it- we google the paper title, check authors, verify the journal and year, maybe search multiple times to find all the details. The AI does exactly that, but systematically for every reference…search grounding nearly eliminates AI hallucinations. The AI isn’t relying on training data, just what it found on the web.” https://jenkleiman.com/reference-checker/ 

Kleiman does not charge to use her tool, but you need a Gemini API key for Google to run your queries from the tool. The tool’s website includes a link and instructions to get an API key, which is not complicated. You need to give Google a payment method such as a credit card, but when I signed up the API key came with $300 of free usage, so I have not paid anything so far.

As powerful as the tool is, it does not address another problem with references that I have seen as an editor and peer reviewer – citations that point to a work that is real but does not substantiate the assertion that the manuscript cites it for. This still requires human judgement.

Jennifer Kleiman’s tool for verifying and correcting references is very impressive, and I think it should be a routine part of the workflow for evaluating manuscripts that are submitted to journals.

Journals’ Strategies to Engage Practitioners

Peer reviewed journals are making some efforts to increase the involvement of practitioners, adjunct faculty, and students. I described the strategies they are using in my presentation at the American Society for Public Administration conference on March 29. The slides are here: https://publishpeerreview.org/wp-content/uploads/2025/03/practitioners-in-peer-reviewed-journals-for-aspa-conference-2025.pdf

Why should practitioners publish in peer reviewed journals?

If you have some knowledge to share, then why publish it in a peer reviewed journal rather than the quicker and easier route of just posting it to a blog? Dr. Annette N. Brown answered this in her excellent article, “Why should practitioners publish their research in journals?”

Brown’s article became a real life example of one advantage of journals. She published it as a blog post in 2017, and it is no longer available on the blog or anyplace else I can find on the internet. This is despite the fact that it is a valuable article and has even been cited in peer reviewed literature. By contrast, articles in journals are preserved permanently in the redundant collections of publishers, libraries and archives. As Brown wrote, “If the question is salient enough to do research, and the research findings are credible enough to disseminate publicly, then shouldn’t we make sure they enter the permanent, searchable, public record?” Since Brown’s article isn’t in that permanent public record, you can read my personal marked-up copy here: https://publishpeerreview.org/wp-content/uploads/2025/03/2017_why-should-practitioners-publish-their-research-in-journals.pdf 

Brown pointed out two other advantages of publishing in peer reviewed journals. First, it increases the credibility of the research. As she wrote, “Submission to a peer-reviewed journal is an important signal, both that the researcher feels the work is of a standard to be published in a journal and that she is willing to undergo rigorous external peer review.”

Second, publishing in a peer reviewed journal increases the credibility of the practitioner. Brown cited her own experience, “Over my career I have recruited for hundreds of short-term and long-term positions for USAID and other funder proposals and projects, and while a record of journal publication is rarely a necessary credential, it is often a favored credential.”