A study of 636 references found fabricated papers and errors in genuine citations. A convincing reference list still needs to be checked.
A reference can look entirely plausible: several authors, a technical title, a journal name, and a publication year. None of that guarantees that the paper exists. An AI system can generate the form of a citation without retrieving an actual publication. This creates extra work for anyone using AI to prepare an article, assignment, or report. You may spend time searching for a nonexistent paper, or publish a claim that appears to have support when it does not. Even finding a real paper with a similar title is not enough; it may have studied a different question.
Researchers who examined 636 AI-generated references found both fabricated sources and errors in citations to real work. The rates came from particular systems tested in 2023, so they should not be treated as the error rate of every current tool. The study still establishes a basic editorial requirement: open the source and check what it supports before including it in your writing.
William Walters and Esther Wilder tested GPT-3.5 and GPT-4. They asked for short reviews of research on 42 topics. They checked 636 references from 84 generated texts. In that set, 55% of the GPT-3.5 references and 18% of the GPT-4 references were fabricated. Some references to real works also contained errors. Their study appeared in Scientific Reports in 2023. These rates apply to those tools and that test. They are not current failure rates for every AI search tool. Read the original paper. The useful lesson is a rule for checking. Find the paper. Read it. Only then can it back your claim.
First, it must identify a real work. Second, that work must support what you say. Passing the first test does not pass the second. Imagine a real paper about adults learning a list of words. An AI answer cites it to say a phone game helps children with math. The paper exists. The claim still has no support from it. Adding references after choosing the argument can encourage a writer to select papers for their apparent relevance rather than their findings. That reverses the research process: the claim should follow from the source, including its limitations.
Read the methods and results before deciding how strongly to phrase the claim. If the source tested a narrow task, keep that task in the sentence. A reference should let a reader verify what was written, not merely signal that research exists on the topic.
Search the exact title. Check the authors, year, and publisher. If there is a DOI, open it and make sure it leads to the same work. A DOI is a code used to find a published work. It is not a quality score. If the title is slightly wrong, the work may still be real. Search a distinctive phrase with an author's name, but do not repair a citation by picking a paper that merely sounds close.
When you cannot find the source, leave the claim unverified. Ask for a source you can inspect, or remove the claim. The fact that the AI repeats the title does not bring the paper into existence.
An abstract gives a short overview. It may be enough to see that the paper is about the wrong question. It is often not enough for a detailed claim about methods or numbers. Look at who took part, what happened, and what was measured. If you report a number, find it in the results and check what it compares. A 20% relative change is different from 20 extra people out of 100. Then read the limits. Do not turn a short task in a lab into a lifelong effect in the world. Do not turn a link between two things into proof that one caused the other.
You can write plainly without losing those distinctions. The study found this in these people is often all the care a sentence needs.
For each main claim, save three lines. What will you say? Which source backs it? What must you not overstate? For example, an invented note might say: Students recalled more words a week later. Source: the named memory experiment. Limit: the study did not test school grades. That note helps when you return to the draft days later. It also makes review easier. The next reader can see why you wrote that sentence. They can check the source too. If a paper is corrected, check whether the correction affects your claim. A changed author address and a changed result require very different edits.
Use AI to suggest search terms, explain unfamiliar terminology, or help organize notes from papers you have inspected. Do not treat a generated bibliography as completed research. Each important claim needs a source that exists, can be located, and actually addresses the claim. If you cannot find a paper, leave the statement unverified rather than replacing it with a vaguely related citation. If the paper exists but the result is weaker than the draft suggests, change the draft. The source should determine what you write; it should not be attached afterward to make an already chosen claim look respectable.
