TL;DR: Before accepting a prediction, break it into three parts: claim, reasoning, and evidence, then ask five questions: who says it, what exactly is said, based on what, are there counterexamples, and does the evidence truly support the conclusion. Common errors include confirmation bias, sunk cost fallacy, ownership bias, and the illusion of a winning streak, and AI models can also fall into them because they learn from human data. At a deeper level, the boundary between science and pseudoscience is also where schools of thought disagree, so when a prediction is labeled scientific or pseudoscientific, we should ask which standard is being used.
Critical thinking before a prediction: five questions to ask and the most common reasoning errors
Why do we need a procedure instead of relying on intuition alone?
Intuition works well in familiar environments, but it is weak when facing new information, ambiguous data or a smoothly delivered prediction. A slower procedure helps separate the feeling of right and wrong from the actual structure of an argument. In critical thinking, the smallest unit is not an opinion but an argument, made up of three components: a claim, reasoning and evidence. Once we can see these three components, we have somewhere to place our questions.
What are the five questions to ask before a prediction?
Question 1: Who is speaking, and what do they stand to gain?
The source does not determine truth, but it determines how much caution is needed. A prediction from someone with expertise in the field, with public data and no significant conflict of interest will be more credible than a prediction from a source that cannot be verified. This question is not meant to dismiss, but to allocate attention.
Question 2: What exactly is the claim?
Many tangled debates happen simply because the two sides are talking about two different claims. Write the claim as a sentence that can be tested: who, does what, under what conditions, to what degree. The words "always", "certainly", "everyone" often hide the difference between a statistical tendency and an absolute law.
Question 3: Where are the reasoning and the evidence?
Reasoning is the bridge between evidence and conclusion. Evidence is data, observations, experiments or texts that can be checked. A prediction that has only a conclusion and no bridge is not yet an argument. A prediction with a bridge that does not hold up is also not enough.
Question 4: Is there a counterexample or another explanation?
This is the strongest question. If we can find a case with the same conditions but the opposite result, the universal claim is refuted. If there is another explanation that also fits the data, we cannot choose the first explanation just because it is more comfortable.
Question 5: Does this evidence actually support the conclusion?
A percentage can be misread if the denominator is missing. A personal story can be very emotionally persuasive but not representative of the whole. This question forces us to re-examine the entire path from data to conclusion.
Quick reference table of the five questions
| Question | Aim | Sign that you should stop |
|---|---|---|
| Who is speaking? | Assess the source and their interests | Anonymous, unverifiable source |
| What exactly is the claim? | Clarify scope and degree | Many absolute words, unclear conditions |
| Where are the reasoning and evidence? | Separate conclusion from data | Only a conclusion, no bridge |
| Is there a counterexample? | Test universality | Ignoring contrary cases |
| Does the evidence support the conclusion? | Check the logical path | Jumping from correlation to causation |
What are the most common reasoning errors?
What is confirmation bias?
We tend to seek out, remember and believe information that matches what we already think, while ignoring contrary information. This is the foundational error, because it corrupts all four of the remaining questions. The practical countermeasure is to actively look for contrary evidence before concluding.
Raymond S. In his 1998 review article in Review of General Psychology, Nickerson described confirmation bias as the tendency to seek, interpret and remember evidence in ways that fit existing beliefs, and he regarded it as one of the most common biases in human reasoning. He gathered many forms of its expression, from the laboratory to everyday life, including science, forensics and superstition.
What is the sunk cost fallacy?
We continue a choice only because we have already put time, money or effort into it, even when new data shows we should stop. What has already been spent is an unrecoverable cost, so it is not a valid reason to continue.
What is the endowment effect?
We value what we already have more highly than what we do not have, even when the two are equivalent. This makes changing our mind harder than it needs to be.
What is the hot-hand illusion?
We believe that a random event will repeat in a streak, for example believing that someone who is "on form" will keep succeeding. For independent events, this belief has no basis in probability.
What is ignoring the base rate?
We judge a specific case while forgetting the background rate of the phenomenon in the whole population. This is a common error when reading test results, news or forecasts.
What are the straw man fallacy and the tu quoque fallacy?
The straw man fallacy is attacking a distorted version of the opponent's argument. The tu quoque fallacy is replying by pointing out that the opponent has also been wrong, instead of answering the substance.
What cognitive vices do people who believe in pseudoscience often have?
Massimo Pigliucci in his article Pseudoscience and the Demarcation Problem on the Internet Encyclopedia of Philosophy notes that the problem of demarcating science from pseudoscience lies not only on the theoretical side, but also on the side of the believer. He discusses the epistemic virtues and vices of the believer in pseudoscience, that is, the habits of thought that make a person prone to accepting unfounded claims. This perspective complements the five questions above: when examining ourselves, we consider not only the claim but also our own cognitive habits.
How do pseudoscience and science denial differ?
Sven Ove Hansson in his article Science and Pseudo-Science on the Stanford Encyclopedia of Philosophy (major 2025 revision) presents an overview of the demarcation problem from Popper through Kuhn, Lakatos, Thagard to Laudan, and also distinguishes pseudoscience from science denial. In the Internet Encyclopedia of Philosophy article, Pigliucci also recounts that Hansson later proposed treating science denial as a form of pseudoscience, with the other form being the promotion of pseudo-doctrines. Along these lines, a statement is considered pseudoscientific when it falls within the broad domain of science, lacks reliability to the point of being unbelievable, and belongs to a doctrine whose own supporters deliberately try to create the impression that it is the most trustworthy knowledge on that issue.
Why is the demarcation problem still unresolved?
Pigliucci recounts that after the volume The Philosophy of Pseudoscience appeared, the number of articles on the demarcation problem increased significantly, and the philosophy of science community regards this as a revival after the period when Laudan doubted the problem itself. In the Stanford Encyclopedia of Philosophy article, Hansson also shows that different demarcation criteria, from Popper's falsifiability criterion to the approaches of Kuhn, Lakatos and Thagard, do not entirely coincide. Therefore, when a statement is called pseudoscientific, we should see it as an assessment with criteria attached, not a simple label.
Where do the schools disagree?
There is no single list of reasoning errors agreed upon by every school. Some texts group confirmation bias under cognitive errors, others group it under reasoning errors. Some fold the hot-hand illusion into the category of probability biases, others treat it as a reasoning error. This article follows the convention common in introductory critical thinking textbooks: separating cognitive errors from reasoning errors, while acknowledging that they often go together.
At a deeper level, the very boundary between science and pseudoscience is also a place where schools disagree. Hansson in the Stanford Encyclopedia of Philosophy article shows that different demarcation criteria yield different conclusions; Pigliucci in the Internet Encyclopedia of Philosophy article notes that after Laudan, this problem was once considered no longer worth discussing, then revived into a vibrant field of research. Therefore, when reading a prediction labelled "scientific" or "pseudoscientific", we should ask which criterion is being used.
Can AI avoid these errors?
Not entirely. A study published in the journal Manufacturing & Service Operations Management tested GPT-3.5 and GPT-4 on 18 familiar cognitive biases. The results showed that in nearly half of the situations, ChatGPT behaved like a human when facing emotionally charged choices. GPT-4 was superior on logic problems, but showed confirmation bias more clearly and was more prone to the hot-hand illusion than GPT-3.5. Conversely, AI avoided some errors such as ignoring the base rate and the sunk cost fallacy. The cause lies in the training data and the mechanism of fine-tuning on human feedback. Therefore, using AI for tasks with a clear procedure is still safer than using AI for strategic or emotional judgement.
