P-Value vs Significance Level: What’s the Difference and Why It Matters
Two of the most commonly confused terms in hypothesis testing are the p-value and the significance level (α). They’re related — you compare one to the other to make a decision — but they represent fundamentally different things. This article clarifies exactly what each one means and how they work together.
The Short Answer
P-Value
Calculated from your data after running the test. It’s the probability of getting a result as extreme as yours if H₀ were true. It changes with every new dataset.
Significance Level (α)
Set by the researcher before the test runs. It’s the threshold below which you’ll reject H₀. It doesn’t change — it’s a fixed decision rule, not a result.
The decision rule is simple: if p < α, reject H₀. If p ≥ α, fail to reject H₀.
What Is the P-Value?
The p-value is the probability of observing a test result at least as extreme as the one you calculated, assuming the null hypothesis is true. It’s a property of the data — it comes out of the calculation and varies from test to test.
A small p-value means the data would be unlikely if H₀ were true. A large p-value means the data is plausible under H₀ — there’s no strong reason to reject it.
You test whether a coin is fair by flipping it 100 times and getting 62 heads. The null hypothesis is that p = 0.5. After running a z-test, you get p = 0.017.
This means: if the coin were truly fair, there would be only a 1.7% chance of getting 62 or more heads in 100 flips just by chance. That’s a surprising result under H₀.
What Is the Significance Level (α)?
The significance level is a threshold you choose before seeing the data. It represents the maximum probability of rejecting H₀ when it’s actually true — called the Type I error rate. The most common choice is α = 0.05 (5%), meaning you’re willing to be wrong 5% of the time when H₀ is true.
α is not calculated — it’s a decision. You set it in advance based on the consequences of making a Type I error. If falsely rejecting H₀ would be costly or dangerous, use a smaller α (0.01 or 0.001).
How They Work Together
Once you have both values, the decision is mechanical:
| Situation | Decision | Meaning |
|---|---|---|
| p < α (e.g., p = 0.02 < α = 0.05) | Reject H₀ | Statistically significant result |
| p ≥ α (e.g., p = 0.12 ≥ α = 0.05) | Fail to reject H₀ | Not statistically significant |
| p = 0.049, α = 0.05 | Reject H₀ | Barely significant — report exact p |
| p = 0.051, α = 0.05 | Fail to reject H₀ | Barely not significant — same caution applies |
Common Misconceptions
“p = 0.03 means there’s a 3% chance H₀ is true”
This is the most widespread misinterpretation. The p-value is not the probability that H₀ is true. It’s the probability of the data given that H₀ is true — a subtle but crucial distinction. To find the probability that H₀ is true, you would need Bayesian analysis, not a frequentist p-value.
“α = 0.05 is the correct threshold”
α = 0.05 is a convention, not a law. It was popularized by Ronald Fisher in the early 20th century and has stuck — but it’s arbitrary. Different fields use different thresholds. What matters is that you set α before seeing the data, not after.
“p > 0.05 means the effect doesn’t exist”
Failing to reject H₀ means the data didn’t provide sufficient evidence against it. The effect may still exist — perhaps the sample size was too small to detect it, or the variability was too high. Absence of evidence is not evidence of absence.
Choosing the Right Significance Level
- α = 0.05: Standard in social science, psychology, education research
- α = 0.01: Used when Type I errors are more costly — medical trials, safety testing
- α = 0.001: High-stakes decisions, preliminary screening for genetic association studies
- α = 0.1: Sometimes used in exploratory research when missing a real effect is the bigger risk
Practical Advice for Reporting
Modern statistical reporting conventions recommend:
- Always report the exact p-value (e.g., p = .023, not just p < .05)
- Report the test statistic and degrees of freedom alongside the p-value
- Report effect size (Cohen’s d, η², R²) so readers can judge practical significance
- Never report p = .000 — write p < .001
- State the α level you used and that it was set before data collection
See p-values calculated step by step
Run any hypothesis test and watch the full calculation with p-value and critical value shown.
Open the Hypothesis Testing Solver →Summary
The p-value comes from your data — it measures how surprising the result is under H₀. The significance level α is chosen by you before the test — it sets the decision threshold. You reject H₀ when p < α. Neither value alone tells you whether an effect is real, large, or important — they only tell you whether the data crosses your pre-set evidence threshold.