Free statistical significance calculator
Compare two groups from an A/B test or a survey. See the rates, the lift, the p-value and each calculation step. No signup. All processing is in your browser.
Compare two variants, A and B. A conversion is a visitor who did the action that you measure.
A two-sided test examines a difference in both directions. A one-sided test examines one direction only.
The threshold for the p-value is 1 minus the confidence level. At 95%, the threshold is 0.05.
Correct this input, then calculate again:
Your result
- Conversion rate, variant A
- Conversion rate, variant B
- Difference
- Lift
- z-score
- p-value
Calculation steps
What is statistical significance?
The p-value is the probability of a result at least as extreme as yours, if no real difference exists between the groups. If the p-value is less than your threshold, the result is statistically significant. For example, 100 of 1,000 people in group A and 130 of 1,000 people in group B give a two-sided p-value of 0.0355. This result is significant at 95% confidence, because 0.0355 is less than 0.05.
This calculator does a two-proportion z-test. It shows the rates, the lift, the p-value, the result at the confidence level that you select and each calculation step. Your numbers stay in your browser.
A/B test labels and survey labels
The test needs two counts for each group: a total and the number of successes. For an A/B test, these are visitors and conversions. For a survey, these are responses and the responses with a selected answer. For example, you can show one ad concept to one group of people and a different ad concept to a different group. Then you compare the number of people in each group who select “Yes” for a question.
The labels do not change the calculation. The same numbers give the same result with both sets of labels.
The test assumes independent random samples. Each person must be in one group only. Do not use this calculator to compare two answers from the same people.
This calculator compares two groups on one answer. To compare more than two groups, or all the answers of a question, use the chi-square calculator for survey crosstabs.
Method: the two-proportion z-test
The calculator uses a two-proportion z-test with a pooled standard error. It does these calculations:
Rate = successes ÷ total
Lift = (rate B − rate A) ÷ rate A
Pooled rate = (successes A + successes B) ÷ (total A + total B)
Standard error = √(pooled rate × (1 − pooled rate) × (1 ÷ total A + 1 ÷ total B))
z = (rate B − rate A) ÷ standard error
Two-sided p-value = 2 × (1 − Φ(|z|))
One-sided p-value (B is higher than A) = 1 − Φ(z)
Φ(z) is the part of the standard normal distribution that is below z.
The threshold is 1 minus the confidence level. The result is significant if the p-value is less than the threshold.
| Confidence level | p-value threshold |
|---|---|
| 90% | 0.10 |
| 95% | 0.05 |
| 99% | 0.01 |
Worked example
Group A has 100 successes of 1,000. Group B has 130 successes of 1,000. The test is two-sided.
- Rates: 100 ÷ 1,000 = 0.10 and 130 ÷ 1,000 = 0.13
- Difference: 0.13 − 0.10 = 0.03. Lift: 0.03 ÷ 0.10 = 30%
- Pooled rate: (100 + 130) ÷ (1,000 + 1,000) = 0.115
- Standard error: √(0.115 × 0.885 × (1 ÷ 1,000 + 1 ÷ 1,000)) = 0.014267
- z-score: 0.03 ÷ 0.014267 = 2.103
- Two-sided p-value: 2 × (1 − Φ(2.103)) = 0.0355
- At 95% confidence, 0.0355 is less than 0.05, so the result is significant. At 99% confidence, 0.0355 is not less than 0.01, so the result is not significant.
How to use the calculator
- Select the A/B test labels or the survey labels.
- Enter the total and the successes for A and for B.
- Select the test type and the confidence level.
- Select Calculate significance. Read the result and the calculation steps. If an entry is not valid, a message tells you what to correct.
Statistical significance FAQ
What does statistically significant mean?
It means that the p-value is less than the threshold for your confidence level. If no real difference exists, a result as extreme as yours has a low probability.
What does the p-value not tell me?
The p-value does not tell you the probability that your hypothesis is true. It also does not tell you the size of an effect or the importance of a result. The American Statistical Association also says that business decisions should not be based only on whether a p-value passes a threshold.
Should I use a one-sided or a two-sided test?
A two-sided test examines a difference in both directions. A one-sided test examines one direction and does not examine the other direction. Do not change to a one-sided test only to get a significant result, or after a two-sided test does not give a significant result.
Which confidence level should I use?
This page does not give one correct level, because the correct level depends on your decision. A higher level needs a lower p-value. For example, a p-value of 0.0355 is significant at 90% and 95%, but it is not significant at 99%.
Can I stop an A/B test when the result becomes significant?
No. Repeated checks during a test increase the rate of false positives. Decide on the sample size before the test, and calculate the result when the test is complete.
Can I use this calculator for survey results?
Yes, if you compare two separate groups of people. Select the survey labels, then enter the responses and the responses with the selected answer for each group. Do not use it to compare two answers from the same people, because the test assumes independent samples.
What if my numbers are small?
The z-test needs large samples. One rule is that each group has more than 10 successes and more than 10 other results. If your numbers do not obey this rule, the calculator shows a warning. For small samples, you can use Fisher's exact test.
Is my data sent to a server?
No. The calculation runs in your browser, and this page does not send or save your numbers. The site uses Plausible analytics to count page visits. Plausible does not receive what you enter.
Do I need an account?
No. The calculator is free and needs no signup.
Related pages
Sources
- Comparing two proportions, NIST/SEMATECH e-Handbook of Statistical Methods.
- Two-proportion Z-test, Wikipedia.
- P-value, Wikipedia.
- Statistical significance, Wikipedia.
- Statistical calculations in A/Bn tests (lift), Adobe Target.
- Statement on p-values, American Statistical Association.
- One-tailed and two-tailed tests, UCLA.
- How Not To Run an A/B Test, Evan Miller.