All tools › From a sample
A/B test significance
Whether the difference between two variants is more than noise.
It runs entirely in this browser tab: what you type is never uploaded, and the tool makes no network request while it works. Open your browser's network panel and try it — nothing leaves the page. There is no sign-up and no account.
Settings
- Confidence
- Ninety-five per cent is the convention and is not a fact about anything. Choices: 95 per cent, 90 per cent, 99 per cent.
- Tails
- Two-tailed asks whether they differ; one-tailed asks whether one is larger, and needs deciding beforehand. Choices: Two-tailed, One-tailed.
- Correct for the count
- More branches mean more chances of a false find, so the threshold is divided by their number.
How to use it
- Paste your text into the box above, or open a file.
- Adjust the settings beside it until the result is what you wanted.
- Copy the result, or save it as a file. The result updates as you type.
Questions
- Why does this disagree with my spreadsheet?
- Usually because one of you is using a different definition, and the tool says which one it used. A percentile has six in ordinary use, a standard deviation has two divisors, and a histogram's class count is a rule with a name. Where the choice moves the answer it is a setting here.
- Does a p-value below 0.05 mean the result is true?
- No. A p-value is the probability of seeing data at least this extreme if there were no real difference. It is not the probability that there is none, and a difference can be distinguishable from noise and far too small to matter.
- How exact are the distributions?
- They are series and continued fractions stopped when a term stops changing the answer, and searches bisected to a stated tolerance — right to more places than anybody reports a p-value to. Where one does not settle, the result says so rather than showing where it stopped as though it were the answer.
- Is what I type uploaded anywhere?
- No. This page carries the whole tool as WebAssembly and runs it on your own device. Nothing you type is sent anywhere, and there is no network request to make while it works.
Other tools for from a sample
All twenty-eight tools