Statistics · Hypothesis Testing · Year 1

Setting Up a Hypothesis Test

A sample result is never exactly what theory predicts, even when theory is correct. A hypothesis test is a formal way of asking: is this difference real, or just noise?

10 min read 3 worked examples

Null and alternative hypotheses

Every hypothesis test starts with two competing claims about a population parameter — here, a binomial probability $p$.

Null hypothesis, $H_0$: the "status quo" claim, usually a specific stated value: $H_0: p=p_0$.

Alternative hypothesis, $H_1$: what you'd conclude if the evidence contradicts $H_0$ — either $p\neq p_0$ (two-tailed), $p>p_0$ (one-tailed), or $p<p_0$ (one-tailed).

A test never proves $H_0$ true. It either finds enough evidence to reject $H_0$ in favour of $H_1$, or it doesn't — in which case $H_0$ is simply not rejected, not confirmed.

One-tailed vs two-tailed — read the question's wording

A one-tailed test is used when you're only interested in a change in one specific direction ("has the proportion increased?"). A two-tailed test is used when you're interested in any change ("has the proportion changed at all?"). The wording of the question — "increased", "decreased", "changed", "is different from" — decides which to use, and must be decided before looking at the data.

Significance level and critical region

The significance level, usually $5\%$ or $1\%$, is the threshold probability used to judge whether a result is unusual enough to doubt $H_0$. The critical region is the set of values of the test statistic so extreme that, if observed, $H_0$ would be rejected — chosen so that the probability of the test statistic falling in it (assuming $H_0$ is true) is as close as possible to the significance level, without exceeding it.

Image pending

hypothesis-test-critical-region-diagram.png

The probability distribution of the test statistic under $H_0$, with the critical region shaded in the tail — its total probability kept as close as possible to the significance level without exceeding it.

Strategy — the standard test procedure

1. State $H_0$ and $H_1$ in terms of $p$.
2. Identify the test statistic and the distribution it follows under $H_0$.
3. Find the critical region at the given significance level.
4. Compare the observed value against the critical region.
5. State a conclusion in the context of the original question.

Worked example 1

Stating hypotheses

A company claims that 60% of its customers are satisfied. A researcher suspects the true proportion is lower, and plans to survey a sample of customers to test this. State suitable null and alternative hypotheses.

MethodIdentify the claimed value and the direction of suspicion

Let $p$ be the true proportion of satisfied customers.

Define the parameter in context.

$H_0: p=0.6$ — the company's claimed value.

The status-quo, stated value.

$H_1: p<0.6$ — the researcher only suspects the proportion is lower, so this is one-tailed.

"Lower" specifies a single direction.

▪ stated

Examiner's eye

Always define $p$ in words first ("the true proportion of satisfied customers") before writing the hypotheses symbolically — a bare "$H_0: p=0.6$" with no definition of $p$ often loses marks even if the numbers are correct.

Worked example 2

One-tailed or two-tailed?

For each scenario, state whether a one-tailed or two-tailed test is appropriate: (a) "Test whether a coin is biased." (b) "Test whether a new drug increases recovery rate compared to the standard treatment."

MethodCheck whether the wording implies a specific direction

(a) Two-tailed. "Biased" could mean the coin favours heads or tails — no direction is specified.

Interested in any deviation from fair.

(b) One-tailed. "Increases" specifies a single direction of interest.

Only concerned with improvement, not any change.

▪ decided

The key insight

This decision must be made from the wording of the research question before any data is collected or examined — choosing the test direction based on which way the data happens to point is a serious methodological error, since it effectively cheats the significance level.

Worked example 3

Identifying the test statistic's distribution · Challenge

A machine is supposed to produce 8% defective items. To test this, a random sample of 50 items is inspected, and $X$ is the number found defective. Stating any assumptions, give the distribution of $X$ under $H_0$.

MethodMatch the scenario to the binomial conditions

A fixed number of items ($n=50$), each either defective or not (two outcomes), with a claimed constant probability under $H_0$ ($p=0.08$).

Matches three of the four binomial conditions.

Assuming each item's defect status is independent of the others (a reasonable assumption if the sample is small relative to total production), the fourth condition holds too.

Independence needs to be assumed/justified explicitly.

Under $H_0$: $X \sim B(50, 0.08)$.

The distribution the test statistic follows if H₀ is true.

▪ identified

Look deeper — the whole test hinges on this distribution

Every later step of a hypothesis test — finding the critical region, computing a p-value — depends entirely on knowing the test statistic's distribution under the assumption that $H_0$ is true. Getting this distribution right, including checking the binomial conditions, is the foundation the rest of the test is built on.

Common mistake

$H_0$ always states a specific, testable value ($p=p_0$) — never a vague claim or an inequality. Inequalities belong only in $H_1$. And "not rejecting $H_0$" is not the same as "proving $H_0$ true" — a test can only ever find evidence against the null hypothesis, never evidence definitively for it.

Check yourself

A website claims 25% of visitors make a purchase. A manager wants to test whether this proportion has changed, in either direction. State $H_0$ and $H_1$, and say whether the test is one-tailed or two-tailed.
(Hint: "changed" — has a direction been specified?)

Method

Let $p$ be the true proportion of visitors who make a purchase.

$H_0: p=0.25$, $H_1: p\neq0.25$.

No direction specified, so ≠.

This is a two-tailed test, since "changed" allows either an increase or a decrease.

▪ solved

← Previous

Cumulative Probabilities

Critical Values