Student's T-Test

Named after William Sealy Gosset, who published under the name "Student" to respect his employer's privacy.

Student's t-test helps us compare two sets of data to see if the difference between their means is just random chance.

There are two main types:

Examples:

• Independent: Comparing recovery rates of patients given a drug vs those on a placebo
• Independent: Comparing test scores of a class taught using Method A vs a class taught using Method B
• Paired: Comparing people's weight before and after a weight-loss program
• Paired: Comparing employee productivity before and after a training program

The data should be normally distributed, and isn't reliable if the sample sizes are too small or the variances are very different.

Note: you might also like to investigate the Chi-Square Test.

Independent Samples T-Test

We typically use this formula (but there are other formulas):

Where:

• x₁ and x₂ are sample means
• s₁² and s₂² are sample variances
• n₁ and n₂ are sample sizes

Note: we use s² for variance, because the variance is the square of the standard deviation s.

To do the t-test:

• Calculate mean, variance and samples size for each set of sample data
• Use the formula above to calculate "t"
• Then we look up a special table to discover how significant that "t" value is

In effect the t-value is a measure of how much the means of the two groups differ from each other compared to the variability of the data.

A higher t-value suggests a more significant difference between the groups (ie less likely due to random chance).

Let's try an example:

Example: Battery Life

This is an experiment you can do yourself (at the cost of 20 batteries).

Two batches of identical batteries are tested:

• One group of 10 were tested in a cool environment
• Another group of 10 were tested in a hot environment

The batteries power identical devices until they run out of charge, with these results (in hours):

• Cool Group: 85, 90, 76, 88, 93, 87, 91, 89, 95, 85
• Hot Group: 78, 82, 79, 88, 91, 85, 90, 87, 84, 86

Let's calculate the mean for each:

• Cool Group Mean = 85+90+76+88+93+87+91+89+95+8510 = 87.9 hours
• Hot Group Mean = 78+82+79+88+91+85+90+87+84+8610 = 85 hours

The Cool Group lasted, on average, about 3 hours longer.

But is this a significant result, or just random in nature?

Now, it's time to calculate the t-statistic.

Our next step is to calculate Variances.

Variance is a measure of how much the data points spread out from the mean. We calculate it by taking the average of the squared differences between each data point, as shown here:

For the Cool Group:

• n = 10
• Mean = 87.9
• Variance = (85-87.9)²+(90-87.9)²+(76-87.9)²+(88-87.9)²+(93-87.9)²+(87-87.9)²+(91-87.9)²+(89-87.9)²+(95-87.9)²+(85-87.9)²9 = 27.88

For the Hot Group:

• n = 10
• Mean = 85
• Variance = (78-85)²+(82-85)²+(79-85)²+(88-85)²+(91-85)²+(85-85)²+(90-85)²+(87-85)²+(84-85)²+(86-85)²9 = 18.89

Start with:

t = x₁ − x₂√(s₁²/n₁ + s₂²/n₂)

Put in our values:

87.9 − 85√(27.88/10 + 18.89/10)

Calculate:

t = 1.341

Now let's compare our t-value to the "critical t-value" from the Student's t-test Table at a chosen significance level, typically 0.05 for a 95% confidence level with degrees of freedom df = n₁ + n₂ − 2 = 18, and using the two-tails value, as the change in battery life could theoretically go either way.

The table shows us that for 0.05 and df=18 we get a "critical t-value" of 2.101

Our actual t value is 1.341, and being less than 2.101 our result does not pass the 95% confidence level.

So there's not enough evidence to say there's a significant difference between the cool and hot groups.

Hypothesis

The t-test is often done more formally using the idea of a hypothesis and null hypothesis.

But we usually take the opposing point of view:

The null hypothesis is the baseline assumption for statistical analysis

Rejecting the null hypothesis gives credibility to (but doesn't prove) our original hypothesis. In other words if we can show the skeptic is likely wrong, we can suggest our original idea might be right.

Hypothesis Steps

1. Formulate the null hypothesis (H₀): There's no significant difference between the two groups
2. Formulate the alternative hypothesis (H_a): There is a significant difference between the two groups
3. Choose a significance level (commonly 0.05), which is the probability of rejecting the null hypothesis when it is actually true
4. Collect the data and calculate the t-statistic based on the two sample means, standard deviations, and sample sizes
5. Compare the calculated t-statistic to the critical value from the t-distribution table to determine whether to accept or reject the null hypothesis

Let's do our original example that way!

Example: Battery Life

Null Hypothesis (H₀): The difference between a cool and hot environment has no effect on a battery's running time

Alternative Hypothesis (H_a): The difference between a cool and hot environment will affect a battery's running time

Using the values calculated earlier:

From before:

87.9 − 85√(27.88/10 + 18.89/10)

Calculate:

t = 1.341

From the Student's t-test Table for 0.05 (which is a 95% confidence level) and df=18 we get a "critical t-value" of 2.101

Since our calculated t-value of 1.341 is less than the critical t-value of 2.101, we don't have enough evidence to reject the null hypothesis.

We can't reject the null hypothesis

This means that based on our sample, we can't confidently say a statistically significant difference exists.

In everyday terms, it's as if we're saying, "Based on what we've seen, we can't conclude that the change is anything more than just random chance."

Paired Samples T-Test

A Paired samples t-test, also known as a dependent samples t-test, is a statistical test that's used to compare two means (averages) when the data is paired in some way. Such as a before and after trial.

So the same subjects are involved in both sets of data.

Step-by-Step Calculation

To do the Paired Samples t-test:

• Calculate the difference (d) for each pair of scores
• Calculate the mean difference (d)
• Calculate the variance of the differences (var)
• Calculate the standard deviation of the differences (s)
• Calculate the t-statistic
• Determine the degrees of freedom (df)
• Compare the calculated t-statistic to the critical value from the t-distribution table to determine whether to accept or reject the null hypothesis

Now let's compare our t value to the critical t-value in the table.

• The degrees of freedom (df) for a paired t-test is n − 1: df = 7−1 = 6
• For a two-tailed test with df=6 at the 0.05 level, we find the critical t-value is 2.447
• Our value of 2.78 is above that, so the Null Hypothesis fails

It is unlikely that the Null Hypothesis is true (less than 5% chance), so we have good reason to believe that the alternative hypothesis "The special math program does affect test scores" is true

One-Tail and Two-Tail T-Tests

• One-Tail T-Test: Tests for a significant effect in one specific direction (either greater than or less than)
• Two-Tail T-Test: Tests for a significant effect in both directions (either greater or less than), without committing to one specific direction before the test

Choose a one-tail test if you have a specific hypothesis about the direction of the effect. Use a two-tail test if you're interested in detecting any significant difference, regardless of direction.

Single Set of Data

For one set of data we can compare our sample mean (x) to a known or hypothesized population mean (μ):