12 Chapter 13: Independent Samples

We have seen how to compare a single mean against a given value and how to utilize difference scores to look for meaningful, consistent change via a single mean difference using a repeated measures design. Now, we will learn how to compare two separate means from separate groups that do not overlap to see if there is a difference between them. The process of testing hypotheses about two means is exactly the same as it is for testing hypotheses about a single mean, and the logical structure of the formulae is the same as well. However, we will be adding a few extra steps this time to account for the fact that our data are coming from different sources.

Difference of Means

Last chapter, we learned about mean differences, that is, the average value of difference scores. Those difference scores came from ONE group and TWO time points (or two perspectives). Now, we will deal with the difference of the means, that is, the average values of separate groups that are represented by separate descriptive statistics. This analysis involves TWO groups and ONE time point. As with all of our other tests as well, both of these analyses are concerned with a single variable.

It is very important to keep these two tests separate and understand the distinctions between them because they assess very different questions and require different approaches to the data. When in doubt, think about how the data were collected and where they came from. If they came from two time points with the same people (sometimes referred to as “longitudinal” data), you know you are working with repeated measures data (the measurement literally was repeated) and will use a paired/dependent samples t-test. If it came from a single time point that used separate groups, you need to look at the nature of those groups and if they are related. Can individuals in one group being meaningfully matched up with one and only one individual from the other group? For example, are they a romantic couple? If so, we call those data matched and we use a matched pairs/dependent samples t-test. However, if there’s no logical or meaningful way to link individuals across groups, or if there is no overlap between the groups, then we say the groups are independent and use the independent samples t-test, the subject of this chapter.

Research Questions about Independent Means

An independent samples t-test is also designed to compare populations. If we want to know if two populations differ and we do not know the mean of either population, we take a sample from each and then conduct an independent sample t-test. Many research ideas in the behavioral sciences and other areas of research are concerned with whether or not two means are the same or different. Logically, we can then say that these research questions are concerned with group mean differences. That is, on average, do we expect a person from Group A to be higher or lower on some variable that a person from Group B. In any time of research design looking at group mean differences, there are some key criteria we must consider: the groups must be mutually exclusive (i.e. you can only be part of one group at any given time) and the groups have to be measured on the same variable (i.e. you can’t compare personality in one group to reaction time in another group since those values would not be the same anyway).

Let’s look at one of the most common and logical examples: testing a new medication. When a new medication is developed, the researchers who created it need to demonstrate that it effectively treats the symptoms they are trying to alleviate. The simplest design that will answer this question involves two groups: one group that receives the new medication (the “treatment” group) and one group that receives a placebo (the “control” group). Participants are randomly assigned to one of the two groups (remember that random assignment is the hallmark of a true experiment), and the researchers test the symptoms in each person in each group after they received either the medication or the placebo. They then calculate the average symptoms in each group and compare them to see if the treatment group did better (i.e. had fewer or less severe symptoms) than the control group.

In this example, we had two groups: treatment and control. Membership in these two groups was mutually exclusive: each individual participant received either the experimental medication or the placebo. No one in the experiment received both, so there was no overlap between the two groups. Additionally, each group could be measured on the same variable: symptoms related to the disease or ailment being treated. Because each group was measured on the same variable, the average scores in each group could be meaningfully compared. If the treatment was ineffective, we would expect that the average symptoms of someone receiving the treatment would be the same as the average symptoms of someone receiving the placebo (i.e. there is no difference between the groups). However, if the treatment WAS effective, we would expect fewer symptoms from the treatment group, leading to a lower group average.Now let’s look at an example using groups that already exist. A common, and perhaps salient, question is how students feel about their job prospects after graduation. Suppose that we have narrowed our potential choice of college down to two universities and, in the course of trying to decide between the two, we come across a survey that has data from each university on how students at those universities feel about their future job prospects. As with our last example, we have two groups: University A and University B, and each participant is in only one of the two groups (assuming there are no transfer students who were somehow able to rate both universities). Because students at each university completed the same survey, they are measuring the same thing, so we can use a t-test to compare the average perceptions of students at each university to see if they are the same. If they are the same, then we should continue looking for other things about each university to help us decide on where to go. But, if they are different, we can use that information in favor of the university with higher job prospects.

As we can see, the grouping variable we use for an independent samples t-test can be a set of groups we create (as in the experimental medication example) or groups that already exist naturally (as in the university example). There are countless other examples of research questions relating to two group means, making the independent samples t-test one of the most widely used analyses around.

Setting up step 1

In chapter 12, it is the same participants tracked over 2 timepoints for the paired samples t-test.  This is an example of a repeated measures or within-group design.  This chapter focuses on two different samples compared, using independent samples t-test, known as a between-group design.  In setting up step 1, there will be a few variations to how to set up the null and alternative (H1 or HA) hypotheses.  You can see these are all ways to set up the two types of research hypotheses (three hypotheses for the 2 directional and 1 for the non-directional).  You can set it up by comparing both groups (first 2 columns) or examining differences (which is similar to how we set up the top of your t-formula).

Research Question	Hypotheses in 3 ways
Are male scores higher than female scores? (between-group design)	H0: μM – μF ≤ 0 H1: μM – μF > 0	in other words: H0: μM ≤ μF H1: μM > μF	in other words: H0: μM ≤ μF H1: μM > μF
Are male scores lower than female scores? (between-group design)	H0: μM – μF ≥ 0 H1: μM – μF < 0	in other words: H0: μM ≥ μF H1: μM < μF	in other words: H0: μM ≥ μF H1: μM < μF
Are male scores different from female scores? (between-group design)	H0: μM – μF= 0 H1: μM – μF ≠ 0	in other words: H0: μM = μF H1 μM ≠ μF	in other words: H0: μM = μF H1: μM ≠ μF
Do athlete performance improve after training? (within-group design)	H0: μD ≤ 0 H1: μD > 0	in other words: H0: μ1 ≤ μ2 H1: μ1 > μ2	in other words: H0: μ1 – μ2 ≤ 0 H1: μ1 – μ2 > 0
Do athlete reaction times decrease after training? (within-group design)	H0: μD ≥ 0 H1: μD < 0	in other words: H0: μ1 ≥ μ2 H1: μ1 < μ2	in other words: H0: μ1 – μ2 ≥ 0 H1: μ1 – μ2 < 0
Does training have an effect on athlete reaction times? (within-group design)	H0: μD = 0 H1: μD ≠ 0	in other words: H0: μ1 = μ2 H1 μ1 ≠ μ2	in other words: H0: μ1 – μ2= 0 H1: μ1 – μ2 ≠ 0

Table 1. Examples of hypotheses set up for 2 samples based on between-group and within-group design.

Hypotheses and Decision Criteria

The process of testing hypotheses using an independent samples t-test is the same as it was in the last three chapters, and it starts with stating our hypotheses and laying out the criteria we will use to test them.

Our null hypothesis for an independent samples t-test is the same as all others: there is no difference. The means of the two groups are the same under the null hypothesis, no matter how those groups were formed. Mathematically, this takes on two equivalent forms:

H₀: µ1 = µ2 or H₀: µ1 − µ2 = 0

Both of these formulations of the null hypothesis tell us exactly the same thing: that the numerical value of the means is the same in both groups. This is more clear in the first formulation, but the second formulation also makes sense (any number minus itself is always zero) and helps us out a little when we get to the math of the test statistic. Either one is acceptable and you only need to report one. The English interpretation of both of them is also the same: H₀: There is no difference between the means of the two groups.

Our alternative hypotheses are also unchanged: we simply replace the equal sign (=) with one of the three inequalities (>, <, ≠):

H_A: µ1 > µ2 or µ1 − µ2 > 0; H_A: Group 1 is more than/stronger than group 2.

H_A: µ1 < µ2 or µ1 − µ2 < 0; H_A: Group 1 is less than/weaker than group 2.

H_A: µ1 ≠ µ2 or µ1 − µ2 ≠ 0; H_A: There is no difference between the two groups.

Whichever formulation you chose for the null hypothesis should be the one that connects to the research or alternative hypothesis. Notice that we are now dealing with two means instead of just one, so it will be very important to keep track of which mean goes with which population and, by extension, which dataset and sample data. We use subscripts to differentiate between the populations, so make sure to keep track of which is which. If it is helpful, you can also use more descriptive subscripts.

To use the experimental medication example:

H₀: There is no difference between the means of the treatment (T) and control (C) groups.  H₀: µ_T = µ_C

H_A: There is a difference between the means of the treatment (T) and control (C) group. H_A: µ_T ≠ µ_C

Step 2

Once we have our hypotheses laid out, we can set our criteria to test them using the same three pieces of information as before: significance level (α), directionality (left, right, or two-tailed), and degrees of freedom.  We will use the same critical value t table, but a new degrees of freedom for the independent samples t-test.

This looks different than before, but it is just adding the individual degrees of freedom from each group (n – 1) together. Notice that the sample sizes, n, also get subscripts so we can tell them apart. For an independent samples t-test, it is often the case that our two groups will have slightly different sample sizes, either due to chance or some characteristic of the groups themselves. Generally, this is not as issue, so long as one group is not massively larger than the other group, and there are not large differences in the variance of each group (more on this later).

Independent Samples t-statistic

The test statistic for our independent samples t-test takes on the same logical structure and format as our other t-tests: our observed effect minus our null hypothesis value, all divided by the standard error:

Our standard error in the denominator (bottom of the formula) is denoting what it is the standard error of (derived from 2 samples). Because we are dealing with the difference between two separate means, rather than a single mean or single mean of difference scores, we put both means in the subscript.

Calculating our standard error, as we will see next, is where the biggest differences between this t-test and other t-tests appears. However, once we do calculate it and use it in our test statistic, everything else goes back to normal. Our decision criteria is still comparing our obtained test statistic to our critical value, and our interpretation based on whether or not we reject the null hypothesis is unchanged as well.

Standard Error and Pooled Variance

Recall that the standard error is the average distance between any given sample mean and the center of its corresponding sampling distribution, and it is a function of the standard deviation of the population (either given or estimated) and the sample size. This definition and interpretation hold true for our independent samples t-test as well, but because we are working with two samples drawn from two populations, we have to first combine their estimates of standard deviation – or, more accurately, their estimates of variance – into a single value that we can then use to calculate our standard error.

The combined estimate of variance using the information from each sample is called the pooled variance and is denoted Sp²; the subscript p serves as a reminder indicating that it is the pooled variance. The term “pooled variance” is a literal name because we are simply pooling or combining the information on variance – the Sum of Squares and Degrees of Freedom – from both of our samples into a single number. The result is a weighted average of the observed sample variances, the weight for each being determined by the sample size, and will always fall between the two observed variances. The computational formula for the pooled variance is:

Once we have our pooled variance calculated, we can drop it into the equation for our standard error:

Looking at that, we can now see that, once again, we are simply adding together two pieces of information: no new logic or interpretation required. Once the standard error is calculated, it goes in the denominator of our test statistic, as shown above and as was the case in all previous chapters. Thus, the only additional step to calculating an independent samples t-statistic is computing the pooled variance. Let’s see an example in action.

Effect Sizes and Confidence Intervals

We have seen in previous chapters that even a statistically significant effect needs to be interpreted along with an effect size to see if it is practically meaningful. We have also seen that our sample means, as a point estimate, are not perfect and would be better represented by a range of values that we call a confidence interval. As with all other topics, this is also true of our independent samples t-tests. Our effect size for the independent samples t-test is still Cohen’s d, and it is still just our observed effect divided by the standard deviation. Remember that standard deviation is just the square root of the variance, and because we work with pooled variance in our test statistic, we will use the square root of the pooled variance as our denominator in the formula for Cohen’s d. We also can still calculate r², the percentage of variance accounted from by the independent variable/treatment effect.

For our example above, M₁ = 24, M₂ = 16.5, s_p² = 144.48, we can calculate Cohen’s d to be:

d =24-16.50/√(144.48)=7.5/12.02 = 0.62

We interpret this using the same guidelines as before, so we would consider this a moderate or moderately large effect.

For our example above, t = 2.48 (thus t² = 2.48*2.48= 6.15), df = 62, we can calculate r² to be:

 r² = 6.15/(6.15+62) = 6.15/68.15 = 0.09

We interpret this using the same guidelines as before, so 9% of the variance in mood (our DV) is from type of movie (our IV).

Our confidence intervals also take on the same form and interpretation as they have in the past. The value we are interested in is the difference between the two means, so our point estimate is the value of one mean minus the other, or M₁ – M₂. Just like before, this is our observed effect and is the same value as the one we place in the numerator of our test statistic. We calculate this value then place the margin of error – still our critical value times our standard error – above and below it.

Because our hypothesis testing example used a one-tailed test, it would be inappropriate to calculate a confidence interval on those data (remember that we can only calculate a confidence interval for a two-tailed test because the interval extends in both directions).

Homogeneity of Variance

Before wrapping up the coverage of independent samples t-tests, there is one other important topic to cover. Using the pooled variance to calculate the test statistic relies on an assumption known as homogeneity of variance. In statistics, an assumption is some characteristic that we assume is true about our data, and our ability to use our inferential statistics accurately and correctly relies on these assumptions being true. If these assumptions are not true, then our analyses are at best ineffective (e.g. low power to detect effects) and at worst inappropriate (e.g. too many Type I errors). A detailed coverage of assumptions is beyond the scope of this course, but it is important to know that they exist for all analyses.

For the current analysis, one important assumption is homogeneity of variance. This is fancy statistical talk for the idea that the true population variance for each group is the same and any difference in the observed sample variances is due to random chance (if this sounds eerily similar to the idea of testing the null hypothesis that the true population means are equal, that’s because it is exactly the same!) This notion allows us to compute a single pooled variance that uses our easily calculated degrees of freedom. If the assumption is shown to not be true, then we have to use a very complicated formula to estimate the proper degrees of freedom. There are formal tests to assess whether or not this assumption is met, but we will not discuss them here.

Exercises – Ch. 13

1. What is meant by “the difference of the means” when talking about an independent samples t-test? How does it differ from the “mean of the differences” in a repeated measures t-test?
2. Describe three research questions that could be tested using an independent samples t-test.
3. Calculate pooled variance from the following raw data:

Group 1	Group 2
16	4
11	10
9	15
7	13
5	12
4	9
12	8

4. Calculate the standard error from the following descriptive statistics

1. s₁ = 24, s₂ = 21, n₁ = 36, n₂ = 49
2. s₁ = 15.40, s₂ = 14.80, n₁ = 20, n₂ = 23
3. s₁ = 12, s₂ = 10, n₁ = 25, n₂ = 25

5. Determine whether to reject or fail to reject the null hypothesis in the following situations:

1. t(40) = 2.49, α = 0.01, one-tailed test to the right
2. X̅̅̅ = 64, ̅X̅̅̅ = 54, n₁ = 14, n₂ = 12, 𝑠_{̅̅̅̅ ̅̅̅̅} = 9.75, α = 0.05, two-tailed test
3. 95% Confidence Interval: (0.50, 2.10)

6. A professor is interest in whether or not the type of software program used in a statistics lab affects how well students learn the material. The professor teaches the same lecture material to two classes but has one class use a point-and-click software program in lab and has the other class use a basic programming language. The professor tests for a difference between the two classes on their final exam scores.

Point-and-Click	Programming
83	86
83	79
63	100
77	74
86	70
84	67
78	83
61	85
65	74
75	86
100	87
60	61
90	76
66	100
54

7. A researcher wants to know if there is a difference in how busy someone is based on whether that person identifies as an early bird or a night owl. The researcher gathers data from people in each group, coding the data so that higher scores represent higher levels of being busy, and tests for a difference between the two at the .05 level of significance.

Early Bird	Night Owl
23	26
28	10
27	20
33	19
26	26
30	18
22	12
25	25
26

8. Lots of people claim that having a pet helps lower their stress level. Use the following summary data to test the claim that there is a lower average stress level among pet owners (group 1) than among non-owners (group 2) at the .05 level of significance. M₁ = 16.25, M₂ = 20.95, s₁ = 4.00, s₂ = 5.10, n₁ = 29, n₂ = 25

9. Administrators at a university want to know if students in different majors are more or less extroverted than others. They provide you with descriptive statistics they have for English majors (coded as 1) and History majors (coded as 2) and ask you to create a confidence interval of the difference between them. Does this confidence interval suggest that the students from the majors differ? M₁ = 3.78, M₂= 2.23, s₁ = 2.60, s₂ = 1.15, n₁ = 45, n₂ = 40

10. Researchers want to know if people’s awareness of environmental issues varies as a function of where they live. The researchers have the following summary data from two states, Alaska and Hawaii, that they want to use to test for a difference. M_H = 47.50, M_A = 45.70, s_H = 14.65, s_A = 13.20, n_H = 139, n_A = 150

Answers to Odd- Numbered Exercises – Ch. 13

1. The difference of the means is one mean, calculated from a set of scores, compared to another mean which is calculated from a different set of scores; the independent samples t-test looks for whether the two separate values are different from one another. This is different than the “mean of the differences” because the latter is a single mean computed on a single set of difference scores that come from one data collection of matched pairs. So, the difference of the means deals with two numbers but the mean of the differences is only one number.

9. M1-M₂ = 1.55, t* = 1.990, s_M1-M2= 0.45, CI = (0.66, 2.44). This confidence interval does not contain zero, so it does suggest that there is a difference between the extroversion of English majors and History majors.