Removing Unaccounted-for Error

As we have seen, whenever we do a research study, there is bound to be some error in our measurements. Some of this error is uncontrolled and due to random factors such as variations in the researcher's appearance on particular days, the weather, and time of the semester. One main source of "unaccounted-for" error is the individual differences among the subjects in the study. Remember the dependent groups t-test allowed us to statistically control for subject differences by calculating a difference score for a subject's two scores? Here, the repeated-measures analysis of variance enables us to eliminate (i.e., partition out) the variance that is attributable to individual differences from the dependent measure. Statistically, this has the effect of removing some unexplained error from the within-groups variance estimate (i.e., the error term) thus, making the denominator for the F smaller, and therefore, making the analysis more powerful.

Often, researchers and therapists will want to record changes in, and repeatedly measure some behavior over a period of time. One of its best features is that, unlike the dependent groups t-test, the repeated-measures ANOVA allows for any number of repeated observations-not just two.

Just like the two-way ANOVA, the repeated-measures ANOVA is economical (fewer participants are required since the error term is reduced). It is efficient (change can be studied as an ongoing process), elegant, and can be generalized.

One Factor

Research which repeatedly measures the same subjects on one or more variables is called a repeated-measures or within-subjects design. Do not confuse the within-subjects design with the within-subjects (or groups) error (we will study more about this later). In this section I will complete an entire example of a single-factor repeated-measures ANOVA for you. Keep in mind that in theory, any number of factors may be manipulated in a single study. These may all be within-subjects variables (all subjects responding to all levels of all factors) or they may be a combination of within-subjects and between-subjects variables, as in a mixed design.

The statistical assumptions of the repeated-measures ANOVA are the same as those for the one and two-way ANOVAs covered previously. In addition, since each subject is experiencing all levels of a single independent variable the repeated-measures ANOVA also assumes that the effect on behavior of each level of the independent variable should be consistent in all subjects. This is simply another way of saying that you think the independent variable manipulation will affect all people in a similar manner. Remember that this factor (all subjects being affected in approximately the same way) makes the dependent groups t-test more powerful than the independent groups t-test.

Let's Do It

A language professor is interested in how students' proficiency of a new language progresses over the course of a semester. He has six students from his Spanish class take a Spanish vocabulary test at the beginning of the semester and then again every month for the remainder of the semester. The vocabulary test is scored on a 10 point scale with 1 being poor and 10 being excellent. Here are the data he collected:

Prior to course	Month 1	Month 2	Month 3
1	3	5	8
3	4	6	9
5	6	8	10
9	9	9	10
6	6	8	9
2	2	5	7
MP = 4.33	M1 = 5	M2 = 6.83	M3 = 8.83

First, calculate the total sum of squares.
ΣΧ = 150
ΣΧ² = 1108
N = 24 (not 6! n = 6)

SS_T = 1108 - 1502/24 = 170.5
df_T = N - 1 = 23

SS_B = 262/6 + 302/6 + 412/6 + 532/6 - 1502/24 = 112.67 + 150 + 280.167 + 468.167 - 937.5 = 73.504
df_B = k - 1 = 3
MS_B = 73.504/3 = 24.50

SS_S = (172 + 222 + 292 + 372 +292 + 162)/4 - 937.5 = (289 + 484 + 841 + 1369 + 841 + 256)/4 - 937.5 = 1020 - 937.5 = 422.5
df_S = n - 1 = 5
MS_S = 84.5
SS_W = 1108 - 1020 - 1011.004 + 937.5 = 14.5
df_W = (n -1 )(k - 1) = 5 x 3 = 15
MS_W = 14.5/15 = .967

F_obt = 24.5/.967 = 25.345
F_subjects = 84.5/.967 = 87.38
F_{crit(3, 15)} = 3.29

So, we can reject the null hypothesis of no difference between means. Clearly, students' vocabulary scores change over the course of the semester. Next we will conduct post hoc tests to see where the real learning (or forgetting!) is going on.

Based on this HSD value you can see that there is no significant difference between the mean for prior to the course and the first month, but all the other pair-wise comparisons are significant.

We can also calculate the effect size for these data.

Mixed Design ANOVA

We can now logically conclude that the number of possible experimental designs and accompanying analyses is almost limitless. One popular combination of what was just illustrated is the mixed-design ANOVA. In a mixed design there are both within-subject factors (variables in which each subject receives all levels of the variable) and between-subject factors (variables in which the subject receives only one of a number of possible levels). While these designs are useful, they are too complex to be examined in detail in an introductory course. Please see Judd and McClelland (1989) for excellent examples and instruction for analyzing data from these and other complex designs.

Internal Validity

A useful aspect of repeated-measures designs is that a great deal of data can be gathered from the same subjects. This comes at a cost, however. Once a subject has contributed data to one level of the repeated-measures variable, he still has to contribute to the other(s). In many ways the subject may have been changed by other influences in the period between the first and second measurements. These other possible sources of change are identified as "threats to internal validity". Thus, in the example of language aptitude presented earlier, grades may have improved over the semester because students were learning more, but it also may have been that they were simply becoming more familiar with the way that the instructor tests (testing), or maybe the instructor's tests became less challenging (instrumentation). Perhaps the students were simply applying themselves more as the semester drew to a close to avoid failing (maturation and/or history). Regardless of whether the data were confounded by these or other factors, the possibility remains that they were, and so the researcher must be aware of their occurrence.