Inferential statistics help us determine whether the difference we find between our experimental and control groups is caused by the manipulation of the independent variable or by chance variation in the performances of the groups. If the difference has a low probability of being caused by chance variation, we can feel confident in the inferences we make from our samples to the populations they represent.
In experiments, psychologists use inferential statistics to test the null hypothesis. The null hypothesis states that the independent variable has no effect on the dependent variable. Consider an experimental study of the effect of overlearning on memory in college students. When we use overlearning, we study material until we know it perfectly, and then continue to study it some more. At the beginning of the experiment, the participants would be selected from the same population (college students) and randomly assigned to either the experimental group (overlearning) or the control group (normal studying). Thus, the independent variable would be the method of studying (overlearning versus normal studying). The dependent variable might be the score on a 100-point exam on the material studied.
Learning Check #21: Identify the independent variable, dependent variable, and the null hypothesis from the following scenario:
A researcher would like to know if highlighting a textbook helps students to score better on the exams. She randomly selects one-half of the students in an introductory class and instructs them to highlight their textbooks as they read. The other students are instructed to do NO highlighting as they read.
If the experimental manipulation has no effect, the experimental and control groups would not differ significantly in their performance on the exam. In that case, we would fail to reject the null hypothesis. If the experimental manipulation has an effect, the two groups would differ significantly in their performance on the exam. In that case, we would reject the null hypothesis. This would indirectly support the research hypothesis, which would predict that overlearning improves exam performance. But how large must a difference be between groups for it to be significant? To determine whether the difference between groups is large enough to minimize chance variation as an alternative explanation of the results, we must determine the statistical significance of the difference between them.
The characteristics of samples drawn from the population they represent will almost always vary somewhat from those of the true population. This is known as sampling error. Thus, a sample of five students taken from your psychology class (the population) would vary somewhat from the class means in age, height, weight, intelligence, grade point average, and other characteristics.
If we repeatedly took random samples of five students, we would continue to find that they differ from the population. But what of the difference between the means of two samples, presumably representing different populations, such as a population of students who practice overlearning and a population of students who practice normal study habits? How large would the differences have to be before we attributed them to the independent variable rather than to chance? In this example, how much difference in the performance of the experimental group and the control group would be needed before we could confidently attribute the difference to the practice of overlearning?
The larger the difference between the means of two samples, the less likely it would be attributable to chance. Psychologists typically accept a difference between sample means as statistically significant if it has a probability of less than 5 percent of occurring by chance. This is known as the .05 level of statistical significance. In regard to the example, if the difference between the experimental group and the control group has less than a 5 percent probability of occurring by chance, we would reject the null hypothesis. Our research hypothesis would be supported: overlearning is effective. Scientists who wish to use a stricter standard employ the .01 level of statistical significance. This means that a difference would be statistically significant if it had a probability of less than 1 percent of being obtained by chance alone.
The difference between the means of two groups will more likely be statistically significant under the following conditions:
1. When the variability within the groups is small.
Note that statistical significance is a statement of probability. We can never be certain that what is true of our samples is true of the population they represent. This is one of the reasons why all scientific findings are tentative. Moreover, statistical significance does not indicate practical significance. A statistically significant effect may be too small or be produced at too great a cost of time or money to be useful. What if those who practice overlearning must study two extra hours each day to improve their exam performance by a statistically significant, yet relatively small, 3 points? Knowing this, students might choose to spend their time in another way. As the American statesman Henry Clay (1777-1852) noted, in determining the importance of research findings, by themselves statistics are no substitute for judgment.
Learning Check #22: Suppose that the researcher in Learning Check #21 rejected the null hypothesis and concluded that there was a significant difference due to highlighting. What would this mean in terms of probability?
Learning Check #23: Can research demonstrate statistical significance, yet have no real practical value?
Learning Check #24: Could two groups have a difference that looked important, yet not be statistically different from one another? That is, could the difference between two groups appear to have practical value, yet not achieve statistical significance?
The independent variable is highlighting/no highlighting. The dependent variable is the exam score. The null hypothesis would be that there is no difference in exam scores between students who highlight in the text and those who do not highlight in the text.
A significant difference indicates that the difference in the means between the two groups (in this case exam scores for the highlighting and not highlighting groups) is probably NOT due to random chance alone. Something is causing a difference between these two groups, probably the highlighting, given that the groups are close to identical in all other respects.
Yes, suppose that the research in Learning Check #22 found that the exam scores differed by 1 point and that a 1 point difference was statistically significant. In this case, the difference is probably not impressive enough to encourage people to spend a great deal of time highlighting.
This is a very important topic for psychology. Many things look at first glance to be different than they really are. Two students may find that one scored very high on an exam by simply looking at the chapter summaries, whereas another student scored low and read every chapter. Although there is a big difference in exam scores that looks important, research would need to be conducted to determine exactly how that difference came about. Just because something looks valuable does not mean that it is valuable.