Inferential statistics help us determine whether the difference we find between our experimental and control groups is caused by the manipulation of the independent variable or by chance variation in the performances of the groups. If the difference has a low probability of being caused by chance variation, we can feel confident in the inferences we make from our samples to the populations they represent.

Hypothesis Testing

In experimental research, sociologists use inferential statistics to test the null hypothesis. The null hypothesis states that the independent variable has no effect on the dependent variable. Consider an experimental study of the effect of overlearning on examination performance in college students. When we use overlearning, we study material until we know it perfectly, and then continue to study it some more. At the beginning of the experiment, the participants would be selected from the same population (college students) and randomly assigned to either the experimental group (overlearning) or the control group (normal studying). Thus, the independent variable would be the method of studying (overlearning versus normal studying). The dependent variable might be the score on a 100-point exam on the material studied.

Learning Check #21: Identify the independent variable, dependent variable, and the null hypothesis from the following scenario:

A researcher would like to know if highlighting a textbook helps students to score better on the exams. She randomly selects one-half of the students in an introductory class and instructs them to highlight their textbooks as they read. The other students are instructed to do NO highlighting as they read.

Click here for Answer.

If the experimental manipulation has no effect, the experimental and control groups would not differ significantly in their performance on the exam. In that case, we would fail to reject the null hypothesis. If the experimental manipulation has an effect, the two groups would differ significantly in their performance on the exam. In that case, we would reject the null hypothesis. This would indirectly support the research hypothesis, which would predict that overlearning improves exam performance. But how large must a difference be between groups for it to be significant? To determine whether the difference between groups is large enough to minimize chance variation as an alternative explanation of the results, we must determine the statistical significance of the difference between them.

Statistical Significance

The characteristics of samples drawn from the population they represent will almost always vary somewhat from those of the true population. This is known as sampling error. Thus, a sample of five students taken from your sociology class (the population) would vary somewhat from the class means in age, height, weight, intelligence, grade point average, and other characteristics.

If we repeatedly took random samples of five students, we would continue to find that they differ from the population. But what of the difference between the means of two samples, presumably representing different populations, such as a population of students who practice overlearning and a population of students who practice normal study habits? How large would the differences have to be before we attributed them to the independent variable rather than to chance? In this example, how much difference in the performance of the experimental group and the control group would be needed before we could confidently attribute the difference to the practice of overlearning?

The larger the difference between the means of two samples, the less likely it would be attributable to chance. Sociologists typically accept a difference between sample means as statistically significant if it has a probability of less than 5 percent of occurring by chance. This is known as the .05 level of statistical significance. In regard to the example, if the difference between the experimental group and the control group has less than a 5 percent probability of occurring by chance, we would reject the null hypothesis. Our research hypothesis would be supported: overlearning is effective. Scientists who wish to use a stricter standard employ the .01 level of statistical significance. This means that a difference would be statistically significant if it had a probability of less than 1 percent of being obtained by chance alone.

The difference between the means of two groups will more likely be statistically significant under the following conditions:

1. When the samples are large.
2. When the difference between the means is large.

1. When the variability within the groups is small.

Note that statistical significance is a statement of probability. We can never be certain that what is true of our samples is true of the population they represent. This is one of the reasons why all scientific findings are tentative. Moreover, statistical significance does not indicate practical significance. A statistically significant effect may be too small or be produced at too great a cost of time or money to be useful. What if those who practice overlearning must study two extra hours each day to improve their exam performance by a statistically significant, yet relatively small, 3 points? Knowing this, students might choose to spend their time in another way. As the American statesman Henry Clay (1777-1852) noted, in determining the importance of research findings, by themselves "statistics are no substitute for judgment."

Learning Check #22: Suppose that the researcher in Learning Check #21 rejected the null hypothesis and concluded that there was a significant difference due to highlighting. What would this mean in terms of probability?

Click here for Answer.

Learning Check #23: Can research demonstrate statistical significance, yet have no real practical value?

Click here for Answer.

Learning Check #24: Could two groups have a difference that looked important, yet not be statistically different from one another? That is, could the difference between two groups appear to have practical value, yet not achieve statistical significance?