The characteristics of samples drawn from the population they represent will almost always vary somewhat from those of the true population. This is known as fluctuations in sample means due to sampling, or sampling error. Thus, a sample of five students taken from your psychology class (the population) would vary somewhat from the class means in age, height, weight, intelligence, grade point average, and other characteristics.

If we repeatedly took random samples of five students, we would continue to find that they differ from the population. But what of the difference between the means of two samples, presumably representing different populations, such as a population of students who practice over-learning and a population of students whom practice normal study habits? How large would the differences have to be before we attributed them to the independent variable rather than to chance? In this example, how much difference in the performance of the experimental group and the control group would be needed before we could confidently attribute the difference to the practice of over-learning?

The larger the difference between the means of two samples, the less likely it would be attributable to chance. Psychologists typically accept a difference between sample means as statistically significant if it has a probability of less than 5 percent of occurring by chance. This is known as the .05 level of statistical significance. In regard to the example, if the difference between the experimental group and the control group has less than a 5 percent probability of occurring by chance, we would reject the null hypothesis. Our research hypothesis would be supported: over-learning is effective. Scientists who wish to use a stricter standard employ the .01 level of statistical significance. This means that a difference would be statistically significant if it had a probability of less than 1 percent of being obtained by chance alone.