The difference between the means of groups will more likely be statistically significant under the following conditions:

1. When the samples are large.
2. When the difference between the means is large.
3. When the variability within the groups is small.

These are all factors involved in the power of a study. Power is the probability of your experiment allowing you to detect an effect that really exists in the world. The difference between the means of your groups is a measure of effect size – how big of an impact your independent variable has on your dependent variable. Larger samples are apt to be more representative of the population in question and, as sample size increases within groups variance typically decreases. Since one rarely has precise control over the difference between means, a good method for improving power is to increase the number of participants in a study.

Whenever hypotheses are tested and the null hypothesis is retained or rejected there are a number of possible outcomes. Specifically, a correct or incorrect decision may be made regarding the null hypothesis. Figure 6 displays the possible combinations of "truth" and "statistical decision" that may be obtained in any hypothesis-testing situation.

Figure 6.

		Truth About H0
		True	False
Statistical Decision	Reject	Type I Error	OK/POWER
About H0	Retain	OK	Type II Error

A Type I or alpha (a) error occurs when the null hypothesis (H₀) is rejected but should not be since it is true. A Type II or beta (b) error occurs when the null hypothesis (H₀) is retained but should not be since it is false. There are several conditions that may lead to the occurrence of a Type I error, including poor sampling and biased group assignment. Typically, Type II errors are the result of insufficient power. Unfortunately, avoiding one type of error increases the probability of the other.

Learning Check #22:
Suppose that the researcher in Learning Check #21 rejected the null hypothesis and concluded that there was a significant difference due to highlighting. What would this mean in terms of probability?

Click here for Answer.

Learning Check #23:
Can research demonstrate statistical significance, yet have no real practical value?

Click here for Answer.

Learning Check #24:
Could two groups have a difference that looked important, yet not be statistically different from one another? That is, could the difference between two groups appear to have practical value, yet not achieve statistical significance?

Click here for Answer.