Hypothesis Testing

Power and Statistical Errors

Error

You can make an error or two when you do a test of significance. You might say things are different when they are not. You may miss a relationship that really exists. These are called Type Iand Type II errors, respectively.

Power

How about finding real differences or relationships? You want that power!

Power is an easy idea in the statistical world. You want to correctly reject a false Null Hypothesis. Power is the probability of correctly rejecting a false Null Hypothesis. It means the chance of finding what you are looking for!

Going to the Doctor as a Metaphor for Significance Testing, Errors and Power

 

In statistics, you are usually looking for some result such as:
  1. A difference between means (as with a t-test or Anova).
  2. A difference between proportions (with a chi-square).
  3. A relationship (a correlation or multiple regression).

Think of it this way. You go to the doctor because you want to know if you are sick. This is like looking for a statistical difference.

However, there are several possibilities.

It might be that you are sick or you are not.

Also, independent of this - the doctor may say you are sick or the doctor may tell you that you are not sick.

This leads to four outcomes. Two are correct and two are wrong.
(See our graphic below)

The one we are interested in is when you are really sick and the doctor says you are really sick.

The above situation is like a statistical test.

You look for a difference. This like seeing if you are sick. It is when the Ho (Null Hypothesis) is false.

The statistical test is like the doctor.

It might say there is a difference when there is one (good - what we want).

This is statistical power.

Many experts recommend that you use a power of .80. This means that you have an 80% chance of finding a difference when you really want to find it. You don't want to miss a real difference or correlation.

(Bad - called a Type II error with probability equal to Beta).

Power is equal to 1 - Beta.

The test might say there is a difference when there is not one.

(Bad - an error called Type I error whose probability equals your alpha rate: .05 or .01).

Depending on conditions you may have a good or bad chance of finding the desired result. To increase power you can:

  1. Try to increase the effect size or the strength of the relationship.
  2. Decrease experimental error.
  3. Use a higher alpha level (say .05 as compared to .01). Note this increases power but also Type I error.
  4. Increase sample size.
  5. Use matched samples or covariance techniques.

Study the graphic at your leisure.

Bottom Line:

Remember this: