Hypothesis Testing

1. You come up with the hypothesis (for example - college students sleep a different amount than other folks)

2. You generate a sample (pick a set of college students)

3. You calculate your summary statistics (for example, the mean and standard deviation of number of hours that college students sleep per week)

4. You determine the statistical test that will compare your summary statistic against the value determined by your null hypothesis. With the college students - you would use the single sample t-test.

5. You calculate the test statistic.

6. You derive the appropriate sampling distribution - see Workshop 3 for the Single Sample t-test - in reality, you use tables in your book or your computer program does this.

7. The test statistic tells you if your sample statistic is very far away from the null hypothesis statistic in the sampling distribution and thus not likely. (For our example, we assume that people sleep eight hours a night. We calculate our student sleep mean. We calculate the test statistic using this mean). Now you test for significance.
You do this by comparing the test statistic against the table statistic for a specific significance level or alpha level (a level).
The alpha level is a cutoff score in the sampling distribution (See Workshop 1).
It marks a cut off score that indicates whether your sample statistic occurred by chance at a probability less or greater than that marked by the alpha level. Typically this probability is set at .01 or .05.
If your test statistic is outside the alpha level's cut off score (or in the tail); you reject the H_o (Null Hypothesis). If not, you fail to reject the H_o. Check out the above graphic in blue and yellow.

8. If the test statistic is unlikely, we reject the null hypothesis and say we have a statistical significant result (see our workshop for this example).

What does statistically significant mean?

It means the result is not by chance or luck. It did not happen because of chance.

It does not necessarily mean that your result has importance in the real world. Significant means not chance and that's it!!

If you wonder about whether your statistically significant result has utility - you need to consider other statistics like effect size.

Big Idea Box Alpha Level (a) or Significance Level Value of a test statistic that leaves .05 or .01 in the tail of a sampling distribution. If the test statistic is bigger than this value (in the vast majority of cases for the beginner), that means your p value is less than .05 or .01. This means your result could only happen by chance 5% or 1% of the time. Since, this means a chance result is unlikely, we say that we don't have a chance result. We reject this idea and reject the Null Hypothesis.

9. If you do not find significance, the current rules of statistics say that you say the following:

I fail to reject the Null Hypothesis!

What does this mean?

It does not mean: There is no difference, relationship or effect.

It means I did not find a difference, effect or relationship.

The logic is that maybe you should have tested more subjects as more subjects might have lead to statistical significance. Or you could have changed the experimental conditions to yield a greater effect.

What! Can I never say that there is no effect, no relationship and no difference?

This is controversial but as you get more into statistics, you will find more advanced techniques to deal with the problem.