Reliability and Validity

There are different kinds of reliability. Here’s a graphic to help you organize and remember these important ideas:

The actual statistics used to test reliability can be quite complex. However, the ideas are simple and really just forms of correlation and regression. We'll just give you a small taste of the procedures.

Say you have a test that measures a personality trait. You would like for all the items to give you consistent information about the trait.

How could you do that?

Here's a clever idea - let's take half the items, compute your score and take the other half of the items and compute a separate score for each. If you found a high Pearson's correlation coefficient between these split halves then it would look like the two parts of the test agree with each other. The whole test would seem to have good internal consistency or reliability.

This is in fact done with the Spearman-Brown Split Half Coefficient (r_sb):

Take your scale or test and divide it in some random manner into two halves. If the sum scale were reliable, you would expect that the two halves would have an r close to 1.0. Reliability will lead to less than perfect correlations. The actual equation for Split Half reliability is:

You might see a problem in that you picked two halves at random. Why not try to take into account all possible split halves. Wouldn't that you give you a better estimate?

In fact, that is done by Cronbach's Alpha:

Cronbach's Alpha (a) is preferred to r_sb .

The s²_i 's indicates the variances for the k individual items; s²_sum indicates the variance for the sum of all items. Bottom line: If a is close to 1.0 your test items are reliable. Programs can claudicate this for you.

Spearman-Brown Split half Coefficient and Cronbach's Alpha are the most common statistics you will see. If you go into the field of testing (Psychometrics), let me assure you that you will learn much more.

Having subjects respond reliably on a measure is a great start, but there is another concept you need to get down really well. That’s validity. There are many kinds of validity, but they all refer to whether or not what you are manipulating, or what you are measuring, truly reflects the concept you think it does.

Here’s a crazy (but true) example: many years ago, people used to believe that if you had a large brain then you were intelligent. Suppose you went around and measured the circumference of your friend's heads because you also believed this theory, (they’d know for sure that you're a psychology major now). Is the size of a person’s head a reliable measure (Think first!)? The answer is YES. If I measured the size of your head today and then next week, I would get the same number. Therefore, it IS reliable. However, the whole idea is wrong! Because we now know that larger headed people are not necessarily smarter than smaller headed ones, we know that the theory behind the measure is invalid.

Moral: When you do research in psychology you have to make sure that you get consistent results that also truly reflect those mysterious concepts that reside in the human mind.