Quantifying Reliability and Measurement Errors

One important goal of a measurement study is to quantify the reliability of a measurement process. We have seen that quantification can be accomplished by two general methods. Using a representative sample of objects we can employ a measurement process using multiple co-occuring observations to estimate internal consistency reliability; or we can repeat a measurement process on separate occasions, which enables estimation of test-retest reliability. A key aspect of reliability, from a measurement perspective, is that any measurement process consisting of multiple observations can reveal the magnitude of its own reliability. This contrasts with estimation of validity, that, as we will see later, requires collection of additional data [120].

Friedman, Charles P. & Wyatt, Jeremy C. Evaluation Methods in Biomedical Informatics, 2nd ed. Springer, 2006 (Health Informatics Series). ISBN 0387258892.