Standardized Test Analysis

Classical test theory describes scores in terms of true score plus error and produces sample-dependent item statistics (difficulty as proportion correct, discrimination as item-total correlation) and reliability coefficients. Item response theory models the probability of each response as a function of a latent ability and item parameters, yielding sample-invariant item parameters, ability estimates, and the test information function. IRT supports equating, adaptive testing, and precise targeting of measurement, at the cost of stronger assumptions and larger sample requirements. Many programs use both.

Reliability concerns the consistency of scores — would a student get a similar score on a parallel form or another occasion. Validity concerns whether the scores support their intended interpretation and use — does the test actually measure what it claims and predict what it should. A test can be highly reliable yet invalid (consistently measuring the wrong thing), and reliability only sets an upper bound on certain validity coefficients. The Standards treat validity, supported by multiple sources of evidence, as the overriding concern, with reliability one necessary component.

The standard error of measurement (SEM) estimates how much an individual's observed score would vary across repeated parallel administrations due to measurement error; it is derived from the score variability and reliability. It matters because it bounds how precisely a single score can be interpreted: differences or classifications based on score gaps smaller than the SEM (or its conditional version across the scale) are not trustworthy. Reporting and respecting the SEM guards against over-interpreting small score differences in high-stakes decisions.