Computerized Adaptive Test Test-Retest Reliability

Cronbach's alpha and KR-20 measure internal consistency within a single administration. In a CAT different examinees receive different items, so there is no single fixed item set whose internal consistency can be calculated across the full sample in the conventional way. You can compute internal consistency within each adaptive administration using the item information contributed by each selected item, but this does not capture temporal stability. Test-retest correlation is necessary to assess score consistency across occasions.

For a Pearson correlation, a sample of at least 100 examinees is commonly recommended to achieve acceptable precision. With fewer than 50 participants the 95% confidence interval around a correlation of 0.85 can span more than 0.20 points, making the estimate difficult to interpret. Power analysis for correlation testing can inform the required N for a given desired precision.

The interval should be long enough for memory effects on specific items to dissipate — typically at least two weeks — but short enough that true change in the measured construct is unlikely. For stable traits such as general cognitive ability or personality dimensions, intervals of two to four weeks are standard. For constructs sensitive to treatment or environmental change, a shorter interval or additional design controls are needed.

Disattenuation removes the attenuation due to measurement error in both theta estimates and therefore raises the reliability coefficient toward the true-score correlation. However, the correction amplifies any sampling error in the conditional standard errors of measurement, and in small samples the corrected coefficient can exceed 1.0 or become unstable. Report both the observed and corrected correlations with their confidence intervals.

No. Pearson or Spearman correlation captures rank-order consistency but is insensitive to systematic mean shifts between occasions. If theta scores drift upward for all examinees between administrations — perhaps due to a practice effect or seasonal variation — the correlation can remain high while scores are systematically biased. Supplement the correlation with a paired-samples test for mean differences and, for clinical contexts, an intraclass correlation coefficient that is sensitive to both rank-order and absolute agreement.