Construct Validity in Computerized Adaptive Testing (CAT)

No. IRT model fit indicates that item response patterns are consistent with the mathematical model used for scoring. It does not address whether the modeled trait corresponds to the intended psychological construct. Construct validity requires additional evidence: internal factor structure, convergence with related external criteria, and discrimination from unrelated ones.

Classical coefficient alpha is not directly applicable because it assumes a fixed item set. The appropriate reliability analogue in CAT is the IRT-based marginal reliability, sometimes called the test information function integrated over the trait distribution. This can be supplemented by the expected a posteriori (EAP) reliability estimate.

Small samples create two problems: IRT parameter estimates become unstable, which undermines the score scale, and external validity analyses lose statistical power. Structural evidence via CFA or IRT fit typically requires at least several hundred respondents; DIF analyses require large enough subgroup sizes (typically n ≥ 200 per group) to detect meaningful differential functioning.

Adding or removing items can change the effective construct coverage of the pool, especially if the item selection algorithm's exposure controls shift which items are actually administered. Construct validity evidence should be revalidated — at minimum, the internal structure of the updated pool and the DIF status of new items should be checked — before deploying the revised instrument.

The conceptual framework is the same, but the implementation differs. Because item sets are personalized, score-level analyses (correlations, factor structures of subscores) are directly applicable, whereas item-level analyses must account for differential exposure and the IRT scale. DIF analyses must cover all pool items rather than a fixed set, and reliability is expressed through the information function rather than classical indices.