Content Validity in Computerized Adaptive Testing (CAT)

In a fixed-form test every examinee receives the same items, so domain coverage is determined once at test design and applies universally. In CAT each examinee receives a different item subset chosen for statistical efficiency, raising the risk that the algorithm ignores less informative content areas. Without explicit content balancing constraints, some blueprint categories may be systematically underrepresented or absent from individual adaptive administrations.

The Content Validity Ratio quantifies the proportion of subject-matter experts who rate an item as essential to the construct, adjusted for chance agreement. Items not reaching the critical CVR for the panel size are revised or excluded. In CAT development this step is applied to every candidate item before calibration, ensuring the bank contains only content-valid items before adaptive selection begins.

It depends on the balance between the strictness of blueprint constraints and the depth of the item bank within each category. With a rich bank, precision loss is modest. With sparse categories, constraints may force selection of less informative items, increasing the test length needed to meet reliability targets. This trade-off is why adequate bank coverage per category is a prerequisite for effective content balancing.

The shadow test approach, developed by van der Linden and colleagues, constructs at each item selection step a complete hypothetical test (the shadow test) that satisfies all blueprint and logistical constraints simultaneously. The most informative item from that feasible shadow test is then administered. This guarantees a globally feasible blueprint-compliant test rather than making greedy local decisions that may become infeasible later in the adaptation.

Partially. Expert panel reviews and CVR values for individual items carry over, since item content does not change with administration mode. However, blueprint adherence evidence must be re-established for the CAT context, as adaptive selection may produce different realized content distributions than the fixed form. New validity evidence documenting that adaptive administrations meet blueprint proportions is required.