Digital Program Evaluation — Assessing Programs Using Digital Data and Methods

Learning analytics is a sub-field focused specifically on educational platforms — it uses learner behavioral data to understand and improve learning processes. Digital program evaluation is broader in scope and purpose: it applies to any program type (health, social services, government, education), retains the classic evaluative goal of rendering a judgment about program merit and worth, and explicitly requires comparison against stakeholder-defined criteria of success. Learning analytics is a tool that digital program evaluators may use, but evaluation also involves stakeholder engagement, ethical review, mixed-method data integration, and consequential recommendations that go beyond analytics.

With caution. Vendor-supplied analytics are a starting point, not a substitute for independent data collection and quality assurance. Platform metrics are designed for product management and may not align with your evaluative constructs, may exclude certain user segments, and may change their calculation methodology without notice. Whenever feasible, request access to raw log data, validate key metrics against independent sources, and supplement with primary data (surveys, interviews, assessments) to ensure you are measuring what the evaluation actually needs to measure.

This depends on jurisdiction and institutional rules. In many contexts, informed consent for program participation does not automatically cover use of behavioral log data for research or evaluation purposes. GDPR (EU), FERPA (US education), and HIPAA (US health) impose specific constraints. Best practice is to address evaluation data use in the initial informed-consent process, obtain IRB or ethics-committee review if required, and anonymize data at the earliest possible stage. Consult your institutional review office before collecting or analyzing any participant data.

Non-digital participants represent a critical subgroup: they are typically the hardest to reach, may have lower digital literacy or access, and are most likely to be underserved by the program. Excluding them from the evaluation produces an optimistic and inequitable picture of program performance. Use supplementary traditional data collection — telephone surveys, in-person interviews, paper records — to capture outcomes for offline participants, and report disaggregated results explicitly. An equity analysis comparing digital and non-digital participants is often one of the most policy-relevant findings in digital program evaluations.

Yes, and digital delivery can make randomization easier than in face-to-face settings. Platform infrastructure can randomly assign users to program versus control conditions (or to alternative program versions for A/B testing) at scale and low cost. However, ethical concerns about withholding a potentially beneficial program, spillover between conditions (users who talk to each other), and attrition in control conditions must be managed carefully. When full randomization is infeasible, quasi-experimental designs such as regression discontinuity or interrupted time series applied to platform log data are well-suited alternatives.