Hallucination Detection — Factual Consistency Checking for LLM Outputs

An intrinsic hallucination directly contradicts information present in the source document — for example, a summary that says a patient was treated with drug A when the source says drug B. An extrinsic hallucination introduces information that is neither supported nor contradicted by the source, such as adding a biographical detail that does not appear anywhere in the input. The distinction matters because extrinsic hallucinations are harder to catch with a simple contradiction check.

For NLI-based and QA-based verification you do need a reference source. The SelfCheckGPT approach is the exception: it draws multiple samples from the model for the same prompt and treats cross-sample consistency as a proxy for factuality, so no external reference is required — but you do need the ability to call the model multiple times, and you need to interpret the resulting consistency score carefully.

If you have a well-defined source document (e.g., in a RAG pipeline or summarisation task), an NLI model such as a DeBERTa-MNLI model is a natural starting point and gives per-claim entailment labels. If question generation is feasible, QA-based verification can be more precise on factual claims. If you have black-box API access only and no reference document, SelfCheckGPT is the appropriate choice. In high-stakes domains, combining methods and requiring agreement between them reduces false negatives.

A labelled dataset of reference-output pairs annotated by human experts for faithfulness is the gold standard. The detector's predictions are compared against these labels using precision, recall, and F1 at the claim or sentence level. Published benchmarks such as FactScore and the SummaC benchmark provide standardised evaluation sets for this purpose.