NLM DIR Seminar Schedule

Abstract:

Can we mechanistically measure the difference in the response generation process of a Large Language Model when producing correct and hallucinated content? This question drives our broader project goal to detect and mitigate hallucinated content generated by large language models, especially in biomedical settings where plausible but incorrect outputs can affect real-world decision support in high-risk scenarios.

This talk is grounded in our systematic literature review on intrinsic knowledge boundary detection, which examined how internally held model knowledge may be identified without external retrieval or prompt augmentation. That work framed hallucination as more than an output-level error, raising the possibility that correct answers and hallucinated answers may follow measurably different internal retrieval and generation pathways.

Our project, ClinIQLink, formalized this question at scale through a novel biomedical question answering benchmark and annotation framework. Using source-grounded medical QA across closed-ended, open-ended, inverse, and multi-hop formats, ClinIQLink benchmarked available open source and open weight LLMs, evaluated model knowledge on a novel dataset, and collected hallucination behavior in biomedical responses. Interactive ClinIQLink resources are available at https://bionlp.nlm.nih.gov/ClinIQLink and https://bionlp.nlm.nih.gov/ClinIQLink2. See https://cliniqlink.org/ and https://aclanthology.org/2025.bionlp-1.32/ for more.

The resulting hallucination corpus supports a structured categorization of biomedical LLM failures. We analyze hallucinations through source orientation and consistency orientation, including intrinsic and extrinsic hallucinations, factuality and faithfulness failures, and cases such as factual, faithful, temporal, amalgamated, nonsensical, and incomplete responses. We also discuss hallucination impact through severity, harmfulness, and obviousness, emphasizing that the most dangerous biomedical hallucinations are often plausible, non-obvious, and actionable.

MechaTerp and MechaTerp-TRACE extend this work into mechanistic interpretability. MechaTerp is an interactive tool for inspecting model internal behavior during generation, available at https://lhc-lx-openidev.nlm.nih.gov/MechaTerp/ (requires access through the NIH Bethesda campus network or NIH VPN). MechaTerp-TRACE uses targeted recall ablation and component analysis to trace the knowledge path of information retrieval during generation. The project goal is to apply MechaTerp-TRACE to the ClinIQLink hallucination dataset, comparing accurate biomedical answer generation with hallucinated biomedical answer generation, so that hallucination detection, categorization, and mechanistic knowledge localization can support safer LLM-based decision support systems.