Research gapComputer science

Hallucination in high-stakes AI: can we measure how often a deployed language model gets it dangerously wrong?

Hospitals, law firms and service companies now run large language models inside real workflows, and both vendors and researchers are racing to reduce fabricated output (Huang et al., 2025; Basu & Huynh, 2026). What is still missing is a validated way to estimate, before and during deployment, how often a system produces errors that actually harm a patient, client or customer.

PhD Research Lab InsightsEditorial team, reviewed by a subject specialist
Posted
5 October 2026
Last checked
5 October 2026
Reading time
10 min
Sources
12
Research proposal support
The gap

We do not yet know how to estimate, for a specific deployed language-model system, the rate of consequential hallucinations (fabricated or unsupported content that would change a clinical, legal or customer decision) in real workflows, nor whether automatic uncertainty signals and human review catch those errors in practice. Benchmarks report rates from about 1% to over 80% (Asgari et al., 2025; Dahl et al., 2024), but only 5% of healthcare evaluations used real patient data (Bedi et al., 2025).

Methodological gapEmpirical gap (real-world data)Evidence gap (inconsistent measures)Practical-knowledge gap

What we know

  • Hallucination has well-developed taxonomies covering factual and faithfulness errors, with causes traced to data, training and decoding (Huang et al., 2025).
  • General-purpose models hallucinated in 58% (GPT-4) to 88% (Llama 2) of answers to verifiable questions about US federal court cases (Dahl et al., 2024).
  • Commercial legal research tools built on retrieval-augmented generation still hallucinated in roughly 17% to 33% of 202 preregistered queries, despite vendor claims (Magesh et al., 2025).
  • In a clinical note-summarisation study of 12,999 clinician-annotated sentences from mock consultations, the hallucination rate was 1.47%, and 44% of hallucinations were judged major (Asgari et al., 2025).
  • Semantic entropy detects confabulations without task-specific training, with an average AUROC of 0.790 across datasets (Farquhar et al., 2024), yet the best model reached an F1 of only 0.625 on hard medical hallucinations (Pandit et al., 2025).
  • Across 44 studies, retrieval reduced healthcare hallucinations by about 30% to 50%, but prospective real-world validation was limited (Basu & Huynh, 2026).

What we do not know yet

  • How hallucination rates measured on benchmarks and mock data translate to rates on real records, queries and customer conversations.
  • Which hallucinations are consequential: there is no shared, domain-calibrated severity scale linking an error to harm.
  • Whether uncertainty signals such as semantic entropy remain useful for long, retrieval-grounded professional documents.
  • How reliably human reviewers catch the errors that remain, and how review quality changes with workload and trust.
  • How to monitor drift in hallucination rates after deployment, when models, prompts and source documents change.

Research questions a PhD could ask

  1. What is the rate of consequential hallucination of a deployed retrieval-augmented system in a defined clinical or legal workflow, measured on real (consented, de-identified) cases rather than benchmarks?
  2. Can a severity scale agreed by domain experts make hallucination rates comparable across studies and settings?
  3. Do sampling-based uncertainty signals flag the specific sentences that experts judge harmful in long-form professional output?
  4. Under realistic workload, what proportion of model errors do clinicians, lawyers or support staff detect, and which interface designs raise that proportion?
  5. Which post-deployment monitoring design detects a rise in hallucination rate early enough to act?
Methods that could close the gap
ApproachWhy it fits
Prospective evaluation on real casesSamples real, de-identified workflow outputs and has blinded experts label errors and severity, addressing the reliance on exam questions and mock data (Bedi et al., 2025; Asgari et al., 2025).
Preregistered audit studyFixes queries, tools and scoring before testing, following the design used for legal research tools (Magesh et al., 2025).
Uncertainty-signal validationTests whether semantic entropy and related signals predict expert-labelled harmful sentences, extending Farquhar et al. (2024).
Human-factors experimentMeasures detection of seeded errors by professionals under time pressure and with different interface cues, the human-in-the-loop strand flagged by Basu and Huynh (2026).
Benchmark and incentive designBuilds domain test sets that reward abstention over confident guessing, following the argument of Kalai et al. (2025) and the "not sure" results of Pandit et al. (2025).
Gap typeIndustry and research both working on it

The problem

Large language models sometimes generate fluent content that is false, unsupported by the source, or invented, such as a non-existent court case or a symptom never mentioned in a consultation (Huang et al., 2025; Dahl et al., 2024). In high-stakes work, one such error can change a decision.

Why it happens

Models are trained to predict plausible text, and common evaluations reward a confident guess over an honest "I do not know" (Kalai et al., 2025). Retrieval reduces but does not remove the problem, because the model can still misread or go beyond the retrieved sources (Magesh et al., 2025; Basu & Huynh, 2026).

Where it is felt most

The risk is felt most where outputs feed directly into decisions: clinical documentation and decision support, legal research, and customer-facing service. A Canadian tribunal held an airline responsible for incorrect advice given by its website chatbot (Moffatt v. Air Canada, 2024).

How big is the problem

  • 78%of organisations surveyed reported using AI in 2024, up from 55% in 2023 (Stanford AI Index 2025) (Maslej, 2025).
  • 66%of US physicians surveyed in 2024 said they use AI in their practice, up from 38% in 2023 (American Medical Association) (American Medical Association, 2025).
  • 233AI-related incidents recorded in 2024 in the AI Incident Database, a record and a 56.4% rise on 2023 (Maslej, 2025).
  • 17% to 33%hallucination rates of leading commercial legal AI research tools in a preregistered test of 202 queries (Magesh et al., 2025).
What studies say, and at what level of evidence
LevelWhat it shows
Systematic reviews and meta-analysesOf 519 healthcare LLM evaluations, 95.4% measured accuracy, but only 5% used real patient data and 1.2% measured calibration and uncertainty (Bedi et al., 2025). A review of 44 mitigation studies found retrieval and human review helped, with limited prospective validation (Basu & Huynh, 2026).
Preregistered and field-like studiesCommercial legal tools hallucinated in 17% to 33% of responses (Magesh et al., 2025); clinical summaries from mock consultations showed 1.47% hallucinations, many of them major (Asgari et al., 2025).
Benchmark studiesGeneral models hallucinated in 58% to 88% of verifiable legal queries (Dahl et al., 2024); a 10,000-item medical benchmark showed hard hallucinations remain difficult to detect (Pandit et al., 2025).
Methods and theorySemantic entropy offers a training-free detection signal (Farquhar et al., 2024), and evaluation incentives have been proposed as a root cause (Kalai et al., 2025).
Industry and policy reportsOrganisational adoption and recorded AI incidents both rose sharply in 2024 (Maslej, 2025).
What is missingProspective, preregistered measurement of consequential errors in real deployments, with agreed severity scales and evidence on whether human reviewers catch what models miss.

Who should take this on

Scholar profile

  • A computer scientist or NLP researcher who can partner with a hospital, law firm or service company for access to real, governed workflow data.
  • A clinician-researcher or health informatician who can recruit expert annotators and secure ethics approval.
  • A legal-technology or information-systems researcher interested in audit design and accountability.
  • A human-computer interaction researcher focused on how professionals review AI output.

Subjects and PhD specialisations

  • Computer science (natural language processing, trustworthy AI)
  • Health informatics or digital health
  • Information systems
  • Human-computer interaction
  • Law and technology
  • Biostatistics (evaluation design)

Countries where the problem is greatest

  • United States: physician use of AI rose from 38% to 66% in a year (American Medical Association, 2025), and the main legal hallucination studies test US case law (Dahl et al., 2024; Magesh et al., 2025).
  • Canada: the Moffatt decision makes companies answerable for what their chatbots tell customers (Moffatt v. Air Canada, 2024).
  • United Kingdom: clinical documentation tools are being built and safety-tested by UK-based teams (Asgari et al., 2025).
  • China, Indonesia and Thailand: majorities see AI products as more beneficial than harmful (83%, 80% and 77%), which points to fast uptake and a strong need for reliability evidence (Maslej, 2025).

Scope for future research

  1. Domain-specific severity scales that link a hallucination to likely harm, agreed by practitioners.
  2. Post-deployment monitoring methods that track hallucination drift as models, prompts and sources change.
  3. Uncertainty estimation for long, multi-source documents rather than short answers.
  4. Human-AI review designs that measurably raise error detection without adding unsafe workload.
  5. Hallucination in multilingual and low-resource language settings, where benchmarks are scarce.
  6. Evaluation incentives and leaderboards that reward calibrated abstention (Kalai et al., 2025).

How a doctoral project could address it

  1. Define the workflow

    Choose one bounded, high-stakes task (for example, discharge summaries or case-law research) and the decision each output feeds into.

  2. Agree the error scale

    Run a Delphi or expert panel to define hallucination types and severity levels for that task, building on existing taxonomies (Huang et al., 2025).

  3. Secure data and ethics

    Partner with an organisation for consented, de-identified real cases; plan governance, data protection and approval early.

  4. Measure prospectively

    Preregister the protocol, sample outputs and use blinded double annotation with agreement statistics (Magesh et al., 2025).

  5. Test detection signals

    Compare uncertainty methods such as semantic entropy against expert labels (Farquhar et al., 2024).

  6. Study the human reviewer

    Run an experiment on how often professionals catch seeded errors under realistic conditions.

  7. Publish openly

    Release the protocol, severity scale and (where permitted) annotated data so others can replicate in new settings.

Why it is worth solving

If deployed systems came with credible, task-specific error rates, clinicians, lawyers and customers could decide how much to rely on them, and organisations could compare products on evidence rather than marketing. The current evidence base is dominated by exam-style tests (Bedi et al., 2025), so even a single well-designed field study would move the field.

For science, a shared severity scale and validated uncertainty signals would make results comparable across studies. For industry, it would show where retrieval, review or abstention is worth the cost (Basu & Huynh, 2026). For the public, it would reduce the chance that a fabricated fact quietly shapes a decision about their health, money or rights.

Why this gap matters now

Language models have moved from pilots to daily work. Organisational use of AI rose from 55% to 78% in a single year (Maslej, 2025), and two-thirds of US physicians surveyed in 2024 reported using AI in practice (American Medical Association, 2025). Vendors have responded by grounding models in retrieved documents and by adding verification layers, and the research community has produced detectors, benchmarks and surveys (Huang et al., 2025; Farquhar et al., 2024).

Yet the numbers that decision-makers need are the hardest to find. Published rates range from about 1% in carefully engineered clinical summarisation (Asgari et al., 2025) to more than half of answers for general models on legal questions (Dahl et al., 2024). These figures are not contradictory: they measure different tasks, data and definitions. That is the gap. Without a standard for what counts as a consequential error, and without measurement on real cases, no one can say how safe a specific deployment is.

Why the gap persists

Three obstacles keep the question open. First, real workflow data are sensitive, so most studies use exam questions, vignettes or mock consultations; only 5% of healthcare LLM evaluations used real patient data (Bedi et al., 2025). Second, hallucination is defined and scored differently from study to study, which a recent review identifies as a barrier to comparison (Basu & Huynh, 2026). Third, systems change quickly, so a result can date before it is published.

  • Commercial tools rarely publish error rates by task or severity, which is why independent audits matter (Magesh et al., 2025).
  • Detection methods are mostly tested on short question-answer pairs, not long professional documents.
  • Evaluation leaderboards can reward guessing, which works against calibrated abstention (Kalai et al., 2025).

The result is a field that is strong on benchmarks and detection methods and thin on evidence from live use. Hard hallucinations, the ones that sit closest to the truth, are also the hardest for automatic detectors to catch (Pandit et al., 2025), which is why human review and its limits belong in the same study.

Is this feasible for a doctorate?

Yes, if the project is scoped to one task in one setting. A realistic thesis could combine an expert consensus study on error severity, a preregistered measurement study on real cases, and an experiment on detection by humans or uncertainty signals. Each strand can stand as a paper, and together they answer one question: how often does this system get it dangerously wrong, and would anyone notice?

Practical planning for a doctoral project on hallucination measurement
ConsiderationWhat to plan for
Data accessA partner organisation, a data-sharing agreement and de-identification before any annotation.
Ethics and governanceApproval covering real records, annotator welfare and any use of external model APIs.
AnnotationAt least two blinded domain experts per item and an agreement statistic.
Model versionsRecord model, prompt and retrieval versions with dates; re-run key tests before submission.
SkillsNLP evaluation, statistics for rare events, and qualitative or mixed methods for the human strand.

What to do now

  • Map 2023 to 2026 studies in your chosen domain by task, data source (real or simulated), definition and rate.
  • Pick one workflow and one partner organisation before refining research questions.
  • Draft a severity scale with practitioners and test it on a small pilot sample.
  • Preregister your protocol and plan for model and prompt version control from the start.
Your next step

Could this gap become your thesis?

  1. 1Read the evidence and the open questions
  2. 2Talk to a mentor about fit, data and method
  3. 3Build a defensible proposal
See proposal support

Questions scholars ask

Has retrieval-augmented generation not solved hallucination already?

No. It reduces hallucination, but audited legal tools built on retrieval still erred in roughly a sixth to a third of queries (Magesh et al., 2025), and reviews report reductions rather than elimination (Basu & Huynh, 2026).

Do I need access to a frontier model or large compute?

Not necessarily. The gap is about measurement in real settings, so careful study design, expert annotation and data access matter more than training models.

Will my findings date before I finish?

Specific models will change, so frame the contribution around the measurement method, the severity scale and the human review findings, and report versions and dates clearly.

Can I study this outside healthcare or law?

Yes. Finance, insurance, public services and customer support all face the same question, and liability for chatbot advice is already being tested (Moffatt v. Air Canada, 2024).

Sources

  1. American Medical Association. (2025, February 12). AMA: Physician enthusiasm grows for health care AI [Press release]. https://www.ama-assn.org/press-center/ama-press-releases/ama-physician-enthusiasm-grows-health-care-ai
  2. Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Au Yeung, J., & Pimenta, D. (2025). A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine, 8, 274. https://doi.org/10.1038/s41746-025-01670-7
  3. Basu, S., & Huynh, B. (2026). Mitigating hallucinations in healthcare AI: A systematic review of evidence-based strategies. BMC Health Services Research, 26, 1115. https://doi.org/10.1186/s12913-026-14851-1
  4. Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A. R., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A., Pfeffer, M. A., & Shah, N. H. (2025). Testing and evaluation of health care applications of large language models: A systematic review. JAMA, 333(4), 319–328. https://doi.org/10.1001/jama.2024.21700
  5. Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1), 64–93. https://doi.org/10.1093/jla/laae003
  6. Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630, 625–630. https://doi.org/10.1038/s41586-024-07421-0
  7. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & Liu, T. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2), 1–55. https://doi.org/10.1145/3703155
  8. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why language models hallucinate (arXiv:2509.04664) [Preprint]. arXiv. https://arxiv.org/abs/2509.04664
  9. Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025). Hallucination-free? Assessing the reliability of leading AI legal research tools. Journal of Empirical Legal Studies, 22(2), 216–242. https://doi.org/10.1111/jels.12413
  10. Maslej, N. (2025, April 7). AI Index 2025: State of AI in 10 charts. Stanford Institute for Human-Centered Artificial Intelligence. https://hai.stanford.edu/news/ai-index-2025-state-of-ai-in-10-charts
  11. Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia, 14 February 2024).
  12. Pandit, S., Xu, J., Hong, J., Wang, Z., Chen, T., Xu, K., & Ding, Y. (2025). MedHallu: A comprehensive benchmark for detecting medical hallucinations in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 2858–2873). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.emnlp-main.143

How to cite this page

Choose your referencing style, then copy.

PhD Research Lab Insights. (2026, October 5). Hallucination in high-stakes AI: can we measure how often a deployed language model gets it dangerously wrong?. PhD Research Lab. https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use/

PhD Research Lab Insights. “Hallucination in High-stakes AI: Can We Measure How Often a Deployed Language Model Gets It Dangerously Wrong?.” PhD Research Lab, 5 Oct. 2026, www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use. Accessed [access date].

PhD Research Lab Insights (2026) Hallucination in high-stakes AI: can we measure how often a deployed language model gets it dangerously wrong?. PhD Research Lab. Available at: https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use/ (Accessed: [access date]).

PhD Research Lab Insights. “Hallucination in High-stakes AI: Can We Measure How Often a Deployed Language Model Gets It Dangerously Wrong?.” PhD Research Lab. October 5, 2026. https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use.

PhD Research Lab Insights, ‘Hallucination in High-stakes AI: Can We Measure How Often a Deployed Language Model Gets It Dangerously Wrong?’ (PhD Research Lab, 5 October 2026) <https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use> accessed [access date].

PhD Research Lab Insights. Hallucination in high-stakes AI: can we measure how often a deployed language model gets it dangerously wrong? [Internet]. PhD Research Lab; 2026 Oct 5 [cited [access date]]. Available from: https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use/

PhD Research Lab Insights, “Hallucination in high-stakes AI: can we measure how often a deployed language model gets it dangerously wrong?,” PhD Research Lab, Oct. 5, 2026. [Online]. Available: https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use. [Accessed: [access date]].

@online{llm_hallucination_high_stakes_use_2026,
  author = {{PhD Research Lab Insights}},
  title = {{Hallucination in high-stakes AI: can we measure how often a deployed language model gets it dangerously wrong?}},
  organization = {PhD Research Lab},
  date = {2026-10-05},
  url = {https://www.phdassistance.com/insights/r-and-d/research-gap/llm-hallucination-high-stakes-use},
  urldate = {[access date]}
}

  • Large language models
  • AI hallucination
  • Trustworthy AI
  • Clinical AI safety
  • AI evaluation
Hallucination in high-stakes AI: can we measure how often a deployed language model gets iResearch gap · Computer science
Proposal support