A Claim-Level Framework for Factual Reliability, Context Adherence, Data Freshness, Access Constraints, and Human Feedback
Kishore Chalakkal Varghese
ORCID: 0000-0002-0356-566X
https://doi.org/10.5281/zenodo.21313179
Abstract
Large Language Models can produce fluent answers that are not fully supported by facts, source documents, or the operational context in which they are used. Existing research evaluates related phenomena through truthfulness benchmarks, attribution frameworks, citation-quality benchmarks, atomic factuality metrics, black-box consistency checks, and retrieval-augmented generation evaluation. This paper proposes a Verifiable Hallucination Risk Index, HRI, for enterprise AI systems. The index is not presented as a universal benchmark, but as an auditable, claim-level scoring framework that combines factual inconsistency, context deviation, unsupported claims, data freshness, scope mismatch, access or permission gaps, and validated human feedback. Compared with model-level hallucination benchmarks, the proposed HRI treats hallucination risk as a system-level reliability problem emerging from the interaction between model behavior, available evidence, governance constraints, and human validation. The paper contributes an operational claim scoring method, a numerical example, a provisional risk-band model, and a source verification protocol. Empirical validation and calibration of thresholds remain future work.
Keywords: hallucination risk, factuality, groundedness, context adherence, RAG evaluation, AI governance, claim verification, human feedback
I. Introduction
Large Language Models, LLMs, have created new opportunities for natural language interfaces over documents, data, and enterprise knowledge bases. Their usefulness, however, depends on whether generated answers remain truthful, attributable, and appropriate to the available evidence. The hallucination problem is commonly described as the generation of plausible but nonfactual or unsupported content; recent survey work treats hallucination as a central reliability issue for LLMs and discusses taxonomy, detection, mitigation, and open questions [1].
In enterprise environments, hallucination should not be reduced to a model-only defect. A response may be unreliable because the model reasons incorrectly, because retrieved context is incomplete, because the request is outside the intended scope, because the system lacks permission to use the relevant source, or because the available data are stale. This paper therefore frames hallucination as a system-level risk emerging from the interaction of model behavior, evidence availability, retrieval quality, governance constraints, and human validation.
The proposed Hallucination Risk Index, HRI, is a claim-level framework. It decomposes an AI answer into atomic claims, labels each claim against evidence and operational constraints, assigns severity values, and aggregates them into an auditable risk indicator. The HRI is not a replacement for existing benchmarks such as TruthfulQA, SelfCheckGPT, RAGAs, ALCE, or FActScore. Rather, it is a governance-oriented layer intended for enterprise use cases where auditability, scope control, permission boundaries, and human review are operational requirements.
II. Related Work
TruthfulQA measures whether language models generate truthful answers to questions designed to expose common human false beliefs. The benchmark contains 817 questions across 38 categories, including health, law, finance, and politics [2]. SelfCheckGPT proposes a black-box, zero-resource method for hallucination detection based on consistency among multiple sampled responses from the same model [3].
RAGAs addresses evaluation of Retrieval-Augmented Generation, RAG, systems, providing reference-free metrics for dimensions such as context relevance, faithfulness, and answer quality [4]. ALCE focuses on citation-enabled long-form generation and evaluates citation quality, correctness, and fluency [5]. FActScore decomposes long-form generations into atomic facts and computes the percentage supported by a reliable knowledge source [6].
Industrial tools and leaderboards also use hallucination-related metrics. Galileo describes a Hallucination Index based on metrics such as Correctness and Context Adherence [7]. The Vectara hallucination leaderboard provides a practical benchmark and repository for measuring hallucination behavior in grounded generation settings [8]. Such sources are useful for industry context, but they should not be treated as independent academic validation of the HRI proposed here.
The governance motivation for the HRI is also consistent with risk-oriented AI management frameworks. In particular, the NIST AI Risk Management Framework provides a reference context for treating AI trustworthiness and reliability as risk-management concerns rather than purely technical accuracy problems [9].
The distinction between factual error and context deviation in the HRI is aligned with the broader separation between intrinsic and extrinsic hallucinations discussed in the hallucination literature [1]. The present framework extends that distinction with enterprise-specific dimensions: data freshness, request scope, access permissions, and validated human feedback.
III. Comparative Positioning
Table I positions the proposed HRI against related evaluation approaches. The comparison is intentionally high level: the objective is not to claim superiority, but to clarify the specific enterprise-governance dimensions addressed by the HRI. The compact column labels correspond to the formal HRI dimensions introduced in Table II.
Table I
Comparative Positioning of the Proposed HRI
| Framework | Factuality | Context / Grounding | Freshness | Scope | Permissions | Human Feedback |
|---|---|---|---|---|---|---|
| TruthfulQA [2] | Yes | No | No | No | No | No |
| SelfCheckGPT [3] | Partial | Partial | No | No | No | No |
| RAGAs [4] | Partial | Yes | Partial | No | No | No |
| ALCE [5] | Partial | Yes, via citation support | No | No | No | No |
| FActScore [6] | Yes | Partial | No | No | No | No |
| Galileo / Vectara [7], [8] | Yes | Yes | Partial | No | No | Partial |
| Proposed HRI | Yes | Yes | Yes | Yes | Yes | Yes, if validated |
IV. Definition of the HRI
The Hallucination Risk Index is defined as a composite indicator estimating the probability and severity of unsupported, incorrect, stale, context-inconsistent, out-of-scope, or access-constrained claims generated by an AI system under a given operational setting.
[
HRI_{add}(r) = \min\left(1, \sum_{k \in K} w_k s_k(r)\right)
]
where (r) is the generated response, (K={F,C,U,O,S,P,H}) is the set of risk dimensions, (w_k) are non-negative weights, (\sum_k w_k=1), and (s_k(r)\in[0,1]) is the normalized score for dimension (k).
Table II
HRI Risk Dimensions
| Symbol | Dimension | Meaning |
|---|---|---|
| (F) | Factual inconsistency | Claim contradicts a reliable source. |
| (C) | Context deviation | Claim deviates from the retrieved or provided context. |
| (U) | Unsupportedness | Claim lacks explicit evidence. |
| (O) | Obsolescence | Claim depends on data that may be stale. |
| (S) | Scope mismatch | Claim or request is outside the approved use case. |
| (P) | Permission gap | Relevant evidence exists but cannot be accessed or used. |
| (H) | Validated human feedback | Previously confirmed human corrections or negative feedback. |
V. Operational Claim-Level Scoring
To make the HRI replicable, a generated response is decomposed into (n) atomic claims:
[
C_r={c_1,\ldots,c_n}
]
Each claim receives one or more labels after comparison with evidence, retrieval context, freshness metadata, scope rules, permission rules, and validated feedback.
A claim may contribute to more than one risk dimension simultaneously. This is an intentional design choice, not an accidental double count: factual inconsistency, context deviation, unsupportedness, obsolescence, scope mismatch, permission gaps, and feedback events are not mutually exclusive in operational settings. For example, a claim contradicted by the retrieved contract may affect both factual inconsistency (F) and context deviation (C).
During pilot annotation, each claim should be labeled independently by at least two annotators when feasible. Disagreements may be resolved by majority vote, expert adjudication, or a documented escalation rule. Inter-annotator agreement should be measured and reported in empirical validation studies, since the reliability of HRI depends on the consistency of claim extraction and labeling.
For each claim (c_i), dimension (k), and criticality weight (v_i), the contribution (m_{ik}\in[0,1]) is assigned according to a labeling rubric. The dimension score is:
[
s_k(r)=\frac{\sum_{i=1}^{n} v_i m_{ik}}{\sum_{i=1}^{n} v_i}
]
where (v_i=1) by default. Higher values of (v_i) should be reserved for claims with higher operational consequence, such as legal, financial, safety, contractual, or compliance-critical claims.
Table III
Default Claim-Level Contributions
| Claim Label | Default Contribution | Typical Dimension |
|---|---|---|
| SUPPORTED | 0.00 | All |
| PARTIAL | 0.50 | (C) or (U) |
| UNSUPPORTED | 0.75 | (U) |
| CONTRADICTED | 1.00 | (F), and (C) if contradicted by context |
| OUTDATED | 0.75 | (O) |
| OUT_OF_SCOPE | 0.75 | (S) |
| ACCESS_GAP | 0.60 | (P) |
| VALIDATED_NEGATIVE_FEEDBACK | 1.00 before decay | (H) |
The above values are initial design parameters, not empirically validated constants. They should be calibrated with domain-specific datasets, expert review, and incident history.
A. Human Feedback Score
Human feedback should not be used as a raw popularity signal. The (H) dimension should include only validated feedback, meaning feedback confirmed by a reviewer, source check, or adjudication process. A time-decayed score can be used:
[
s_H=\frac{\sum_{j=1}^{m} q_j e^{-\lambda \Delta t_j}}{\max(1,N_{reviewed})}
]
where (q_j) is the severity of the validated feedback event, (\Delta t_j) is its age, (\lambda) is a decay parameter, and (N_{reviewed}) is the number of reviewed responses in the relevant evaluation window. This avoids indefinitely penalizing a corrected system and reduces the risk of accepting erroneous feedback as ground truth.
VI. Additive Model, Safety Floors, and Limitations
A linear weighted aggregation is adopted as an initial design choice because it is interpretable, auditable, and easy to explain in enterprise governance contexts. However, a purely additive formula may dilute a single severe contradiction when many low-risk claims are present. It may also hide correlations among dimensions. For example, unsupportedness and factual inconsistency are not independent: a claim without evidence may be more likely to be false, but it is not necessarily contradicted.
To reduce dilution of critical errors, the additive score can be combined with a claim-level safety floor:
[
HRI^*(r)=\max\left(HRI_{add}(r),\max_i \phi(c_i)\right)
]
where (\phi(c_i)) is a minimum risk floor associated with severe labels and their criticality. Table IV provides provisional floor values intended to make the rule replicable during pilots. They require empirical calibration before operational adoption.
Table IV
Provisional Claim-Level Safety Floors
| Label | Criticality Condition | Floor (\phi(c_i)) | Rationale |
|---|---|---|---|
| CONTRADICTED | High, (v_i\geq3) | 0.70 | Severe contradiction in a critical claim. |
| CONTRADICTED | Medium, (v_i=2) | 0.50 | Confirmed contradiction requiring review. |
| ACCESS_GAP | High, (v_i\geq3) | 0.55 | Evidence exists but cannot be accessed or lawfully used. |
| OUT_OF_SCOPE | High, (v_i\geq3) | 0.55 | Request exceeds approved use case or policy boundary. |
| VALIDATED_NEGATIVE_FEEDBACK | Repeated or unresolved | 0.50 | Confirmed unresolved feedback indicates persistent system risk. |
These floor values complement rather than replace the additive score. Future work should compare this hybrid approach with max-based aggregation, multiplicative penalties, and interaction-aware models.
VII. Numerical Example: Supplier-Contract RAG Scenario
Consider an enterprise RAG assistant answering a question about a supplier contract. The answer is decomposed into six claims. Criticality weights are assigned by the evaluator, with higher values for operationally relevant claims.
Table V
Illustrative Claim-Level Scoring for a Supplier-Contract RAG Answer
| # | Atomic Claim | Label | Risk Dimension | Contribution | Criticality (v_i) |
|---|---|---|---|---|---|
| 1 | The contract expires on 31 December 2026. | SUPPORTED | — | 0.00 | 1 |
| 2 | The supplier has ISO 27001 certification. | UNSUPPORTED | (U) | 0.75 | 2 |
| 3 | The SLA includes 24/7 support. | CONTRADICTED | (F,C) | 1.00 | 3 |
| 4 | The pricing is still valid for the current year. | OUTDATED | (O) | 0.75 | 2 |
| 5 | The assistant can disclose the supplier security questionnaire. | ACCESS_GAP | (P) | 0.60 | 2 |
| 6 | The answer gives legal interpretation of liability clauses. | OUT_OF_SCOPE | (S) | 0.75 | 1 |
With:
[
\sum v_i=11
]
the dimension scores are:
[
s_F = 3/11 = 0.273
]
[
s_C = 3/11 = 0.273
]
[
s_U = 1.5/11 = 0.136
]
[
s_O = 1.5/11 = 0.136
]
[
s_P = 1.2/11 = 0.109
]
[
s_S = 0.75/11 = 0.068
]
[
s_H=0
]
Using provisional weights:
[
w_F=0.25
]
[
w_C=0.20
]
[
w_U=0.15
]
[
w_O=0.10
]
[
w_S=0.10
]
[
w_P=0.10
]
[
w_H=0.10
]
the additive score is:
[
HRI_{add}=0.175
]
or:
[
17.5/100
]
However, because claim 3 is a high-criticality contradiction, the safety-floor rule applies. If:
[
\phi(c_3)=0.70
]
then:
[
HRI^*=\max(0.175,0.70)=0.70
]
The final operational interpretation is therefore high risk requiring human review, despite the lower additive score. This illustrates why the HRI should not rely on an additive average alone.
VIII. Risk Bands and Calibration
Table VI proposes initial risk bands and recommended actions.
Table VI
Provisional HRI Risk Bands
| HRI Score | Risk Band | Recommended Action |
|---|---|---|
| 0–20 | Low | May be used with normal logging. |
| 21–50 | Medium | Require source visibility or user caution. |
| 51–75 | High | Require human verification before use. |
| 76–100 | Critical | Do not use without formal review. |
The proposed bands are initial design thresholds and should not be interpreted as empirically validated cut-offs. Future calibration should rely on datasets with known outcomes, such as confirmed human corrections, audit findings, incident reports, or domain-expert adjudications. Receiver operating characteristic analysis, precision-recall analysis, and cost-sensitive threshold selection may be used to set deployment-specific thresholds.
IX. Weight Elicitation
In the absence of historical incident data, weights may be elicited through structured expert judgment. Practical methods include Delphi rounds with risk, compliance, IT, legal, and business stakeholders; Analytic Hierarchy Process, AHP, pairwise comparisons; or risk workshops using representative scenarios. When enough labeled cases become available, weights should be recalibrated empirically.
Weights should also reflect domain risk. In safety, legal, financial, or compliance settings, factual contradiction and permission gaps may deserve higher weights. In knowledge-search settings, unsupportedness and context adherence may be more important. The HRI should therefore be documented as a domain-specific measurement protocol, not as a fixed universal score.
X. Enterprise Evaluation Workflow
Figure 1 shows the proposed evaluation workflow for enterprise AI systems.
Figure 1
HRI Evaluation Workflow for Enterprise AI Systems
User Request
↓
Scope Check
↓
Permission Check
↓
Retrieval and Evidence Collection
↓
Atomic Claim Extraction
↓
Claim Labeling Against Evidence
↓
HRI Scoring and Safety Floor
↓
Risk Band
↓
Release, Caution, Block, or Review
↓
Validated Human Feedback
↺
Feedback returns to HRI scoring and future calibration
The dashed path in the formal figure represents validated human feedback feeding subsequent scoring and calibration.
XI. Source Verification Protocol
A paper or system using the HRI should maintain a source verification register. Each factual claim should be linked to a specific source, and the source should be classified by type: peer-reviewed publication, official framework, vendor documentation, repository, or original proposal. Claims that cannot be verified should be removed, softened, or explicitly marked as hypotheses.
The following principles are recommended: use primary sources where possible; do not cite vendor material as independent validation; distinguish empirical evidence from design proposals; record access date for mutable web sources; and separate source-supported claims from author interpretations.
XII. Source Verification Register
Table VII lists major claims used in this paper, their source type, and their verification status.
Table VII
Source Verification Register
| Paper Claim | Source Type | Verification Status |
|---|---|---|
| TruthfulQA contains 817 questions across 38 categories. | ACL paper | Verified in source [2]. |
| SelfCheckGPT is a black-box, zero-resource hallucination detection approach based on sampled-response consistency. | EMNLP paper | Verified in source [3]. |
| RAGAs provides reference-free evaluation for RAG pipelines. | EACL paper / arXiv | Verified in source [4]. |
| ALCE evaluates citation-enabled generation, including citation quality. | EMNLP paper | Verified in source [5]. |
| FActScore decomposes text into atomic facts and estimates source-supported factual precision. | EMNLP paper | Verified in source [6]. |
| The proposed HRI combines factuality, context, unsupportedness, freshness, scope, permissions, and feedback. | Original contribution | Not a literature claim; explicitly proposed here. |
| Risk bands and scoring constants are valid thresholds. | — | Not claimed. They are provisional design parameters requiring empirical validation. |
XIII. Minimum Evidence Rules
The source verification protocol should be supported by minimum evidence rules. These rules define the lowest acceptable evidence type for different classes of claims and prevent the paper or the deployed evaluation process from treating unverifiable assertions as established facts.
Table VIII
Minimum Evidence Rules
| Claim Type | Minimum Acceptable Evidence | Required Handling if Evidence is Missing |
|---|---|---|
| Scientific or methodological claim | Peer-reviewed paper, recognized conference proceedings, or clearly marked preprint when no peer-reviewed source exists. | Remove, soften, or mark explicitly as hypothesis or open question. |
| Benchmark property or numerical result | Primary benchmark paper, official dataset card, or official repository documentation. | Do not report the number; state only the qualitative point if supported. |
| Vendor or product capability | Official vendor documentation, release note, or product page with access date. | Classify as vendor claim, not independent validation. |
| Governance or risk-management framework | Official framework document from the issuing body. | Do not paraphrase as regulatory requirement unless the source itself states it. |
| Internal enterprise policy or permission rule | Approved policy, access-control record, system log, or authorized process owner confirmation. | Classify as ACCESS_GAP or UNKNOWN rather than infer authorization. |
| Original HRI design choice | Explicit statement that the element is proposed by this paper. | Do not present it as an established standard or empirically validated result. |
XIV. Pre-Submission Claim-Verification Checklist
Before publication or internal deployment, the HRI protocol should be accompanied by a lightweight audit checklist. The checklist is not a substitute for empirical validation, but it helps ensure that the paper, benchmark report, or enterprise evaluation record does not introduce unsupported claims.
Table IX
Pre-Submission Claim-Verification Checklist
| No. | Check |
|---|---|
| 1 | Every factual claim is linked to a primary or acceptable secondary source. |
| 2 | Every numerical value, benchmark property, and threshold is either sourced or explicitly marked as provisional. |
| 3 | Vendor claims are labeled as vendor claims and are not presented as independent validation. |
| 4 | Original elements of the HRI are described as proposals, design choices, or hypotheses, not as established standards. |
| 5 | Ambiguous or unverifiable claims are removed, softened, or moved to future work. |
| 6 | Mutable web sources include access dates in the reference list or verification register. |
| 7 | Claim labels, criticality weights, and safety floors are reproducible from the documented rubric. |
XV. Limitations
This paper proposes a methodological framework, not a validated benchmark. The scoring constants, weight values, safety floors, and risk bands are provisional. Human feedback can be biased or incorrect if not validated. Claim extraction may itself introduce error, especially when answers contain implicit claims, compound claims, or ambiguous wording. Permission and scope rules may vary across organizations and jurisdictions. Finally, the relationship among HRI dimensions may be non-linear, requiring future empirical study.
XVI. Future Work
Future work should validate the HRI on labeled enterprise datasets, compare additive and non-additive aggregation models, calibrate risk bands against known outcomes, measure inter-annotator agreement for claim labels, test the role of criticality weights, and evaluate how HRI-guided feedback loops affect answer quality over time. Empirical studies should also compare HRI against existing RAG and factuality metrics under controlled settings.
XVII. Conclusion
The Hallucination Risk Index proposed in this paper reframes hallucination as an enterprise reliability problem. Instead of asking only whether a model hallucinated, the framework asks whether each claim is supported, current, in scope, permitted, context-consistent, and responsive to validated human correction. The contribution is an auditable claim-level method that combines established concerns from factuality and RAG evaluation with enterprise governance dimensions. The framework is intentionally conservative: it distinguishes verified evidence from original proposal, marks its thresholds as provisional, and identifies empirical validation as the next necessary step.
References
[1] L. Huang et al., “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” ACM Transactions on Information Systems, 2025.
[2] S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” in Proceedings of ACL 2022, 2022.
[3] P. Manakul, A. Liusie, and M. J. F. Gales, “SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models,” in Proceedings of EMNLP 2023, 2023.
[4] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “RAGAs: Automated Evaluation of Retrieval Augmented Generation,” in Proceedings of EACL 2024 Demo, 2024.
[5] T. Gao, H. Yen, J. Yu, and D. Chen, “Enabling Large Language Models to Generate Text with Citations,” in Proceedings of EMNLP 2023, 2023.
[6] S. Min et al., “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation,” in Proceedings of EMNLP 2023, 2023.
[7] Galileo, “Introducing the Hallucination Index,” 2023. Accessed: July 1, 2026.
[8] Vectara, “Hallucination Leaderboard,” GitHub repository. Accessed: July 1, 2026.
[9] NIST, “Artificial Intelligence Risk Management Framework, AI RMF 1.0,” 2023.