Veridi methodology: what the numbers actually mean

Veridi is an LLM-grounded fact-verification system that produces evidence-quality-graded verdicts with calibrated confidence and abstains when the evidence will not support a verdict. The system reports four numbers on its public calibration page:

• 99.5% overall accuracy (99 correct, 1 partial across the 100-claim Veridi validation set: 95 original validation claims plus 5 targeted extension claims dated 2026-05-04)
• 0.0253 selective Brier, 95% CI [0.0193, 0.0320] (calibration on the 89 verdicts where the system committed)
• 89% coverage (the rate at which the system commits rather than abstains)
• 11/11 abstention correctness (when the system declined to commit, it was the right call)

These are baseline measurements on the curated Veridi validation set, covering the Veridi verification methodology at the time of measurement (the live methodology is now v1.2). The Pragma and Praxis assessment frameworks, which run on top of Veridi for policy and individual-action synthesis, have their own 95-claim benchmark and their own version lines: 87 of 95 combined claims passed at the v1.0 baseline (2026-03-21), and a focused regression dated 2026-05-26 under the current Pragma v1.6 and Praxis v1.4 resolved six of the eight prior partial passes with no regressions, for 93 of 95. The two residual partials are Praxis pathway-ranking edge cases, root-caused with a methodology-refinement ticket open; they are documented as methodology gaps, not waved through.

The Veridi suite consists of three independent methodology lines, all current as of May 2026:

• Veridi (verification): v1.2. What is true?
• Pragma (policy synthesis): v1.6. What should be done?
• Praxis (action synthesis): v1.4. What can I do?

Two independent benchmarks:

• Veridi calibration set (100 claims): Smoke Tests (1) + Golden Test Sets A-C (70) + GTS-D Wave 1 targeted extension (5, 2026-05-04) + Adversarial Test Suites v1-v2 (24).
• Pragma + Praxis combined benchmark (95 claims): Pragma 55 (15 adversarial + 25 golden + 15 boundary) + Praxis 40 (10 adversarial + 20 golden + 10 boundary).

Veridi is in invite-only beta as of May 2026, run as a Canadian cooperative in formation. The methodology is published; the production service runs with admin-issued access codes. The verification pipeline operates primarily in English and outputs, as nearly as is practical, in the language of the submitted claim; the user-facing app supports English and French.

What a “claim” means here

A claim is a discrete proposition that a statement asserts and that can be evaluated against evidence. “The 2024 election in Country X had a turnout of Y%” is one claim. “X is undermining democracy” is not a claim in the schema; it is an interpretive frame that decomposes into multiple claims.

Decomposition is itself a methodological step. The pipeline takes a piece of content, identifies the claims it asserts, and routes each claim to the appropriate verification track. Decomposition errors show up as misrouting (a claim sent to the wrong specialist) or as incomplete coverage (an asserted claim missed entirely). Both are tracked.

The 95-claim benchmark is split between the Pragma and Praxis frameworks. Pragma claims (n=55) test policy-relevant assertions across domains. Praxis claims (n=40) test individual-action recommendations. Each suite is structured: 15 adversarial + 25 golden + 15 boundary in Pragma; 10 adversarial + 20 golden + 10 boundary in Praxis. Both share the verification methodology; they differ in what evidence counts as load-bearing and what threshold resolves “supported.”

What the numbers mean separately

Fact-checking systems that can answer “I don’t know” should be evaluated on two questions, not one:

1. When the system commits to a verdict, is it calibrated? Does it assign higher confidence when it is right and lower confidence when it is wrong, in a way that tracks observed frequency?
2. When the system abstains, was that the correct call? Did it correctly identify, and decline to commit on, claims that genuinely could not be resolved with the available evidence?

A single composite Brier score lumps these together. It penalizes a correct abstention (the system says UNVERIFIABLE at low confidence on a claim that genuinely has no resolution) the same as a wrong commitment (the system says FALSE at high confidence on a claim that turns out true). For a fact-checker that treats abstention as a first-class output, that is the wrong measurement.

Veridi follows the selective-classification framing from Fisch et al. (2022), Calibrated Selective Classification, and reports the two questions separately:

• Selective Brier on committed verdicts: 0.0253 [0.0193, 0.0320]. Computed only on the 89 verdicts where the system committed; abstentions are excluded.
• Abstention correctness on the 11 abstained claims: 11/11. The question is whether the abstention was right, not whether the system was calibrated on a verdict it did not produce.

The raw Brier across all 100 claims (0.0745, 95% CI [0.0456, 0.1087]) is reported alongside, since prior literature uses it, but it is the lower-quality signal. The selective framing is the load-bearing one.

Reference frame for the selective Brier number: a calibration metric where 0.0 is perfect and 1.0 is worst. Crowd-level forecasting accuracy sits around 0.15. Superforecaster baseline is around 0.10. Naive 50%-on-binary baseline is 0.25.

What we measured

The 100-claim Veridi calibration set (95 original validation claims plus 5 targeted extension claims) was selected for verifiability. Claims were drawn from cases where ground truth could be established with reasonable confidence by independent reviewers using documented evidence trails. This biases the set toward claims that are resolvable in principle and against the genuinely-ambiguous cases that real-world fact-checkers face daily. The bias is disclosed in the limitations section below; the test set’s job is to measure how well the methodology does on the resolvable cases, not to be representative of all claims encountered.

Composition and methodology versions at validation:

• Smoke Tests (1 claim). Methodology sanity check.
• Golden Test Set A (25 claims). Validated against Veridi methodology v1.0.
• Golden Test Set B (25 claims). Validated against v2.2.
• Golden Test Set C (20 claims). Validated against v2.4.
• GTS-D Wave 1 (5 claims, targeted extension dated 2026-05-04). Adds coverage on recent-ambiguity intent attribution, non-US legal status, manipulated-media provenance, hazard-versus-risk framing, and policy-label compression. Validated against v1.1 current runtime instructions.
• Adversarial Test Suite v1 (12 claims). Targets the 11 documented gaming vectors. Validated against v1.0.
• Adversarial Test Suite v2 (12 claims). Same vectors with improved abstention discipline. Validated against v2.2.

Production methodology is currently v1.2; the v2.x labels above are as-recorded validation tags from the retired system-version stream (see the version history below). Re-validation against v1.2 is planned; the published numbers reflect the as-recorded validation at the version each suite was originally evaluated against.

The Pragma + Praxis benchmark was validated separately against its own suites; the PARTIAL → PASS arc is documented later in this piece.

Calibration data: tables and decomposition

Per-confidence-band calibration (committed verdicts only):

Calibration gap = observed accuracy − mean confidence; positive values indicate under-confidence (system was right more often than it claimed).

The pattern across bands is systematic under-confidence: at every band, observed accuracy exceeds mean confidence. The 90-100% band is closest to perfectly calibrated; lower bands are progressively more under-confident. This is partly an artifact of the test set being too uniformly correct (99/100 verdicts correct, 1 partial, weighted accuracy 99.5%). A system that is almost always right cannot meaningfully be over-confident at any band. The under-confidence finding is real, the magnitude is partly inflated by test-set selection, and harder validation claims are needed to separate the two effects (see the methodology refinement backlog).

By test suite (the gaming-detection improvement story, made quantitative):

The ADV-v1 → ADV-v2 arc is the lived gaming-detection improvement story: same 11 vectors, same 100% verdict accuracy, but Brier dropped 21x as the methodology learned to abstain confidently rather than commit hesitantly on adversarial claims. ADV-v1’s high Brier with 100% accuracy is the structural penalty raw Brier imposes on correct-but-low-confidence verdicts, which the selective framing does not penalize.

By domain (committed verdicts):

The “general” domain (claims that don’t map to a specialist) is the weakest at Brier 0.2093 and 65.0% mean confidence. Technology is the next-weakest. Medical, electoral, and historical (n<5) are the strongest. The differential is itself a feature for an evaluating reader: the system’s weak spots are reported, not hidden.

Murphy decomposition (Brier ≈ Reliability − Resolution + Uncertainty; Murphy 1973), on committed verdicts. Murphy splits the Brier score into three components: Reliability (how well stated confidence tracks observed frequency), Resolution (whether confidence discriminates correct from incorrect outcomes), and Uncertainty (baseline variability from outcome unpredictability). On a near-uniformly-correct test set, Resolution is expected to approach zero; that is the finding, not a flaw.

The Resolution ≈ 0 finding is structurally the most important data point on this page, and it is a finding about the test set rather than the methodology. Resolution measures whether confidence discriminates between outcomes the system gets right versus wrong. With 99/100 verdicts correct + 1 partial, there is almost no wrong-cohort against which confidence can discriminate. The selective Brier framing plus decomposition made this visible; classical raw Brier would have hidden it. A harder validation suite is needed to exercise Resolution.

By verdict type (committed verdicts; UNVERIFIABLE row included for contrast):

The UNVERIFIABLE row’s Brier of 0.4922 is the structural penalty raw Brier imposes on correct abstentions: a verdict at 30.8% mean confidence on a claim that is in fact unverifiable scores as nearly as wrong as a high-confidence wrong verdict, even though the abstention itself was correct. This row is the artifact the selective-classification framing exists to remove from the headline number.

Comparison to the human baseline

The closest peer to Veridi is professional fact-checking at IFCN-certified organizations (the International Fact-Checking Network’s certified professional fact-checkers). The published time and cost numbers for that workflow:

• Time per claim: 1.5 to 4 hours, simple to complex, excluding editorial review (the second-analyst review pass).
• Cost per claim: $75 to $200 USD at professional analyst rates ($40 to $60 per hour loaded, varying by organization and geography; $50 per hour as a midrange).
• Inter-rater agreement: Lim 2018 found that when PolitiFact and the Washington Post Fact Checker both rated the same claim (n=77 Overlap statements), they agreed on the rating in 49 of 77 cases: 63.6% raw agreement on a converted 5-point comparable scale, 36.4% disagreement. Cohen’s κ on that scale is 0.467 unweighted, 0.750 weighted. Disagreements were predominantly minor (78% off by one scale-point). Agreement drops further in the ambiguous middle of the scale (“Half True” / “Mostly False”): 14 of 23 cases, 60.9%. On a binary collapse (Amazeen 2016 scheme, True vs. all else), Cohen’s κ rises to 0.787.
• Bottleneck: evidence gathering is the largest time sink; gaming detection requires the most expertise and is the most inconsistently applied. Trained analysts develop gaming-detection judgment over years.

Veridi’s targets, set against that baseline:

• Time per claim: under 5 minutes at Standard tier; under 10 at Full; under 15 at Forensic.
• Cost per claim: target less than or equal to 5% of human analyst cost at equivalent tier.
• Verdict accuracy target: at least 90%, with an explicit caveat from the system specification: 90% is achievable for binary or 3-point collapsed verdicts. On the full 10-verdict taxonomy (TRUE / MOSTLY TRUE / MISLEADING / LACKS CONTEXT / MIXED / MOSTLY FALSE / FALSE / OUTDATED / UNVERIFIABLE / CONTESTED DEFINITION), 90% exceeds known human inter-rater agreement and should be treated as aspirational.

The 99.5% measured accuracy on the validation set sits in that aspirational zone. The number is real on the test set as constructed, and the test set was selected for verifiability (claims where ground truth could be established). On the full distribution of claims a human fact-checker would handle, the realistic accuracy ceiling is bounded by human inter-rater agreement on the same claims. The selective-classification framing addresses this directly: when a claim is genuinely ambiguous, the right output is abstention with calibrated low confidence, not a forced verdict.

The gaming-detection result is independently meaningful. Across both adversarial test suites (24 claims targeting the 11 documented gaming vectors), the methodology achieved 100% verdict accuracy with full vector-detection coverage, against a target of at least 85%. The system specification identifies gaming detection as the most expertise-dependent and most inconsistently applied step in human fact-checking. The methodology’s structured countermeasure scan exists to address that inconsistency directly.

Methodology versioning

Three methodology lines are in production, each on its own release cadence:

Veridi (verification methodology). v1.0 (foundational) → v2.2 (post-audit hardening of source-independence checks) → v2.4 (added specialist routing for nine domain agents) → v2.6 (added Step 0 conditional claim decomposition, a methodology self-reference / conflict-of-interest gate, value-judgment classification, and a promotional-framing checklist) → v2.7 (added the substrate-self-reference trigger, which flags claims where the assessor model evaluates a claim about itself or its developer) → v2.8 (search-mandate hardening, a retrieval-grounding gate, declared bot user-agent on outbound web requests, and retrieval-bypass output discipline). The v2.x system-version stream was retired in May 2026: Veridi now carries a single v1.x methodology version line, and a v2.x number is not comparable to (and not newer than) a v1.x entry. Current production: v1.2, promoted 2026-05-20.

Pragma (policy synthesis methodology). v1.0 → v1.1 (initial audit remediation) → v1.2 (closed the audit backlog with three structural-framing fixes, integrated Olson collective-action dynamics, and added dynamic implementation-risk factors) → v1.3 (statistical-discipline pass: tightened triggers in the cross-policy comparison section) → v1.4 (test-retest variance protocol; reconciliation of the adversarial test cohort) → v1.5 (added an adoption-readiness ladder modeled on StrongREJECT, a published adversarial-jailbreak benchmark) → v1.6 (added a multi-trait multi-method (MTMM) protocol document for construct validity, and a multi-value-frame panel design). v1.2.1 through v1.2.3 are app-side cross-reference patches with no methodology content changes. Current methodology version: v1.6, dated 2026-05-01.

Praxis (action synthesis methodology). v1.0 → v1.1 → v1.2 (golden-scenario fixes: action-ranking reform, a financial-capacity rule, a leverage-tier-one bonus, and a decision-authority modifier) → v1.3 (close-out manifest: WHO checklist generalization, Brier-lite operational threshold tuning, and Three-Gate runtime expansion) → v1.4 (StrongREJECT readiness pass; MTMM protocol document; verbalized-confidence-channel adoption with Spotlighting input-datamarking, plus five new indirect-prompt-injection (IPI) scenarios). v1.2.1 through v1.2.3 are app-side cross-references. Current methodology version: v1.4, dated 2026-05-01.

The published Veridi calibration numbers reflect validations at v1.0, v2.2, and v2.4 against their original test suites. Production has since moved to the v1.2 methodology line. Re-validation against v1.2 is on the methodology refinement backlog; the calibration page reports under as-recorded framing rather than as-current until that re-validation lands.

The Pragma + Praxis combined benchmark stands at 93 of 95 claims under the current versions, measured by a focused regression dated 2026-05-26: the eight partial passes recorded at the v1.0 baseline (2026-03-21, 87/95) were fully re-evaluated under Pragma v1.6 and Praxis v1.4, six resolved to PASS, two Praxis pathway-ranking cases remain partial (root-caused; refinement ticket open), and six spot-checks confirmed no regressions. The v1.2 changelog had predicted all six Praxis partials would resolve; the measured regression corrected that prediction, and the correction is recorded in the validation results. Both current versions also add new evaluation surface (the StrongREJECT-readiness ladder and the MTMM construct-validity protocol) that has not yet been run end-to-end against the benchmark.

Source hierarchy

Veridi uses a 4-tier source hierarchy. Each tier carries different weight; not every claim needs all tiers; mixing tiers is the common case.

• Tier 1: Primary peer-reviewed. Original research, replicated where possible. Weighted highest. Coverage gaps are real for non-English-language sources and for emerging domains where the peer review pipeline has not caught up.

• Tier 2: Trustworthy institutional. Reports from institutions with track records (multilateral bodies, established research organizations, statistical agencies). Weighted high. Failure mode: institutional capture, tracked as one of the thirteen documented gaming vectors.

• Tier 3: Reasonable secondary. Reporting from publications with documented editorial standards, cited primary sources. Weighted moderately. Used for breadth where tier 1 and 2 are sparse.

• Tier 4: Other. Everything else, including social media, commentary, and unverified claims. Used as a flag that something exists; not load-bearing for verification.

A claim is supported when load-bearing evidence is concentrated in tiers 1 and 2, with no contradiction in tier 1 sources. Claims supported only in tiers 3 and 4 are flagged as such; the system does not represent them as verified.

Confidence ceilings by sourcing. Each tier mix carries a maximum confidence the verdict can claim, regardless of how strong the evidence-quality assessment was on its own:

The verdict’s final confidence is min(tier_ceiling, evidence_assessment). The model’s evidence-quality assessment can lower the confidence below the ceiling but cannot exceed it.

Field reliability is a disclosure annotation, not a multiplier. Some fields produce more replicable findings than others; the methodology reports the field reliability coefficient alongside the confidence number, but does not fold it into the math. Disclosing the variability without pretending to quantify it inside a single number is the discipline call.

Breaking-event ceiling. Claims about events less than 72 hours old are capped at 75% confidence on peripheral facts and 65% on narrative framing, regardless of source tier. The temporal ceiling addresses the failure mode where rushed verdicts on developing stories propagate before counter-evidence surfaces.

Pipeline architecture

The Veridi system runs as a two-tier model dispatch with seven system-level agents. The workhorse is Claude Sonnet 5 (migrated from Sonnet 4.6 in June 2026), handling the bulk of verification. The advisor is a higher-capability model tier, invoked only on uncertain or high-risk claims for review. Escalation logic references tiers (workhorse / advisor), not vendor names; swapping models or providers is a config change, not a rewrite.

The seven system-level agents (per AGENT-PRODUCT-SPEC.md v1.4, dated 2026-05-01):

• A1 (Claim Normalizer). Python. Normalizes input text, computes a SHA-256 hash for cache lookup, enforces per-user rate limits, creates the database record.
• A2 (Tier Dispatcher). Python. Loads the methodology files for the requested verification tier (Quick, Standard, Full, or Forensic), constructs the system prompt, dispatches to the processing backend.
• A3 (Fact-Check Orchestrator). Workhorse LLM. The core agent. A single LLM call with an agentic tool-use loop (web_search, web_fetch, up to 25 tool-use rounds). The methodology stages execute sequentially within this one call (described below).
• A4 (Result Parser + Escalation). Python. Extracts verdict and confidence via regex, evaluates escalation triggers. Two escalation paths: auto-escalate from Standard to Full when gaming flags fire, or invoke the advisor for boundary verdicts and high-uncertainty claims.
• A5 (Follow-Up Handler). Workhorse LLM, reasoning only. Handles user follow-up questions against existing evidence. No new tool use.
• A6 (Advisor). Advisor LLM, reasoning only. Reviews workhorse output on uncertain or high-risk claims, concurs or revises with explicit reasoning delta. Target advisor revision rate: 10 to 25%.
• A7 (Brier Accumulator). Python. Records the prediction tuple for calibration tracking. Brier score target: less than or equal to 0.15 (crowd-level forecasting accuracy; superforecaster baseline, where superforecasters are expert human forecasters from Tetlock’s Good Judgment Project research, is around 0.10).

Inside A3, the methodology runs as sequential stages within the single LLM call:

• A3a (triage): classify domain, complexity, and mode (breaking event vs. standard).
• A3b (evidence gatherer): targeted web search and source reading; trace citation chains to origin.
• A3c (source evaluator): 4-tier hierarchy assignment, independence verification, IRI (Institutional Reliability Index) check for government sources.
• A3d (gaming scanner): 15-item quick checklist at Standard tier; full 13-vector scan at Full or Forensic.
• A3e (specialist): conditional, for complex claims or Forensic tier. Eight domain specialists are available at Forensic tier: Scientific, Medical, Legal, Financial, Electoral, Historical, Technology, Propaganda. A Breaking Event Analyst loads on a separate trigger (events less than 72 hours old) regardless of tier.
• A3f (verdict and confidence): decision tree traversal; confidence is the minimum of the tier ceiling and the evidence assessment; field reliability is annotated.
• A3g (output composer): structured format with verdict, evidence, gaming flags, field reliability, and limitations.

The workhorse-then-advisor split is the central architectural decision. Most claims do not need the advisor tier. Routing only the uncertain ones up costs less than running every claim at the higher tier; a fraction of the bill produces most of the calibrated-confidence value. Trade-off: this design assumes the workhorse’s uncertainty signal is itself well-calibrated. If workhorse confidence is poorly calibrated, the advisor either gets too much false-escalation noise or misses claims that should have been reviewed. The Brier Accumulator (A7) is partly there to detect calibration drift over time.

Methodology loading is cost-proportional by tier. The Tier Dispatcher (A2) loads only the methodology files needed for the requested verification depth:

The token cost of methodology loading scales with the rigor demanded; a Quick-tier verification does not pay the cost of a Forensic-tier specialist load.

Auto-escalation as post-hoc re-dispatch. When a gaming flag fires at Standard tier, the system cannot load additional methodology mid-conversation (the system prompt is immutable once an API call begins). The mechanism is a fresh A2-to-A3 dispatch at Full tier with re-gathered evidence; the Standard-tier output is discarded rather than carried forward, preventing evidence contamination. Cost implication: auto-escalated claims pay for two workhorse calls. At an approximately 15% auto-escalation rate on Standard claims, this adds about $0.03 amortized per Standard claim; the cost is already accounted for in the Full-tier frequency rows of the cost model.

Vendor-neutral architecture. The escalation logic references model tiers (workhorse / advisor), not vendor names. The configuration maps tiers to vendor-specific model IDs at runtime via environment variables: VERIDI_PROVIDER selects the vendor, VERIDI_WORKHORSE_MODEL and VERIDI_ADVISOR_MODEL select the model IDs. The current Anthropic mapping is claude-sonnet-4-6 (workhorse) and claude-opus-4-6 (advisor); a possible OpenAI mapping is gpt-4o-mini (workhorse) and o3 or gpt-4o (advisor). Swapping vendors means changing environment variables, not code.

Audit-trail discipline in storage. A4 stores the workhorse result immediately upon parsing. If the advisor is invoked and revises the verdict, the workhorse output and the raw API response are preserved in full_output and raw_response columns; only the verdict and confidence fields update. The original assessment is recoverable. If the advisor concurs, no fields change but advisor_used=1 and advisor_action='concur' mark that the review happened.

Gaming countermeasures

Veridi documents thirteen specific manipulation vectors in Gaming_Countermeasures.md. Five examples that show the shape of the territory:

Confidence Laundering. A claim cites multiple sources that all trace back to a single origin (a single study, a single press release, a single Wikipedia edit), inflating apparent independent support. Countermeasure: trace every citation chain to its terminus; when sources resolve to the same origin, collapse them to one for confidence purposes.

Citogenesis. A claim circulates through a citation loop where Wikipedia, social media, and news coverage cite each other without an independent investigation anywhere in the chain. Countermeasure: check Wikipedia edit history against citation timestamps; flag wire-service-only coverage (one AP story republished by twelve outlets) as one source; identify suspiciously clustered publication times.

Tier Inflation. A low-tier source (commentary, social media, press release) is presented as if it were a tier-1 primary source. Countermeasure: tier-aware source assignment with explicit reasoning when a source is downgraded.

Framing Manipulation. A claim is structured to make a contested term load its premise, or to frame the absence of a unifying name as the substantive issue when the actual situation is naming competition. Countermeasure: term-resolution and definitional decomposition before verification.

Anchoring. A piece presents true, verifiable claims (anchors) that prop up weaker or false claims around them. The reader extends credibility from the anchor to the surrounding material. Countermeasure: evaluate each sub-claim independently; flag anchoring patterns where strong claims share scope with weak claims.

The full thirteen cover eight additional vectors (UNVERIFIABLE-by-Design, Preprint Pump-and-Dump, Selective Skepticism Exploitation, Coordinated Legitimate Sourcing, Data Disappearance Exploitation, Institutional Capture, Substrate Self-Reference, and Warm-up-then-Defect). The two newest show where the frontier sits: Substrate Self-Reference flags claims where the assessor model evaluates a claim about itself or its developer; Warm-up-then-Defect (added 2026-05-19) covers per-user trust gaming across submissions, where benign early requests build standing for a later adversarial one. The Veridi adversarial test suite covers these vectors with a target detection rate of at least 85%. Suite size has grown from the 26 claims at the spring-2026 validation (the substrate-self-reference test among them): five indirect-prompt-injection (IPI) scenarios were added in May 2026, mapped to five NIST AI RMF subcategories with corresponding MITRE ATLAS technique IDs. Current suite: 31 claims, verified 2026-07-03 against the suite files.

Non-English source coverage (research-stage)

Production-grade AI systems document their coverage boundaries explicitly. Here is one of ours.

Verification accuracy depends heavily on whether the system can actually read the sources a claim cites. The Veridi app currently supports English and French as user-facing languages. The verification pipeline operates in English; non-English source coverage is research-stage planning, not implemented capability.

The research file non-english-claims.md (compiled 2026-02-25) identifies test-claim candidates in five language groups, each chosen because the load-bearing evidence for the claim lives in non-English sources:

• Chinese / Mandarin (3 candidate claims). Examples include the China youth-unemployment-data suspension and Xinjiang demographic statistics. Foundational evidence sits in Chinese-language government statistical yearbooks; English coverage typically cites derived figures without showing original source data.
• Spanish / Portuguese (3 candidate claims). Examples include the Venezuela 2024 election dispute and Brazil 2022 election claims.
• Arabic (3 candidate claims). Examples include the Al-Ahli hospital explosion attribution and religious-disinformation claims. Direct authoritative documents are available only in Arabic.
• Japanese / Korean (2 candidate claims). Fukushima treated-water-release framing and the South Korea martial law crisis disinformation.
• Hindi / Urdu (2 candidate claims). “Muslim population explosion” claims in India and India GDP-data-inflation claims.

What this is not: in-language verification has not been implemented. There are no in-language specialist agents in the codebase. The 95-claim Pragma + Praxis benchmark is English. The non-English research is a roadmap for how the methodology would extend to languages where source-content match cannot be approximated by translation.

What this is: documentation of a coverage gap and a planned-but-not-shipped capability. A claim arriving in Mandarin today routes through English-source paths; the verifier returns a calibrated answer that is, on inspection, only as confident as the English source coverage justifies. This is a design choice: scrupulous factual bounds to claimed or implied coverage.

In-language verification is needed for genuine source-content match where translation cannot approximate the original. The path to ship it is not on the immediate roadmap; it sits behind the harder-validation-suite work in the methodology refinement backlog.

What the methodology fixed

The validation report identifies specific test failures and the methodology updates that resolved them. Documenting the changes is the maturity claim.

The eight PARTIALs in the earlier benchmark (Pragma 53/55, Praxis 34/40 at v1.0) traced to specific methodology gaps. v1.2 ships targeted fixes referenced in the report by ID: improvements to verdict-decision-tree edge handling, boundary-case clarifications, and specific test-case pattern fixes. All eight prior PARTIALs trace to v1.2 PASS. No new regressions. The two residual non-passes are the documentation reconciliations noted at the top.

The PARTIAL designation had been working as a soft hide before v1.2: a claim could pass as PARTIAL when the methodology did not have a clean answer, and the soft hide masked the methodology gap. The fix at v1.2 was treating PARTIAL as a flag for follow-up rather than as a terminal verdict. The eight PARTIALs that resolved to PASS in v1.2 are the audit trail.

That changelog is the discipline. A methodology that does not identify specific failures and specific fixes is not measuring itself. The Veridi validation report does, and the sample is small enough (eight resolved PARTIALs across 95 claims) to be inspectable.

Limitations

The 100-claim validation set has explicit limitations. The first five are imported from the public calibration page; the rest carry forward from the prior version of this piece.

Selection bias. All 100 claims have known ground truth, selected for verifiability. Real-world claims include more genuinely ambiguous cases than any test set can capture. The dataset measures calibration on resolvable claims, not on the full distribution of claims the system would encounter in production.

The test set is near-uniformly correct. 99 of 100 verdicts correct plus 1 partial. Murphy Resolution lands at approximately zero because there is almost no wrong-cohort against which confidence can discriminate. Harder validation claims are needed to exercise Resolution. The methodology refinement backlog (next section) flags this as priority 1. GTS-D Wave 1 (2026-05-04) is the first targeted extension toward harder claims; it added five claims at boundary verdict categories, all of which passed. A larger harder-validation suite (working name GTS-D Wave 2 or RES-v1) remains the priority-1 backlog item.

Same-model evaluation, partially addressed. The validator (Claude Opus 4.6) and the system under test share a model family. Systematic shared biases would not be detected by Claude-validates-Claude. A first cross-model run against Gemini 2.5 Pro on an 8-claim subset (with 2 timeouts) returned 50% exact verdict agreement, 9.1-point mean confidence delta, and zero contradictory verdicts. The 50% agreement rate is below the 85% acceptance threshold; the disagreements are themselves informative (Veridi tended toward MISLEADING where Gemini said FALSE on directional cases; Veridi tended toward UNVERIFIABLE where Gemini said LACKS CONTEXT on structural cases). Cross-model expansion is methodology refinement backlog priority 3.

Small abstention sample. 11 abstentions is suggestive of good abstention discipline but statistically thin. Test cases designed to tempt over-abstention and over-commitment are planned.

No temporal drift data. Validation runs were conducted in a few sessions across 2026-02 to 2026-03. There is no data on whether calibration drifts over extended use or across model updates.

Domain coverage beyond the tested set. The eight domain specialists at Forensic tier cover Scientific, Medical, Legal, Financial, Electoral, Historical, Technology, and Propaganda. A Breaking Event Analyst loads conditionally on a separate trigger (events less than 72 hours old). Coverage in pure science (especially fast-moving subfields), claims at the intersection of two specialists, and historical claims with contested interpretations is partial. The methodology generalizes; the domain-specific tooling does not do so automatically.

Cost-quality trade-offs at production scale. The production service runs invite-only with admin-issued codes because cost performance has not been validated under unrestricted load. Monetization will be needed for sustainability.

Drift handling at the per-source level. Source hierarchies are not static. Trustworthy institutional sources change. The methodology has a documented review cadence; drift handling at the per-source level is currently manual. Worth automating in the future.

Methodology bias. I am the lead architect. The framework prioritizes evidence types and source hierarchies that match my training and understanding (Western academic-track epistemology, English-primary). The framework documents this bias and routes around it where possible. It cannot eliminate it.

In-language verification. Documented above as research-stage. The gap between the methodology’s documented capability (“we plan to handle non-English claims via in-language source search with specialist routing”) and its shipped capability (“the pipeline runs in English”) is the most visible deferred work.

Methodology refinement backlog

The 2026-04-22 refinement backlog (Veridi/fact-checker-files/validation-results/methodology-refinement-2026-04-22.md) lists three prioritized refinements with concrete acceptance criteria. They surfaced from the selective-Brier and Murphy decomposition analysis above.

Priority 1: Harder validation claims to exercise Murphy Resolution. Build a new validation suite (working name GTS-D or RES-v1) of approximately 20 claims deliberately calibrated so the methodology is expected to land at 70 to 85% accuracy at best. Genuinely contested topics, evidence-ambiguous cases, borderline verdict-category cases. Acceptance: at least 5 with divided-reviewer ground-truth labels; selective Brier on the new suite has Resolution greater than 0.03 (real discriminative power), or the methodology cannot discriminate and that itself is publishable.

Priority 2: Confidence floor for committed verdicts. The calibration-coverage curve shows Brier improves smoothly as the commitment threshold rises (≥50% Brier 0.0253; ≥70% Brier 0.0199; ≥85% Brier 0.0096). The methodology has tier-specific confidence ceilings but no floor. Three candidate rules under evaluation: auto-escalate Standard commitments below 60% to Full; force abstention below a 50% floor at Full or Forensic; or display a “provisional” badge on commitments below 70% with explicit limitation note. Acceptance: pick one (likely the first or second), re-run on a 20-claim subset that previously produced sub-70% commitments, adopt if selective Brier improves by more than 0.005 without losing good commitments.

Priority 3: Cross-model validation expansion. The 8-claim Gemini run is the first pass. Expand to a 20-claim representative mix across test suites and domains; prioritize contested or definitional claims, politically-sensitive domains, and domains where training-data skew is plausible. Run through a non-Claude model (GPT-4 / GPT-5, Gemini 2.x, or Llama 3 for open-weight reproducibility). Acceptance: verdict agreement greater than 90% on non-contested claims; disagreements on contested claims are documented, not explained away. Estimated cost: $20 to $40 in API calls.

The backlog itself is published; effort estimates and dependencies are documented per priority. The methodology improves through inspection, not through claims about its rigor.

Verify and reproduce

The validation data and methodology files are published. To check or replicate the numbers in this piece:

• Raw 100-row JSONL data. One claim per line; includes verdict, confidence, and outcome. Available from the Veridi calibration page. The first 95 rows are the original validation set; rows 96 through 100 are the GTS-D Wave 1 targeted extension (2026-05-04).
• Validation reports. Veridi calibration analysis (calibration-analysis-2026-04-05.md) and the methodology refinement backlog (methodology-refinement-2026-04-22.md) are checked in to the dev tree. The Pragma + Praxis v1.2 validation lives across the suite-specific results files (adv-results.md, gts-results.md, bnd-results.md for Pragma; adversarial-results.md, golden-results.md, boundary-results.md for Praxis) plus the scoping plan at plan-pragma-praxis-v1.2-2026-04-24.md. The cross-model raw data against Gemini 2.5 Pro (cross-model-raw-2026-04-09.json) is alongside.
• Compute script. Veridi/fact-checker-files/validation-results/compute-calibration.py produces the published calibration numbers from the JSONL.
• Methodology corpus. Approximately 22,500 lines across the Veridi, Pragma, and Praxis frameworks, 74 documents total (verified against dev tree 2026-04-30; Veridi ~7,000 lines / 31 documents + Pragma ~8,500 lines / 26 documents + Praxis ~7,000 lines / 17 documents). Recent additions through April and May 2026 include the StrongREJECT-readiness ladder, the MTMM construct-validity protocol, an Inspect AI evaluation specification, and the indirect-prompt-injection test scenarios. The system specification is a 1,968-line agent product spec at v1.4, dated 2026-05-01.

References:

• Fisch, A., Jaakkola, T., and Barzilay, R. (2022). Calibrated Selective Classification. The selective Brier framing for systems that can abstain.
• Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology 12: 595-600. The Reliability / Resolution / Uncertainty decomposition cited above.
• Dimitriadis, T., Gneiting, T., and Jordan, A. I. (2021). Stable reliability diagrams for probabilistic classifiers. PNAS. Cited for future work on stable reliability diagrams under sparse data.
• Lim, C. (2018). Checking how fact-checkers check. Research and Politics 5(3). Cohen’s κ for PolitiFact vs. Washington Post Fact Checker on a converted 5-point comparable scale (0.467 unweighted, 0.750 weighted); raw agreement on Overlap statements 49 of 77 (63.6%); agreement in the ambiguous range 14 of 23 (60.9%); binary collapse κ 0.787 using Amazeen 2016’s coding scheme.
• Amazeen, M. A. (2016). Checking the fact-checkers in 2008: Predicting political ad scrutiny and assessing consistency. Journal of Political Marketing 15(4): 433-464. The binary True-vs-all-else coding scheme that Lim 2018 applies to compute κ = 0.787.

Net read

These are real measurements against published benchmarks. The selective Brier of 0.0253, the abstention correctness of 11/11, the Pragma + Praxis v1.2 pass rate of 93 of 95, and the documented limitations and refinements all live in checked-in files in the dev tree. The benchmarks are published; the methodology corpus (approximately 22,500 lines across Veridi, Pragma, and Praxis, 74 documents total, verified against dev tree 2026-04-30) is published; the architecture is published in a 1,968-line product specification at v1.4.

The fact-checking ecosystem is full of unverified claims about verification systems. The full semantic content of these numbers is: here are the benchmarks, here is the framing under which they should be read, here is the architecture, here is what is shipped versus research-stage, here is what got fixed, here is what is still open, here is what is on the refinement backlog, here are the references.

If you are building verification systems, the through-line is: specify the benchmarks, document the architecture, separate shipped from planned, publish the failure modes, list the refinements, and let other people check. I think the verification industry would benefit from broader adoption of this discipline. The case is in the artifacts above; check, then decide.

Work with me

If your team is sizing a Reliability Layer for a production AI deployment, I take on a small number of contract reviews per quarter through Nettercap Productions (my independent consulting practice); see availability for scope, fit, and pricing.

If you are sizing a senior-to-staff Applied AI engineering hire and the methodology above maps to your runtime concerns, availability covers role types, locations, and the hiring-inquiry email path.