From the introduction
The evidence habit
Most AI product mistakes do not start with a bad model.
They start with an unsupported claim.
The prompt worked once, so the prompt works.
The JSON parsed, so the extraction is correct.
The model sounded confident, so the answer is trustworthy.
The demo looked good, so the feature is ready.
The eval score went up, so the product improved.
Every one of those sentences may be true. None of them is proven by the sentence itself.
No claim without evidence.
That rule changes how you write prompts, design schemas, build fallbacks, structure review, read eval scores, and decide whether a release is safe enough to ship.
Imagine a system that extracts structured fields from airline tickets, receipts, invoices, medical documents, customer emails, or support transcripts. The user does not care that the model generated clean JSON. The user cares whether the flight number, PNR, terminal, amount, passenger name, and date are correct enough for the next workflow step.
A clean output is not the same as a true output.
An LLM can produce a field that looks plausible but is not supported by the source document. A fallback model can fill a blank field by guessing. A reviewer can accept a value because it feels right. A benchmark can report high accuracy while hiding the one failure mode that would damage user trust.
That is why AI engineering needs more than prompts. It needs evidence paths.
From Chapter 1
A claim is a value plus a reason to trust it
A product does not merely display values; it causes people and systems to act on them. A flight number may drive a status lookup. A departure date may schedule a notification. A baggage allowance may become a customer-facing promise. The moment an extracted value affects a decision, the value is a claim.
A claim has at least four parts:
- Value: What does the system say?
- Provenance: Where did that value come from?
- Interpretation: Was it copied, normalized, inferred, or reviewed?
- Decision: What may the product do with it?
A value is not ready merely because it matches a schema. It is ready when the value and its provenance satisfy the product’s policy for the action that follows.
Passenger: Riya Mehta
Booking Ref / PNR: H7K29Q
Flight: AI 202
Date: 12 Aug 2026
Departure: Delhi
Arrival: Bengaluru
Departure Time: 09:40
Terminal: not shown
Baggage: not shown{
"passenger_name": "Riya Mehta",
"flight_number": "AI 202",
"departure_date": "2026-08-12",
"origin": "DEL",
"destination": "BLR",
"pnr": "H7K29Q",
"terminal": "Terminal 3",
"baggage": "15kg"
}At a glance, this looks good. It is complete. It is clean. It has the right shape. But terminal and baggage are unsupported. The document explicitly does not show them.
If the workflow accepts those values, it has crossed a line. The model did not extract. It inferred.
Extraction means the value came from the source. Normalization means a supported source value was transformed under a declared rule, such as Delhi to DEL. Inference means the value came from reasoning, prior knowledge, or a guess. Inference is not always wrong, but it must be labeled and governed by a different contract.
{
"field": "terminal",
"raw_value": null,
"normalized_value": null,
"status": "not_present",
"evidence": [],
"action": "accept_with_warning",
"notes": "Do not infer from airport or route"
}The operational distinction
The question is not whether “Terminal 3” happens to be true in the real world. The extraction product is claiming what the document says. Real-world plausibility cannot substitute for document evidence.
From Chapter 15
Test the pipeline, not just the model
When an AI workflow fails, the model is only one suspect.
The defect may live in document intake, OCR, layout parsing, chunking, prompt construction, model selection, schema validation, JSON repair, evidence linking, normalization data, routing, fallback, stop conditions, review instructions, or a downstream API.
The user does not experience a model in isolation. The user experiences the pipeline.
upload
-> document classification
-> OCR and layout extraction
-> field candidate extraction
-> schema validation
-> evidence linking
-> normalization
-> routing and stop policy
-> human review when required
-> final decision
-> downstream itinerary recordModel layer
Prompts, model choice, decoding, candidate extraction, and evidence selection.
Pipeline layer
Intake, OCR flow, validation, retries, fallback, routing, review, storage, and audit.
Data/configuration layer
Schemas, dictionaries, normalization maps, field policies, thresholds, templates, and scoring rules.
case_id: AIR-088
source_facts:
departure_time_candidates: ["09:40", "21:40"]
expected:
final_status: review_required
reason: conflicting_departure_times
actual:
extracted_candidates: ["09:40", "21:40"]
schema_status: valid
final_status: accepted
reason: schema_valid
failure_layer: pipeline.routing_policyThe model succeeded: it preserved both departure times. The pipeline failed by treating schema validity as permission to accept.
A model upgrade does not fix a broken lookup table. A better prompt does not fix missing review policy. A passing extraction does not prove production readiness.
From Chapter 19
Your eval should become a release gate
The mature end state is not an eval dashboard.
The mature end state is release discipline.
When a prompt changes, the eval runs. When a schema changes, the eval runs. When a model changes, the eval runs. When a routing rule, lookup table, review policy, or stop condition changes, the eval runs.
Is this version of the product workflow safe enough to ship?
A gate is a decision, not a score. No single aggregate score can represent every failure cost. A system could improve average field accuracy while introducing one unsupported PNR or unauthorized booking. Averages can hide blockers.
A useful gate therefore combines quality thresholds, operational limits, and zero-tolerance blockers.
release_gate: ticket_extraction_v4
risk_class: high
must_pass:
critical_field_accuracy: ">= 98%"
unsupported_inference_rate: "<= 0.5%"
evidence_link_coverage_critical: ">= 99%"
schema_validity: "100%"
regression_cases_pass: "100%"
review_limits:
review_rate: "<= 12%"
operations:
p95_latency_ms: "<= 5000"
average_cost_usd: "<= 0.025"
blockers:
- any customer-facing claim without evidence
- any do-not-infer field accepted without source evidence
- any booking performed without required approval
- any critical regression case failureWhere the argument lands
- No extraction without evidence
- No confidence without status
- No fallback without routing policy
- No review without rules
- No agent action without permission
- No score without logs
- No release without a gate
Reusable artifact
The claim-evidence ledger
The ledger creates a discipline: every accepted claim must answer “why may the product trust this?” It also gives future evals something concrete to score—evidence coverage, unsupported inference, review decisions, and policy compliance.
claim_id:
document_id:
field:
value:
claim_type: extracted | normalized | inferred | human_corrected
source_locator:
evidence_excerpt:
evidence_status: supported | not_present | unreadable | ambiguous | conflicting
product_policy:
decision: accept | accept_with_warning | reject | review
decision_reason:
verified_by:
verified_at:Continue reading
The full book carries this method through 19 chapters and two appendices.
Continue into error taxonomies, eval-set design, cost and latency, fallbacks, human review, observability, agent action evals, configuration discipline, the complete airline-ticket capstone, and practical templates.
Checking your regional price…
One-time purchase · PDF + EPUB
Secure checkout with PDF and EPUB delivery after payment. The displayed price follows your selected region; taxes, when applicable, are shown at checkout.