# Preparing documents for ingestion

> How to write, structure and manage source documents so retrieval works: single-document design, multi-document corpora, metadata, versions, format pitfalls and an ingestion pipeline with tests.

Source: https://conversationalaidesign.com/modules/07-ingestion-documents.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026.

## What you will learn
- Write a single source document that retrieves and answers well
- Design a multi-document corpus with metadata, versions and supersession
- Avoid the extraction pitfalls of PDF, Word, slides, spreadsheets, HTML and tickets
- Build an ingestion pipeline with stable IDs, validation and re-ingestion
- Test a corpus with golden questions before launch


## Why this is the highest-leverage work in RAG

In module 6 we saw that most retrieval quality comes from the content, not the model. This module is the practical guide to that content: how to write one document well, how to run a whole library of them, and how to get them into an index safely.

> **The Air Canada lesson, at corpus scale**
> In the Air Canada case, two parts of one website disagreed and the assistant repeated the wrong one. Ingestion is where you prevent that: one source of truth, version control and explicit scope.

## Scenario 1: a single document

Imagine one return-policy document that must answer questions well. Write it so each section can be understood alone, because a chunk is often all the model sees.

**Structure**

- **A clear title** that names the topic and audience: "Return policy for online orders, United States."
- **A metadata block** at the top: document ID, owner, version, effective date, region, audience, status.
- **A heading hierarchy** with one topic per section. Headings should be descriptive: "Returns after 30 days," not "Exceptions."
- **The answer first.** Open each section with the rule, then the detail.
- **Scope in every section** where it could be ambiguous: "Applies to items bought online. In-store purchases follow the store policy."
- **Definitions** for terms you rely on ("delivery date means the carrier's confirmed delivery").
- **Explicit conditions** written as full sentences, not fragments or tables of abbreviations.
- **Numbered steps** for procedures, with one action per step.
- **Questions as headings** for FAQs, with the answer directly beneath.

**Content hygiene**

- Use real text, not screenshots of text. Add alt text and a text description for any image that carries information.
- Put tables in a machine-readable form (a Markdown table or a CSV) and add a sentence that states what the table means.
- Spell out acronyms on first use.
- Remove repeated headers, footers, page numbers and navigation text.
- Keep links explicit: "See the Warranty policy (document WAR-004)" is better than "click here."

A compact template that works well:

```markdown
---
doc_id: RET-001
title: Return policy for online orders (United States)
version: 4.2
status: current
effective_from: 2026-09-01
supersedes: RET-001 v4.1
region: US
audience: customers
language: en
owner: customer-care
---

# Return policy for online orders (United States)

## Can I return an item after 30 days?
No. Returns are accepted within 30 days of the delivery date, with proof of purchase.
This applies to items bought online in the United States. In-store purchases follow the store policy (STO-002).

## What counts as the delivery date?
The delivery date is the date the carrier confirms delivery, not the order date.

## How do I start a return?
1. Open your order in your account.
2. Choose "Return an item" and select the reason.
3. Print the prepaid label or choose a drop-off location.
```

## Scenario 2: many documents

A real knowledge base has hundreds or thousands of documents, written by different people over years. Now the problems are conflict, freshness and findability.

**Governance**

- **One source of truth per topic.** Name an owner. If three documents describe the same policy, merge them or mark two as derived.
- **Versions and effective dates.** Every document carries a version and a validity window. Retrieval filters to the current version unless the question is historical.
- **Supersession.** A new version names what it replaces, and the old one is marked superseded, not silently deleted. Keep history for audits.
- **Lifecycle.** Draft, reviewed, current, superseded, retired. Only "current" is retrievable by default.

**A metadata schema**

| Field | Purpose | Example |
|---|---|---|
| `doc_id`, `version` | Identity and history | `RET-001`, `4.2` |
| `status` | Lifecycle filter | `current` |
| `effective_from` / `effective_to` | Validity window | `2026-09-01` |
| `region`, `market` | Correct jurisdiction | `US` |
| `audience` | Customers, staff, partners | `customers` |
| `language` | Retrieval and answer language | `en` |
| `product` or `topic` | Scoping | `online-orders` |
| `doc_type` | Policy, how-to, FAQ, price list | `policy` |
| `owner` | Who fixes it | `customer-care` |
| `sensitivity` and access groups | Retrieval-time permissions | `public` |

**Design decisions**

- **Split or merge?** Split huge documents by topic so chunks stay focused. Merge many tiny fragments that only make sense together.
- **Cross-references.** Use stable document IDs, not "see the page on the left."
- **Multiple languages.** Prefer parallel documents with the same `doc_id` and a `language` field, translated from an approved source, over machine-translating at question time.
- **Permissions.** If some content is staff-only, enforce access at retrieval time using the user's permissions, not just in the prompt.

## Format pitfalls

| Source | What breaks | What to do |
|---|---|---|
| **PDF** | Scanned pages have no text. Multi-column layouts interleave. Tables flatten. Headers and footers repeat. Words hyphenate across lines | Prefer the original source. Use OCR for scans and check it. Use layout-aware extraction and inspect the output |
| **Word** | Tracked changes and comments leak into text. Styles are inconsistent | Accept or reject changes first. Use real heading styles |
| **Slides** | Text is fragmentary and depends on visuals | Convert to prose, add speaker notes and an explicit summary |
| **Spreadsheets** | Merged cells and implicit headers lose meaning | Convert to tidy tables with one header row, add a description |
| **HTML and wikis** | Navigation, cookie banners and macros pollute text | Extract the main content only and strip boilerplate |
| **Email and tickets** | Quoted replies, signatures and threads duplicate text | Keep the resolution and the customer question, drop quoted history |
| **Transcripts** | Filler, no structure, speaker confusion | Summarize into structured notes and keep the transcript as a source |

## The ingestion pipeline

- Acquire → Extract text and structure → Clean and normalize → Enrich with metadata → Chunk with context → Embed and index → Validate with test questions → Monitor and re-ingest changes

A sketch of the middle steps. It validates required metadata, builds chunks that carry their context, and gives each one a stable ID so updates and citations stay consistent:

```python
import hashlib

REQUIRED = {"doc_id", "version", "status", "effective_from", "region", "audience", "language", "owner"}

def validate_meta(meta: dict) -> list[str]:
    missing = sorted(REQUIRED - meta.keys())
    problems = [f"missing metadata: {', '.join(missing)}"] if missing else []
    if meta.get("status") == "current" and meta.get("effective_to"):
        problems.append("current document has an end date")
    return problems

def chunk_id(doc_id: str, version: str, section_path: str, index: int) -> str:
    raw = f"{doc_id}|{version}|{section_path}|{index}"
    return hashlib.sha1(raw.encode("utf-8")).hexdigest()[:16]

def build_chunks(meta: dict, sections: list[tuple[str, str]], max_words: int = 220):
    """sections: list of (section_path, text). Returns chunk records ready to embed."""
    problems = validate_meta(meta)
    if problems:
        raise ValueError(f"{meta.get('doc_id', '?')}: " + "; ".join(problems))
    records = []
    for path, text in sections:
        words, i = text.split(), 0
        for n, start in enumerate(range(0, max(len(words), 1), max_words)):
            piece = " ".join(words[start:start + max_words])
            records.append({
                "id": chunk_id(meta["doc_id"], meta["version"], path, n),
                "text": f"{meta['title']} > {path}\n{piece}",     # context travels with the chunk
                "meta": {**meta, "section": path},
            })
    return records
```

**Re-ingestion.** Hash each source document. Re-process only what changed, delete the chunks of superseded versions (or mark them non-current), and keep a log of what was added, changed and removed.

## Test the corpus before launch

1. **Golden questions.** For each important document, write the questions customers actually ask, with the expected passage.
2. **Retrieve, then read.** Run each question through retrieval alone and read the top results. Is the right passage first? Is a stale one present?
3. **Conflict scan.** Search for the same fact across documents and check agreement.
4. **Freshness check.** List documents past their review date.
5. **Access test.** Confirm a user without permission can't retrieve restricted chunks.
6. **Not-found test.** Ask questions the corpus can't answer, and confirm the assistant says so (see module 11).

Turn the questions into specs (module 15) and run them on every content release.

## Sensitive content

Strip or tokenize personal data that doesn't belong in a knowledge base, carry access-control metadata through every chunk, honor deletion and retention requests all the way to the vector index, and don't ingest content you aren't permitted to use. See module 18.


## Key takeaways
- Retrieval quality is decided by the documents long before the model sees them.
- Write for retrieval: one topic per section, the answer first, scope and dates explicit.
- At corpus scale, metadata, versioning and a single source of truth matter more than volume.
- Treat ingestion as a pipeline with validation, stable IDs and tests, not a one-time upload.

## Sources
- Ragas: evaluation for RAG: https://docs.ragas.io
- OWASP GenAI Security Project: vector and embedding weaknesses: https://genai.owasp.org
- Case study: Air Canada: https://conversationalaidesign.com/case-studies/air-canada-chatbot.html