How to build a legal knowledge base whose answers can be backed up: parsing with Docling, chunking along section and subsection, citation metadata, and the checks that should run after every ingest.
At Erst Recht, we are building an AI platform for initial legal assessments under German law. One building block is a retrieval-augmented knowledge base, and we convert legal PDFs to Markdown with Docling. I have fundamentally revised this post. Instead of a workshop report, it is now a checklist for legal knowledge bases, collected from practice and primary sources.
Why Legal Texts Are Different
A legal AI is only as good as its sources. A hit "somewhere in the BGB" (German Civil Code) is not enough. The answer has to point to the provision, subsection and sentence, and for court decisions to the court, date, docket number and paragraph number. Three properties make this hard:
- Deep hierarchy: statute, chapter, section, subsection, sentence, item. A subsection without its section is worthless as a source.
- Dense abbreviations: "Abs.", "S.", "Nr.", "i.V.m.", "a.F.". Every naive sentence splitter trips over them.
- Chains of references: provisions refer to provisions, judgments cite judgments. A chunk without context breaks the chain.
Ignore this and you get a search that finds similar words but no citable source.
The Pipeline at a Glance
From PDF to semantic search, there are five stations. The domain logic in the middle decides whether you end up with citable chunks:
Parsing: Docling Instead of Plain Text Extraction
Docling comes out of the "AI for Knowledge" group at IBM Research Zurich, is MIT-licensed and is now hosted by the LF AI & Data Foundation. It is not "better OCR" but a document understanding pipeline:
| Plain text extraction / OCR | Docling |
|---|---|
| Returns flat text | Returns a structured document |
| Loses headings and hierarchy | Detects headings, lists, outline |
| Reads multi-column pages line by line | Detects reading order |
| Tables turn into garbled text | Tables stay tables |
Three decisions are worth making before the first run:
- OCR only where needed. Digitally created PDFs have a text layer, scans do not. Docling supports several OCR engines, including EasyOCR, Tesseract and RapidOCR. Decide per document source, not across the board.
- Table detection on. Deadline, fee and overview tables show up in legal documents more often than you would think. Docling has a dedicated table model with a fast and an accurate mode.
- Markdown for review, DoclingDocument for chunking. The Markdown export is ideal for checking results by eye. For chunking, the structured DoclingDocument is the better basis, because it knows the hierarchy and origin of every element, including the page.
Structure-Aware Chunking
Statutes: Fetch Them Structured Instead of Parsing
For federal law, the PDF is a detour. Gesetze im Internet offers the laws as XML downloads. The DTD structures them into norm elements with metadata such as the abbreviation (jurabk), the unit designation (enbez), title, outline unit and version note. The statute text sits in paragraph elements (P).
The chunking almost follows by itself: one chunk per subsection, numbered lists stay with their subsection, and sentences are recorded as positions within the chunk. This is what a chunk for a provision can look like:
{
"quelle": "gesetz",
"gesetz": "BGB",
"norm": "§ 626",
"titel": "Fristlose Kündigung aus wichtigem Grund",
"absatz": 1,
"gliederung": "…",
"stand": "…",
"text": "Das Dienstverhältnis kann von jedem Vertragsteil …"
}Two things save you a lot of pain later. First, the version: laws change, so the version note is stored, and on an update the new chunk replaces the old one instead of landing next to it. Otherwise two versions of the same provision sit in the index. Second, the references: if a subsection refers to another provision via "i.V.m." (in conjunction with), that provision belongs on the chunk as metadata. Then retrieval can load the referenced provision on purpose.
Judgments: Along the Structure of the Decision
German civil judgments follow a structure set by law: § 313 ZPO requires, among other things, the operative part, the statement of facts ("Tatbestand") and the reasons for the decision ("Entscheidungsgründe"). That is where you cut: chunks are created within these sections, never across their boundaries. Headnotes, where available, become chunks of their own, and paragraph numbers ("Randnummern") serve as anchors for citations.
The boundary between the statement of facts and the reasons matters most. According to § 313 ZPO, the statement of facts presents the claims raised and the means of attack and defense put forward, while the reasons contain the court's considerations. If both end up in the same chunk, the model may well cite a party's submission as the court's legal opinion.
The metadata should include the court, date, docket number and, where assigned, the European Case Law Identifier. According to the Federal Administrative Court, an ECLI consists of a country code, the court, the year and a local identifier. In Germany, the Federal Constitutional Court, the supreme federal courts, the Federal Patent Court and the state courts participate. That makes every decision with an ECLI uniquely addressable.
Everything Else: Docling's Chunkers
For articles, information sheets or contracts, Docling ships its own chunkers. The HybridChunker cuts along the document structure, splits sections that are too long, merges short ones that share the same headings, and takes the tokenizer into account. contextualize() prepends each chunk's headings, and the metadata points to the source document items, including page numbers. A neutral example:
from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker
doc = DocumentConverter().convert("info_sheet.pdf").document
chunker = HybridChunker()
for chunk in chunker.chunk(dl_doc=doc):
text = chunker.contextualize(chunk=chunk) # headings + text for the embedding
path = chunk.meta.headings # outline path
pages = {p.page_no for item in chunk.meta.doc_items for p in item.prov}The Docling docs recommend using the same tokenizer for the chunker and the embedding model. If you additionally split by sentences, you need a list of legal abbreviations, or a chunk will end in the middle of a statutory reference.
Citation Metadata: The Citation Is a Field
Citability is a property of the data, not of the prompt. Every chunk should carry:
- Source reference: statute, section, subsection, sentence, or court, date, docket number, ECLI, paragraph number.
- Outline path: the headings from the top down to the chunk, not just the last one.
- Version: which version of the statute, which date of the decision.
- Origin: document, page, link to the source.
- Controlled vocabulary: areas of law and document types from a fixed list, validated at ingest time. Otherwise a filter with a typo silently returns nothing.
The metadata belongs in the context so the model can cite. And it belongs in a check after generation: every section cited in an answer should resolve against the statute index. Whatever does not resolve gets flagged, not shipped. An invented section is the most expensive failure mode for a legal AI (more on hallucinations and calibration).
Quality Checks After Every Ingest
Parsers and chunkers almost always return something. Whether it is the right thing only shows up when you check:
- Recount. Documents in, documents with at least one chunk out. Every file without chunks belongs in a report, not in a log line.
- Sample against the original. Put random chunks next to the PDF page: are headings, tables and reading order right?
- Find repeated lines. Anything that appears identically on many pages is usually a header or footer and should be removed before embedding.
- Character hygiene. Characters from Unicode's Private Use Area (often from symbol fonts), hyphenation at line ends, ligatures.
- Metadata completeness. How many chunks have no outline path, no page, no source reference? Frequent titles without a parent node, such as "Tatbestand", point to lost hierarchy.
- Abbreviation endings. Chunks that end in "Abs." or "Nr." give away a sentence splitter without a legal dictionary.
Monitor Retrieval
- Health check with a real query. Do not just check whether the collection exists. Embed a known test question and require the expected hit.
- Measure empty results. Track the share of queries with no hits or below the score threshold, broken down by filter value. A filter that never matches is a bug.
- Hybrid search. For "§ 626 BGB", the exact match is what counts. Common vector databases offer full-text filters and hybrid search combining dense and sparse vectors for this.
- Model lifecycle. Providers retire embedding models. Store the model name with the vector and treat a full re-index as a planned operation, not an emergency.
- Golden set. Real questions with expected source references. You measure whether the right chunks land on top and whether answers with retrieval are better than without. How we evaluate systematically is described in our search for the best model for legal analysis.
When the Effort Is Not Worth It
- The corpus is small. If it fits into the context window, it belongs in the prompt, not in a vector database.
- You only need statute text. Then process the XML directly. PDF parsing is a detour here.
- The scans are poor. Then text recognition is the project first, not chunking.
- Nobody measures. Without a golden set, RAG is added latency and complexity with no evidence that it helps.
Conclusion
Docling solves the parsing problem well. The real work comes afterwards: carrying the structure all the way into the chunk, keeping source references as metadata, recounting after every ingest and monitoring retrieval. Citability is created at ingest time, not in the prompt.
Erst Recht uses AI to make initial legal assessments accessible. Our multi-agent system analyzes your situation and shows concrete next steps.
Tools Used:
Working on a similar problem? Preparing large document collections for RAG? Get in touch, I am happy to help with architecture or implementation.