How We Found the Best AI Model for Legal Analysis

A three-tier evaluation approach: LLM-as-a-Judge, Crew-as-a-Judge, and human validation. Parallel, not sequential.

The Challenge: Making Quality Measurable

Which AI model delivers the best legal analyses? At Erst Recht, this was the first question we faced. Gut feelings and marketing promises are not enough. We needed a systematic, data-driven approach.

The result: A three-tier evaluation where AI assessment and human expertise work together. In this article, I show how we approached it.

The Problem: How Do You Measure "Good" Legal Analysis?

With math problems, there is a clear ground truth: 40+2=42, done. (And no, we are not talking about JavaScript here, where '1' + '1' = '11'.) With legal analyses, this clarity is missing. The requirements are multifaceted:

• Factually correct: Correct laws, current case law
• Complete: All relevant aspects, no missed deadlines
• Understandable: Comprehensible even without a law degree
• Action-oriented: Concrete next steps

Manually evaluating each test case is time-consuming and subjective. We needed a scalable yet reliable approach.

Why Public Benchmarks Do Not Answer This Question

A leaderboard would be the shortcut. For laypeople's questions under German law, it does not work:

• Wrong language, wrong law: LegalBench collects 162 tasks on legal reasoning in English. Whether a model tells a layperson, in German, the correct notice period under the BGB is not something it measures.
• Contamination: Public test questions end up in training data. On a fresh twin of the GSM8k math benchmark, accuracy for some model families dropped by up to 8%, while the strongest models were barely affected (Zhang et al., 2024).
• Marketing is not a measurement: Commercial legal research tools whose vendors promised to avoid hallucinations hallucinated in 17 to 33% of cases in a Stanford study (Magesh et al., 2024).

If you want to know which model works for your own use case, you need your own test cases.

Our Approach: Three Parallel Judges

Important: The three evaluation tiers work in parallel, not sequentially. Each judge evaluates the same outputs independently, and the results are aggregated at the end.

1. LLM-as-a-Judge

The concept: A powerful language model evaluates the outputs of all other models according to clearly defined criteria. Automated and scalable.

We evaluate each analysis across four specialized dimensions:

2. Crew-as-a-Judge

A single judge has blind spots. The solution: Multiple specialized AI agents evaluate from different perspectives.

The key: The agents "discuss" and consolidate into an overall verdict. The result is more robust than any single evaluation.

3. Human-as-a-Judge (Human Validation)

AI evaluation does not replace human expertise, but it also does not need to review every single case. Instead, a lawyer from the founding team validates a sample of the results:

• Blind tests: The reviewer does not know which model produced which output
• Edge cases: Targeted testing of borderline scenarios
• Calibration: Do human judgments align with the AI evaluation?

The goal: Ensure that the automated evaluation is reliable. The human sample serves as a sanity check, not a complete re-evaluation.

Result Aggregation

All three judges evaluate every case for every model. The results are combined into:

• Winner model per case: Which model performed best on this specific case?
• Winner model overall: Overall winner across all cases

Consensus among the judges increases confidence in the result. When judgments diverge, we analyze the case more deeply.

The Testing Process in Detail

Systematic evaluation requires systematic tests:

1. Test case creation: Cases across the areas of law the product covers
2. Identical conditions: Exactly the same prompts for each model
3. Multiple runs: Consistency check per scenario
4. Parallel evaluation: Both AI judges evaluate simultaneously, humans review a sample

Results and Insights (as of February 2026)

Historical snapshot: These results come from the first version of this article in February 2026. I deliberately do not name the models we compared, and the ranking says nothing about today's models. What remains are the patterns.

"The AI judges rated Model A as the winner by majority. The human samples confirmed this result. A strong signal for the reliability of our automated evaluation."

The evaluation delivered clear results: Both LLM-as-a-Judge and Crew-as-a-Judge consistently identified the same model as the winner. The human validation via sampling reached the same conclusion.

Key insights from the evaluation:

• Custom approaches pay off: Our specialized methods outperformed generic chat assistants
• Trade-offs exist: Some approaches are faster, others more precise. We optimized for quality
• Consistency matters: Approaches with fluctuating quality were eliminated

What We Learned

The Method to Reuse: Eight Steps for Today's Models

This is how I would repeat the evaluation today. First: what you test is not the model but the system of model, prompts, and pipeline. Pipeline variants therefore belong in the same comparison as model candidates. If you only test new models in a chat window, you are measuring the wrong system.

1. Derive test cases from real cases

Anonymized, across the areas of law your product actually covers, plus deliberately uncomfortable cases: incomplete facts, out-of-scope questions, obvious nonsense. Keep the test set under wraps. The termination case above is public with this article and no longer works as a test case.

2. A reference checklist per case

Before the first run, someone with legal training defines what a correct answer must contain and which errors disqualify it immediately, such as a wrong deadline. The judge receives this checklist. In the MT-Bench study, the judge's failure rate in a small test (ten math questions, 20 judgments) fell from 70% to 15% once the prompt contained a reference answer the judge had generated itself beforehand (Zheng et al., 2023).

3. Grade pass/fail instead of on a 1-to-5 scale

The rubric above rates comprehensibility on a 1–5 scale, but nobody knows what to do with a 3. Binary grading works better: mark each answer correct or incorrect, with a detailed critique. Hamel Husain recommends this in his guide to LLM-as-a-Judge. If you need a ranking, pick the best among the correct answers afterwards. I would add one check: does the analysis state what it does not know for sure? (Why models otherwise prefer to bluff.)

4. Compare blind and pairwise

Reviewers only see neutral labels such as A to D, in random order per case. LLM judges grade every pairwise comparison twice, once with the order swapped. A win only counts if the same answer wins both times; otherwise it is a tie. That is how Zheng et al. handle position bias.

5. Check the judge before you trust it

• Self-preference: LLMs recognize their own text and rate it higher (Panickssery et al., 2024). The judge and the candidates should come from different model families, or several judges from different vendors decide.
• Length bias: Judges tend to prefer longer answers. Check whether your scores simply rise with length.
• Calibration: Lawyers grade a sample blind with the same rubric. What matters is how many of the answers humans marked as wrong the judge also catches as wrong, and how many correct ones it lets pass (Husain recommends reporting both rates).

6. Measure cost and latency too

Tokens, cost, and runtime per case belong in the same table as quality. A few percentage points more quality at several times the cost is not automatically the better choice for affordable initial assessments. The survey of 247 agent security papers also criticizes how rarely security evaluations measure latency and cost.

7. Take statistics seriously

A worked example with made-up numbers: with 40 cases and 80% correct answers, the 95% confidence interval is roughly ±12 percentage points. Small differences are noise. So run each case several times, compare models paired on the same cases, and report error bars (Anthropic, 2024).

8. Schedule re-evaluation

Re-evaluate on every model, prompt, or pipeline change, when the law changes (checklists age too), and additionally about once a quarter. Pin model versions instead of using aliases that point to whatever model is newest. Vendors also retire older models regularly; see, for example, the deprecation pages from Anthropic and OpenAI. The exact model ID therefore belongs in the evaluation config, and every change to it is a reason for a new run.

Three Principles for a New Eval Project

• A judge from a different model family than the candidates, from the start: Otherwise self-preference kicks in.
• Log cost per case from day one: If you only ask about cost after choosing a model, you are comparing with incomplete data.
• Treat the eval setup as part of the product: Test cases, checklists, judge prompts, and model IDs belong versioned in the repo and run as a gate before every model change. An eval tool that lives next to the product goes stale with every rebuild.

When It Is Not Worth the Effort

• A prototype without real users: A dozen hand-picked cases and your own judgment are enough.
• A human edits every output anyway: Then that editing is your evaluation.
• No experts for the references: Without vetted checklists, an LLM judge mostly produces false precision. References first, automation second.
• Crew-as-a-Judge for every case: Several judge agents cost a multiple of a single judge. I would only use them where the judges disagree.

Erst Recht uses AI to make initial legal assessments accessible. See the quality of our AI analysis for yourself.

Planning your own LLM evaluation project? Get in touch. I am happy to help with the concept and implementation.