One person. Three days. A verified legal appeal.

01The problem

Legal drafting with chatbots has one famous failure mode and several quieter ones. The famous one is invented citations, and it is getting worse: the AI Hallucination Cases database, which logs rulings where a court found a party relied on hallucinated material, counted around 200 by mid-2025 and more than 1,500 by mid-2026, with the first bar suspensions over AI filings already handed down. The quieter failures are the ones that lose cases. Accuracy drops sharply once the work requires synthesis across many documents, and nothing raises a flag when the decisive fact in the record is read and passed over.

The first phase described here produced examples of each. Six drafts written the traditional way with a commercial chatbot produced a structurally excellent appeal that also contained 18 numbered defects, including a quoted passage that does not exist in the source, an appeal-court decision dated on a Sunday inside judicial vacations, and the case's single most important fact sitting unnoticed on a scanned page.

None of this is visible from the text alone. All of it is disqualifying in front of a judge or rapporteur. The product opportunity is not "AI that writes legal documents". It is AI that can prove what it wrote is grounded, cheaply enough for small firms and in-house teams.

02What we actually did

The process ran in three distinct regimes. Each one maps to a product module, and each was audited from full transcripts after filing:

SVD-025-2026/
├─ 00_INSTRUCTIONS   mission charter, state handoffs          4 files · 11k words
├─ 01_SOURCES        the case record, read-only inputs       62 docs · 211 pages
├─ 02_LEGISLATION    downloaded statutes, version-checked    28 files · 33k words
├─ 03_CASE_LAW       case law, per-lens source logs          19 files · 25k words
├─ 04_ANALYSES       case map + specialist analyses          10 files · 63k words
├─ 05_ADVERSARIAL    45 attack vectors + verdicts             3 files · 27k words
├─ 06_VERIFICATION   192-row citation matrix, audits          6 files · 62k words
├─ 07_OPINION        technical opinion + corrections          3 files · 19k words
└─ 08_DELIVERY       v7…v13, filing kit, dated change log    28 files · 78k words

03What the audit proved

After filing, we ran a forensic audit: a deterministic parser over every session transcript, ten analyst agents, and an adversarial completeness critic that corrected the analysts' own errors. Four findings define the product:

• Human judgment is the scarce input, and it is tiny. 96 typed interventions, median ≈15 words. Only 19% were instructions; 49% were judgment calls: skepticism, factual corrections, scope decisions. One word of human input yielded ≈840 tokens of model output and ≈43 words of persisted work product.
• The catches that saved the filing came from skepticism, never from the producer. A fabricated quotation embedded since the chat phase was found by a vision agent re-reading scanned pages, a check that only ran because the user challenged an AI-claimed limitation, under 24 hours before the deadline. Most users would not have pushed. The interface must engineer that skepticism: make the agent declare what it checked and what it did not, with "exhaustive vs. sampled" as a blocking contract, independent verifiers on every deliverable.
• Keeping state in files is what made it robust. The system survived a client crash that killed 4 agents mid-flight (auto-recovered in ≈20s), account limits that killed 2 more, and an API outage that killed the longest session, each time resuming from disk in minutes, with zero context loss across ≈25 hours of sessions.
• Legal deliverables need transactional editing. One Word file was silently corrupted by an XML-level edit that passed every content check; the fix became a permanent format-validation guard. The product ships with anchor-verified edits, container validation and a dated change log: the evidence behind the certification a growing number of US courts now require from the filing lawyer, that a human verified the AI-assisted content.
• Reading is not seeing. The fact that became the appeal's central argument, a detail in a stamp that gave away the delivery method used and carried material weight in the proceedings, sat in a document with no text layer at all. The vision pass rendered that page and transcribed the stamp character by character between 00:00 and 01:19 on 28 July, then moved on: it was reading for another purpose, and marked a different passage on the same page as decisive. The reviewer supplied the fact from memory eleven hours later, and confirmation took seconds because the transcription already existed. The machine had read the sentence, stored it, and not seen its significance. The missing step is the one that recognizes that significance: sweep every new transcription against a lexicon of legally loaded terms (dispatch and delivery methods, mentions of means and deadline, stamps, handwritten dates, reference numbers) and raise a flag even when the agent is reading for something else, with the flag surviving the change of phase and session.

"Is everything really finished? Even the agents that died mid-run with API errors?" — the user question that exposed silently degraded verification, and the reason declaring what was checked is a core product feature

04Unit economics

Three pricing observations. First, at this scale the work had no price at all. Verification against 192 primary sources, and 45 adversarial attack vectors scored one at a time, is what a well-funded defence produces over weeks; no small firm or sole practitioner buys it for a single matter at any figure. Second, where a professional-fee equivalent does exist, the compute sits at least an order of magnitude below it, and two in the most expensive jurisdictions, which puts this kind of work within reach of small firms and in-house teams. What constrains price is qualified review time, not compute. Third, cost is dominated by context re-reading (cache reads/writes), not generation, meaning workflow and caching architecture, which we control, are the main cost levers. Nothing here was optimized for cost: the objective was to finish. Those levers are untouched, and we expect the same work for less than half the compute in the next iterations.

05The product

The audit distilled the process into nine replicable steps. Six were designed upfront, two emerged under fire, and one the audit itself found missing:

• 1 · Intake & guided drafting: chat with automatic version history and a citation ledger where every reference is born "unverified".
• 2 · Case file constitution: templated dossier: read-only sources, rules, phases, known traps; the user fills in a form.
• 3 · Autonomous analysis pipeline: mapping, specialist lenses, isolated adversarial attack + judge agent, citation matrix. Human gates between blocks.
• 4 · Synthesis: a technical opinion with surgical corrections, each traceable to sources.
• 5 · Completeness critic: numbered open items; verification mode (exhaustive/sampled) is a declared, blocking contract.
• 6 · Transactional application: native-format editing with anchors, container validation, immutable versioning, dated change log.
• 7 · Paired review: inline annotation for the lawyer; the AI verifies before opining; structured decisions with a binding decision log. (emerged in production)
• 8 · Filing module: exhibit stamping, independent fact-checking of the cover email, human-only signature and submission checklist. (emerged in production)
• 9 · Salience sweep: every new transcription checked against a lexicon of legally loaded terms, raising a flag that survives the change of phase and session. (found missing by the audit)

Target user: lawyers and in-house legal teams without technical skills. What they buy is verification, auditability and process: the citation matrix, the adversarial test, the dated audit trail they can put their name to.

06Scaling

Four properties of the architecture are what make the result repeatable rather than lucky, and they are the ones we intend to keep through any rewrite:

• State lives in files, never in a context window. One owner per file, sources read-only, every phase handing off on disk. The project survived a session fork, six agent deaths, an API outage and two changes of model without losing a line of reasoning, and after one session was cleared the synthesis phase rebuilt the entire case state from disk in 5 minutes.
• The task journal is idempotent. Work is keyed per task, so a relaunch redoes only what did not finish. That is why the 4 agents killed mid-flight were back in ≈20 seconds with nobody at the keyboard.
• Verifiers return structured records, not prose. Each check comes back as pass, issues and detail, so results aggregate deterministically. Nine verifiers produce a table; nine hundred produce the same table. Nobody reads nine hundred reports.
• The producer never grades its own work. The two adversarial agents were kept informationally isolated from each other and from the drafting, a separate judge agent scored their 45 attacks one at a time, and the filing check runs one independent agent per exhibit. That last check caught a factual error in the covering email written by the orchestrator itself.

Fan-out today follows the legal questions: six specialist lenses in parallel, two attackers, a judge agent. Fan-out per document exists as well, and it is the part that scales linearly with volume, but it sits at the end of the pipeline, verifying what was produced. The next iteration moves that same pattern to intake: one agent per document, each returning a structured record of entities, dates, acts, cross-references and page ranges, so the case map comes from querying that index. The citation matrix already carries claim → exact quotation → source and pinpoint across all 192 rows; adding entity and event identifiers turns cross-document reasoning into a join, and moves the matrix to the front of the pipeline, where it feeds the analysis it currently audits. Partitions then derive from the index, and synthesis runs as a tree, by legal question first. None of that is new machinery. It is the machinery above, wired in a different order.

07Limitations

• The longest unattended run so far. ≈8 hours end to end under a mission charter is the longest the pipeline had been left to run without a human at the keyboard. It draws on nearly 20 prior legal filings produced with AI assistance (several signed off by lawyers), until now built with a mix of tools and heavily manual workflows. It is still one matter and one technically sophisticated user: the product thesis is precisely to make the author's two decisive behaviors (skepticism and domain facts) available to users who don't bring them.
• 62 documents is a small volume. A case this size fits comfortably in context, so nothing here tests what happens at hundreds or thousands of input documents, which is where retrieval and partitioning start to decide the outcome. That is the next thing we intend to run. The widest parallelism we have actually run is 9 agents at once, so the section above is design analysis backed by working parts, not a load test.
• The AI declared two capabilities impossible that it in fact had, silently degraded one verification to sampling, and fabricated one citation in the chat phase. Every one was caught by the process. That is the point. But the catalog of 12 failure/recovery pairs, listed in full in the process audit, is part of the spec.
• Account limits stopped work for ~2 hours mid-project; quota management is a first-class product problem.
• The first part of the drafting, up to v6, ran unaudited, including the use of a commercial chatbot. The useful part is the analogy: it stands in for the state a matter is usually in when it reaches us, drafted somewhere else and unverified. The product takes that input and also owns the drafting natively: versioned drafts, a citation ledger, an audit trail from the first prompt.