How I went from parsing messy contracts to building an evidence-grounded AI platform on Google Cloud—and why evaluating the alternatives mattered more than choosing the fanciest technology.

The Problem: Documents Are Everywhere. Structured Data Isn't.

Every organization deals with documents: contracts, invoices, agreements, compliance reports, and legal filings. They contain valuable information, but much of it remains buried in paragraphs, tables, and legal language.

Imagine someone asking:

"Which of our vendor contracts are governed by Delaware law and contain a change-of-control clause?"

Answering that question manually could mean opening hundreds of PDFs, reading legal clauses, interpreting their meaning, and compiling the findings.

Modern LLMs make it tempting to solve this with a simple prompt:

"Read this PDF and return the important fields as JSON."

That works surprisingly well for a prototype. But turning it into a dependable platform raises questions:

• What happens when a PDF is scanned rather than digitally generated?
• How do we distinguish a real legal clause from text that merely mentions similar words?
• How do we know whether the model's answer is correct?
• Should we send the entire document to an LLM or retrieve relevant sections?
• Is a larger model worth its additional cost?
• How do we handle failures, retries, hallucinations, authentication, and observability?
• How can other AI agents consume the extracted information safely?

These questions became the foundation of my Document Intelligence Platform (DIP).

The objective was straightforward: turn unstructured commercial contracts into validated, structured JSON records, with verifiable evidence supporting the extracted fields.

The engineering challenge was everything surrounding that objective.

What I Built

DIP is an AI-powered document processing platform that extracts 41 structured fields from commercial contracts.

These include 33 legal clause categories, such as change of control, audit rights, liability caps, and non-compete provisions, plus eight typed fields covering parties, dates, durations, governing law, and document name.

Each extracted answer includes supporting evidence from the original document.

Rather than treating an LLM response as the final answer, the platform validates the structure, normalizes values, verifies quotations against source documents, and makes the resulting records accessible through REST APIs, analytics, and the Model Context Protocol (MCP).

The results at a glance

The final configuration was evaluated on 102 held-out contracts from the Contract Understanding Atticus Dataset (CUAD).

One particularly interesting finding: none of the 316 false positives in the born-digital test had fabricated, unsupported evidence. The errors were primarily interpretations of real text that did not satisfy the target legal category.

That distinction shaped much of the architecture.

1. Architecture: Separate the Intelligence from the Infrastructure

I designed the platform around independently deployable components, using Google Cloud Run as the primary compute environment.

The high-level flow looks like this:

            REST Clients       AI Agents
                 |                 |
              FastAPI          MCP Server
                 |                 |
                 +--------+--------+
                          |
                   Shared Service
                          |
                +---------+---------+
                |                   |
         Google Cloud Storage    Firestore
         (Source Documents)     (Job Status)
                |                   |
                +---------+---------+
                          |
                      Cloud Tasks
                          |
                  Private AI Worker
                          |
                 Document Inspection
                          |
                +---------+---------+
                |                   |
           Digital PDF          Scanned PDF
                |                   |
              pypdf            Multimodal LLM
                |                   |
                +---------+---------+
                          |
                  Structured Extraction
                          |
                 Validation & Grounding
                          |
                   JSON Records (GCS)
                          |
                    Parquet Compaction
                          |
                     DuckDB Analytics

The architecture follows a few principles.

AI extraction should be independent of transport. A contract should produce the same structured result whether it arrives through a CLI, REST API, or MCP client.

Document processing should be asynchronous. LLM calls can take seconds or minutes. They should not tie up an HTTP request or require a client to maintain a connection.

Raw documents and extracted records should be durable. A container should be replaceable without losing work or state.

Analytics and operational workloads should be separated. Document extraction is transactional, while querying thousands of historical records is analytical.

And importantly, expensive AI decisions should be backed by empirical evidence.

That last principle drove the project from its earliest phase.

2. Building the Evaluation System Before the AI Pipeline

The first major design decision was not about which model to use.

It was about how to measure whether any model was useful.

Why CUAD?

I selected the Contract Understanding Atticus Dataset (CUAD), which contains 510 commercial contracts with expert-annotated legal information.

The dataset provided two advantages.

First, the contracts reflect real-world legal-document complexity. Second, the annotations allow predictions to be compared with reference answers rather than evaluated subjectively.

I divided the dataset into:

I used multi-label stratification because several legal clauses appear infrequently. A random split could leave rare clause types entirely absent from the small experiment set.

Why not simply measure accuracy?

An important discovery was that the average clause appears in only about 21.9% of contracts.

A system that always says "clause absent" could achieve approximately 78% accuracy without doing useful extraction.

This is the classic class-imbalance problem.

I therefore evaluated the platform primarily using precision, recall, and F1, broken down by individual field and field type.

The evaluation system also distinguishes three different questions:

1. Presence: Did we identify the field or clause?
2. Value: Did we extract the correct normalized value?
3. Evidence: Did we identify the correct supporting passage?

These are different failure modes. A model might correctly locate an expiration clause but misunderstand the expiration date.

Establishing simple baselines

Before evaluating an LLM, I implemented two baselines:

• An all-absent classifier.
• A keyword and regular-expression extractor.

The keyword baseline achieved approximately 55.8 F1 on the development set. It had high clause recall but relatively poor precision.

This established an important requirement: the LLM needed to do more than locate keywords. It needed to interpret what the language meant.

Tradeoff: rigorous evaluation versus development speed

Building evaluation infrastructure first delayed the initial demonstration.

However, it made every subsequent decision measurable.

I also measured run-to-run variability. Two identical LLM configurations differed by approximately 1.6 F1 points. That meant small improvements could be noise rather than meaningful progress.

For important comparisons, I used paired evaluations and bootstrap confidence intervals.

The lesson: before optimizing an AI system, establish that your measurement system can detect an improvement reliably.

3. PDF Processing: Why I Didn't Use the Most Sophisticated Parser Everywhere

PDFs create one of the first architectural challenges in document intelligence.

A born-digital PDF often contains a machine-readable text layer. A scanned PDF may contain only images of pages.

I evaluated three extraction approaches:

pypdf: Extract text already embedded in the PDF.

Docling: Use layout-aware parsing and OCR for document understanding.

Multimodal extraction: Pass the PDF to a vision-capable model that can interpret the pages directly.

My initial inclination was to use Docling because of its sophisticated document-processing capabilities.

But the experiments told a different story.

Experimental results

The comparison used 30 contracts, with synthetic scanned versions generated so that the same expert labels could be reused.

On born-digital PDFs, the quality differences were within experimental noise.

pypdf was simpler, faster, and required fewer dependencies.

For scans, pypdf could not extract the missing text layer. Docling's OCR was reasonably effective overall, but date extraction deteriorated significantly—by approximately 11 F1 points relative to Docling's born-digital results.

That matters because dates are often among the most operationally important fields in a contract.

Multimodal extraction preserved date quality considerably better.

My decision: a hybrid document router

Rather than adopting one parser for every document, I implemented a document-aware routing strategy.

The pipeline inspects the PDF's text coverage:

• Documents with usable text layers follow the pypdf route.
• Documents with predominantly image-only pages follow the multimodal route.

This avoids paying multimodal inference costs when the text is already available.

It also allowed me to remove heavyweight Docling dependencies, including PyTorch-related packages and substantial OCR model assets, from the core production image.

What did I give up?

The system no longer benefits from Docling's richer layout analysis on every document. Complex tables, unusual page structures, or forms could benefit from a layout-aware parser.

The multimodal route is also more expensive. On the final test set, scanned contracts cost approximately $0.0149 versus $0.0080 for born-digital contracts.

And synthetic scans are not a complete substitute for real-world low-quality scans.

I would revisit this decision for a document corpus dominated by forms, complex tables, or genuinely poor-quality images.

The lesson: choose the cheapest reliable processing route for the document characteristics you actually have, rather than forcing every document through the most capable parser.

4. LLM Architecture: One Call or Forty-One?

With the document content available, the next problem was extracting 41 fields.

I considered three strategies.

A single model call: Give the model the contract and ask it to extract all 41 fields into one structured response.

Grouped calls: Divide fields into categories such as dates, parties, and legal clauses.

Per-field calls: Run a separate extraction for every field.

The tradeoff initially seemed straightforward.

Smaller calls might improve focus, enable parallelism, and simplify field-specific prompting.

But they also repeat document context, require more API calls, and increase orchestration complexity.

What the experiment showed

All three strategies delivered broadly comparable extraction quality.

However, the single-call strategy was approximately 3.2 times cheaper than grouped extraction and 18 times cheaper than per-field extraction in the evaluated setup.

It was also the fastest.

So I selected one structured extraction call per contract, followed by targeted validation and repair.

Why direct OpenAI SDK instead of an orchestration framework?

For the core extraction path, I used the OpenAI SDK directly with Structured Outputs and Pydantic.

A framework could have abstracted parts of this workflow, but the abstraction would not materially reduce complexity for a single extraction call.

More importantly, direct SDK integration gave explicit control over:

• Input and output schemas.
• Token usage.
• Retries and error handling.
• Model configuration.
• Latency and cost recording.
• Prompt caching behavior.

LlamaIndex was retained for retrieval experiments rather than made a dependency of the core pipeline.

An unexpected prompt-caching lesson

I discovered that changing the response schema affected cache reuse.

Initially, grouped calls used different schemas, preventing the expected shared prompt caching.

Standardizing schemas improved the situation, but parallel requests still did not guarantee cache hits.

This led to a practical cost-management decision: budget using conservative uncached estimates rather than assuming caching discounts.

The tradeoff

One large extraction call creates more coupling between fields. A poorly behaved response can affect the entire contract, and output size limits need careful handling.

The architecture mitigates these risks with structured schemas, bounded error handling, and targeted re-asks after validation.

If I were extracting hundreds of fields or processing documents that exceed the model context window, I would reconsider grouping.

For this workload, though, one call was the clear cost-performance winner.

5. Context Engineering: Why Full-Document Extraction Beat RAG

Retrieval-Augmented Generation (RAG) is a natural architectural choice for document applications.

Instead of sending a long contract to the model, retrieve only the most relevant passages.

That sounds more efficient.

But relevance depends on the task.

A question about one clause might require only a few paragraphs. Extracting 41 different fields can require information from almost every section of the document, including definitions, schedules, and appendices.

I evaluated:

• Full-document context.
• Retrieved sections.
• Retrieved sections plus document-specific definitions.

Retrieval reduced cost by only about 11%, while losing approximately 3.4–4.6 F1 points against the full-context approach.

The problem was not necessarily retrieval quality in isolation.

It was the mismatch between retrieval and a broad, multi-field extraction task.

A chunk that seems irrelevant to one field may contain a defined term essential to interpreting another.

My decision

Use full-document context by default, retaining retrieval as a fallback for documents too large to fit in a single model context.

The tradeoff

Full context increases input tokens and can become prohibitively expensive for extremely long documents.

It also requires the model to reason over more irrelevant material.

However, in the measured workload, the improvement in completeness was worth the relatively small additional cost.

This reinforced an important distinction:

RAG is an architectural technique, not a universal requirement for applications that process documents.

Its usefulness depends on whether the task is narrow information retrieval or comprehensive document understanding.

6. Prompt Engineering: Improving Legal Judgment, Not Just Keyword Recognition

The first extraction system performed reasonably well, but evaluation exposed a consistent problem.

The LLM often found real passages and interpreted them too broadly.

For example:

• An indemnification clause was classified as uncapped liability.
• A survival clause was mistaken for post-termination service obligations.
• Volume-based pricing was interpreted as a volume restriction.
• An assignment clause mentioning a merger was mistaken for a change-of-control provision.

These errors were not failures to find text.

They were failures to interpret the legal boundaries of a category.

Moving beyond field descriptions

I experimented with four prompt versions.

The initial prompt used field definitions.

Subsequent versions added real phrasing variants, positive examples, hard negative examples, and definitions extracted from each contract.

A hard negative is particularly valuable.

It is a passage that resembles the target category but does not qualify for it.

For example, a contract might mention assignment and mergers without creating the particular change-of-control obligation being sought.

Including such examples teaches the model where a category ends.

What improved?

On the development set:

The improvement from the enhanced prompt was approximately +1.18 F1, with a paired 95% confidence interval of +0.07 to +2.38.

That came with approximately 25% higher inference cost.

Interestingly, individual prompt additions did not consistently produce improvements greater than run-to-run noise. The complete bundle showed the measurable gain.

The tradeoff

Longer prompts increase token usage, maintenance effort, and the risk of confusing or conflicting examples.

The examples were sourced exclusively from training data to avoid contaminating evaluation.

I also learned that more examples cannot fix every problem. Some legal categories have boundaries that are broader or narrower than their names suggest.

A few example phrasings cannot replace a precise operational definition.

The lesson: prompt engineering should be informed by categorized errors, not driven by adding more instructions indiscriminately.

7. Model Selection: Why Bigger Wasn't Better

After improving the extraction workflow, I compared three models using the same production-style configuration.

The larger models cost approximately 11 and 22 times more without a demonstrated overall accuracy improvement.

The largest model was actually worse on legal clauses in this evaluation, principally because it identified additional passages that did not qualify under the dataset's definitions.

This illustrates an important distinction between broad reasoning capability and precision on a specific extraction task.

A model can provide a sophisticated interpretation that differs from the narrower classification convention used by the application.

My decision

Select luna for the production extraction pipeline.

It occupied the best position on the measured quality-versus-cost frontier.

The caveat

The prompt had been optimized primarily for luna.

Therefore, the experiment compared the complete model-plus-prompt configurations as built. It did not prove that luna is universally better than the larger models.

Retuning prompts for each model could change the ranking.

I would consider a larger model for selected difficult fields if an appropriately controlled evaluation demonstrated enough value to justify the extra cost.

The lesson: optimize for measured task performance per dollar, not model size or reputation.

8. Validation and Grounding: A Valid JSON Response Is Not Necessarily Correct

Structured Outputs solve an important problem: producing data in the expected shape.

But syntactically valid JSON can still contain incorrect information.

I separated validation into multiple layers:

Schema validation: Does the output conform to the expected structure?

Value normalization: Are dates, durations, names, and jurisdictions represented consistently?

Evidence grounding: Can each supporting quote be located in the original document?

Consistency validation: Do the extracted fields violate domain-specific consistency rules?

When validation detects particular problems, the platform can perform one targeted re-ask, passing the relevant validation errors back to the model.

This is more economical than repeating extraction for every document.

Evidence grounding

The grounding layer checks quotations using progressively more flexible matching:

1. Exact text matching.
2. Whitespace and punctuation normalization.
3. Fuzzy matching for minor differences.
4. Matching across page breaks and inserted footers.

Where possible, the extracted quotation is replaced with the exact corresponding source text.

This produces a record that is more useful to downstream consumers because answers can be accompanied by verifiable evidence.

The surprising discovery

On the development set, approximately 99.9% of model quotations could be located in the source.

The model's false positives were not generally fabricated passages.

Instead, the model quoted real text and reached incorrect conclusions.

For example, it could cite an indemnification clause perfectly while incorrectly classifying it as uncapped liability.

This taught me that there are at least two distinct failure classes:

Evidence fabrication: The model provides support that does not exist.

Semantic misclassification: The model finds genuine evidence but interprets it incorrectly.

Grounding is highly valuable for detecting the former. It cannot, by itself, prevent the latter.

Should unsupported answers be dropped?

I compared dropping answers whose quotations could not be matched against keeping them with a warning.

Dropping removed a correct answer whose quotation crossed a redacted section of text.

The measurements did not show a corresponding benefit from aggressive deletion.

So the platform retains such answers while exposing their grounding status.

The choice is then explicit for downstream consumers.

The cost of repair

Targeted re-asking added about 12% to extraction cost in the development evaluation and produced fully valid records on the evaluated set.

That was an acceptable tradeoff for better structural reliability and inspectability.

The lesson: structured output, factual grounding, and semantic correctness are three different concerns. A trustworthy AI system needs to measure them separately.

9. Backend Architecture: Designing for Failures, Retries, and Cost

Once the extraction pipeline was established, I built the operational platform using FastAPI, Cloud Run, Cloud Tasks, Firestore, and Google Cloud Storage.

Why asynchronous processing?

Extracting a long document can take tens of seconds or longer.

Holding an HTTP connection open during extraction introduces timeout risks and makes it harder to manage bursts of traffic.

Instead, the API accepts a document and returns 202 Accepted with a job identifier.

The client polls for status and retrieves the completed record.

Cloud Tasks delivers processing requests to a separate private worker service.

The separation allows API scaling and extraction concurrency to be managed independently.

Idempotency is essential when AI calls cost money

Cloud Tasks provides at-least-once delivery semantics.

That means a worker may receive the same job more than once.

Without safeguards, retries can trigger duplicate model calls—and duplicate charges.

DIP uses document content hashing combined with the pipeline version to reuse existing results. It also supports client-supplied idempotency keys so network retries do not create duplicate jobs.

Including the pipeline version matters: an identical document processed with a newer prompt or model may legitimately produce a different record.

Completed jobs are not reprocessed simply because a task is delivered again.

Handling stranded jobs

Normal retries do not cover every failure.

For example, a job can become stranded if task delivery repeatedly fails before processing begins.

I introduced a scheduled sweeper to identify such jobs.

The timing was deliberate.

Cloud Tasks stops retrying after a 30-minute window. The sweeper looks for jobs idle for more than 35 minutes.

If the sweeper acted while Cloud Tasks was still retrying, both systems might dispatch the same expensive work.

Here, a five-minute difference is not just an operational configuration. It is part of the correctness design.

Backpressure

The worker processes no more than four queued jobs concurrently, protecting the model provider's rate limits.

The API also rejects new uploads with HTTP 429 when the waiting backlog exceeds 200 jobs.

This bounds the queue rather than allowing unlimited work to accumulate.

The tradeoff

Asynchronous processing increases system complexity: job states, queues, retries, deduplication, and recovery must all be designed and tested.

For low-volume prototypes, direct synchronous calls may be entirely sufficient.

For a production-oriented platform handling costly, variable-duration work, asynchronous processing provides much stronger control over reliability and spend.

10. Analytics: Why I Used DuckDB and Parquet Instead of Another Database

The platform has two different data-access patterns.

Operationally, it needs to store and retrieve individual extraction jobs and records.

Analytically, users may want to ask:

"How many contracts contain liability caps?"

"Which governing laws appear most frequently?"

"How do extraction failures vary by clause type?"

These are fundamentally different workloads.

My design

The original extracted JSON records remain the source of truth in Google Cloud Storage.

A scheduled compaction process converts them into partitioned Parquet datasets.

DuckDB then queries those Parquet files directly from cloud storage.

This creates a lightweight analytical layer without provisioning a continuously running warehouse or database cluster.

Why Parquet?

JSON is convenient for producing and retrieving individual records.

Parquet is column-oriented and better suited for scanning selected fields across many records.

Separating the formats allows each to serve the workload it handles best.

Why not let clients submit SQL?

A general SQL interface creates unnecessary security and resource-control risks.

Instead, the platform exposes a fixed set of named analytical queries with validated parameters.

DuckDB connections are restricted to the authorized data lake.

The tradeoff

Compaction introduces data freshness lag. Analytics can be behind the latest extracted record until the next scheduled refresh.

Queries were also significantly slower than operational status requests, taking roughly two seconds in the measured setup.

That was acceptable because analytics is a separate workload with different latency expectations.

A larger, more frequently queried corpus might justify BigQuery or another dedicated analytics platform.

For this project's scale, DuckDB and Parquet offered an effective combination of simplicity, performance, and low infrastructure cost.

11. MCP: Making the Platform Useful to AI Agents

REST APIs are natural interfaces for applications and backend services.

But a growing class of consumers is made up of AI agents.

An agent might need to discover which contracts satisfy several legal conditions, retrieve the relevant evidence, and explain its findings to a user.

I added support for the Model Context Protocol (MCP) to make the platform accessible to such clients.

Importantly, MCP was added as a thin adapter rather than becoming the core architecture.

Both REST and MCP share the same underlying service layer.

This avoids maintaining two independent implementations of document search and retrieval.

Coarse-grained versus fine-grained tools

An interesting design question emerged.

Should an agent receive many specialized tools, such as separate searches by governing law and clause category?

Or should it receive fewer, more expressive tools that support several filters in one call?

I tested both.

The evaluation covered 25 tasks, repeated across tool configurations and different record stores.

Fine-grained tools increased costs by approximately 25–42% without delivering meaningful accuracy improvements.

They also required agents to combine intermediate results themselves.

I chose coarse-grained tools with richer descriptions.

The more descriptive tool definitions improved citation accuracy by approximately eight percentage points.

The most valuable experiment: an oracle comparison

I wanted to distinguish mistakes caused by the agent from mistakes caused by the extracted data it consumed.

So I ran the same agent tasks using two record stores:

• Records generated by DIP.
• Perfect reference records derived from the dataset's expert labels.

The comparison was revealing.

The approximately 29-point difference indicated that extraction errors were responsible for much of the downstream agent failure.

The agent could use its tools correctly and still produce the wrong answer because a contract had been incorrectly classified upstream.

This is why evaluating an agent in isolation is insufficient.

The quality of an agent's final answer depends on the quality of every intermediate system it trusts.

MCP security decisions

Exposing tools to autonomous agents introduces additional risks.

For remote access, the platform uses Google OAuth identity verification, per-principal authorization, audit logging, rate limits, and spending caps.

Document submission URLs are protected against server-side request forgery (SSRF), including attempts to access internal cloud metadata addresses.

Document content is treated as untrusted input.

That matters in both directions: malicious instructions may appear inside uploaded documents, and extracted quotations later enter an agent's context.

Returning capped, explicitly labeled evidence fields helps downstream agents distinguish source data from instructions.

I also tested with actual MCP clients, including Claude Desktop and MCP Inspector.

Those tests surfaced problems that unit tests missed, including a long-held request stream, deployment host validation, and cold-start behavior.

The broader lesson was familiar from backend engineering: interoperability should be verified using real clients in the deployed environment, not just mocked interfaces.

12. Production Engineering: Deployment, Security, and Observability

An AI pipeline isn't production-oriented simply because it runs inside a Docker container.

It needs operational controls.

Infrastructure as code

I used Terraform to define Google Cloud resources, including Cloud Run services, service accounts, storage, and task queues.

However, I deliberately separated infrastructure administration from application deployment.

Infrastructure changes are applied through a controlled human process. GitHub Actions is limited to building and deploying application images.

Why?

A CI identity capable of modifying IAM and infrastructure could have a much larger blast radius if compromised.

Limiting CI to application rollout reduces that risk.

Workload Identity instead of stored Google credentials

GitHub Actions authenticates to Google Cloud through Workload Identity Federation.

Google Cloud services use assigned service identities.

This avoids distributing long-lived Google service-account JSON keys.

Third-party API credentials, including model-provider and observability keys, are still required and are stored using Secret Manager.

Observability

The worker emits traces that follow an extraction through its important processing stages, including the model call.

Arize is used for production tracing, while local tracing supported development.

Cost and usage are recorded separately so experiment spending can be audited even if a run crashes.

This last detail came from a real failure: an early experiment recorded spend only when the complete run finished. A crash meant the consumed tokens were not reflected in the ledger.

The fix was to record usage after every paid call.

Reliability measured in the cloud

In a 50-contract load test against the deployed development environment:

• All 50 extraction jobs completed.
• End-to-end p95 latency was 63 seconds.
• Status API p95 latency was 160 milliseconds.
• Throughput reached 13.9 contracts per minute.

Those results demonstrate a working deployment under the measured workload, not a universal availability guarantee. Fifty successful requests are insufficient to prove a 99% success-rate SLA.

The deployment also exposed unexpected cold-start costs, eventually traced partly to heavyweight libraries being imported unnecessarily.

Removing experimental dependencies from startup paths improved service initialization.

The key lesson was that deployment performance is part of architecture, not an optimization to consider only after everything is built.

13. What Didn't Work—and What I Would Improve Next

Perhaps the most useful part of this project was understanding the limitations.

The platform achieved 84.0 F1 on the held-out test set, but that still leaves meaningful errors.

I manually categorized a sample of 50 development-set errors.

The largest category involved over-triggering: the model found legitimate text but applied the wrong legal classification.

Other categories included missed clauses, incorrect values, normalization issues, and suspected annotation errors in the reference dataset.

Three improvements stand out.

First: fix deterministic normalization errors in code

The model sometimes returned a governing-law jurisdiction such as "Delaware, United States" when the expected value was simply "Delaware."

That should not require another LLM call.

A deterministic jurisdiction normalizer can handle it more reliably and cheaply.

This is a reminder that not every problem in an AI application needs an AI solution.

Second: improve the boundaries of difficult legal categories

I would expand training-only hard negative examples for common misclassifications, such as indemnity versus uncapped liability and survival provisions versus post-termination services.

Any improvement would need another evaluation on the development set before being accepted.

Third: compute derived dates deterministically

Some expiration dates do not appear explicitly.

A contract might say "three years after the Effective Date."

Instead of expecting the model to perform all such calculations, I would have it extract the source expression and its anchor date, then calculate the final value in code where the evidence is sufficient.

This separates language understanding from deterministic computation.

The evaluation-data caveat

Manual error analysis also identified cases where the model appeared correct but the dataset's annotation did not capture the relevant clause.

These were suspected label errors, not independently adjudicated corrections.

I kept the official scores unchanged.

A second reviewer and a documented re-annotation process would be needed before reporting adjusted performance.

The goal is to understand model quality accurately, not to improve metrics by selectively changing the reference answers.

Final Thoughts: Good AI Engineering Is an Exercise in Making Tradeoffs Explicit

Building this platform reinforced an important lesson for me.

The most interesting architectural choices were rarely about adopting a new library or a more powerful model.

They were about understanding what the system actually needed.

I chose pypdf over heavyweight parsing when the text was already available.

I chose multimodal processing over OCR for scanned contracts because OCR damaged important fields.

I chose one structured extraction call over dozens because the quality was comparable and the cost substantially lower.

I chose full context over retrieval because comprehensive extraction required information from across the document.

I chose a smaller model because larger alternatives did not justify their cost.

I chose to validate evidence without automatically discarding every imperfectly grounded answer, because the data showed that aggressive filtering could remove correct information.

And I kept MCP as an adapter over the existing platform rather than redesigning the entire system around agents.

None of these decisions is universally correct.

They are appropriate for the workload, requirements, and measurements of this project.

For me, that is the central principle of AI systems engineering:

A strong AI architecture is not defined by how many sophisticated technologies it uses. It is defined by how clearly its decisions are justified, how reliably its outputs can be evaluated, and how honestly its tradeoffs are understood.

The final result is an evidence-grounded document intelligence platform, deployed on Google Cloud, with measurable extraction quality, controlled inference costs, recoverable processing, and interfaces for both traditional applications and AI agents.

But the more valuable outcome is the engineering process behind it: establish a baseline, isolate a design decision, run an experiment, measure the tradeoff, document the outcome, and remain willing to change the architecture when the evidence changes.

That's the process I'd carry into the next AI system I build.

Project: Document Intelligence Pipeline — GitHub

Core technologies: Python, FastAPI, OpenAI Structured Outputs, Pydantic, pypdf, Docling (evaluated as an alternative), LlamaIndex (retrieval experiments), DuckDB, Parquet, Docker, Google Cloud Run, Cloud Tasks, Firestore, Cloud Storage, Terraform, Arize, and MCP.

Evaluation scope: CUAD commercial contracts, with measurements from the project's October 2026 development and held-out test runs. Results may differ for other document types, annotation conventions, and workloads.