Re-Architecting Finnie: From a Prototype AI App to a Durable, Faster, Multi-User System

Building an AI application is surprisingly easy.

Building an AI application that is fast, persistent, safe, multi-user, inexpensive to run, and maintainable is a very different problem.

That is the journey I went through while re-architecting Finnie, a personal-finance education application built around multiple AI agents.

The original system worked. Users could ask questions, agents could retrieve financial knowledge, external APIs could provide market data, and an LLM could generate useful responses.

But as I looked at the architecture more critically, several problems started becoming obvious:

every request made multiple sequential LLM calls;
user data disappeared after a refresh;
conversation history was not durable;
retrieval lived inside the application process;
prompts were structured in a way that prevented effective prompt caching;
every application instance had its own state;
repeated questions unnecessarily went through the whole AI pipeline;
and the application was not really designed for multiple isolated users.

None of these problems necessarily matter when you are building the first version of an application.

They matter a lot once you start thinking about running it as a real service.

This post walks through the re-architecture and, more importantly, explains why each decision was made.

The Starting Point

The original request path looked roughly like this:

User
  ↓
Guardrail LLM
  ↓
Router LLM
  ↓
Agent LLM
  ↓
FAISS + external APIs

Before the user saw an answer, Finnie often made three serial LLM calls:

a guardrail classified whether the question was allowed;
a router decided which agent should answer;
the selected agent generated the answer.

The knowledge base lived in an in-process FAISS index using MiniLM embeddings. Live financial data came from services such as yfinance, FRED, Alpha Vantage, and NewsAPI. The Streamlit frontend ran behind Google OAuth on Cloud Run.

For an early architecture, this was perfectly reasonable.

The problem was that a prototype architecture had gradually become the application architecture.

That created some interesting engineering challenges.

Problem 1: Too Many LLM Calls Before Anything Useful Happened

The guardrail and router each required their own LLM request.

Worse, both produced free-form text that the application then tried to parse as JSON using regular expressions.

That meant two things.

First, latency accumulated serially. If the guardrail took one second and the router took another second, the actual agent could not even begin until both finished.

Second, parsing model output with regex is fragile. A model may produce valid information with slightly different formatting and suddenly the application logic breaks.

The design document estimated roughly two seconds could be spent before the agent even started working.

Decision: Merge Guardrail and Routing

Instead of:

Guardrail LLM
     ↓
Router LLM

Finnie now uses a single classifier that returns structured output:

{
  on_topic,
  agent,
  needs_macro,
  reason
}

The model therefore answers two questions at once:

Is this request allowed?
Which agent should handle it?

The classifier uses structured model output rather than parsing JSON from arbitrary text. It also retains the existing blocklist fast path and fails closed: malformed output, errors, or an off-topic result cause the request to be rejected instead of accidentally bypassing the guardrail.

Why?

Because an architecture should not use two expensive probabilistic operations when one operation can produce both answers.

This is a useful rule for junior engineers working with LLM systems:

Before adding another model call, ask whether the information can be produced by an existing call.

The best optimization is often not making an AI call at all.

That leads to the next change.

Problem 2: Why Ask an LLM the Same Question Twice?

A large percentage of educational questions have reusable answers.

For example:

What is dollar-cost averaging?

and:

Can you explain dollar cost averaging?

are semantically almost the same question.

Running retrieval, routing, prompt construction, and generation again is unnecessary if the system has already produced a reusable answer.

Decision: Put a Semantic FAQ Cache Before the AI Pipeline

The new request path begins with a FAQ cache.

User
  ↓
FAQ Cache
  ├── hit → return answer
  │
  └── miss
        ↓
     Classifier
        ↓
       Agent

The cache first checks for an exact normalized query match.

If that fails, Finnie generates an embedding and searches cached questions using cosine similarity.

A sufficiently similar question—with similarity of at least 0.92—can reuse the stored answer.

A cache hit means:

LLM calls = 0

instead of three calls in the original architecture.

Why exact match first?

Because exact matching is cheaper.

There is no reason to calculate an embedding if:

"What is an ETF?"

is already in the cache.

This pattern comes up frequently in performance engineering:

cheap check
   ↓
expensive check

Do the cheapest reliable operation first.

But Semantic Caching Has a Dangerous Edge Case

Suppose somebody asks:

What is Apple's stock price?

Caching that answer for seven days would be a terrible idea.

Financial information can become stale very quickly.

The solution was not simply to cache everything.

Finnie caches only reusable answers from the finance_qa and tax_education agents that did not depend on:

live market data;
macroeconomic data;
user memories.

Cached entries are also stamped with a knowledge-base version and tax year.

This is an important caching principle:

The difficult part of caching is usually not storing data. It is deciding when cached data is still valid.

The design explicitly identifies semantic-cache false positives on time-sensitive financial questions as one of the largest risks, mitigated with the 0.92 threshold, version stamps, tax-year stamps, and exclusion of live-data answers.

Problem 3: FAISS Was Useful, but It Wasn't a System of Record

The original knowledge base was an in-process FAISS index.

FAISS is excellent at vector similarity search.

But an application eventually needs more than similarity search.

For Finnie, that included:

durable storage;
metadata filters;
incremental knowledge updates;
keyword search;
user persistence;
conversation checkpoints;
memories;
and a shared semantic cache.

Using separate databases for all of these would significantly increase operational complexity.

Decision: Use PostgreSQL + pgvector as the System of Record

The redesigned architecture uses Cloud SQL for PostgreSQL 16 with pgvector.

One database now supports:

Knowledge chunks
FAQ cache
User data
Conversation checkpoints
Long-term memories
LangGraph state

This was one of the most important architectural decisions.

Why PostgreSQL?

Not because PostgreSQL is the fanciest vector database.

Because it solved several problems at once.

The system needed relational persistence anyway.

LangGraph also provides PostgreSQL-backed checkpoint and store implementations.

Adding pgvector meant that vector retrieval could live in the same infrastructure.

Instead of:

Postgres
+ vector database
+ cache database
+ memory service

the architecture could start with:

Postgres + pgvector

and keep Redis only where it still made sense.

For a relatively small knowledge base—hundreds or thousands of chunks—introducing another dedicated stateful service would have added more operational complexity than value. The design explicitly treated a separate Qdrant, Pinecone, Weaviate, or Milvus deployment as unnecessary for this scale.

This is a broader architecture lesson:

The best technology is not necessarily the technology that is best at one task. It may be the technology that solves several important problems adequately while keeping the whole system simpler.

Problem 4: Vector Search Alone Wasn't Enough

Semantic embeddings are excellent for conceptual similarity.

But imagine somebody asks about:

P/E ratio

Keyword matching can sometimes locate exact terminology more reliably than embeddings.

That suggested another change.

Decision: Hybrid Retrieval

The new retriever performs two searches:

Vector similarity search
+
Postgres full-text search

Each produces a ranked list.

The lists are then combined using Reciprocal Rank Fusion, or RRF.

Conceptually:

          Vector Search
               ↓
            ranking
               \
                → RRF → final results
               /
            ranking
               ↑
        Keyword Search

RRF is useful because it combines rankings without requiring the scores from the two retrieval systems to be numerically comparable.

The implementation retrieves candidates from exact cosine search and Postgres full-text search, then fuses them. Category filtering happens directly in SQL through a WHERE clause.

The measured result on the application's real corpus was interesting:

MRR@5

pgvector hybrid retrieval: 0.950
FAISS:                   0.867

with equal Hit@5.

Why No HNSW Index?

This is perhaps one of my favorite decisions in the redesign because it illustrates an important engineering habit:

Do not optimize for scale you don't have.

Vector databases often use approximate nearest-neighbor indexes such as HNSW.

They are incredibly useful when searching very large collections.

But Finnie's knowledge base is relatively small.

At tens to thousands of chunks, exact vector search is already extremely fast.

So the architecture deliberately does not create an HNSW index.

Exact search gives perfect recall and avoids some of the complications approximate indexes can introduce when combined with metadata filtering.

The document proposes revisiting that decision once the corpus grows beyond roughly 10,000 rows.

This is a useful engineering mindset:

Build for the scale you reasonably expect next—not the scale of Google Search.

Problem 5: Every Refresh Made Finnie Forget the User

The original application stored user information in Streamlit session state.

The user ID was effectively "default".

So things like:

profile;
holdings;
financial goals;

could disappear after a refresh.

Conversation history was similarly stored as a Streamlit list capped at 40 messages.

That cap was not summarization.

It was simply deleting old history.

Decision: Add Durable User Identity and Persistence

Users authenticate through Google OAuth.

Finnie derives the internal user ID from Google's stable sub value:

g-<Google sub>

It is recomputed from the authenticated session instead of being accepted from page input.

Persistent user data then lives inside LangGraph's Postgres-backed store.

Conceptually:

("users", user_id)
    ├── profile
    ├── holdings
    ├── plan
    ├── session
    └── settings

Conversation threads receive IDs such as:

user_id:uuid

So a thread carries ownership information directly in its identifier.

Why does this matter?

Because identity should be part of the storage model—not something the UI happens to remember.

A multi-user system should be able to answer this question for every piece of state:

Which user owns this?

Why Application-Level Isolation Instead of Postgres Row-Level Security?

The design considered database-level row security.

It ultimately uses application-level isolation instead.

All access passes through:

UserData(user_id)

which binds operations to a user's namespace.

One reason was practical: LangGraph manages some of its own tables, and the design did not have a clean per-request hook for applying database-level policies.

Instead, cross-user isolation is explicitly tested with a leakage test suite.

This is an important nuance.

Architecture decisions are rarely:

perfect solution
vs
bad solution

More often they are:

Solution A fits the framework better
Solution B provides stronger guarantees
Solution C adds too much complexity

You choose based on the constraints of the actual system.

Problem 6: Prompts Were Accidentally Defeating Prompt Caching

LLMs often receive a large amount of repeated information:

system instructions;
policies;
domain knowledge;
agent behavior.

Modern model providers can cache stable prompt prefixes.

But Finnie's original system prompt mixed everything together:

stable instructions
+ retrieved documents
+ market data
+ user profile

Because dynamic information changed on every request, the prompt prefix changed too.

That effectively prevented prompt caching.

Decision: Split the Prompt into Three Layers

The new prompt structure is:

1. Shared Core
2. Agent Role
3. Request Context

Block 1 — Shared Core

Contains:

common Finnie instructions;
a digest of the knowledge base.

This block is identical across all six agents.

Block 2 — Agent Role

Contains:

agent name;
agent-specific instructions;
static reference material.

Block 3 — Request Context

Contains things that change:

user profile;
memories;
retrieved documents;
live financial data.

The first two blocks are cacheable.

The last one is not.

The ordering matters because prompt caching matches from the beginning of the prompt.

So the stable content goes first.

This is a simple but powerful optimization.

Instead of thinking of a prompt as:

one giant string

think of it as:

stable prefix
+
semi-stable role
+
dynamic request

Problem 7: Independent External Calls Were Running Serially

Imagine an agent needs:

stock data
+
sector data
+
macro data

If these calls are independent, this is inefficient:

fetch A → wait
fetch B → wait
fetch C → wait

If each takes 500 ms:

total ≈ 1500 ms

Instead, the redesigned system runs independent provider calls concurrently using gather():

fetch A ─┐
fetch B ─┼→ continue
fetch C ─┘

The total time becomes closer to the slowest individual request.

The architecture also avoids fetching the FRED macro snapshot unless the classifier determines that macro data is actually required.

Again, the lesson is simple:

Don't fetch data just because you can. Fetch it when the current request needs it.

Problem 8: Users Want an Assistant That Remembers

Persistence solves:

"Can I reopen my conversation?"

Memory solves a different problem:

"Does the assistant remember something important I told it last week?"

Those are not the same feature.

Decision: Add Selective Long-Term Memory

After an on-topic conversation, Finnie can extract durable facts such as:

goal
situation
preference
constraint

For example:

"I want to save for my daughter's college."

could become a memory.

The extraction runs in a background thread, which means the user does not wait for the memory model before receiving the answer.

Several safety filters were added.

The extractor:

skips the model unless the message contains a first-person word;
keeps only facts with confidence ≥ 0.8;
removes things resembling account numbers, card numbers, SSNs, email addresses, or passwords;
replaces sufficiently similar existing facts instead of endlessly duplicating them.

Users can also:

delete an individual memory
turn memory off
"Forget everything"

Why build this instead of using Mem0 or Zep?

Because Finnie's memory requirements were fairly narrow.

It needed durable personal facts, not a complicated temporal knowledge graph.

Using the existing LangGraph store also meant the data stayed inside the application's Postgres database.

The design evaluated Mem0 and Zep but chose an in-house extractor plus LangGraph Store because it was native to the existing stack, simple to delete from, and avoided moving user data into another system.

The Resulting Architecture

After these changes, the request path looks much closer to:

                   ┌──────────────────────┐
                   │      Streamlit       │
                   │   + Google OAuth     │
                   └──────────┬───────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │ Semantic FAQ     │
                    │ Cache            │
                    └───────┬──────────┘
                        HIT │      MISS
                            │        │
                            ▼        ▼
                          END    Hydrate user
                                   │
                                   ▼
                          Guardrail + Router
                              1 LLM call
                                   │
                                   ▼
                                Agent
                         ┌─────────┼─────────┐
                         ▼         ▼         ▼
                      pgvector   Live APIs   LLM
                         │      concurrent   cached
                         │                   prompt
                         └─────────┬─────────┘
                                   ▼
                                Answer
                                   │
                                   ▼
                          background memory
                              extraction

The design doc summarizes the architectural change succinctly: Finnie moved from an in-process FAISS index and session-only state to a PostgreSQL/pgvector system of record, with a semantic FAQ cache, merged classifier/router, prompt caching, and long-term user memory.

Deployment Architecture Matters Too

Architecture is not only about code.

A system can have perfect Python abstractions and still have unsafe deployments.

The redesigned Finnie deployment runs on Cloud Run using a dedicated least-privilege service account.

Secrets live in Google Secret Manager.

Cloud Build creates immutable images tagged by build ID.

Database migrations run before a new revision receives traffic.

That final point is particularly important.

Suppose new application code expects:

new database column

but the migration did not run.

Letting that revision receive traffic can cause immediate failures.

Instead:

build
 ↓
migration
 ↓
revision ready
 ↓
traffic

If the database cannot be reached during deployment, the new revision never becomes ready and the previous revision continues receiving traffic.

That is a much safer failure mode.

The Re-Architecture Was Intentionally Incremental

Another lesson from this exercise was not to rewrite everything at once.

The work was divided into phases:

Phase 1 — Fast path
Phase 2 — Prompt caching
Phase 3 — Persistence and identity
Phase 4 — pgvector knowledge layer
Phase 5 — Memory + Google Cloud
Phase 6 — Optional Meko curation

Each phase was protected by configuration flags and designed to be reversible. With no DATABASE_URL, the application could continue operating using the previous path.

At the time of the design document:

phases 1–5 were merged;
retrieval evaluation showed MRR@5 = 0.950 for the pgvector approach compared with 0.867 for FAISS;
cross-user isolation tests passed;
memory recall across sessions worked;
cross-agent prompt-cache reuse had been verified;
but production performance numbers were still waiting for the first deployment.

I think that distinction is worth making.

There is a difference between:

implemented
tested
deployed
measured in production

A design should say which one it means.

Things I Deliberately Did Not Build

A good architecture document should explain not only what was added, but also what was rejected.

Several technologies were considered.

A dedicated vector database

Qdrant, Pinecone, Weaviate, and Milvus are all capable vector systems.

But introducing another stateful service for only a few hundred vectors would have increased operational complexity without solving a pressing problem.

Redis as the primary vector store

Redis offers very fast lookup, but it was not a good fit as the application's durable system of record.

It remains useful as a data-client cache.

YugabyteDB

The PostgreSQL-compatible API meant it could theoretically fit.

But Finnie did not require distributed database scale.

Neon

Technically suitable, but Cloud SQL was selected because the rest of the application already runs on Google Cloud.

Mem0 and Zep

Both offered stronger dedicated memory capabilities.

But Finnie's memory model was simple enough that another platform would have added more infrastructure than benefit.

This kind of analysis is one of the most valuable parts of architecture work.

Architecture is not:

Which technology is coolest?

It is:

Which technology best fits the problem, scale, constraints, and operational model I actually have?

What I Learned From This Exercise

The most interesting part of this re-architecture was that no single change was revolutionary.

The improvement came from removing many small inefficiencies and inconsistencies.

A few principles stood out.

1. Eliminate work before optimizing work

Going from three LLM calls to two matters.

Going from three LLM calls to zero on a FAQ cache hit matters even more.

2. Durable state changes how you design an application

Once the system has multiple users, every piece of state needs ownership, persistence, and deletion semantics.

3. AI architecture is still distributed-systems architecture

Most of the difficult questions were familiar backend questions:

Where does state live?
What happens when a dependency fails?
How is data isolated?
How do we invalidate caches?
How do migrations happen?
What work belongs on the request path?
What can run asynchronously?

The fact that an LLM exists inside the application does not make those questions disappear.

4. Don't prematurely optimize for enormous scale

Exact vector search was good enough.

One PostgreSQL database was good enough.

An HNSW index was unnecessary.

A separate vector database was unnecessary.

Those decisions can change later.

Architecture should leave room for growth without forcing you to pay today's complexity bill for tomorrow's hypothetical traffic.

5. Make expensive operations earn their place

An LLM call, embedding operation, API request, or database hop should exist because the request actually needs it.

That principle drove several improvements:

semantic cache before LLM
exact match before embeddings
macro data only when requested
parallel external calls
background memory extraction
prompt caching for stable text

What Is Still Unfinished

The redesign also intentionally records unresolved problems.

Among them:

the first production deployment had not happened yet;
the merged classifier still needed a dedicated routing-accuracy evaluation;
tax tables still contained 2024 figures and needed to be refreshed from a verified source;
chat and memory retention policy still needed to be defined;
a runtime Postgres circuit breaker with FAISS fallback had not been built;
and the proposed Meko integration needed to be reconsidered because its available tools could not export datapack documents in the way the original design required.

I actually like having this list in the architecture document.

Engineering design should not pretend uncertainty doesn't exist.

A good design makes uncertainty visible.

Closing Thoughts

Finnie started as an AI application.

After this exercise, I increasingly thought of it as a distributed backend system that happens to use AI.

That change in perspective influenced almost every architectural decision.

Instead of asking:

Which AI framework or database should I add?

I started asking:

What is the minimum amount of work this request needs?

Which state must survive this process?

Who owns this data?

Can this operation fail safely?

Can the same infrastructure solve several problems?

Can I measure whether the new design is actually better?

Those questions led Finnie from:

FAISS
+ session state
+ serial LLM calls
+ transient conversations

toward:

Postgres + pgvector
+ durable user state
+ semantic caching
+ hybrid retrieval
+ prompt caching
+ long-term memory
+ concurrent I/O
+ safer deployments

And perhaps the biggest lesson was this:

Re-architecture does not necessarily mean replacing everything.

Some of the original pieces were already good—the LangGraph workflow shape, retry logic, circuit breakers, deterministic planning engine, and evaluation suite were retained.

Good architecture work is often less about rewriting the system and more about identifying which assumptions from version one no longer hold.

Then changing only those.