:
August 20, 2026
How we built a 12-step enrichment pipeline that turns every LLM decision into a supervised, deterministic rule — and gets cheaper and safer with every run.
In most AI products, a wrong answer is a bad user experience. In accounting, a wrong answer is a misstated financial record.
If a model files an internal transfer as revenue, your income statement is wrong. If it merges two vendors that happen to share a name, every bill and payment attached to them is wrong — and untangling that later is brutal. There is no “regenerate response” button on a general ledger.
We still wanted the leverage of an LLM. Modern models are genuinely good at reading a messy bank memo and telling you it's an annual software subscription. What they are not is accountable — and accounting is, definitionally, the domain of being accountable.
So we built our AI enrichment pipeline around a single organizing idea:
The LLM is a source of judgments, never a source of record. Every judgment it makes is captured, verified by code, put on probation, and only then allowed to touch the books.
This post is about the three mechanisms that make that idea real: rule shadowing (the LLM answers each question once, ever), programmatic validation (we never trust the model's arithmetic), and precision-first defaults (when in doubt, the system does less, not more). Along the way we'll share the numbers that convinced us it works.
Our platform ingests raw financial activity in both directions — bank feeds, card transactions, bills and expenses on the spend side; invoices and customer payments on the revenue side — from many upstream sources. Before any of it can flow into downstream recognition, every piece of activity has to answer four questions:
Each question sounds simple. Automating them is not. Names arrive spelled a dozen different ways across platforms, bank memos are cryptic, and the same-looking line can be a real expense one day and a transfer leg the next. Answering all four correctly, consistently, with no human in the loop by default — that's the whole game.
The answers come from a pipeline of twelve ordered steps in six phases, running per entity, where every step deliberately sets up the next.
We won't walk through all twelve steps here. What's more interesting is the architecture that sits underneath them, because it answers the question every fintech team building with LLMs eventually hits: how do you let a probabilistic model make decisions in a system that must be deterministic, auditable, and right?
Here's the pattern we see most teams reach for: put an LLM behind every classification, add a cache, ship it. It works — and it quietly commits you to paying inference costs forever, re-rolling the dice on every cache miss, and having no artifact a human can review except a pile of individual predictions.
We inverted it. In our system, the LLM's job is not to classify transactions. Its job is to author rules.
When a transaction reaches the categorization step, it walks a waterfall:
The interesting part is what happens after tier 2. The LLM's decision doesn't just get applied — it gets written down as a candidate rule: “transactions that look like this belong in that category.” And that rule enters shadow mode.
A shadow rule never touches a transaction. It just watches. Every time the LLM independently makes the same call on a matching transaction, the rule earns an observation. Every time the LLM disagrees with it, it earns a contradiction. Promotion is earned, not granted.
A taxonomy rule with fifteen clean, unanimous observations auto-promotes. One with even a single contradiction can never auto-promote — the best it can do is surface for human review, and only after overwhelming agreement (20+ observations at a 20:1 ratio).
Taxonomy isn't the only thing the LLM learns, though. Counterparty alias rules — “this raw bank string means this vendor” — go through the same lifecycle, but promote on stricter terms: an alias never auto-applies at all, no matter how spotless its record. Five clean observations earns it a place in the review queue; only an explicit human promotion makes it effective. The asymmetry is deliberate. A wrong category is a filing error you can re-file; a wrong identity link quietly corrupts every transaction attached to it.
Once promoted, a rule is just a deterministic match. Same input, same output, forever, explainable in one sentence, reviewable in one admin screen, disable-able with one click.
This architecture makes a measurable prediction: over time, the deterministic tier should eat the LLM tier. That's exactly what we observe.
[FIGURE 1 — The deflection curve. Weekly share of categorization decisions resolved deterministically (lookup + promoted rules) vs. requiring an LLM call, since launch. Placeholder: deterministic share climbing from [XX%] at launch to [XX%] today; per-transaction inference cost down [X]×.]
Today, [XX%] of categorization decisions never touch a model. Of the [X,XXX] rules the LLM has authored, [XX%] promoted cleanly, [XX%] are still in shadow or pending review, and [X.X%] were contradicted out of contention. Most telling: of all rules that ever reached effective status, only [X.X%] were later overturned or disabled by a human. Probation works.
There's a subtle second-order benefit here, too. Because the LLM keeps “voting” on transactions that shadow rules also match, the shadow ledger doubles as a continuous consistency audit of the model itself — we know, per rule, exactly how often the LLM agrees with its own past judgment. Which brings us to trust.
The newest phase of our pipeline answers the fourth question — is this a prepaid, and what service period does it cover? — and it's where our verification philosophy is at its most paranoid, because dates are where LLMs are at their most confidently wrong.
A prepaid is money paid now for value consumed later: a $24,000 annual subscription paid in January shouldn't land as a $24,000 January expense; it should spread across twelve months. The same problem exists in mirror image on the revenue side — cash a customer pays you up front for a year of service you haven't delivered yet — and our data model treats both as one concept: value that belongs to future periods. We'll use the spend side, prepaids, as the running example. Either way, downstream recognition needs two facts: the flag itself, and the exact service period. Get the period wrong by a month and every affected financial statement is wrong by a month.
So we drew a hard line: the LLM identifies evidence; the program does math.
The model never returns a computed end date. It returns a structured verdict — a start date, a term (“12 months”), and crucially an attribution: a pointer to where in the source text it found those facts. Except for the rare case where both dates are literally printed in the document, we compute the end date in code, with deterministic anniversary-exclusive date arithmetic (2025-02-09 + 12 months → 2026-02-08). The model isn't allowed to do calendar math because the model is bad at calendar math — and unlike a category, a hallucinated date looks exactly like a real one.
Then every single verdict passes through a programmatic validation gate before anything is persisted:
A validation failure persists nothing. The object stays unclassified, flagged for human review. We would rather have an honest NULL than a plausible fiction.
The third mechanism isn't a component — it's a posture, and it shows up everywhere in the pipeline.
Counterparty identity: only exact matches auto-merge. Counterparties span both sides of the ledger — the vendors you pay and the customers who pay you — and the same real company arrives as Stripe, Inc., Stripe Inc, and Stripe LLC. We fold those into one canonical identity — but only via deterministic name normalization that strips legal suffixes. A fuzzy match, even a very strong one, never auto-merges. Even a trigram similarity of 0.90+ is recorded only as a suggestion for a human — never merged automatically. Our favorite cautionary pair: cathay pacific and pacific catch — a fuzzy matcher's dream, an airline and a seafood restaurant. A missed merge costs a duplicate row you can join later; a wrong merge poisons every attached bill and payment. The asymmetry dictates the policy. In a manual audit of [X,XXX] auto-linked vendors, we found [zero] incorrect merges.
Ambiguity defers, automatically. Before the transfer matcher links two bank lines as an internal transfer, it checks whether an open bill or invoice could also explain either leg. If one exists, the match is ambiguous — so it isn't made. The pair is surfaced to a human instead of auto-linked into a half-explained story.
Confidence gates are calibrated, not vibes. LLM category assignments below 0.7 confidence are never written — recorded, with the reason, but never applied. And the threshold isn't folklore: because every prediction is logged alongside the eventual human decision, we can continuously measure whether the model's stated confidence tracks its actual agreement with reviewers, and move the gate if it drifts.
Every one of these defaults trades recall for precision, and that's the point. In this domain, the expensive failure mode is never “the system didn't act.” It's “the system acted wrong, confidently, at scale.”
A few decisions that raised eyebrows internally and turned out to be the right call:
No pre-filters in front of the prepaid classifier — yet. The obvious cost optimization is to filter before the LLM: skip small amounts, skip categories that are “never” prepaid, skip anything that looks monthly-recurring. We shipped v1 with none of that. Category, amount, and merchant type are passed to the model as signals, never used as gates — because a “software” charge can be a monthly seat or an annual prepaid, and a filter that's wrong is a prepaid silently dropped with no audit trail. The filters are planned, but they'll be proven safe against v1's own labeled results before they're switched on. You can't validate a shortcut until you've walked the long way once.
No separate recurring-transaction detector. Instead, each prepaid prompt carries the vendor's recent history — dates, amounts, prior prepaid periods. “24 charges, same amount, ~30 days apart” reads as a monthly subscription; one charge 12× the usual size reads as an annual prepaid. The recurrence detector didn't disappear — it moved into the prompt, where the model can weigh it against everything else instead of being overruled by it.
No watermarks for the backfill. is_prepaid is tri-state: NULL means not yet evaluated. The pipeline only ever scans NULL rows — which means the column itself is the resume cursor. Every object is classified exactly once, ever; the backfill is trivially resumable; and there is no watermark to corrupt, drift, or reason about. The cheapest infrastructure is the infrastructure encoded in your data model.
Where this lands, as of this writing:
The industry conversation about LLMs in high-stakes domains tends to collapse into two camps: “the models aren't ready” and “the models are ready, ship it.” Our experience is that both camps are asking the wrong question. The models are ready to judge. They are not ready to decide — and the distance between those two words is exactly the system you have to build.
Ours came down to three commitments. Convert every model judgment into a reviewable, deterministic, probationary artifact — so intelligence compounds instead of being re-purchased per token. Verify mechanically everything that can be verified mechanically — attribution over trust, computed dates over claimed ones. And make doing less the default under uncertainty — because in accounting, an honest gap is a task for tomorrow, but a confident error is a restatement.
The result is a system where the LLM gets smarter about our data every week while touching it less — and where, when an auditor asks “why is this transaction categorized this way?”, the answer is never “the model said so.” It's a named rule, with a history, a probation record, and a human who could have vetoed it.
That, we think, is what it actually means to put AI into production finance.
If these problems sound fun — teaching models to be accountable, building the deterministic scaffolding around probabilistic cores — we're hiring. [Link placeholder.]
