Automated Financial Data Pipeline

The question

Anyone who has pulled a transaction export knows the problem: dates in four formats, amounts written as (1,250.00) or -48.2K or 1.200,50 €, the same vendor spelled six ways, categories missing, rows duplicated. Language models are good at this kind of tidying, but they are priced by the token and the biggest ones cost fifty times more than the smallest. Nobody tells you where the extra money stops buying accuracy.

So the thesis asks one main question, with five smaller ones behind it:

1. Is a bigger model worth it? How much accuracy does each step up a provider's ladder buy, in points per extra dollar, and where do the returns flatten out?
2. Is any of it better than plain rules? A free, hand-written cleaner is the baseline every model has to beat.
3. Which errors do bigger models fix? Formatting mistakes a rule could catch, or the ones that need context, like knowing that AAPL is Apple?
4. How often do they make things up? Some fields are deliberately unrecoverable. The right answer is a blank, and a model that guesses gets caught.
5. Does size matter more when the data is messier? The same records at 10%, 25% and 40% error rates.
6. Can a cheap model with good tools match an expensive one? The same model, once on its own and once with my cleaning functions handed to it as tools.

How it works

Data with the answers in hand

Real financial data comes without an answer key, so I generate my own. The generator builds 500 clean transactions across 30 real companies, 10 categories and 5 currencies, with log-uniform amounts so small charges are common and huge ones are rare, the way real ledgers look. Then it makes three messy copies of the same records, hitting each field with errors at a low, medium or high rate, and logs every single error it adds. That log is what makes honest scoring possible: for every run I know exactly which values were fixed, which were missed, which clean values got broken, and which blanks were filled in with a guess.

There is also a held-out set: the same companies and categories, but with new memo wordings and new name variants that neither my rules nor my prompt examples were written against. Nothing gets tuned on it. The same random seed always produces the same data, and each run records a fingerprint of the exact files it saw, so a result can never be quietly compared against data that has since changed.

Three kinds of cleaner, one output format

Every cleaner has to return the same seven columns, so the scorer never cares which one produced them.

• Rules. Plain Python, no AI, $0. My own parsing functions read six date formats including Excel serials, understand accounting negatives, K and M suffixes and European decimals, match company names by spelling similarity, and guess categories from keywords in the memo. Anything they cannot read, they leave blank rather than guess.
• LLM, direct. The model gets the formatting rules, the official company list, a worked example and a batch of 25 messy rows, and returns structured records that are validated against a schema. LangChain gives every provider the same interface, so swapping models is one line.
• LLM with tools. The same model, but my cleaning functions are handed to it as tools. The model decides what to call and makes the judgment calls, such as recognizing a ticker or a brand name, while the functions do the exact formatting. The idea under test is that a cheap model with good tools might match an expensive one working alone.

The model lineup is a price ladder from each of three providers, from the cheapest tier to the most expensive, plus a free 8-billion-parameter model running locally on my own machine as the floor. The main ladder is run three times at every strategy and messiness level, because models at their default randomness do not give the same answer twice, and the spread is part of the result.

Guardrails, because every call costs money

The thesis budget is small, so the pipeline is built to make it impossible to overspend by accident.

• Every API call's cost is written to a ledger the moment it is paid for, and a run stops before any batch that would push total spending past the cap, even across runs in different windows.
• A dry run estimates tokens and cost with no API calls at all. A finished run cannot be repeated by accident, and a failed one is retried as a clearly labeled new attempt.
• Every batch's prompt, tool calls, tool results, answer and any error are saved as an audit trail, along with token counts, the exact model version reported by the provider, and the prices in effect that day.

Scoring that is hard to game

A record only counts as right if it comes back exactly once and every field matches the truth, amounts to the cent, with trap fields left blank. Beyond plain accuracy, the scorer separates the things a business would actually worry about: how many errors were fixed, how many clean values were broken in the process, how many unrecoverable blanks were filled in with a made-up value, how many records were invented or dropped, and whether the column totals per currency still add up. That last check matters because a cleaner that is 95% accurate but quietly doubles a payroll line is worse than one that is 90% accurate and leaves the totals alone.

The analysis, in R

The scored runs go into an R script that produces the thesis results: accuracy against cost on a log scale, one line per model family and strategy, with the rules baseline as a dotted line; the accuracy points bought per extra dollar at each step up a ladder; a heatmap of which error types each cleaner fixes; and a total-cost model that adds the time a person would spend fixing the records the cleaner got wrong, with a sensitivity grid over how long a fix takes and what that person is paid. Because the same 500 transactions appear in every run, the logistic regressions cluster their standard errors by transaction, and adjacent model sizes are compared with McNemar tests so a one-point difference is not mistaken for a finding.

Where it stands