How We Taught Our Agent to Learn

That sounds like a diagram on a slide, and it would have stayed one, except that it settled arguments. Each rung has one job and each transition has one owner. Restating is not interpreting. Interpreting is not verifying. When a claim comes out wrong, you can say which rung produced it, which is never true of a design that does all three at once inside a single prompt.

Faithful is not the same as complete

Here is the idea that shaped the rest of the architecture, and it took us a while to see it clearly.

The information layer is faithful. It never invents or distorts. It is also extremely lossy: roughly 0.036 statements survive per row, so about 96% of rows are discarded at compression time. Faithful and complete are different properties, and the two kinds of thing we wanted to learn sit on opposite sides of that line.

Descriptive insights tolerate loss. "Revenue is roughly 70% concentrated in two customers" is exactly the right shape for a human building a mental model. Approximation is the point. These climb the ladder in one direction, data to information to knowledge, and stop. They never need the raw rows again.

Normative insights do not. A rule an agent acts on, paired with what to do when a record breaks it, is only trustworthy with its exact track record, and that count was thrown away during compression. No better extractor recovers it, because completeness was lost by construction rather than by accident. So norms travel a V shape through the ladder: up to the information layer to generate a candidate rule cheaply, then back down to the raw records for the exact count. The information layer is a hypothesis generator on that path, not a source of truth.

That single asymmetry generated most of the design. Two stores, because the two kinds of claim are consumed differently. Two consumption paths. And a deterministic aggregator that walks the rows and computes the ratio itself, because the whole point of descending is to get a number nobody guessed.

The pipeline

Stage one compresses every row into short factual statements. It does no judgment, no inference, no ranking. It will write "£42,300 paid to a vendor across 11 invoices." It will not write "this vendor is a key dependency," because that is an interpretation, and interpretation belongs to a later, named stage that owns it.

Keeping that stage stupid turned out to be load-bearing in a way we did not anticipate. We run it on a small model with thinking switched off. Turning thinking on made it worse: the model started improving the text, and improved text came back truncated. The dumb stage has to stay dumb.

Everything downstream reads that one shared information layer, built from every row rather than the interesting ones, so coverage is a property of the pipeline rather than something an agent decides.

Descriptive synthesis then runs in buckets grouped by topic and record type, and splits across two agents with one responsibility each. The generator proposes candidates from this run's new information plus a topic index of what is already known, so it hunts new angles instead of restating old ones, and it is judged on generation quality alone. The consolidator folds those candidates into the living set by emitting a transaction of lifecycle operations, add, update, merge, drop, demote, keep, applied deterministically. It never rewrites the whole set, and anything it does not touch carries forward byte-identical.

All of it lands in typed tables. An early design used a generic key-value blob, which is the natural thing to reach for and precisely the thing that cannot support a verifier, a denominator, or a per-consumer query. One consequence is worth naming: prose stops being where truth lives. Any guidance an agent eventually reads is generated from the stores and regenerable from them, and our last stage curates exactly that and files it as proposals.

Grounding

Then we added the piece that mattered most. Every claim has to cite the records it rests on. A cheap deterministic pre-check fails anything with no valid citation before a model sees it, and then a verifier from a different model family re-rates each surviving claim against only its cited sources. The verifier's number becomes the confidence. We built a field to hold the generator's own confidence in its output and then deleted it, because we could not name a question it answered honestly.

In testing, the verifier and the citation gate changed the outcome without changing a word of the prose: nothing about the writing got better, the claims simply got attached to their evidence.

The second thing citations bought us was not designed for. Every rung cites the rung below, so a claim walks back to the statements it cites and those walk back to raw rows. Nothing the system believes is a dead end.

Where we stopped asking the model

Once you start looking, the design turns out to be a list of places we took discretion away.

The extractor does not judge. The aggregator has no model in it at all: it walks raw records, computes the exact n/N, and that computed ratio is the confidence. The synthesizer may propose any pattern over any field, value, or population, but it has to express it as a typed predicate, and each shape has exactly one deterministic evaluator over raw rows, which turns scoring into a finite dispatch instead of text-to-SQL chaos.

The proposing model is bounded to a published catalog of fields per record type, so open-ended can never mean inventing a column. And for BOUNDED, the band is computed from the population's own mean and standard deviation rather than chosen by the model, because scoring a model against its own guess is circular.

Open-ended in what gets proposed. Closed-form in how it gets checked.

The reversals

The interesting part of the build was watching production data correct us.

One agent holding a contradiction. Our first descriptive synthesis was a single running reviser: one agent that read the new information and rewrote the whole insight set each run. It silently reworded stable claims, and worse, it forked. A single transaction of $64,692 got written up as both a customer payment and an expense, and the reviser held both claims happily, because nothing in the loop was responsible for reconciling them. Splitting it into a generator and a consolidator gave contradictions an owner. The consolidator now resolves them in the rows: it re-reads the handful of records behind the conflicting citations, drops the wrong claim, and says why, citing the row. It is the one point where the descriptive path goes back to raw data, and it goes there to disambiguate, never to count.

Trusting one clock. Every ledger row carries two dates with little to do with each other: when the transaction happened, and when someone typed it in. Our scheduler tracked the latest business date seen and re-read a trailing window for stragglers, which fails both ways. A bookkeeper reclassifies a March expense in November and the change is never seen, sitting far behind the watermark; meanwhile a new company imports three years of invoices in an afternoon, so one day's writes would have you re-reading three years. Neither clock knows what you haven't processed, but the pair does. Extraction became two passes: a catch-up pass with no model in it scans by system time and marks three-day business-date slots dirty, then a period extract re-reads each dirty slot by business date. A failed slot does not advance the watermark.

Rules that quietly evaporated. Norms were decaying across incremental runs: a rule found once would vanish on the next run that did not rediscover it. A rule like "payroll runs on the 28th, 12 months out of 12" would be found on a full pass, and then a quiet week with three new invoices and no payroll rows in it would fail to re-derive the rule, and the rule would simply be gone. Nothing had contradicted it. It just wasn't in that run's evidence. Fixed by routing norms by record type and persisting them across runs, so a rule is only removed by counter-evidence rather than by absence.

Running it

The learning logic is only half the build. Extraction is cheap and frequent, synthesis is expensive and rare, and decoupling the two costs nothing because each row is still extracted exactly once. The service owns no scheduler: it exposes idempotent single-company jobs driven by a per-company cursor, and an external orchestrator owns the clock and the fan-out across companies. That split is what makes it safe to run often: because each job is idempotent and driven by its own cursor, a crashed or re-run job resumes exactly where it left off without double-counting, and because companies are processed independently, one slow or broken company never holds up the rest.

The honest cost of those decoupled cadences is staleness. Knowledge can lag by up to about two weeks. That was a price we were willing to pay: descriptive insights are mental-model shaped, and a mental model that lags real activity by a couple of weeks is still a useful one.

What teaching actually meant

The system is not free. It does not decide what is worth noticing. It does not rate its own work. It does not estimate anything it could count. Most of the engineering was removing discretion from places where discretion quietly produces confident, unverifiable claims.

That is the part I would take to a different stack entirely. Teaching an agent to learn was mostly a matter of deciding where it was not allowed to think.

See how Astra, our always-on analyst, works inside Light, the agentic ERP. See the product or book a demo.