Compact AML Validation: weighted test sets and rerun checks for LLM transaction monitoring
Builds small, statistically valid test sets and run-to-run flip rates for LLM AML triage; bank model-risk teams pay.
Competitive · 271x smaller AML eval sets
Written for: Small startup team
Next step
Call the head of model risk management at a fintech sponsor bank such as Cross River Bank. Ask how they validated the last LLM-based alert-triage tool a partner brought them, and what it cost in analyst hours.
Why now
The earlier Eval Error Bars card argued for confidence intervals on agent evals in general. Today there is a published, AML-specific sampling protocol and hard numbers on rerun instability. Model validators can now run a cheap, defensible test of an LLM alert-triage system on the bank's own alert history and state how often its verdicts flip. That fits exactly into the 'verification tax' surveys describe.
What became possible
AMLworld-Compact keeps every illicit transaction, samples benign ones by supervised-score strata, and reweights with Horvitz-Thompson estimators. That cuts evaluation targets 271x and 611x while keeping precision and F1 estimates unbiased. A separate study of 1.2 million LLM calls shows 0.07% to 10.2% of rows change their answer on rerun at temperature 0, so variance can now be measured and reported.
Two ways in
- Validation harness
Takes a bank's labelled alert history, builds a stratified, HT-weighted test set, runs the vendor or in-house LLM triage model, and reports weighted precision and recall with intervals plus the per-case flip rate over N reruns.
- Customer
- Model risk management teams at sponsor banks and mid-size banks piloting LLM alert review
- Model
- Per-validation engagement fee ($15-40k), then a subscription for revalidation whenever the model or prompt changes
- Vendor evidence pack
The same harness sold to AML AI vendors, so they arrive at bank procurement with validation results already in the format model risk teams expect.
- Customer
- AML and fraud AI startups selling to banks
- Model
- Annual license per product
Competition
The statistics engine is already taken. There are at least five small open-source packages (errorbars, evalsig, evalci, abeval, agent-regress-cli) that put CIs, paired tests, clustered SEs and power analysis on top of eval runs.
Full competition notes (10 competitors)
The statistics engine is already taken. There are at least five small open-source packages (errorbars, evalsig, evalci, abeval, agent-regress-cli) that put CIs, paired tests, clustered SEs and power analysis on top of eval runs. Inspect AI and lm-eval-harness already report standard errors themselves. A standalone evalstats CLI would be a commodity, and AI engineering teams are unlikely to pay for it. The one technical part I found no product for is splitting variance by prompt format, run order, temperature and judge (a generalizability-theory study plus a decision study). That exists only in 2026 papers, so it could set a tool apart, but it is a feature rather than a company. The more defensible opening is the Validation Pack for regional banks and fintech lenders. ValidMind and Monitaur sell MRM documentation and governance workflow, but I found nothing that produces a statistically rigorous uncertainty statement for agentic and LLM vendor evals that a validator can sign. That means sample-size justification, paired comparisons and a variance breakdown written in effective-challenge language. A new entrant could win by selling validation evidence through MRM consultancies or alongside ValidMind rather than against it. The main risk is demand. A July 2026 CRA analysis reports that the new Fed guidance SR 26-2 explicitly leaves generative and agentic AI out of scope, which weakens the regulatory push. I have not checked that against Fed or OCC sources. Banks would then be validating GenAI by choice, not because a regulator requires it, so budgets may be smaller and harder to win. ValidMind could also add CI and power-analysis tests to its library with little effort.
- errorbars (PyPI)
An open-source package that says it implements Evan Miller's Anthropic paper. It covers CLT and clustered standard errors, paired variance reduction and power analysis, which is most of what the evalstats core would do.
- evalsig (PyPI)
Apache-2.0, at v0.1.0. It does paired tests, clustered SEs, minimum detectable effect and power, sequential testing and release gating. It presents itself as a layer on top of Inspect AI and lm-eval-harness, which is very close to the evalstats pitch.
- evalci
Gives confidence intervals on benchmark accuracy, paired tests, power analysis, clustered SEs and multiple-comparison correction for comparing LLMs. It is installed from GitHub source.
- abeval
Treats an eval as an A/B test. It provides single-run CIs, paired significance tests and sample-size planning, using only the standard library.
- agent-eval / agent-regress-cli
A CLI for statistical regression testing of agents. It runs each version many times and reports Mann-Whitney p-values, Cohen's d and bootstrap CIs. It was launched on Product Hunt.
- Adding Error Bars to Evals (Anthropic paper)
The methodology this idea rests on is published and free: the super-population framing, paired differences, clustered SEs and power planning. This makes the statistics a commodity.
- ValidMind
A model risk management platform with a developer framework that runs validation tests and generates documentation aligned with SR 11-7, SS1/23 and E-23, including for LLMs and LLM judges. It is the most direct threat to the Validation Pack.
- Monitaur
An AI governance and assurance platform that links policies and controls to validation evidence through a model and use-case inventory. It sells to regulated industries such as insurance and banking.
- FairPlay MRM Manual for Agentic Systems
A fair-lending AI vendor that publishes an MRM handbook for agentic systems covering validation reports and evidence preparation. It already sells to lenders.
- Generalizability-theory eval research (TEE, CyclicJudge)
2026 papers that split eval variance by prompt variant, judge, replication and temperature, with decision studies for choosing replicate counts. The method behind the variance-splitting feature exists, but I found no shipped product built on it.
Checked 2026-10-11
Risks
- Banks may not share labelled alert data with a three-person vendor without a long security review
- Validation consultancies can copy the published method
- Labels in alert history are noisy, which limits how far the statistics can be trusted
Strong team fit; open space is the weak point (5).
tech 7 · pull 6 · space 5 · fit 9 · build 7
Exposed
Companies that could benefit or be threatened. News and context, not investment advice.
Momentum of its main topic
Sources
- AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error DiagnosisHF Daily Papers
- AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error DiagnosisarXiv
- Same Query, Different Answer: Run-to-Run Variance in Semantic Query ProcessingarXiv
- As economic and corporate research shows limited AI efficiency gains, surveys by Workday, Google, MIT, and others detail a human "verification tax" bottleneck (Sarah Kessler/New York Times)Techmeme
What happened next
- +7d · 18 Oct 2026 nothing yet. No products, funding rounds, acquisitions, or press coverage matching the AML validation opportunity were found for the period after October 11, 2026. A seven-day window combined with the specialized nature of LLM-based model risk validation for banks makes this an unlikely timeframe for market activity to surface.
Status history
- 2026-10-11 Flagged as buildable