Retest Audit: does your AI adviser's allocation swing when the risk answer wobbles?

I found no commercial product built specifically to perturb the risk-tolerance answer by realistic amounts and report how far an LLM adviser's allocation moves, which client profiles are most fragile, and which guardrails reduce the swing. The nearby players come in two kinds. General LLM eval and model-risk platforms (Promptfoo, Openlayer, Patronus, ValidMind, CIMCON, QuantPi) offer the plumbing, and some market FINRA or SR 11-7 mapping. Academic papers from 2026 document that the instability is real. So the gap is a narrow, finance-native methodology: realistic noise models for questionnaire answers, allocation-distance metrics, fragility rankings by profile, and guardrail comparisons, packaged as an evidence artifact for Reg BI reasonable-basis and FINRA 3110 supervision files. Regulators have not prescribed a consistency methodology, so a credible, citable standard audit could become the reference. The Model Change Retest is the weaker of the two ideas on its own: Openlayer, ValidMind and Promptfoo already sell version-linked regression and sign-off records, and any of them could add a stability test cheaply. The main risks are that the market is still small (few robo-advisers ship LLM-generated allocations; most still use deterministic models with an LLM layer on top), and that firms may build this in-house on Promptfoo. A new entrant would win on domain credibility and on publishing benchmark fragility results that create demand, not on tooling.

• Promptfoo (finance solutions)

  An open-source LLM testing and red-teaming tool. Its finance page maps test scenarios to FINRA 3110, 2210, Reg BI and the Advisers Act, and names robo-advisers and planning copilots as targets. Its focus is generic compliance and safety tests, not measuring how much allocations move when the risk answer is perturbed. A firm could build such a suite on it, which makes it the most likely do-it-yourself substitute.

• Openlayer

  An eval and monitoring platform aimed at regulated industries. It stresses version-linked performance records and sign-off evidence for auditors, which overlaps directly with the Model Change Retest idea. It has no finance-specific allocation stability metrics.

• Patronus AI (financial services)

  An LLM evaluation vendor with a financial services offering, focused on hallucination and accuracy tests. Not allocation stability.

• ValidMind

  A model risk management and validation documentation platform for banks (SR 11-7, EU AI Act). It is the natural home for compliance-file validation reports and could add a stability test module.

• CIMCON AIValidator

  A model risk platform that tests LLMs for hallucinations and vulnerabilities and generates validation documentation. It is not finance-specific.

• QuantPi PiCrystal

  A black-box AI testing engine covering robustness, bias and fairness, used with Unique, a wealth-management AI vendor. Its robustness testing is the closest commercial analogue to a perturbation-based audit, but it is generic.

• BiasMix-Finance (academic)

  A 2026 preprint on post-generation KYC guardrails. It treats LLM allocations as drafts and applies deterministic caps and repairs. This is related to the allocation-band guardrails the audit would evaluate. Research only, not a product.

• Post-Challenge Fragility of LLMs in Financial Decisions (NeurIPS 2026)

  Academic work showing that financial recommendations flip under 'are you sure?' prompts at rates from 4% to about 50% depending on the model. This supports the problem, but it is a study, not a commercial service.

• One Size Fits None: Heuristic Collapse in LLM Investment Advice (academic)

  A paper finding that LLMs overweight stated risk willingness relative to risk capacity. This is the fragility the audit would measure. Research only.

Checked 2026-10-19