Cue Check: test whether your document pipeline drops strikethroughs and highlights before agents read policy

The problem is already recognized and partly solved. LlamaIndex's ParseBench names dropped strikethrough as a critical failure and publishes per-parser formatting scores. Reducto ships an opt-in change_tracking mode aimed at redlines and policy edits. LlamaParse extracts Word tracked changes. Tensorlake, undocx and pandoc already turn insertions and deletions into explicit tags. That makes the cue-preserving shim the weak half of the idea. It wraps converters that are adding the same feature natively, and for DOCX tracked changes it competes with free tools. The audit is the more defensible half. I found nobody selling a vendor-neutral test that generates formatting-dependent questions from a lender's or insurer's own policy documents, runs them through their actual converter and agent, and reports loss by cue type. Public benchmarks run on generic corpora, are built by parser vendors, and stop at parser output rather than agent answers. A new entrant could win by selling that audit to model-risk and AI-governance teams at banks and insurers as independent validation evidence, under SR 11-7-style model risk management and examiner scrutiny. Coverage could also go beyond strikethrough, for example highlighted exceptions, colour-coded applicability, comments, and scanned redlines in PDFs. The risks are real. Buyers could run ParseBench themselves or simply turn on Reducto's or LlamaParse's formatting options, and parser vendors could release customer-corpus evaluation tools. The audit probably works best as a recurring compliance or regression-testing service, not a one-off report.

• ParseBench (LlamaIndex)

  Open-source parser benchmark for agents. Its Semantic Formatting dimension scores strikethrough, superscript, bold and title hierarchy, and it measures highlight and underline at rule level. It explicitly calls a silently dropped strikethrough a critical error. Published scores range from about 1% for Docling to 85% for LlamaParse Agentic. It runs on a fixed public corpus, not the customer's own policies, and the vendor that built it also tops it.

• ParseBench evaluation skill

  Community agent skill for running ParseBench to compare parsers before ingestion. It shows that do-it-yourself parser evaluation is already packaged for free.

• LlamaParse

  Commercial parser that leads ParseBench on semantic formatting. It reportedly added extraction of Word tracked changes and reviewer comments as structured metadata, so it is a direct substitute for the shim.

• Reducto change_tracking

  Opt-in parse option that tags strikethrough as <s> and underline as <u> and wraps each revision in <change>. It uses a segmentation model for scanned pages and pitches compliance tracking of policy edits. This is close to the cue-preserving shim built into a leading converter, though it is off by default, which is the kind of setting an audit would catch.

• Tensorlake DOCX parsing

  DOCX parser that keeps tracked changes as HTML <ins> and <del> in its output.

• undocx

  Open-source Rust crate that converts DOCX to markdown, rendering insertions as <ins> and deletions as strikethrough, with formatting options.

• Pandoc --track-changes=all

  Widely used free option for keeping DOCX insertions and deletions when converting. It covers the Word tracked-changes part of the shim at no cost. (This link is Tiptap's tracked-changes import docs, not pandoc's own.)

• Unstructured parsing evaluation guidance

  Converter vendor publishing guidance on scoring parsers against labelled ground truth. It covers text, tables, reading order and structure but not formatting-borne meaning specifically.

Checked 2026-10-22