Chat Leak Scan: find the analytics and session-replay scripts reading your chatbot's conversations
Headless scanner that types canary prompts into a chat UI and catches third-party scripts sending them out. Paid by AI chat app founders.
Early · First measurement of chatbot leakage
Written for: Solo builder
Next step
This week, build the Playwright canary scan, run it against 50 AI chat apps listed on Product Hunt, privately email the founders of the ones that leak with their results, and post the aggregate numbers (no names) on Hacker News.
Why now
Every small AI chat app ships with analytics and session replay by default, and users type medical, financial and intimate details into them. Lawsuits are already testing what chatbots owe their users. Separately, the Cathedral breach of Anthropic through third-party software shows third-party code is where exposure sits. The scanner itself is a weekend of Playwright work, and founders will pay to know before a researcher or a journalist finds out.
What became possible
'Penny for Your Prompts' is the first systematic measurement of AI chat platforms leaking conversation content to third-party analytics, advertising and session-replay scripts running alongside the chatbot. The leakage channel is now documented and can be measured the same way on any site.
Two ways in
- Chat Leak Scan
Enter a chatbot URL. A headless browser types a unique canary string, captures every outbound request, and reports which third-party domains received the canary or its keystrokes, with the script responsible and a suggested config fix (masking rules, replay exclusions).
- Customer
- Founders of consumer AI chat, companion, health and finance assistant apps
- Model
- Free single scan. $29/month for monitoring after every deploy, with a shareable 'no leaks found' badge
- Who's Reading
Browser extension that flags, while you chat, when a third-party script on the page reads the chat input.
- Customer
- Privacy-minded consumers and journalists
- Model
- Free as the top of the funnel for the paid scanner
Competition
The risks are that Blacklight's open-source collector makes the core easy to copy, so the moat is chat-specific depth and the fix advice, not detection. Enterprise vendors could add a chatbot preset. Session-replay vendors themselves may start masking chat content by default. Founders may also see this as a one-time check rather than a subscription unless it monitors continuously after each deploy. The consumer extension idea, Who's Reading, has no polished direct competitor beyond research prototypes, but consumer privacy extensions are hard to make money from. It probably works best as a free marketing tool feeding the paid scanner.
Full competition notes (8 competitors)
I found no product that does exactly this: type a canary into a chatbot UI, catch which third-party domains receive it, name the script responsible, and give the config fix. The nearby players split into three groups. Free generic scanners like Blacklight catch session recorders and key logging in general, but they are not chat-aware, do not track the canary through streamed bot replies, and give no fixes. Enterprise client-side security tools (F5, Feroot, Reflectiz, c/side) and tag-governance and CIPA tools (ObservePoint, Lokker) are sold to security or legal teams at large companies. Several 2026 academic studies already ran this exact method by hand across dozens of chatbots, found prompts leaking to Microsoft Clarity, and got press for it. The opening is a cheap, self-serve tool aimed at small AI chat, companion, health and finance app founders. It should handle the details of chat apps: login walls, streaming replies, prompts that show up in page titles and sidebars, and vendor-specific masking settings for Clarity, Hotjar, FullStory, PostHog and others. It could run as a CI check and produce a report a lawyer can use as evidence, given the wave of CIPA chatbot wiretap lawsuits. The risks are that Blacklight's open-source collector makes the core easy to copy, so the moat is chat-specific depth and the fix advice, not detection. Enterprise vendors could add a chatbot preset. Session-replay vendors themselves may start masking chat content by default. Founders may also see this as a one-time check rather than a subscription unless it monitors continuously after each deploy. The consumer extension idea, Who's Reading, has no polished direct competitor beyond research prototypes, but consumer privacy extensions are hard to make money from. It probably works best as a free marketing tool feeding the paid scanner.
- The Markup Blacklight (and open-source blacklight-collector)
Free real-time website privacy scanner. A headless browser visits a URL and reports trackers, session recorders and key logging on forms. The collector is open source on GitHub (github.com/the-markup/blacklight-collector). It is generic: it types into forms but does not drive a chat UI, follow a canary through replies, or suggest masking fixes. It is the closest existing free alternative and a ready-made base to build on.
- Tracking Conversations: Measuring Content and Identity Exposure on AI Chatbots (UC Davis)
2026 research covering 20 chatbots. Three of them (Genspark, SeaArt, ChatOn) sent plaintext prompts and responses to Microsoft Clarity session replay. The method matches the product idea, but it is an academic study, not a product. It shows the problem is real and in the news (project page privacy-genai.github.io).
- Penny for Your Prompts (arXiv 2610.37467)
Very recent measurement study of 42 AI chat platforms. 21 leaked conversation titles and 9 leaked verbatim prompts to third-party trackers, with Clarity reading visible page text. Research, not a tool. It adds public pressure that a paid scanner can sell into.
- F5 Distributed Cloud Client-Side Defense
Enterprise client-side security product. Its docs say it detects form-field reads and network exfiltration of PII by scripts. It is aimed at Magecart-style malicious scripts on enterprise sites, not at well-meaning analytics on AI chat apps, and it is sold to security teams.
- Feroot Security
Client-side security and privacy platform. It maps where sensitive fields are collected and where browser data is sent, and it blocks unauthorized scripts. It is priced for enterprise, PCI and HIPAA compliance buyers. It could add a chatbot preset.
- Reflectiz and c/side
Third-party script monitoring vendors. They do continuous behavioral analysis of external scripts and the data they touch. They are enterprise web security tools, not chat-specific, and have no canary-prompt workflow.
- ObservePoint and Lokker
Privacy and tag-governance scanners marketed for CIPA wiretap risk. ObservePoint publishes lists of chat vendors to flag. They scan for which tags and pixels are present, not whether chat content actually leaves the page. Sold to legal and marketing ops at larger companies.
- chatbot-scanner (Yusuf-Hridoy)
Open-source CLI that drives a website's chat widget in a real browser and uses canary strings. Its canaries test system-prompt leakage and PII in bot replies, not third-party network exfiltration. It shows that automating chat UIs for audits is already being done in the open.
Checked 2026-10-28
Risks
- Logged-in flows and bot protection make automated scanning harder
- Findings about named apps need careful disclosure
- Low urgency until regulators or press act on it
Strong tech readiness and team fit; market pull is the weak point (5).
tech 9 · pull 5 · space 7 · fit 9 · build 9
Exposed
Companies that could benefit or be threatened. News and context, not investment advice.
Momentum of its main topic
Sources
Status history
- 2026-10-28 Flagged as buildable