Zero-bill docs assistant: a 2B model running in the visitor's browser
Embeddable docs Q&A widget where inference runs on the reader's GPU via WebGPU. Maintainers pay nothing per question.
Early · Ternary Bonsai 2 at 500 tok/s
Written for: Solo builder
Next step
Fork the webml-community Bonsai WebGPU space, add in-browser keyword search over one project's docs, and time first answer on a mid-range Windows laptop. Pitch the result to one open-source project in its Discord.
Why now
A ternary 2B in the browser shifts the inference cost of a docs assistant onto the visitor's machine. That matters to open-source projects and indie SaaS that can't justify a Kapa- or Inkeep-style bill. It's a watch because 500 tok/s is a top-end Mac figure, and answer quality at 2B over retrieved docs is unproven.
What became possible
Ternary Bonsai 2 ships in 2B, 4B and 9B sizes on Hugging Face. The 2B runs entirely in the browser at up to 500 tokens per second on an M4 Max, using custom WebGPU kernels, and a public demo space already exists.
Two ways in
- docs-ask widget
Script tag that loads the model on first question, runs keyword search over a prebuilt docs index in the browser, and answers with links to sources.
- Customer
- Open-source maintainers with static docs sites
- Model
- Free widget, $15 a month for analytics on unanswered questions
- Server fallback
Routes to a cheap hosted model when the visitor's device lacks WebGPU or is too slow.
- Customer
- Indie SaaS docs teams
- Model
- Metered fallback pricing
Competition
The most defensible version is probably a thin, open-source abstraction that picks the best available engine on the reader's device: Chrome's built-in model, then WebLLM, then a cheap server fallback. That should be paired with excellent static indexing and citation quality, and sold to indie SaaS teams through a paid fallback tier rather than to open-source maintainers, who will expect it to be free.
Full competition notes (10 competitors)
I found no packaged product that does exactly this: a drop-in script tag that runs a small LLM on the reader's GPU over a prebuilt static docs index. Hosted docs Q&A is crowded. Kapa, Inkeep, Algolia Ask AI, Mintlify, Biel and Tluma all compete there, and the WebGPU building blocks (WebLLM, Transformers.js, BrowserAI) plus demos like web-llm-embed are free and open source. So this is early as a product but easy to copy technically. The opening is a zero-config build plugin for Docusaurus, MkDocs, VitePress, Starlight and Sphinx that produces the index and a widget, aimed at open-source maintainers who cannot justify $499 a month and do not want to run a server. Pricing-wise, the "no per-question bill" pitch is weaker than it looks. Tluma already offers free unlimited hosted answers to open-source projects, and Algolia gives DocSearch away for free and could easily do the same with Ask AI. There are also hard user-experience costs. The reader has to download a 1 to 2 GB model on their first question, WebGPU support is uneven (especially on mobile), and a 2B model gives noticeably worse answers than hosted ones. Many maintainers may prefer a free hosted tier for those reasons. Chrome's built-in Gemini Nano Prompt API could also remove the need to ship a model at all. The most defensible version is probably a thin, open-source abstraction that picks the best available engine on the reader's device: Chrome's built-in model, then WebLLM, then a cheap server fallback. That should be paired with excellent static indexing and citation quality, and sold to indie SaaS teams through a paid fallback tier rather than to open-source maintainers, who will expect it to be free.
- Kapa.ai
The market leader for hosted docs Q&A. It installs with one script tag on Docusaurus, Mintlify, GitBook and ReadMe. Third-party sites quote about $499 a month, which is too expensive for most small open-source projects. Inference runs on its servers.
- Inkeep
A hosted docs assistant with headless UI components. It is source-available and has a free self-hosted Docker tier, but that still means running your own server and paying for model calls.
- Algolia DocSearch Ask AI
DocSearch is free for open-source and technical docs and is built into Docusaurus through an askAi config option. Ask AI is sold as an optional add-on and runs on Algolia's servers. This is the most natural incumbent to add a free AI tier.
- Tluma
A Docusaurus GitHub discussion says Tluma offers a free, unlimited Ask AI widget for public open-source projects. It is hosted, but it already competes on the zero-cost-to-maintainer pitch.
- Biel.ai
A hosted Ask AI chatbot plugin for Docusaurus. The widget loads from a CDN and indexing runs against your live site.
- ask-my-site
An open-source Docusaurus plugin with a floating Ask AI button that renders in the browser and needs no API keys at runtime. Without build-time keys it falls back to a keyword-only index. It does not run an LLM in the browser.
- Orama
An open-source search engine that runs in the browser and has an answer engine. Answers go through Orama Cloud's Secure Proxy to OpenAI models, so search happens on the client but generation happens on a server.
- web-llm-embed
An open-source demo of document chat that uses WebLLM for generation and Transformers.js for embeddings, with a static GitHub Pages version on its roadmap. It is the closest technical match, but it is a demo rather than an embeddable docs widget.
- WebLLM / BrowserAI
Open-source libraries for running LLMs in the browser over WebGPU, which this product would be built on. They are not docs products. The BrowserAI author admitted it adds little on top of WebLLM and transformers.js.
- Chrome Prompt API (Gemini Nano)
Google's on-device model built into Chrome. Its docs name answering questions about page content as a use case. It is a platform-level threat: maintainers could call it for free with no 2B model download, though it is desktop Chrome only and needs a lot of storage and a strong GPU.
Checked 2026-10-16
Risks
- Model download size and first-load time may drive visitors away
- Speed on mid-range laptops and phones is far below the M4 Max figure
- 2B models hallucinate even with retrieval
- Funded docs-assistant vendors already exist
Strong team fit; tech readiness and market pull are the weak point (5).
tech 5 · pull 5 · space 6 · fit 9 · build 7
Momentum of its main topic
Sources
Status history
- 2026-10-16 Added to the watchlist