Topic · AI
Small and efficient models
82 developments in the last 30 days.
Small and efficient models AI
42 developments this week
+3 vs last week
Stage: Coverage
Opportunities
- Bulk Pass: batch extraction over CSVs on the new fast, cheap modelsA CLI that tests a sample, picks the cheapest model that passes, then runs your whole file. Data-heavy small teams pay.First flagged 2026-10-17
- Diffusion and Chinese flash models squeeze the small-model tier, and the DeepSeek ban splits buyers by originFast, cheap models for routine agent work. Inference platforms and regulated buyers pay, sorted by where the model came from.First flagged 2026-10-17
- Frontier MoE models stream experts from NVMe, adding pressure on HBMInference that pulls only the experts each token needs from DRAM and flash. Cost-sensitive operators running their own hardware pay.First flagged 2026-10-13
- Rig Fit, now for expert streaming: size RAM and NVMe, not just VRAMA calculator and shared benchmark list that predicts tok/s for MoE models spread across VRAM, RAM and NVMe. Local-LLM buyers pay.First flagged 2026-10-11
- Barge Test: interruption and disclosure testing for AI debt-collection callsSimulated borrowers who talk over, dispute and say 'stop' to AI collection voice agents. Collection agencies and voice vendors pay.First flagged 2026-10-17
- Memory Write: a local memory server running a 3B-active writer modelAn open-source MCP memory server that uses Hy-Memory-30B-A3B for the write path. Developers pay for sync.First flagged 2026-10-17
- Origin Swap: move a Qwen or DeepSeek fine-tune onto US open weights and prove parityA CLI that re-runs your fine-tune on Nemotron, Muse or Beam and compares results on your evals. Startups selling to US buyers pay.First flagged 2026-10-17
- Turn Kit: drop-in full-duplex turn-taking for open-source voice agentsA plugin that runs Tyger's answer/stop/chime-in model inside voice-agent frameworks. Indie voice-agent builders pay for the hosted version.First flagged 2026-10-17
- Escalate Only When Lost: small-model drafts with frontier checks for back-office agentsA local 24GB model drafts tool-call trajectories and a frontier model checks them only at risky points. Fintech ops teams pay.First flagged 2026-10-16
- China's export rule hands the open-weight enterprise market to US labsRegulated enterprises standardise on US open-weight models they host themselves. Revenue comes from fine-tuning, serving and support contracts.First flagged 2026-10-10
- Meta and Apple make the Mac a home for open frontier weightsMuse Spark Open on Apple silicon and Nvidia's 4GB–24GB Nemotron models let Mac apps skip API bills.First flagged 2026-10-16
- Shelf Shots: consistent product photos from ten reference images on an 8GB GPUProduct-photo generator for Etsy and Shopify sellers that keeps the item exact across scenes. Sellers pay per batch.First flagged 2026-10-16
- Zero-bill docs assistant: a 2B model running in the visitor's browserEmbeddable docs Q&A widget where inference runs on the reader's GPU via WebGPU. Maintainers pay nothing per question.First flagged 2026-10-16
- Runway Bands: probabilistic MRR forecasts from Stripe data already in your PostgresSelf-serve forecasting that reads synced Stripe tables and shows indie SaaS founders revenue ranges. $19–49 a month.First flagged 2026-10-16
- Nvidia ships its own 24GB models, and the local agent stack takes shapeSparse models that run agents on phones and 24GB cards. Device makers and privacy-bound enterprises pay.First flagged 2026-10-14
- Statement Desk: local agent over your own bank exports, now that JPMorgan says noDesktop app plus MCP server that lets a local on-device model reason over your bank CSVs. Prosumers pay.First flagged 2026-10-14
- On-device coached-scam check inside the payment flowA small model on the phone, inside the bank app, flags coached payments before they send. Neobank fraud teams pay.First flagged 2026-10-07
- Scam Second Opinion: on-device check of suspicious chats and screenshotsBrowser extension and phone app that judge suspicious messages locally with small models. Worried families pay.First flagged 2026-10-15
- Open-vocabulary speech on $5 chips threatens cloud voice licensingOffline speech recognition for appliances and devices without cloud fees, paid for by hardware OEMs per unit.First flagged 2026-10-12
- Model Origin Scan: find Chinese-developed models in a fintech's stackOpen-source scanner that lists every model, weight file and AI endpoint in a codebase, with lineage and origin. Fintechs pay.First flagged 2026-10-13
- Microsoft enters decision models, and the category becomes a platform featureFast classify-and-route models are being bundled by hyperscalers, which squeezes independents while Jev shows real revenue.First flagged 2026-10-10
- Stack Bill: spend and overlap tracker for people paying hundreds a month on AITracks every AI subscription and API account, shows cost per tool and overlap, suggests cuts. Heavy prosumers pay.First flagged 2026-10-11
- Haiku-priced E2E testing for indie web appsPlain-English end-to-end tests run by a cheap computer-use model against your staging site on every deploy, which removes most of the work of writing Playwright tests.First flagged 2026-10-09
- Decision Swap: codemod and router that moves classify-style LLM calls to decision modelsCLI that finds yes/no and pick-one LLM calls in your code and reroutes them to cheaper decision APIs; dev teams pay.First flagged 2026-10-10
- Invoice Sense: does this invoice fit this business, checked on every paymentFlags invoices whose goods, prices or routes don't fit the payer's business. FCA-registered EMIs and payment fintechs pay.First flagged 2026-10-10
- Weights Witness: proof your inference provider serves the model you validatedStatistically checks that a hosted open-weight endpoint is the exact model risk teams approved. Bank model-risk teams pay.First flagged 2026-10-10
- Guardrail Fuzzer for typed decision modelsAn open-source red-team harness that tests whether your allow/deny decision-model guardrail can be pushed into failing open with one-word option and definition attacks.First flagged 2026-10-09
- Fair Shake: matched-pair fairness testing for AI banking assistantsAutomated test personas that probe a bank's AI assistant or adviser for unequal pricing, offers or advice by wealth, age or location, then produce an evidence report regulators will accept.First flagged 2026-10-08
- Green-screen agents for credit union and mortgage back officesHaiku 5.5 computer-use agents that work old core-banking and loan-origination screens (the systems with no API) and check every action against what the screen shows afterwards.First flagged 2026-10-08
Developments
Last 30 days- Inception debuts Mercury 3, a diffusion LLM that it says tops GPT-6 Luna, Claude Haiku 5.5, and GLM-5.3 Flash on agentic benchmarks while being 4x to 8x faster (Jason Hiner/The Deep View)Inception's Mercury 3 diffusion LLM says it beats small frontier models on agentic tasks while running 4-8x faster. Techmeme · 2026-10-20
- TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token PruningTokenGate is a small drop-in add-on that learns to skip unimportant input tokens, cutting prefill compute and memory on long documents and code. arXiv · 2026-10-20
- A bare IPython harness ties production coding agents with Qwen3.5-9B on TB-Lite, and tool-output normalization fixed most of the run-to-run varianceA bare single-tool IPython harness matched production coding agents with a 9B model, and cleaning up tool output removed most run-to-run variance. r/LocalLLaMA · 2026-10-20
- Tandem: Serving Hybrid Linear-Attention GUI Agents on Mobile NPUsA system for running phone GUI agents on mobile NPUs, so tasks run on-device and screen contents stay private. arXiv · 2026-10-20
- We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.A small team built its own inference engine to fit a 32B model with 262K context into a 16GB GPU. r/LocalLLaMA · 2026-10-20
- Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language ModelsReplacing a small model's embedding table, the bulk of its size, with one frozen 16 MB vocabulary sharply shrinks small and ternary models. HF Daily Papers · 2026-10-19
- AuroraASR-1M Releases!AuroraASR-1M is a 1MB Apache-licensed speech model that runs 436x real-time on one CPU thread. r/LocalLLaMA · 2026-10-19
- Kapitsa: open-source Apple Silicon optimized Metal engine built for Qwen and Nemotron. See benchmarks of Qwen 27B, Nemotron + 8 other models on an M1 Pro (16GB), M4 Pro (48GB) and M5 Max (128GB)Kapitsa is an open-source Metal inference engine for running local coding agents on Macs in air-gapped enterprises. r/LocalLLaMA · 2026-10-19
- Xiaomi-Lens-0 Technical ReportXiaomi's Lens-0 is a 4B-parameter multimodal model built on Qwen3.5 that predicts masks to understand specific regions of an image. arXiv · 2026-10-19
- nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 · Hugging FaceNVIDIA released Dory, a 4B looped-architecture reasoning model with 256K context under an open license. r/LocalLLaMA · 2026-10-18 NVDA
- RTX Spark Ignites Global Celebration as New Windows PCs ArriveLaptops with Nvidia's RTX Spark chip and up to 128GB of unified memory are now on sale from $2,399, making strong local AI possible on slim Windows PCs. NVIDIA Blog · 2026-10-18 NVDA
- Xolotl (Strata based) - DeepSeek-V4.1-Flash (552B) on a single 4090 at ~27 tok/s, Uncensored 1017 quant coming in a couple of daysA hobby inference engine runs the 552B DeepSeek V4.1 Flash on a single RTX 4090 at about 27 tok/s by streaming experts from RAM and NVMe. r/LocalLLaMA · 2026-10-18
- Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV StreamingAdaptive KV streaming in llama.cpp lets Qwen 27B use its full 262k context on a 16GB AMD card at 80+ tok/s. r/LocalLLaMA · 2026-10-18
- Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardwareLlmspeed, a popular Show HN, lets people feel how fast a local LLM runs before they buy the hardware. Hacker News · 2026-10-18
- RoleQ: Role-Aligned Scales for Converting Language Models to TernaryRoleQ converts existing language models to ternary weights without training from scratch, cutting memory needs for inference. arXiv · 2026-10-18
- Open From Data Center to Desktop: NVIDIA Expands Nemotron Family With Highest Efficiency Super VL and Bolt ModelsNVIDIA releases open Nemotron 3.5 Super VL 120B plus edge Bolt models down to 4GB for local agents. NVIDIA Blog · 2026-10-17 NVDA
- Meta launches an open-weight AI model that runs on MacsMeta released Muse Spark Open, an open-weight model built with Apple to run locally on Apple silicon. TechCrunch AI · 2026-10-17 META AAPL
- Black Forest Labs' open-weights image model FLUX 3 Image Klein runs on consumer GPUs with just 8 GB of VRAMFLUX 3 Klein 5.6B open image model does native 4K on 8GB GPUs. The Decoder · 2026-10-17
- Granite Switch 4.1: Chat, Decisions, and Speech from One EndpointIBM Granite Switch 4.1 serves chat, decisions and speech from one endpoint. Hugging Face Blog · 2026-10-17 IBM
- nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 · Hugging FaceNVIDIA releases Nemotron Labs Dory 4B small model. r/LocalLLaMA · 2026-10-17 NVDA
- Qwen3.8-Flash-Next-GSQ-RCO (IQ3_S): ~40-50 tok/sec decode & 1300-1500 tok/sec prefill at 131k ctx on 12GB VRAM + 32GB RAM + NVMEQwen3.8 Flash Next runs at 40-50 tok/s at 131k context on 12GB VRAM. r/LocalLLaMA · 2026-10-17
- Ternary Bonsai 2 (2B, 4B, and 9B) just released on Hugging Face. The 2B can run entirely locally in the browser at up to 500 tok/s on an M4 Max with custom WebGPU kernels.Ternary Bonsai 2 small models are out, and the 2B version runs fully in the browser at up to 500 tokens per second via WebGPU. r/LocalLLaMA · 2026-10-16
- mistralai/TimestralMistral released Timestral, open time-series foundation models from 5M to 209M parameters for zero-shot forecasting, useful for fintech forecasting work. r/LocalLLaMA · 2026-10-16
- ALoDLM 8b looped diffusion language model: ~3x inference speed vs its base model qwen3-8b with better benchmark resultsALoDLM is an 8B looped diffusion language model that runs about 3x faster than its Qwen3-8B base and scores better on benchmarks. r/LocalLLaMA · 2026-10-16
- What a £300, 35W Snapdragon mini PC can actually do 👀: 35B coder at ~30 tok/s, and an 80B model that doesnt even fit in RAM running at 17 tok/sA £300 Snapdragon mini PC runs a 35B mixture-of-experts coding model at about 30 tokens per second on CPU alone, showing how cheap local inference has become. r/LocalLLaMA · 2026-10-16
- QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releasesQUASAR released 4-bit quantization-aware checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer that stay closer to full precision than Google's and NVIDIA's 4-bit releases. r/LocalLLaMA · 2026-10-16
- NVIDIA Nemotron Bolt 3.5 74B A9B releases Oct. 15thNVIDIA's Nemotron Bolt 3.5 is a 74B-parameter MoE (9B active) that fits in 24GB using 2-bit quantization-aware training. r/LocalLLaMA · 2026-10-15 NVDA
- LFM2.5-24B-A2B: Built for fast on-device agentsLiquid AI's LFM2.5-24B-A2B is a sparse model with about 2B active parameters, built for fast on-device agents, which cuts the cost of running agents locally. Hacker News · 2026-10-15
- A 35B multimodal model running on a phone with ~3GB memory? MiniCPM-V 4.7 demo is pretty wildMiniCPM-V 4.7, a 35B-A3B vision-language model, runs on a phone in about 3GB of memory. r/LocalLLaMA · 2026-10-15
- Run fast, private agents on-device with LFM2.5-24B-A2BLiquid AI's LFM2.5-24B-A2B, a model with only 2B active parameters, makes fast, private on-device agents practical. Hugging Face Blog · 2026-10-15
- Google DeepMind and Singapore Government collaborate to build fine-tuned AI model for scam detectionGoogle DeepMind and GovTech Singapore fine-tuned small on-device Gemma models to catch chat scams without sending data to the cloud. Google DeepMind · 2026-10-15 GOOGL
- New model from Telperion: Qwen3.8-27B-MLA & Qwen3.8-27B-GLA-g2. 4x smaller KV cache, smarter, and more efficient reasoning.Telperion's latent-attention variants of Qwen3.8-27B cut KV cache memory 4x. r/LocalLLaMA · 2026-10-15
- Web search for a 1.7B model running fully on a phone, no search API and no backend (open source)Private Mind, an open-source app, adds web search with citations to a 1.7B model running fully on a phone, with no search API or backend. r/LocalLLaMA · 2026-10-15
- Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding AgentsKeeping context compaction transparent, rather than hiding it behind model-written summaries, helps lightweight coding agents retain facts and commitments in long sessions, a practical tip for agent builders. arXiv · 2026-10-15
- Your Transformer Is Secretly ElasticA method without retraining lets one pretrained transformer run at flexible sizes, so deployers can match model size to hardware and load instead of picking from fixed S/B/L options. arXiv · 2026-10-15
- Very impressed with LFM2.5-24-A2B in Pi AgentAn early user reports LFM2.5-24B-A2B performs well inside the Pi Agent setup. r/LocalLLaMA · 2026-10-15
- JetBrains Mellum 2.1 12B A2.5B MoE is a very good model.JetBrains' Mellum 2.1, a 12B MoE with 2.5B active, codes fast on consumer GPUs. r/LocalLLaMA · 2026-10-15
- LiquidAI/LFM2.5-24B-A2BLiquid AI's LFM2.5-24B-A2B uses only 2B active parameters out of 24B, built to run fast, private agents on-device. r/LocalLLaMA · 2026-10-14
- We turned EmbeddingGemma 2 into a speech-to-text modelA team turned the small EmbeddingGemma 2 model into a speech-to-text model. Hacker News · 2026-10-14
- DeepSeek Solved AI’s Memory ProblemAn explainer on DeepSeek-OCR's approach of compressing context into images to ease AI memory limits. Two Minute Papers · 2026-10-14
- Show HN: Pulse Local, a Mac budgeting app whose AI models run on your machinePulse Local is a Mac budgeting app whose AI models run entirely on the user's machine. Hacker News · 2026-10-14
- DoubtLess: Training-Free Contrastive Decoding for Efficient ReasoningDoubtLess shortens reasoning chains without any training, which cuts inference token costs. arXiv · 2026-10-14
- Agaram 2.5B: The First Tamil-First Language Model Trained from Scratch with Tool Calling. A Playbook for Under-Represented LanguagesA 2.5B Tamil-English model with tool calling was trained from scratch on a single node of eight H100s in about six days, a low-cost playbook for under-represented languages. arXiv · 2026-10-14
- openbmb/MiniCPM-V-4.7-35B-A3BMiniCPM-V 4.7 adds an efficient 35B-A3B mixture-of-experts version, the first in the series, built for image and video understanding on devices. r/LocalLLaMA · 2026-10-13
- Oído: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural AcceleratorRuns open-vocabulary speech recognition on a $5 ESP32 microcontroller, crossing a cost threshold for edge voice products. arXiv · 2026-10-13
- Running unsloth/Qwen3.8-27B-UD-IQ4_XS 262k q4_0 KV + vision on 16GB VRAM @ 80t/s and 1500ppA user reports running a quantized Qwen3.8-27B with a 262k context and vision on a 16GB GPU at about 80 tokens per second. r/LocalLLaMA · 2026-10-13
- Boole-1.7B: A 1-bit Qwen3-1.7B model that outperforms Bonsai-1.7BBoole-1.7B is a 247MB 1-bit version of Qwen3-1.7B that keeps 86% of the full model's average score across 10 tasks. r/LocalLLaMA · 2026-10-13
- DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs4-bit quantization of both weights and activations without calibration data, for low-bit accelerators. arXiv · 2026-10-13
- Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked!A builder got 8% faster output from a MoE model streamed off SSD on a Mac Mini by predicting which experts to load two layers ahead. r/LocalLLaMA · 2026-10-13
- Update: Converting dense models into Mixture-of-ExpertsA hobbyist is converting small dense Qwen models into mixture-of-experts models and releasing GGUF versions. r/LocalLLaMA · 2026-10-13
- openbmb/MiniCPM-V-4.7-1BMiniCPM-V 4.7 also comes in a 1B on-device vision model. r/LocalLLaMA · 2026-10-13
- Receipt scanner that runs a fine-tuned 0.8B vision model on the phone (free) - detailed cost trackingA solo developer built an on-device receipt scanner using a fine-tuned 0.8B vision model for item-level spending tracking. r/LocalLLaMA · 2026-10-12
- Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boardsDecision 3.0 is an Apache-2.0 family of multimodal decision models from 0.6B to 27B, built on Qwen, for routing and decisions. r/LocalLLaMA · 2026-10-12
- Whistle: speech to text in 16.9 MBWhistle is a speech-to-text model in just 16.9 MB. r/LocalLLaMA · 2026-10-12
- Microsoft unveils Microsoft-Decision-1, a fast decision-scoring model trained on Qwen3.5-9B, and says it will soon rebase it on MAI, OpenAI, and other models (Achint Srivastava/Command Line)Microsoft launched Decision-1, a fast decision-scoring model built on Qwen3.5-9B, and plans to rebase it on MAI, OpenAI and other models. Decision models are emerging as their own category. Techmeme · 2026-10-11 MSFT
- Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.Qualcomm's CEO says AI companies want phones running 100B-parameter models continuously by 2028. r/LocalLLaMA · 2026-10-11 QCOM
- The Most Important Trends in New AI ProductsThis week brought ChatGPT's intelligent UI, Claude Haiku 5.5 and GrokBot's use of rival models, plus Nous Research reaching a $1.5B valuation, all pointing to cheaper intelligence and competition shifting to the experience around the model. The AI Daily Brief · 2026-10-11
- Microsoft's Decision-1 model enters the fast-growing AI decision model raceMicrosoft's Decision-1 is a 9B model built for fast classification and routing (85 ms), feeding demand for cheap model routers. The Decoder · 2026-10-11 MSFT
- Converting dense models into Mixture-of-ExpertsA hobbyist is converting dense models into cheaper Mixture-of-Experts models without pretraining from scratch. r/LocalLLaMA · 2026-10-11
- The Best New AI Products and the Key Trends They RepresentWeekly roundup: Claude Haiku 5.5 retakes the cheap-model frontier, ChatGPT now generates its own interfaces, and Ramp spending data shows where enterprise AI money goes. The AI Daily Brief · 2026-10-11