Topic · AI
Inference cost
96 developments in the last 30 days.
Inference cost AI
53 developments this week
+10 vs last week
Stage: Coverage
Opportunities
- Bulk Pass: batch extraction over CSVs on the new fast, cheap modelsA CLI that tests a sample, picks the cheapest model that passes, then runs your whole file. Data-heavy small teams pay.First flagged 2026-10-17
- Diffusion and Chinese flash models squeeze the small-model tier, and the DeepSeek ban splits buyers by originFast, cheap models for routine agent work. Inference platforms and regulated buyers pay, sorted by where the model came from.First flagged 2026-10-17
- Amazon opens PyTorch on Trainium, and CUDA's moat is attacked at the framework layerRunning PyTorch code unchanged on non-Nvidia chips, paid for by hyperscalers and model labs buying cheaper compute.First flagged 2026-10-12
- PRD Trace: a requirement-by-requirement check that an agent's PR did what the spec askedA GitHub app that maps each spec requirement to diff lines and tests and flags gaps. Small product teams pay per repo.First flagged 2026-10-20
- Origin Check, now with a federal DeepSeek ban: model-origin scans for government vendorsFinds Chinese-developed models in code, gateways and laptops and produces attestations; fintechs selling to government pay.First flagged 2026-10-13
- Port Desk: VB6 and Access ports for small businesses, now that OpenAI claims a legacy-language leapAgent-assisted rewrites of a small firm's VB6 or Access app as a web app. Owners pay a fixed price per port.First flagged 2026-10-11
- The neocloud shakeout sorts survivors by who holds a lab contractGPU clouds are being repriced, and frontier-lab offtake is now the credit that decides which ones get financed.First flagged 2026-10-09
- Labs bundle web search for agents, and standalone search APIs lose pricing powerModel labs now sell search and fetch alongside tokens. Agent builders pay less, and independent search-API vendors get squeezed.First flagged 2026-10-18
- Search Shim: a drop-in search-and-fetch tool over DeepSeek's $0.75 API, with fallbackAn open-source MCP server and hosted router that defaults agent web search to DeepSeek's cheap API. Indie agent builders pay.First flagged 2026-10-18
- Google open-sources self-tuning agent harnesses, and the app layer's hand-built moat thinsAgents that rewrite their own prompts, tools and memory for each task. Durable value moves to whoever owns the grader and replay data.First flagged 2026-10-19
- Harness Tune: AGENTS.md edits proposed by Harness2, kept only if replays improveA CLI that reads failed coding-agent sessions, suggests instruction edits, replays them and keeps only the edits that win. Developers pay.First flagged 2026-10-18
- Trail Report: a verifiable stolen-funds trace for small crypto theft victimsAn agent traces stolen funds to exchange deposit addresses and writes a hop-by-hop report for police and exchanges. Victims and small DAOs pay per report.First flagged 2026-10-19
- Coding platforms swap default models, and lab revenue looks rentableData on how much frontier-lab revenue comes through app platforms that can change default models overnight. IPO investors would pay.First flagged 2026-10-11
- Frontier MoE models stream experts from NVMe, adding pressure on HBMInference that pulls only the experts each token needs from DRAM and flash. Cost-sensitive operators running their own hardware pay.First flagged 2026-10-13
- Rig Fit, now for expert streaming: size RAM and NVMe, not just VRAMA calculator and shared benchmark list that predicts tok/s for MoE models spread across VRAM, RAM and NVMe. Local-LLM buyers pay.First flagged 2026-10-11
- Escalate Only When Lost: small-model drafts with frontier checks for back-office agentsA local 24GB model drafts tool-call trajectories and a frontier model checks them only at risky points. Fintech ops teams pay.First flagged 2026-10-16
- Review Bill: predict what an agent code review will cost before the PR landsGitHub Action that scores PR code health and estimates agent-review token spend for teams paying per token.First flagged 2026-10-16
- Origin Swap: move a Qwen or DeepSeek fine-tune onto US open weights and prove parityA CLI that re-runs your fine-tune on Nemotron, Muse or Beam and compares results on your evals. Startups selling to US buyers pay.First flagged 2026-10-17
- China's export rule hands the open-weight enterprise market to US labsRegulated enterprises standardise on US open-weight models they host themselves. Revenue comes from fine-tuning, serving and support contracts.First flagged 2026-10-10
- Bill Replay, now with a harness swap: test Argon in your own agent loop before switchingReplays your real agent tasks across models and harness settings to report cost per completed task. Dev teams pay.First flagged 2026-10-09
- Policy Check: automated credit-policy review of SME loan filesChecks each SME loan application against the lender's own credit policy and logs exceptions. Neobank credit teams pay.First flagged 2026-10-15
- Nvidia ships its own 24GB models, and the local agent stack takes shapeSparse models that run agents on phones and 24GB cards. Device makers and privacy-bound enterprises pay.First flagged 2026-10-14
- Agent sandboxes commoditise, and the value moves to saved stateRuntimes that save, branch and restore an agent's workspace, paid for by teams running long coding and ops agents.First flagged 2026-10-13
- Flush Runner: fork-and-flush restarts for coding agentsA wrapper for Claude Code and Codex that clears chat history, keeps workspace checkpoints and branches retries. Heavy agent users pay.First flagged 2026-10-13
- Swarm Check: does your multi-agent setup beat one agent on your own tasks?Harness that replays your tasks through solo and team agent configs, reporting pass rate and spend; agent builders pay.First flagged 2026-10-12
- Multi-agent swarms hit a cost wall, and token-cutting middleware formsGateways and audits that cut agent token spend, paid for by enterprises whose agent bills outrun the value delivered.First flagged 2026-10-09
- Tool Fuzz: argument fuzzing for MCP servers, built on the ToFu methodCLI and GitHub Action that attacks MCP tools with model-written arguments; server authors pay for private repos.First flagged 2026-10-12
- Tool Fuzz: CI security tests for MCP servers that move moneyGitHub Action that fuzzes agent tool endpoints with adversarial LLM arguments; fintechs publishing MCP servers or agent toolkits pay.First flagged 2026-10-12
- Agent skills need a package registry, and nobody runs one yetProvenance, versioning and security scanning for SKILL.md files that coding agents run with user permissions.First flagged 2026-10-10
- Skill Ledger: provenance and fix tracking for agent skills in your reposInventories every SKILL.md in a company's repos, traces where it was copied from, and flags missed fixes. AppSec teams pay.First flagged 2026-10-10
- Stack Bill: spend and overlap tracker for people paying hundreds a month on AITracks every AI subscription and API account, shows cost per tool and overlap, suggests cuts. Heavy prosumers pay.First flagged 2026-10-11
- Microsoft enters decision models, and the category becomes a platform featureFast classify-and-route models are being bundled by hyperscalers, which squeezes independents while Jev shows real revenue.First flagged 2026-10-10
- Decision Swap: codemod and router that moves classify-style LLM calls to decision modelsCLI that finds yes/no and pick-one LLM calls in your code and reroutes them to cheaper decision APIs; dev teams pay.First flagged 2026-10-10
- Invoice Sense: does this invoice fit this business, checked on every paymentFlags invoices whose goods, prices or routes don't fit the payer's business. FCA-registered EMIs and payment fintechs pay.First flagged 2026-10-10
- Weights Witness: proof your inference provider serves the model you validatedStatistically checks that a hosted open-weight endpoint is the exact model risk teams approved. Bank model-risk teams pay.First flagged 2026-10-10
- Schema Lint: catch the structure tax in JSON-mode promptsLinter and A/B harness that reorders output schemas to reasoning-first and measures the accuracy gain; LLM app developers pay.First flagged 2026-10-10
- Model Swap Lab: cost-per-task re-validation for regulated AI workflowsWhen a cheaper model such as Haiku 5.5 ships, replays a fintech's real workloads on it and measures true cost per completed task, including tokenizer effects. The output is a model-change memo for risk sign-off.First flagged 2026-10-09
- AI pipeline leakage audit for regulated dataScans a fintech's AI stack for leaks that teams treat as safe: router and gateway logs that reveal sensitive topics, and multi-vector document indexes that can be turned back into the original pages.First flagged 2026-10-08
- Feature Forge: turning unstructured SMB data into validated credit featuresUses an LLM offline to write deterministic feature code from long text such as shipment notes, broker emails and statements, so credit models can use those signals without calling an LLM at decision time.First flagged 2026-10-09
- Haiku-priced E2E testing for indie web appsPlain-English end-to-end tests run by a cheap computer-use model against your staging site on every deploy, which removes most of the work of writing Playwright tests.First flagged 2026-10-09
- Lookback Engine: AI re-review of historical transactions for AML remediationA cheap first-pass AI review of years of past transactions and closed alerts. It produces the case files and audit evidence that banks under AML enforcement must hand to regulators.First flagged 2026-10-07
- Green-screen agents for credit union and mortgage back officesHaiku 5.5 computer-use agents that work old core-banking and loan-origination screens (the systems with no API) and check every action against what the screen shows afterwards.First flagged 2026-10-08
- Secure Pay Capture for voice and chat agentsLets any AI voice or chat agent take a card or bank payment without card data ever reaching the model, transcripts or the merchant's servers.First flagged 2026-10-07
- Generated interfaces threaten BI and dashboard incumbentsGPT-6's Intelligent UI and Claude Dashboards build live, interactive views from a prompt, which pushes BI value toward governed data and metric definitions.First flagged 2026-10-09
- Fair Shake: matched-pair fairness testing for AI banking assistantsAutomated test personas that probe a bank's AI assistant or adviser for unequal pricing, offers or advice by wealth, age or location, then produce an evidence report regulators will accept.First flagged 2026-10-08
- MAS-ready AI pre-implementation review packA review workbench that turns a bank's planned AI project into the evidence package an independent reviewer needs: risk sizing, test results, agent action logs and sign-offs.First flagged 2026-10-08
Developments
Last 30 days- MiniMax M3.1 FlashMiniMax released M3.1 Flash, a fast, cheap model aimed at lowering inference cost. Hacker News · 2026-10-20
- Inception debuts Mercury 3, a diffusion LLM that it says tops GPT-6 Luna, Claude Haiku 5.5, and GLM-5.3 Flash on agentic benchmarks while being 4x to 8x faster (Jason Hiner/The Deep View)Inception's Mercury 3 diffusion LLM says it beats small frontier models on agentic tasks while running 4-8x faster. Techmeme · 2026-10-20
- Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!DeepSeek open-sourced DeepInfer, a high-performance large-scale inference framework for its models. r/LocalLLaMA · 2026-10-20
- MiniMax M3.1 Flash shows China's AI price race has no floor in sightMiniMax M3.1 Flash matches DeepSeek's coding scores at under $0.30 per task, with open weights coming Oct 23, so inference prices keep falling. The Decoder · 2026-10-20
- Mercury 3: Diffusion at scaleInception's Mercury 3 runs diffusion language models at scale, which could cut how long models take to respond. Hacker News · 2026-10-20
- DeepSeek V4.1 Flash 1017 (Max)DeepSeek put out V4.1 Flash, an updated efficient model. Hacker News · 2026-10-20
- The Programmable Data Center — Evan Conrad, SF ComputeSF Compute explains why running a GPU marketplace forced it to build its own cloud operations. Latent Space · 2026-10-20
- Parallel Language Models via Variational DecodingParallel language models generate several consistent tokens per forward pass, which could cut inference latency. arXiv · 2026-10-20
- TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token PruningTokenGate is a small drop-in add-on that learns to skip unimportant input tokens, cutting prefill compute and memory on long documents and code. arXiv · 2026-10-20
- Which Model for Which Role? Cost-Efficient Pareto Recommendation for LLM Agentic WorkflowsA cost-aware search method recommends which model to use for each role in multi-model agent workflows, trading off quality, cost and latency. arXiv · 2026-10-20
- RouterWorld: Scaling Conversational AI with Latency-Aware LLM RoutingRouterWorld is an LLM routing benchmark that treats latency as a first-class metric alongside answer quality. arXiv · 2026-10-20
- DeepSeek search and fetch – $0.75 per 1000 successful requestsDeepSeek launched a search-and-fetch API at $0.75 per 1,000 successful requests, which cuts the cost of grounding agents in web results. Hacker News · 2026-10-19
- The Most Important AI News This Week Is Getting OverlookedMicrosoft's MAI Base 2 and Meta's Muse Spark Open could change how businesses customize and pay for AI, alongside Anthropic's embedded safety evaluators and Cognition's performance-based token budgets. The AI Daily Brief · 2026-10-19 MSFT META
- DeepSeek Deploys What It Calls Faster Version of Its Open-Source ModelDeepSeek released V4.1-Flash-1017, an open model that is faster and gives shorter outputs. PYMNTS · 2026-10-19
- DLite: History-free Speculative Decoding for Agentic AIDLite is a speculative decoding method that speeds up inference for agentic workloads with long, changing contexts. HF Daily Papers · 2026-10-19
- DLite: History-free Speculative Decoding for Agentic AIDLite is a speculative-decoding method built for agent workloads with long, changing contexts, aimed at lowering inference cost. arXiv · 2026-10-19
- Hybrid Reasoning AgentsTogether released HybridAgent, whose agents reason in latent space to use fewer tokens. r/LocalLLaMA · 2026-10-19
- FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on HopperFuTransformer runs Llama-style models on Nvidia Hopper GPUs with FP8 precision for prefill and INT4 for decode, a way to speed up inference. arXiv · 2026-10-19 NVDA
- Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decodeSonicBoom is a compiler-first inference runtime with no Python that claims faster decode than llama.cpp. r/LocalLLaMA · 2026-10-19
- Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costsDeepSeek put out V4.1-Flash-1017 as an open-weight model that beats larger rivals on agentic tasks at lower cost per task, and opened Search/Fetch APIs at $0.75 per 1,000 calls. The Decoder · 2026-10-18
- Xolotl (Strata based) - DeepSeek-V4.1-Flash (552B) on a single 4090 at ~27 tok/s, Uncensored 1017 quant coming in a couple of daysA hobby inference engine runs the 552B DeepSeek V4.1 Flash on a single RTX 4090 at about 27 tok/s by streaming experts from RAM and NVMe. r/LocalLLaMA · 2026-10-18
- DeepSeek-v4.1-Flash-1017DeepSeek released V4.1 Flash, a new cheap open model that is already on OpenRouter, which pushes inference prices down further. Hacker News · 2026-10-18
- Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV StreamingAdaptive KV streaming in llama.cpp lets Qwen 27B use its full 262k context on a 16GB AMD card at 80+ tok/s. r/LocalLLaMA · 2026-10-18
- Trajectory-Level Speculation with Semantic Verification for Tool-Using AgentsThis method lets a small model propose whole sequences of tool calls that a strong model then checks, cutting agent costs. HF Daily Papers · 2026-10-18
- LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use AgentsLEASH cuts computer-use agent costs by letting a small model act by default and hand off to a frontier model only when it is uncertain. HF Daily Papers · 2026-10-18
- LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use AgentsLEASH lets a cheap model run a computer-use agent and hand off to a frontier model only when it is uncertain, cutting inference cost. arXiv · 2026-10-18
- CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull RequestsA pre-registered study of 465 agent-written pull requests finds that code health predicts how many tokens agentic code review costs. arXiv · 2026-10-18
- When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel PrefillA study of GLM-5.3 on B300 nodes finds a shared remote KV-cache tier can become a bottleneck in long-context prefill when it is slower than the GPUs. HF Daily Papers · 2026-10-18
- RoleQ: Role-Aligned Scales for Converting Language Models to TernaryRoleQ converts existing language models to ternary weights without training from scratch, cutting memory needs for inference. arXiv · 2026-10-18
- Replit is no longer working with Anthropic, CEO saysReplit has stopped working with Anthropic, citing cost among other reasons, a sign that coding platforms are sensitive to model pricing. TechCrunch AI · 2026-10-17
- [Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)Open-source ORL gets diffusion LLMs to 2,788 tok/s on a single RTX 5090. r/LocalLLaMA · 2026-10-17
- deepseek-ai/DeepSeek-V4.1-Flash-1017 · Hugging FaceDeepSeek released V4.1-Flash-1017 open weights refresh. r/LocalLLaMA · 2026-10-17
- Apple ML Research: Muse Spark Open with MLX on Mac StudioApple ML Research shows Muse Spark Open running locally with MLX on a Mac Studio. Hacker News · 2026-10-17 AAPL META
- CEO Amjad Masad says Replit is no longer working with Anthropic, citing cost and saying "ChatGPT is better now"; source: the partnership is still in place (TechCrunch)Replit CEO says it dropped Anthropic over cost, preferring ChatGPT, signaling model switching on price. Techmeme · 2026-10-17
- Qwen3.8-Flash-Next-GSQ-RCO (IQ3_S): ~40-50 tok/sec decode & 1300-1500 tok/sec prefill at 131k ctx on 12GB VRAM + 32GB RAM + NVMEQwen3.8 Flash Next runs at 40-50 tok/s at 131k context on 12GB VRAM. r/LocalLLaMA · 2026-10-17
- New paper from Stanford, Together AI, UIUC, and Meta introduces Hybrid Reasoning Agents. These AI models think mostly in a hidden internal way (faster and cheaper), and only use normal text when they need tools or give the final answer.Hybrid Reasoning Agents reason mostly in hidden internal form and score about 10% higher while using half the tokens and running twice as fast. r/LocalLLaMA · 2026-10-16
- ALoDLM 8b looped diffusion language model: ~3x inference speed vs its base model qwen3-8b with better benchmark resultsALoDLM is an 8B looped diffusion language model that runs about 3x faster than its Qwen3-8B base and scores better on benchmarks. r/LocalLLaMA · 2026-10-16
- What a £300, 35W Snapdragon mini PC can actually do 👀: 35B coder at ~30 tok/s, and an 80B model that doesnt even fit in RAM running at 17 tok/sA £300 Snapdragon mini PC runs a 35B mixture-of-experts coding model at about 30 tokens per second on CPU alone, showing how cheap local inference has become. r/LocalLLaMA · 2026-10-16
- capy-flash: Qwen3.8-Flash-Next (125B MoE) in full NVFP4 on two 16 GB consumer GPUscapy-flash runs the 125B Qwen3.8-Flash-Next model in NVFP4 on two 16GB consumer GPUs by offloading experts to RAM. r/LocalLLaMA · 2026-10-16
- Virtual Head AttentionVirtual Head Attention is a new attention design for long-context models that aims to cut KV cache size without the quality loss or extra compute of current alternatives. HF Daily Papers · 2026-10-16
- Prime Intellect acquires Moondream, a vision AI startup founded in Seattle by AWS vetsAI cloud company Prime Intellect bought vision-AI startup Moondream, mainly for its technology for running models faster. GeekWire · 2026-10-16
- Gemini 4 Argon is now available worldwide as Google's most capable frontier modelGemini 4 Argon is now available worldwide through the API at aggressive prices, but trails Claude Opus 5.5 and Meta's Muse Wonder Full on independent benchmarks. The Decoder · 2026-10-15 GOOGL
- A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML ConversionAn industrial legislation-to-XML pipeline found that a strong harness narrows the gap between models, so cheaper models can do legal-document work. arXiv · 2026-10-15
- Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUsDelta Tokenizer re-tokenizes only the new text in each agent turn instead of the whole conversation, removing a CPU bottleneck in LLM serving. arXiv · 2026-10-15
- JointServe: Serving Multi-Model Agents on Memory-Constrained GPUsJointServe serves agents that call several expert models on GPUs too small to hold all the models at once, aimed at private and edge deployments. arXiv · 2026-10-15
- New model from Telperion: Qwen3.8-27B-MLA & Qwen3.8-27B-GLA-g2. 4x smaller KV cache, smarter, and more efficient reasoning.Telperion's latent-attention variants of Qwen3.8-27B cut KV cache memory 4x. r/LocalLLaMA · 2026-10-15
- Your Transformer Is Secretly ElasticA method without retraining lets one pretrained transformer run at flexible sizes, so deployers can match model size to hardware and load instead of picking from fixed S/B/L options. arXiv · 2026-10-15
- 4x R9700 builds with $127 used EPYCs: $10.7k all-in per machine. I built three (384GB VRAM)A builder shows $10.7k boxes using 4x R9700 GPUs and cheap used EPYC CPUs, giving 384GB of VRAM across three machines. r/LocalLLaMA · 2026-10-15 AMD
- Tessera-27B-B-CJK, Zellige AIs new finetune of Qwen with novel approaches to how we think about token savingsTessera-27B is a Qwen-based coding agent that reasons in compressed CJK text to use fewer tokens. r/LocalLLaMA · 2026-10-15
- cuda, hip: GDN chunked kernel by am17an · Pull Request #29353 · ggml-org/llama.cppA new llama.cpp kernel speeds up prompt processing by up to about 20%. r/LocalLLaMA · 2026-10-15
- Why Amazon might pay billions for Decart, a 3-year-old AI startup that hasn’t disclosed its revenueAmazon is reportedly in talks to buy Decart for about $7B; Decart makes software that speeds up AI models across chips, including Amazon's Trainium. GeekWire · 2026-10-14 AMZN
- My local Qwen3.8FN agent is significantly cheaper than cloudA user's cost breakdown shows a local Qwen3.8 agent can beat most cloud models on price, depending on electricity rates. r/LocalLLaMA · 2026-10-14
- A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon DevicesA distributed virtual GPU cloud pools idle Apple Silicon devices for compute. Hacker News · 2026-10-14
- DoubtLess: Training-Free Contrastive Decoding for Efficient ReasoningDoubtLess shortens reasoning chains without any training, which cuts inference token costs. arXiv · 2026-10-14
- FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE ServingFastEEP lets mixture-of-experts serving scale capacity up and down within a running instance, which could cut serving costs under fluctuating demand. arXiv · 2026-10-14
- Microsoft plans to scale its own Maia AI chips to gigawatt levels by 2027Microsoft's Maia 300 chip is claimed to deliver 40% more tokens per megawatt than leading GPU systems and will be deployed at multi-gigawatt scale from 2027, reducing reliance on Nvidia. The Decoder · 2026-10-13 MSFT NVDA
- Janas: Run LLMs larger than RAM by streaming experts from SSDJanas runs mixture-of-experts LLMs larger than RAM by streaming experts from SSD, making local inference of big models cheaper. Hacker News · 2026-10-13
- AI agent teams waste massive tokens for barely measurable quality gains, research findsVals AI found that teams of agents cost up to 5.1x more than a single agent for barely measurable quality gains. The Decoder · 2026-10-13
- Microsoft's Maia 300 chip: Can it cut its reliance on Nvidia?Microsoft's Maia 300 chip, built by TSMC, claims 40% more tokens per megawatt and is meant to cut reliance on Nvidia. Yahoo Finance · 2026-10-13 MSFT NVDA TSM
- Ask Your Advisor: Learning When to Consult Stronger Models in Agentic SystemsCheap executor models learn when to escalate to stronger advisors, cutting agent inference cost. arXiv · 2026-10-13