DeepSeek open-sources its serving stack, so inference margins are no longer secret
DeepSeek's open, vLLM-based serving engine lets any GPU operator run cheap inference. Closed-API labs and neoclouds lose margin.
Competitive · DeepInfer open-sourced, margin math published
Written for: Investor or analyst
Next step
Thesis: open serving stacks lower the price floor for tokens and squeeze closed-API gross margins; confirming signal: vLLM merges DeepInfer kernels and a US lab cuts list prices within a quarter; already building: DeepSeek, the vLLM project, Nvidia with Nemotron open models.
Why now
This updates the watchlist item on cheap flash models. Before, the evidence was cheap models. Now the serving software and a public estimate of its unit economics are open too. Efficient serving was the hidden part of an API lab's margin, and it is now public weeks before Anthropic's November IPO and alongside a reported $80B+ first-half net loss at Anthropic. The US federal DeepSeek ban splits buyers. Regulated buyers cannot use the model, but the serving techniques carry over to vLLM and US open weights. The split comes back as a porting job, not a barrier.
What became possible
DeepSeek open-sourced DeepInfer, the vLLM-based engine behind V4.1-Flash, which it says cuts compute and energy costs. Latent Space estimates it runs 155T tokens a day per cluster and earns about $22M a day at roughly 80% margin, which pays back the cluster in 81 days. Other speed-ups landed the same week. Eagle-4 speculative decoding reports 3.59x over vanilla decoding on Qwen3.8-27B, and Specter speeds up MoE expert offloading.
Three ways in
- Token Cost Marks
Independent, repeatable benchmarks of tokens per GPU-hour and cost per million tokens by serving stack, model and chip. Updated monthly.
- Customer
- Credit funds lending to neoclouds and enterprise procurement teams negotiating API contracts
- Model
- Data subscription, $50-150K a year per seat group
- Clean-Origin Serving
Ports the DeepInfer techniques into vLLM builds for US open weights such as Nemotron. Ships with documentation showing no DeepSeek code or weights are in the deployment.
- Customer
- Federal contractors and banks covered by the DeepSeek removal order
- Model
- Enterprise support licence plus a porting fee
- Sovereign DeepInfer hosting
Managed DeepInfer deployments for national and EU clouds that want frontier-cheap inference without US API dependence.
- Customer
- European and Gulf sovereign cloud operators
- Model
- Managed service, per-cluster fee
Competition
Sovereign DeepInfer hosting is the weakest of the three. Scaleway, OVHcloud and Mistral already offer EU-hosted open-weight inference, and several European governments restrict DeepSeek. Gulf operators may be more receptive, especially given the Huawei Ascend support, but I did not verify any Gulf competitors.
Full competition notes (9 competitors)
I confirmed both triggers. DeepSeek released DeepInfer, a vLLM-based engine, on October 20, 2026, and DHS made the DeepSeek removal order public on October 19, 2026. But the release is narrower than the opportunity assumes. It runs only DeepSeek-V4.1-Flash in offline mode, and it leaves out the caching, routing and disaggregation layers where much of the serving margin comes from. So "inference margins are no longer secret" is overstated. Token Cost Marks has the most room, but it is not empty. SemiAnalysis InferenceX already publishes open cost-per-million-token benchmarks by chip and framework. Artificial Analysis benchmarks providers and sells custom hardware benchmarking to enterprises. A newcomer could still win by serving buyers who need numbers they can rely on in contracts: credit funds underwriting neocloud loans need numbers that can be audited and reproduced, with realistic utilization and real negotiated GPU-hour prices rather than list prices or vendor-sponsored runs. That means selling a lender-grade data product, not a public leaderboard. Clean-Origin Serving has real demand from the DHS order and the DoD NDAA rule. However, NVIDIA NIM already ships hardened, FedRAMP-ready vLLM containers for Nemotron. The order also covers products built on DeepSeek implementations, so advertising "DeepInfer techniques" works against the clean-origin pitch. A viable version would be provenance attestation and audit services (SBOMs and code-lineage reports) for banks and contractors on whatever vLLM build they run, rather than a DeepInfer port. Sovereign DeepInfer hosting is the weakest of the three. Scaleway, OVHcloud and Mistral already offer EU-hosted open-weight inference, and several European governments restrict DeepSeek. Gulf operators may be more receptive, especially given the Huawei Ascend support, but I did not verify any Gulf competitors.
- DeepSeek DeepInfer (the trigger itself)
Confirmed real. Released October 20, 2026 under Apache-2.0 and built mostly on vLLM. The first release supports only DeepSeek-V4.1-Flash in offline mode, on Nvidia and Huawei Ascend. Reports say the cache, routing and disaggregation layers were held back, so it does not fully reveal how DeepSeek gets its margins.
- SemiAnalysis InferenceX (formerly InferenceMAX)
Open, regularly updated benchmarks of throughput and cost per million tokens by chip, model and serving framework, with a published formula based on TCO per GPU-hour. Directly overlaps Token Cost Marks, and NVIDIA already cites it heavily in marketing.
- Artificial Analysis
Benchmarks API providers on speed and blended price per million tokens. Its enterprise tier covers custom benchmarking of models, inference and hardware, and it is hiring an AI hardware PM to compare price-performance across accelerators.
- MLCommons MLPerf Inference
The established industry throughput benchmark. It is vendor-submitted rather than independent and does not report costs, but procurement teams still use it as a reference.
- NVIDIA NIM / NVIDIA AI Enterprise
Ships hardened vLLM containers for Nemotron and customer-supplied weights. Includes CVE patching, open-source review compliance, model signature verification and FedRAMP-ready branches for government. This is the default US-origin answer the Clean-Origin idea would compete with.
- DHS Exclusion and Removal Order 2026-01
Confirmed real. Made public October 19, 2026. Removes DeepSeek products, including third-party products built on DeepSeek implementations, from civilian federal IT and procurement. Separately, Section 1532 of the FY2026 NDAA covers DoD contractors. The order creates the demand behind Clean-Origin, but its third-party clause makes porting DeepSeek techniques legally risky.
- Scaleway Managed Inference
EU-hosted managed inference for open-weight models behind an OpenAI-compatible API, with dedicated GPU deployments. Competes with the sovereign hosting idea.
- Mistral regional endpoints and Mistral Compute
Generally available EU inference endpoints, plus a European compute buildout and coalition aimed at sovereign capacity. This is the leading European sovereign inference option.
- OVHcloud AI Endpoints
EU-hosted catalog of open-weight models with zero data retention beyond billing data. Another sovereign competitor for EU cloud buyers.
Checked 2026-10-21
Risks
- The $22M/day and 80% margin figures are Latent Space's estimates, not audited disclosures
- DeepInfer is tuned to DeepSeek's architecture and may not carry over cleanly to other models
- National-security rules may expand from DeepSeek products to DeepSeek-derived code
- Western labs may hold prices because of brand and enterprise contracts, not cost
Strong tech readiness and team fit; open space is the weak point (5).
tech 8 · pull 7 · space 5 · fit 8 · build 7
Exposed
Companies that could benefit or be threatened. News and context, not investment advice.
Momentum of its main topic
Sources
- Deepseek open-sources the inference engine that powers its fastest and cheapest AI modelThe Decoder
- DeepInfer: High-performance inference framework for DeepSeek modelsHacker News
- [AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day paybackLatent Space
- Eagle-4 Released: Faster than DFlash2 & DSparkr/LocalLLaMA
- Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative OffloadingHF Daily Papers
- US orders federal agencies to remove China’s DeepSeek AI from their systemsTechCrunch AI
- Anthropic logged net loss of over $80B in first half of 2026Hacker News
Status history
- 2026-10-21 Flagged as buildable