117 updated first flagged14 Oct 2026

Nvidia ships its own 24GB models, and the local agent stack takes shape

Sparse models that run agents on phones and 24GB cards. Device makers and privacy-bound enterprises pay.

Crowded · Nemotron Bolt, LFM2.5, MiniCPM-V 4.7

Written for: Investor or analyst

Next step

Thesis: sparse models with few active parameters move agent workloads onto devices, and chip vendors that ship tuned models will set the default runtime. Confirm: a phone OEM or bank app ships one of these models by default, or independent tests show 2-bit Nemotron Bolt holding quality. Already building: NVIDIA (Nemotron Bolt, RTX Spark), Liquid AI (LFM2.5), OpenBMB (MiniCPM-V), and Google with GovTech Singapore.

Why now

The watchlist asked whether chip vendors would own the local runtime. This week NVIDIA shipped both the silicon (RTX Spark, with 128GB of unified memory) and models sized to its consumer memory tiers, with a 4GB variant due October 23. Google and GovTech Singapore already run Gemma on-device to catch scams with no cloud data leaving the phone, a reference deployment in a regulated setting. Telperion's 4x KV-cache cut adds headroom.

What became possible

Sparse mixture-of-experts models store many parameters but run only a few per token. NVIDIA's Nemotron Bolt 3.5 packs 74B parameters (9B active) into 24GB using 2-bit quantization-aware training. LFM2.5-24B-A2B runs 2B active parameters. MiniCPM-V 4.7, a 35B vision model, runs on a phone in about 3GB using Lookahead Expert Scheduling.

Three ways in

Expert-scheduling runtime for OEMs
A licensed runtime that pages experts from flash ahead of use, so large sparse models fit low-memory devices.

Customer
Android OEMs and PC makers
Model
Per-device royalty
Private on-device agent kit for regulated apps
A packaged local model plus a policy layer for bank and government apps that cannot send chats to the cloud.

Customer
Banks and public-sector app teams
Model
Annual licence per app
Local model fleet manager
Deploys, updates and audits local models across enterprise laptops and RTX Spark boxes.

Customer
Enterprise IT
Model
Per-endpoint SaaS

Competition

Crowded

The core idea of one SDK across AMD, Apple, Qualcomm and Nvidia with tool calling is already taken. NexaSDK does almost exactly this.

Full competition notes (9 competitors)

The core idea of one SDK across AMD, Apple, Qualcomm and Nvidia with tool calling is already taken. NexaSDK does almost exactly this. RunAnywhere and Cactus do similar things. Microsoft's Foundry Local and Windows ML handle the multi-vendor layer for free on Windows, and llama.cpp, Ollama, ExecuTorch and MLC LLM give the basic runtime away. Selling hardware neutrality alone would mean competing with free tools and the operating system. What none of these products appear to offer is a compliance layer for regulated desktop software: audit logs of agent actions and tool calls, encrypted agent memory with retention and redaction policies, tool-call permissions tied to HIPAA or attorney-client privilege rules, signed and pinned model versions that can be validated, central fleet management, and the vendor security paperwork and HIPAA business associate agreements that hospital and law-firm buyers expect. A new entrant could win by building on top of llama.cpp or Windows ML rather than competing with them, and selling governed private agents to legal and healthcare software vendors. It would also need to move quickly, before Nexa or RunAnywhere build enterprise compliance tiers.

Nexa AI NexaSDK
The closest match. Its README describes an on-device runtime for CUDA, Metal and Vulkan plus Qualcomm, Intel and AMD NPUs. It has an OpenAI-compatible server with JSON-schema function calling and agent demos, and supports GGUF, MLX and its own .nexa format. Its Apple NPU claim and the scope of its AMD NPU support (image generation only in its changelog) were not confirmed in primary docs.
RunAnywhere SDK
A startup SDK for running AI locally across platforms. It is built on llama.cpp and ONNX and adds hybrid cloud routing, over-the-air model distribution and fleet management. The licence is source-available with a commercial-use cap.
Cactus Compute
An on-device LLM framework with grammar-constrained generation and function calling for agent apps, plus a cloud fallback. It started mobile-first, uses a proprietary model format, and has a source-available licence with a commercial-use cap.
Microsoft Foundry Local / Windows ML
The operating-system layer on Windows. It detects the machine's silicon and downloads the matching execution providers for Nvidia CUDA, AMD, Qualcomm and Intel. It is free, only works on Windows, and needs models in ONNX format.
NobodyWho
Another startup making an on-device inference SDK, aimed at app developers.
llama.cpp
The free open-source engine underneath many of these products. It supports CUDA, Metal, Vulkan, ROCm and other backends, so the basic runtime layer is effectively free.
Ollama
A popular free local LLM runtime with tool-calling support on Mac, Windows and Linux. Many desktop apps already build on it.
ExecuTorch (Meta)
Meta's open-source on-device runtime, with backends for Apple Core ML, Qualcomm, Arm and others.
MLC LLM
An open-source compiler and runtime that targets Metal, Vulkan, CUDA, ROCm and WebGPU from a single codebase.

Checked 2026-10-14

Risks

Quality claims at 2-bit are unverified by independent tests.
Apple and Qualcomm may keep runtimes closed and in-house.
Concern about self-spreading small models could bring device-level restrictions.

Strong team fit; open space is the weak point (5).

tech 7 · pull 6 · space 5 · fit 8 · build 7

Exposed

NVDA QCOM AAPL AMDARM

Companies that could benefit or be threatened. News and context, not investment advice.

Momentum of its main topic

Small and efficient models AI

42 developments this week +3 vs last week

Stage: Coverage

Sources

Status history

2026-10-14 Added to the watchlist
2026-10-15 Promoted from the watchlist: Today's evidence made it buildable
2026-10-15 Flagged as buildable

Open in Earlyink