Frontier MoE models stream experts from NVMe, adding pressure on HBM

The hardware side is already crowded. Intel, Qualcomm, HyperExcel, Chinese vendors like Xiwang, and the SanDisk/SK hynix HBF consortium are all building LPDDR- or flash-based inference parts, so a new hardware entrant would be late and capital-starved. Tiered MoE serving is competitive but still unfinished. KTransformers, llama.cpp and a pending vLLM RFC already cover GPU plus CPU-DRAM expert placement, and the academic predict-and-prefetch techniques are public. What I did not find is a production engine that treats flash (NVMe, and later HBF) as a real expert tier, or one that targets the new LPDDR accelerators such as Crescent Island and AI200 instead of Nvidia GPUs with host RAM. A startup could win by being the serving layer for those non-Nvidia, cheap-memory boxes, with per-model expert-heat profiles and a flash tier. The risk is that vLLM merges expert caching upstream and the chip vendors ship their own stacks. The benchmark tracker has the clearest gap. InferenceX publishes cost per million tokens only for HBM GPUs. LPDDR vendors publish unverified claims like 90% cheaper. I found no independent, same-workload comparison of HBM vs LPDDR vs unified memory vs flash. A neutral tracker could become the reference for buyers and analysts in 2026–27 as these parts begin sampling. The catch is that it is hard to monetize on its own, and SemiAnalysis could add these platforms whenever it chooses.

• KTransformers (Tsinghua MADSys / kvcache-ai)

  An open-source CPU/GPU hybrid MoE inference framework (SOSP '25). It keeps attention and the KV cache on the GPU and the experts in system RAM, and it has run DeepSeek-R1 671B on a single 24 GB GPU. This is the closest existing version of the tiered MoE serving idea, though it uses DRAM rather than flash.

• vLLM CPU offload and GPU expert-cache RFCs

  Upstream proposals to keep a hot subset of experts on the GPU and stream or offload the rest to CPU memory. A related RFC (#38256) adds an LFRU-evicted expert cache with cross-layer prediction. If these merge, the most popular serving engine gets the feature for free.

• llama.cpp selective tensor offload

  The --override-tensor option lets users pin MoE experts to CPU RAM by hand. This is common in hobbyist and small-enterprise setups.

• FineMoE / HybriMoE / TriMoE research line

  A line of academic systems for predicting, prefetching and placing hot, warm and cold experts across GPU, CPU and near-data-processing tiers. TriMoE is at https://arxiv.org/pdf/2603.01058. The techniques are published and easy to copy.

• Intel Crescent Island

  An Xe3P inference card that uses LPDDR5X instead of HBM. Board partners can fit up to 480 GB, at roughly 1.5 TB/s, and it is air-cooled. Customer sampling was planned for 2H 2026.

• Qualcomm AI200 / AI250

  Rack-scale inference cards, each with 768 GB of LPDDR, explicitly pitched as a lower-cost-per-token alternative to HBM. Humain is the first customer and availability is targeted for 2026.

• HyperExcel (HyperAccel)

  A Korean startup building an LPDDR5X-based LPU for LLM inference, co-developed with Naver Cloud, with a data-center chip due in 2027.

• Xiwang Qwang S3

  A Chinese LPDDR6-based inference chip. The company claims, without independent verification, a 90% lower cost per token than HBM-based hardware.

• SanDisk / SK hynix High Bandwidth Flash

  A flash-based HBM alternative published as an OCP spec: up to 512 GB per stack at 0.4 to 3 TB/s, aimed at read-heavy inference. Memory samples are due 2H 2026 and devices in early 2027.

• SemiAnalysis InferenceX (formerly InferenceMAX)

  The leading open benchmark of TCO per million tokens. Its published lineup (H100, B200, GB200/GB300, MI355X) is all HBM GPUs. It is the obvious incumbent if it decides to add LPDDR or flash systems.

Checked 2026-10-13