Amazon opens PyTorch on Trainium, and CUDA's moat is attacked at the framework layer

The portability layer and the agentic front-end are both up against free tooling from the hyperscalers and Nvidia, so the opening is narrow. On portability, Google funds the vLLM TPU backend and AWS ships Autoport, which already does GPU-parity checks for Trainium. A startup would be building on platforms whose owners want this layer to be free. The weakest spot is Maia: its SDK is still in preview and I found no evidence that open models are served on it for customers. What may be left is a single cross-cloud parity and certification harness that tells an enterprise whether a given open model runs identically, and how much cheaper, on TPU, Trainium and Maia. That is closer to an independent validation and benchmarking service than a compiler. On the agentic front-end, prefix-cache-aware routing is now standard in Dynamo, llm-d, Ray Serve and SGLang/vLLM routers, and Shepherd Model Gateway already offloads tokenization. GPU-side tokenization exists only in research (BlockBPE), so a productized version tuned for agent traces is a real but thin feature gap. It would probably get folded into the open-source stacks rather than support a standalone company. On the share tracker, SemiAnalysis already sells shipment models by SKU to institutions, and Epoch covers chip sales publicly. What I could not find is a tracker of workload share, meaning what fraction of each cloud's inference tokens are actually served on in-house chips versus Nvidia, as opposed to chip units. Building one from pricing, capacity and API-latency fingerprinting could set an analyst product apart. Even so, the buyers are few and the incumbents could add it.

• vLLM TPU (tpu-inference plugin)

  Official vLLM backend that runs PyTorch and JAX model definitions on Google TPU through a single lowering path, keeping the standard vLLM interface. Free and open source, so it already covers TPU portability for open-weight models.

• AWS Neuron Autoport for vLLM Neuron

  AWS's agentic workflow that generates vLLM Neuron implementations of new model architectures for Trainium and checks their accuracy against the GPU reference. This is very close to the 'parity tests against GPU outputs' idea, and AWS ships it for free.

• Microsoft Maia SDK

  Preview SDK with a Triton compiler, PyTorch support, a simulator and a cost calculator. Microsoft invites open-source model contributors to optimize their models on it. I found no public evidence that Maia serves open models to Foundry customers yet.

• NVIDIA Dynamo KV router

  KV-cache-aware router in which workers report cache blocks and requests go to the best overlap, with a tunable trade-off between time to first token and inter-token latency. Free from Nvidia.

• llm-d (Red Hat and others)

  Precise, event-driven prefix-cache-aware scheduling built into the Kubernetes Gateway API Inference Extension. Red Hat explicitly pitches it for agentic and RAG workloads with large shared contexts.

• Ray Serve PrefixCacheAffinityRouter

  Alpha prefix-affinity router for vLLM with automatic prefix caching.

• Shepherd Model Gateway (LightSeek)

  Rust gateway that takes tokenization and detokenization off SGLang and vLLM, where they had become bottlenecks, and also does routing. It overlaps directly with the 'serving front-end that handles tokenization' idea.

• BlockBPE GPU tokenizer (research)

  Research paper on a GPU-resident BPE tokenizer for long-context serving. It shows that GPU tokenization is being worked on but is not yet a product.

• SemiAnalysis Accelerator Industry Model

  Paid institutional model tracking production, shipments, ASPs and revenue by SKU for TPU, Trainium and Inferentia, MTIA, Maia and others. This is the main incumbent for the ASIC share tracker idea.

• Epoch AI chip sales data

  Public dataset estimating AI chip sales, including Trainium. A free alternative for estimating installed base.

Checked 2026-10-12