Vocab Trim: cut a small model's vocabulary to your traffic and measure the saving

I found plenty of open-source tools and research on vocabulary trimming, but no packaged product that does the full vocab-trim loop. The closest tools (vocabtrimmer, VocabSlim, TextPruner, LLMPruner) trim by language or retrain a tokenizer. None of them take your production logs, run your eval set before and after, and print cost per task and the accuracy change. NVIDIA's TensorRT-Edge-LLM ships a reduce-vocab feature, but it only covers the LM head, only runs on that NVIDIA stack, and makes you build the token mapping yourself. A new entrant could still win with the full workflow for current decoder models (Qwen, Llama, Gemma, Phi). That means profiling your traffic, keeping a safety margin of tokens your logs haven't used yet, exporting to vLLM, llama.cpp and GGUF, an optional short fine-tune to recover accuracy, and an honest report of the trade-off. The hub idea is weaker. The vocabtrimmer org already publishes per-language trimmed models, and anyone can copy domain-trimmed uploads for free, so it works better as marketing for the CLI than as a business. The bigger risk is whether customers will pay. Embedding and LM-head savings matter most below about 2B parameters and shrink quickly at 7 to 8B. Published papers report modest speedups and accuracy drops that need fine-tuning to fix. Teams can also get much of the speed benefit through constrained decoding or logit masking in their serving engine without changing the model. That makes this look more like a feature or open-source tool than a paid product, unless the measured savings for extraction and classification on 1 to 3B models turn out to be clearly large.

• lm-vocab-trimmer (vocabtrimmer)

  pip-installable CLI from the EMNLP 2023 vocabulary trimming paper. It removes tokens a multilingual model doesn't need for a target language, using mC4 token frequencies, and can run before or after fine-tuning. It is built around languages, not your own logs, and targets mT5, mBART and XLM-R rather than modern decoder-only SLMs.

• vocabtrimmer Hugging Face org (pre-trimmed models)

  Already publishes many pre-trimmed variants, such as mT5-small, XLM-RoBERTa and mBART trimmed per language, with reported compression rates. This is the trimmed-model-hub idea done for languages and older architectures. I found nothing for Qwen, Llama or domains like legal or e-commerce.

• VocabSlim

  Hugging Face package that trains a smaller BPE tokenizer and prunes the embedding weights to match. Its example uses Qwen2.5-0.5B, so it already reaches modern small decoder models. I found no evidence of profiling your own traffic or reporting cost and evals.

• TextPruner

  PyTorch toolkit for vocabulary pruning (dropping unused tokens from the tokenizer and embeddings) and transformer pruning. It supports encoder models like BERT and RoBERTa only.

• LLMPruner (yangjianxin1)

  Small open-source project, about 300 stars, that removes rarely used tokens from multilingual LLMs while keeping the language tokens you need.

• NVIDIA TensorRT-Edge-LLM vocabulary reduction

  Shipped feature with a reduce-vocab script that limits the LM head to a task-specific token subset. The user has to define the vocabulary mapping, and only the LM head gets faster. This is the closest big-company feature to the idea.

• COMPACT (IJCAI 2026)

  Research method with public code that prunes rare vocabulary tokens together with FFN channels, for models from 0.5B to 70B. It finds embeddings and the LM head matter most in small models, which supports the idea's premise.

• Efficient Vocabulary Reduction for Small Language Models (COLING 2025 Industry)

  Industry paper that measured GPU memory, speed and accuracy after cutting an SLM's vocabulary. Accuracy usually drops, but fine-tuning recovers most of it.

• VocabTrim (Qualcomm) / FR-Spec / NanoSpec

  Methods that prune the vocabulary of speculative-decoding draft models based on token frequency, with modest reported speedups of about 1.1 to 1.3x. Their existence shows that frequency-based LM-head pruning is well-known research.

Checked 2026-10-22