# Expert substitution (MoNE) [Mixture of Novices and Experts (MoNE)](https://arxiv.org/abs/2507.00390) prunes MoE layers by replacing redundant experts with "novices" that return a constant vector. vLLM loads the retained experts into compact weights and adds the novices' output separately, so pruned experts use no MLP weight memory and no MoE backend needs to know about them. ## Checkpoint format The checkpoint keeps the original architecture. `config.json` lists the substituted experts of each decoder layer: ```json { "architectures": ["DeepseekV2ForCausalLM"], "n_routed_experts": 64, "approximate_experts": { "1": [1, 6, 8, 12], "2": [0, 3, 17, 40] } } ``` For VLMs, `approximate_experts` may live on the text config. Retained experts keep their logical checkpoint names. Each substituted expert stores one `[hidden_size]` tensor instead of its MLP: ```text model.layers.1.mlp.experts.0.gate_proj.weight model.layers.1.mlp.experts.1.approx_value ``` Alternatively, `compression_config.transform_config.expert_substitution` declares the same information with explicitly named constant tensors: ```json { "compression_config": { "transform_config": { "expert_substitution": { "version": 1, "router_semantics": { "preserve_logical_expert_ids": true, "preserve_router_weights": true, "renormalize_after_substitution": false }, "targets": { "model.layers.1.mlp.experts": { "num_logical_experts": 64, "weight_layout": "compact_retained_experts", "replacements": { "1": { "format": "constant-v1", "tensors": {"value": "model.layers.1.mlp.expert_replacements.1.value"} } } } } } } } } ``` Only the values shown for `version`, `router_semantics`, `weight_layout`, and `format` are supported. Each target path must contain exactly one layer index. This form is not yet accepted alongside compressed-tensors quantization, whose `transform_config` schema does not include `expert_substitution`. ## Semantics For a substituted expert `j` with constant `v_j` and router weight `w_j(x)`, the expert contributes `w_j(x) v_j`. Routing is unchanged: router weights are neither renormalized nor redistributed to retained experts. ## Execution `FusedMoEFactory` matches each MoE layer to `approximate_experts` by the layer index in its prefix and, for a match, builds `SubstitutedRoutedExperts`, which: 1. allocates weights only for retained experts, in ascending logical order; 2. after routing, sums `w_j(x) v_j` over substituted routes in FP32; 3. rewrites substituted routes as zero-weight routes to physical expert 0 and retained routes as compact physical IDs, preserving the `[num_tokens, top_k]` contract of every decomposed backend; 4. adds the constant output on one tensor-parallel rank, before the partial outputs are all-reduced. Zero-weight routes may schedule some unnecessary GEMM work; the memory savings are unaffected. Monolithic backends route internally and are not selected. ## Weight loading Retained experts are loaded through the standard expert mapping by logical ID and placed in their compact row. Substituted IDs map to no local row, the same way expert parallelism skips non-local experts. Constant tensors are consumed at the model's `load_weights` boundary, so initial loading and direct weight updates share one path. `approx_value` tensors of layers that the model does not build (other pipeline stages, MTP layers) are passed through to the model's loader, which skips them like any other weight it does not own. Explicitly named constants are always consumed, and dropped when their layer is built elsewhere. Loading fails if a local substituted expert has no constant. Direct `load_weights` updates may change individual constants. Layerwise reloads (`reload_weights` in checkpoint format) are rejected, and the model keeps its previous weights. ## Supported configurations - Models whose MoE layers use `FusedMoEFactory` with the default `RoutedExperts`, and whose MoE prefix contains exactly one layer index. - Unquantized experts. - Tensor and pipeline parallelism, including MTP draft models. - Any decomposed MoE backend. Expert and data parallelism, EPLB, MoE LoRA, fused shared experts, quantized experts, and routed input/output transforms are rejected at initialization. Deferred MoE finalize and layerwise weight reloads are rejected when used.