[REQUEST] Inference V2: add one model family on the existing kernels (dense Qwen3 proposed) #8762

Is your feature request related to a problem? Please describe.

deepspeed/inference/v2/model_implementations is a hand-written backend, not a generic Hugging Face runner. build_hf_engine only accepts opt, llama, mistral, mixtral, falcon, phi, phi3, qwen, qwen2, qwen2_moe, exaone4, and exaone4_5. Every other model_type raises Unsupported model type. The factory still says # TODO: generalize this to other models.

Each added model needs a parameter container, a forward on DSTransformerModelBase, and an InferenceV2Policy, then a branch in the factory. The kernels underneath stay the same: fp16/bf16, LayerNorm or RMSNorm, GELU/ReLU/SiLU or the gated forms, RoPE (rotate_half / rotate_every_other) or ALiBi, causal ragged attention, and an optional MoE GEMM.

Describe the solution you'd like

I would like to add one family, and only after you say the contribution is welcome.

Proposed family: dense Qwen3 (model_type=qwen3). Qwen, Qwen2, and Qwen2-MoE are already implemented, so the container and policy pattern is the one you already review. This would not include Qwen3-MoE or Qwen3-Next. Qwen3-Next is a hybrid attention model and does not fit these kernels.

Qwen3's decoder matches the Qwen2 block except for one step: RMSNorm on q and k, per head, after the QKV projection and before RoPE. The Qwen2 forward already has a boundary between self.qkv(...) and self.attn(...). The plan is to norm q and k there with the existing RMSNorm module, then call attention unchanged. No new CUDA op.

Scope if you agree:

• model_implementations/qwen_v3/: container, model, policy, following AddingAModel.md and the Qwen2 files
• one qwen3 branch in build_hf_engine
• a check that the container mapping accepts a Qwen3 state dict, and a small forward comparison against Hugging Face on a tiny random config

Out of scope unless you ask for it: new CUTLASS or ragged-attention kernels, MLA, sliding-window or hybrid attention, multimodal towers, and Qwen3-MoE.

Describe alternatives you've considered

• A family that is only a checkpoint remap of the Llama/Qwen2 block, with no forward change. That is smaller, and I will switch to it if you would rather not have the QK-norm step in the model forward.
• Leaving Inference V2 as it is. The older injection path under deepspeed/model_implementations is a different engine and does not cover this either.

Additional context

Questions for the inference owners (@hwchen2017 @tohtana):

1. Is a contribution that adds one Inference V2 family on the existing kernels welcome?
2. Is dense Qwen3 the family you want, or should this be a pure remap of a model you name?
3. Is applying the existing RMSNorm to q and k between qkv and attn acceptable, or does QK-norm need to stay inside the attention kernel?

Happy to adjust the family or drop the QK-norm approach if it does not match how you want this code to grow.