Skip to content

fix: bypass Switch expert overflow through the outer residual - #319

Open
Afloat16 wants to merge 1 commit into
labmlai:masterfrom
Afloat16:fix/switch-overflow-residual
Open

Afloat16 wants to merge 1 commit into
labmlai:masterfrom
Afloat16:fix/switch-overflow-residual

Conversation

@Afloat16

@Afloat16 Afloat16 commented Oct 2, 2026

Copy link
Copy Markdown

When an expert exceeds capacity, SwitchFeedForward currently writes the dropped token's input into its FFN output. SwitchTransformerLayer then adds this branch to the original token through its residual connection. Dropped tokens therefore receive an extra normalized-input update (also weighted by the router when is_scale_prob=True), instead of bypassing the expert computation.

Leave capacity-overflow positions at zero in the FFN branch. The outer residual already preserves their representations. Count dropped token indices explicitly without changing pre-capacity routing counts, router probabilities, retained expert outputs or auxiliary load-balancing gradients.

This follows Switch Transformers, Section 2.2, where overflow bypasses expert computation through the residual. The official Mesh TensorFlow implementation also zeros the combine gate when capacity is exceeded.

Validation: python -m pytest tests/test_switch.py -q gives 10 passed with native package imports and PyTorch 2.14.1 CPU. The original implementation gives 8 failed, 2 passed. Tests compare actual transformer-layer outputs and input/router/expert gradients to an explicit residual-plus-retained-expert reference across zero and partial capacity, both probability-scaling modes, and multiple overflowing experts. Disabling dropping or giving sufficient capacity remains equivalent, and auxiliary router gradients remain active. git diff --check passes.

The standalone FFN's overflow outputs change from input copies to zeros, matching its role as a residual branch. GPU/distributed execution, model pretraining and the full repository suite were not run. Existing documentation escape-sequence warnings remain unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant