feat: add native LFM2 and PhiMoE bridge support#1420
Conversation
jlarson4
left a comment
jlarson4
left a comment
There was a problem hiding this comment.
Thanks for putting this together @randommm, these adapters are very well built. Just one small testing comment below. Other than that, you'll just need to get the CI checks to pass.
You're failing two tests right now which are effecting Full Coverage Test and all three compatibility tests:
FAILED tests/unit/model_bridge/test_bridge_generate_no_tokenizer.py::test_generate_without_tokenizer_stop_at_eos_requires_eos_id - Failed: DID NOT RAISE <class 'AssertionError'>
FAILED tests/unit/model_bridge/test_bridge_generate_no_tokenizer.py::test_generate_stream_without_tokenizer_stop_at_eos_requires_eos_id - Failed: DID NOT RAISE <class 'AssertionError'>
Both of these are related to your eos_token_id change, they will need to be adjusted to account for the presence of eos_token_id.
You'll also need to run make check-format and uv run mypy . & resolve any issues they surface to ensure you can pass the Format Check and Type Check.
If you run into any 429 errors caused by too many tests running at once, or any stderr output errors in our Notebook tests, just let me know and I can rerun those for you. Those errors are caused by too many CI tasks running at once.
And of course, if you have any other questions do not hesitate to reach out to me via the Mech Interp slack or by tagging me in a comment.
| # Some chat models use a turn-end token that differs from the | ||
| # tokenizer's primary EOS. Let adapters provide the full stop | ||
| # set via cfg.eos_token_id; otherwise fall back to the tokenizer. | ||
| eos_token_id = getattr(self.cfg, "eos_token_id", None) |
There was a problem hiding this comment.
Can we get a couple tests for this adjustment to the EOS token system? Since this effects every model's generate, we want to be extra careful when making changes.
Ideally we'd want to test stop logic for both int and list eos_token_id values.
TL;DR: add TransformerBridge support for LiquidAI LFM2 MoE and Microsoft PhiMoE, including registry wiring, config passthrough, unit coverage, native PhiMoE loading, and correct chat EOS stopping. Adds native architecture adapters for Lfm2MoeForCausalLM and PhiMoEForCausalLM. LFM2 is exposed conservatively as a residual-probing adapter over the HF decoder layers, avoiding fake attention/MLP internals for the hybrid conv/full-attention architecture. PhiMoE targets the native Transformers implementation with trust_remote_code=False rather than the archived remote implementation, which is incompatible with current generation/cache semantics. Registers both architectures across the TransformerBridge adapter factory, supported-architecture exports, model-registry architecture lists, report metadata, and supported_models.json. Adds explicit registry entries for LiquidAI/LFM2.5-8B-A1B and microsoft/Phi-mini-MoE-instruct. Extends HF config passthrough for hybrid/MoE and tokenizer-related attributes used by these adapters, including LFM2 layer/norm/MoE metadata, PhiMoE routing/bias metadata, and eos_token_id. TransformerBridge generation now prefers cfg.eos_token_id when present, allowing architecture adapters to provide multiple default stop tokens. Handles PhiMoE chat generation correctly by stopping on both the tokenizer EOS (<|endoftext|>) and the chat-template turn terminator (<|end|>). This prevents generation from continuing into another assistant turn after the model has ended its answer. Adds unit tests covering both adapters' config flags, component mappings, weight conversions, and PhiMoE EOS defaults.
2 participants
Description
add TransformerBridge support for LiquidAI LFM2 MoE and Microsoft PhiMoE, including registry wiring, config passthrough, unit coverage, native PhiMoE loading with correct chat EOS stopping (a small change on
transformer_lens/model_bridge/bridge.pywas needed for that).Type of change
Checklist: