Add Phi-3.5-MoE

[Blaizzy]

Still WIP

[Blaizzy]

@awni ready ✅

[Blaizzy]

It got 2x faster generation all the sudden 🔥

[awni]

Yes, the old SuRope was really slow so we made our fast RoPE more flexible and implement SuRoPE using it., which really speeds things up for the Phi models that use it.

[awni]

Thanks for the addition!

[Blaizzy]

My pleasure!

I'm always happy to help :)

[nickludlam]

I'm using an M1 Ultra, and the speed discrepancy between 8-bit and 4-bit is enormous.

4bit

Prompt: 10 tokens, 13.931 tokens-per-sec
Generation: 100 tokens, 59.524 tokens-per-sec

8bit

Prompt: 10 tokens, 4.455 tokens-per-sec
Generation: 100 tokens, 0.476 tokens-per-sec

Is this an expected outcome? It seems disproportionally large!

[Blaizzy]

That's looks odd.

I will investigate tomorrow 👌🏾

[Blaizzy]

Can you share the link to the 8bit model you used?

[awni]

This looks like an issue with swapping / too much memory use. Once that happens the generation time plummets.

If you're machine has enough RAM for the 8-bit model (likely 64GB is the minimum, maybe 48 but it's a stretch with 8bit) then it could be related to memory wiring issues. You could check out this related issue #776.

The most consistent solution has been to upgrade to Sequoia (macOS 15.0) and set sudo sysctl iogpu.disable_wired_collector=1.

For older OS sometimes setting iogpu.wired_lwm_mb and iogpu.wired_limit_mb to some large value is helpful.

[nickludlam]

Can you share the link to the 8bit model you used?

I downloaded https://huggingface.co/mlx-community/Phi-3.5-MoE-instruct-8bit with
huggingface-cli download mlx-community/Phi-3.5-MoE-instruct-8bit

I'm using a 128GB M1 system so it's not a swap issue.