Edge0-AI · Streaming MoE Inference

edge0

A streaming mixture-of-experts inference framework for Apple Silicon on MLX, combining SSD expert offload for bounded memory, int4 distillation-trained Recover-LoRA adapters to claw back quality, and a prerouter predicting routing for up to 59% more decode throughput. Ships two 4-bit models, edge0-35b with 256 experts over 40 layers and edge0-8b with 128 over 24. On a Mac mini M4 Pro the 35B runs 14.9-17.7 tokens a second at 2.9GiB peak memory, the 8B 23.9-25.3 at 1.0GiB, at a stated average quality loss against fp16 of 3.9 and 2.8 points. Apache-2.0.