Mixture of Experts at Home: Running MoE Models Locally
-1
moe models are efficient enough to run locally now. guide to setup and real performance numbers
the main thing ive noticed is running llama 3 locally and its basically as good as the api for most tasks
6 replies
moe routing is the key thing people miss. active params during inference are way lower than total params, so a 47B moe model might only use 12B per forward pass. that's why it fits in less ram than you'd expect.