model-architecture

Nvidia Breaks Down Mixture-of-Experts Model Efficiency

Nvidia published a technical explainer contrasting dense and mixture-of-experts architectures, showing how a 30-billion-parameter model can run using only 3 billion active parameters per token.