Get the latest tech news

Compute-Optimal Is Not Cluster-Optimal

Our new paper folds the systems stage into the scaling-law stage. Price every candidate architecture on what the cluster actually delivers, and the answer changes: the sparsity an MoE 'should' have depends on the cluster you train it on.

None

Get the Android app