Get the latest tech news
Attention Through Arithmetic Intensity
Reduce MHA to its arithmetic intensity and you get exactly 1. GQA, MQA and MLA come out just as clean -- head counts and nothing else. That last constant is what puts MLA at odds with MTP.
None
Or read this on Hacker News
