Get the latest tech news
How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server
seek Ai Rig Build for Local Inference Let’s start with the good news. I got very solid performance off the same baseline AMD EPYC Rome system that has been at the core of our entire journey 😁 That initial parts selection has remained fantastic! Owners of that system are going to get some great news today also as they can hit between 4.25 to 3.5 TPS (tokens per second) on the Q4 671b full model.
I got very solid performance off the same baseline AMD EPYC Rome system that has been at the core of our entire journey 😁 That initial parts selection has remained fantastic! The MZ32-AR0 was also a very good board recommendation to start with as well as it lowers the price of hitting 512GB to 1TB of System Ram dramatically with 16 dimm slots that can run at a full 3200 speed. Basically you should install this on a bare metal Ubuntu 24.04 server base if you want to eliminate extra things and are setting this up new and fresh, or follow the prior proxmox guide.
Or read this on Hacker News