Get the latest tech news
Serving AI from the Basement – 192GB of VRAM Setup
Dedicated LLM server powered by 8x RTX 3090 Graphic Cards, boasting a total of 192GB of VRAM.
AI from The Basement: My latest side project, a dedicated LLM server powered by 8x RTX 3090 Graphic Cards, boasting a total of 192GB of VRAM. Now that I kinda have everything in order, I’m working on a series of blog posts that will cover the entire journey, from building this behemoth to avoiding costly pitfalls. The challenges of assembling this system: from drilling holes in metal frames and adding 30amp 240volt breakers, to bending CPU socket pins (don’t try this at home, kids!).
Or read this on Hacker News