Get the latest tech news
Red Hat Announces The llm-d Open-Source Project For Gen AI
In addition to rolling out Red Hat Enterprise Linux 10, Red Hat used their annual developer summit today for introducing llm-d as their newest open-source project.
The llm-d project is supported by Red Hat along with NVIDIA, AMD, Intel, IBM Research, Google Cloud, CoreWeave, Hugging Face, and other vendors and AI organizations. Llm-d also employs LMCache for key-value cache offloading, AI-aware network routing, high performance communication APIs, and other features to help come up with a compelling solution for distributed Gen AI inference at scale. With llm-d, users can operationalize gen AI deployments with a modular, high-performance, end-to-end serving solution that leverages the latest distributed inference optimizations like KV-cache aware routing and disaggregated serving, co-designed and integrated with the Kubernetes operational tooling in Inference Gateway (IGW)."
Or read this on Phoronix
