Planning to join us in NYC next week? 🏙️
Registration for the llm-d Distributed Inference Meetup closes this Tuesday, March 10th.
Don't miss out on a night of technical talks and networking with the community at the IBM 1 Madison office. Grab your spot now!
🎟️ luma.com/0crwqwg4
#llmd #NYCMeetup
What’s on the agenda for next Wednesday's NYC meetup?
🛠️ Intro to llm-d 0.5
⚡️ Distributed LLM serving on AMD
🧠 Lessons scaling Wide-EP and MoE
💾 KV-cache offloading & prefix scheduling
Join the engineers building the future of open-source inference.
Details: luma.com/0crwqwg4
Join us next week in NYC with the llm-d community for a deep dive into distributed inference.
We’re talking llm-d 0.5, scaling MoE models, and KV-cache offloading.
If you're building LLM infra, don't miss this.
📅 March 11th
📍1 Madison Ave
Register: luma.com/0crwqwg4
In the latest llm-d release, we’re tackling high hardware costs with the new GPU Recommendation Tool! 📈
Evaluate throughput, latency, and cost-effectiveness before requesting expensive cluster resources.
Check out the full demo: www.youtube.com/watch?v=Y26i...
There are many more sessions and community meetups happening throughout the year.
Check the full calendar for session details, room numbers, and the complete list of talks from the llm-d community:
🔗 llm-d.ai/docs/communi...
📍 Stop 3: PyTorch Conference Europe
📅 April 7–8 | Paris
Deep technical tracks on chunked decoding, preemptive scheduling, and disaggregated tokenization. We'll be sharing the latest on state-aware serving with vLLM + llm-d.
Full Schedule: events.linuxfoundation.org/pytorch-conf...
📍 Stop 2: KubeCon Europe
📅 March 23–26 | Amsterdam
From Istio Day to the main stage, we’re talking AI-aware routing and KV-cache scheduling. Don't miss our tutorial on building resilient LLM gateways with Kubernetes.
Details: events.linuxfoundation.org/kubecon-clou...
📍 Stop 1: NYC Distributed Inference Meetup
📅 March 11 | IBM Innovation Studio
We’re diving into the weeds of llm-d 0.5, Wide-EP, and MoE model scaling. Perfect for anyone in the city looking to optimize LLM serving on AMD and beyond.
Register: luma.com/0crwqwg4
Where to find the llm-d community over the next 2 months 🧵
We have a busy Spring ahead with sessions in NYC, Amsterdam, and Paris. If you're building open-source infrastructure for distributed inference, come join the conversation. ⬇️
The agenda is still evolving, and we’ve got even more awesomeness in the works! 📈
Whether you're running GenAI in production or building the platforms to support it, this is the room to be in.
📅 March 11 | 4:30 PM
📍 1 Madison Ave, NYC
🎟️ RSVP: luma.com/0crwqwg4
Hosted by Red Hat AI, IBM Research, and AMD. 🤝
If you're building or scaling models, this event is for you.
We’re bringing together maintainers and engineers working on:
🔹 llm-d project roadmap
🔹 Optimizing for AMD hardware
🔹 Scaling MoE (Mixture-of-Experts)
🔹 KV-Cache & Prefix-caching performance
NYC: Ready to go deep on Distributed Inference? 🗽
The llm-d community is hitting Manhattan on March 11th!
Join us at the IBM Innovation Studio for a technical deep dive into the infra powering the next generation of LLM serving. 🧵
We'd like to announce that @kubernetes.io WG Serving has succeeded and will be disbanded! Thank you everyone who have participated and contributed to the discussions and initiatives!
More details: groups.google.com/a/kubernetes...
In case you missed it, last week the llm-d community shipped the v0.5 release.
Check out the post from the llm-d project owners to learn more about all the features we've included in this release.
llm-d.ai/blog/llm-d-v...
👉 Check out the February newsletter here: https://inferenceops.substack.com/p/state-of-the-model-serving-communities-f81
👉 Subscribe to get future issues in your inbox: https://inferenceops.substack.com/
🚀 Thanks to everyone who subscribed so far!
Kudos to all contributors to this edition!
Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from vLLM, KServe, @llm-d.ai, @kubernetes.io, and Llama Stack.
This release is built on collaboration—from NIXL 0.9 merges to vLLM integrations.
We are building an open, hardware-agnostic inference control plane.
Ready to build? 🧱
GitHub: github.com/llm-d/llm-d
Website: llm-d.ai
Community Calls: Wed 12:30pm ET
🌐 In disaggregated serving, network congestion kills tail latency.
We’ve integrated the UCCL backend into NIXL, demonstrating 2.4x greater resilience to network contention than standard transports.
🎯 Multi-tenant workloads often suffer from the "thundering herd" problem.
v0.5 introduces LoRA-precise prefix caching, routing requests based on specific cache locality to maximize efficiency.
⚖️ Standard deployments collapse when memory is saturated. Our new Hierarchical KV Offloading creates a "performance floor" using a three-tier hierarchy (GPU, CPU, Disk).
We sustained ~185k tok/s during high concurrency—a 13.9x improvement.
Realized production performance shouldn't be a mystery.
We've adopted the "Research Paper Principle": every claim in v0.5 is backed by a reproducible, version-controlled configuration you can validate with one command. ⚙️
🏗️ llm-d v0.5: Sustaining Performance at Scale In our last release, we focused on breaking latency records.
With v0.5, we’re shifting from peak performance to the operational rigor required to sustain those gains in production.
🧵👇
llm-d.ai/blog/llm-d-v...
Check out this recent talk from IBM Distinguished Engineer and llm-d project lead Carlos Costa at NVIDIA Dynamo Day. Breaking down our approach to open-source inference, moving beyond theory to provide verified blueprints for scaling LLMs in production.
red.ht/4kjqGty
We put the stack to the test on NVIDIA A100s:
🔹 KServe + llm-d: Optimizing distribution across replicas.
🔹 vLLM Production Stack: Testing cache-aware routing and MIG partitions for mixed workloads.
Open source for the most demanding AI use cases
youtu.be/OinY1Oooke0
The demo shows how llm-d serves as the core layer in a modular architecture, integrating with key upstream projects:
🚀 vLLM: High-perf engine
⚖️ llm-d: Multi-replica load balancing
📦 KServe: Orchestration
🌐 Envoy Gateway: Intelligent request routing
📊 Grafana: Deep o11y
Scaling prod LLM inference shouldn't require a proprietary ecosystem.
We’re demonstrating how a fully open-source stack handles high-performance, distributed workloads on Kubernetes.
Huge shout-out to Sean Condon from @RedHat for this deep dive! 📺 youtu.be/OinY1Oooke0
A huge shoutout to the contributors in SIG-benchmarking for making performance transparency a core pillar of the llm-d project!
🚀 Check out the full demo here: youtu.be/TNYXjZpLCN4
#AI #Kubernetes #Benchmarking
⚫ 100% Reproducibility: We aim for a world where if you see a benchmark in an llm-d blog post, you can run the exact same template on your cluster and see the same results. Transparency is key to scaling AI.
Why does this matter for the community?
⚫ Verified, Not Just Documented: Every community-tested guide is now backed by standardized benchmarking templates.
If the guide says it performs, we provide the tools to prove it.
This new contribution allows anyone to benchmark a pre-existing or pre-installed stack. It is specifically designed for stacks deployed via official llm-d guides to ensure your setup matches our verified community baselines.
