Presentation: Beyond the Prototype: Scaling Framework Agnostic AI Agent Infrastructure with Ray

Deepak Mohanakumar Chandramouli: Before we kick off, our focus today is basically building an agent and running it at scale with Ray. How many of you are familiar with Ray? Ray is an open-source distributed computing framework. It's mainly specializing in machine learning workloads, such as training, fine-tuning, running inference. Today what we are taking you through is also running agents on Ray at scale. That's the focus. This is the lineup for our session today. We are going to start with a quick refresher of an agent's building block. I think we've had a lot of sessions talking about prompts, context, memory, and so on. We'll take a quick refresher of that. Followed by that, we will walk through the developer lifecycle of building an agent and running it at scale, locally and in a remote environment. We'll do the same by running it with Ray. Finally, we want to call out a few things and share some of the learnings that we have from production. This may be familiar grounds. The agentic runtime generally is composed of a few key building blocks. First one is LLM. LLMs drive the execution lifecycle of a request within the agentic runtime.…