Presentation: From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

Ben O'Mahony: First thing I want to bring up is this quote. We shape our tools, and thereafter our tools shape us. John Culkin paraphrased Marshall McLuhan with this quote decades ago, but I think it's never been more relevant. When you build an AI tool, you're creating something that will reshape your workflow, your habits, your thinking. The question isn't just, does it work, but actually, does this make you a better person? What kind of person does this tool create? Look at how we work as developers. I use Neovim, so I've spent hours shaping my editor exactly how I want it, hundreds of lines of Lua, lots of plugins, color schemes. Every key binding, I want to shape my tools. Continuing this theme, I built an AI-powered Language Server Protocol. I wanted that to give me semantic code analysis, inline hints and intelligent suggestions. I crafted the harness, the agent, the instructions, the interactions, but I couldn't do much with the AI API. I could change models if I needed better model quality, but that was it. I couldn't customize the actual brain of this tool and actually improve it over time in ways that were a…