Iterating on LLM apps at scale Learnings from Discord: Ian Webster
Discover best practices for rapid evaluation and iteration on LLM apps in large-scale applications, with a first-hand account from Discord's engineering team. This talk covers development workflow and evaluation methodology in order to measure model & prompt improvements, mitigate risks, and speed up development. We'll discuss the best practices that we refined and implemented internally, the tooling and automation that got us shipping improvements consistently, and some of the strange and wonderful things that happen with LLMs in the wild
Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/20... & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025
About Ian
Ian Webster is a Senior Staff Engineer at Discord and the maintainer of Promptfoo, a popular LLM evaluation tool. At Discord he leads teams that successfully scaled an AI-based products to millions of users while navigating the many new challenges presented by LLMs.
