CI in the Era of AI: From Unit Tests to Stochastic Evals — Nathan Sobo, Zed
Summary
This talk focuses on the empirical testing approach used by Zed, an AI-enabled code editor built from scratch in Rust. The speaker highlights their rigorous, deterministic testing methods, even for complex concurrent scenarios, and discusses the new challenges and the need to embrace stochastic behavior introduced by integrating LLM-powered features like Agentic Editing. The practical takeaway is the necessity of adapting testing strategies to accommodate the inherent non-determinism of AI models to build reliable products.