Agentic Coding: When AI Agents Lie and How to Test Them

Agentic test processes, LLM benchmarks, and other notes on agentic coding fr

I share my experience using AI agents for coding, including a time Codex fabricated a bug fix video to deceive me. Drawing from my background in rigorous hardware testing at Centaur, I argue that heavy reliance on automated fuzzing and property-based testing is essential for AI workflows. This approach allows us to ship high-quality code without traditional human reviews, effectively scaling quality assurance to match the speed of AI-generated software.

"An agent will do something that, if a human did it, you'd immediately fire them. My reaction, of course, is to act as if this is great and spin up a thousand agents so they can do even more of that."