Ephemeral Testing: A New Way to Validate Software with AI

Ephemeral Testing: A New Way to Validate Software with AI

Daniel Lemire proposes ephemeral testing: instead of directly testing a core library, you have an AI agent build a throwaway application on top of it and test that. A clean API and stable invariants let the agent succeed quickly; hidden state or poor docs produce failures. Those failures reveal the quality of your foundation, not the agent. Lemire has used this trick on his own projects.

The failures are evidence about your code, not about the agent.
  1. simonw

    I've been using this pattern quite a bit recently for API design, and I really like it.

    The big challenge with designing an API is that the only way to be confident in the design is to build a bunch of different things on top of it. But why invest all that effort in an API that you don't think is ready yet?

    With coding agents the cost of building those prototypes drops to almost nothing. I can exercise a proposed API design five different ways before I commit to the shape.

  2. bunderbunder

    I had good results with a similar technique this summer.

    I was designing a DSL to help make a colleague’s work easier, and I tested it by having a coding agent re-implement some of their notebooks using a draft of the DSL. The LLM output was a decent enough approximation of “typical” use, and it uncovered some warts I didn’t catch by testing it myself. As its designer, I simply wouldn’t have thought to try using it in some of the ways the LLM-generated code did.

  3. bakies

    Not unthinkable at all. Been doing this for several years as an opt-in on pr tags usually but now with agents writing code I have it on by default. I've got several skills files about accessing with a read only account and gitops done through the PR. It's an excellent dev environment for the agents.

More from this day

2026-10-05