Momentic's AI agents have caught 117,000 bugs by actually using your app
Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app
Momentic, a YC W24 startup, builds an AI QA engineer called Mo that tests apps end to end from plain-English prompts. Teams write flows in YAML, run them locally and in CI, and let the agent re-find controls when the UI shifts. So far it has logged 77,813 tests, 70 million runs, and 117,010 bugs caught before production, with customers including Notion, Quora, Webflow, and Retool.
Having E2E QA be part of the development process, run this fast (6.3x parallel), and at this scale (for every single pull request) is a true dream. Seriously, a green CI now has a measurable impact on my sleep quality.
- ice_cream_man_9
Congratulations on the launch! It looks interesting. I ran a check on one of the pages of my site. Here are a few points that caught my eye -hopefully, you’ll find them useful:
- It’s a bit confusing: the window below "Live Mo's screen" says "No bugs yet," while the tabs at the top (Agents / Bugs / Mo's screen) indicate "Bugs: 11."
Once the process finished, the window stated, "Mo didn't find any bugs," yet the tab at the top showed "Bugs: 12."
Perhaps this is because they are sitting in the "Needs review" list? Nevertheless - If they are awaiting review, why does it say there are no bugs?
- All the bugs found are labeled as "medium," even though they clearly vary in severity. It would be helpful to add a tooltip explaining how the program assigns these labels. Even better, it would be great to have the option to customize which bugs are flagged as, for example, "high priority" for my report.
- The "stop testing" bttn only appeared right at the very end of the process. Maybe I missed something, but if I changed my mind, wanted to refine the prompt, got tired of waiting, or simply didn't want to burn through my budget - there are plenty of reasons why I might want to stop - yet the button wasn't there for the duration of the test.
- It would be excellent to have an option to export the list of discovered bugs. Since I can manually select and copy the headings anyway, why not provide a proper export feature? It would allow me to share the results of an initial audit with the […]
- beavisringdin
Questions:
- I would always pick an open source tool I can strap to my custom harness over using a service like this. Do you somehow expect that no such competitor will arrive that performs as well?
- If you are YC 2024 what have you been doing for 2 years?
- The real edge here would be supporting mobile development builds on real devices, is this on your roadmap and if so what's your plan to support it? In that case I would pay for this if you handled the real device on your end.
- JulianPaul
Why is this better than running Opus 5.5 directly against my app with a custom Playwright-based harness for in my own subscription?
- daniiel
Not sure precision/recall is the number I'd look at first, because the thing I'd actually want to know is how often the tool says pass on a flow that is already broken, and a silent pass is worse than a flaky test, since the flaky one at least makes some noise.
I'm building something in the same space and this bit me quite early, so what I ended up doing was basically mutation testing: take a handful of open source apps, plant bugs on purpose, and count how many of them the tool waves through anyway. Seems that the pass/fail counts were not the interesting part at all, the interesting part was letting the engine answer with a third option, "I can't tell", because once that was allowed the false passes mostly disappeared, it simply stopped guessing. I would say a guessed pass is worse than having no test, and I keep every run replayable for the same reason, otherwise I wouldn't trust my own numbers either.
Curious how Mo (nice name) handles the middle ground - the docs say a bug it can't reproduce is not reported as confirmed, but does it show up anywhere or is it just dropped? And for QA Bench, did you plant bugs yourselves or only label existing ones?
- podoman
> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.
is this good? so Mo costs more money per bug? I'm confused.