Automating AI Away: Turning Clumsy LLMs into Reliable Deterministic Workflows
I develop Beagle SCM using Anthropic's Fable, a brilliant but often clumsy LLM that struggles with deterministic tasks like parsing. Instead of fighting its non-deterministic nature, I embed it within fast, formal tools and workflows. By letting the AI automate its own repetitive failures into reliable scripts, we effectively automate the AI away, replacing inconsistency with simple, deterministic processes.
Essentially, we let the LLM automate itself away, in favor of simple reliable deterministic tools.
- bob1029
I think semi-automation with contextual and domain-specific tooling is the key to the best quality outcomes.
For example, with browser automation, giving the LLM raw access to the literal DOM generally results in disaster for tasks that need to be stable across more than 5-10 interactions. The better approach is to write an intermediate layer that understands each view and can provide a list of tools that are precisely tailored for each case. E.g.:
- <raw dom - hundreds of kb>
- Available Tools: <arbitrary javascript>
vs
- We detected that this is the application's login page.
- It has the following visible elements:
+ Username
+ Password
+ Login Button
- Available Tools:
+ PerformLogin
+ Quit
The later case takes a lot more effort, but it also reduces a Turing complete problem space into a binary decision at this particular step.
- contextfree
A dumber but related habit I've gotten into is that if I want to use AI to do some sort of refactoring on a C# codebase, instead of asking it to edit the code directly I ask it to write a code transformation using the Roslyn compiler API, then run that on the code. The result is less likely to have subtle bugs if it appears to work and gets through a light code review on the transformation (i.e., attempts to cheat with weird special-casing are more likely to stand out amongst the Roslyn API code, and if there isn't such weird special-casing but the code is wrong, the result is more likely to be completely broken rather than subtly broken)
- lubujackson
Makes sense, I have had the biggest wins with AI by attacking nondeterminism whenever possible.
BTW, you should probably fix the Beagle link on your homepage: https://replicated.live/beagle/
- derdi
This is a very interesting introduction to a blog post, but... I'm somehow missing the actual blog post. How does this stuff work in practice? What are some concrete examples? How does one get from JavaScript tokenizing things in a commit hook to validating that the LLM didn't disable tests it didn't agree with, or any other helpful property?
- Animats
This makes sense, although it's not well described here.
Formal methods, as in proof of correctness, have been around for decades (I was doing that stuff in the 1980s) but pushing the proofs through was too laborious. The seL4 verification effort reportedly used over a decade of people time.
The idea is that if you have a formal specification of what you want to happen, you can get a LLM to do the struggling with the proof system to get it right. It's a good task for an LLM, because there's feedback from the prover.
I'd like to see more non-trivial examples of this. People keep republishing verifications of greatest common divisor or stack algorithms, which was done decades ago.