AI migrated legacy COBOL programs to Java, bugs included
Migrating legacy COBOL to Java is notoriously tricky, especially without test data. Researchers propose the 'Locksmith Loop,' an agentic method that instruments both the COBOL source and generated Java with mocks, then iteratively searches for inputs to penetrate branches and applies parity-preserving mutations. When stuck, an analyzer identifies a 'Locked Paragraph' blocking deeper exploration. Across three programs (430–4,114 lines), it achieved near-complete branch coverage on open-source code and 91.90% on an internal production-like program, with all accepted test cases matching the COBOL reference under deterministic parity checks.
When routing boundaries are reached, an analyzer identifies a Locked Paragraph: a condition preventing deeper exploration.
- matsemann
Before, only the senior cobol programmers at the company understood and knew the codebase.
Now, no one does.
I can see the allure of moving away from a legacy cobol system. But an AI rewrite doesn't actually solve any of the issues with having a legacy cobol codebase. You just have a new system no one knows or understands.
- toplinesoftsys
The biggest problem is not that bugs are migrated with COBOL, but that lots of new bugs are going to be introduced. AI is not deterministic, it will be making tons of mistakes. The only realistic low-error approach is incremental step-by-step migration using Cursor or similar tools. However, it requires much more time as each step must be prompted, tested and committed manually. Any hope that one-shot migraton of a large code base will not introduce enormous number of bugs is very naive. LLM is very bad on handling long context - it is their nature unfortunately. There is no answer to this problem yet.
- pacaro
The largest test case was 4kloc.
There are hundreds of billions of lines of cobol in production.
The IRS alone has approx 160 cobol programs, averaging 230kloc each.
- mtct88
Why should I convert COBOL to Java?
LLM can write COBOL just as fine.
- kukkeliskuu
While not everything can be easily converted (IMS, CICS, reports, batch processing etc.), there are many situations where automated tooling can be helpful in migrations.
Related to this, I created a tool for situations where you want to compare COBOL code with Java code. It includes a preprocessing step where IMS etc. calls are converted to mocks that return JSON (from file), and also use JSON for input/output, and GnuCOBOL to run the program. More a proof-of-concept than production, but here is a link if somebody finds it helpful.