LLMs Asked to Modernize Legacy PHP Break More Working Files Than They Fix
When Modernization Breaks Working Code: A Study of LLMs on Legacy PHP
Five LLMs were each asked to modernize 100 WordPress 4.0 PHP files to PHP 8.3 while preserving behavior, with no project context or execution feedback. Evaluated against 630 hidden behavioral checks, the results are sobering: 83 untouched files already pass everything, but the best model leaves only 59. Each model breaks 32-51 working files while repairing at most 10; the rule-based tool Rector breaks 7 and repairs 6. Most damage is avoidable, often from type declarations PHP itself rejects.
Only 37 findings (7 percent) require a change under PHP 8.3, yet in compatible files with no required change, the LLMs break 36-60 percent (Rector: 8 percent).