CrucibleBench: Using a MUD to Evaluate LLM Behavior for $99
Can a MUD evaluate LLMs? A $99 proof of concept
I built CrucibleBench, a proof-of-concept using a persistent MUD to test how language models handle trust and social objectives over 50 turns. By placing models in a constrained text world where mistakes leave traces, we identified specific failure modes like dialogue looping. Our $99 experiment revealed that LLM judges can drastically reorder rankings, proving that aggregate reliability statistics often hide critical measurement instability.
We did not choose a MUD because it is charming. We chose it because its constraints make behavior measurable.
- Varelion
A little irrelevant, but I need to vent about something MUD adjacent.
I really wish MUDs were still a thing. The text-based roleplaying communities were largely swallowed by Discord, where as before they existed scattered over proboards, jcink, and a few others.
Discord, being the imperfect text platform, limits the text and leaves most interaction limited by awkward formatting. To boot, even when a better ecosystem is invented, players refuse to give it a shot even if directly invited, because they got comfortable with Discord's terribleness.
It's so frustrating, and I don't know how to voice it, or even what can be done.
- dataviz1000
> the measurements of each LLM
I think you have the correct idea.
If you ask an LLM to solve a multiplication problem using reasoning without code tools, depending on the model, it will get 15 digit (15D) * 15 digit multiplication correct (123456789012345 x 998765432109876). It will take between 4000 and 8000 tokens if Sonnet. Eventually with enough digits, it will start only solving the problem 80% ... then 70% of the time until it has so many digits it will never solve. There will be a certain number of digits where it will not converge on a solution nor will it stop working thinking it can solve it. That is very, very expensive. It takes a lot of runs to determine the probability it will solve it.
What you can do now, this is likely the most important thing, is change the prompt and evaluate how many tokens and at what speed it takes to accomplish the task. Sure the measurements of each LLM are important! Nonetheless, if you can say to a company that you have a technique to tune prompts and evaluate them so instead of spending $1 X 100,000 times a day, they can instead spend $0.90 X 100,000 times a day, you will make a ton of money.
- rufasterisco
I am having a hell of a lot of fun letting agents play and understand a (still alive, human populated) MUD, which I have also played for the last 30 years.
It’s mostly my way to play with local llm inference (m5 64gb, gwen3.6 27).
It’s amazing. They build maps, classify events (building a grammar for a parser), run experiments (to verify the grammar).
They are now (given the correct tools/infrastructure) trying to fine-train a 3b model for fighting (where you need a decision for 5 seconds rounds). Basically autonomously!
Overall, a MUD does prove a great constrained sandbox for them to play in.
What started as an experiment to test local inference landed in a sweet spot for seeing models strength/weaknesses/tradeoffs. And it’s really fun.
Only problem is that Claude gets really jealous when I ask him to code their po harness running local inference. Weird world.
- krzyk
I was a bit afraid that MUD was an acronym for something else, I clicked in about 10% hope that this will be for multi user dungeons and behold, thank you. Interesting read.
That brought back memories when we were trying to find free terminal in few departments at our uni to get access to one that has internet connectivity.
- aphexairlines
It's interesting that you chose to measure how well each LLM did in talking to other NPCs, and having each NPC also use an LLM to react to each input. Why not have the LLM fight NPCs and loot items (both a significant part of the MUD experience, neither requiring LLMs on the server side), then measure character progression in experience and stats?