Why I Moved My 35KB Prompts Off Claude and Onto a Local LLM
Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama
After watching frontier providers get caught training on user sessions, I tried migrating my 35KB preprompts from Anthropic and OpenAI to a self-hosted Ollama setup on a 128GB AMD Ryzen AI MAX+ 395. The prompts that worked cleanly on frontier APIs immediately thrashed locally: repeated tool calls, re-read files, and context saturation within minutes. The culprit wasn't the smaller model—it was the 65K context window. A 35KB prompt eats 14% of it, so the agent is essentially briefing a man reincarnated every ninety seconds. I share the single-objective prompting fixes that made it work.
With limited context window, the pre-prompt is basically briefing a man who is reincarnated every ninety seconds. It performs your last instructions without any awareness of the 15 preceding demands.
- DiabloD3
The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context.
At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.
If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.
The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
- cube00
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
- andai
> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.
- fghorow
I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills.
I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do?
(I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
- robotswantdata
Why are you using Ollama?
Just use llama.cpp