Infinite-Parameter LLMs Write Live Data Straight into Their Weights
Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Mixture-of-Experts models activate only a fraction of a huge static parameter bank, but they cannot learn from live interaction because their weights are frozen. This paper proposes the Infinite-Parameter LLM: a compact hypernetwork turns run-time data into a low-rank modulation of a shared base network, so feed-forward weights are generated from live data rather than stored. A Bayesian belief over the generator's latent code is updated online, keeping the stored footprint fixed while the compilable weights become effectively infinite.
The stored footprint stays fixed, yet the weights the model can compile are effectively infinite.
- lubujackson
Think about this in context of the Navier-Stokes math discovery controversy.
Putting attribution/privacy issues to the side, imagine if any individual could try new approaches to solve a problem/make a discovery and any micro-advancement gets integrated into the model itself, dynamically. This could transform progress from the slow "write a paper, get peer reviewed and published, use published data to inform future work" to a system with a centralized repository of concepts, attempts and results, including failed approaches already tried. How much work do humans waste replicating failed approaches?
Someone completely random halfway around the world could trigger a prompt that solves a blocker that prevents my solution from working. Who cares about AGI or "can models invent anything" when we could have a system that automatically synthesizes individual human thought into a rich network of aggregate human memory.
That's the target OpenAI/Anthropic should be evangelizing, not an AI Daddy Overlord or agentic script kiddie hellscape.
- wood_spirit
Continuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?
- juancn
I wonder how (and if) continuous learning models will achieve stability.
They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run
- alightsoul
So would it be a 42 trillion parameter model, because that's how many tokens there are in the training data?
- alightsoul
I see a new version of the web, web 4.0, being exactly this. Web 4.0 is the source of the live data. Web 4.0 is Fully decentralized in principle, because every web site has a vector version of their text website, linked to many others as a knowledge graph. The knowledge graph's links are like hyperlinks.
The vector databases might be free or paid or have ads. There is no difference between web 4.0 and an ai model that has all of it as its live data.
It is up to every client to turn that data into ai responses so in theory anyone would instantly have frontier scale ai which sounds as crazy as everyone having a computer, and the internet and the www in the 1980s.
It also, by coincidence, fully fulfils the goals of web 3.0 aka the semantic web.
Moderation of content and ranking content like a search engine would be left to the client. Or an instance like a fediverse mastodon instance or atproto with bluesky. There will be a few large models like web browsers or operating systems or social networks or fediverse instances today that will compile web 4.0 into a result in an ai agent like say Hermes and perform content moderation and ranking. You could in theory use a decensored model to perform crime like you can with the web today with Tor like selling drugs or piracy, with search engines that are not Google or defederated mastodon instances
Humans can't read raw html easily and can't read vector databases easily either. It would be read by an ai model connected to an ai agent, li […]