Needle 2 - 14MB Agentic LLM for Tiny Devices
Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
Needle 2 is a 45M-parameter language model compressed to a 14MB binary, designed for on-device tool calling and structured extraction on phones, wearables, smart home devices, and robots. It runs in just 28MB of RAM, achieving 500+ tokens/sec on a Raspberry Pi 5 and fitting on microcontrollers like ESP32-S3. Built on Simple Attention Network and Cactus Quants, it delivers performance comparable to models 5-70x larger while using 7-85x less compute per token. Apache 2.0 licensed, with weights on Hugging Face and a GitHub repo for easy deployment.
The model's footprint is tiny and the performance never lets us down.
- nater5000
This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
- dbeardsl
My first query:
> Make it a little warmer in here.
The reply:
> "name": "set_thermostat",
> "arguments": {
> "temperature": 65,
> "mode": "cool",
> ...
> "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.",
Maybe I'm doing it wrong?
- Tiberium
Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:
Query: HN
Result:
{
"function_calls": [
{
"name": "lock_door",
"arguments": {
"door": "front door"
}
}
],
"reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.",
"confidence": 0
}
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
- arthuqa
That's really cool - I was already thinking of compressing `functiongemma-270m-it` down to 1-2 bits so it would work flawlessly in the browser.
Your `Fine-tuning` feature is even much more convenient.
- profsummergig
Could someone please share how such open source micro-LLMs might have been created?
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
- prmoustache
How many languages does it supports in such a small size?
- hathym
I tested with
import needle
@needle.tool
def add(a: int, b: int):
"Add two numbers."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("calculate 1 + 1?")["reasoning"])
python main.py
No calculator or math tool available.
conclusion: completly useless
- redrix
This is cool!
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.