Cactus Hybrid - On-device models that know when they're wrong
Show HN: Cactus Hybrid: We taught Gemma 4 to know when it's wrong
Cactus Hybrid introduces a breakthrough in on-device AI by post-training models like Gemma 4 E2B to self-assess their accuracy. Every generated answer includes a structured confidence score, enabling intelligent routing to cloud models only when necessary. This approach balances speed and privacy with high reliability, matching larger models like Gemini 3.1 Flash-Lite while processing most queries locally. With seamless integration for Cactus, Transformers, llama.cpp, and MLX, developers can easily implement confidence-aware AI that dynamically decides when to hand off complex tasks.
The probe was trained on zero audio data, yet achieves high accuracy on audio benchmarks, proving it reads a modality-independent correctness signal from the hidden state rather than memorizing patterns.
- BugsJustFindMe
> "post-trained to know when it's wrong"
Is it also post-trained to know when it's wrong about when it's wrong?
> "Every response comes with a confidence score between 0 and 1"
How confident is it in its confidence?
Please, I'm sure that what you're doing is very neat and useful, but use other language to describe it. I beg you. You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct.
- maxgashkov
I don't really get why you need handoff if your score is accurate. If it is, and it is low for a given response, just let the harness re-run the prompt with a different seed until the score is high enough. If this approach doesn't work, your score is most likely garbage.
- dmrivers
Cool idea, some feedback:
1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction
2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result?
3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC?
- mncharity
> the hidden state for different layers carry meaningful self-awareness signal for various situations.
Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?
If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?
- astrobiased
Is this in any way similar to Goodfire's work? https://www.goodfire.ai/research/rlfr#