Neutrino-1 8B: A Single Model for Datacenter GPUs and Laptops
We introduce Neutrino-1 8B, a transformer model using a proprietary ternary format to shrink its footprint to just 3.88 GB. This design allows the same artifact to run efficiently on everything from H100 datacenter GPUs to 16 GB MacBook laptops without conversion. By decoding weights directly inside matrix kernels, we achieve high throughput and enable speculative decoding on consumer hardware while maintaining bit-exact performance across all platforms.
The bits of fp16 72.1 MMLU 763 tok/s Spec decode, H100
- kamranjon
There's a really interesting trend of labs using "proprietary" methods to convert existing models to a compressed ternary format.
PrismML actually targeted the same Qwen 8b model and got it down to 1.75gb here: https://prismml.com/news/ternary-bonsai
I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: https://arxiv.org/abs/2402.17764
From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."
The section from huggingface is here: https://huggingface.co/blog/1_58_llm_extreme_quantization#fi...
I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.
- wincondition
hey guys, founder here. I am working completely solo right now running on grant funding, so things slipped through on the website quality, it is largely written by AI, I'll deslopify the blogs and be much clearer about methodology. Neutrino-1 is a starting point for Fermion, It's a density experiment on how much capability survives per byte. rn working on really fast and efficient TTS and realtime models to which this architecture is generalizing pretty well.
Sorry for all the confusion from reading through the blogs, should be fixed in a couple of days!
- moinism
The content on that page is too AI-generated to make sense to me; I don't understand what the model is for.
- secult
There is not a single person mentioned on the website, github created 3 days ago, no real contact, everything hidden. Completely anonymous. Domain owner hidden.
- pavel_lishin
Damn, I was hoping this was about an actual neutrino.