Ollaya runs decision models locally and answers in 10 ms
Ollaya – Ollama for open-source, Jev-style decision models
Ollaya is an open-source tool that runs decision models on your own hardware, answering typed questions about text or JSON in milliseconds. It's drop-in compatible with TypeSafe's API, so the official Python SDK works unchanged. Models like Laya answer five questions in about 10 ms on an RTX 4090, with no token-by-token generation. Available for macOS, Windows, Linux, and Docker under Apache-2.0.
Fast Decisions in milliseconds. A decision model answers in a single forward pass, with no token-by-token generation.
- pradn
I'm not sure what this means for AI startups if their innovations can be copied by OSS so quickly (what, like 2 weeks?). There's "consumer surplus" for everyone, to borrow an economic concept. But we do ideally want some of the surplus to flow to the innovator, too. I know there were precursors, but that's fine - it's hard to have a totally novel idea in such a popular field. I don't know what the end game is for TypeSafe - they'd need to demonstrate perpetually better results, or compete in another axis: UX, support, custom solutions, etc. So much of the time, someone proving a concept, or it simply getting enough publicity, is enough for a "Cambrian explosion" of follow-ups and copies. Famously, that was true for "Attention is All You Need", and the general idea of "next-token prediction" being so powerful.
We've stumbled into general differentiable models..
- george_max
Has anyone actually seen better or the same results with Laya compared to Jev? From my experience, Laya performs significantly worse. It's less confident and often makes wrong decisions with more complex queries.
- alex7o
Guys I have a real q, what is the difference between an instruct based re-ranker and laya/jev I just don't see it.
Edit: One is that jev/laya are tuned to have better probabilities, but a reranker can be fine tuned to do that as well. And jev/laya use RLCD?
- solaire_oa
I installed it, I tried the examples, it works.... But forgive my lack of imagination... what is this useful for?
Like, their example is of classification for a support interface.... `refund_requested`. Pretty convenient bool given the example is about a refund- what if 99% of submissions don't ask about a refund? Also, is that user not a `churn_risk`? What could possibly qualify as a churn risk if not a user asking for a refund?
https://ollaya.dev/library/laya The examples suffer the same problem of why I'd prefer to use a string column vs an enum. Changing an enum means you need to update the db, using a string you can do whatever.
I'm not trying to be negative, I genuinely want to know about some practical examples (that don't require tons of backwards maintenance).
- fooker
For everyone dismissing Jev's innovation as being trivial, no it's not.
It is definitely not the MNIST classifier you had trained in 2019.
The difference is that you only train it once and the modern LLM machinery sort of takes care of that with large contexts.
It's great that Jev proved this is a viable product. I'd expect a great many research innovations coming from making this work better/faster/cheaper, and around interfacing modern agents with it.
- ranyume
>Run decision models locally.
>example is a text classification task instead of a decision
- mococa
It would be really cool to have LLMs and System One in a single tool - in this case, if Ollama implemented it.
- zahlman
Ollama is for large language models, so this is for large language... yodels?