libargus - Zero-allocation native AI runtime for Java 22
Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)
libargus is a high-performance, model-agnostic inference engine designed for Java 22+ developers. It unifies LLM text generation, speech-to-text, and multimodal vision pipelines into a single, zero-allocation native runtime. Leveraging Project Panama's Foreign Function & Memory API, it eliminates JVM heap overhead and GC pauses by enabling direct, zero-copy memory access to GGML and llama.cpp backends. This solution provides a frictionless, thread-safe C API for orchestrating complex AI tasks like video analysis and real-time transcription with absolute hardware efficiency.
libargus provides a unified, thread-safe C API designed explicitly for frictionless, zero-copy compilation alongside modern unmanaged orchestration frameworks, featuring out-of-the-box structural alignment for the JDK 22+ Project Panama Foreign Function & Memory API.
- RandomBK
I'm curious to hear what bottlenecks you encountered in the traditional path. Of all the compute and data shuffling involved in LLM inference, I would have thought shuffling the raw input/output around would have been a trivial part of the overall cost, and thus not a big optimization target?
- hi_hi
What are the benefits of this. I get the impression it improves speed of…something? Most of the time when using AI comes from the LLM, so I’m curious what this improves?
- exabrial
This is pretty impressive.