aiqbench takeAI Products

This release marks a significant step in edge AI by demonstrating that specialized tool-calling tasks can be executed on constrained hardware like Raspberry Pis and smartphones without GPUs or NPUs. The model's efficiency, achieved through quantization to CQ2-bit and a custom Simple Attention Network architecture, enables fully offline sessions within 28MB of RAM for specific device control and data extraction workflows.

Source coveringMarkTechPost

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Meet Needle 2: An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary and Runs a Full Session in 28MB of RAM

MarkTechPostAugust 14, 20264 min read

Cactus Compute released Needle 2, an open 45M-parameter model for tool calling, device use, and structured extraction. The full model is a single 14MB binary that runs a session in about 28MB of RAM. It leads both Seal-Tools splits while targeting hardware with no GPU and no NPU. The post Meet Needle 2: An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary and Runs a Full Session in 28MB of RAM appeared first on MarkTechPost.

Cactus Compute has released Needle 2, an open 45M-parameter model for tool calling, device use, and structured extraction. The entire model ships as a single 14MB binary that runs a full session in about 28MB of RAM. Weights are trained and deployed at CQ2-bit using Cactus Quants, and the model is sealed inside the company’s own C++ engine, so there is no runtime to install and no download at inference time. Reported decode throughput is 500 tokens/sec on a Raspberry Pi…

Excerpt shown under the publisher's RSS terms · 81 words · full text and figures are on the publisher's site

Read at MarkTechPost ↗Ask about this article