Welcome Inkling by Thinking Machines

Inkling is a large (1T params!) open model designed natively to accept image, text, and audio inputs.

TLDR; Inkling by Thinking Machines is now available on Hugging Face. Inkling is a massive multimodal LLM that understands all modalities (image, audio, text), possesses agentic capabilities, and supports 1M context window. It comes in full BF16 format as well as a well-calibrated NVFP4 variant, and includes speculative MTP layers for faster inference. There's day-0 support available in transformers, SGLang, and llama.cpp.

What makes Inkling special?

Inkling is the first large open model with approximately 1T parameters and a 1M context window to natively receive image, text, and audio inputs; it was trained on 45 trillion tokens of text, images, audio, and video. It focuses on reasoning across modalities such as audio, images, and text, and is intended for domain adaptation via fine-tuning. We've tinkered with this model to build some demos and explore the architecture, and we think it's great for building a new wave of multimodal reasoning apps.

Inkling is a decoder-only multimodal Mixture-of-Experts model with 975B total parameters and 41B active parameters. There are many components at play, so let's break each part down:

Decoder-only: This means that the architecture supports causal autoregressive generation, similar to most state-of-the-art LLMs.

Multimodal: The model can ingest text, audio, and images.

Mixture of Experts (MoE): The feed forward networks inside each layer are sparse, achieving faster inference because only 41B parameters are active at any given time. The model has 256 experts, as we'll see later.

Here's a quick overview of the architecture.

Relative attention: Instead of RoPE, which is the usual method to inject positional information in transformer models, Inkling uses relative attention to encode position information. Each attention layer learns position directly from the attention logits. Aside from key-query-values, there's a fourth projection producing a per-token, per-head relative feature R. This projection tensor is then adjusted with distance information (the distance between the key and the query vector) and propagated into the attention module.

Hybrid attention: The decoder layers alternate between global attention (attending to the full context length at once) and sliding window attention (attending to a fixed context window in a sliding fashion). The architecture follows a pattern of 5:1 for sliding window to global attention layers. This hybrid attention scheme provides efficiency in computation. The final layer uses global attention to help build feature-rich representations.

Short convolution: The model uses a distinctive short 1D convolution, or SConv, over the hidden states. SConv reads the current token and the previous W-1 hidden states, with W being the sliding window size. The intuition here is that SConv helps with local attention while freeing the attention and MoE modules from handling local representations.

MoE with shared experts sink: In Inkling, the router scores both routed experts and shared experts. Top-k selection is performed over 6 experts, plus 2 shared experts that are always active.

Vision understanding: The model includes a simple hierarchical MLP patchifier consisting of several linear layers. Each layer merges pixels progressively until the final layer produces one embedding per patch.

Audio understanding: The architecture employs a discretized mel spectrogram, where each audio chunk (of 100 ms) is converted to the mel scale and then classified into the exact mel spectrogram bin.

The multimodal towers are relatively simple modules, unlike other models that employ separate encoders for each modality. Each image patch passes through the image embedding tower and the audio chunk is passed through the audio embedding tower to obtain both media embeddings. Image inputs also include an additional temporal dimension for video processing. We expect this capability to be useful for downstream fine-tuning, but we haven't evaluated out-of-the-box video performance yet. The tower folds the patch grid; a small local block of neighboring tokens is stacked into the channel dimension and goes through hMLP. The audio waveform is converted to mel scale, which is then classified into discrete mel bins. These mel bin values are embedded in the audio embedding tower, and the embeddings are summed to construct the final audio input.

Inkling comes with day-0 transformers support and is supported by major inference engines like SGLang and vLLM.

This model is huge. The bf16 checkpoint requires 2 TB of VRAM, while the nvfp4 version requires 600 GB of VRAM. You can try the model through serverless inference routers like Inference Providers, or use ggml quants for local deployment with llama.cpp.

The easiest way to infer using transformers directly is to use the any-to-any pipeline. You can either use the 16-bit "thinkingmachines/Inkling" on Hopper GPUs and later models, or the quantized NVFP4 checkpoint "thinkingmachines/Inkling-NVFP4" on Blackwell Nvidia GPUs. Make sure you have the latest version of transformers (5.14.0 was released today) by running `pip install -U transformers`.

After initializing the pipeline, you can pass in the prompt as follows: [Code snippet omitted for brevity].

Going one level lower, you can use Auto classes. For inference, you can use the AutoModelForMultimodalLM class for models and the AutoProcessor class for processors. For different reasoning tasks, the tokenizer takes a `reasoning_effort` argument. Existing options for reasoning effort are "none", "minimal", "low", "medium", "high", "xhigh", and "max".

For multimodal inference, you can use the same classes. We provide example snippets for each different modality in the model card.

Inkling also takes audio input. Below is an example inference snippet, which still uses the same AutoModelForMultimodalLM class: [Code snippet omitted for brevity].

For more realistic parallel deployment across a cluster of several nodes, please refer to the Slurm section below.

SGLang is one of the fastest deployment frameworks for Inkling at the time of release, as it includes a custom model implementation. The launch command below shards the model across 8 GPUs and serves an OpenAI-compatible API on port 30000: [Command snippet omitted]. Match `--tp-size` to your GPU count. Add `--mem-fraction-static` (e.g., 0.85) if you need to leave more headroom for the KV cache.

vLLM is strong for production serving. A single vllm serve command downloads the weights from the Hub, shards the model across your GPUs with tensor parallelism, and starts an OpenAI-compatible server on port 8000: [Command snippet omitted]. In practice, you will need multiple nodes and a distribution tool like SLURM (see below). Key parameters are `--tensor-parallel-size` set to the number of GPUs on your node, and use `--max-model-len` to cap the context window if you hit KV-cache memory limits.

You can infer using this model via several inference providers through Hugging Face. You can see all code snippets for consumption here: [Link omitted]. Below is how to use it with the OpenAI client: [Code snippet omitted]. Using the ":auto" suffix routes to your preferred provider in your settings; you can also use "cheapest" or ":fastest". For this release, we cover inference costs for 2 hours within the release period for everyone.

Note: audio support in Inference Providers is work in progress and will be added shortly.

Local Inference with llama.cpp and Unsloth

You can use llama.cpp to run quantized versions of the model on limited hardware. Unsloth has quantized the model down to 1-bit precision, reducing VRAM consumption by 95% over the original model: [Command snippet omitted]. This starts an OpenAI-compatible server running at http://localhost:8080/v1 that you can connect to in your preferred tool or clients. Heading there, you can start chatting with the model and set it up with your favorite MCPs, pass images or files conveniently, and more!

Llama.cpp also ships with a built-in UI that supports tools, mcp, and agentic workloads. Check out Inkling running at: [Link omitted].