The production platform for open-weight AI inference

Open-weight models now match closed models on quality, run at a fraction of the cost, and are fully customizable for your task. That combination has made them the foundation for teams building serious AI products and agents. But the strategic motivation for adopting open-weight models is still the ability to exercise control. Unlike their closed alternatives, open-weight models give teams control over performance, quality, and functionality. Many commercial AI applications have adopted a majority open-weight endpoint strategy so they can maintain complete control over their user experience. Many enterprises use this control to incorporate their valuable proprietary IP into models without risking exposure to third parties.

While open-weight model users want maximum control over performance, functionality, and quality, few teams want to test their ability to correctly align a matrix of decisions across quantization levels, parallelism schemes, engine parameters, and draft model architectures—to name just a few—on a weekly basis. Meanwhile, frontier techniques for achieving the best quality and performance are advancing just as quickly. In a rapidly evolving AI landscape, no one has the time or desire to stop and read an almanac’s worth of trivia about inference internals or a mountain of research papers—not even the teams running the world’s most successful models and agents.

This update brings together advances from our research, model optimization, and platform engineering teams in a single service designed to give companies an easier, more reliable, and more optimal way to run open-weight, licensed closed-weight, and fine-tuned models in production.

Delivering that combination of control and simplicity requires treating inference as more than hosting a model behind an API. The precision you run, the hardware you deploy on, the serving configuration you use, and the way you scale and route traffic all have a meaningful impact on performance, reliability, and cost.

We built this platform so that moving from experimentation to production does not require changing platforms or rebuilding your deployment. Production readiness is built into the foundation, even when you are still testing and iterating. That means you can roll out new versions safely, test changes against real traffic, shadow production requests, route traffic across deployments, scale with demand, and roll back if something does not work as expected.

Together already got our latency where it needed to be for voice. What their new inference platform changes is how we ship the next model version: we can canary a new fine-tuned model on a small percentage of live traffic and automatically roll back if key metrics regress. We will go from ‘fast inference’ to ‘fast inference we can safely iterate on every week.’

That is ultimately what we wanted to unlock: not just fast, efficient inference, but the ability to continuously improve what you are serving without introducing unnecessary risk. You can start with the paths Together has already optimized, then take on more control as your workload becomes more specialized. That principle is reflected across the platform—from how you bring in and configure a model to how you scale it, test changes, observe performance, and move new versions into production.

Deploy models from Together’s model platform, or bring your own open-weight or fine-tuned model. You can upload full model weights or adapters from Hugging Face, S3, or your local machine.

The platform is designed to support the full lifecycle of a model, from an early fine-tune through the versions that eventually serve production traffic.

The model itself is only one part of the deployment. Hardware type, quantization, tensor parallelism, speculative decoding, and the serving engine all affect the performance and economics of the workload.

You should not have to become an expert in every one of those choices to get strong results. Together deployment profiles package configurations our research and engineering teams have already tested and optimized, so you can start with a proven setup rather than assembling one from scratch. When you need more control, those choices are still available. You can select from different hardware and optimization profiles and, over time, choose whether you want to optimize primarily for latency, throughput, or a balance between the two.

The larger models become, the more startup time is dominated by moving hundreds of gigabytes of weights into place. We rebuilt our model caching and distribution layer so that those weights can be shared across the fleet and proactively warmed before a deployment needs them. In internal testing, this has delivered roughly 4× faster warm starts across several frontier models.

Llama 3.3 70B, Qwen 3 235B

Kimi K2.7, MiniMax M3, GLM 5.2

Choose where your model runs by pinning it to a specific region or leaving placement flexible. You can also define how the deployment responds to changes in traffic.

Different workloads have different signals for demand, so autoscaling should not be limited to a single generic metric. The platform starts with a sensible default, while allowing you to scale based on inflight requests, GPU utilization, time to first token, latency, decoding speed, or throughput when your workload calls for something more specific.

An inference endpoint no longer has to map to a single, static deployment. You can create multiple deployments behind the same endpoint, each with different model weights, hardware, or serving configurations.

This lets you change what is running behind the API without forcing your application to change how it calls the model. The endpoint remains stable while the deployments behind it evolve—making it easier to test, roll out, and replace models or configurations without reworking the application layer.

Use canary, blue-green, or rolling updates to introduce new model versions and configurations gradually.

The platform handles the traffic movement and deployment lifecycle, while giving you control over how quickly a change progresses and when it should stop or roll back. You get the production controls you need without having to build the rollout machinery yourself.

Use A/B testing to route a percentage of traffic across different deployments and compare their behavior directly. Use shadow traffic to mirror production requests to a new deployment without affecting the response returned to the user.

This makes it possible to evaluate new weights, configurations, and hardware using the traffic that actually matters, rather than relying only on offline benchmarks or synthetic tests.

The new platform gives you an organization-level, scrapeable Prometheus endpoint that you can connect to the observability tool of your choice to build dashboards, set alerts, monitor rollout health, compare A/B test variants, and measure shadow traffic against production.

We have also refreshed the in-product analytics experience to make deployment health, traffic, performance, and scaling behavior easier to understand directly in Together.

Alongside the inference platform, we're launching a closed beta for custom training, including full-weight and LoRA reinforcement learning as well as advanced supervised fine-tuning. You can configure training through the Python SDK and granular primitives, and run multiple LoRA experiments concurrently on dedicated capacity. Custom training connects experimentation directly to production. When a checkpoint is ready to evaluate, you can deploy it natively to your inference deployments, with no separate handoff between training and serving and no rebuilding your setup to move from a post-training experiment to production evaluation on the same platform. Access is limited during the closed beta.

‍Request access to get started.

This update is just the beginning. It is the foundation for a platform where models can be trained, optimized, deployed, measured, and continuously improved without stitching together a different system for every part of the lifecycle.

Open-weight models give companies an enormous amount of control over quality, performance, and cost. Our job is to make that control accessible without turning inference into a DIY infrastructure project. That means providing defaults that work, configurations we have already optimized, and deeper control when you need it.

Over the coming weeks, we’ll go deeper on the systems behind the platform, including deployment profiles, model startup, autoscaling, observability, rollouts, A/B testing, and shadow traffic. For now, the new Dedicated Model Inference platform is available to help you move from early iteration to production and keep improving what you serve once you get there.