NVIDIA Dynamo-Triton speeds up generative recommenders with PyTorch

NVIDIA Dynamo-Triton now supports end-to-end Hierarchical Sequential Transduction Unit (HSTU) Generative Recommender inference, combining technologies including FlexKV-backed KV caching and NV embedding cache. The system achieved speedups of up to 5.93x for an eight-layer HSTU model on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU, compared to the same configuration without KV caching, under a 100% GPU KV-cache hit rate, Dynamo-Triton says. This result demonstrates a practical path for serving HSTU ranking models with low latency despite long user histories and large item catalogs. The workflow allows for model deployment without rewriting for a separate runtime. The system tackles the computational expense of processing long user histories common in modern recommendation systems by enabling reuse of previously computed attention state, a key innovation for large-scale personalization.…