Watermarking in vLLM

What is text provenance, and why is it challenging? Establishing the provenance of digital content is essential to build trust and accountability in the information we share. Watermarking offers a solution: it alters a work to embed a message about it [1], for instance its provenance. Images and audio can embed watermarks as a subtle perturbation, but these techniques are not applicable to text due to its discrete nature. Instead of trying to perturb the text itself, we influence the generation process in a way that is detectable but does not modify the expected output distribution. Adding a watermark should not modify the expected output distribution of a model [2]: it should not systematically favour certain words, writing styles, or solutions to problems. Generated text can be altered in many ways: shortened, edited, mixed with other corpora, or even rephrased by another LLM. A good watermarking system aims to preserve its provenance signal even under such modification. Watermarking should add minimal latency and memory overhead to generation and detection. This is particularly important in a high-throughput serving engine such as vLLM. Slowdowns arise both when serving small models, sensitive to changes in scheduling latency, and models with large vocabulary sizes.…