How to Measure Video Similarity: 6 Techniques I Tested (and the One I Shipped)

Two short clips. One question: how alike do they look? Sounds trivial, it isn’t, and I learned that the slow way. My setup: one reference clip, eight others to rank against it, all waterfalls (more on why in a second). I figured this was an afternoon job, grab a model, compute a number, move on. Instead I watched supposedly-smart methods rank near-identical clips in nonsense orders, and the one that looked best on paper was too slow to actually use. So I benchmarked six methods, same clips, same rules, and judged them accuracy first. A wrong answer in a millisecond is still wrong, so speed only gets a say once a method proves it can rank correctly. Accuracy first, speed as the tiebreaker. Here’s what I found. Does sampling frames even help? So which one should you actually use? My first instinct was to just compare pixels. That’s the trap: you’re actually comparing meaning, the subject, colors, light, whether the water crashes or trickles. Chase meaning and you’re picking from three families, each costing you something. Embeddings sample frames through a model that knows what images mean, smart, but costs milliseconds or API credits. Fingerprints crush each frame to a…