Why Transformers Need Positional Encoding For Time Series: A Visual Guide

While digging into foundation models for time series, I realized that I could not really understand them without first understanding transformers. I did not want to use these models as black boxes, so I started tracing the ideas backward, from foundation models to transformers, and from transformers to self-attention. What made the transition interesting is that although transformers were originally built for language, the core idea carries naturally to time series. The two modalities are very different, but they share something fundamental: both are sequences, and in both cases, order changes meaning. In language, dog bites man is very different from man bites dog. Time series are no different. A temperature of 30∘30^\circ30∘ yesterday and 20∘20^\circ20∘ today tells a different story from 20∘20^\circ20∘yesterday and 30∘30^\circ30∘ today. The values may be the same,…