Knowledge cutoff is a poor proxy for model capability

A model’s knowledge cutoff marks the latest date its training data could cover. But its actual knowledge of a product might end years earlier or be absent entirely. I learned this while trying to answer what seemed like a straightforward question: how much does a model know about a product? If the model’s knowledge ended around a particular date, I should have been able to translate that date to a product version. The model knows the product up to roughly version X. Anything introduced after that needs additional context. The model knows the product up to roughly version X. Anything introduced after that needs additional context. On paper, that would be useful when building agent extensions. You could establish what the model already knows and supply the missing documentation, saving the context window for the work. In practice, the results showed no consistent boundary. To test it, I used GPT-5.6…