What Can I Actually Do with a Small Language Model?

The most common objection to running a small language model (SLM) is that it does not know enough, an objection that isn't well calibrated. In reality, no model on the market is a dependable store of facts, and general capability benchmark scores turn out not to predict knowledge reliability at all. The two measures are not correlated. A smaller model knows less than a large one, certainly, but "knows less" isn't a metric of much use, it turns out. If capability is not the reason to run a model on your own hardware, then something else is, and there are three candidates: data that cannot leave the building; volume you have already paid for; and latency that is the whole product. Each of these calls for a specific kind of work that a competent off-the-shelf model of eight billion parameters or fewer (the rough ceiling for a non-specialized piece of hardware that you…