Hyperparameters vs Parameters
A parameter is a number the model learns; a hyperparameter is a number you choose before training starts. One test separates them, and only about five of them are worth your time.
A parameter is a number the model learns. A hyperparameter is a number you choose. That is the whole distinction, and the only reason it causes trouble is that statistics uses "parameter" to mean something slightly different.
Who sets the number
A weight is a parameter: training discovers it, and nobody types its value. The learning rate is a hyperparameter: no amount of training will set it, because it is one of the things that governs how training happens at all.
Ask "could gradient descent have found this?" Weights, yes — that is exactly what the arithmetic does. The learning rate, no: it is a term inside that arithmetic. You cannot learn the rule you are learning by.
The counts are wildly different. A modern model has billions of parameters and perhaps a couple of dozen hyperparameters. The billions are set automatically; the couple of dozen are your job, and they decide whether the billions come out any good.
Which hyperparameters actually matter
Most of them do not matter much. A few dominate.
The learning rate deserves its position at the top. Too high and the loss diverges into NaN; too low and you wait days for what should take an hour. There is a live demonstration of both failure modes in gradient descent, by hand.
How to search
- Random search beats grid search. This is counter-intuitive and well established. A grid wastes most of its budget varying hyperparameters that do not matter; random sampling gives you far more distinct values of the ones that do.
- Search in log space. Learning rates live on a multiplicative scale — 1e-4, 3e-4, 1e-3 — not an additive one. Sampling uniformly between 0.0001 and 0.1 puts almost every sample near the top of the range.
- Change one thing at a time when diagnosing, even though the best final combination usually requires changing several.
- Use a validation set, never the test set. Tuning against your test set means your final number is optimistic and you have no honest estimate left. See train, validation, test for why that third split exists.
Every hyperparameter you tune against a dataset quietly fits the model to that dataset. This is why the test set must be touched once, at the very end. Tune on validation, report on test.
The vocabulary trap
In statistics, a hyperparameter means a parameter of a prior distribution — the Bayesian sense. Search for the bare word and that is frequently what you will find, and it is not what machine-learning practitioners mean.
In ML the meaning is simply: a setting you chose rather than learned. Same word, different field, genuinely different concept.
The short version
- Parameters are learned by training. Hyperparameters are chosen by you, before it.
- Test: could gradient descent have found this number? If no, it is a hyperparameter.
- Billions of the first, a couple of dozen of the second.
- Learning rate matters most. Then batch size, then regularization strength.
- Random search, log scale, validation set — and never tune on test.
Related
- Gradient descent, by hand — where the learning rate does its damage.
- Epoch vs batch vs iteration — two more hyperparameters, in detail.
- SGD vs momentum vs Adam — the optimiser is a hyperparameter too.
- Overfitting vs underfitting — what bad tuning looks like.
- Bias vs variance — what you are really tuning for.