Learning Rate: The First Hyperparameter to Tune

Why this one knob matters more than the rest

When people start tuning a model, there is a temptation to reach for the exciting levers first: add a layer, try a fancier architecture, bolt on some regularisation. I understand the appeal, but in practice almost none of that matters if the learning rate is wrong. Get it wrong and a beautifully designed network will either sit there barely learning or diverge into useless noise within a few steps. Get it roughly right and even a plain baseline will train sensibly, giving you something honest to compare against.

The reason learning rate deserves first place is that it controls the step size in gradient descent, and every other hyperparameter is downstream of that step size. Batch size interacts with it. Weight decay interacts with it. The effective capacity of a schedule, warmup, or optimiser like Adam versus plain SGD all interact with it. If you tune architecture or regularisation while the learning rate is badly chosen, you are optimising on top of a broken foundation, and any conclusion you draw about what helped or hurt is unreliable.

There is also a practical, time-based argument. Learning rate sweeps are cheap. You can often tell within a few hundred or a few thousand steps whether a rate is hopeless, because loss curves diverge or flatline early. Compare that to tuning something like network depth, where each configuration might need a full training run to judge fairly. Spending your limited compute budget on learning rate first gives you the best information per pound spent.

A worked intuition: three rates, three outcomes

Picture a simple regression problem where you are fitting a curve to noisy data using gradient descent, and suppose the loss surface has a fairly narrow valley. Say a sensible learning rate for this problem, given the scale of the gradients, sits somewhere around 0.01. Now imagine trying three different values around that point.

At a learning rate of 0.0001, training is stable but painfully slow. After a thousand steps, loss has barely moved from its starting value, and you might reasonably conclude the model architecture is too weak or the features are unhelpful. In fact the model is fine; it is simply crawling towards the minimum in steps so small it will take tens of thousands of iterations to get anywhere useful. If you stop early and judge the architecture based on this run, you draw the wrong conclusion entirely.

At a learning rate of 0.01, the loss drops sharply for the first few dozen steps and then settles into a steady, mostly monotonic decline. This is the behaviour you want: fast early progress followed by fine adjustment as the loss surface narrows. Small oscillations near the minimum are normal and expected, not a sign of failure.

At a learning rate of 1.0, the update step is so large that each gradient step overshoots the valley and lands on the opposite wall, often with a higher loss than before. A few steps in, the loss can start increasing rather than decreasing, sometimes diverging to very large values or producing numerical overflow. This is the classic instability signature: a loss curve that looks like it is bouncing or growing rather than settling. It has nothing to do with the model's capacity and everything to do with step size.

How to search for it without guesswork

A sensible first move is a learning rate range test: start training with a very small rate and increase it, often on a logarithmic scale, over the course of a few hundred to a couple of thousand steps, plotting loss against the rate used at each step. The resulting curve typically shows a flat or slowly decreasing region at small rates, a sharp drop where the model is learning fastest, and then a sudden explosion once the rate is too high. A good starting point for full training is usually somewhere just before that explosion point, not at the very edge of it, since the edge is often unstable across random seeds.

It is worth doing this range test on a held-out slice of training data that never touches your validation or test sets. The point is purely to characterise the loss surface's sensitivity to step size, not to select a final model, so leakage here is less about test contamination and more about wasting the diagnostic on data you will need later for honest comparisons. Keep the range test lightweight and separate from your main evaluation protocol.

Once you have a rough working rate, the next decision is whether to keep it fixed or schedule it. In my experience a short warmup followed by a decay, whether step-based, cosine, or linear, tends to be more forgiving than a single fixed value, especially for larger models or larger batch sizes where the initial gradients can be noisy. But schedules are a refinement, not a substitute for the initial search. If the base rate is wrong, no schedule will rescue the run; it will just delay or reshape the failure.

One caution worth stating plainly: learning rate is not independent of batch size. Increase the batch size substantially and the effective noise in each gradient estimate drops, which often means the learning rate that worked before is now too conservative. If you change batch size, treat the learning rate as needing a fresh look rather than assuming the old value still applies.

The practical takeaway

Before you tune architecture, regularisation strength, or anything else, spend a small, cheap experiment establishing a workable learning rate. Use a range test or a simple grid across orders of magnitude, watch for the signatures of too small (flat, slow loss) and too large (diverging, oscillating loss), and settle on a value in the stable, fast-decreasing region rather than at its unstable edge. Everything else you tune afterwards will be more meaningful, because you will be improving a model that is actually learning, rather than papering over a training process that was broken from the first step.