Gradient Clipping and Exploding Gradients in RNNs
Why recurrent networks blow up during training, and why clamping the gradient norm is often the simplest fix that actually works.
The problem hiding inside backpropagation through time
When I first trained a vanilla RNN on a moderately long sequence, the loss looked fine for a few steps and then became NaN with no warning. This is one of the most common early lessons in sequence modelling, and it is worth understanding precisely rather than just patching it away. Recurrent networks apply the same weight matrix repeatedly, once per timestep, and backpropagation through time multiplies gradients by that matrix's derivative at every step going backwards. If the dominant eigenvalue of that matrix is even slightly above one, the gradient magnitude grows exponentially with sequence length rather than linearly.
Concretely, imagine a toy recurrent weight matrix whose largest eigenvalue is 1.2. Over 50 timesteps, a naive estimate of the compounding effect is 1.2 raised to the 50th power, which is well over five thousand. In practice the real dynamics are messier because of nonlinearities and varying eigenvalues at each step, but the qualitative behaviour is the same: small multiplicative excesses compound into gradients so large that a single weight update can overshoot and destroy the parameters entirely. This is the exploding gradient problem, and it is the mirror image of the vanishing gradient problem, where an eigenvalue slightly below one causes gradients to shrink towards zero instead.
What makes this particularly frustrating in practice is that the symptom often looks like a bug rather than a numerical instability. Losses that suddenly jump to NaN, weights that saturate, or training that diverges after looking stable for many epochs are all consistent with exploding gradients, and it is easy to waste time checking data loading or label encoding before realising the actual cause is the recurrent structure itself.
Why clipping the norm works better than clipping each value
The standard fix is gradient clipping, and the version that matters most in practice is global norm clipping rather than clipping individual gradient values. The idea is to compute the L2 norm of the entire gradient vector across all parameters, and if that norm exceeds some threshold, rescale the whole vector down so its norm equals the threshold exactly. Crucially, this rescaling preserves the direction of the gradient; it only shrinks the step size, so the optimiser still moves the parameters the same way, just less aggressively.
Say the gradient norm on a given batch turns out to be 40, and the threshold is set to 5. Norm clipping multiplies every component of the gradient by 5/40, which is 0.125, so the update direction is untouched but the magnitude is controlled. Contrast this with clipping each gradient element independently to some fixed range, say minus one to one. That approach distorts the direction of the update, because components that were large relative to others get truncated disproportionately, effectively changing which parameters get emphasised. For recurrent networks in particular, where the relationship between weights across timesteps matters, preserving direction while controlling magnitude is usually the more sensible compromise.
It is worth being honest about what clipping does and does not fix. It does not solve exploding gradients at their root; the underlying recurrent dynamics still amplify signals across time. What clipping does is prevent any single batch from producing a catastrophic update that knocks the network out of a reasonable region of parameter space. It is a safety valve, not a cure, and that distinction affects how you should think about tuning it.
Choosing a threshold without fooling yourself
Picking the clipping threshold is often treated as an afterthought, but it deserves the same care as any other hyperparameter. A threshold set too low will cap the network's ability to make large, legitimate updates when they are genuinely needed, effectively slowing learning across the board. A threshold set too high defeats the purpose and lets occasional spikes through anyway. A practical habit is to log the unclipped gradient norm for a few hundred batches early in training and look at its distribution. If the typical norm sits around 2 to 4 but occasional batches spike to 30 or more, a threshold somewhere around the 90th or 95th percentile of the typical range is a reasonable starting point, adjusted by watching whether training stabilises.
One thing I am careful about is not letting gradient clipping mask a different problem entirely. If gradients are exploding on nearly every batch rather than occasionally, that is often a sign of a learning rate that is too high, a poorly scaled input, or an architecture choice that is fundamentally unstable for the sequence lengths involved. In that situation, clipping might make training limp along without NaNs, but the model may still learn poorly or slowly, and the honest fix is elsewhere: better initialisation, shorter truncated sequences during backpropagation through time, or switching to gated architectures such as LSTMs or GRUs, which are specifically designed to keep gradient flow closer to well behaved across long sequences.
It is also worth remembering that gradient clipping addresses explosion, not vanishing. If your recurrent model is failing to learn long range dependencies because gradients are shrinking towards zero rather than growing, clipping will do nothing useful, and you need architectural or optimisation changes aimed at the opposite problem.
A practical takeaway
If you are training a recurrent model and see sudden divergence, treat exploding gradients as a first suspect before assuming a data or labelling bug. Add global norm clipping early rather than as a late fix, log the unclipped norm so you can see what is actually happening rather than guessing at a threshold, and be honest with yourself about whether clipping is stabilising a fundamentally sound setup or quietly papering over a learning rate or architecture problem that deserves a real fix. Used this way, clipping is a small, well understood tool that buys you stability without pretending to solve problems it was never designed to solve.
