Batch Size: The Quiet Hyperparameter Nobody Tunes

The hyperparameter that hides in plain sight

Ask most practitioners which hyperparameter matters most and they will say learning rate, and they are not wrong to prioritise it. But ask them why they chose their batch size and the answer is usually something like 'it fit on the GPU' or 'it was the default in the example notebook'. Batch size gets treated as a memory constraint rather than a modelling decision, which is strange given how much it actually influences the shape of training.

The reason it gets ignored is partly practical. Larger batches often mean faster epochs on modern hardware, so there is a natural pull towards 'as large as fits'. But batch size is not just an efficiency knob. It changes the statistical properties of the gradient estimate at every step, which in turn changes how the optimiser moves through the loss landscape. Treating it as fixed while you carefully tune learning rate is a bit like tuning the steering on a car while ignoring that you have swapped the tyres.

I want to be precise about what batch size actually controls, because vague appeals to 'noise' do not help anyone make a decision. It controls the variance of the gradient estimate used at each update. A small batch gives a noisy but cheap estimate of the true gradient over the full dataset; a large batch gives a smoother but more expensive one. That single trade-off cascades into almost everything else: how large a learning rate you can safely use, how many updates you need, and even which minima the model tends to settle into.

A worked intuition: same data, three batch sizes

Suppose you are training a moderate-sized classifier on a dataset of fifty thousand examples. With a batch size of 32, one epoch gives roughly 1,560 gradient updates. With a batch size of 512, the same epoch gives about 98 updates. That is a sixteen-fold difference in how many times the model's weights get nudged per pass through the data, even though the total amount of data seen is identical.

Now think about what each update actually represents. At batch size 32, each gradient is estimated from a small, noisy sample, so consecutive updates can point in noticeably different directions. This noise acts almost like a mild regulariser: it discourages the optimiser from settling too quickly into sharp, narrow minima that fit the training batch precisely but generalise poorly. At batch size 512, the gradient estimate is much closer to the true full-batch gradient, updates are smoother and more consistent, but you get far fewer of them per epoch, and the optimiser can glide confidently into a sharper minimum simply because there is less noise to knock it out.

This is why doubling batch size and leaving the learning rate untouched so often produces a worse-performing model, even though the loss curve looks perfectly well-behaved. The learning rate that suited the noisy, frequent updates of batch size 32 is now too timid for the smoother, less frequent updates of batch size 512. A common rule of thumb is to scale the learning rate roughly in proportion to the batch size when moving into this larger regime, but that scaling has limits and needs verifying with a held-out validation curve rather than assumed blindly.

The practical consequence is that batch size and learning rate are not two independent knobs, they are coupled. If you tune learning rate at batch size 32 and then later decide to increase the batch size for speed, you have quietly invalidated your tuning. Any serious hyperparameter search should treat batch size as part of the same search space as learning rate, not as a separate, earlier decision made once and forgotten.

Why this matters beyond training curves

The generalisation gap between small and large batch training is not folklore, it shows up reliably enough in careful ablations that it is worth taking seriously as a default expectation, even without citing a specific number. Models trained with very large batches and no compensating adjustments (learning rate warm-up, adjusted regularisation, longer training) tend to reach lower training loss but validation performance that lags behind an equivalent small-batch run. If your evaluation pipeline is leakage-aware and your validation split is trustworthy, this gap is exactly the kind of thing a proper batch size sweep will reveal, and exactly the kind of thing that a single fixed batch size will hide from you entirely.

There is also a reproducibility angle that I think gets underweighted. If you report results from a model trained at batch size 256 without mentioning that you never checked batch size 64 or batch size 1,024, you are implicitly claiming that 256 was the right choice, when in fact it may have just been the largest size that comfortably fit in memory that afternoon. Reproducing your result later, on different hardware with different memory limits, someone else might reasonably pick a different batch size and get meaningfully different numbers, through no fault of their method.

Practically, this means batch size deserves the same treatment as any other hyperparameter: a small, deliberate sweep, evaluated on a proper validation split, ideally alongside learning rate rather than in isolation. You do not need an exhaustive grid. Trying two or three batch sizes that differ by a reasonable factor, say 32, 128, and 512, each paired with a learning rate scaled appropriately, will usually tell you whether your model is sensitive to this choice at all. Some architectures and datasets are fairly robust to it; others are surprisingly fragile.

The practical takeaway

Batch size is not a memory afterthought, it is a control on gradient noise that shapes both how quickly you train and where you end up in the loss landscape. Treating it as fixed while you tune everything else around it is a subtle form of leakage in your experimental design: your conclusions about learning rate, regularisation, or architecture are all conditioned on a batch size you never actually validated.

The fix is simple and cheap relative to the risk. Include batch size explicitly in your hyperparameter search, pair it with learning rate rather than tuning them separately, and check validation performance rather than training loss alone when judging whether a larger batch has actually helped. It will not always change your conclusions, but when it does, it is usually because you were about to publish or deploy a model that was quietly undertrained for the batch size you happened to pick.