Multi-Task Learning: When a Shared Backbone Helps or Hurts

Why the idea matters

Multi-task learning promises something that sounds almost too good: train one backbone, attach a few task-specific heads, and get several models for the price of one. Fewer parameters to store, faster inference, and in the best case, tasks that help each other by sharing statistical structure. It is a genuinely attractive idea, and it does work in plenty of settings. The trouble is that the same architecture can also quietly make every task worse than a set of separate models would have been, and the failure is easy to miss because the aggregate loss curve can still look fine.

I think the honest way to frame multi-task learning is not as a technique that helps or hurts in general, but as a bet on task relatedness. When you share a backbone, you are asserting that the features useful for task A are also useful, or at least not actively harmful, for task B. That assertion is sometimes true and sometimes false, and the only way to know is to measure each task against its own single-task baseline, not against some averaged multi-task score that can hide one task quietly propping up another.

A worked intuition

Imagine a shared convolutional backbone feeding two heads: one predicting whether an image contains a cat or a dog, the other predicting the breed. These tasks are closely related. The low-level and mid-level features that help distinguish a cat from a dog, edges, fur texture, ear shape, are largely the same features that help distinguish a beagle from a labrador. If the species classifier alone reaches 96 percent accuracy trained solo, and the breed classifier alone reaches 78 percent, a shared backbone trained on both jointly might push species to 96.5 percent and breed to 80 percent. The shared representation acts as a regulariser: the breed task, which has less data per class, borrows statistical strength from the species task's more abundant labels. This is the case where multi-task learning genuinely earns its keep.

Now change one task. Keep the species head, but replace the breed head with a task predicting the time of day the photo was taken, inferred from lighting and shadows. These tasks want different things from the backbone. Species classification wants invariance to lighting, since a cat is a cat whether it is noon or dusk. Time-of-day estimation wants the backbone to preserve exactly the lighting cues that species classification is trying to discard. Trained jointly, you might see species accuracy drop from 96 to 93 percent and time-of-day accuracy come in worse than a dedicated model trained alone, because the shared layers are being pulled toward a compromise representation that serves neither task well. This is negative transfer, and it is not a bug in the implementation, it is the architecture faithfully reflecting a genuine conflict in what the tasks need.

The practical lesson from this pair of examples is that task similarity is not something you can eyeball from task names. Species and breed sound related and are related at the feature level. Species and time-of-day sound unrelated in a way that turns out to matter mechanically, not just semantically, because the features that help one actively hurt the other.

Where the sharing quietly breaks

Beyond outright conflicting objectives, there are subtler failure modes worth naming. Data imbalance across tasks is one of the most common. If one task has ten times more labelled examples than another, gradients from the larger task will dominate backbone updates unless you carefully weight losses or balance sampling. I have seen setups where a supposedly shared backbone was, in practice, being shaped almost entirely by the dominant task, with the smaller task's head bolted onto features it had little say in forming. The smaller task's reported gain over its own single-task baseline was real but modest, while its ceiling was clearly being capped by features it did not get to influence.

Another quiet failure is task difficulty mismatch combined with shared learning rates. A task that is easy to overfit will drive the backbone toward representations that memorise its idiosyncrasies early in training, while a harder task is still trying to extract general structure. Without per-task loss weighting or gradient balancing schemes, the easy task can effectively hijack early training dynamics. This is not the same as negative transfer from feature conflict, but it produces a similar symptom: one task does fine, the other underperforms its solo baseline, and it is tempting to blame the whole multi-task premise when the real issue is optimisation imbalance.

Evaluation discipline matters enormously here, and it is where I think a lot of practitioners cut corners. You need single-task baselines trained with the same data splits, same amount of tuning effort, and same compute budget as the multi-task model, otherwise you are not measuring transfer, you are measuring whether you tuned one setup more carefully than the other. You also need per-task metrics reported separately, never just an averaged score, because averaging can mask one task losing three points while another gains three, which nets to zero but is a genuine regression for whoever depends on the losing task. And you need to check this at multiple points in training, since negative transfer sometimes only appears after enough epochs for the backbone to specialise unhelpfully toward the dominant task.

Practical takeaway

Before committing to a shared backbone, ask whether the tasks would plausibly benefit from the same invariances and the same feature granularity, not just whether they sound like they belong together. Run single-task baselines with genuinely equal care, report per-task metrics rather than averages, and watch for the imbalance and difficulty-mismatch failure modes separately from true feature conflict, since they call for different fixes: loss weighting and sampling balance for the former, task selection or partial sharing for the latter. If, after all that, every task still beats or matches its solo baseline, the shared backbone has earned its efficiency. If even one task quietly loses, that is not a rounding error, it is the model telling you the tasks disagree about what a good representation looks like.