Synthetic Data for Vision: Closing the Domain Gap
Synthetic imagery can slash labelling costs, but only if you treat the domain gap as a measurable quantity rather than a hope.
Why the domain gap is the real question
Synthetic data generation gets pitched as a shortcut around the expensive, slow business of labelling real images. In principle you get infinite data, perfect ground truth for segmentation masks, depth, and pose, and full control over lighting, occlusion, and rare events. In practice, a model trained purely on rendered scenes often collapses when it meets a real camera feed. This is the domain gap: the statistical mismatch between the distribution the model learned and the distribution it is tested on. Textures are too clean, noise is absent, shadows fall in unnaturally consistent ways, and the renderer's idea of a cluttered warehouse is not the same as an actual cluttered warehouse.
The mistake I see most often is treating synthetic data as a binary choice: either it works or it does not. It is more useful to treat the domain gap as a quantity you measure directly, then close incrementally. A model trained on synthetic images only and evaluated on synthetic images only will report excellent numbers that tell you nothing about deployment. The only honest test is training on synthetic data and evaluating on a held-out set of real, unseen images that were never touched during rendering, texture design, or hyperparameter tuning. That gap, real-test accuracy minus synthetic-test accuracy, is the thing you are actually trying to shrink.
Consider a simple, realistic scenario: detecting damaged packaging on a conveyor belt. You render ten thousand synthetic boxes with procedurally generated dents, tears, and label misprints, train a detector, and get ninety-six percent precision on a synthetic validation set. Deployed against the real camera, precision drops to sixty-one percent. That thirty-five point gap is not a rounding error, it is the entire story, and no amount of synthetic-only tuning will reveal it because the synthetic validation set is drawn from the same generative process as the training data, so it cannot expose distribution shift by construction.
What actually closes the gap
Three techniques reliably narrow that gap, and it is worth being precise about why each one works rather than treating them as interchangeable tricks. Domain randomisation deliberately over-diversifies the synthetic distribution: instead of trying to make textures, lighting, and camera noise photorealistic, you randomise them wildly, far beyond what real deployment conditions would show. The logic is that if the model has seen a thousand implausible lighting conditions, the one real lighting condition it meets at deployment sits comfortably inside the training distribution rather than outside it. This works well for geometry-dominated tasks like pose estimation or object detection where shape matters more than surface appearance.
Domain adaptation takes the opposite approach: rather than randomising the synthetic side, you nudge synthetic and real feature distributions towards each other, either by adding an adversarial loss that penalises a discriminator for telling synthetic and real feature embeddings apart, or by using image-to-image translation networks to give synthetic renders a real-world texture and noise profile before training. This is more compute-hungry and more fragile, since the translation network can introduce its own artefacts, but it tends to outperform domain randomisation on tasks where fine surface detail genuinely carries the discriminative signal, such as detecting corrosion or fine cracks.
The third and most underrated technique is simply mixing a small amount of real, labelled data into an otherwise synthetic-heavy training set. Going back to the packaging example: adding just three hundred real, hand-labelled images, roughly three percent of the synthetic volume, to the training set closed the observed gap from thirty-five points down to around eight in comparable published sim-to-real studies. The intuition is that the model does not need thousands of real examples to recalibrate; it needs enough real signal to anchor its decision boundary in the correct part of feature space, and synthetic data does the heavy lifting of teaching it the general shape of the problem.
Where synthetic data quietly fails
Synthetic data closes the gap fastest on tasks that are fundamentally geometric or structural: bounding box detection, keypoint localisation, depth estimation, and pose. These tasks care about shape, relative scale, and spatial arrangement, all of which a renderer can produce faithfully even without photorealism. It closes the gap slowest, and sometimes not at all, on tasks that depend on fine-grained surface statistics: material classification, subtle defect detection, or anything where the signal lives in high-frequency texture that current renderers approximate poorly, like fabric weave, rust patina, or skin texture.
There is also a subtler failure mode worth naming explicitly: label leakage through the rendering pipeline itself. If your synthetic scenes always place damaged boxes under slightly dimmer lighting because that was an artefact of how the damage-generation script was written, the model will learn to detect lighting rather than damage, and it will look excellent on synthetic validation while being useless in the real world. This is a leakage problem in the same family as train-test contamination, just dressed differently, and it is caught the same way: by auditing what correlates with the label in your generation process, not just what the label is supposed to mean.
The practical takeaway is to never trust synthetic-only evaluation numbers, always hold out a small real-world test set that is genuinely representative of deployment conditions, and measure the real-versus-synthetic performance gap as a first-class metric rather than an afterthought. If that gap is small and your task is geometric, synthetic data alone might carry you most of the way. If the gap is large or the task depends on fine texture, budget for a modest amount of real labelled data rather than chasing photorealism in the renderer indefinitely, since a few hundred well-chosen real examples usually buy more accuracy than another month spent perfecting shader code.