Underpainting is basically outpainting turned inward.
Instead of extending the edges of an image, you’re trying to reconstruct what should exist underneath the visible composition. A kind of sleight of hand, if you think about it.
Let's walk through how I'm doing it.
SAM can give you masks. It can pull an image apart into usable pieces. But it doesn’t really understand the scene.
A mask is not a judgment call.
Those layers also tend to come through in pieces. For example, if the image has a cat in it, you probably don't get one giant “cat” layer.
At first, that’s annoying.
Then you realize it’s helpful.
This breakdown gives you control: the ability to push and pull on things, adjust tilt, move objects around, and play with the fun chonky art bits that make the process enjoyable.
That’s why we need to bring in reasoning.
Separate and reconstruct
You start by separating the image into layers, then ask a reasonably reliable VLM what it sees. The VLM identifies the objects, sorts them into foreground, midground, or background, and names the layers so the rest of the pipeline can use them.
Once the layers are separated and labeled, you isolate whatever counts as background and flatten those pieces onto a scratch pad. Think of a scratch pad like an imaginary image in computer memory that only exists while you're working on it.
At this step, you get a nice dirty background with holes in it.
The missing areas, broken textures, edge artifacts, and half-erased shadows tell the model where the image needs repair.
Good news: that’s usually an easy prediction.
Just inpaint and outpaint through the gaps until you have a better guess at the world underneath the foreground.
Then you pull it apart again.
Remove the leftovers
This second pass is where the VLM earns its donuts.
You show it the original image, the layer packet, and the reconstructed background candidate.
From there, it can help you spot anything from the original composition that is still contaminating the background.
Remove those leftovers.
Paint over the gaps again.
Now you're getting close. Mostly clean is perfectly acceptable at this point. Clean just means the background can sit behind movable layers without looking like a Photoshop crime scene.
Reconstruct the overlaps
After that, you repeat the same basic trick anywhere layers collide.
When one object overlaps another, the hidden part of the lower object becomes a reconstruction problem.
You isolate the visible piece, infer what's missing, paint behind the obstruction, then put the scene back together as independent movable layers.
Upper layers are usually easier because there is less hidden underneath them. Backgrounds are the hard part because they carry the whole illusion.
That’s underpainting.
It’s a kind of scene reconstruction by negotiation between segmentation, visual reasoning, and generative repair.
SAM gives you scissors.
The VLM gives you judgment.
Diffusion gives you paint.
Your app’s job is to make them cooperate without turning the whole thing into soup, then make it look easy enough that the user never thinks to ask how or why any of it happened.
Creative AI apps are a kind of magic that way.