Inpainting
What Is Inpainting?
Inpainting fills in an area of an image that is missing, damaged or masked, using content that matches what surrounds it. The word comes from art restoration, where conservators repaint the damaged areas of a painting, and image software took it over for repairing scratches, dust and tears in photos.
In AI tools, you brush a mask onto the area, with white marking the pixels to fill and black the pixels to keep, and you can add a prompt describing what belongs there. People use it to remove objects, text and date stamps, to repair old photos, and to redo a hand or a face that came out wrong.
How Does Inpainting Work?
Methods have moved from reusing the image’s own pixels to generating new ones, and the older ones still suit small repairs.
Spreading color in from the edges
Early digital methods, modeled on how conservators restore paintings, carry the colors and lines at the border of the gap inward. Researchers call them diffusion-based. They work well on scratches and other thin gaps, but anything wider comes out blurred.
Copying patches from the same image
Exemplar-based methods, such as one Microsoft Research published in 2003, search the photo for small patches that match the area around the gap and copy the best match in, working from the edges inward so lines carry through. They handle grass, sand and brick well, but they can only reuse what the photo already contains.
Neural networks trained on photos
Deep learning models learn from example photos what belongs in a hole. Partial convolutions, introduced in 2018, read only the valid pixels, so the empty area's color doesn't leak into the fill. In IMG.LY's 2020 tests, a GAN-based network rebuilt the grid of a metal door hidden behind a person, where the partial-convolution network it was compared with painted the area a uniform black. LaMa, from 2021, was trained on large masks and copes with big gaps and higher resolutions.
Diffusion models that generate from noise
These fill the mask with noise and remove it step by step, putting the unmasked pixels back after every step so the new content keeps fitting them. In Stable Diffusion inpainting, a prompt steers what appears. Despite the name, they have nothing to do with the diffusion-based methods above, since here the word refers to adding and removing noise.
What Do Inpainting Mask Settings Do?
AI inpainting tools share a handful of controls, though each tool names them its own way. The names below are the ones Stable Diffusion tools use.
| What it controls | When to change it | |
|---|---|---|
| Mask blur | How soft the edge of the mask is | Raise it when a seam shows, but too much lets the fill alter pixels you meant to keep |
| Strength | How far the fill may move from the pixels under the mask | Low to recolor or retouch, high to put something new there |
| Only masked area | Fills a crop around the mask at the model's full size | Turn it on for small areas such as faces, which come out sharper |
Why Do Large Masks Give Weaker Inpainting Results?
IMG.LY ran into two limits in 2020 while testing inpainting networks on high-resolution photos with the Bochumer Institute of Technology.
The middle of a large mask is far from real pixels
A model fills a hole from what borders it, so quality drops toward the center of a big mask, where the nearest real pixels are farthest away. The 2020 tests countered this by filling in rounds, shrinking the mask from its edge toward the center each time. The fill looked better for it, though every round ran the network again and added computing time.
Models work at one fixed size
One of the networks in those tests was trained at 512 × 680 pixels, so a much larger photo had to be scaled down, filled and scaled back up, which softened the result. Small masks on large photos still came out well, and the only-masked setting above works on the same principle.
How Layered Editing Works in CE.SDK
In a CE.SDK design, text, logos and photos are separate blocks, so a change to one of them leaves the pixels of the others alone.
Edit text and logos as blocks
A headline is a text block and a logo is its own image, so your users retype a line or swap a logo without a mask or a model call, and the type stays sharp. In IMG.LY’s test of a layered AI ad, swapping the logo took 10 milliseconds.
Cut the background off a photo
On the web, the background removal plugin cuts the subject from a photo inside the browser, so the image never leaves the device. The cutout sits over whatever background the design uses, and moving it leaves no hole to fill.
Regenerate one image and keep the rest
When one photo needs new pixels, your users send only that block to an image-to-image model, and every other block stays exactly as it was. In the same test, regenerating only the background took 38 seconds and left the headline, product and logo as they were, while regenerating the whole ad as one image changed the product, logo and layout from one attempt to the next.
FAQs
Inpainting fills an area inside the image, with real pixels on every side to match. Outpainting adds new area past the original edges, so the model has to extend the scene from context on one side only.
No. Generative fill is a name for prompt-driven inpainting that Adobe introduced in Photoshop in 2023.
Models trained for inpainting blend the fill into its surroundings more naturally, but they are also more likely to alter pixels outside the mask. Pasting the original unmasked pixels back on top keeps them untouched, though the edge around the fill can then look harder.
Yes. Segmentation models such as Segment Anything draw the mask when you click an object, and tools like CLIPSeg build one from a text description. Instruction-based editing skips the mask altogether, as when your users add or remove objects by describing them.
Links
Documentation and references for this concept.
Related Terms
Generative Fill
An AI editing feature that fills a selected part of an image with newly generated pixels, guided by a text prompt or, wh…
Background Removal Plugin
A CE.SDK plugin that removes the background from images entirely in the user's browser using a client-side machine learn…
AI Editor
A CE.SDK Starter Kit pre-configured to surface generative AI tools (image generation, video generation, text generation,…