Generative Fill
What Is Generative Fill?
Generative fill replaces whatever is inside a selected area of an image with new pixels drawn by an AI image model. A text prompt says what should appear there, and with the prompt left blank, the model fills the area from what surrounds it, which is how most removals are done.
Adobe introduced the name in May 2023 with a Photoshop beta built on its Firefly models, and Cloudinary uses it for an API feature that pads an image to a new aspect ratio and fills the padding.
How Does Generative Fill Work?
Generative fill tools share one loop, from drawing the selection to keeping the version that fits best.
Select the area to change
The selection marks what the model is meant to replace, from a rough lasso around a tourist to a precise mask around a person. Everything outside it is context the model reads but should leave alone.
Name what should appear
When adding or replacing, a short, specific prompt works best. “Red canoe pulled up on the sand” gives the model a clear target where “boat” could come back as almost anything, and a long, elaborate sentence rarely beats a few precise words.
Let the model generate the fill
The model reads the pixels around the selection and generates content that matches their lighting and perspective. How long that takes depends on the model: when IMG.LY timed 26 models generating an image from a prompt in its AI benchmark, the median ran from 2 seconds to over 2 minutes.
Choose one of the results
Each run comes out different, so tools usually return several versions to compare. If none of them fits, generating again with the same selection and prompt gives a fresh set.
What Can You Do With Generative Fill?
One tool covers four jobs, and what you select and type changes with each. Filling the blank canvas past an image’s edge is also called outpainting, and Photoshop gives it a separate tool, Generative Expand.
| What to select | What to type | |
|---|---|---|
| Remove an object | The object, its shadow and a margin | Nothing |
| Add an object | The spot where it should go | The object to add |
| Replace a background | Everything behind the subject | The new setting |
| Extend the frame | The empty canvas past the edge | Nothing, or the scene |
How Is Generative Fill Different From Content-Aware Fill?
Both can remove an object from a photo, but they get the replacement pixels from different places.
Content-aware fill copies from the photo
Photoshop has offered content-aware fill since CS5 in 2010. It rebuilds the selection from pixels found elsewhere in the same image, which works well on sky, grass or a plain wall. The copied pieces can repeat, though, so a distinctive object next to the selection may turn up twice, and on buildings, crowds and other structured scenes the pattern breaks or smears.
Generative fill creates new pixels
Generative fill pastes in nothing from the photo itself. Its model draws every pixel of the selection, so it can rebuild a background the object was hiding or add an object the photo never had. The result is the model’s best guess at what belongs there, so the small details are worth a closer look before you keep it.
How Do You Get Better Generative Fill Results?
Most failed fills come down to the selection, the source image or a detail that image models still draw badly.
Select a little more than the object
Include its shadow, any reflection and some of the background around it, so no trace of the object is left at the edges.
Work on one small area at a time
Large selections tend to come back softer than small ones, and a low-resolution photo gives a blurry fill. Remove two objects in two passes.
Check faces, hands and text
These are the hardest details for a fill to get right: hands and faces can come out distorted, and lettering garbled. Text accuracy varies by model, and 15 of the 26 in IMG.LY’s text-to-image benchmark got at least one required string wrong. Zoom in before keeping a result.
How AI Photo Editing Works in CE.SDK
Your users remove, add and replace by writing what should change, and an image editing model applies it to the photo they selected.
Describe the edit for a selected photo
On the web, image edits sit in the canvas menu of a selected photo. On iOS and Android, an Edit button in the inspector bar takes the prompt. The instruction and the photo go to an image editing model, which sends back the edited version.
Name the object in the instruction
Users write which object should go or come, and the model finds it in the photo, as the Add & Remove Objects page shows. On the web, a separate action replaces the background of the selected photo from a prompt.
Choose the model behind the edit
On iOS and Android, edits go to the IMG.LY AI Gateway, where FLUX.2 is the default and GPT Image 2 can replace it. In IMG.LY’s benchmark, which tested generation from a prompt and not editing, GPT Image 2 came closest of all 26 models to requested brand colors. On the web, you choose which of the Gateway’s image editing models handles it, or connect a provider of your own.
FAQs
Inpainting is the general technique of filling a missing or masked part of an image. Generative fill is a product feature built on AI inpainting, and when the selection reaches past the edges of the image, the same idea is called outpainting.
It is meant to change only the selection, but the model blends the new content into its surroundings, so a fill can spread a little past the selection edge. In Photoshop the result sits on its own layer, and hiding or deleting that layer brings the original back.
Yes, by extending it. Widen the canvas, select the empty strip, and the model continues the scene into it, which can turn a tall photo into a wide one. Results get weaker as the new area grows, because the model has less of the original to go on.
Links
Documentation and references for this concept.
Related Terms
AI Editor
A CE.SDK Starter Kit pre-configured to surface generative AI tools (image generation, video generation, text generation,…
Visual Prompting
Prompting an image model with an arranged canvas of photos and written notes. In CE.SDK everything on the canvas goes to…
Background Removal Plugin
A CE.SDK plugin that removes the background from images entirely in the user's browser using a client-side machine learn…
Inpainting
Filling in an area of an image that is missing, damaged or masked, so it matches its surroundings. Older methods reuse t…