Background Removal – IMG.LY Blog

Plugin Release Part 1: Customizing the UI & Quick Actions

Jan — Tue, 13 Aug 2024 07:16:36 GMT

At IMG.LY we are driven by a commitment to empower creativity for end users worldwide, we do so by giving our customers the tools to build the ideal creative experience for their customers. For this reason, we are shaping our CreativeEditor SDK to be built for modification and extension. IMG.LY’s customers are not limited by our roadmap. To the contrary, plugins allow customers to not only extend editor functionality, but to also benefit from third-party plugins and open source development based on IMG.LY’s CE.SDK. This new plugin system is designed to give our customers unparalleled autonomy, allowing them to tailor the user experience to their specific needs and accelerate product innovation.

This first release of CE.SDK’s plugin API enables you to customize the UI at certain extension points and locations, as well as change the order and location of UI elements. At these points you will not only be able to change the appearance of existing features but add entirely custom features in the form of quick actions - simple one-click actions to edit canvas elements like removing an image background or vectorizing a graphic.

Extension Points in the UI

Before customizing the UI, it is crucial to understand the user experience we have designed to facilitate intuitive manipulation and creation of a range of designs. The following key locations in the editor UI are extension points for your plugins. We provide a visual example for the type of customization possible at each location:

Canvas: The core interaction area for design content, controlled by the Creative Engine.

Dock: The primary entry point for user interactions not directly related to the selected block. It is primarily, though not exclusively, used to open panels with asset libraries or add elements to the canvas. This release allows you to customize the buttons in the dock. The customization of panels will be possible following the next release.

Canvas Menu: Provides block-specific settings and actions such as deletion or duplication. Here, we customized the menu by adding a background removal action and actions for layer management.

Inspector Bar: Main location for block-specific functionality. Any action or setting available for the currently selected block that does not appear in the canvas menu will be shown here.

Navigation Bar: For actions affecting browser navigation and global scene effects such as zoom or undo/redo. The customization below shows more advanced navigation patterns for multi-steps designs that can be implemented via plugins.

Canvas Bar: For actions affecting the canvas or scene as a whole, such as adding pages. This is an alternative place for actions such as zoom or redo/undo.

Customizing UI Components

You can control the order and layout of components within these locations using Order APIs, which allow you to add, remove, or reorder components. We also provide special layout components like separators and spacers to help organize your custom components effectively.

Registering New Components

Beyond predefined components, you can register your own custom components using builder elements such as buttons, dropdowns, and inputs. This enables you to integrate unique behaviors and logic into the web editor seamlessly. This is how you would implement custom “quick actions” such as background removal

Feature-level Control

In addition to defining custom components, you can use the new feature API to control components on the feature level based on the current context. For instance, you might want to hide the duplicate and delete button for a new “Avatar” block type, you can use the Feature API to selectively disable these components for that specific block.

Examples: Background Removal and Cut-out lines

A straightforward example of a plugin that extends the default editor UI with a quick action that acts on an element is our image background removal plugin.

Here, we customized the canvas menu of image elements to prominently display a “BG Removal” button acting on the currently selected block. To not place so much emphasis on background removal, this feature could have been placed in the inspector bar instead.

The next example is an outlook on what is to come.

Sometimes a plugin manipulates an element but requires additional inputs from the users so that a single button click does not suffice. Such is the case for our cut-out lines plugin, allowing users to create custom cut-out or perforated line shapes on the canvas. The button is placed in the dock and a panel opens on click where the user can select shapes for the cut-out or opt for a dynamic shape that is generated from the selection.

Get Started

Start developing your own plugins by visiting our plugin documentation. We’re excited to see what you’ll build!

What’s Next

The next plugin release will allow you to customize the UI with panels, enabling the extension of increasingly complex behaviors that require additional user input or interaction with external services.

Stay tuned for more updates, and don’t hesitate to reach out if you have any questions or need assistance. Together, we are redefining the future of creative editing.

Thank you for reading. Join 3,000+ creative professionals—subscribe to our newsletter for updates on new features, plugins, and early access.

IMG.LY Announces Open Source JavaScript Library for In-Browser Background Removal

Jan — Wed, 28 Jun 2023 11:38:49 GMT

UPDATE October 2023: We’ve recently introduced Node.js support for the @imgly/background-removal npm package. This enhancement enables you to remove backgrounds from images not only in the user’s browser but also on your server. This opens up the potential for batch processing and allows you to bypass memory constraints, especially for larger images.

Just before the summer lull engulfs us all, we have something exciting to share! We are thrilled to announce the release of @imgly/background-removal, an innovative npm package that empowers developers to seamlessly remove backgrounds from images directly in the browser.

Gone are the days of relying on server-side processing or sacrificing data privacy. With IMG.LY’s Background Removal, you can now harness the power of in-browser background removal with ease. Let’s dive into the key features that make this library truly exceptional:

In-Browser Background Removal

Our one-of-a-kind solution performs the entire background removal process directly in the user’s browser, eliminating the need for additional server costs. By leveraging the computing power of the local device, users can enjoy a fast and efficient background removal process that streamlines their workflow.

Data Protection

Rest assured that your images and sensitive information remain secure within your own devices. As IMG.LY’s Background Removal runs entirely in the browser, there are no data transfers to external servers, ensuring robust data privacy and alleviating any concerns you may have.

You can experience it in action on our background removal showcase.

Seamless Integration with IMG.LY’s CE.SDK

IMG.LY’s Background Removal seamlessly integrates with IMG.LY’s CE.SDK, making it easier than ever to incorporate powerful in-browser image matting and background removal capabilities into your projects. Boost your creative endeavors with this intuitive integration.

The Neural Network (ONNX model) and WebAssembly (WASM) files used by IMG.LY’s Background Removal are hosted on UNPKG, making them readily available for download to all users of the library. However, if you prefer to host the data on your own servers, our library also supports custom asset serving. The choice is yours!

Background removal is often the first step in any creative workflow involving image composition. Whether you are developing e-commerce applications that need real-time background removal, enhancing user experience in image editing applications, or simplifying the creative process with web-based graphic design tools, IMG.LY’s Background Removal is your go-to solution.

Empower Yourself with IMG.LY’s Background Removal

Whether you are a professional developer or a hobbyist, this open source JavaScript library empowers you to deliver impressive applications and services with ease. Join the growing community of developers who are revolutionizing the way background removal is done.

Get started with @imgly/background-removal today by visiting our official npm package page and our GitHub repository. You’ll find comprehensive documentation, examples, and everything you need to unlock the full potential of in-browser background removal.

Update: Node.js Support

We have just shipped Node.js support for the @imgly/background-removal npm package. In addition to the users’ browser, you can now remove backgrounds from images on your server, making it possible to perform batch processing or bypass memory constraints for larger images.

Thank you for your support, and we can’t wait to see the incredible projects you’ll create with IMG.LY’s Background Removal. Stay tuned for more updates and enhancements as we continue to improve and expand this powerful library. Happy coding!

Don’t forget to subscribe to our newsletter to stay in the loop with the latest news and updates.

How to Remove Backgrounds Using Core ML

Walter — Tue, 28 Jun 2022 15:26:53 GMT

In this tutorial, you will learn how to use machine learning to identify an image background and mask it out. This is particularly useful for making stickers or avatars or adding a fake background to a video call. The process of assigning the pixels in an image to a specific object is called “segmentation”. Apple provides an optimized method with pictures of people in its Vision framework. For performing the same tasks with non-human subjects, you can use the DeepLabV3 machine learning model with Core ML. The code examples in this tutorial have been tested using Xcode 13 and Swift 5. Because of the use of Core ML, Vision, and CoreImage in this tutorial, you should run the demo code on a device, not on the Simulator. An iOS project with the demo code is on GitHub.

Image segmentation is a different process and requires other machine learning models than image recognition. With recognition, the model produces bounding rectangles that the system believes to contain the entire object. With segmentation, the model identifies the actual pixels of the object.

In this tutorial, you’ll start with an image of your subject. Then you’ll generate a mask for the background using Vision. Finally, you’ll use CoreImage filters to blend the original image, image mask, and the new image background.

Whether using the Vision framework alone or supplementing Vision with another Core ML model – the process will be the same:

Create a Vision request object with some parameters.
Create a Vision request handler with the image to be processed.
Process the image with the handler and object.
Process the results into the final image using CoreImage.

When working with still images, Apple’s CoreImage framework is usually the best option, mainly because of the large number of available filters and the ability to create reusable pipelines that you can run on the CPU or the GPU.

Using Vision to Segment People

Without an external model, Vision can only segment documents or people in an image. It is vital to understand that Vision will only identify a pixel in the image as “this pixel is part of a person” or “this pixel is not part of a person.” If an image contains a group of people, Vision will not be able to separate individuals.

To segment the image, start by creating an instance of a VNGeneratePersonSegmentationRequest.

var segmentationRequest = VNGeneratePersonSegmentationRequest()
segmentationRequest.qualityLevel = .balanced

This type of request has a few options you can set. .qualityLevel can be .fast, .balanced or .accurate. The level of .accurate is the default. This will determine how closely the mask conforms to the boundaries of the original image. The different levels process at different speeds. The .fast setting is intended for use in a video so that frames don’t get dropped. Using .balanced or .accurate causes noticeable delay in an app processing the image on most devices. Experiment with different settings depending on your needs.

The output of the segmentation request will be a CVPixelBuffer. This is a structure that contains information for each pixel of the image. Most of Apple’s video frameworks as well as CoreImage can work with pixel buffers. The default for VNGeneratePersonSegmentationRequest is a buffer where the color of each pixel is represented by an 8-bit number. Any pixel that Vision thinks contains part of a person will be white and any not-a-person will be black and represented by zero. This will be exactly what you want for generating a mask to work with CIBlendWithMask.

Next, create a VNImageRequestHandler with the image to be processed. The handler class has a number of initializers for different types of data. In this example we will use CGImage, but you could also start with CIImage, CVPixelbuffer, Data, or others. You can also specify options for the handler. In this tutorial we will not specify any, but one of the options is to pass in a CIContext. This can help with performance as you can tell your app to do all of the processing with a single context on the GPU using Metal.

guard let originalCG = originalImage?.cgImage else { abort() }
let handler = VNImageRequestHandler(cgImage: originalCG)
try? handler.perform([segmentationRequest])
guard let maskPixelBuffer =
  segmentationRequest.results?.first?.pixelBuffer else { return }
let maskImage = CGImage.create(pixelBuffer: maskPixelBuffer)

In the code above, the originalImage (which happens to be a UIImage) gets converted to a CGImage. Then we use the image to initialize a request handler. The VNImageRequestHandler has a method .perform which takes an array of all of the Vision requests you want to use to process the image. The .perform method will not return until all of the requests in the array have been completed. Depending on how you want to structure your code, you can either provide a completion handler to use with each of the requests or just process the results in-line. In this example, we’ll process in-line.

If the segmentationRequest found any people in the image, the results array will contain a .pixelBuffer. Create the mask we need by converting the pixelBuffer into a CGImage. To convert to a CGImage, the example uses some helper methods that Matthijs Hollemans published to GitHub specifically for working with Core ML inputs and outputs.

Now that we have the mask image, use CIFilter to compose the final image. It will take a few steps. First, resize the new background, mask, and original image to the same size. Then, blend the three images.

//Convert main image to a CIImage and get the size
let mainImage = CIImage(cgImage: self.originalImage!.cgImage!)
let originalSize = mainImage.extent.size
//Convert the maskimage to CIImage and set the size
//to be the same as the original
var maskCI = CIImage(cgImage: maskImage!)
let scaleX = originalSize.width / maskCI.extent.width
let scaleY = originalSize.height / maskCI.extent.height
maskCI = maskCI.transformed(by: .init(scaleX: scaleX, y: scaleY))
//Convert the new background to a CIImage and set the size
//to be the same as the original
let backgroundUIImage = UIImage(named: "starfield")!.resized(to: originalSize)
let background = CIImage(cgImage: backgroundUIImage.cgImage!)
//Use CIBlendWithMask to combine the three images
let filter = CIFilter(name: "CIBlendWithMask")
filter?.setValue(background, forKey: kCIInputBackgroundImageKey)
filter?.setValue(mainImage, forKey: kCIInputImageKey)
filter?.setValue(maskCI, forKey: kCIInputMaskImageKey)
//Update the UI
self.filteredImageView.image = UIImage(ciImage: filter!.outputImage!)

The above code resizes the images to match the size of the original image and converts them to CIImage. Many machine learning models commonly resize inputs and outputs. For example, the output of the VNGeneratePersonSegmentationRequest using the demo image is 384x512.

Both CIImage and UIImage formats describe an image but do not always have a bitmap representation. That is why each image gets converted to a CGImage first. Though this guarantees that the demo code will work, converting formats makes the code run slower. In your app, you should experiment with different methods to get your images into CIImage format for filtering. Also, the above code uses Matthijs’ resized helper method to do some of the resizing.

With everything resized, the CIBlendWithMask filter stitches the images together. In place of the black pixels in the mask, the final image will show the background – for white pixels, the foreground. Wherever the mask image has a gray pixel, the final image will blend background and foreground.

Now let’s see how to use an additional CoreML model to process images that don’t have a person as the main subject.

Choosing a Model

Apple provides a number of pre-built models for text and image processing. The DeepLab v3 Machine Learning model can perform segmentation requests on images that have subjects like dogs. You can download it from Apple directly or also find it at the DeepLab repo on GitHub. Once you have downloaded the model, add it to your Xcode project the same as any other file. The DeepLab model will recognize people, the same as the VNPersonSegmentationRequest, but also other objects. This does come at a cost in an increased filesize for your application. You may notice that Apple provides multiple versions of the model of different file sizes. They all perform the same task, but differ in how they represent the output and a few other things.

Models on Apple’s website are packaged to work with Core ML and Xcode, so you can easily try a different model. The input and output method names will be the same. It is beyond the scope of this tutorial, yet Apple provides tutorials and example scripts on how to convert TensorFlow and other machine learning models to work with Core ML.

Core ML and Xcode

When you are working with a Core ML model, Xcode provides some convenient tools. Access them by highlighting the name of the model in the File Navigator pane of Xcode. For instance, clicking “Preview” will let you test the model with your data.

You can also use the “Predictions” tab to determine how the input needs to be formatted and what to expect for the output.

Here you can see that the DeepLabV3 model expects images to be 513 x 513 and will output a multidimensional array of integers that is 513 x 513 in size.

Finally, on this screen, you can see an entry for the “Model Class” in the headers. Double-click on the name to jump into the Swift wrapper class for the model to see how to call the model in your code and work with the inputs and outputs.

The DeepLabV3 Model

The DeepLab model input will be a color image that is 513 x 513 pixels. The Vision framework will handle resizing the input, but you can provide options on how that resize should work. The DeepLabV3 model has been trained to recognize and segment these items:

aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
dining table
dog
horse
motorbike
person
potted plant
sheep
sofa
train
tv or monitor

Anything that the model does not recognize, it will consider a background. After performing the recognition, the model returns a two-dimensional 513 x 513 array. Each entry in the array corresponds to one pixel in the original 513 x 513 input image. For instance, every pixel in the original image that shows a dog will be represented in the output by a 12 in the corresponding array entry. Any pixel that is not a recognized object will be given a 0 in the output array to represent a background.

If the model recognizes multiple objects, there will be multiple numbers in the array. If that is not what you want, you need to make changes to the array before creating the mask. For example, using an image of a dog riding a horse, some entries in the array will be 12, others will be 13, and the rest will be 0. To filter out the horse, you need to loop through the array and change any 13s to 0s.

Segmentation with DeepLab

To use the DeepLab model in your code, you again use a request to the Vision framework but this time you can specify a model.

let config = MLModelConfiguration()
var segmentationModel = try! DeepLabV3(configuration: config)
if let visionModel = try? VNCoreMLModel(for: segmentationModel.model) {
  self.request = VNCoreMLRequest(model: visionModel)
  self.request?.imageCropAndScaleOption = .scaleFill
}

In the code above, we create an instance of the DeepLab Core ML object and then initialize a generic VNCoreMLRequest with its model. The only option we set on the request is to tell it how to modify the image when it resizes it for input.

Now the code is very similar to the first example.

let cgImage = originalImage?.cgImage
let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
try? handler.perform([request])
if let observations = request.results as? [VNCoreMLFeatureValueObservation],
  let segmentationmap = observations.first?.featureValue.multiArrayValue {
  guard let maskUIImage = segmentationmap.image(min: 0.0, max: 1.0) else { return }
  applyBackgroundMask(maskUIImage)
}

Because we are using a generic VNCoreMLRequest we have to cast the results as VNCoreMLFeatureValueObservation. The DeepLab model returns a .multiArrayValue instead of a .pixelBuffer so we will again rely on the helper methods to convert it to an image. Now that the mask image has been created, applying the background is the same as in the original example.

Going Further

This tutorial focused on still images, yet both the standard Vision segmentation requests and DeepLabV3 allow processing video input as they can work with CVPixelBuffer and VNSequenceRequestHandler.
The rest of the process will be almost identical to our example. You will need to finetune the performance, or else you will experience frame drops.

Apple provides another method for identifying the background and primary subject in a still image: Portrait mode. When a device renders Portrait mode, it takes pictures with all cameras on the device and stitches them together. This way, the depth of field can change, and you can adjust what items are in focus or blurred.

If the DeepLab model does not cover your subject matter, you can train the DeepLabV3 model to recognize other objects using your own data.

However, you may consider using SDKs such as PE.SDK and VE.SDK. Enabling background removal for your users is easy — all it takes is a few lines in your configuration as described in the official PE.SDK documentation for iOS and Android, which adds a control to the photo editor to toggle the setting.

Integrate a fully customizable photo editor with Background Removal into your app with PE.SDK.

The latest VE.SDK release introduced Background Removal for stickers. This feature recognizes people in pictures and removes the background with one tap – no need for manual outlines or masks.

Let your users create beautiful visuals with Sticker Background Removal.

This feature is available on Android and iOS 15.0 and higher only. See the official documentation for Sticker Background Removal on Android or iOS.

Using an SDK will simplify and accelerate your app development, as you can save time and resources, and focus on the growth and innovation of your application instead.

Wrapping Up

In this tutorial, you saw how to use Vision and Core ML to segment an image of a person and remove the background. You also saw how to use DeepLab to work with images of non-persons.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.