Image Processing – IMG.LY Blog

Background Removal in the Browser Using ONNX Runtime with WebGPU

Daniel — Tue, 11 Jun 2024 16:26:17 GMT

TL;DR: Using ONNX Runtime with WebGPU and WebAssembly leads to 20x speedup over multi-threaded and 550x speedup over single-threaded CPU performance. Thus achieving interactive speeds for state-of-the-art background removal directly in the browser.

Removing background from an image is a typical job to be done in creative editing. We have come a long way from manually knocking out the background from an image to full automation with Neural Networks.

Most state-of-the-art background removal solutions work by offloading the task to the server with a GPU as it was simply infeasible to run the NN on the client.

However, running background removal directly in the browser offers several advantages over server-side processing:

Reduced server load and infrastructure costs by offloading heavy lifting to the client.
Enhanced scalability by distributing the workload across client devices.
Easier compliance with data protection and security policies by not transferring data across a network to a server.
Offline processing without needing a reliable internet connection.

It caters to a wide range of use cases, including but not limited to:

E-commerce applications that need to remove backgrounds from product images in real time.
Image editing applications that require background removal capabilities for enhancing user experience.
Web-based graphic design tools that aim to simplify the creative process with in-browser background removal.

In general, two factors influence the feasibility of running background removal directly on the client.

The execution performance, and
the download size of the Neural Network.

The performance or overall runtime is the major factor to be useful in interactive applications, if a user has to wait several minutes or hours for a neural network to execute, this is in many cases far too long in terms of good user experience. From experience, there are three factors to consider.

The initial first-time execution. The major factor is that neural networks come with the drawback of generally being several MB to GB in size, thus the time to download the neural network into the browser cache is considerable. In subsequent browser page reloads this has no impact anymore.
The neural network or session initialization time, cannot be cached and has to run with every reload of the page in the browser.
The neural network or session inference time, largely depends on the longest path inside the neural network and most importantly the execution time of each operator in the neural network.

Towards Real-time Background Removal in the Browser

Neural networks are commonly trained in frameworks like PyTorch, which is a neural network library for Python, as such not usable directly in the browser. The best option to run neural networks directly in the browser is converting the neural network into the Open Neural Network Exchange (ONNX) format, which is a widely supported standardized format by Microsoft used extensively in the industry.

These ONNX-formatted neural networks can then be reconverted into a platform-specific format or directly executed by a supported runtime.

The ONNX Runtime by Microsoft is a high-performance inference engine designed to run ONNX models across various platforms and languages. One notable feature is ONNX Runtime Web, which allows JavaScript developers to execute ONNX models directly in the browser. ONNX Runtime Web offers several execution providers for hardware acceleration. For instance, its WebAssembly execution provider enhances CPU execution performance using multiple Web Workers and SIMD instructions.

More importantly, starting from version 1.7.0, ONNX Runtime Web includes official support for the WebGPU execution provider. WebGPU is a modern web API that enables developers to utilize GPU power for high-performance computations, offering a significant performance boost over CPU-based in-browser machine learning. WebGPU support has been available by default since Chrome 113 and Edge 113 on Mac, Windows, and ChromeOS, and Chrome 121 on Android. For the latest browser support updates, you can track them here.

Implementation Details

For the implementation of the open-source package @imgly/background-removal-js, we started the journey with a neural network implementation in PyTorch written in Python. The network was then converted to ONNX. You can see our live showcase here.

The original model was using 32-bit floating point (fp32) precision, which is fine, but results in a file size of 168 MB after converting it to ONNX. As mentioned earlier, the size of the network has a large impact on perceived first-time execution performance as the download time tends to be longer than the execution time, but more to that later.

To reduce the size of the model, we converted the model to use fp16 (16-bit floating point) and QUINT8 (Quantized 8-bit) datatypes. Thus, effectively reducing the size to half (84MB) and a fourth (42MB) of the original size. Additionally to the download size, the operators used will be converted corresponding to the datatype and different data types and depending on the hardware may lead to speed improvements or even deteriorating due to specialized hardware being used or not being present.

Note, that while that sounds great, the conversion has a potential negative impact on the quality of the output as we are removing information in the neural networks and since we are working with images, artifacts might become visible and the quality of the resulting background mask is reduced.

Evaluation

When dealing with neural networks in the browser we can identify three different scenarios

First Use
First Run
Consecutive Runs

The first-use scenario occurs when the web application is started the first time. The neural network has to be downloaded from the server into the browser sandbox, as neural networks are several 10-1000 MB in size this is not neglectable.

The first run assumes that the neural network is already present in the browser cache, however, when the neural network ought to be used, the network has to initialize before execution.

In consecutive runs, the neural network is already in memory and the execution time is largely determined by the neural network depth and operators only.

First Use – Neural Network Download

While download or time is a large factor in the first-time execution time, this is largely dependent on the available network bandwidth, as such it is not part of the evaluation but in order to get an idea here are the expected download times for the fp32 (168MB) and fp16 (84MB) neural network subject to various common network bandwidth:

Network Bandwidth	Filesize	Download Time
10 Mbps	84 MB	67s s
100 Mbps	84 MB	6.7 s
1 Gbps	84 MB	0.67 s
10 Gbps	84 MB	0.067

Network Bandwidth	Filesize	Download Time
10 Mbps	168 MB	134.4 s
100 Mbps	168 MB	13.4 s
1 Gbps	168 MB	1.34 s
10 Gbps	168 MB	0.13 s

Based on the assumption that the median download bandwidth is ~100 Mbps, we see that it’s in the 5-15 second range.

First Run – Neural Network Initialization / Compilation

Before the neural network can be executed it has to be initialized. Initialization includes several execution provider-specific steps. Most prominent are the time to upload or convert the data to the execution provider and execution provider-specific ONNX graph optimization passes. This all adds to the first run experience. We have evaluated the average session initialization time for the CPU (WASM) and WebGPU provider on a MacBook Pro 13” from 2024 with an Apple M3 Max 16 cores to get an idea of the general impact on first run execution time:

Device	Datatype	Session Initialization Time
CPU (WASM)	fp32	~320ms
CPU (WASM)	fp16	~320ms
WebGPU	fp32	~400ms
WebGPU	fp16	~200ms

Note, that session initialization is not negligible and adds significant runtime overhead and might be subject to additional optimization possibilities like caching the optimized model.

Consecutive Runs – Neural Network Execution

Independent of the download and initialization time, we evaluated different execution providers on a MacBook Pro 13” from 2024 with an Apple M3 Max 16 cores. While this is top-end consumer hardware, the general trends will probably apply to various hardware configurations. As a reference, we choose the single thread performance with the neural network running on the CPU with 1 worker thread and SIMD disabled.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
CPU (WASM)	No	1	fp32	~53000 ms	1.0 x
CPU (WASM)	No	16	fp32	~6300 ms	8.4 x
CPU (WASM)	Yes	1	fp32	~15000 ms	3.5 x
CPU (WASM)	Yes	16	fp32	~2000 ms	26.5 x

The data above reveals that running the neural network without any acceleration such as SIMD and threading in the browser results in almost 53s runtime, and as such is for most interactive use cases too slow to use. Due to the optimizations of the ONNX Runtime that uses SIMD and threads, we can achieve an overall speedup of roughly ~26 times compared to the baseline performance. Thus decreasing the session runtime to around 2s and making it usable for interactive applications.

As mentioned before, we neglected the download time and size of the network. Leveraging the fp16 model compression, we re-ran the benchmarks and got similar results as before, but with half the bandwidth required to download the network for the first time application. The general visual performance of the fp16 and fp32 is similar or visually not perceivable. The quint8 model led to artifacts and as such unusable for visual processing, thus we excluded from the following benchmark.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
CPU (WASM)	No	1	fp16	~55000 ms	1.0 x
CPU (WASM)	No	16	fp16	~7300 ms	7.2 x
CPU (WASM)	Yes	8	fp16	~15000 ms	3.5 x
CPU (WASM)	Yes	16	fp16	~2300 ms	23.9 x

The results are a little worse than the fp32 version, which might be because the fp16 datatype has no direct specialized hardware in modern CPUs like the M3 Max CPU, and as such additional fp16 to fp32 conversion operations have to be performed.

Our final benchmark measures the WebGPU performance, which led to the following impressive results. To have a fair comparison and understanding of the impact of the WebGPU technology, we compare it with the best CPU version with 16 threads and SIMD enabled with the fp32 model, which was around ~2 s.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
WebGPU	Not applicable	Not applicable	fp32	~120ms	16.6 x
WebGPU	Not applicable	Not applicable	fp16	~100ms	20.0 x

The WebGPU performance varies around 16 to 20 x improvements over the best CPU execution time. The GPU has specialized hardware for fp16 instructions, as such the model performs better than the fp32 model. Also, note that the session initialization time is also reduced due to half the required bandwidth to upload the network data to the GPU.

Therefore, the first run of the network will take ~300 ms and consecutive runs will be ~100 ms, leading to near real-time performance in the browser.

Note, that the WebGPU performance is an astonishing 550 times faster than the single thread, with no SIMD performance.

Try the live implementation in our background removal showcase.

Conclusion

WebGPU is a major leap in establishing the browser as a factor to be reckoned with as a true Application platform. With IMG.LY, we are striving to leverage modern technology to make design tools accessible, this includes on-device execution, on-premise, but also on-cloud execution of design tools leveraging neural networks.

As a next step, we will port background removal to all our supported platforms, ONNX Runtime seems the best choice, as it is already available for all the potential platforms we support.

Furthermore, we are evaluating the feasibility of in-browser background removal for videos to be included with our video-editing suits.

Try our tools today to see how we help bring unique creative editing experiences to any application. Or contact us to discuss your project.

How To Add a Sticker to a Texture With Three.js

Antonello — Fri, 25 Feb 2022 15:10:58 GMT

Disclaimer: This article has been updated in June 2023 to reflect the changes introduced by Three.js r150

In this article, you will learn to add a sticker to a texture with WebGL. In detail, you will see how to use the Three.js JavaScript library to add an overlay image to a texture. Check out our guide to explore other image editing operations, such as resizing an image in React.

Dealing with 3D graphics used to be a challenging, complex, and time-consuming task. Thanks to libraries like three.js, it has become more accessible than ever. Follow this easy step-by-step how-to and achieve the following result:

Prerequisites

Here are all the prerequisites you need to meet to follow this tutorial and build the demo application shown above:

Explore the official documentation page to learn about how to install three.js.

What is Three.js?

As stated in the GitHub page of the project, the goal of the Three.js project is to create a lightweight 3D JavaScript library with a very low level of complexity. In other words, Three.js aims to make 3D graphics more accessible, especially considering that Three.js only requires a browser to run.

To be more specific, it allows you to create a 3D graphics project where you can modify elements in your browser and see visual feedback. The library is currently built on top of a WebGL renderer, but it also supports WebGPU, SVG, and CSS3D renderers.

If you are not familiar with these concepts, WebGL stands for “Web Graphics Library” and is a JavaScript API for rendering high-performance interactive 3D and 2D graphics. You can use it in many web browsers without external plugins or libraries.

As you can verify from the MDN compatibility page, browsers currently support the most popular WebGL features. However, keep in mind that the user’s device must meet hardware requirements for some features to work.

WebGL is undoubtedly a powerful tool but requires a lot of skill and involves a lot of boilerplate code. Here is where Three.js comes into play, making 3D development much easier.

Now, let’s see it in action!

Adding an Overlay Image to a Texture in WebGL

Clone the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/how-to-add-a-sticker-with-threejs-imgly

Additionally, try the demo application by launchingthe command below:

npm install
npm run start

Then, visit http://localhost:5000 in the browser.

Now that you know what the demo app looks like, keep following this tutorial and learn how to build it.

Specifically, this is the entire JavaScript code required to achieve the goal:

import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

// set up the scene, camera, and renderer
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(
  40,
  window.innerWidth / window.innerHeight,
  1,
  1000
);
camera.position.set(150, 100, 150);
new OrbitControls(camera, document.body);

// initialize the WebGL renderer
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// add light to the scene to make the texture visible
const light = new THREE.AmbientLight(0xffffff);
scene.add(light);

// add the axes helper indicator to the scene
scene.add(new THREE.AxesHelper(20));

// load the base texture
const texture = new THREE.TextureLoader().load(
  'https://threejs.org/examples/textures/hardwood2_diffuse.jpg'
);

// create a new buffer geometry and place it
// to the horizontal plane
const textureGeometry = new THREE.PlaneGeometry(50, 50);
textureGeometry.rotateX(-90 * THREE.MathUtils.DEG2RAD);
// translate the plane to the center of the scene
textureGeometry.translate(25, 0, 25);

// create a MeshBasicMaterial with the base texture
const textureMaterial = new THREE.MeshBasicMaterial({ map: texture });

// create a mesh using the geometry and material
// and add it to the scene
const textureMesh = new THREE.Mesh(textureGeometry, textureMaterial);
scene.add(textureMesh);

// repeat the same logic to place the sticker image
// over the texture
const sticker = new THREE.TextureLoader().load(
  'https://i.imgur.com/IYh17Rv.png'
);

const stickerGeometry = new THREE.PlaneGeometry(20, 20);
stickerGeometry.rotateX(-90 * THREE.MathUtils.DEG2RAD);
stickerGeometry.translate(25, 0, 25);
const stickerMaterial = new THREE.MeshBasicMaterial({
  map: sticker,
  transparent: true,
});
// create a mesh using the image geometry and material
const imageMesh = new THREE.Mesh(stickerGeometry, stickerMaterial);
scene.add(imageMesh);

// adjust the position and rotation of the sticker mesh
// by setting the y-coordinate to 1
// in order for the sticker to rest on the texture
imageMesh.position.set(0, 1, 0);

// render the scene
function launchThreeJs() {
  requestAnimationFrame(launchThreeJs);
  renderer.render(scene, camera);
}

launchThreeJs();

First, the code imports the necessary components from the Three.js library. Next, it initialize a Scene and set up a PerspectiveCamera with OrbitsControl, allowing the user to move around. Then, it creates a WebGL renderer and sets its size to match the window’s dimensions. The scene is lit by an AmbientLight instance, which illuminates all objects in the scene equally.

The snippet loads a base texture using TextureLoader from the given URL, creates a PlaneGeometry, defines a mesh with some geometries, and then add the resulting element to the scene. This logic will take care of rendering the texture image in the scene.

To add the sticker to the texture, the same procude gets repeated. This time, Three.js is instructed to load a sticker image and position it over the original texture. Note that the position of the sticker mesh is adjusted by setting its y-coordinate to 1 to ensure the new image rests on top of the old one.

Et voilà! You just learned how to add a sticker to a WebGL texture with three.js!

Adding a Sticker With `PHOTOEDITORSDK`

PhotoEditorSDK uses WebGL as the main renderer. You can effortlessly add a sticker to your textures without writing any line of code. WebGL will be used behind the scene. Consequently, you do not need to know WebGL or code in GLSL to harness its power.

Adding a sticker to an image only takes a few seconds here. Just jump to this article from the official documentation and learn how to get started with PhotoEditorSDK in Vanilla JavaScript. Then, upload your texture image and a sticker with the Sticker feature. As shown below, you can position a sticker on top of the original image with a point-and-click UI:

Check out this feature on the PhotoEditorSDK demo page.

Conclusion

Today we learned how to add an overlay image to a WebGL texture in JavaScript with three.js. This tool makes it easier to deal with WebGL and 3D graphics. In particular, by using one of the many plugins supporting the Three.js project, we were able to add a sticker to a texture with just a bunch of lines of code.

3D graphics remains a complex topic, especially when it comes to developing in the OpenGL Shading Language. Three.js tries to hide this encumbrance as much as possible, but sometimes you still require it. If you want to avoid using WebGL directly to achieve your goal, consider an all-in-one and easy-to-use solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

On the Democratization of Design

Daniel — Mon, 24 Jun 2019 00:00:00 GMT

Over the last century we have experienced the democratization of various domains of our lives:

Key drivers have been technological advancements and lower barriers to leveraging those technologies. In turn, technology became more accessible to more people resulting in products, services, and tools that improve our lives every day.

“Computers are a bicycle for the mind” — Steve Jobs

Tools, such as computers, allow us to finish more tasks in a shorter amount of time and with fewer resources. Even more importantly, said tools also rid us from boring and repetitive work, thus enabling us to shift our attention to exciting and more creative tasks, paving the way for us to experiment and create rather than repeat.

Evolve Design with better Tools

Proper visual design attracts, engages, and persuades; it helps convey stories and facilitates communication.

Nowadays, design in all its facets like advertising, social stories, product presentations, or merchandise is omnipresent in our daily lives. As there is no arguing that appealing and beautiful design has an impact on us and our surrounding, consequently there is also a growing demand for good design and easy-to-use tools.

“Use a picture. It’s worth a thousand words!” — Arthur Brisbamne

Although the importance of design is unquestioned, creating appealing designs remains hard for most of us, and there are many reasons for that.

First of all, design tools are still challenging to master, and while they empower professional designers to create stunning assets, they remain a closed book for most people.

Secondly, even if we leave most of the creational part to professionals, the average user is still unable to use existing designs as a basis to create new assets or refine and update small details. Even for the slightest changes, one has to get familiar with the inner workings of sophisticated design tools.

Also, when it comes to collaboration, the process with most tools is far from frictionless, to put it mildly. Still, most designs get shared as Photoshop, Sketch, or PowerPoint files, and there is no tool or platform that grants unhindered access for multiple stakeholders with varying degrees of experience with design solutions.

As today’s design processes regularly involve multiple stakeholders like designers, marketers, copywriters, or product engineers, one has to acknowledge that prevalent design solutions simply lack features and workflows that reflect daily routines.

Make Design Accessible through Technology

As we pointed out, the overall access to good visual design is still very limited, but with the growing need for compelling creatives in virtually every aspect of the digital and non-digital realm this shouldn’t be and it doesn’t have to. We believe in the democratization of design.

Better tools enable the democratization of design!

We believe that:

We need to overhaul design processes and methods to cope with the ever growing design needs and the ever-rising number of stakeholders.
Achieving elegant design must be made simple.
Design can be made accessible through technology.

The question remains: How do we achieve that?

We came to the conclusion that there are four key elements design tools should have that will make design more accessible:

Collaboration,
building blocks of intelligent design components,
automation of tedious design tasks, and
bringing together Editing, Compositing, and Layouting.

Collaboration

We hold as an opinion that design tools have to facilitate the collaboration between professionals and non-professionals, thus allowing an effective and frictionless design creation and adaption process. This includes direct collaboration on a design as well as the exchange of whole design templates, design components, and presets.

While some solutions already offer templates for content creation, we think that design components as a more modular approach are going to be a necessity to simplify the creation of new complex assets for everyone.

Building Blocks of Intelligent Design Components

If Lego™ taught us anything, then it is that creativity can flow when you base your work on a solid foundation.

That said, one has to admit that while the basic elements of design such as text, shapes, or images can be creatively combined in millions of ways to create stunning designs, they also can be used in a trillion ways to create underwhelming output.

Similar to modern UI design, complex building blocks will play a central role in the visual design process, allowing creators to be more efficient by removing repetitive tasks from the workflow and also allowing average users to create and adopt complex designs quickly. Intelligent design components will shift the focus to the composition by guiding the user in carefully chosen constraints.

Automation of Tedious Design Tasks

Speaking of intelligent building blocks, we believe that said blocks must not be static but adaptive to fit into an effective and frictionless design process. That may include the automation of certain design tasks like text layout or even abstracting away the complexity of image editing.

We envision that with the help of technology and specifically artificial intelligence, recurring and taunting tasks like for example, the separation of a picture into fore- and background can become totally interaction- and hassle-free. Even automatically choosing the right adjustments based on the image content will be possible.

Altogether, intelligent design components are a necessity and will be the next step in design automation, fostering a flexible and fail-safe adaption and creation process and thus ensuring that even without professional knowledge designs will be created faster and look better.

Bringing together Editing, Composition, and Layouting

Photo editing and layouting are both central to the design process. That is why, in a holistic approach, combining both will yield exciting new possibilities, improved visual consistency, and a reduced amount of friction while going back and forth between layout and photo editing.

Closing words

To sum it up: To democratize design, we need intelligent design tools and ecosystems that allow for the cooperation between all stakeholders, as well as a community that is open to share and learn from one another by building and sharing reusable and intelligent design components.

Lightweight Image comparison with Rembrandt.JS

Felix — Wed, 08 Mar 2017 00:00:00 GMT

Rembrandt.JS is a lightweight, client- and server-side image comparison library that we developed for our internal tests for the PhotoEditor SDK. Since we’re adherents of test-driven development, we devote a lot of time and effort to writing robust tests and finding the right tools. A lot can go wrong in the process of refactoring and adding new features to our product. Since we want to make sure that our SDK performs at a top level 100% of the time, solid unit tests are crucial to our work. However, when it came to image comparison, we were not happy with the existing options. We tried to work with Blink-Diff from Yahoo, but since the tool isn’t maintained anymore and furthermore has a dependency on another unmaintained library, we chose to develop our own solution. Rembrandt.JS has no dependencies because it works client-side with the Canvas API of the browser and server-side with Node-Canvas.

The primary function of Rembrandt is quite simple; it compares two pictures. One image is the expected output of our SDK and the other one is the actual output of the PhotoEditor. After the comparison, Rembrandt outputs a numeric value between 0 and 1 indicating the percentual error rate. With an additional feature of Rembrandt, it is also possible to highlight the incorrect pixels on the image which helps to see instantly where the errors occur. Furthermore, Rembrandt.JS can compensate defective pixels that originate through compression artifacts. By comparing every pixel with its adjacent pixels with an adjustable tolerance, Rembrandt evaluates whether there’s a discrepancy or not. Said feature was critical to us since due to the sheer amount of features and filters in PhotoEditor SDK we’re always testing on a large scale. If we would’ve been using loss-free test material (like .png for example) the amount of disk space that the images would’ve been consuming would have been quite noticeable.

Rembrandt.JS could also be leveraged for UI tests. With screenshots of both the design and the actual website or app, you could verify whether every object loads and behaves correctly. With the rendering option, you could quickly detect smallest discrepancies in even large sites or apps. Or, you could solve every spot the difference game within seconds.

If Rembrandt is something potentially useful for your project, check out the Github repositories for Javascript, iOS and Android.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Image Processing – IMG.LY Blog

Background Removal in the Browser Using ONNX Runtime with WebGPU

Towards Real-time Background Removal in the Browser

Implementation Details

Evaluation

First Use – Neural Network Download

First Run – Neural Network Initialization / Compilation

Consecutive Runs – Neural Network Execution

Conclusion

How To Add a Sticker to a Texture With Three.js

Prerequisites

What is Three.js?

Adding an Overlay Image to a Texture in WebGL

Adding a Sticker With PHOTOEDITORSDK

Conclusion

On the Democratization of Design

Evolve Design with better Tools

Make Design Accessible through Technology

Closing words

Lightweight Image comparison with Rembrandt.JS

Adding a Sticker With `PHOTOEDITORSDK`