Plugin – IMG.LY Blog

Plugins Part 2: Custom Panels

Jan — Wed, 23 Oct 2024 12:53:27 GMT

In Part 1 of our series, we introduced a plugin system for customizing UI elements and quick actions within the CreativeEditor SDK.

Now, in Part 2, we’re introducing Customizable Panels, a new feature that gives developers even more control over the editor. These panels can handle tasks like guiding users through complex workflows, interacting with external APIs or gathering detailed user input—allowing for anything from simple tweaks to fully custom workflows.

Let’s take a quick look at where panel plugins fit within the CE.SDK Web UI. To dive in instead, consult the documentation or explore one of the examples below.

Extension Points in the UI

Before customizing the UI, it’s important to understand the user experience that makes the editor intuitive for creating and editing designs. Below are the key areas of the editor UI where your plugins can be integrated:

The quick actions we introduced in part one are used to implement simple one-click actions to edit canvas elements, primarily inside the inspector bar and canvas menus. This release opens up the panel as additional UI element to facilitate more involved features requiring UI interactions beyond a simple click.

New Possibilities with Panels

Panels serve as customizable UI elements that provide plenty of room for your plugin’s unique requirements. Here’s how developers can utilize them:

Guide Users Through Workflows
Implement panels to assist users in navigating workflows or controlling specific plugin settings, tailoring the editor experience to meet individual needs.

Complex Interaction with External AI or Services
While it’s possible to connect to third-party service using quick actions, some use cases require more user interaction. To support these cases you can now create panel UIs that connect with third-party APIs to gather data or trigger processes, such as AI-driven design tools or content generation.

Collect Detailed User Input
Use panels to collect more complex input from users, such as advanced export options, template customization, theming or batching design adjustments.

These new capabilities round out our goal of empowering developers to shape CE.SDK themselves, tailoring the user experience to fit perfectly with their particular use case.

Example Plugins

To demonstrate the potential of panels, we’ve developed three example plugins that showcase different approaches to extending the CreativeEditor SDK’s functionality.

Plugins can be grouped into two categories:

Functionality-Enhancing Plugins: These integrate external services, assets, or custom code. For example, the QR embed plugin adds a feature by interacting with a third-party API.

Custom Editor Behavior Plugins: These plugins interact directly with the CE.SDK Engine API, enhancing the SDK’s existing functionality by introducing new modes of interaction or extending the user interface.

QR Code Embed Plugin

The first category of plugins is exemplified by this QR plugin, which add the ability to embed a QR code directly in the canvas. The plugin interacts with a third-party QR code generator API and displays a panel where users can enter a URL and customize the color of the QR code.

The next two examples illustrate plugins from the second category, which interface with the Engine API to enhance CE.SDK functionality:

Export Options Panel

This showcase provides a reference implementation for a panel that allows users to configure export options including export formats and file types, such as PNG, JPEG, or PDF, which pages to export and at what quality and resolution. A corresponding showcase is also available for video scenes, allowing users to control resolution and quality of the exported video file.

Template Adaptation UI

In this example, we add a panel that collects user input via a form, which is then used to dynamically adapt a design template. Users can input text, select colors, or upload images, and the template is modified based on these choices. This demonstrates how panels can streamline workflows, making it easier for users to customize templates to fit their specific needs.

Outlook

With the release of panels, we are advancing our mission to empower developers to fully customize and extend the CreativeEditor SDK. These new features for creating sophisticated, interactive plugins will unlock exciting opportunities for innovation, whether integrating external services, designing dynamic user interfaces, or automating complex workflows.

We encourage you to explore the power of panels in your own projects. Visit our plugin documentation to get started, and let us know what you’re building! As always, we’re here to support your journey in redefining creative editing.

Stay tuned for more updates, and feel free to contact us if you have any questions or need assistance.

Over 3,000 creative professionals gain early access to new features and updates—don’t miss out, and subscribe to our newsletter.

Plugin Release Part 1: Customizing the UI & Quick Actions

Jan — Tue, 13 Aug 2024 07:16:36 GMT

At IMG.LY we are driven by a commitment to empower creativity for end users worldwide, we do so by giving our customers the tools to build the ideal creative experience for their customers. For this reason, we are shaping our CreativeEditor SDK to be built for modification and extension. IMG.LY’s customers are not limited by our roadmap. To the contrary, plugins allow customers to not only extend editor functionality, but to also benefit from third-party plugins and open source development based on IMG.LY’s CE.SDK. This new plugin system is designed to give our customers unparalleled autonomy, allowing them to tailor the user experience to their specific needs and accelerate product innovation.

This first release of CE.SDK’s plugin API enables you to customize the UI at certain extension points and locations, as well as change the order and location of UI elements. At these points you will not only be able to change the appearance of existing features but add entirely custom features in the form of quick actions - simple one-click actions to edit canvas elements like removing an image background or vectorizing a graphic.

Extension Points in the UI

Before customizing the UI, it is crucial to understand the user experience we have designed to facilitate intuitive manipulation and creation of a range of designs. The following key locations in the editor UI are extension points for your plugins. We provide a visual example for the type of customization possible at each location:

Canvas: The core interaction area for design content, controlled by the Creative Engine.

Dock: The primary entry point for user interactions not directly related to the selected block. It is primarily, though not exclusively, used to open panels with asset libraries or add elements to the canvas. This release allows you to customize the buttons in the dock. The customization of panels will be possible following the next release.

Canvas Menu: Provides block-specific settings and actions such as deletion or duplication. Here, we customized the menu by adding a background removal action and actions for layer management.

Inspector Bar: Main location for block-specific functionality. Any action or setting available for the currently selected block that does not appear in the canvas menu will be shown here.

Navigation Bar: For actions affecting browser navigation and global scene effects such as zoom or undo/redo. The customization below shows more advanced navigation patterns for multi-steps designs that can be implemented via plugins.

Canvas Bar: For actions affecting the canvas or scene as a whole, such as adding pages. This is an alternative place for actions such as zoom or redo/undo.

Customizing UI Components

You can control the order and layout of components within these locations using Order APIs, which allow you to add, remove, or reorder components. We also provide special layout components like separators and spacers to help organize your custom components effectively.

Registering New Components

Beyond predefined components, you can register your own custom components using builder elements such as buttons, dropdowns, and inputs. This enables you to integrate unique behaviors and logic into the web editor seamlessly. This is how you would implement custom “quick actions” such as background removal

Feature-level Control

In addition to defining custom components, you can use the new feature API to control components on the feature level based on the current context. For instance, you might want to hide the duplicate and delete button for a new “Avatar” block type, you can use the Feature API to selectively disable these components for that specific block.

Examples: Background Removal and Cut-out lines

A straightforward example of a plugin that extends the default editor UI with a quick action that acts on an element is our image background removal plugin.

Here, we customized the canvas menu of image elements to prominently display a “BG Removal” button acting on the currently selected block. To not place so much emphasis on background removal, this feature could have been placed in the inspector bar instead.

The next example is an outlook on what is to come.

Sometimes a plugin manipulates an element but requires additional inputs from the users so that a single button click does not suffice. Such is the case for our cut-out lines plugin, allowing users to create custom cut-out or perforated line shapes on the canvas. The button is placed in the dock and a panel opens on click where the user can select shapes for the cut-out or opt for a dynamic shape that is generated from the selection.

Get Started

Start developing your own plugins by visiting our plugin documentation. We’re excited to see what you’ll build!

What’s Next

The next plugin release will allow you to customize the UI with panels, enabling the extension of increasingly complex behaviors that require additional user input or interaction with external services.

Stay tuned for more updates, and don’t hesitate to reach out if you have any questions or need assistance. Together, we are redefining the future of creative editing.

Thank you for reading. Join 3,000+ creative professionals—subscribe to our newsletter for updates on new features, plugins, and early access.

Background Removal in the Browser Using ONNX Runtime with WebGPU

Daniel — Tue, 11 Jun 2024 16:26:17 GMT

TL;DR: Using ONNX Runtime with WebGPU and WebAssembly leads to 20x speedup over multi-threaded and 550x speedup over single-threaded CPU performance. Thus achieving interactive speeds for state-of-the-art background removal directly in the browser.

Removing background from an image is a typical job to be done in creative editing. We have come a long way from manually knocking out the background from an image to full automation with Neural Networks.

Most state-of-the-art background removal solutions work by offloading the task to the server with a GPU as it was simply infeasible to run the NN on the client.

However, running background removal directly in the browser offers several advantages over server-side processing:

Reduced server load and infrastructure costs by offloading heavy lifting to the client.
Enhanced scalability by distributing the workload across client devices.
Easier compliance with data protection and security policies by not transferring data across a network to a server.
Offline processing without needing a reliable internet connection.

It caters to a wide range of use cases, including but not limited to:

E-commerce applications that need to remove backgrounds from product images in real time.
Image editing applications that require background removal capabilities for enhancing user experience.
Web-based graphic design tools that aim to simplify the creative process with in-browser background removal.

In general, two factors influence the feasibility of running background removal directly on the client.

The execution performance, and
the download size of the Neural Network.

The performance or overall runtime is the major factor to be useful in interactive applications, if a user has to wait several minutes or hours for a neural network to execute, this is in many cases far too long in terms of good user experience. From experience, there are three factors to consider.

The initial first-time execution. The major factor is that neural networks come with the drawback of generally being several MB to GB in size, thus the time to download the neural network into the browser cache is considerable. In subsequent browser page reloads this has no impact anymore.
The neural network or session initialization time, cannot be cached and has to run with every reload of the page in the browser.
The neural network or session inference time, largely depends on the longest path inside the neural network and most importantly the execution time of each operator in the neural network.

Towards Real-time Background Removal in the Browser

Neural networks are commonly trained in frameworks like PyTorch, which is a neural network library for Python, as such not usable directly in the browser. The best option to run neural networks directly in the browser is converting the neural network into the Open Neural Network Exchange (ONNX) format, which is a widely supported standardized format by Microsoft used extensively in the industry.

These ONNX-formatted neural networks can then be reconverted into a platform-specific format or directly executed by a supported runtime.

The ONNX Runtime by Microsoft is a high-performance inference engine designed to run ONNX models across various platforms and languages. One notable feature is ONNX Runtime Web, which allows JavaScript developers to execute ONNX models directly in the browser. ONNX Runtime Web offers several execution providers for hardware acceleration. For instance, its WebAssembly execution provider enhances CPU execution performance using multiple Web Workers and SIMD instructions.

More importantly, starting from version 1.7.0, ONNX Runtime Web includes official support for the WebGPU execution provider. WebGPU is a modern web API that enables developers to utilize GPU power for high-performance computations, offering a significant performance boost over CPU-based in-browser machine learning. WebGPU support has been available by default since Chrome 113 and Edge 113 on Mac, Windows, and ChromeOS, and Chrome 121 on Android. For the latest browser support updates, you can track them here.

Implementation Details

For the implementation of the open-source package @imgly/background-removal-js, we started the journey with a neural network implementation in PyTorch written in Python. The network was then converted to ONNX. You can see our live showcase here.

The original model was using 32-bit floating point (fp32) precision, which is fine, but results in a file size of 168 MB after converting it to ONNX. As mentioned earlier, the size of the network has a large impact on perceived first-time execution performance as the download time tends to be longer than the execution time, but more to that later.

To reduce the size of the model, we converted the model to use fp16 (16-bit floating point) and QUINT8 (Quantized 8-bit) datatypes. Thus, effectively reducing the size to half (84MB) and a fourth (42MB) of the original size. Additionally to the download size, the operators used will be converted corresponding to the datatype and different data types and depending on the hardware may lead to speed improvements or even deteriorating due to specialized hardware being used or not being present.

Note, that while that sounds great, the conversion has a potential negative impact on the quality of the output as we are removing information in the neural networks and since we are working with images, artifacts might become visible and the quality of the resulting background mask is reduced.

Evaluation

When dealing with neural networks in the browser we can identify three different scenarios

First Use
First Run
Consecutive Runs

The first-use scenario occurs when the web application is started the first time. The neural network has to be downloaded from the server into the browser sandbox, as neural networks are several 10-1000 MB in size this is not neglectable.

The first run assumes that the neural network is already present in the browser cache, however, when the neural network ought to be used, the network has to initialize before execution.

In consecutive runs, the neural network is already in memory and the execution time is largely determined by the neural network depth and operators only.

First Use – Neural Network Download

While download or time is a large factor in the first-time execution time, this is largely dependent on the available network bandwidth, as such it is not part of the evaluation but in order to get an idea here are the expected download times for the fp32 (168MB) and fp16 (84MB) neural network subject to various common network bandwidth:

Network Bandwidth	Filesize	Download Time
10 Mbps	84 MB	67s s
100 Mbps	84 MB	6.7 s
1 Gbps	84 MB	0.67 s
10 Gbps	84 MB	0.067

Network Bandwidth	Filesize	Download Time
10 Mbps	168 MB	134.4 s
100 Mbps	168 MB	13.4 s
1 Gbps	168 MB	1.34 s
10 Gbps	168 MB	0.13 s

Based on the assumption that the median download bandwidth is ~100 Mbps, we see that it’s in the 5-15 second range.

First Run – Neural Network Initialization / Compilation

Before the neural network can be executed it has to be initialized. Initialization includes several execution provider-specific steps. Most prominent are the time to upload or convert the data to the execution provider and execution provider-specific ONNX graph optimization passes. This all adds to the first run experience. We have evaluated the average session initialization time for the CPU (WASM) and WebGPU provider on a MacBook Pro 13” from 2024 with an Apple M3 Max 16 cores to get an idea of the general impact on first run execution time:

Device	Datatype	Session Initialization Time
CPU (WASM)	fp32	~320ms
CPU (WASM)	fp16	~320ms
WebGPU	fp32	~400ms
WebGPU	fp16	~200ms

Note, that session initialization is not negligible and adds significant runtime overhead and might be subject to additional optimization possibilities like caching the optimized model.

Consecutive Runs – Neural Network Execution

Independent of the download and initialization time, we evaluated different execution providers on a MacBook Pro 13” from 2024 with an Apple M3 Max 16 cores. While this is top-end consumer hardware, the general trends will probably apply to various hardware configurations. As a reference, we choose the single thread performance with the neural network running on the CPU with 1 worker thread and SIMD disabled.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
CPU (WASM)	No	1	fp32	~53000 ms	1.0 x
CPU (WASM)	No	16	fp32	~6300 ms	8.4 x
CPU (WASM)	Yes	1	fp32	~15000 ms	3.5 x
CPU (WASM)	Yes	16	fp32	~2000 ms	26.5 x

The data above reveals that running the neural network without any acceleration such as SIMD and threading in the browser results in almost 53s runtime, and as such is for most interactive use cases too slow to use. Due to the optimizations of the ONNX Runtime that uses SIMD and threads, we can achieve an overall speedup of roughly ~26 times compared to the baseline performance. Thus decreasing the session runtime to around 2s and making it usable for interactive applications.

As mentioned before, we neglected the download time and size of the network. Leveraging the fp16 model compression, we re-ran the benchmarks and got similar results as before, but with half the bandwidth required to download the network for the first time application. The general visual performance of the fp16 and fp32 is similar or visually not perceivable. The quint8 model led to artifacts and as such unusable for visual processing, thus we excluded from the following benchmark.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
CPU (WASM)	No	1	fp16	~55000 ms	1.0 x
CPU (WASM)	No	16	fp16	~7300 ms	7.2 x
CPU (WASM)	Yes	8	fp16	~15000 ms	3.5 x
CPU (WASM)	Yes	16	fp16	~2300 ms	23.9 x

The results are a little worse than the fp32 version, which might be because the fp16 datatype has no direct specialized hardware in modern CPUs like the M3 Max CPU, and as such additional fp16 to fp32 conversion operations have to be performed.

Our final benchmark measures the WebGPU performance, which led to the following impressive results. To have a fair comparison and understanding of the impact of the WebGPU technology, we compare it with the best CPU version with 16 threads and SIMD enabled with the fp32 model, which was around ~2 s.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
WebGPU	Not applicable	Not applicable	fp32	~120ms	16.6 x
WebGPU	Not applicable	Not applicable	fp16	~100ms	20.0 x

The WebGPU performance varies around 16 to 20 x improvements over the best CPU execution time. The GPU has specialized hardware for fp16 instructions, as such the model performs better than the fp32 model. Also, note that the session initialization time is also reduced due to half the required bandwidth to upload the network data to the GPU.

Therefore, the first run of the network will take ~300 ms and consecutive runs will be ~100 ms, leading to near real-time performance in the browser.

Note, that the WebGPU performance is an astonishing 550 times faster than the single thread, with no SIMD performance.

Try the live implementation in our background removal showcase.

Conclusion

WebGPU is a major leap in establishing the browser as a factor to be reckoned with as a true Application platform. With IMG.LY, we are striving to leverage modern technology to make design tools accessible, this includes on-device execution, on-premise, but also on-cloud execution of design tools leveraging neural networks.

As a next step, we will port background removal to all our supported platforms, ONNX Runtime seems the best choice, as it is already available for all the potential platforms we support.

Furthermore, we are evaluating the feasibility of in-browser background removal for videos to be included with our video-editing suits.

Try our tools today to see how we help bring unique creative editing experiences to any application. Or contact us to discuss your project.

Unleash Creativity with CE.SDK’s New Plugin System

Daniel — Mon, 03 Jun 2024 08:42:25 GMT

At IMG.LY, we have always believed that a superb design editor should be effortlessly customizable and extensible. We are thrilled to roll out a brand-new Plugin system for CE.SDK in the upcoming months—to take creative editing and feature development to the next level.

Starting with one-click features like background removal or vectorizers, smart design tools like QR codes or subtitle generators, and deeply interactive features like generative AI for text and images; all these tools can be used or built by customers soon.

Additionally, our upcoming Plugin system will bring you unparalleled autonomy by making feature development for our SDK accessible and offering extensive options to reconfigure our editor’s UI.

Start exploring our Plugin System rollout now to immediately benefit from upcoming features.

Built for Modification

Let’s dive deep into some of the opportunities CE.SDK plugins help unlock.

Unlocking the AI Revolution

While the AI transformation is already fully underway, much of the tech is still not very accessible to product builders, often requiring deep technical knowledge to get started. At the same time, AI features become significantly more valuable when integrated with other editing functionalities in workflows or automation. With our plugins, we aim to make it effortless to leverage innovation and put it to use.

Boosting Customer Autonomy

Key to our success is providing maximum flexibility and autonomy to our customers about product decisions. Ultimately, you shouldn’t depend on our product roadmap; rather, you should be able to add features when you like. While our SDKs are already highly configurable, plugins allow tailoring the whole user experience not only on a look & feel level but through custom functionalities and editing experiences.

Accelerate Product Expansion

Many ecosystems witnessed explosive growth in added value to the user after releasing plugin mechanisms. Currently, only IMG.LY core developers can contribute to the SDK. We have started to extend this to solution engineers and even designers on our team who don’t have much knowledge of the inner workings of the SDK. Ultimately, we will push this more and more into a community of contributors, making the community’s innovation accessible to everyone.

Key Concepts

Three important concepts have driven the development of our Plugins:

Customizable Menu Bars
We are extending our API so that it allows easy hooking into various parts of the UI. Our editor has key components like the inspector, toolbar, and on-canvas menu. These are now all accessible through an API, so you can hook your feature anywhere in the editor.

UI Building Blocks to Provide Consistency
To reach a high level of consistency and speed up time for development, we will be providing out-of-the-box UI components such as buttons, sliders, text inputs, etc.

Escape Hatches
From experience, we know that sometimes unique functionality needs unique solutions, so we have added escape hatches to add custom elements via HTML whenever needed.

What Can You Build with our Plugin System?

Let’s explore some potential use cases of the plugin system.

Custom Actions

Adding custom actions is a great option to make simple third-party APIs accessible within the editor. This can be one-click edits such as background removal, vectorizers, or auto-enhancement for images. You can also add custom actions for text in combination with Large Language Models (LLM) to provide features like autocorrection and improved writing, etc.

Custom Tools

Some custom functionality will require more than just a single button, e.g., to generate AI images, background patterns, QR codes, or maps. In these cases, you’ll require sliders, text input, drop-downs, and many other UI elements. With plugins, you can easily create panels with your own UI to bring any custom tool to life.

Custom Assets & Presets

Apart from building custom tools, you can also bundle and group effects into presets and make them accessible in a custom panel. This is especially useful to simplify the design process or create standard design components: for example, providing beautiful text presets will enable your users to create instantly great text designs without any design knowledge.

Custom Libraries

Integrate third-party libraries, such as Unsplash, Getty Images, Pexels, or your own.

Custom Editor Behavior

Some of our customers asked us how they could move a functionality from one place in the editor to another. Let’s say you wish to move the function ‘move to front’ from the inspector to the canvas menu. This is not a problem! You can do this by using the internal API endpoints of our editor.

As for custom editor behavior, you can do far more than just move functionality. Here is a demo of layer lists we built as a plugin.

Custom User Feedback

Additionally, you can enhance the canvas with overlays, which are useful for providing alerts, instructions, or feedback directly on the canvas.

What’s Next

We are now rolling out Plugins and building an initial set of features through Plugins ourselves—available for the web first, and mobile SDKs will follow. Keep your eyes peeled for our next releases and don’t hesitate to get in touch with us to learn more about plugins, and how your product benefits from integration without losing time and resources.

Thank you for reading. Join 3,000+ creative professionals—subscribe to our newsletter for updates on new features, plugins, early access, and more!