App Development – IMG.LY Blog

CE.SDK v1.44 Release Notes

Neslihan — Thu, 06 Feb 2025 07:31:00 GMT

Welcome to CE.SDK v1.44! This release brings a range of updates designed to enhance your editing workflow across platforms. Enjoy more vibrant and true-to-life visuals with expanded P3 color support on iOS and Android. On the web, streamline your editing process with the new ability to add multiple clips per track, making timeline management more efficient.

Let’s dive into the updates!

Simplify Your Video Editing Workflow

Add multiple clips to a single track on your video timeline! With CE.SDK video editing for the web, you can insert video clips, text, overlays, and more on one track—resulting in a more organized and efficient workspace.

This update empowers you to create richer, more dynamic sequences while keeping your timeline neat and easy to navigate.

Maximize Efficiency
Simplicity and speed are key. By reducing the need for multiple tracks, this feature helps users to work smarter and faster—allowing them to focus on creative storytelling and deliver engaging content.

Get started with our Video Editing for Web documentation.

Get Vibrant Colors with P3/Wide Gamut Support

Unlock a broader color range with P3 color space support on iOS and Android devices. This enhancement ensures your designs and content are more vibrant, accurate, and true to life—giving you richer color options for all your projects.

Set up color spaces with our documentation for iOS and Android.

💡 The Canvas menu appears when you select a design block, letting you edit its settings, like duplicating a selected image.

Building on mobile configuration introduced in our last release, we’ve extended customization to the editor Canvas Menu for Android. This update gives you full control over how the menu and its items appear and behave in the editor:

Customization: Configure visibility, transition effects, and decoration. Adjust the style of the divider between buttons.
Item List Control: Modify, add, reorder, or hide menu items.
Flexible Button and Divider Configuration: Adjust button appearance, functionality, and divider styles to match your design requirements.

More customizations will be available soon, including for iOS.
Explore the documentation.

Improvements

Switch the Editor Language without Restarting

Easily switch languages in our web editor without restarting it. This update lets you update the UI language instantly via API, streamlining localization. Check our documentation, and translate right away!

Upcoming

Record Reactions to Videos on Android
We’re excited to announce that Reactions will soon be available on Android. Already live in CE.SDK on iOS, this feature—popularized by platforms like TikTok and Instagram—allows users to capture and share their real-time responses to videos. Enhance engagement and provide content creators with immediate, interactive feedback, making your video content even more dynamic.

View the Changelog. Thanks for reading!

3,000+ creative professionals gain early access to new features and updates—don’t miss out, and subscribe to our newsletter.

CE.SDK v1.33 Release Notes

Neslihan — Thu, 29 Aug 2024 12:08:57 GMT

Welcome to the latest update of CE.SDK! We are bringing you new features that enhance your app’s creative capabilities and user experience. This release includes Configurations for a Design Editor or Photo Editor on iOS and Android, Plugins, Voiceover Capabilities, and more.

With this release, you can:

Integrate a Design Editor on iOS and Android

Get started with our new Design Editor configuration for iOS and Android. The out-of-the-box package is designed for easy creation and management of multipage designs.

Users can switch between editing and the page overview mode to review and adjust their work.

Keep the design process straightforward and efficient: the editor features a dock at the bottom, offering convenient access to essential tools like adding text, images, stickers, and uploading new assets. Additionally, you can insert a photo directly from your camera.

Key Features

Page Management
Create and manage your design pages within a template. It’s easy to create fun carousel posts, as seen on Instagram and LinkedIn, or create a series. Head to the ‘Pages’ icon on the top right bar. In the page overview, you can create, duplicate, and edit a page. Lastly, you can change the order of pages by moving them up and down.

Image Settings
Selecting an image lets you fine-tune adjustment settings, and add filters, effects, or blur. Easily crop, straighten, rotate, or flip images.

Try the Design Editor Configuration on iOS and Android.

Use Plugins to Customize UI and Add Quick Actions

We’re excited to introduce the first part of our plugin release for CE.SDK on the web: with this release, you can tailor the editor’s UI elements, reorder components, and add quick actions like background removal.

We’ve enabled extension points within key areas of the UI: the Canvas, Navigation bar, Inspector Bar, Dock, and Canvas Menu. Modify the appearance of existing features, register new components, and integrate quick actions. For example, you can add a “Background Removal” button directly in the Canvas Menu or create complex interactions like custom cut-out lines.

For more details on this first batch release and the upcoming plugins, please read our dedicated article.

Add Voiceovers to Videos on iOS

Add voiceover tracks directly within your mobile app, making your projects more dynamic and engaging. Whether creating a tutorial, narrating a slideshow, or adding commentary to a video, this feature allows you to seamlessly record, edit, and integrate voiceovers.

The popular feature, as enjoyed on TikTok and Instagram, is a staple for content creation and adds a personal touch to video creations.

To get started, navigate to “Add Audio”, select “Voiceover” section and start recording. You can preview the recording, re-record it, and add more audio, such as music. Adjust the audio levels for a balanced background track.

Try recording a voiceover with our Demo App for iOS.

Integrate a Photo Editor for iOS and Android

Our new Photo Editor Configuration is designed to offer an intuitive, out-of-the-box solution that bundles a range of key photo editing capabilities. Enhance individual photos, add text and shapes, or apply filters—this configuration is a kick-start to provide high-quality photo editing for iOS and Android with CE.SDK.

Key Features

Crop, Resize
Crop, straighten, flip, resize, and rotate your photos.

Adjustments
Fine-tune your photos by adjusting brightness, contrast, and other key settings, ensuring your images look their best.

Text, Shapes, and Stickers
Enhance your photos by adding and customizing text, shapes, and stickers. Whether you’re creating eye-catching titles, intricate shapes, or playful stickers—our configuration gives you the flexibility to adjust fonts, font colors, sizes, and positions to perfectly match your vision.

Effects, Filters, Blur, Blending Modes
Transform your images with fun effects, blur, and filters. Apply overlays to add depth and texture to your photos by adjusting blend modes and opacity to achieve the perfect effect.

Additionally, use our range of photo filters to set the mood, from warm and nostalgic tones to vibrant, modern effects, giving your images a distinctive and professional look.

Check our documentation to get started with our photo editor configuration, or try the photo editor on iOS and Android.

Improvements

Preview Audio on iOS and Android

We’ve made it easier to select the perfect audio clip for your projects. Listen to audio tracks directly within the interface before adding them to your video. Tap the play button of an audio clip in the asset library to preview it, and adjust your audio clip within the timeline.

Resize Pages

Resize pages in your web editor by simply grabbing corners and side handles, moving beyond the limitations of input fields or preset format sizes. This intuitive resizing integrates seamlessly with other adjustments, such as image cropping, ensuring a smooth workflow. Enable or disable this behavior as needed.

Create Rounded Corners

You can now easily create rounded corners for images on the web, iOS and Android. Within your designs, you can add a sleek and modern touch to your projects.

Drag and Select Multiple Elements

You can now click and drag with your mouse to draw a selection rectangle, instantly selecting all blocks within the covered area. This feature saves you time and effort, working with complex templates.

Thanks for reading! Don’t hesitate to reach out if you have any questions or need assistance.

Over 3,000 creative professionals gain early access to our new features, demos, and updates—don’t miss out, and subscribe to our newsletter.

Please note that these release notes also include improvements from previous versions.

Designing A Timeline For Mobile Video Editing

Frank — Tue, 20 Aug 2024 13:08:58 GMT

This article documents the most surprising insights from our journey building timeline user interface. We’ll explore everything from overcoming usability challenges, visualizing data, prioritizing information, and refining touch interactions.

IMG.LY’s CE.SDK for iOS now comes with a native video editor, built from scratch in SwiftUI. I was responsible for coding and polishing the timeline user interface, working with both our design team and our cross-platform engine team. When I started implementing the timeline, we had the basic design concept sketched out in Figma. However, for an advanced UI like this, nothing beats rapid prototyping and iteration between design and development to make it feel right.

I. Visualizing Time

Timelines are a well-known convention, at least to modern humans. Timelines show durations in time as segments in space, distributed along the horizontal axis, with multiple tracks for things that happen in parallel. This is a simple yet effective way of representing moments in time visually. As obvious as it may seem today, this flavor of data visualization first appeared only 250 years ago. One of the earliest known examples is Joseph Priestley’s A Chart of Biography, published in 1765. Here is Priestley’s A New Chart of History from 1769, which is even more impressive:

Video editing timelines are built on this exact principle of mapping time to space: A digital video composition consists of tracks for simultaneous audio and video layers. These tracks contain individual clips that can be rearranged, trimmed, and transformed. Navigating the video’s time dimension is as easy as scrolling left and right. The tracks and clips serve as landmarks for orientation, forming the mental model of what the video composition is.

Timelines map time to space

https://www.youtube.com/embed/cKcksLW95-Y?feature=oembed

Horizontal Direction

Modern iOS user interfaces appear flipped in right-to-left languages: the buttons and controls are in different horizontal positions. The layout follows the reading direction of the language. Even the familiar drill-down navigation animates to the left when drilling deeper into the hierarchy, and the Back button sits in the upper right corner, its chevron pointing to the right. So, naturally, one of the first things we checked was whether timelines should run right-to-left in writing systems like Arabic or Hebrew. According to a trustworthy source, this is not the case: we found an old version of Apple’s developer guides stating that editor timelines in software are universally left-to-right, even if the surrounding interface layout is flipped by the operating system for right-to-left system languages.

Cognitive Load vs. Discoverability

When designing our video timeline, we experimented with both the size of the clips and with the dimensions of the timeline itself. How much can you fit on a screen before it looks cluttered? How large can the controls be without being clumsy and inefficient? The only way to find out is to try it out right on the device.

Hiding features in dropdown menus is easy. Banishing them to separate screens is even easier. But hiding functionality hurts discoverability. Out of sight, out of mind. We think user interfaces in productivity apps should be the opposite. They should prioritize high information density and avoid extra navigation steps as much as possible.

We increased the density of the toolbar that sits below the timeline: Our goal was to make all the important features easily accessible, despite the small screen. We achieved this by changing the single floating Add button from the initial designs to a more granular set of buttons to add specific asset types, like stickers and text. This not only increases efficiency for frequent users, it also makes the capabilities of the video editor more obvious and approachable.

Adaptive Height

As space is at a premium on a small screen, we wanted to make the video preview as large as possible. We decided to allow the user to collapse the timeline at any time to focus on the video canvas. And even without manually minimizing the timeline, it doesn’t waste space: the timeline adjusts its height to fit snugly around its contents. It only grows as you add more clips.

Units and Scale

The numeric scale is an essential part of any timeline. It helps you see the in and out time of a clip at a glance, and the permanently visible ruler makes changes in the zoom level easy to understand. We added dots as subdivisions between the ruler’s numeric labels, which gives it a nice tool-like character. At full zoom—when the ruler shows 5-second intervals—we decided to use four dots in between, so there is one dot for each second. We think this detail adds a welcome sense of precision.

The clips themselves have a duration label when they are selected. If the duration is below 10, you get a fractional digit (e.g., 4.2s). For the ruler, we went with the same short format (5s, 10s, 30s, etc.) for all values below one minute, and the technical format (1:00, 1:30, etc.) for the rest. Of course, we use iOS’s built-in number formatters to get properly formatted and localized strings for our labels.

An important detail for the label that shows the current playback timecode: The seconds must always be rounded down. If the playhead sits between two seconds, the lower number must be displayed until the playhead crosses the indicator for the higher second. Simple rounding to the nearest integer would be confusing, because it doesn’t match our expectations when dealing with timelines. Five seconds do not count as five seconds until they have elapsed in their entirety.

II. Manipulating Time

What looks good is not guaranteed to feel good when you interact with it. A video editor timeline is more than just a visualization; it’s also a user interface. As such, it should be self-explanatory and provide constant feedback as the user manipulates it.

Fingertip-sized

The clips must be large enough to be easy to touch and at the same time detailed enough to give a decent overview. The size that can be easily tapped by most people is around one square centimeter, which matches the 44 × 44 pt default size of iPhone system controls. Our testing showed that we could go a little smaller. Editing video is a focused task, so the tap targets don’t need to be as robust as for tasks that happen while you, say, run for the bus. And all user interfaces should be scalable for accessibility anyhow, so we didn’t hard-code any sizes and we also made the dimensions adjustable. And, speaking of adapting to individual user needs: of course, all the text labels in our timeline are automatically scaled depending on your device settings.

Handles

Selected clips in the timeline get a bold blue frame with large handles on either side. These handles are used to trim the clip’s length. For comfortable tapping and dragging, the handles must be larger than their visual size. In fact, we made the hit area more than twice as wide as the visual appearance suggests.

Early on in our design process, we realized that the trim handles had to sit on the outside of the clip, extending beyond the left and right edge. This made it hard to see the clip’s actual size, so we added little grooves: little triangular indents that point to the start and end of the selected clip.

Some clips in the timeline can be extended to any duration, like text or stickers. Other clips have a finite footage duration, like video and audio clips. To make this difference visible, clips with a finite footage duration reveal a ghost outline in the background while they are resized, featuring marching ants. The handle icon changes from an outward-pointing chevron to a vertical line when the clip can’t be made longer.

Scrolling and Zooming

A rather invisible detail was one of the trickiest things to get right: the coordination of all the different finger gestures on the timeline.

Horizontal Scrolling: Timeline scrolls horizontally to move in time.
Vertical Scrolling: Timeline scrolls vertically providing room for tracks.
Clip Movement: Dragging horizontally on a selected clip moves it.
Duration Adjustment: Dragging the trim handles adjusts the clip’s duration.
Pinch-to-Zoom: Pinching with two fingers changes the zoom level of the visualization.

All these gestures must behave as expected and not interfere with each other. We found it unexpectedly difficult to fine-tune and harmonize these interactions with pure SwiftUI, so we used some proven legacy iOS techniques to get it right. If you’re interested in the technical details of our solution with SwiftUIIntrospect and UIGestureRecognizers, check the source code.

Snapping

Video editing must feel precise, even on a small screen. When you trim or move clips, they should snap to other clips, to the playhead, and to the start and the end of the timeline. To show which position the selected clip snaps to, we designed animated dotted lines. We use the iPhone’s haptic engine to provide subtle feedback for snapping, which really helps to make the interaction feel natural—like a notch snapping into a detent. Something that we didn’t get right at first: Snapping should only respect those snapping points that are currently visible in the viewport. Of course! Otherwise, a partially visible clip may snap to something offscreen, which is very confusing.

Scrubbing Preview

For editing a video clip to feel precise, you need to be able to see exactly where you’re trimming. But the playback position in our mobile timeline can’t change while you’re trimming, because that would mean scrolling the timeline—which would move the very clip you’re trimming. The trim preview needs to be independent of the composition’s playback position. We solved this by adding a temporary overlay with its own playback time to the canvas. When you start dragging one of the handles on either side of the clip, the overlay shows the exact trim position of the video clip. When you release, the preview returns to the playback position.

Try it out!

“The only way to experience an experience is to experience it,” said interaction designer Bill Moggridge, and we couldn’t agree more! If you’re curious how our new video timeline feels, download our iOS app and try editing your own videos in our brand-new timeline experience.

If you’re a developer, check out the video editor documentation. With IMG.LY’s SDK, you can add all of these video editing capabilities to your own apps in a breeze.

Background Removal in the Browser Using ONNX Runtime with WebGPU

Daniel — Tue, 11 Jun 2024 16:26:17 GMT

TL;DR: Using ONNX Runtime with WebGPU and WebAssembly leads to 20x speedup over multi-threaded and 550x speedup over single-threaded CPU performance. Thus achieving interactive speeds for state-of-the-art background removal directly in the browser.

Removing background from an image is a typical job to be done in creative editing. We have come a long way from manually knocking out the background from an image to full automation with Neural Networks.

Most state-of-the-art background removal solutions work by offloading the task to the server with a GPU as it was simply infeasible to run the NN on the client.

However, running background removal directly in the browser offers several advantages over server-side processing:

Reduced server load and infrastructure costs by offloading heavy lifting to the client.
Enhanced scalability by distributing the workload across client devices.
Easier compliance with data protection and security policies by not transferring data across a network to a server.
Offline processing without needing a reliable internet connection.

It caters to a wide range of use cases, including but not limited to:

E-commerce applications that need to remove backgrounds from product images in real time.
Image editing applications that require background removal capabilities for enhancing user experience.
Web-based graphic design tools that aim to simplify the creative process with in-browser background removal.

In general, two factors influence the feasibility of running background removal directly on the client.

The execution performance, and
the download size of the Neural Network.

The performance or overall runtime is the major factor to be useful in interactive applications, if a user has to wait several minutes or hours for a neural network to execute, this is in many cases far too long in terms of good user experience. From experience, there are three factors to consider.

The initial first-time execution. The major factor is that neural networks come with the drawback of generally being several MB to GB in size, thus the time to download the neural network into the browser cache is considerable. In subsequent browser page reloads this has no impact anymore.
The neural network or session initialization time, cannot be cached and has to run with every reload of the page in the browser.
The neural network or session inference time, largely depends on the longest path inside the neural network and most importantly the execution time of each operator in the neural network.

Towards Real-time Background Removal in the Browser

Neural networks are commonly trained in frameworks like PyTorch, which is a neural network library for Python, as such not usable directly in the browser. The best option to run neural networks directly in the browser is converting the neural network into the Open Neural Network Exchange (ONNX) format, which is a widely supported standardized format by Microsoft used extensively in the industry.

These ONNX-formatted neural networks can then be reconverted into a platform-specific format or directly executed by a supported runtime.

The ONNX Runtime by Microsoft is a high-performance inference engine designed to run ONNX models across various platforms and languages. One notable feature is ONNX Runtime Web, which allows JavaScript developers to execute ONNX models directly in the browser. ONNX Runtime Web offers several execution providers for hardware acceleration. For instance, its WebAssembly execution provider enhances CPU execution performance using multiple Web Workers and SIMD instructions.

More importantly, starting from version 1.7.0, ONNX Runtime Web includes official support for the WebGPU execution provider. WebGPU is a modern web API that enables developers to utilize GPU power for high-performance computations, offering a significant performance boost over CPU-based in-browser machine learning. WebGPU support has been available by default since Chrome 113 and Edge 113 on Mac, Windows, and ChromeOS, and Chrome 121 on Android. For the latest browser support updates, you can track them here.

Implementation Details

For the implementation of the open-source package @imgly/background-removal-js, we started the journey with a neural network implementation in PyTorch written in Python. The network was then converted to ONNX. You can see our live showcase here.

The original model was using 32-bit floating point (fp32) precision, which is fine, but results in a file size of 168 MB after converting it to ONNX. As mentioned earlier, the size of the network has a large impact on perceived first-time execution performance as the download time tends to be longer than the execution time, but more to that later.

To reduce the size of the model, we converted the model to use fp16 (16-bit floating point) and QUINT8 (Quantized 8-bit) datatypes. Thus, effectively reducing the size to half (84MB) and a fourth (42MB) of the original size. Additionally to the download size, the operators used will be converted corresponding to the datatype and different data types and depending on the hardware may lead to speed improvements or even deteriorating due to specialized hardware being used or not being present.

Note, that while that sounds great, the conversion has a potential negative impact on the quality of the output as we are removing information in the neural networks and since we are working with images, artifacts might become visible and the quality of the resulting background mask is reduced.

Evaluation

When dealing with neural networks in the browser we can identify three different scenarios

First Use
First Run
Consecutive Runs

The first-use scenario occurs when the web application is started the first time. The neural network has to be downloaded from the server into the browser sandbox, as neural networks are several 10-1000 MB in size this is not neglectable.

The first run assumes that the neural network is already present in the browser cache, however, when the neural network ought to be used, the network has to initialize before execution.

In consecutive runs, the neural network is already in memory and the execution time is largely determined by the neural network depth and operators only.

First Use – Neural Network Download

While download or time is a large factor in the first-time execution time, this is largely dependent on the available network bandwidth, as such it is not part of the evaluation but in order to get an idea here are the expected download times for the fp32 (168MB) and fp16 (84MB) neural network subject to various common network bandwidth:

Network Bandwidth	Filesize	Download Time
10 Mbps	84 MB	67s s
100 Mbps	84 MB	6.7 s
1 Gbps	84 MB	0.67 s
10 Gbps	84 MB	0.067

Network Bandwidth	Filesize	Download Time
10 Mbps	168 MB	134.4 s
100 Mbps	168 MB	13.4 s
1 Gbps	168 MB	1.34 s
10 Gbps	168 MB	0.13 s

Based on the assumption that the median download bandwidth is ~100 Mbps, we see that it’s in the 5-15 second range.

First Run – Neural Network Initialization / Compilation

Before the neural network can be executed it has to be initialized. Initialization includes several execution provider-specific steps. Most prominent are the time to upload or convert the data to the execution provider and execution provider-specific ONNX graph optimization passes. This all adds to the first run experience. We have evaluated the average session initialization time for the CPU (WASM) and WebGPU provider on a MacBook Pro 13” from 2024 with an Apple M3 Max 16 cores to get an idea of the general impact on first run execution time:

Device	Datatype	Session Initialization Time
CPU (WASM)	fp32	~320ms
CPU (WASM)	fp16	~320ms
WebGPU	fp32	~400ms
WebGPU	fp16	~200ms

Note, that session initialization is not negligible and adds significant runtime overhead and might be subject to additional optimization possibilities like caching the optimized model.

Consecutive Runs – Neural Network Execution

Independent of the download and initialization time, we evaluated different execution providers on a MacBook Pro 13” from 2024 with an Apple M3 Max 16 cores. While this is top-end consumer hardware, the general trends will probably apply to various hardware configurations. As a reference, we choose the single thread performance with the neural network running on the CPU with 1 worker thread and SIMD disabled.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
CPU (WASM)	No	1	fp32	~53000 ms	1.0 x
CPU (WASM)	No	16	fp32	~6300 ms	8.4 x
CPU (WASM)	Yes	1	fp32	~15000 ms	3.5 x
CPU (WASM)	Yes	16	fp32	~2000 ms	26.5 x

The data above reveals that running the neural network without any acceleration such as SIMD and threading in the browser results in almost 53s runtime, and as such is for most interactive use cases too slow to use. Due to the optimizations of the ONNX Runtime that uses SIMD and threads, we can achieve an overall speedup of roughly ~26 times compared to the baseline performance. Thus decreasing the session runtime to around 2s and making it usable for interactive applications.

As mentioned before, we neglected the download time and size of the network. Leveraging the fp16 model compression, we re-ran the benchmarks and got similar results as before, but with half the bandwidth required to download the network for the first time application. The general visual performance of the fp16 and fp32 is similar or visually not perceivable. The quint8 model led to artifacts and as such unusable for visual processing, thus we excluded from the following benchmark.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
CPU (WASM)	No	1	fp16	~55000 ms	1.0 x
CPU (WASM)	No	16	fp16	~7300 ms	7.2 x
CPU (WASM)	Yes	8	fp16	~15000 ms	3.5 x
CPU (WASM)	Yes	16	fp16	~2300 ms	23.9 x

The results are a little worse than the fp32 version, which might be because the fp16 datatype has no direct specialized hardware in modern CPUs like the M3 Max CPU, and as such additional fp16 to fp32 conversion operations have to be performed.

Our final benchmark measures the WebGPU performance, which led to the following impressive results. To have a fair comparison and understanding of the impact of the WebGPU technology, we compare it with the best CPU version with 16 threads and SIMD enabled with the fp32 model, which was around ~2 s.

Device	SIMD	Threads	Datatype	Session Runtime	Speedup
WebGPU	Not applicable	Not applicable	fp32	~120ms	16.6 x
WebGPU	Not applicable	Not applicable	fp16	~100ms	20.0 x

The WebGPU performance varies around 16 to 20 x improvements over the best CPU execution time. The GPU has specialized hardware for fp16 instructions, as such the model performs better than the fp32 model. Also, note that the session initialization time is also reduced due to half the required bandwidth to upload the network data to the GPU.

Therefore, the first run of the network will take ~300 ms and consecutive runs will be ~100 ms, leading to near real-time performance in the browser.

Note, that the WebGPU performance is an astonishing 550 times faster than the single thread, with no SIMD performance.

Try the live implementation in our background removal showcase.

Conclusion

WebGPU is a major leap in establishing the browser as a factor to be reckoned with as a true Application platform. With IMG.LY, we are striving to leverage modern technology to make design tools accessible, this includes on-device execution, on-premise, but also on-cloud execution of design tools leveraging neural networks.

As a next step, we will port background removal to all our supported platforms, ONNX Runtime seems the best choice, as it is already available for all the potential platforms we support.

Furthermore, we are evaluating the feasibility of in-browser background removal for videos to be included with our video-editing suits.

Try our tools today to see how we help bring unique creative editing experiences to any application. Or contact us to discuss your project.

Unleash Creativity with CE.SDK’s New Plugin System

Daniel — Mon, 03 Jun 2024 08:42:25 GMT

At IMG.LY, we have always believed that a superb design editor should be effortlessly customizable and extensible. We are thrilled to roll out a brand-new Plugin system for CE.SDK in the upcoming months—to take creative editing and feature development to the next level.

Starting with one-click features like background removal or vectorizers, smart design tools like QR codes or subtitle generators, and deeply interactive features like generative AI for text and images; all these tools can be used or built by customers soon.

Additionally, our upcoming Plugin system will bring you unparalleled autonomy by making feature development for our SDK accessible and offering extensive options to reconfigure our editor’s UI.

Start exploring our Plugin System rollout now to immediately benefit from upcoming features.

Built for Modification

Let’s dive deep into some of the opportunities CE.SDK plugins help unlock.

Unlocking the AI Revolution

While the AI transformation is already fully underway, much of the tech is still not very accessible to product builders, often requiring deep technical knowledge to get started. At the same time, AI features become significantly more valuable when integrated with other editing functionalities in workflows or automation. With our plugins, we aim to make it effortless to leverage innovation and put it to use.

Boosting Customer Autonomy

Key to our success is providing maximum flexibility and autonomy to our customers about product decisions. Ultimately, you shouldn’t depend on our product roadmap; rather, you should be able to add features when you like. While our SDKs are already highly configurable, plugins allow tailoring the whole user experience not only on a look & feel level but through custom functionalities and editing experiences.

Accelerate Product Expansion

Many ecosystems witnessed explosive growth in added value to the user after releasing plugin mechanisms. Currently, only IMG.LY core developers can contribute to the SDK. We have started to extend this to solution engineers and even designers on our team who don’t have much knowledge of the inner workings of the SDK. Ultimately, we will push this more and more into a community of contributors, making the community’s innovation accessible to everyone.

Key Concepts

Three important concepts have driven the development of our Plugins:

Customizable Menu Bars
We are extending our API so that it allows easy hooking into various parts of the UI. Our editor has key components like the inspector, toolbar, and on-canvas menu. These are now all accessible through an API, so you can hook your feature anywhere in the editor.

UI Building Blocks to Provide Consistency
To reach a high level of consistency and speed up time for development, we will be providing out-of-the-box UI components such as buttons, sliders, text inputs, etc.

Escape Hatches
From experience, we know that sometimes unique functionality needs unique solutions, so we have added escape hatches to add custom elements via HTML whenever needed.

What Can You Build with our Plugin System?

Let’s explore some potential use cases of the plugin system.

Custom Actions

Adding custom actions is a great option to make simple third-party APIs accessible within the editor. This can be one-click edits such as background removal, vectorizers, or auto-enhancement for images. You can also add custom actions for text in combination with Large Language Models (LLM) to provide features like autocorrection and improved writing, etc.

Custom Tools

Some custom functionality will require more than just a single button, e.g., to generate AI images, background patterns, QR codes, or maps. In these cases, you’ll require sliders, text input, drop-downs, and many other UI elements. With plugins, you can easily create panels with your own UI to bring any custom tool to life.

Custom Assets & Presets

Apart from building custom tools, you can also bundle and group effects into presets and make them accessible in a custom panel. This is especially useful to simplify the design process or create standard design components: for example, providing beautiful text presets will enable your users to create instantly great text designs without any design knowledge.

Custom Libraries

Integrate third-party libraries, such as Unsplash, Getty Images, Pexels, or your own.

Custom Editor Behavior

Some of our customers asked us how they could move a functionality from one place in the editor to another. Let’s say you wish to move the function ‘move to front’ from the inspector to the canvas menu. This is not a problem! You can do this by using the internal API endpoints of our editor.

As for custom editor behavior, you can do far more than just move functionality. Here is a demo of layer lists we built as a plugin.

Custom User Feedback

Additionally, you can enhance the canvas with overlays, which are useful for providing alerts, instructions, or feedback directly on the canvas.

What’s Next

We are now rolling out Plugins and building an initial set of features through Plugins ourselves—available for the web first, and mobile SDKs will follow. Keep your eyes peeled for our next releases and don’t hesitate to get in touch with us to learn more about plugins, and how your product benefits from integration without losing time and resources.

Thank you for reading. Join 3,000+ creative professionals—subscribe to our newsletter for updates on new features, plugins, early access, and more!

CE.SDK v1.25 Release Notes

Neslihan — Tue, 07 May 2024 11:43:59 GMT

Welcome to the latest update from our team! Since our last release, we have been working to bring more creative capabilities to your app and improve user experience. With this release, you can:

Recolor Images like T-shirt Mockups

Replace colors in any image with our new recoloring feature. Select a color you want to change, choose its replacement, and apply the changes. Recoloring is ideal for tasks like altering the color of specific elements, such as t-shirts in product photos.

Simply choose your image, navigate to Effects, and select “Recolor”. Choose the color you want to change (Source Color), and the color you want to replace it with (Target Color). For a more refined result, adjust “Smoothness”, “Color Match”, and “Brightness Match”.

This feature is available on Web, Android, and iOS platforms.

Check our documentation on filters and effects to learn more.

Create Transparency with a Green Screen

Easily make green screen backgrounds transparent in your images and videos on the Web, and for images on Android and iOS. Our Green Screen Effect is tailored to identify and eliminate green backdrops—or any other color you specify. Perfect for content creators and videographers who need to isolate subjects without manually editing backgrounds.

Select an image or video clip with a color background you wish to remove. Then choose “Green Screen” in the Effects, and our effect handles the rest. Fine-tune the effect to your liking.

User Interface Improvements

In our latest update, we’ve focused on enhancing the usability and functionality of our interface. Here’s what’s new:

Audio Preview

Choosing the right audio clip is now easier with our new Audio Preview feature. Preview audio clips directly within the interface and make the perfect selection for your projects.

Selection Frame Improvements

We’ve upgraded our selection frames with new icons, hover effects, enhanced drag functionality, and shadow rendering. These improvements make manipulating elements within our platform more intuitive and visually appealing. Whether you’re rearranging components or resizing elements, our refined handling ensures a smoother interaction.

Please note that these release notes also include improvements from v1.24.

Thanks for reading. Join over 3000 specialists with powerful apps, and subscribe to our newsletter. We keep you in the loop with brand-new features, early access, and updates.

CE.SDK v1.19 Release Notes

Neslihan — Fri, 22 Dec 2023 14:26:48 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we’re thrilled to introduce CE.SDK v1.19! With this release, you can:

Unify Design Power with Block Unification

Introducing Block Unification for CE.SDK on Mobile and Web—a feature that simplifies design versatility. Videos, images, shapes, and stickers now seamlessly come together as one Graphic Block, easily modified within a unified inspector. All supported settings for your selected block, including strokes, filters, and adjustments, are conveniently displayed in a single, user-friendly space.

Unified Graphic Block
Experience a singular Graphic Block supporting diverse fills (images, videos, colors, gradients) and shapes. Switch fill types (video, image, color) effortlessly, enabling dynamic designs in one place.

Easy Fine-Tuning
Block Unification integrates effects, filters, adjustments, and blur uniformly across all design elements. Elevate your designs with precision and consistency.

A Treat for Users and Developers
For developers, Block Unification means streamlined code maintenance. The consolidation of design blocks into one Graphic Block simplifies development, allowing more focus on crafting exceptional design experiences with IMG.LY.

Enhance Interaction with Mouse Wheel Support

In our latest update, our engine now directly registers mouse wheel events on the canvas, eliminating the need for manual camera manipulation. This seamless integration brings a host of new features, including smooth zooming, centered around the cursor, and support for pinch gestures on multitouch trackpads. Enjoy a more intuitive and unified interaction experience across various inputs, simplifying and enriching your design process.

Migrate to CE.SDK v1.19

The structural changes of CE.SDK v1.19 bring a more composable and powerful editing experience. To benefit from our new features, and to attain to licensing changes, head to our documentation on migrating to v.1.19 for an easy transition.

Wait. There’s more. As we explore the extensive capabilities of our SDK, we’re unveiling new showcases to demonstrate your possibilities. Plus, stick around until the end for an exciting announcement that will elevate your video creation experience!

New Showcase: Automated Resizing

In our new live showcase for web, effortlessly generate size variations of your designs and seamlessly scale your marketing materials across platforms. The automated resizing feature empowers you to easily leverage marketing materials without requiring the involvement of a design team, ensuring efficiency and adaptability.

New Showcase: Version History

Discover the convenience of Version History, enabling you to monitor changes made to each design and seamlessly restore past versions when needed. Explore our straightforward Version History showcase for a practical design experience.

Outlook: Offer TikTok-Inspired Video Editing

Video remains the reigning champion for attracting and retaining your audience. Yet, building a video editor from the ground up is a pain.

To simplify your journey and save valuable time and resources, we are excited to unveil a groundbreaking SDK for creating captivating short-form videos in January 2024!

Empower your users to arrange video, audio, text, and graphics seamlessly on a sleek video timeline. Our new camera introduces popular features like Voiceover, Zoom, Tap to Record, and more. Plus, bring the beloved Split Screen Modes for Reactions and Duets, akin to TikTok and Instagram!

Be the first to access our new IMG.LY SDK—join our waitlist now!

Thanks for reading! Subscribe to our newsletter to stay in the loop with the latest news and updates.

CE.SDK v1.17 Release Notes

Neslihan — Tue, 17 Oct 2023 12:48:42 GMT

In the ever-evolving landscape of app development, staying ahead of the curve is crucial. Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we’re thrilled to introduce CE.SDK v1.17. With this release, you can:

Support WebP File Formats

Interfaces: Engine
Platforms: All

We’re delighted to announce that our editor now supports WebP images, delivering you an array of benefits. WebP is gaining popularity for its unique ability to strike the perfect balance between high image quality and smaller file sizes. This means your visuals will look stunning while optimizing website performance.

Capture and Export Videos with Camera for Android

Interfaces: Engine
Platforms: Android

Video content creation has become the lifeblood of today’s app ecosystem. Social entertainment, user-generated content, and the demand for immersive experiences have transformed the way we engage with audiences.

We’re excited to introduce Camera Recording for Android, seamlessly integrating your live camera preview with our engine’s extensive capabilities. This means you can apply a wide range of effects, strokes, and drop shadows to your live camera feed, all while ensuring it harmoniously blends into the composition of your scene.

This feature allows you to seamlessly capture, edit, and export high-quality videos directly within your applications. Whether you’re building a social platform, a video editing app, or a storytelling tool, this feature will redefine your users’ video creation experience. Learn how to use your camera in our Engine for Android in our documentation.

Stay tuned for more exciting updates and the full reveal: CE.SDK v1.17 is just the beginning of a new era in app-driven video content.

Thank you for being a part of this journey with us. Subscribe to our newsletter to ensure you never miss out on updates! ?

CE.SDK v1.16 Release Notes

Neslihan — Tue, 26 Sep 2023 14:26:37 GMT

It’s getting colorful! Since our last release, we’ve been busy crafting new features and enhancements to empower your user’s creative journey. We are excited to announce that CE.SDK v1.16 is now available.

With this release, you can:

Maintain Color Consistency with CMYK & Libraries

Interfaces: Engine, Editor
Platforms: Web

We understand the importance of precise color reproduction and the need for brand consistency. That’s why we’re enhancing your editing experience with greater color control.

You can now seamlessly integrate CMYK and Pantone color spaces, along with other customized color libraries, including your brand’s unique colors.

This ensures that your designs not only meet but exceed expectations across all mediums, delivering a consistent and visually engaging experience. It also helps eliminate color inconsistencies and product returns, while ensuring color accuracy.

Learn more about adding color libraries to CE.SDK in our documentation.

Improve Web Interaction with Touch Gestures

Interfaces: Engine, Editor
Platforms: Web

With this update, we’re bringing a more interactive and responsive experience to our web applications. Let your users experience seamless multitouch interactions, even on mobile web. This ensures that the content and functionality of our web applications remain accessible and user-friendly across various devices.

Thank you for reading! Never miss out on updates and subscribe to our newsletter.

CE.SDK v1.14 Release Notes

Neslihan — Fri, 25 Aug 2023 15:37:00 GMT

Since our last release, we’ve been hard at work developing new features and improvements to enhance your creative editing experience. We are excited to announce that CE.SDK v1.14 is now available!

With this release, you can:

Generate Thumbnails Fast for Smooth Performance

Interfaces: Engine
Platforms: All

We have made significant enhancements to the way video thumbnails are generated, resulting in a smoother and more efficient workflow. Our intelligent engine now renders thumbnails seamlessly across multiple frames, ensuring there are no delays or interruptions while you work. Instead of slow JPEG compression, we have implemented a faster method using raw pixel buffers that maintain high-quality results.

If you’re using web applications, the thumbnail width adapts to fit your screen, providing a seamless and enjoyable user experience. Additionally, our dynamic thumbnail generation adjusts based on the size of your clips and zoom level, optimizing resources and improving overall performance.

Enhance Templating with Improved Placeholders

Interfaces: Engine, Editor
Platforms: All

Introducing an enhanced placeholder behavior for text and image content, allowing for seamless editing and replacement. With the new PlaceholderBehavior concept, you can define whether a block should initially look and behave as a placeholder or allow editing/replacement. This feature provides a smooth editing experience for users and offers flexibility for template designers to differentiate between placeholder and desired content.

Bring Creative Editing to Your macOS App

We’re excited to announce the addition of macOS platform support to our lineup. Extend your application’s reach to macOS users and provide a seamless editing experience across different platforms. Get started, and initialize our engine in your macOS app with our documentation.

Expand Editing Capabilities: macOS & Catalyst Binding

With this new feature, we provide a comprehensive solution for developers looking to enhance their iOS applications with our CE.SDK’s editing capabilities. Learn how to initialize our creative engine in your iOS app in our documentation.

Thank you for reading! Never miss out on updates and subscribe to our newsletter.

IMG.LY Announces Open Source JavaScript Library for In-Browser Background Removal

Jan — Wed, 28 Jun 2023 11:38:49 GMT

UPDATE October 2023: We’ve recently introduced Node.js support for the @imgly/background-removal npm package. This enhancement enables you to remove backgrounds from images not only in the user’s browser but also on your server. This opens up the potential for batch processing and allows you to bypass memory constraints, especially for larger images.

Just before the summer lull engulfs us all, we have something exciting to share! We are thrilled to announce the release of @imgly/background-removal, an innovative npm package that empowers developers to seamlessly remove backgrounds from images directly in the browser.

Gone are the days of relying on server-side processing or sacrificing data privacy. With IMG.LY’s Background Removal, you can now harness the power of in-browser background removal with ease. Let’s dive into the key features that make this library truly exceptional:

In-Browser Background Removal

Our one-of-a-kind solution performs the entire background removal process directly in the user’s browser, eliminating the need for additional server costs. By leveraging the computing power of the local device, users can enjoy a fast and efficient background removal process that streamlines their workflow.

Data Protection

Rest assured that your images and sensitive information remain secure within your own devices. As IMG.LY’s Background Removal runs entirely in the browser, there are no data transfers to external servers, ensuring robust data privacy and alleviating any concerns you may have.

You can experience it in action on our background removal showcase.

Seamless Integration with IMG.LY’s CE.SDK

IMG.LY’s Background Removal seamlessly integrates with IMG.LY’s CE.SDK, making it easier than ever to incorporate powerful in-browser image matting and background removal capabilities into your projects. Boost your creative endeavors with this intuitive integration.

The Neural Network (ONNX model) and WebAssembly (WASM) files used by IMG.LY’s Background Removal are hosted on UNPKG, making them readily available for download to all users of the library. However, if you prefer to host the data on your own servers, our library also supports custom asset serving. The choice is yours!

Background removal is often the first step in any creative workflow involving image composition. Whether you are developing e-commerce applications that need real-time background removal, enhancing user experience in image editing applications, or simplifying the creative process with web-based graphic design tools, IMG.LY’s Background Removal is your go-to solution.

Empower Yourself with IMG.LY’s Background Removal

Whether you are a professional developer or a hobbyist, this open source JavaScript library empowers you to deliver impressive applications and services with ease. Join the growing community of developers who are revolutionizing the way background removal is done.

Get started with @imgly/background-removal today by visiting our official npm package page and our GitHub repository. You’ll find comprehensive documentation, examples, and everything you need to unlock the full potential of in-browser background removal.

Update: Node.js Support

We have just shipped Node.js support for the @imgly/background-removal npm package. In addition to the users’ browser, you can now remove backgrounds from images on your server, making it possible to perform batch processing or bypass memory constraints for larger images.

Thank you for your support, and we can’t wait to see the incredible projects you’ll create with IMG.LY’s Background Removal. Stay tuned for more updates and enhancements as we continue to improve and expand this powerful library. Happy coding!

Don’t forget to subscribe to our newsletter to stay in the loop with the latest news and updates.

IMG.LY Partners with Soundstripe to Infuse Video Editing with Epic Royalty-Free Music & SFX

Jan — Wed, 24 May 2023 07:31:24 GMT

We are excited to announce our exciting new partnership with Soundstripe, the leading provider of royalty-free music for video creators. Our VideoEditor SDK now features a seamless integration with Soundstripe, offering users access to a vast collection of over 9,000 hand-curated songs from 150+ musicians, along with an extensive library of 70,000 sound effects.

Starting from version 11, our VideoEditor SDK now comes pre-equipped with a seamless Soundstripe integration, as detailed in our documentation.

Streamlining Music Licensing for Digital Platforms

Finding the right music for digital platforms hasn’t been easy. Music licensing complexities and the challenge of finding high-quality tracks that align with your brand standards can be overwhelming.

But now, with our partnership with Soundstripe, you can effortlessly enhance your app with royalty-free music and video editing.

Our mission has always been to provide the ultimate toolkit for building captivating creative experiences, and video editing plays a central role in achieving this. We recognize the significant role that music plays in video storytelling, as it has the power to evoke emotions, set the mood, tone, and atmosphere, and greatly influence the pacing and energy of a video. With Soundstripe’s diverse range of over 50 genres, your users will have no trouble finding the perfect track to complement their visual narrative.

Effortless Music Selection

We have developed an intuitive search interface that allows users to explore music by title, genre, and description effortlessly. Once users have found their desired song, they can easily position any section of the track over their video. Furthermore, we are delighted to provide every one of our customers with seven complimentary sample songs, allowing them to get a taste of the fantastic library available. To unlock the full audio library, simply head over to Soundstripe and request an API key.

We are eagerly looking forward to working closely with Soundstripe to deliver the most exceptional video editing experience on the market. We invite you to try out VideoEditor SDK in your Android or iOS app and witness the transformative impact it will have on your users’ creativity. Together, we can take your video editing capabilities to new heights.

Thanks for reading! Stay ahead of the curve with our newsletter and sign up now!

CE.SDK v1.11 Release

Neslihan — Tue, 16 May 2023 15:37:49 GMT

We are introducing CE.SDK v1.11! Elevate your editing game with new features like Cutout Lines, Advanced Text Styling, HEVC/H.265 codec, and transparent video support. We’ve also improved asset resource management, and added user interfaces to enhance your template creation and editing workflow. See how CE.SDK v1.11 can bring your creative vision to life.

With this release, you can:

Create Cutout Stickers and Shapes
Choose Letter Cases
Keep Font Size Consistent in Templates
Enhance and Highlight Text
Ensure Effortless Layouting with Alignment Options
Support More Video Codecs
Use Less Memory
Enjoy Smoother Editing

Create Cutout Stickers and Shapes

Interfaces: API, Default UI, Advanced UI
Platforms: Web, Server, iOS, Android

We are excited to announce that CE.SDK now supports cutout lines, which includes a new type of block that enables users to create these lines and perform cutout line combinations. A future update will allow dynamic generation of such lines for existing blocks. This feature is based on the technology used for named/spot colors and is available for use with a range of printers.

Create Cutout and Adjust Offset
By providing precise control over the offset, we enable print businesses to ensure accurate and high-quality output for their users. Empower your users to create designs with exceptional precision, perfect for applications like stickers.

Combine Cutout Shapes (Boolean Operations)
These boolean operations provide designers with the ability to manipulate and combine Cutout shapes in various ways, enabling the creation of complex designs and precise control over the final output.

Union: Combines two or more shapes into a single shape, creating a merged outline of the combined areas.
Subtract: Removes one shape from another, resulting in a new shape formed by subtracting the overlapping region.
Intersect: Creates a new shape that represents the overlapping area between two or more shapes.
Exclude: Achieve artistic flair by excluding the overlapping area between shapes, resulting in a unique and captivating composition.

Visit our documentation on using cutouts to get started.

Choose Letter Cases

Interfaces: API, Advanced UI, Default UI
Platforms: Web, iOS, Android, Server

In our Advanced Text Styling, users can now adjust multiple new settings to enhance the visual appeal of their designs:

Letter case settings, including uppercase, lowercase, and title case.

Using the new styling settings enhances your design’s visual appeal while reducing implementation time and effort.

Keep Font Size Consistent in Templates

Interfaces: API, Default UI, Advanced UI
Platforms: Web, iOS, Android, Server

CE.SDK’s new Automatic Font Size feature adjusts font size to fit within text frame constraints, enabling users to create visually appealing designs with ease. For example, a text placeholder for names will now adjust the font size for longer names, ensuring they fit perfectly within a print design. Users can now enter as much text as they need, without worrying about it overflowing out of the designated frame.

Enhance and Highlight Text

Interfaces: API, Advanced UI, Default UI
Platforms: Web, iOS, Android, Server

Easily manipulate individual segments of text within a text block with Text Run. That means, you can modify the formatting of specific words or phrases, such as font weight and changing colors, without having to manipulate the entire text block. This feature is particularly useful for creating visually appealing text effects or highlighting specific words or phrases within a larger body of text.

Ensure Effortless Layouting with Alignment Options

Interfaces: API, Advanced UI, Default UI
Platforms: Web, Server, iOS, Android

We are introducing the Block Align feature, which transforms the editing experience by enabling users to effortlessly align blocks to the page or each other, empowering them to achieve their desired layout with ease. Read our documentation on Block Alignment for more details.

Support More Codecs and Transparent Videos

Interfaces: All
Platforms: Web

We have added support for hardware-accelerated video decoding of the HEVC/H.265 codec. This new feature will allow users to decode high-quality videos with better compression and smaller file sizes. It enables CE.SDK to handle a wider range of video files with improved efficiency.

Please note: At the moment, we do not support transparent videos on some major platforms due to a limitation on Windows. However, we’re excited to announce preliminary MOV container support, allowing you to work with a wider range of video files.

Use Less Memory

Loading and unloading assets, including images and fonts, has been updated to allow for the unloading of unused items. This unused asset usage optimization frees up memory and reduces the engine’s memory footprint, resulting in more stable and longer editing sessions.

Smoother Editing

This update addresses issues with the undo/redo stack not being built properly in certain scenarios, which could cause delays in updating the video timeline. With these improvements, users can now benefit from a smoother and more efficient editing experience.

Thanks for reading. Visit our changelog for more details and start creating with confidence.

IMG.LY Partners with Getty Images to Revolutionize Image Integration

Klaudia — Wed, 03 May 2023 13:07:31 GMT

We are thrilled to announce our latest partnership with Getty Images, the world’s leading provider of images, clips, and illustrations for visual communication. This collaboration has brought forth the Getty Images Library SDK - a bridge that seamlessly integrates image searching, licensing, and editing into your application. This revolutionary toolkit grants access to millions of stock images and enables your teams and users to browse, license, and edit images without ever leaving your application or website.

By integrating Getty Images Library SDK into your application, you can save precious resources, streamline your workflows, and ensure legal compliance with a broad range of licensed stock images at your fingertips. Imagine not having to switch between different platforms to search, license, and edit images – all while staying in your application.

Empower your Developers and Users

At IMG.LY, we are passionate about delivering powerful editing tools with an intuitive interface, providing a creative outlet for users of all skill levels. Teaming up with Getty Images, we strive to provide a toolkit that empowers content creators to communicate their message in the most efficient and effective way possible, while upholding Getty Images’ commitment to quality and diversity in visual images.

Designed by developers, for developers, the integration is quick and easy to set up. Once integrated, users can effortlessly browse through images, license the visuals they want, and make quick edits such as cropping or rotating images, all within the same application.

We are excited to share this news and look forward to working with Getty Images to bring tools and empower creative expression!

Get Started

Learn more about the Getty Images Library SDK, how to integrate it, or contact Getty Images to obtain a license key.

How To Resize Images in Flutter

Natalia — Tue, 27 Sep 2022 15:03:53 GMT

Static images are a core part of mobile applications. Usually, you store them in the directory in their original sizes, which requires you to adjust the sizes now and then, depending on where they are displayed. This article discusses how to resize images in Flutter and adjust their width, height, and size with efficient lines of code.

Resizing Methods

For image editing tools, Flutter offers an excellent and easy-in-use Boxfit property that works within the fit parameter, and you can easily integrate it into your application.

child: Image.asset( 'images/pexels.jpg', fit: BoxFit.cover)

Depending on your needs, you can choose between multiple attributes. For example, .cover and .fitHeight properties are similar when called, and both result in maximum frame coverage. Yet, although the image is widened proportionally, these methods can significantly affect the quality of crop image borders if they overfill. On the contrary, with .fitWidth or .scaledown, the asset is resized to the container’s width boundaries, which could be helpful when multiple images are needed to be displayed simultaneously. If you are dealing with a static image of small size, you can also employ .fill attribute that helps stretch your assets without cropping any critical information.

Alternatively, by calling .scale 0.5 the fit parameter will return you a graphical asset based on the scale you define. Note that any value less than 1 would reduce the image size.

Adjusting Image Size with Flutter

Now, let’s look at how Flutter allows applying these image manipulation techniques. The crucial difference from the other frameworks is that in Flutter, images should be stored in a specific folder. In other words, once you upload graphic files, go to the pubspec.yaml file and add the path to the directory around under assets(Note: by default the dependencies corresponding to ****assetsare usually placed around 45-46th lines in the code and need to be uncommented first by eliminating # and one space).

# Don't forget to define the assets folder in your directory
# To do so, find lines 45 - 46 in the **pubspec.yaml** file
# and uncomment the assets section like this:
assets:
  - images/ #Note that this should correspond to the name of your folder

Alternatively, you can specify the path to each image in the list:

assets:
  - assets/images/your_image.jpg
  - assets/images/your_image2.jpg

When the image is uploaded to Flutter, it seeks to occupy as much size as possible. But let’s presume that we have a “frame” or container that differs in size from the visual asset. One way to see the differences in sizes of both canvases is to color one frame, like color: Colors.indigo. Then we provide Flutter with the size specification and render our image in a child node child: Image.asset(‘images/pexels.jpg’). Thus, now we can see the image is placed inside the frame, and we need only to assign one of the filling methods (for example, fit: BoxFit.cover) discussed above.

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';

void main() {
  runApp(const ImgApp());
}

class ImgApp extends StatelessWidget {
  const ImgApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      debugShowCheckedModeBanner: false,
      home: const ResizePage(),
    );
  }
}

class ResizePage extends StatelessWidget {
  const ResizePage({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text('How to Resize Images'),
          backgroundColor: Color(0xffe55586),
        ),

        body: Center(
            // Enabling the Image Frame
            child: Container(
                color: Colors.indigo, // To see the difference between the image's original size and the frame
                height: 550,
                width: 300,

                // Uploading the Image from Assets
                child: Image.asset(
                  'images/pexels.jpg',

                  // Resizing the Image to the Frame Size
                  fit: BoxFit.cover,
                ))));
  }
}

You can find the associated to this tutorial Git repository ― here. Otherwise, run the following command to clone the complete code to your system:

git clone <https://github.com/nataliakzm/Resizing_Images_with_Flutter>

Conclusion

This article covered Flutter’s approach to the image resizing problem and saw how to use it in practice. However, despite the diversity of suggested methods that could fulfill basic users’ needs, Flutter fails to provide a comfortable integration. Just imagine you are building an app similar to Instagram in its functionality and you need not only upload various images from different assets but also resize them differently. If you want to avoid spending your time writing and maintaining a ton of code, then you may consider using PhotoEditor SDK in your next project.

PhotoEditor SDK is available for various frameworks: first, read this article from the official documentation to set up dependencies for your Flutter-based project. You can also follow our guide on how to integrate video editor for Flutter into your app.

In case you encounter any difficulties with the installation process, don’t hesitate to contact our support who will be happy to help you. Then, as shown in the example below, you can efficiently resize, crop, rotate or flip your visual assets with the transform tool.

How to Crop and Trim Videos in Flutter

Michael H. — Tue, 27 Sep 2022 08:45:16 GMT

If you are looking for a package that crops and trims videos in Flutter, you must have already come across the video_trimmer Flutter package. This package can trim videos but does not provide video cropping (at least not out-of-the-box). In fact, none of the packages on pub.dev, as of today, allow cropping a video in Flutter. If you have dug deeper, you might have come across FFmpeg — a powerful video editing command line tool, that is not the easiest to get started with. See the Ultimate Guide to FFmpeg for help.In this article we will use the FFmpeg library to crop and trim a video in a Flutter project.

Here is a list of packages that we will be using; ffmpeg_kit_flutter package for cropping and trimming videos, the path_provider package to get the path to the application or external directory where the video files will be stored, and the video_player package to play the video preview.

This article will not cover building any UI for cropping and trimming. But, it will discuss in brief how to crop the video using the video_trimmer library as well. If you want to package this workflow for production, check out our guide on running FFmpeg inside a Docker container. Also, you can try out the code used in this article from this GitHub repository.

Get Started

The end result of this tutorial will be a simple app that would look like the following screenshot.

When you tap the Save Video, the preview will refresh with the cropped and trimmed video replacing the original video. Follow the instructions in this article, and you will be able to develop a similar Flutter app.

First, let us start by adding dependencies to a new Flutter project.

Add Dependencies

In your project’s pubspec.yaml file add the following 3 dependencies.

dependencies:
  ...
  ffmpeg_kit_flutter: ^4.5.1
  path_provider: ^2.0.11
  video_player: ^2.4.6

Then, execute flutter pub get command from the project folder root.

Set Minimum SDK Version and Platform Version

The ffmpeg_kit_flutter plugin runs on Android SDK API level 24+ and iOS SDK 12.1+. Therefore, modify the module level build.gradle file to declare the minSdkVersion.

android {
  defaultConfig {
    ...
    minSdkVersion 24
    ...
  }
}

Then, modify the Podfile to declare the minimum global platform.

# Uncomment this line to define a global platform for your project
platform :ios, '12.1'

Add a Video Asset

For simplicity, we will be using a video asset instead of implementing a file picker. Choose a video file you would like to work with or download this sample video file. Next, copy the video file to a new directory named assets at the root of your project folder.

And then reference the video file from the pubspec.yaml file.

flutter:
  assets:
    - assets/file1.mp4

Execute flutter pub get again.

Having added the required dependencies, let us move on to coding.

Implement Video Crop and Trim

Replace the code in your main.dart file with the following code in the code block. This will be our starter code.

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Crop and Trim Demo',
      home: const MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({Key? key}) : super(key: key);

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text("Flutter Crop and Trim"),
      ),
      body: Column(
        children: [

        ],
      ),
    );
  }
}

Initialize the Input Player

Add the following code to the _MyHomePageState class.

import 'dart:io';

...
import 'package:flutter/services.dart';
import 'package:path_provider/path_provider.dart';
import 'package:video_player/video_player.dart';

...

class _MyHomePageState extends State<MyHomePage> {
  late String inputPath;
  VideoPlayerController? controller;
  String outputPath = "";

  ...

  @override
  void initState() {
    super.initState();
    copyVideoToApplicationDirectory().then((path) async {
      inputPath = path;
      controller = VideoPlayerController.file(File(inputPath));
      await controller?.initialize();
      await controller?.play();
      setState(() {});
      outputPath = await getOutputPath();
    });
  }

  ///Copy input file to ApplicationStorage Directory
  ///returns path to copied video
  Future<String> copyVideoToApplicationDirectory() async {
    const filename = "file1.mp4";
    var bytes = await rootBundle.load("assets/file1.mp4");
    String dir = (await getApplicationDocumentsDirectory()).path;
    writeToFile(bytes, '$dir/$filename');

    return '$dir/$filename';
  }

  ///Write to Path.
  Future<void> writeToFile(ByteData data, String path) {
    final buffer = data.buffer;
    return File(path).writeAsBytes(
        buffer.asUint8List(data.offsetInBytes, data.lengthInBytes));
  }
}

The copyVideoToApplicationDirectory method is copying the video from Assets to the Application directory on the phone’s file system. The resulting path is then stored in the inputPath variable which is then supplied to the FFmpeg command. For large‑scale workloads, you can move this FFmpeg command to AWS Spot Instances; our AWS guide shows how to configure a cloud environment for scalable video processing.

Next, add the VideoPlayer widget to the Column widget.

body: Column(
  children: [
    (controller != null)
        ? AspectRatio(
        aspectRatio: controller!.value.aspectRatio,
        child: VideoPlayer(controller!))
        : const SizedBox(),
  ],
),

Run the app and the video should start playing.

In the next step, we will add the FFmpeg command, but before that add the following 2 more methods to the _MyHomePageState class.

...

import 'package:ffmpeg_kit_flutter/ffmpeg_kit.dart';
import 'package:ffmpeg_kit_flutter/log.dart';
import 'package:ffmpeg_kit_flutter/return_code.dart';

...

class _MyHomePageState extends State<MyHomePage> {
  ...

  /// Output path with a file name where the result will be stored.
  Future<String> getOutputPath() async {
    final appDirectory = Platform.isAndroid
        ? await getExternalStorageDirectory()
        : await getApplicationDocumentsDirectory();
    final externalPath = '${appDirectory?.path}/out_file.mp4';
    return externalPath;
  }

  ///Executes the FFMPEG [command]
  ///Note: Green bar on the right is a Flutter issue. <https://github.com/flutter/engine/pull/24888>
  ///Should get fixed in a 3.1.0+ stable release <https://github.com/flutter/engine/pull/24888#issuecomment-1212374010>
  Future<void> ffmpegExecute(String command) async {

    final session = await FFmpegKit.execute(command);

    final returnCode = await session.getReturnCode();
    if (ReturnCode.isSuccess(returnCode)) {

      print("Success");
      //Replace the preview video
      await controller?.pause();
      await controller?.dispose();
      controller = VideoPlayerController.file(File(outputPath));
      await controller?.initialize();
      await controller?.play();
      setState(() {});

    } else if (ReturnCode.isCancel(returnCode)) {

      print("Cancel");

    } else {

      print("Error");
      final failStackTrace = await session.getFailStackTrace();
      print(failStackTrace);
      List<Log> logs = await session.getLogs();
      for (var element in logs) {
        print(element.getMessage());
      }

    }
  }
}

The getOutputPath method provides the path where the resulting video will be saved. The outputPath will be passed to the FFmpeg command whereas the ffmpegExecute method is where the FFmpeg magic takes place.

The ffmpegExecute method expects a valid FFmpeg String command. The String command is then passed to the FFmpegKit.execute method which returns an instance of FFMpegSession . It will tell us whether our command was executed successfully or not. If it was not executed successfully, we can extract error logs from it.

FFmpeg Execute Command

Add a TextButton from where the FFmpeg command shall be sent.

TextButton(
  child: Text('Save Video'),
  onPressed: () async {
    //TODO: Call FFMPEG Execute
  },
),

Next, call the following ffmpegExecute method from the onPressed property.

ffmpegExecute('-ss 0:00:15 -to 0:00:45 -y -i $inputPath -filter:v "crop=320:150" -c:a copy $outputPath');

Run the app and tap the Save Video button. You will notice that the player is now playing the new cropped and trimmed video.

If you were able to successfully follow the instructions up till here, it is time to dig a little deeper into the FFmpeg command that we just ran earlier.

Understanding the FFmpeg Command

Because command FFmpeg is a command line tool, it expects string-only commands. For this reason, we do not have any dart Classes, Methods, or Parameters to work with that we usually get when working with a dart plugin. Let us dissect the above command line command that is helping us crop and trim a video.

Command	Description
-ss 0:00:15	Seeks to position on the input video. The trim starts from this position.
-to 0:00:45	Stops reading at the position in the input video. The trim stops at this position. The total length of the video is 0:02:05.
-y	Overwrite output files. Helpful when the Save Video button is tapped more than once.
-i $inputPath	Input file location. This is the file on which the crop and trim are applied.
-filter:v	Apply a filter to the video stream.
”crop=320:150”	Apply a crop filter from the center of the video that is 320 pixels wide and 150 pixels tall. The original dimensions of the video were 320 x 240.
-c:a copy $outputPath	Specifies the codec with which the output file must be encoded. Here, copy is a special value to indicate the stream is not to be re-encoded. The a after the colon is a stream specifier for the audio stream.

That will be either the application directory or the external directory of the application, depending on whether the app is running on iOS or Android.

You can check out the app’s code and play with it by downloading it from this GitHub Repository. This project allows you to tweak the duration of the trim and dimensions of the crop, so it has more code, but at the heart of it, it still uses the above FFmpeg command.

Cropping Using the video_trimmer Plugin

As a bonus, here is a tip on how to use the video_trimmer plugin for cropping videos.

The video_trimmer plugin also uses FFmpeg at its core and allows us to pass on an FFmpeg command while saving the video. For this reason, it is an easy task for us to apply the crop to a video using the video_trimmer plugin.

To do this, pass the following command to the saveTrimmedVideo method’s ffmpegCommand parameter as shown in the code block below. The video_trimmer package expects the customVideoFormat parameter argument when the ffmpegCommand parameter is used.

await _trimmer
    .saveTrimmedVideo(
        startValue: _startValue,
        endValue: _endValue,
        ffmpegCommand:
            '-filter:v "crop=320:150"',
        customVideoFormat: '.mp4')
    .then((value) {
  setState(() {
    _value = value;
  });
});

Limitations of this Approach

The crop that we are applying is from the center of the frame. The present command would also need starting coordinates to crop a non-center frame.

The project will need a refined UI for cropping and trimming videos to offer a complete app experience to users. The current approach is miles away from creating that experience.

But perhaps the most significant limitation is FFmpeg’s licensing. FFmpeg is available with both LGPL and GPL licenses, so you must ensure your project is compatible with those licenses. For many commercial projects, this is a non-starter. Alternatively, you can run FFmpeg on Google Cloud Platform; our GCP tutorial walks you through setup.

Commercial Alternative

VideoEditor SDK (VE.SDK) from IMG.LY provides powerful video editing features, including cropping and trimming videos in a Flutter project. You will receive staples of video editing, including straightening videos, filters, brightness, color adjustments, and more. Follow our guide to learn how to integrate IMG.LY’s video editor for Flutter into your app.

Conclusion

While still a complex topic, video manipulation is more attainable to implement on Flutter than on native Android. Nevertheless, FFmpeg is the only open-source, free-to-use option to edit videos on Flutter right now. To automate cropping and trimming across many files or build a transcoding server, see our article on building a production‑ready batch video processing server.

How to Remove Backgrounds Using Core ML

Walter — Tue, 28 Jun 2022 15:26:53 GMT

In this tutorial, you will learn how to use machine learning to identify an image background and mask it out. This is particularly useful for making stickers or avatars or adding a fake background to a video call. The process of assigning the pixels in an image to a specific object is called “segmentation”. Apple provides an optimized method with pictures of people in its Vision framework. For performing the same tasks with non-human subjects, you can use the DeepLabV3 machine learning model with Core ML. The code examples in this tutorial have been tested using Xcode 13 and Swift 5. Because of the use of Core ML, Vision, and CoreImage in this tutorial, you should run the demo code on a device, not on the Simulator. An iOS project with the demo code is on GitHub.

Image segmentation is a different process and requires other machine learning models than image recognition. With recognition, the model produces bounding rectangles that the system believes to contain the entire object. With segmentation, the model identifies the actual pixels of the object.

In this tutorial, you’ll start with an image of your subject. Then you’ll generate a mask for the background using Vision. Finally, you’ll use CoreImage filters to blend the original image, image mask, and the new image background.

Whether using the Vision framework alone or supplementing Vision with another Core ML model – the process will be the same:

Create a Vision request object with some parameters.
Create a Vision request handler with the image to be processed.
Process the image with the handler and object.
Process the results into the final image using CoreImage.

When working with still images, Apple’s CoreImage framework is usually the best option, mainly because of the large number of available filters and the ability to create reusable pipelines that you can run on the CPU or the GPU.

Using Vision to Segment People

Without an external model, Vision can only segment documents or people in an image. It is vital to understand that Vision will only identify a pixel in the image as “this pixel is part of a person” or “this pixel is not part of a person.” If an image contains a group of people, Vision will not be able to separate individuals.

To segment the image, start by creating an instance of a VNGeneratePersonSegmentationRequest.

var segmentationRequest = VNGeneratePersonSegmentationRequest()
segmentationRequest.qualityLevel = .balanced

This type of request has a few options you can set. .qualityLevel can be .fast, .balanced or .accurate. The level of .accurate is the default. This will determine how closely the mask conforms to the boundaries of the original image. The different levels process at different speeds. The .fast setting is intended for use in a video so that frames don’t get dropped. Using .balanced or .accurate causes noticeable delay in an app processing the image on most devices. Experiment with different settings depending on your needs.

The output of the segmentation request will be a CVPixelBuffer. This is a structure that contains information for each pixel of the image. Most of Apple’s video frameworks as well as CoreImage can work with pixel buffers. The default for VNGeneratePersonSegmentationRequest is a buffer where the color of each pixel is represented by an 8-bit number. Any pixel that Vision thinks contains part of a person will be white and any not-a-person will be black and represented by zero. This will be exactly what you want for generating a mask to work with CIBlendWithMask.

Next, create a VNImageRequestHandler with the image to be processed. The handler class has a number of initializers for different types of data. In this example we will use CGImage, but you could also start with CIImage, CVPixelbuffer, Data, or others. You can also specify options for the handler. In this tutorial we will not specify any, but one of the options is to pass in a CIContext. This can help with performance as you can tell your app to do all of the processing with a single context on the GPU using Metal.

guard let originalCG = originalImage?.cgImage else { abort() }
let handler = VNImageRequestHandler(cgImage: originalCG)
try? handler.perform([segmentationRequest])
guard let maskPixelBuffer =
  segmentationRequest.results?.first?.pixelBuffer else { return }
let maskImage = CGImage.create(pixelBuffer: maskPixelBuffer)

In the code above, the originalImage (which happens to be a UIImage) gets converted to a CGImage. Then we use the image to initialize a request handler. The VNImageRequestHandler has a method .perform which takes an array of all of the Vision requests you want to use to process the image. The .perform method will not return until all of the requests in the array have been completed. Depending on how you want to structure your code, you can either provide a completion handler to use with each of the requests or just process the results in-line. In this example, we’ll process in-line.

If the segmentationRequest found any people in the image, the results array will contain a .pixelBuffer. Create the mask we need by converting the pixelBuffer into a CGImage. To convert to a CGImage, the example uses some helper methods that Matthijs Hollemans published to GitHub specifically for working with Core ML inputs and outputs.

Now that we have the mask image, use CIFilter to compose the final image. It will take a few steps. First, resize the new background, mask, and original image to the same size. Then, blend the three images.

//Convert main image to a CIImage and get the size
let mainImage = CIImage(cgImage: self.originalImage!.cgImage!)
let originalSize = mainImage.extent.size
//Convert the maskimage to CIImage and set the size
//to be the same as the original
var maskCI = CIImage(cgImage: maskImage!)
let scaleX = originalSize.width / maskCI.extent.width
let scaleY = originalSize.height / maskCI.extent.height
maskCI = maskCI.transformed(by: .init(scaleX: scaleX, y: scaleY))
//Convert the new background to a CIImage and set the size
//to be the same as the original
let backgroundUIImage = UIImage(named: "starfield")!.resized(to: originalSize)
let background = CIImage(cgImage: backgroundUIImage.cgImage!)
//Use CIBlendWithMask to combine the three images
let filter = CIFilter(name: "CIBlendWithMask")
filter?.setValue(background, forKey: kCIInputBackgroundImageKey)
filter?.setValue(mainImage, forKey: kCIInputImageKey)
filter?.setValue(maskCI, forKey: kCIInputMaskImageKey)
//Update the UI
self.filteredImageView.image = UIImage(ciImage: filter!.outputImage!)

The above code resizes the images to match the size of the original image and converts them to CIImage. Many machine learning models commonly resize inputs and outputs. For example, the output of the VNGeneratePersonSegmentationRequest using the demo image is 384x512.

Both CIImage and UIImage formats describe an image but do not always have a bitmap representation. That is why each image gets converted to a CGImage first. Though this guarantees that the demo code will work, converting formats makes the code run slower. In your app, you should experiment with different methods to get your images into CIImage format for filtering. Also, the above code uses Matthijs’ resized helper method to do some of the resizing.

With everything resized, the CIBlendWithMask filter stitches the images together. In place of the black pixels in the mask, the final image will show the background – for white pixels, the foreground. Wherever the mask image has a gray pixel, the final image will blend background and foreground.

Now let’s see how to use an additional CoreML model to process images that don’t have a person as the main subject.

Choosing a Model

Apple provides a number of pre-built models for text and image processing. The DeepLab v3 Machine Learning model can perform segmentation requests on images that have subjects like dogs. You can download it from Apple directly or also find it at the DeepLab repo on GitHub. Once you have downloaded the model, add it to your Xcode project the same as any other file. The DeepLab model will recognize people, the same as the VNPersonSegmentationRequest, but also other objects. This does come at a cost in an increased filesize for your application. You may notice that Apple provides multiple versions of the model of different file sizes. They all perform the same task, but differ in how they represent the output and a few other things.

Models on Apple’s website are packaged to work with Core ML and Xcode, so you can easily try a different model. The input and output method names will be the same. It is beyond the scope of this tutorial, yet Apple provides tutorials and example scripts on how to convert TensorFlow and other machine learning models to work with Core ML.

Core ML and Xcode

When you are working with a Core ML model, Xcode provides some convenient tools. Access them by highlighting the name of the model in the File Navigator pane of Xcode. For instance, clicking “Preview” will let you test the model with your data.

You can also use the “Predictions” tab to determine how the input needs to be formatted and what to expect for the output.

Here you can see that the DeepLabV3 model expects images to be 513 x 513 and will output a multidimensional array of integers that is 513 x 513 in size.

Finally, on this screen, you can see an entry for the “Model Class” in the headers. Double-click on the name to jump into the Swift wrapper class for the model to see how to call the model in your code and work with the inputs and outputs.

The DeepLabV3 Model

The DeepLab model input will be a color image that is 513 x 513 pixels. The Vision framework will handle resizing the input, but you can provide options on how that resize should work. The DeepLabV3 model has been trained to recognize and segment these items:

aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
dining table
dog
horse
motorbike
person
potted plant
sheep
sofa
train
tv or monitor

Anything that the model does not recognize, it will consider a background. After performing the recognition, the model returns a two-dimensional 513 x 513 array. Each entry in the array corresponds to one pixel in the original 513 x 513 input image. For instance, every pixel in the original image that shows a dog will be represented in the output by a 12 in the corresponding array entry. Any pixel that is not a recognized object will be given a 0 in the output array to represent a background.

If the model recognizes multiple objects, there will be multiple numbers in the array. If that is not what you want, you need to make changes to the array before creating the mask. For example, using an image of a dog riding a horse, some entries in the array will be 12, others will be 13, and the rest will be 0. To filter out the horse, you need to loop through the array and change any 13s to 0s.

Segmentation with DeepLab

To use the DeepLab model in your code, you again use a request to the Vision framework but this time you can specify a model.

let config = MLModelConfiguration()
var segmentationModel = try! DeepLabV3(configuration: config)
if let visionModel = try? VNCoreMLModel(for: segmentationModel.model) {
  self.request = VNCoreMLRequest(model: visionModel)
  self.request?.imageCropAndScaleOption = .scaleFill
}

In the code above, we create an instance of the DeepLab Core ML object and then initialize a generic VNCoreMLRequest with its model. The only option we set on the request is to tell it how to modify the image when it resizes it for input.

Now the code is very similar to the first example.

let cgImage = originalImage?.cgImage
let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
try? handler.perform([request])
if let observations = request.results as? [VNCoreMLFeatureValueObservation],
  let segmentationmap = observations.first?.featureValue.multiArrayValue {
  guard let maskUIImage = segmentationmap.image(min: 0.0, max: 1.0) else { return }
  applyBackgroundMask(maskUIImage)
}

Because we are using a generic VNCoreMLRequest we have to cast the results as VNCoreMLFeatureValueObservation. The DeepLab model returns a .multiArrayValue instead of a .pixelBuffer so we will again rely on the helper methods to convert it to an image. Now that the mask image has been created, applying the background is the same as in the original example.

Going Further

This tutorial focused on still images, yet both the standard Vision segmentation requests and DeepLabV3 allow processing video input as they can work with CVPixelBuffer and VNSequenceRequestHandler.
The rest of the process will be almost identical to our example. You will need to finetune the performance, or else you will experience frame drops.

Apple provides another method for identifying the background and primary subject in a still image: Portrait mode. When a device renders Portrait mode, it takes pictures with all cameras on the device and stitches them together. This way, the depth of field can change, and you can adjust what items are in focus or blurred.

If the DeepLab model does not cover your subject matter, you can train the DeepLabV3 model to recognize other objects using your own data.

However, you may consider using SDKs such as PE.SDK and VE.SDK. Enabling background removal for your users is easy — all it takes is a few lines in your configuration as described in the official PE.SDK documentation for iOS and Android, which adds a control to the photo editor to toggle the setting.

Integrate a fully customizable photo editor with Background Removal into your app with PE.SDK.

The latest VE.SDK release introduced Background Removal for stickers. This feature recognizes people in pictures and removes the background with one tap – no need for manual outlines or masks.

Let your users create beautiful visuals with Sticker Background Removal.

This feature is available on Android and iOS 15.0 and higher only. See the official documentation for Sticker Background Removal on Android or iOS.

Using an SDK will simplify and accelerate your app development, as you can save time and resources, and focus on the growth and innovation of your application instead.

Wrapping Up

In this tutorial, you saw how to use Vision and Core ML to segment an image of a person and remove the background. You also saw how to use DeepLab to work with images of non-persons.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.

PE.SDK and VE.SDK 10 for Android and 11 for iOS Release

Sascha — Wed, 04 May 2022 12:46:21 GMT

We are thrilled to announce the release of PhotoEditor SDK and VideoEditor SDK 10 for Android and 11 for iOS. This new major version is packed with features and improvements.

This release is adding:

Custom Watermarks
Background Removal for Photos
Background Removal for Stickers
Custom Sticker Libraries
Giphy Integration
And More Updates for Android and iOS

Custom Watermarks

With our new support for custom watermarks, you can add your company logo or any other image to photo and video exports. You can place your watermark in either corner or the center of your photo and video. Additionally, you can specify the size of the watermark and determine its distance from the outline if you choose to place it near a corner. Your users will not be able to remove nor modify the watermark.

For more details on Custom Watermarks, take a look at our documentation for Android or iOS.

Background Removal for Photos

You have unwaveringly requested this feature: we are excited to offer the automatic removal of backgrounds for static content! Background Removal is fully running on-device and currently specializes in images containing a person only. PE.SDK enables this feature exclusively when it detects a person in the photo.

The removed background will turn transparent if your selected output supports transparency, such as PNG. In other cases, such as JPG, your background removal will be tinted black.

This feature is available on Android and iOS 15.0 and higher only. Easily enable Background Removal for Photos with our Android guide or iOS guide.

Background Removal for Stickers

A more common use case for automatic background removal is with stickers, so we’ve also added this functionality. Select a sticker including a person and tap the Remove BG button. Simple!

This feature is available on Android and iOS 15.0 and higher only. Easily enable Background Removal for Stickers with our Android documentation or iOS documentation.

Custom Sticker Libraries

We wanted to make it easier for you to provide your users with more sticker tool content. So far, you were able to create multiple sticker categories with different stickers that could either be part of your app or stored on a server. Additionally, you could enable our personal stickers feature on iOS or Android), giving your users the option to select stickers from their camera roll.

We have now added more ways to provide your users with stickers.

Android

You can now create sticker categories that do not present our sticker selection interface, but will instead load any arbitrary Fragment. That gives you complete control over the sticker selection screen, and you can use it to load and display any content. When the user selects any of your stickers, all you have to do is pass it back to the SDK with a callback. Get started with your Custom Sticker Library with our Android documentation.

iOS

You can now create sticker categories that do not present our sticker selection interface, but will instead load any arbitrary view controller that conforms to the StickerCollection protocol. That gives you complete control over the sticker selection screen, and you can use it to load and display any content. When the user selects any of your stickers, simply pass it back to the SDK using a delegate method.
Additionally, we’ve created a new StickerProviderCategory that you can pass any object conforming to the StickerProvider protocol. This category provides most of the features required to interact with sticker content providers, such as search and pagination. You can use this category to very quickly integrate any sticker service.

Find more information on Custom Sticker Libraries in the iOS documentation.

GIPHY Integration

This online database and search engine for funny GIFs and beautiful animations is a staple in many social applications. Easily integrate GIPHY into your app:

Android

Using our new Custom Sticker Library, we’ve also created an integration for GIPHY on Android. All you have to do is head over to GIPHY, create your API key, pass it to your GiphySettings and create a GiphyStickerCategoryItem. Your users will then immediately have access to all the content GIPHY has to offer.

For more details, visit our documentation on adding GIPHY for Android.

iOS

Using the new StickerProviderCategory, we added GIPHY support on iOS. All you have to do is head over to GIPHY, create your API key, and pass this to the GIPHY sticker category. Your users will then immediately have access to all the content GIPHY has to offer.

For more details, visit our documentation here.

More Updates for Android

Additionally, we have:

Increased the minimum API level for PhotoEditor SDK from 16 (Android 4) to 21 (Android 5), which ensures support for 98% of all phones in use.
Added support for headless video exporting.
Increased video playback stability.
Fixed a number of issues with degraded audio quality in videos.
And added the ability to choose a theme at runtime.

More Updates for iOS

Additionally, we have:

Increased our deployment target from iOS 9 to iOS 13.
Migrated all usage of OpenGL to Metal.
Added a better result API.
Added support for SwiftUI.
Enhanced our video export to display a progress indicator and the option to cancel the export.

How To Apply Image Filters in WebGL

Antonello — Fri, 08 Apr 2022 13:06:16 GMT

In this article, you will learn how to apply custom filters to an image in WebGL. You can achieve this goal without using any external library. That is because the HTML5 canvas element natively supports WebGL and does not require the use of plug-ins.

As we have already covered in previous articles, <canvas> is a powerful tool that equips you with everything required to manipulate images. This is true regardless of the use of WebGL. So, you can filter images also without using WebGL.

Now, let’s see how to filter images in Vanilla JavaScript with WebGL in your browser. Follow this step-by-step tutorial and learn how to build the following demo:

Getting Started with WebGL

If you are not familiar with WebGL, you need to learn a few things before approaching it. To build an application in WebGL, start with the following three concepts.

1. Textures

Keep in mind that to draw an image in WebGL you have to use a texture. To use a texture, WebGL requires you to define the texture coordinates. These coordinates go from 0.0 to 1.0, regardless of the texture size.

2. Vertex shader

The vertex shader is a function you have to write in GLSL that is in charge of computing the vertex positions. Thanks to it, WebGL can rasterize the draw primitives, which include points, lines, and triangles. When rasterizing these primitives, WebGL calls another user-defined function called fragment shader. In other words, WebGL interpolates the values provided in the vertex shader function while it draws each pixel using the fragment shader function execution.

3. Fragment shader

The fragment shader is a function you have to write in GLSL whose goal is to generate a color for each pixel of the draw primitive currently being drawn. This function has little info per pixel, but you can provide it with everything required by using the varyings variables. These allow you to pass values from the vertex shader function to the fragment shader function.

Filtering Images with Kernels in WebGL with `<canvas>`

There are many ways to filter images, but the most common one involves the convolution operation. This is because when used on images, convolution applies a filter by taking the weighted sum of a square of pixels and assigning the resulting value to the current pixel. This logic is applied to every pixel the image consists of. Therefore, you can now imagine why convolution is one of the most relevant concepts when it comes to image processing.

The coefficients used to perform the weighted sum come from a matrix called kernel. The kernel represents the filter you want to apply through the convolution operation. So, by changing the kernel, the resulting image will change accordingly. Some kernels are more useful than others and can be used for blurring, sharpening, performing edge detection, and other operations. You can find a list of the most popular kernels on Wikipedia.

Now, let’s see how to implement kernel image filtering in WebGL.

Clone the GitHub repository that supports this article with the following command:

git clone https://github.com/Tonel/how-to-filter-an-image-in-webgl-imgly

Additionally, try the demo application by launching how-to-filter-an-image-in-webgl-imgly/index.html in your browser.

Otherwise, you can find the JavaScript function taking care of implementing the filter logic in WebGL below:

function filterImage(canvas, originalImage, kernel) {
  // assuming the kernel is a square matrix
  const kernelSize = Math.sqrt(kernel.length);
  const gl = canvas.getContext('webgl');

  // clearing the canvas
  gl.clearColor(1, 1, 1, 1);
  gl.clear(gl.COLOR_BUFFER_BIT);

  const vertexShaderSource = `
  attribute vec2 position;
  varying vec2 v_coordinate;
  void main() {
    gl_Position = vec4(position, 0, 1);
    v_coordinate = gl_Position.xy * 0.5 + 0.5;
  }
`;

  const fragmentShaderSource = `
  precision mediump float;
  // the varible defined in the vertex shader above
  varying vec2 v_coordinate;
  uniform vec2 imageSize;
  uniform sampler2D u_texture;
  
  void main() {
    vec2 position = vec2(v_coordinate.x, 1.0 - v_coordinate.y);
    vec2 onePixel = vec2(1, 1) / imageSize;
    vec4 color = vec4(0);
    mat3 kernel = mat3(
      ${kernel.join(',')}
    );
    // implementing the convolution operation
    for(int i = 0; i < ${kernelSize}; i++) {
      for(int j = 0; j < ${kernelSize}; j++) {
        // retrieving the sample position pixel
        vec2 samplePosition = position + vec2(i - 1 , j - 1) * onePixel;
        // retrieving the sample color
        vec4 sampleColor = texture2D(u_texture, samplePosition);
        sampleColor *= kernel\[i\][j];
        color += sampleColor;
      }
    }
    color.a = 1.0;
    gl_FragColor = color;
 }
`;

  const vertexShader = compileShader(gl, gl.VERTEX_SHADER, vertexShaderSource);
  const fragmentShader = compileShader(
    gl,
    gl.FRAGMENT_SHADER,
    fragmentShaderSource
  );

  // iniziailing the program
  const program = createProgram(gl, vertexShader, fragmentShader);

  const positionAttributeLocation = gl.getAttribLocation(program, 'position');

  const imageSizeLocation = gl.getUniformLocation(program, 'imageSize');

  // binding the position buffer to positionBuffer
  const positionBuffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, positionBuffer);

  // using the program defined above
  gl.useProgram(program);
  // enabling the texcoord attribute
  gl.enableVertexAttribArray(positionAttributeLocation);
  // setting up the size of the image
  gl.uniform2f(imageSizeLocation, canvas.width, canvas.height);
  // telling positionAttributeLocation how to retrieve data out of positionBuffer
  gl.vertexAttribPointer(positionAttributeLocation, 2, gl.FLOAT, false, 0, 0);
  // provide the texture coordinates
  gl.bufferData(
    gl.ARRAY_BUFFER,
    new Float32Array([-1, -1, -1, 1, 1, -1, 1, 1, 1, -1, -1, 1]),
    gl.STATIC_DRAW
  );

  // loading the original image as a texture
  const texture = gl.createTexture();
  texture.image = new Image();

  // setting the anonymous mode
  // Learn more about it here:
  // https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/crossOrigin
  texture.image.crossOrigin = '';
  texture.image.src = originalImage.src;
  texture.image.onload = function () {
    gl.bindTexture(gl.TEXTURE_2D, texture);
    // setting the parameters to be able to render any image,
    // regardless of its size
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
    // loading the original image as a texture
    gl.texImage2D(
      gl.TEXTURE_2D,
      0,
      gl.RGBA,
      gl.RGBA,
      gl.UNSIGNED_BYTE,
      texture.image
    );
    gl.drawArrays(gl.TRIANGLES, 0, 6);
  };
}

function compileShader(gl, type, shaderSource) {
  const shader = gl.createShader(type);
  gl.shaderSource(shader, shaderSource);
  gl.compileShader(shader);

  const outcome = gl.getShaderParameter(shader, gl.COMPILE_STATUS);

  if (outcome === false) {
    // logging the error message on failure
    console.error(gl.getShaderInfoLog(shader));
    gl.deleteShader(shader);
  }

  return shader;
}

function createProgram(gl, vertexShader, fragmentShader) {
  const program = gl.createProgram();
  gl.attachShader(program, vertexShader);
  gl.attachShader(program, fragmentShader);
  gl.linkProgram(program);

  const outcome = gl.getProgramParameter(program, gl.LINK_STATUS);

  if (outcome === false) {
    // logging the error message on failure
    console.error(gl.getProgramInfoLog(program));
    gl.deleteProgram(program);
  }

  return program;
}

The filterImage() function is where the magic happens. It takes the following three parameters:

canvas: an Element object representing an HTML canvas element
originalImage: an Element object representing an HTML img element
kernel: an array containing the values of the kernel to use in the convolution operation

The first part of the function takes care of extracting the WebGLRenderingContext object representing a three-dimensional rendering context from the canvas element. Then, the vertex shader and fragment shader functions are defined as strings written in GLSL. Next, they are compiled and finally used to create a WebGL program.

The last part of the function creates a texture starting from the original image passed as a parameter and passes it to the WebGLRenderingContext to produce the final result. This represents the filtered image and is finally displayed by the browser in the canvas element.

The results produced by the filterImage() function depend on the kernel chosen, as you can verify by playing with the live demo you can find at the beginning of the article.

Et voilà! You just learned how to filter images in WebGL!

Final Considerations

As shown above, you can filter images with WebGL in Vanilla JavaScript with a hundred lines. At the same time, this cannot be considered an easy task to achieve. The reason is that getting into WebGL takes time and effort. Plus, you have to learn how to use GLSL to write the vertex shader and the fragment shader functions. So, things can get more complicated than expected.

Plus, you learned how to perform non-complex filters, but several filtering techniques are complex and do not involve the convolution operation. That also means that implementing them can be challenging and result in inefficient algorithms.

To avoid a headache, you should consider a complete and all-in-one solution like PhotoEditor SDK. Commercial solutions make things easier and shield you from all difficulties, offering features that would be complex, time-consuming, and challenging to implement. That is particularly true when it comes to using WebGL, which is PhotoEditor SDK’s main renderer.

You will be able to harness WebGL’s power without writing a single line of GLSL or knowing about its existence. WebGL will be used behind the scene for you! Keep also in mind that you would not be alone, since developers at IMG.LY are happy to provide support.

Filtering Images With PhotoEditor SDK

Read the article from the official documentation to learn how to get started with PhotoEditor SDK in HTML and Vanilla JavaScript. In detail, the Filters feature gives you more than 60 high-quality filters to play with. Achieve the following result with a few clicks:

Check out this feature on the PhotoEditor SDK demo page.

Conclusion

In this article, we looked at how to filter an image in WebGL with the HTML canvas element. Implementing a feature allowing users to filter images through a kernel-based approach in WebGL involves only a dozen of lines of code. However, understanding how WebGL works and coding in GLSL cannot be considered easy tasks.

As a result, you might want to avoid dealing with WebGL entirely. In this case, consider a commercial and easy-to-adopt solution using WebGL behind the scene – such as PhotoEditor SDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.‌

CE.SDK v1.4.0 Release

Daniel — Tue, 22 Feb 2022 12:34:56 GMT

Six weeks have passed since our last release, where we shipped our new and improved default UI, Placeholders, and a preview for our new Dashboard.

This time, we focused heavily on making content discovery and provisioning better for you and your customers, as well as automation and API-driven design. Our release includes:

Custom Asset Libraries & Management
Internationalization
Headless Mode & APIs
Other
Outlook on upcoming releases

Asset Libraries & Management

With this release, we improve our library system and overall usability. CE.SDK 1.4.0 introduces a new toolbar to let your users quickly access their favorite images, shapes, stickers, or text assets.

Even more importantly, we added a General Asset Source API that allows you to add any external library sources you like. This way, you can provide custom assets to your users, such as images, stickers, or anything else.
Check out our Documentation on Custom Asset Source Integration.

Internationalization (i18n)

CE.SDK now supports the easy translation of the default UI into any language, including full i18n support. We ship with English and German locales, but have successfully tested it with other languages.
Learn more about adding translations to CE.SDK in our Documentation.

Headless Mode & APIs

We are finally extending CE.SDK with a full headless mode that is usable without a user interface in a browser. You can use it to load, modify, and export scene templates programmatically without the need for any user interaction. Use it to create previews of your scenes with custom user content without opening the editor, or export your designs in multiple resolutions on demand.

Included APIs

Scene API lets you create, load, save, or export scenes to a rasterized image format.
Block API lets you create, read, update and delete any item (so-called blocks) in the scene; modifying properties like position, size, or content is guaranteed easy.
Layouts API lets you use advanced layout capabilities: choose from modes like absolute positioning, relative positioning in percent relative to the parent, or even auto layout. An even auto-layout lets you auto-adapt the size of a block according to its parent size and siblings.

Other

Snapping now works with multiple selected items.
Slider input values now allow manual input to give you more control.
Easy asset library access in the UI.
Changing content in Preview Mode to preview your template settings does not alter your template anymore.

Upcoming

We are currently working hard to improve the overall CE.SDK Editor experience for you and your customers. That includes adding features like strokes, gradients, grouping, but also improving the rendering performance to create more complex designs. Additionally, we are improving asset discovery within the editor.

Excitingly, we are working hard to bring our VE.SDK video features to CE.SDK! In the near future, you can use videos and animated GIFs inside any template and export short videos from within CE.SDK for Web.

App Development – IMG.LY Blog

CE.SDK v1.44 Release Notes

Simplify Your Video Editing Workflow

Get Vibrant Colors with P3/Wide Gamut Support

Tailor the Android Canvas Menu to Your Workflows

Improvements

Upcoming

CE.SDK v1.33 Release Notes

Integrate a Design Editor on iOS and Android

Key Features

Use Plugins to Customize UI and Add Quick Actions

Add Voiceovers to Videos on iOS

Integrate a Photo Editor for iOS and Android

Key Features

Improvements

Designing A Timeline For Mobile Video Editing

I. Visualizing Time

Horizontal Direction

Cognitive Load vs. Discoverability

Adaptive Height

Units and Scale

II. Manipulating Time

Fingertip-sized

Handles

Scrolling and Zooming

Snapping

Scrubbing Preview

Try it out!

Background Removal in the Browser Using ONNX Runtime with WebGPU

Towards Real-time Background Removal in the Browser

Implementation Details

Evaluation

First Use – Neural Network Download

First Run – Neural Network Initialization / Compilation

Consecutive Runs – Neural Network Execution

Conclusion

Unleash Creativity with CE.SDK’s New Plugin System

Built for Modification

Unlocking the AI Revolution

Boosting Customer Autonomy

Accelerate Product Expansion

Key Concepts

What Can You Build with our Plugin System?

Custom Actions

Custom Tools

Custom Assets & Presets

Custom Libraries

Custom Editor Behavior

Custom User Feedback

What’s Next

CE.SDK v1.25 Release Notes

Recolor Images like T-shirt Mockups

Create Transparency with a Green Screen

User Interface Improvements

Audio Preview

Selection Frame Improvements

CE.SDK v1.19 Release Notes

Unify Design Power with Block Unification

Enhance Interaction with Mouse Wheel Support

Migrate to CE.SDK v1.19

New Showcase: Automated Resizing

New Showcase: Version History

Outlook: Offer TikTok-Inspired Video Editing

CE.SDK v1.17 Release Notes

Support WebP File Formats

Capture and Export Videos with Camera for Android

CE.SDK v1.16 Release Notes

Maintain Color Consistency with CMYK & Libraries

Improve Web Interaction with Touch Gestures

CE.SDK v1.14 Release Notes

Generate Thumbnails Fast for Smooth Performance

Enhance Templating with Improved Placeholders

Bring Creative Editing to Your macOS App

Expand Editing Capabilities: macOS & Catalyst Binding

IMG.LY Announces Open Source JavaScript Library for In-Browser Background Removal

In-Browser Background Removal

Data Protection

Seamless Integration with IMG.LY’s CE.SDK

Empower Yourself with IMG.LY’s Background Removal

Update: Node.js Support

Filtering Images with Kernels in WebGL with `<canvas>`