Malte – IMG.LY Blog

CE.SDK v1.7.0 Release and Roadmap

Malte — Mon, 08 Aug 2022 12:07:47 GMT

Today, we are thrilled to highlight the best new features and changes in CE.SDK since our v1.6.0 Release. We have also published our Product Roadmap for CE.SDK to share our vision and direction for upcoming releases. This roadmap will let you know ahead of time what features you can expect from us and accelerate your product planning and development. We are excited to hear your feedback and suggestions to build more powerful features for your application!

CE.SDK v1.7.0

This release includes:

Auto Close Option in Web UI Configuration for Asset Libraries
Custom URI Resolver
Improving Mobile Touch Support
Rotation of Groups and Hierarchies
Custom Asset Source UI
Disconnecting UI Variants from Roles
Asset Source ApplyAsset API
Blur API
Effects API
Image Fills API
Image Straighten API
Improved Group Selection and API
Scopes API
Shadows API
Zoom API

Auto Close Option in Web UI Configuration for Asset Libraries

There are no one-size-fits-all solutions for the behavior of asset library panels after an action was triggered. It depends on the use case and might vary between the panels and the available screen space. Thus, this is now configurable.

Custom URI Resolver

Custom URI Resolvers give you full control over how URIs should be resolved. You can create custom storage backends and even resolve different resolution images in different instances. Find more information in our documentation.

Improving Mobile Touch Support

Input handling was streamlined to handle both touch and mouse input efficiently. On touch devices, all handles maintain the recommended minimum size for touch input and are intelligently hidden to minimize overlap. Thanks to this mechanism, resizing and rotating are always a breeze.

Rotation of Groups and Hierarchies

The engine and user interface add options for rotating groups and design blocks with children. This allows changing the rotation of multiple elements at once quickly.

Custom Asset Source UI

Instead of fixed entry points such as “Images,” “Stickers,” or “Shapes,” we want to enable custom and flexible entry points. Until today, we tied them to a block type, which is currently only relevant when users add an asset to the scene.

Moving from Roles to Abilities: Disconnecting UI Variants from Roles

Before the change, the Advanced UI was tied to the creator role and the Default UI was tied to the Adopter Role. This is now no longer the case. The scopes and editing options are still controlled by the abilities of the current role.

Shadows in Advanced UI

With this update, we roll out the long-awaited ability to apply shadows to elements in our Advanced UI. Realistic drop shadow effects add more depth to your designs with a few clicks – no other editor is needed.

Blur API

We introduce new APIs to modify the blur for images and other blocks. This change allows programmatic access to add blur effects in automation cases and when building your own UI.

Effects API

We introduce new APIs to modify the effect stack for shapes and other blocks. This allows programmatic access to effects like adjustments, LUT filters, and many more.

Image Fills API

Currently, you can only access and use images with the image block. However, pages or vector shapes could use more than just a color or gradient fill. Therefore, we broaden the concept of fills by allowing every block to have an image fill, e.g., pages. This change will enable building a more natural photo- or video-editor experience. Documentation.

Image Straighten API

We are extending the Image Crop API to allow straightening, flipping, and rotating images during cropping. This change allows more fine-grained control.

Improved Group Selection and API

There are multiple options to select blocks inside a group:

selecting a group first and with each click select one block deeper in the hierarchy
selecting an element under the cursor directly, neglecting the hierarchy

Previously, we defaulted to different behaviors depending on the user’s role. Now, this is configurable and also customizable during runtime via APIs. Therefore, it’s up to the implementor how the selection should work.

Scopes API

We introduce new APIs to modify the scopes for blocks. This update allows programmatic access to change the scopes for users and allows control over the possible edits a user can make to a scene.

Shadows API

We introduce new APIs to modify shadows for images and other blocks. This change allows programmatic access to add shadow effects to be used in automation cases and when you build your UI.

Zoom API

We introduce new APIs to modify the current zoom level. This allows programmatic access to change the current zoom easily for automation use-cases or for building your own user interface.

Product Roadmap

As mentioned before, we are happy to publish our Product Roadmap to let you in on our new features and enhancements for CE.SDK. This way, we can elaborate our decisions and plans for the product and how you can benefit while allowing you to give us feedback and indicate the importance of particular features.
Now that you know what’s coming down the pike, you can prepare for the release of significant improvements and new features such as Video Support for Web or Native iOS Support well in advance.

CE.SDK v1.6.0 Release

Malte — Fri, 20 May 2022 09:21:48 GMT

Today, we are highlighting the best new features and changes in CE.SDK since our v1.5.0 Release six weeks ago. CE.SDK v1.6.0 is adding:

Linear Gradient Fills
Strokes, Outline, Insets, and Stroke Patterns
Blend Modes
Scale Groups, Multi-Selection, and Hierarchies
UI Template Library
Floating and Static Asset Library Panels
Placeholders API
Editing State APIs
API for Image Fit Modes

Linear Gradient Fills

With this update, we are introducing linear gradient fills. This feature creates a gradual blend between two colors, including transparency. Use the Color Picker to switch between solid color and linear gradient fill modes and determine the order of colors.

Strokes, Outline, Insets, and Stroke Patterns

This feature introduces strokes and stroke patterns. Strokes apply to any block, including images, text, and shapes. Use our APIs or UIs to set the stroke width, color, style, and more options affecting the stroke appearance.

Blend Modes

Blend modes are a popular feature in creative and image editing tools. They blend two blocks or layers to get different types of effects. This feature is available in both the API and the UI.

Scale Groups, Multi-Selection, and Hierarchies

Our engine and user interface now provide options to scale groups, multi-select and create design blocks with children. You can easily change the scaling of multiple elements at once. This also simplifies resizing and rescaling your template designs for other page and art board sizes.

UI Template Library

We are introducing the template library in all UIs. Allow users to switch between multiple templates while editing.

Floating and Static Asset Library Panels

For some screen sizes and use-cases, it might be tedious that the asset library floats over your canvas and covers your design. Therefore, you can now configure the asset library panel to be either floating or static. A static asset library won’t overlap with your design and adjusts to your screen size perfectly.

Placeholders API

CE.SDK 1.6 introduces an additional API that allows you to interact with all placeholders on the canvas. The API queries the number of placeholders in a template or document and lists all available placeholders in the current scene.

Editing State APIs

We extend the API to allow querying and modifying the current editing mode. That includes changing the on-canvas editing mode to Crop, Text, or others. You can also query the editor state for helpful information like the desired cursor type and rotation and the text cursor position. To observe these values more comfortably, an additional endpoint allows registering for callbacks whenever the editing state changes.

API for Image Fit Modes

The API now allows switching between fit modes for images:

Crop applies crop properties, and ignores the aspect ratio.
Cover resizes the image aspect-aware to fill the frame.
Contain allows resizing the image aspect-aware to fit into its frame.

CE.SDK v1.5.0 Release

Daniel — Thu, 14 Apr 2022 12:53:45 GMT

Another six weeks have passed since our v1.4.0 release that brought an updated asset library and headless APIs. Since the initial CE.SDK release, we’ve been listening to your feedback and continuously releasing updates to improve and expand CE.SDK to better serve your needs. Here’s a short overview of what’s happened in the last quarter.

CE.SDK v1.5.0 is adding:

Hierarchies (Groups)
Template Imports & Exports
More Performance, More Power
High-Performance Rendering
Improved Touch Support
PDF Export
Subscription/Events API

Hierarchies (Groups)

With this release, we finally extended our flat hierarchy model to the typical parent-child relationship model between items on the canvas. That allows you to build an even more complex design: Group items to form a single unit and handle those items as one or control their opacity.

Template Imports & Exports

Export your template scenes as .scene-files that pack all assets that were used. You can also share your template by URL - all dependencies will resolve accordingly.

More Performance

We have replaced our underlying rendering engine with a new high-performance implementation, resulting in the performance of distinct scenarios improving by 100%. Likewise, as a result of polishing up various logic, CE.SDK runs faster than ever.

Improved Touch & Mobile Support

After evaluation, CE.SDK v1.5.0 expands upon our mobile advancement efforts by improving our web UI for touch inputs on mobile devices.

PDF Export

CE.SDK now supports PDF export. Vector paths and images are embedded, and rasterization will only be applied when necessary. Scenes of multiple pages can be exported as a multipage document.

Build Your Custom UI

We vastly improved support for building custom UIs on top of CE.SDK engine:

The CreativeEngine offers a powerful set of APIs to inspect, modify and interact with a scene.
A new set of UI-specific APIs, including history and zoom management, was added as a preview.

Events API

An Event API allows subscribing to scene changes and updates custom UI components accordingly.

Other

For an even smoother UI experience, we have improved the visual quality of element state indicators, and they now respect the element order.

From 2D to 3D Photo Editing

Malte — Tue, 26 Jun 2018 00:00:00 GMT

Last November, we released Portrait, an iOS app that helps create amazing, stylized selfies and portraits instantly.

With over a million downloads and many more portrait images created, we feel that the idea and vision of Portrait was more than confirmed. The central component of Portrait is an AI that is trained to clip portraits from the background, a technique we are eager to further improve and refine. In fact, Portrait helped us to explore a novel technique for image editing, as we were able to leverage a new powerful data set in photography: depth data.

We began feeding our AI models with the depth data from the iPhone Xs TrueDepth camera and had one goal in mind: to infer depth information for portrait imagery, or bringing three-dimensionality into a two-dimensional photo. Along the way, we created a new architecture concept, that allows performance and memory improvements through modularizing and reusing neural networks.

In the following article, we’d like to present some of our results along with the insights we made.

The New Cool: Depth Data

The usage of depth data in image editing initially became available with the iPhone 7 Plus when Apple introduced ‘Portrait Mode’. By combining a depth map and face detection, the devices are able to blur our distant objects and backgrounds, mimicking a ‘bokeh’ or depth of field effect, which is well known from DSLRs cameras.

While the actual implementation varies, all major manufacturers nowadays offer a similar mode by incorporating depth data into their image editing pipeline. This is either achieved through the conventional dual or even triple camera on the back of a phone, dual-pixel offset calculations combined with machine learning or dedicated sensors like Apples TrueDepth module. In fact, for a modern flagship phone, some sort of depth based portrait mode is almost a commodity.

From a developers perspective, things look a little different: Depth data became a first-class citizen throughout the iOS APIs in iOS 11 and such data is now easily accessible on supported devices. Android users obviously have access to depth data as well, either by utilizing multiple cameras or by Googles dual-pixel based machine learning approach, seen in the newer Pixel 2 phones. But contrary to iOS, Android doesn’t yet offer a common developer interface to access such data. In fact, developers aren’t able to access any of the depth information Google or other manufacturers collected within their camera apps. This means developers would either need to implement the algorithm to infer depth from two images themselves or try to rebuild Googles sophisticated machine learning powered system. Neither of these options is practical and probably not even possible given the usual limitations to camera APIs.

So although being quite common, depth data isn’t as easily accessible for developers as one might think. Right now you’re out of luck on Android, dependent on hardware on iOS and even then limited to the 1.000$ flagship if you’re interested in depth for images taken with the front camera. And last but not least, across all devices and platforms, there is no way for you to generate a depth map for an existing image.

Deep Possibilities

Despite the restrictions, we decided to first explore the power of depth for image editing, as depth data provides many new exciting creative possibilities:

If we have a depth map for a given image, our editing possibilities are increased dramatically. Instead of a 2D image, a flat plane of color values, we suddenly have a depth value for each individual pixel, which translates into a 3D landscape highlighting distinct objects in the foreground and a clear indication of background.

Depth-aware Editing

Instead of relying on color and texture differences to determine fore- and background, one could literally edit these regions individually. This allows adjustments like darkening the background while lightening the foreground, which makes portraits ‘pop’. If we’d be able to generate a high-resolution depth map, we could easily replace the AI currently used in Portrait and would allow even more sophisticated creatives. Thanks to the new APIs, there are already some awesome iOS apps available that specialize in depth based editing. One famous example is Darkroom with their “depth-aware filters”:

Depth of Field Effects

As a depth of field or bokeh effect was the initial motivation for Apple to incorporate depth sensing technology, it is one of the most obvious applications. Depth is crucial for such an effect, as the amount of bluriness of any given region directly depends on its distance to the camera lens.

3D Asset Placement

As mentioned above, a depth map gives us a 3D understanding of the image. We’re able to tell if subject A is positioned in front of or behind subject B. This allows placement of digital assets like stickers or text in a ‘depth-aware’ fashion, but could also be used to apply ‘intelligent’ depth of field, e.g. a bokeh effect that ensures all faces are in focus.

Enter Deep Learning

Motivated by the possibilities enabled by depth maps, we were wondering if we could bring this magic to any type of portrait image. We consulted existing literature on depth inference and found various papers¹ and articles on the topic, some of which even presented results that seemed sufficient for our use cases. In our case, we didn’t need accurate, as in ‘this pixel is 30cm in front of the camera’, results, but we were only interested in getting the general distance relations correct. For us, knowing that region A was slightly behind but definitely way in front of region B was enough to generate a visually pleasing effect and by constraining our domain to portrait imagery, we were able to further reduce the tasks complexity.

Given our experience with deep learning and our current focus on introducing machine learning powered features to the PhotoEditor SDK, we immediately decided to tackle the new challenge with deep learning or more specifically convolutional neural networks. Having a huge dataset of image and depth map pairs available, made this choice even easier. We stuck to a system similar to our previous segmentation model but decided to put more emphasis on allowing the reuse of individual parts, as this would come in handy when adding additional features in the future. To achieve this, we created a new modularized neural network approach named Hydra, which will be presented in an upcoming blog post.

During development, we followed our tried and tested workflow of starting with a complex custom model, which is then tweaked and refined to match our performance requirements while maintaining the prediction quality we need. Once that was done, we had a fast and small model, trained on thousands of iPhone front camera selfies and capable of inferring high fidelity depth maps from a plain RGB image in under a second.

The Prototype

After creating a small model capable of inferring depth maps for any given portrait image, we immediately wanted to evaluate its performance in a ‘real-world’ environment. We decided to build a prototype that applies a depth of field effect to a portrait image, by using the model and its outputs. With our long-term goal of deploying the model to iOS, Android and the web in mind, we built the prototype using TensorFlowJS to explore this newly released library. Our browser demo consists of a minimal ‘Hydra’ implementation with individual modules, one for extracting features and one for the actual depth inference, which can both be executed individually.

While being optimized for performance and memory footprint, the trained weights of the model still add up to ~18MB, which we will improve by further fine-tuning or even applying pruning or quantization. Once the models are loaded, all further processing happens on the device though, so you may try out all the samples without worrying about your data plan.

Results

Seeing our vision come to life was quite a stunning experience. Suddenly our browser was able to perform a complex depth of field effect without the need for special hardware, manual annotations or anything else apart from our image. And the best part was manually moving the focal plane through the image, either by sliding or tapping on different regions. Although being trained on ‘just’ selfies the model handles turned heads, silhouettes and multiple people pretty well and isn’t as restricted to its domain as we initially expected.

And while our initial prototype is still weighing in at ~18MB, we’re certain to slim that down further in order to use the model in production. Performance wise we were very impressed with the TensorFlowJS inference speed. Even though everything is happening on the client side and is therefore dependent on the clients hardware, we saw inference speed below one second right of the bat and those greatly improved after the initial run, as the resources were already allocated. While not being immediately helpful for the depth inference part, this allowed us to further confirm our theory behind Hydra: Re-running inference once the necessary resources on the machine have been allocated greatly increases performance and might even allow real-time performance after an initial setup-time.

To summarise, we’re definitely eager to further explore the use of depth data in image editing and think we have found a way to overcome the access restrictions on different platforms and hardware with our custom model. Combined with our new Hydra approach we can see lots of potential features that will delight both our users and customers and we will keep you updated right here.

(1)
The papers we extracted most knowledge for our use case from were:
“Depth Map Prediction from a Single Image using a Multi-Scale Deep Network” (arXiv)
“Deeper Depth Prediction with Fully Convolutional Residual Networks” (arXiv)

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

Almost Getting Sherlocked by Apple’s Core Image Team

Malte — Tue, 12 Jun 2018 00:00:00 GMT

During each World Wide Developer Conference keynote, app developers all over the world are fearing their app may get sherlocked by Apple. This has happened to flashlight apps when Apple introduced a simple toggle in the control center, to f.lux when Apple added exactly the same functionality to macOS and iOS, in parts to Dropbox with the launch of iCloud Drive and to many other developers. This year there was only one obvious of such things, the Workflow app which was acquired a few month ago and instantly turned into Siri Shortcuts by Apple. So no real ‘sherlocking’. But when we skimmed the newest APIs, release notes and session descriptions after the keynote, we found out that the technology behind our Portrait by img.ly app might have gotten sherlocked by the Core Image team. The reason is perfectly summarized in this tweet:

“Portrait Segmentation API

A new API for third-party developers allows for the separation of layers in a photo, such as separating the background from the foreground.”

Hi @halidecamera
— Preshit Deorukhkar (@preshit) June 4, 2018

How we found out about the new Portrait Segmentation API.

Quick spoiler: While Core Images new API is impressive, our technology is still ahead in terms of general availability, required hardware and cross-platform compatibility. But let’s start from the beginning:

We started working on automated image segmentation in 2016 and decided to focus on portraits in 2017. After spending most of the year on building a custom deep learning model, running hundreds of experiments and tweaking our post processing pipeline, we finally released the Portrait app in fall 2017. The app is able to generate portrait segmentations in real-time and allows the user to take a nice selfie, which is then automatically stylized using the segmentation mask and some post processing. This allows for sophisticated effects and got great reception all over the world. The app even got featured multiple times in many different countries. Recently we started looking into inferring depth, as we wanted to bring depth based features to our PhotoEditor SDK, but needed support on more iPhone devices, Android and especially required depth data for existing images.

Reading the tweet above, it looked like Apple just added the Portrait apps core functionality to the Core Image framework, making it available to all developers and users running or targeting iOS 12. A little worried about the new competition, we immediately looked into the docs and eagerly waited for the session on last Thursday to see if we’d soon need to find a new unique way of using technology to empower creativity.

We found out that all you need to do in order to get a segmentation mask along with your image is toggle the following flags when requesting a photo capture from your AVCapturePhotoOutput:

let settings: AVCapturePhotoSettings = AVCapturePhotoSettings()
settings.isDepthDataDeliveryEnabled = true
settings.isPortraitEffectsMatteDeliveryEnabled = true
...

After the photo is captured, you’re then able to extract the matte in the didFinishProcessingPhoto callback. And voila, you now have the captured image, a depth map and the mask separating fore- and background available:

Of course, we quickly spun up our own models and algorithms and compared the portrait matte, as well as the depth map to Apples results. As there is no way to rerun Apples matting algorithm for an existing image, we took one of Apples samples and compared their mask with results from our model:

While it may not have been the greatest idea to pick Apples shiny developer example, Apples mask is a little more detailed and for this particular image our model missed parts of the neck on the left. But overall we were still very happy with our results, especially when considering, that everything was generated from the plain image and didn’t require any dedicated hardware like dual cameras or a TrueDepth sensor. We did of course expect a superior depth map from Apple, as we’re clearly lacking data and are still actively working on the depth model used to generate the image above. But Apples depthmap interestingly has some issues around the neck as well, despite their use of a dual camera system. And keep in mind, that our results were all processed on the mobile device, entirely based on the RGB data contained in the image, and could be repeated on an Android phone, older iOS devices and even your browser.

When we wanted to try more samples, we quickly noticed the major limitations of Apples API: As the portrait matte capture is only available in combination with depth data, it’s limited to the iPhone 7 Plus, 8 Plus and iPhone X. And, most important to us, portrait mattes taken using the front camera are exclusively limited to the iPhone X and it’s TrueDepth sensor array. So for our Portrait app, switching to the new API would require us to ditch our live preview and go iPhone X and iOS 12 only. This was enough to calm our minds and we started thinking deeper about Apples technology:

While the depth requirement is annoying, it also explains the higher quality of Apples predictions. Our model is trained to solve the individual problems of portrait matting and depth map inference as a whole, but Apple is able to focus on improving the edges, while the ‘rough’ foreground/background segmentation is handled by masking based on the depth of the face detected within the image. We might be able to combine both models as well, but that would make the segmentation model currently used in the Portrait app obsolete and would most certainly kill the real-time functionality.

Overall, Apples technology is, as almost always, pretty impressive and the portrait matte generation works flawless, but we can now say, that for the use within our app, we reach a good quality with our current algorithms and think that a real-time preview is more important to our users. For our PhotoEditor SDK we’d happily use the high fidelity maps generated by iOS 12, but the hardware requirements are not yet suitable for SDK deployment. Our algorithms on the other hand, neither require a depth map, nor are we limited to processing after the image was captured, but can perform real-time inference on devices as old as the iPhone 6S. And the best of all: Once a TrueDepth camera is more common and everyone is running iOS 12, we can still use the Portrait Segmentation API and enjoy it’s simplicity.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

How to build Instagram’s Story Editor in a Day

Malte — Wed, 13 Dec 2017 00:00:00 GMT

A common question we get asked from our customers, is whether they’ll be able to create an entirely different user interface using our PhotoEditor SDK. The SDK comes with it’s own customizable UI, but customization are of course limited to a certain extent. As our SDK is used in many different use cases and contexts, we like to explore what’s possible with the PhotoEditor SDK and its included components. Here, we decided to build a UI similar to Instagram’s Stories or Snapchat using our SDK. By now, this ‘Story UI’ has become a popular way of quickly designing with different elements (stickers, brush and text) rather than enhancing and styling the image only.

So we grabbed our own docs and headed out to create a demo in which we recreate the Instagram Stories using components from the PhotoEditor SDK. This article presents our approach by starting with a general overview and then diving into the different view controllers to look at specific implementation details. You can follow along by downloading the accompanying code.

Architecture and Overview

Any image editing interface is naturally centered around some sort of canvas or preview. This is where the image with all operations applied is rendered and any changes are shown to the user. All tools add their own interface elements to allow modifications like adding stickers, text or brush strokes to the image. To create such a hierarchy of tools, the PhotoEditor SDK makes heavy use of an iOS pattern called view controller containment. The root view controller, an EditViewController in this case, manages a series of child view controllers:

A PhotoEditPreviewController that handles the internal model and all rendering, as well as the rendering canvas itself
A TextSpriteEditController above the PhotoEditPreviewController that allows selection and manipulation of text sprites
A StickerSpriteEditController above the PhotoEditPreviewControllermanages selection and manipulation of stickers

The child view controllers may manage additional view controllers themselves, but we only need to manage the topmost objects and wire their interfaces together. This is done within the EditViewController who is also responsible for presenting the view controllers that implement the different tools, StickerViewController, BrushViewController and TextViewController.

These are mainly responsible for managing the interface and adjusting the model accordingly, while the ‘real’ work is being done in the background by the PhotoEditPreviewController whenever the model gets updated. Thanks to the SDK, creating our BrushViewController boils down to creating and wiring a BrushEditController and adding its view, color and size controls to the BrushViewControllers view. This creates a fully fledged brush tool with OpenGL rendering, color and brush size adjustments, and great performance in just 89 lines of code.

EditViewController

As described in the previous section, the EditViewController is the root view controller of our demo application and is responsible for showing a preview, presenting tools and rendering the final output. To do so, we add a PreviewEditViewController and SpriteEditControllers to the root view and register the EditViewController as their delegate. Registering as a delegate, allows the EditViewController to wire all child view controllers and pass data between them. In order to do so, we just need to return objects of other view controllers or react to model changes. All controllers then use the EditViewController to ask for objects or data they need, for example the current size of the preview view, in order to rearrange interface elements or calculate model updates. As an example, the spriteEditControllerPreviewView(_ spriteEditController:) method of the SpriteEditControllerDelegate protocol, asks for the current preview view. All we need to do in our EditViewController is to get this view from our PreviewEditViewController and return it:

Sprite Handling

Stickers and text sprites are both sprites, but there are different SpriteEditController subclasses for each sprite type, because of differing gestures and UI elements. The Instagram UI allows the editing of both sticker and text sprites within a single view, so we need to add both specific controllers, StickerSprite- and TextSpriteEditController to our EditViewController and dynamically enable the appropriate one depending on the currently selected sprite.

StickerViewController

For adding stickers, the Instagram Stories UI offers a single button that presents a grid of all available stickers. Replicating this using the PhotoEditor SDK is easy to do, as the SDK uses a collection view for presentation as well. This collection view is embedded in a StickerSelectionController which we add to our StickerViewController. Once again, after registering as the StickerSelectionControllers delegate, we get notified upon selection of any sticker and just need to change the PhotoEditModel accordingly. To do so, we create a link between the root and the currently presented view controller using the StickerViewControllerDelegate protocol. Once a sticker has been selected, we create or update a StickerSpriteModel, pass it to the EditViewController, which updates the PreviewEditViewController model:

This triggers a new rendering pass and the sticker appears on screen. All that’s left to do now is closing the StickerViewController. Again, we use the delegate pattern to ask the EditViewController about the referenceSize that’s needed to calculate the sticker’s normalized size.

BrushViewController

This view controller, just as described above, is just a wrapper around the BrushEditController that adds a color and size selection. Once again, all interaction is handled through delegates and upon closing of the brush tool, the internal model is updated. To support undo, we need to pass the PreviewEditViewController’s undoController and begin a new group, whenever we fire up the tool.

TextViewController

The text editor Instagram created within their Stories UI is essentially a fullscreen textfield with additional controls for color and size selection. Ignoring the customized design, this can easily be recreated using stock iOS components and essentially only requires listening for keyboard notifications to handle the layout and adjusting the textfield’s properties according to user interactions. Once finished, we create a corresponding TextSpriteModel that is positioned below the UITextField and notify the EditViewController using the delegate. After dismissing the TextViewController the text rendering is handled by the PhotoEditPreviewController. When reselecting an existing label, we just prefill the UITextField with it’s contents and update the model when the label textfield changes. To ensure the UITextField is the only place the text is currently visible, we hide the spriteView we previously selected and show it again, once editing has finished.

The Result

Recreating the Instagram Stories UI seemed rather challenging at first, but using our PhotoEditor SDK, it quickly turned into a very rewarding project. Being able to create something like a brush tool without the need to actually implement any drawing or gesture handling is amazing and helped to create a working prototype in no time. Tricky details like sprite transformations, remote loading of stickers or undo management were entirely handled by the SDK and we could completely focus on the interface itself.

For more details and some actual code, take a look at the repository. We documented all tricky parts and you should be able to start your own implementation using the code. Please keep in mind, that the code may target a slightly older PhotoEditor SDK version, so check for a new version before you start your journey.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

PhotoEditor SDK + Cordova & Ionic

Malte — Wed, 14 Jun 2017 00:00:00 GMT

Despite the growing trend towards React Native, Cordova (formerly PhoneGap) is still going strong and creates frequent plugin requests from our customers. Although we can’t offer full support, we set out to create a demo app and put together a guide on how to easily set up the PhotoEditor SDK with Cordova. And as Ionic is built on top of Cordova, the finished plugin is usable in Ionic as well.

The idea is to create a Cordova app that allows the user to open an image from their photo library and edit this image using the PhotoEditor SDK on iOS and Android. This is done by using an existing library to access native photo pickers on both platforms and passing the path to the found image to a plugin. The plugin manages the configuration and opening of the PhotoEditor SDK’s editor and passes the edited image back to Cordova.

In this article we’ll start with the basic setup of the plugin and example app code, explain the plugin code in depth and cover how to integrate the plugin into your app. All code is available in the corresponding GitHub repository and after reading this article, you’ll be able to add a the PhotoEditor SDK to your Cordova app, customize it to your needs and delight your customers.

Setup

As we’ll have to build a Cordova plugin that wraps the PhotoEditor SDK into a JavaScript interface, we start with creating the following folder structure within our project:

demo
- src
- example

The src folder contains the actual Cordova plugin consisting of JavaScript, Objective-C and Java code, while the example folder contains a conventional Cordova project that uses our plugin to open the PhotoEditor SDK and manages the image selection from the users library.

The demo app

To start with our demo, we just use the cordova create example command, to create a new Cordova application. We add our target platforms, iOS and Android, using the cordova platform add <platform> command and are ready to head into actual coding.

Due to the focus on integrating the PhotoEditor SDK in this blog post, we’ll skip the details on how to open the native image pickers and dive directly into plugin development. For now we’re just left with the following snippet in our example app, that’s supposed to open the PhotoEditor SDK at some point in the future:

Creating a Cordova Plugin

Any Cordova plugin starts with a plugin.xml manifest file and then covers the supported platforms in separate files. All these files get wrapped in a JavaScript interface, that exposes the functionality to Cordova. The bridging between JavaScript and the native source files is managed by Cordova and made possible by subclassing CDVPlugin on iOS and CordovaPlugin on Android.

Plugin Configuration

Our plugin.xml becomes quite long due to the involved platforms, so we’ll just link it here and explain some specific details.

The first lines configure the plugins name, version etc. and define the main pesdk.js file containing the interface we’ll later call from our Cordova application. The <clobbers> tag specifies the namespace under which the exports are later available in the window object.

<js-module src="src/www/pesdk.js" name="PESDK">
  <clobbers target="PESDK" />
</js-module>

The file then defines the plugin details for both platforms. Starting with Android, we list the parameters the plugin will add to the config.xml and AndroidManifest.xml files. These specify the Java package, when the plugin should load, register activities etc. The remaining lines for Android embed a custom build.gradle file, ensuring all features are enabled for the PhotoEditor SDK and finally copy our source files PESDKPlugin.java and CameraActivity.java into the application folder.

Moving on with iOS, which behaves a little more picky in this case, we repeat the initial Android steps like defining the package, copying files and linking all frameworks required by the plugin. The last embedded framework is the PhotoEditor SDK:

<framework src="src/ios/PhotoEditorSDK.framework" custom="true" embed="true" />

Sadly Cordova doesn’t correctly handle the embedding of our SDK, as it doesn’t add the SDK to the embedded binaries tab in the Xcode project. Therefore we adapted a resolveDependencies.js script from StackOverflow that uses node-xcode to configure the project correctly during plugin installation. This script is saved in the /hooks/resolveDependencies.js file and called using specific hooks:

<hook type="after_platform_add" src="hooks/resolveDependencies.js" />

The last iOS specific step is setting two keys in the applications Info.plist file that describe our camera and photo library usage when asking the user for access permissions. In this example we won’t actually be requesting these permissions from the PhotoEditor SDK, but when using the integrated camera implementation without these keys, the plugin would never gain access to the iOS camera and photo roll:

<config-file
  parent="NSCameraUsageDescription"
  target="*-Info.plist">
  <string>
    Uses your camera to snap pictures.
  </string>
</config-file>
<config-file
  parent="NSPhotoLibraryUsageDescription"
  target="*-Info.plist">
  <string>
    Accesses your photo library to save and open pictures.
  </string>
</config-file>

With this plugin.xml file in place, we can actually start implementing our plugin for iOS and Android.

Platform Specific Implementations

As we fiddled around with iOS last, we’ll start with the iOS specific code. The PhotoEditor SDK is written in Swift, but Cordova prefers Objective-C instead. We could write our plugin in Swift as well and add an Objective-C wrapper, but for the sake of simplicity we’ll just stick with Objective-C for our plugin, as the PhotoEditor SDK is fully usable from Objective-C anyway.

We need to add three files to our ios directory: The framework itself and our plugin classes header and implementation files PESDKPlugin.h & PESDKPlugin.m. The plugin exposes a single method, present:, taking a filepath as a simple string. The method then loads the image at the given path, configures the PhotoEditor SDK and creates and presents a PESDKPhotoEditViewController that takes the image:

To ensure the SDK is fully loaded and unlocked, we need to add a small initialize method as well. And as you can see, we need to add our iOS license as well. This has to be done by opening the Xcode project and dragging the license file into the sidebar, as it won’t be copied into the app bundle otherwise. Once this is done, we can initialize the SDK in the following way:

These methods contain all the native Objective-C code you need in order to open the PhotoEditor SDK from a Cordova plugin. Except for some boilerplate code for bridging between JavaScript and Objective-C. If you want to configure the PhotoEditor SDK’s style or available functionality you may modify the configuration object according to our documentation.

Moving on with Android, we create a build.gradle according to our docs and the PESDKPlugin.java file that will contain our plugin code. Once again the native code boils down to opening the PhotoEditor SDK and passing a file path and some boiler plate code required by Cordova:

The code extracts the file path we passed from Cordova and prepares the PhotoEditor SDKs editor activity using a PhotoEditorBuilder. Important, but easily overseen parts are saving the callbackContext for notifying Cordova when the editor closes in line 2 and registering the plugin for activity callbacks in line 18. As we did on iOS, we need to initialize the SDK and pass our license file, but in this case we can just copy the file to the example/platforms/android/assets folder:

To finish up, we now need to define an interface using JavaScript. This interface defines the methods available in the plugin and will later be called from our Cordova application. Cordova then handles the delegation to the native implementations for us. In our case, the interface is rather minimal and only consists of our present method:

And thats it! We now have successfully created a Cordova plugin, which will allow us to open the PhotoEditor SDK editor from our own application, while passing an already existing image. Now we just need to wire everything in order to get our final result, the PhotoEditor SDK running on both platforms:

Wiring it all up

We’re now ready to put our fresh plugin to good use. We use the cordova-image-picker plugin to let our users pick an image from their photo library and pass the path of the image to our plugin. To add the plugin to our example project, we just need to call cordova plugin add .. --link. This will use the plugin.xml file to add the plugin and its files to our example project. Now we just need to replace the TODO from the beginning of this article, with the following:

This looks a little crazy at first, but doesn’t do anything magical: We first use the imagePicker object to open the native image pickers and take the first image path from the returned results array. We then use the present: method of our PESDK plugin to open the PhotoEditor SDK and pass the found image path. Everything else is handling errors and returns by presenting them using a conventional JavaScript alert.

When called, the plugin prepares the inputs using native Objective-C or Java and presents the PhotoEditor SDK’s editor with the loaded image. A user can then add filters, stickers and frames, adjust the images colors or transform it to a different size. All with native performance and a polished UI. Once finished, the image is returned by the editor and our plugin saves the file to the native photo rolls. Finally, the plugin returns the path to the edited image to Cordova. From there you could display the image, offer sharing functionality or anything else. On iOS, the running example looks like this:

And thats it! You have now successfully created a Cordova plugin that wraps the PhotoEditor SDK’s native components for use with Cordova. If you need to configure the SDK, e.g. changing behaviour and colors or adding/removing stickers, you can adjust the plugin to modify the configuration object. Take a look at our configuration docs for iOS and Android for more details.

Ionic

Ionic is built on Cordova, so you’re able to use the plugin from your Ionic application as well. Just take a look at our demo repository to get started. The repository contains a Ionic demo app that uses the plugin we just built to open the PhotoEditor SDK.

Summary

Creating a Cordova plugin that wraps the PhotoEditor SDK requires some upfront work, mostly caused by Cordova quirks, but quickly yields very usable results. Combined with the SDK’s configuration options, one can easily embed a photo editor into his Cordova or Ionic app and move on to other parts. For all details, take a look at the corresponding GitHub repository, our iOS and Android docs and the Ionic demo repository.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Deep Learning for Photo Editing

Malte — Thu, 20 Apr 2017 00:00:00 GMT

Deep learning, a subfield of machine learning, has become one of the most known areas in the ongoing AI hype. Having led to many important publications and impressive results, it is applied to dozens of different scenarios and has already yielded interesting results like human-like speech generation, high accuracy object detection, advanced machine translation, super resolution and many more.

There is a steady flow of papers and publications that describe the latest advances in network design, compare existing architectures or describe unseen approaches leading to even better results than the current state-of-the-art. At the same time more and more companies and developers jump on the deep learning bandwagon and deploy the ideas and architectures to real world production systems.

This article describes our approach to applying deep learning to our image editing product, the struggle we had with finding the right architecture and the experiences we made while developing a system that can be deployed to mobile devices.

Our vision

At 9elements, we’ve had various AI topics on our radar for quite some time now. With deep learning, we finally found a tremendous opportunity for our product, the Photoeditor SDK: We believe AI-based algorithms could be the ideal approach to boost our users creative output and simplify complex design tasks.

Given the hype and results, we decided to dip our toes into deep learning, which quickly lead to some research regarding the most common challenges in interactive image editing. We quickly surfaced image segmentation as a major challenge that could be solved using deep learning and started investigating further.

If you have ever tried to select a distinctive region in a picture, say your best friend on the beach or your cute pet, you know the struggle of carefully moving your cursor along the object’s outer bounds until you eventually miss a part or accidentally select something that doesn’t belong to the object. Professional image editing tools can be quite helpful in accomplishing such tasks, but on the one hand, they aren’t available on your mobile device, where you take and publish the images, and on the other hand, can be quite expensive and usually require some hands-on time, before you can produce anything usable.

Our goal was to finally remove the hassle from image clipping. We wanted to reduce the required user interaction to a minimum while offering an intuitive solution that doesn’t require any manuals or online courses. On top of that, as we provide native SDKs for web, iOS, and Android, the solution had to be deployable to all of these systems without relying on a powerful backend or being limited to certain features.

Having formulated our rather ambitious goals, we started our journey by looking into the most common research papers and classic techniques for image segmentation. We then focused on the deep learning part and quickly had an idea on how to design our approach.

Our journey

Image segmentation, the process of classifying each pixel in a picture to be rather fore- or background, is a popular research field and still perceived as quite challenging due to the complicated nature of the task. We, humans, are extremely well trained at perceiving scenes, identifying objects and making logical assumptions based on the visual input we receive.

For a long time, all approaches were based on colors, edges, and contrast and relied heavily on fine-tuned parameters, which had to be adjusted to every new scenario. That changed in 2012 when Krizhevsky et al. presented astonishing object classification results on the ImageNet benchmark using a neural network. Suddenly a system was able to classify objects with unprecedented accuracy and no need for any human fine-tuning. The neural network was ‘just’ trained on the dataset by seeing images combined with their corresponding labels and adjusting its internal representation until it couldn’t learn any further.

As we had already decided on using deep learning for our task, using a neural net clearly was our way to go. We started by examining the existing solutions and approaches, created our first prototype based on our findings and refined our approach and implementation until everything met our expectations.

Scene Labeling

The first approaches we examined focused on segmenting the whole image. This is a common task called scene labeling or semantic labeling, because it allows robots and other systems to understand a scene. The goal is to classify each pixel in an image to a particular object category. An example could be a self-driving car that searches for the road and tries to determine whether any pedestrians are crossing the street. Such a car would try to classify each pixel as road, pedestrian, tree, traffic sign, etc.:

While offering lots of possibilities, the existing solutions were lacking the desired accuracy we needed to provide visually pleasing image segmentations. For a self-driving car, it doesn’t matter if the ‘person’ region for some pedestrian accurately covers the person’s outlines. However, for us it does.

To overcome these issues we experimented with post processing techniques that used the segmentations we found as a base for further optimisations. This lead to our first approach where we would initially segment the entire image using a convolutional neural network, offer the found regions as selectable regions to a user and then try to refine the user’s selection using conventional image segmentations to find the best possible mask.

While already yielding some useful results the system did not quite match our requirements. If the initial segmentation was too coarse or off in critical regions, the user could never select an area that would lead to his desired segmentation.

Image segmentation based on user inputs

We went back to the drawing board and searched for other approaches that would fit our use case. It didn’t take long, and we stumbled upon Deep Interactive Object Selection, a paper that presents an interactive system which creates image segmentations based on user clicks. It looked like a good fit for our requirements, and we updated our existing system to generate fake user inputs and train on combinations of these inputs and images.

To train the net, we used the publicly available COCO dataset, which contains around 300.000 images with more than 2 million annotated object instances. To handle the amount of data, we limited our training data to a subset of the full dataset. This subset was made up of images that contain objects from certain categories and cover a minimum area within the image. As we generated the inputs artificially by adding clicks on the object mask, we could generate as many training data from the COCO subset as we wanted. After some experiments, we settled for three different strategies to create user inputs and trained the net with roughly 300.000 training records.

The masks generated by the updated system were quite impressive already. The neural net could infer which object the user wanted to mask in the image, just by looking at raw pixel data and the user’s clicks on the object. Happy with the first results, we tried to tackle the next hurdle. Before diving deeper into optimizing the neural net, which is a rather error prone process and consumes lots of time, we wanted to deploy the net to a mobile device. We wanted to make sure that such a tool is usable on any device and the performance would match our expectations.

Neural nets on mobile devices

Neural nets are sets of operations, executed in a specific order and based on millions of parameters. Therefore one “run” of such a net requires a lot of computation power, as millions of calculations have to be carried out. At the same time, the millions of parameters need to be deployed, as they represent the model or the representation the neural net has learned during training. So, to deploy our neural net, we had to solve these two requirements on an iPhone.

The first requirement, computing power, was thankfully solved by Apple. With the latest iOS version a specialised framework, called Metal Performance Shaders, was introduced. It offers the all required operations and is tailored to run these on the phones GPU, which is fast and efficient. To execute our net using the framework we had to translate our TensorFlow network code to Swift and rebuild the net’s architecture using Metal Performance Shader operations. Sadly Apple only supports a subset of todays common neural network operations, so we were forced to write some shader code to reconstruct the full network.

The second requirement, extracting the trained parameters and deploying them to the device was much easier. We just had to restore our previously trained model from a TensorFlow checkpoint, write all trained variables into a file and deploy this file with our iOS app. When needed, the iOS app would load the file into memory, and our network implementation would use the given parameters to run an inference pass.

Having met the two requirements, our network worked fine on an iPhone. We added the postprocessing operations and were able to segment images by a single tap without the need for a backend or any network communication. But there were some caveats.

While our neural net was a very common and widely used network, it was huge regarding the trainable variables. A trained model contains ~134 million parameters, which translates to about half a gigabyte of data that needs to be deployed with the app. This was obviously a showstopper for a mobile image editing app, as we couldn’t justify a 500MB download just to be able to segment images with your finger.

Furthermore, the results were still very coarse. If your colleague waved his arms in an image, the net usually could easily detect his torso, head and maybe his legs, but almost never the arms or hands. Fixing this using our postprocessing algorithms wasn’t that much of an option as it would have required lots of computing power and why bother using a neural net with millions of parameters if we fall back to conventional image processing techniques anyway?

So all in all, we had already learned a lot: Our approach of processing user inputs combined with raw image data as neural net input led to usable outputs, although quite coarse. Deploying such a net to mobile devices was possible, and the performance was good enough for using it in an interactive tool. The next step was to optimize the system to fix the parameter size and get finer results.

Combining SqueezeNet and SharpMask

We decided to tackle the network size first, as laying a proper foundation for optimizing the coarseness seemed like a sane thing to do. When looking for small nets with few parameters and fast inference its hard not to stumble across the SqueezeNet architecture by Iandola et al. which was published in November 2016. It met our use case, didn’t use any exotic operations that would be hard to implement on mobile and the results looked promising, so we removed the original network from our system and replaced it with an altered SqueezeNet implementation. And to our surprise, it worked almost right away. We had to tweak our training pipeline, and the results differed slightly, but all in all the small network with only ~5 million parameters matched the performance of our previous behemoth with ~134 million parameters. We quickly updated our conversion script and found out that our deployable model file just shrunk from ~500mb to 2.9mb. What a happy day!

Having solved the network size issue, we went ahead and thought about increasing the precision of our predictions. A loss of resolution is unavoidable in convolutional neural networks, as later layers acquire a larger “view” of the inputs by reducing their input size with so-called “pooling” layers. These layers take for example four values from the previous layer and merge them into a single one. Therefore our new SqueezeNet-based system created a 32 by 32-pixel image mask from a 512 by 512-pixel input image. Up to now we just scaled these up by using a transposed convolution. This allowed the net to learn how the upscaling worked best, but the fine details from the initial input image were already lost at this point.

We remembered Facebooks SharpMask system introduced in summer 2016 and revisited the accompanying paper. Their refinement modules seemed like a good fit, as they were able to gradually incorporate features from lower levels, but with higher resolution, into the coarse outputs. We adopted the idea and altered the refinement modules to take the final SqueezeNet output. The modules then combined the coarse SqueezeNet output with the pooling layers intermediate results and were able to refine the result. This increased our model size and the computation costs by a fair amount, but lead to much finer and more detailed results.

Once we settled on our architecture, we started an extensive training run, in which we tested more than one hundred different variations of hyperparameters, architectural details, and resizing techniques. Evaluating the results, we selected a variation, which made the best compromise between accuracy and inference speed/model size.

Our results and prototype

Having managed to fix all the issues, we were eager to see how the whole system performed on a mobile device with limited computing power and inputs. We updated our mobile app to use the new network architecture and the freshly trained model to compare the refined system to our previous approach. The results were amazing. When selecting objects that matched the categories of our training data and were fully visible in the image, we were able to generate fine-grained selection masks with just a single tap. More complex or larger objects required a few more taps, but we could always find a selection mask for our object, that was at least a solid starting point for further optimizations.

We decided to build a more polished prototype based on our existing img.ly iOS app. This app uses our PhotoEditor SDK to offer advanced image editing including focus and filter operations. As we were now able to create masks based on objects in the image we quickly settled on enhancing our filter and focus tools with selective masking.

Retrospective

Looking back at our journey into deep learning, it was one of the more frustrating yet fascinating ones. The sheer amount of possible applications is exciting, and once you get the hang of training something on your data, you immediately want to start experimenting with new things. On the other hand, you’re usually building huge black boxes with millions of float values, which makes debugging a pain. Especially when trying to replicate an already implemented architecture on other platforms, this can quickly become rather frustrating. If your outputs don’t match the expected results, your only option is to repeatedly go over your code, check all parameters and hope you stumble upon the wrong number somewhere. But once you manage to set everything up and start seeing some good results, you instantly want to tweak and optimise the bits and pieces of your system.

Overall, deep learning is a pain to debug, but yields great results, opens up a new field of photo editing applications and we’ll definitely keep exploring the new possibilities of applying the techniques in our product. Stay tuned for upcoming features!

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

PhotoEditor SDK + React Native

Malte — Mon, 20 Mar 2017 00:00:00 GMT

Often our users ask whether it’s possible to use the PhotoEditor SDK for iOS and Android with React Native (the good news right away: Yes, it is possible and fairly easy as well). So, we set out to create a demo app and put together a guide on how to easily set up the PhotoEditor SDK with React Native and how to avoid eventual pitfalls.

Setup

We started our demo project by creating a new React Native project using react-native init PESDKDemo. We then added react-native-navigation, a navigation library from the awesome folks at Wix, to get a base for our demo app. However, react-native-navigation is not required for embedding the PESDK into your React Native application.

Launching the PhotoEditor SDK from React Native

To successfully launch our editor from React Native we needed to do three things:

Add the PESDK library to our iOS project.
Create a native module that bridges between React Native and the PhotoEditor SDK.
Add a method to create a ToolbarController, push a PhotoEditController and present them from the current view controller.
Call the method, wherever we want to edit an image in our React Native code

The first step was rather easy. We just followed the steps for a manual installation and had the PESDK library ready in minutes.

Creating a native module in React Native was fairly easy as well. We simply created PESDKModule.h and PESDKModule.m and defined a PESDKModule class that inherits from NSObject and implements the RCTBridgeModule protocol. In the classes implementation we registered our module with React Native by calling RCT_EXPORT_MODULE(PESDK):

In order to create a new photo editor view controller we needed to create a new ToolbarController and push a PhotoEditController that loads a sample image. As we’ll always use such a hierarchy, we bundled this setup into the present() method of our native module and exported it to React Native using the RCT_EXPORT_METHOD macro:

The final step was putting our newly created module to good use and actually pushing our photo editor from React Native. To do so, we imported the PESDK native module using var PESDK = NativeModules.PESDK. We could then call PESDK.present() and the native module created the ToolbarController and PhotoEditViewController objects and pushed them from the current view controller. If you were to use a simple TouchableHighlight, this could look like this:

Bringing it all together, we were now able to open the native iOS PESDK PhotoEditController from our React Native code. Surprisingly easy, isn’t it?

Passing an image to the editor

Opening an image editor with a sample image is nice but wasn’t really our intention, when we started integrating the PhotoEditor SDK. Instead, we wanted to be able to edit any image and save the result into our camera roll or share it with our friends. For our little demo, we decided to present a grid of images from unsplash.com to the user, so he can choose any image from the list and edit it with our editor.

The JavaScript for creating the image grid is a little out of scope for this article, so we won’t cover it in detail. It basically uses a ListView to create rows of three cells and fills them with images loaded from the unsplash.it service. All image fetching, scrolling, etc. is handled by React Native, so we only needed to handle the user’s taps on an image:

We used [react-native-fs](https://github.com/itinance/react-native-fs) to download a larger resolution image to the local filesystem, pass the path of the local file to our present() call and modify our iOS native module:

We then had a nice little app, that shows a grid of images, loads a high-resolution image upon selection and opens the PhotoEditor SDK:

Android implementation

As we have seen, opening the PESDK from React Native can easily be done on iOS. But why use React Native, if the app only runs on iOS? It was time to add Android support to our little demo app. To accomplish this we needed to repeat some of the previous steps for Android:

Add the PESDK to our Android project.
Create a native module that bridges between React Native and the PESDK.
Add a method to launch an ImglyIntent using the PhotoEditorBuilder from the current Activity.

Installing the SDK is again done by following the instructions for integrating the PESDK and shouldn’t take more than a few minutes. Creating a native module on Android is quite similar to iOS, although a little more setup code is required: We created our PESDKModule that recreates the present(path)method from iOS, a PESDKPackage containing our module and finally added the package to our Application:

This time, we prepared the desired settings for our editor, added our image path and passed everything to a PhotoEditorBuilder. The builder creates an intent and we launch it from our current Activity. That’s it! Running our sample app on an Android device now leads to this:

Nifty details

Experimenting with our new app, we noticed an issue: While the Android app closed the editor just fine when canceling the editor, the iOS app did nothing at all. That is because the iOS module doesn’t handle the PESDKs delegate methods. However, this can be fixed easily:

It could be taken even further by turning our native iOS module into a RCTEventEmitter and sending events to React Native when the delegate methods are called. Just take a look into our GitHub repository to find the corresponding code.

Summary

To sum it up: Using the PhotoEditor SDK from React Native was rather easy to do and we were very happy with the results. Handling the data flows between React Native and the Java or Objective-C classes can be tricky, but when dealing with a specific use case, it should be easy to adjust the native modules to meet your requirements. Feel free to adapt our code and add the PhotoEditor SDK to your React Native app. We’re looking forward to your feedback and any pull requests, that further optimize our implementation.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Malte – IMG.LY Blog

CE.SDK v1.7.0 Release and Roadmap

CE.SDK v1.7.0

Auto Close Option in Web UI Configuration for Asset Libraries

Custom URI Resolver

Improving Mobile Touch Support

Rotation of Groups and Hierarchies

Custom Asset Source UI

Moving from Roles to Abilities: Disconnecting UI Variants from Roles

Shadows in Advanced UI

Blur API

Effects API

Image Fills API

Image Straighten API

Improved Group Selection and API

Scopes API

Shadows API

Zoom API

Product Roadmap

Thanks for reading! Let us know what you think on Twitter! To stay in the loop, subscribe to our Newsletter.

CE.SDK v1.6.0 Release

Linear Gradient Fills

Strokes, Outline, Insets, and Stroke Patterns

Blend Modes

Scale Groups, Multi-Selection, and Hierarchies

UI Template Library

Floating and Static Asset Library Panels

Placeholders API

Editing State APIs

API for Image Fit Modes

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

CE.SDK v1.5.0 Release

Hierarchies (Groups)

Template Imports & Exports

More Performance

Improved Touch & Mobile Support

PDF Export

Build Your Custom UI

Events API

Other

Thanks for reading! To stay in the loop with our latest articles and releases, subscribe to our Newsletter.

From 2D to 3D Photo Editing

The New Cool: Depth Data

Deep Possibilities

Depth-aware Editing

Depth of Field Effects

3D Asset Placement

Enter Deep Learning

The Prototype

Results

Almost Getting Sherlocked by Apple’s Core Image Team

How to build Instagram’s Story Editor in a Day

Architecture and Overview

EditViewController

Sprite Handling

StickerViewController

BrushViewController

TextViewController

The Result

PhotoEditor SDK + Cordova & Ionic

Setup

The demo app

Creating a Cordova Plugin

Plugin Configuration

Platform Specific Implementations

Wiring it all up

Ionic

Summary

Deep Learning for Photo Editing

Our vision

Our journey

Scene Labeling

Image segmentation based on user inputs

Neural nets on mobile devices

Combining SqueezeNet and SharpMask

Our results and prototype

Retrospective

PhotoEditor SDK + React Native

Setup

Launching the PhotoEditor SDK from React Native