Instagram – IMG.LY Blog

How to build Instagram’s Story Editor in a Day

Malte — Wed, 13 Dec 2017 00:00:00 GMT

A common question we get asked from our customers, is whether they’ll be able to create an entirely different user interface using our PhotoEditor SDK. The SDK comes with it’s own customizable UI, but customization are of course limited to a certain extent. As our SDK is used in many different use cases and contexts, we like to explore what’s possible with the PhotoEditor SDK and its included components. Here, we decided to build a UI similar to Instagram’s Stories or Snapchat using our SDK. By now, this ‘Story UI’ has become a popular way of quickly designing with different elements (stickers, brush and text) rather than enhancing and styling the image only.

So we grabbed our own docs and headed out to create a demo in which we recreate the Instagram Stories using components from the PhotoEditor SDK. This article presents our approach by starting with a general overview and then diving into the different view controllers to look at specific implementation details. You can follow along by downloading the accompanying code.

Architecture and Overview

Any image editing interface is naturally centered around some sort of canvas or preview. This is where the image with all operations applied is rendered and any changes are shown to the user. All tools add their own interface elements to allow modifications like adding stickers, text or brush strokes to the image. To create such a hierarchy of tools, the PhotoEditor SDK makes heavy use of an iOS pattern called view controller containment. The root view controller, an EditViewController in this case, manages a series of child view controllers:

A PhotoEditPreviewController that handles the internal model and all rendering, as well as the rendering canvas itself
A TextSpriteEditController above the PhotoEditPreviewController that allows selection and manipulation of text sprites
A StickerSpriteEditController above the PhotoEditPreviewControllermanages selection and manipulation of stickers

The child view controllers may manage additional view controllers themselves, but we only need to manage the topmost objects and wire their interfaces together. This is done within the EditViewController who is also responsible for presenting the view controllers that implement the different tools, StickerViewController, BrushViewController and TextViewController.

These are mainly responsible for managing the interface and adjusting the model accordingly, while the ‘real’ work is being done in the background by the PhotoEditPreviewController whenever the model gets updated. Thanks to the SDK, creating our BrushViewController boils down to creating and wiring a BrushEditController and adding its view, color and size controls to the BrushViewControllers view. This creates a fully fledged brush tool with OpenGL rendering, color and brush size adjustments, and great performance in just 89 lines of code.

EditViewController

As described in the previous section, the EditViewController is the root view controller of our demo application and is responsible for showing a preview, presenting tools and rendering the final output. To do so, we add a PreviewEditViewController and SpriteEditControllers to the root view and register the EditViewController as their delegate. Registering as a delegate, allows the EditViewController to wire all child view controllers and pass data between them. In order to do so, we just need to return objects of other view controllers or react to model changes. All controllers then use the EditViewController to ask for objects or data they need, for example the current size of the preview view, in order to rearrange interface elements or calculate model updates. As an example, the spriteEditControllerPreviewView(_ spriteEditController:) method of the SpriteEditControllerDelegate protocol, asks for the current preview view. All we need to do in our EditViewController is to get this view from our PreviewEditViewController and return it:

Sprite Handling

Stickers and text sprites are both sprites, but there are different SpriteEditController subclasses for each sprite type, because of differing gestures and UI elements. The Instagram UI allows the editing of both sticker and text sprites within a single view, so we need to add both specific controllers, StickerSprite- and TextSpriteEditController to our EditViewController and dynamically enable the appropriate one depending on the currently selected sprite.

StickerViewController

For adding stickers, the Instagram Stories UI offers a single button that presents a grid of all available stickers. Replicating this using the PhotoEditor SDK is easy to do, as the SDK uses a collection view for presentation as well. This collection view is embedded in a StickerSelectionController which we add to our StickerViewController. Once again, after registering as the StickerSelectionControllers delegate, we get notified upon selection of any sticker and just need to change the PhotoEditModel accordingly. To do so, we create a link between the root and the currently presented view controller using the StickerViewControllerDelegate protocol. Once a sticker has been selected, we create or update a StickerSpriteModel, pass it to the EditViewController, which updates the PreviewEditViewController model:

This triggers a new rendering pass and the sticker appears on screen. All that’s left to do now is closing the StickerViewController. Again, we use the delegate pattern to ask the EditViewController about the referenceSize that’s needed to calculate the sticker’s normalized size.

BrushViewController

This view controller, just as described above, is just a wrapper around the BrushEditController that adds a color and size selection. Once again, all interaction is handled through delegates and upon closing of the brush tool, the internal model is updated. To support undo, we need to pass the PreviewEditViewController’s undoController and begin a new group, whenever we fire up the tool.

TextViewController

The text editor Instagram created within their Stories UI is essentially a fullscreen textfield with additional controls for color and size selection. Ignoring the customized design, this can easily be recreated using stock iOS components and essentially only requires listening for keyboard notifications to handle the layout and adjusting the textfield’s properties according to user interactions. Once finished, we create a corresponding TextSpriteModel that is positioned below the UITextField and notify the EditViewController using the delegate. After dismissing the TextViewController the text rendering is handled by the PhotoEditPreviewController. When reselecting an existing label, we just prefill the UITextField with it’s contents and update the model when the label textfield changes. To ensure the UITextField is the only place the text is currently visible, we hide the spriteView we previously selected and show it again, once editing has finished.

The Result

Recreating the Instagram Stories UI seemed rather challenging at first, but using our PhotoEditor SDK, it quickly turned into a very rewarding project. Being able to create something like a brush tool without the need to actually implement any drawing or gesture handling is amazing and helped to create a working prototype in no time. Tricky details like sprite transformations, remote loading of stickers or undo management were entirely handled by the SDK and we could completely focus on the interface itself.

For more details and some actual code, take a look at the repository. We documented all tricky parts and you should be able to start your own implementation using the code. Please keep in mind, that the code may target a slightly older PhotoEditor SDK version, so check for a new version before you start your journey.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

The IMG.LY Photo Roll

Felix — Thu, 21 Sep 2017 00:00:00 GMT

After today, we at IMG.LY are going to divert some of our energy to a new endeavour, the img.ly Photo Roll. We feel deeply rooted in the open-source movement and we know that the internet and most of the tools that we use today wouldn’t be nearly the same if it wasn’t for the awesome and creative work of countless people that shared their assets for free. Still today we heavily rely on open-source technology and creative work to give us guidance and inspire us to create new and unseen things.

Working on our product PhotoEditor SDK for the last two years gave us a lot of useful and valuable insights about photography and the various possibilities to create exceptional visual content. As some of us are enthusiastic photographers, it only seems logical to join the community of photographers and creative folks that enrich the internet with extraordinary and free content every single day. Since we have always gratefully used the work of other people, we would like to give something back to the community. Therefore, we are going to start sharing pictures we took around the globe via our Unsplash and Instagram account. Please feel free to download, use, modify and share them at your will. The photos that we’re going to post over the following weeks and months will also be available in our photo roll that ships with the PhotoEditor SDK.

We certainly hope that you’re going to make use of them. And we’d be happy to hear from you what you created with our pictures, and we’d love to see them in a new guise. After all, their real value only shows if someone is inspired by and can get creative with them.

Thanks to all the people that contribute to our initiative: Malte Baumann, Tommi Gutscher, Niklas Priddat, Cem Selcuk and Eray Basar. Below we have already compiled a few pictures that we are going to post over the next few days. Stay tuned.

Cheers!
Your friends at IMG.LY

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Bringing Wide Color to PhotoEditor SDK

Sascha — Tue, 10 Jan 2017 00:00:00 GMT

Inspired by Mike Krieger’s great post on supporting wide color images in Instagram, I decided to challenge myself by introducing the same wide color support into our PhotoEditor SDK within a day.

Mike did an excellent job describing everything that is needed to support wide color images so I am not going to reiterate his explanation. Instead, here I’m going to share two findings I made in the process.

Image Export

Mike suggests that in order to preserve the color space while converting an UIImage into a JPEG, the UIImageJPEGRepresentation(_:_:) method has to be replaced with the new UIGraphicsImageRenderer.jpegData(withCompressionQuality:actions:) method. Checking the color profile of the generated JPEG by UIImageJPEGRepresentation(_:) I noticed that it seemed unnecessary and resulting in more work than needed. To verify this, I started Hopper and took a look at the internals of the new method. Here’s what it basically does:

It passes actions to UIGraphicsRenderer.runDrawingActions(_:completionActions:).
It obtains the resulting image using UIGraphicsImageRendererContext.currentImage.
It passes that image to UIImageJPEGRepresentation(_:_:).

As you can see, it makes use of UIImageJPEGRepresentation(_:_:) internally too, so there really is no need to go the extra mile. If you are actually doing any drawing within the actions block using wide color UIColors it absolutely makes sense to use Mike’s apoproach. But if you only want to convert an UIImage into a JPEG image, using the old method is far easier.

Core Image

We are using CIImage, CIFilter and CIContext for most of our processing. In our live preview we use a CIContext to directly render a CIImage into a GLKView and I’ve run into exactly the same problem as Mike has — namely not being able to get that view to be color space-aware. Using the offscreen buffer workaround that he suggested was going to be too much work though and also I didn’t want to sacrifice any performance.

After doing some research and digging further into the GLKit and Core Image assembly I found a nice solution. It looks like in order to render wide color images into OpenGL, OpenGLES 3 and a GLKViewDrawableColorFormat (that is currently not defined as an enum case) has to be used. The following code works like a charm for me:

let api: EAGLRenderingAPI
let colorFormat: GLKViewDrawableColorFormat

if #available(iOS 10.0, *) {
  if UIScreen.main.traitCollection.displayGamut == .P3 {
    api = .openGLES3
    colorFormat = GLKViewDrawableColorFormat(rawValue: 10)!
  } else {
    api = .openGLES2
    colorFormat = .RGBA8888
  }
} else {
  api = .openGLES2
  colorFormat = .RGBA8888
}

let context = EAGLContext(api: api)
let previewView = GLKView(frame: CGRect.zero, context: context!)
previewView.delegate = self
previewView.drawableColorFormat = colorFormat

This code creates a GLKView that is backed by a CAEAGLLayer with the following drawableProperties set:

[
 "EAGLDrawablePropertyRetained": 0,
 "EAGLDrawablePropertyColorFormat": EAGLColorFormatRGBA_XR10_64BPP
]

Unfortunately EAGLColorFormatRGBA_XR10_64BPP also doesn’t seem to be documented at this point, but my guess is that setting this color format on Instagram’s EAGLView would also solve their problem of making the view color space-aware.

Thanks to Core Image we can just switch to OpenGLES 3 without having to do any other modifications to our code. However, I am currently not sure if this is considered a private API use. I would greatly appreciate if anyone could shed some light on this.

I have not yet been able to test this on any device other than the iPhone 7 Plus, but I will update this post if I run into any problems with future tests.

Final Thoughts

Getting an app to support wide color images is definitely worth it and not as difficult as I had imagined. Unfortunately, it currently looks like iOS is still missing (public) support for wide color in some parts of its frameworks. I hope that Apple addresses these problems soon. Nevertheless we’re optimistic that we’ll be able to include wide color image support in the next release of our SDK. Feel free to check out our demo app in the App Store which we will update as soon as possible.