Core Image – IMG.LY Blog

Almost Getting Sherlocked by Apple’s Core Image Team

Malte — Tue, 12 Jun 2018 00:00:00 GMT

During each World Wide Developer Conference keynote, app developers all over the world are fearing their app may get sherlocked by Apple. This has happened to flashlight apps when Apple introduced a simple toggle in the control center, to f.lux when Apple added exactly the same functionality to macOS and iOS, in parts to Dropbox with the launch of iCloud Drive and to many other developers. This year there was only one obvious of such things, the Workflow app which was acquired a few month ago and instantly turned into Siri Shortcuts by Apple. So no real ‘sherlocking’. But when we skimmed the newest APIs, release notes and session descriptions after the keynote, we found out that the technology behind our Portrait by img.ly app might have gotten sherlocked by the Core Image team. The reason is perfectly summarized in this tweet:

“Portrait Segmentation API

A new API for third-party developers allows for the separation of layers in a photo, such as separating the background from the foreground.”

Hi @halidecamera
— Preshit Deorukhkar (@preshit) June 4, 2018

How we found out about the new Portrait Segmentation API.

Quick spoiler: While Core Images new API is impressive, our technology is still ahead in terms of general availability, required hardware and cross-platform compatibility. But let’s start from the beginning:

We started working on automated image segmentation in 2016 and decided to focus on portraits in 2017. After spending most of the year on building a custom deep learning model, running hundreds of experiments and tweaking our post processing pipeline, we finally released the Portrait app in fall 2017. The app is able to generate portrait segmentations in real-time and allows the user to take a nice selfie, which is then automatically stylized using the segmentation mask and some post processing. This allows for sophisticated effects and got great reception all over the world. The app even got featured multiple times in many different countries. Recently we started looking into inferring depth, as we wanted to bring depth based features to our PhotoEditor SDK, but needed support on more iPhone devices, Android and especially required depth data for existing images.

Reading the tweet above, it looked like Apple just added the Portrait apps core functionality to the Core Image framework, making it available to all developers and users running or targeting iOS 12. A little worried about the new competition, we immediately looked into the docs and eagerly waited for the session on last Thursday to see if we’d soon need to find a new unique way of using technology to empower creativity.

We found out that all you need to do in order to get a segmentation mask along with your image is toggle the following flags when requesting a photo capture from your AVCapturePhotoOutput:

let settings: AVCapturePhotoSettings = AVCapturePhotoSettings()
settings.isDepthDataDeliveryEnabled = true
settings.isPortraitEffectsMatteDeliveryEnabled = true
...

After the photo is captured, you’re then able to extract the matte in the didFinishProcessingPhoto callback. And voila, you now have the captured image, a depth map and the mask separating fore- and background available:

Of course, we quickly spun up our own models and algorithms and compared the portrait matte, as well as the depth map to Apples results. As there is no way to rerun Apples matting algorithm for an existing image, we took one of Apples samples and compared their mask with results from our model:

While it may not have been the greatest idea to pick Apples shiny developer example, Apples mask is a little more detailed and for this particular image our model missed parts of the neck on the left. But overall we were still very happy with our results, especially when considering, that everything was generated from the plain image and didn’t require any dedicated hardware like dual cameras or a TrueDepth sensor. We did of course expect a superior depth map from Apple, as we’re clearly lacking data and are still actively working on the depth model used to generate the image above. But Apples depthmap interestingly has some issues around the neck as well, despite their use of a dual camera system. And keep in mind, that our results were all processed on the mobile device, entirely based on the RGB data contained in the image, and could be repeated on an Android phone, older iOS devices and even your browser.

When we wanted to try more samples, we quickly noticed the major limitations of Apples API: As the portrait matte capture is only available in combination with depth data, it’s limited to the iPhone 7 Plus, 8 Plus and iPhone X. And, most important to us, portrait mattes taken using the front camera are exclusively limited to the iPhone X and it’s TrueDepth sensor array. So for our Portrait app, switching to the new API would require us to ditch our live preview and go iPhone X and iOS 12 only. This was enough to calm our minds and we started thinking deeper about Apples technology:

While the depth requirement is annoying, it also explains the higher quality of Apples predictions. Our model is trained to solve the individual problems of portrait matting and depth map inference as a whole, but Apple is able to focus on improving the edges, while the ‘rough’ foreground/background segmentation is handled by masking based on the depth of the face detected within the image. We might be able to combine both models as well, but that would make the segmentation model currently used in the Portrait app obsolete and would most certainly kill the real-time functionality.

Overall, Apples technology is, as almost always, pretty impressive and the portrait matte generation works flawless, but we can now say, that for the use within our app, we reach a good quality with our current algorithms and think that a real-time preview is more important to our users. For our PhotoEditor SDK we’d happily use the high fidelity maps generated by iOS 12, but the hardware requirements are not yet suitable for SDK deployment. Our algorithms on the other hand, neither require a depth map, nor are we limited to processing after the image was captured, but can perform real-time inference on devices as old as the iPhone 6S. And the best of all: Once a TrueDepth camera is more common and everyone is running iOS 12, we can still use the Portrait Segmentation API and enjoy it’s simplicity.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Bringing Wide Color to PhotoEditor SDK

Sascha — Tue, 10 Jan 2017 00:00:00 GMT

Inspired by Mike Krieger’s great post on supporting wide color images in Instagram, I decided to challenge myself by introducing the same wide color support into our PhotoEditor SDK within a day.

Mike did an excellent job describing everything that is needed to support wide color images so I am not going to reiterate his explanation. Instead, here I’m going to share two findings I made in the process.

Image Export

Mike suggests that in order to preserve the color space while converting an UIImage into a JPEG, the UIImageJPEGRepresentation(_:_:) method has to be replaced with the new UIGraphicsImageRenderer.jpegData(withCompressionQuality:actions:) method. Checking the color profile of the generated JPEG by UIImageJPEGRepresentation(_:) I noticed that it seemed unnecessary and resulting in more work than needed. To verify this, I started Hopper and took a look at the internals of the new method. Here’s what it basically does:

It passes actions to UIGraphicsRenderer.runDrawingActions(_:completionActions:).
It obtains the resulting image using UIGraphicsImageRendererContext.currentImage.
It passes that image to UIImageJPEGRepresentation(_:_:).

As you can see, it makes use of UIImageJPEGRepresentation(_:_:) internally too, so there really is no need to go the extra mile. If you are actually doing any drawing within the actions block using wide color UIColors it absolutely makes sense to use Mike’s apoproach. But if you only want to convert an UIImage into a JPEG image, using the old method is far easier.

Core Image

We are using CIImage, CIFilter and CIContext for most of our processing. In our live preview we use a CIContext to directly render a CIImage into a GLKView and I’ve run into exactly the same problem as Mike has — namely not being able to get that view to be color space-aware. Using the offscreen buffer workaround that he suggested was going to be too much work though and also I didn’t want to sacrifice any performance.

After doing some research and digging further into the GLKit and Core Image assembly I found a nice solution. It looks like in order to render wide color images into OpenGL, OpenGLES 3 and a GLKViewDrawableColorFormat (that is currently not defined as an enum case) has to be used. The following code works like a charm for me:

let api: EAGLRenderingAPI
let colorFormat: GLKViewDrawableColorFormat

if #available(iOS 10.0, *) {
  if UIScreen.main.traitCollection.displayGamut == .P3 {
    api = .openGLES3
    colorFormat = GLKViewDrawableColorFormat(rawValue: 10)!
  } else {
    api = .openGLES2
    colorFormat = .RGBA8888
  }
} else {
  api = .openGLES2
  colorFormat = .RGBA8888
}

let context = EAGLContext(api: api)
let previewView = GLKView(frame: CGRect.zero, context: context!)
previewView.delegate = self
previewView.drawableColorFormat = colorFormat

This code creates a GLKView that is backed by a CAEAGLLayer with the following drawableProperties set:

[
 "EAGLDrawablePropertyRetained": 0,
 "EAGLDrawablePropertyColorFormat": EAGLColorFormatRGBA_XR10_64BPP
]

Unfortunately EAGLColorFormatRGBA_XR10_64BPP also doesn’t seem to be documented at this point, but my guess is that setting this color format on Instagram’s EAGLView would also solve their problem of making the view color space-aware.

Thanks to Core Image we can just switch to OpenGLES 3 without having to do any other modifications to our code. However, I am currently not sure if this is considered a private API use. I would greatly appreciate if anyone could shed some light on this.

I have not yet been able to test this on any device other than the iPhone 7 Plus, but I will update this post if I run into any problems with future tests.

Final Thoughts

Getting an app to support wide color images is definitely worth it and not as difficult as I had imagined. Unfortunately, it currently looks like iOS is still missing (public) support for wide color in some parts of its frameworks. I hope that Apple addresses these problems soon. Nevertheless we’re optimistic that we’ll be able to include wide color image support in the next release of our SDK. Feel free to check out our demo app in the App Store which we will update as soon as possible.