Bringing Wide Color to PhotoEditor SDK

Inspired by Mike Krieger’s great post on supporting wide color images in Instagram, I decided to challenge myself by introducing the same wide color support into our PhotoEditor SDK within a day.

Mike did an excellent job describing everything that is needed to support wide color images so I am not going to reiterate his explanation. Instead, here I’m going to share two findings I made in the process.

Image Export

Mike suggests that in order to preserve the color space while converting an UIImage into a JPEG, the UIImageJPEGRepresentation(_:_:) method has to be replaced with the new UIGraphicsImageRenderer.jpegData(withCompressionQuality:actions:) method. Checking the color profile of the generated JPEG by UIImageJPEGRepresentation(_:) I noticed that it seemed unnecessary and resulting in more work than needed. To verify this, I started Hopper and took a look at the internals of the new method. Here’s what it basically does:

1. It passes actions to UIGraphicsRenderer.runDrawingActions(_:completionActions:).
2. It obtains the resulting image using UIGraphicsImageRendererContext.currentImage.
3. It passes that image to UIImageJPEGRepresentation(_:_:).

As you can see, it makes use of UIImageJPEGRepresentation(_:_:) internally too, so there really is no need to go the extra mile. If you are actually doing any drawing within the actions block using wide color UIColors it absolutely makes sense to use Mike’s apoproach. But if you only want to convert an UIImage into a JPEG image, using the old method is far easier.

Core Image

We are using CIImage, CIFilter and CIContext for most of our processing. In our live preview we use a CIContext to directly render a CIImage into a GLKView and I’ve run into exactly the same problem as Mike has — namely not being able to get that view to be color space-aware. Using the offscreen buffer workaround that he suggested was going to be too much work though and also I didn’t want to sacrifice any performance.

After doing some research and digging further into the GLKit and Core Image assembly I found a nice solution. It looks like in order to render wide color images into OpenGL, OpenGLES 3 and a GLKViewDrawableColorFormat (that is currently not defined as an enum case) has to be used. The following code works like a charm for me:

let api: EAGLRenderingAPI
let colorFormat: GLKViewDrawableColorFormat
 
if #available(iOS 10.0, *) {
  if UIScreen.main.traitCollection.displayGamut == .P3 {
    api = .openGLES3
    colorFormat = GLKViewDrawableColorFormat(rawValue: 10)!
  } else {
    api = .openGLES2
    colorFormat = .RGBA8888
  }
} else {
  api = .openGLES2
  colorFormat = .RGBA8888
}

let context = EAGLContext(api: api)
let previewView = GLKView(frame: CGRect.zero, context: context!)
previewView.delegate = self
previewView.drawableColorFormat = colorFormat

This code creates a GLKView that is backed by a CAEAGLLayer with the following drawableProperties set:

[
 "EAGLDrawablePropertyRetained": 0,
 "EAGLDrawablePropertyColorFormat": EAGLColorFormatRGBA_XR10_64BPP
]

Unfortunately EAGLColorFormatRGBA_XR10_64BPP also doesn’t seem to be documented at this point, but my guess is that setting this color format on Instagram’s EAGLView would also solve their problem of making the view color space-aware.

Thanks to Core Image we can just switch to OpenGLES 3 without having to do any other modifications to our code. However, I am currently not sure if this is considered a private API use. I would greatly appreciate if anyone could shed some light on this.

I have not yet been able to test this on any device other than the iPhone 7 Plus, but I will update this post if I run into any problems with future tests.

Final Thoughts

Getting an app to support wide color images is definitely worth it and not as difficult as I had imagined. Unfortunately, it currently looks like iOS is still missing (public) support for wide color in some parts of its frameworks. I hope that Apple addresses these problems soon. Nevertheless we’re optimistic that we’ll be able to include wide color image support in the next release of our SDK. Feel free to check out our demo app in the App Store which we will update as soon as possible.