Apple – IMG.LY Blog

How to Add a Filter to a Video Stream in iOS

Walter — Mon, 08 Nov 2021 15:33:55 GMT

Using filters and effects with video streams requires different strategies than applying them to files. In this tutorial, you will see how to apply effects and filters to video streams in real-time. The code in this tutorial compiles with Xcode 13.

As with most AVFoundation code, the Xcode simulator is not the best platform for running the code. Test on an actual device. A project with code that supports this tutorial can be found on GitHub.

Streams or Files

For video files stored on the device, you can use an AVMutableVideoComposition to apply filters. When streaming video from a remote location, this strategy will not work. The AVMutuableVideoComposition classes are not designed to work in real-time with streams. Apple provides a way to extract the pixels from the stream at regular intervals. You can manipulate the pixels and then render them to the screen. An AVPlayerItemVideoOutput object gives access to the pixels that make up a video frame. Apple also supplies a CADisplayLink object to ensure you are extracting the right pixels at the right time. The display link is a specialized timer that will fire in sync with the redraw rate of the current display.

So, to filter a video stream, the strategy becomes:

Set up an AVPlayer as normal and assign it the URL of the stream
Attach an AVPlayerItemVideoOutput object to the stream
Attach a CADisplayLink object to the run loop
Extract the current pixel buffer from the AVPlayerItem every time the screen redraws
Convert the pixel buffer to a CoreImage object
Apply filters
Display the image

Setting up AVPlayer

You won’t use the AVPlayerLayer or AVPlayerViewController to display the video, but the AVPlayer plays a central role. The AVPlayer keeps the video in sync with the audio, handles decoding whatever format the stream uses, controls buffering, and more. The basic setup is the same as with any other project:

//create a player
let videoItem = AVPlayerItem(url: streamURL)
self.player = AVPlayer(playerItem: videoItem)

The URL can either point to a local resource or a stream.

Using AVPlayerItemVideoOutput

The AVPlayerItemVideoOutput allows you to query the AVPlayerItem for the pixel buffer (one screen worth of pixels) at any given time. You can specify different pixel formats and other options for the output. For this tutorial, you will you can create the object with a simple instantiation using the default formats.

let playerItemVideoOutput = AVPlayerItemVideoOutput()

Later you will add this video output to your player item. Then you can ask it to generate the pixel buffers as needed. In this tutorial, you will ask for the pixel buffer of the current frame. You could ask for the frame for any timestamp though.

Creating the Display Link

The CADisplayLink class is a specialized NSTimer that synchronizes itself to the screen refresh rate of the display. This ensures that the pixel buffer you extract will be from the correct time of your video stream. It will get rendered at the correct time in the screen refresh. Syncing a standard NSTimer to the screen refresh has always been problematic. With the new variable refresh rate screens that Apple makes, it becomes impossible. However, CADisplayLink adjusts its rate as the screen refresh rate changes. Apple explains how this works in the 2021 WWDC Session Optimize for Variable Refresh Rate Displays. When you create the display link, you give it the name of a function to call each time it executes. A sample for creating a link could look like this:

lazy var displayLink: CADisplayLink = CADisplayLink(target: self,
                                                  selector: #selector(displayLinkFired(link:)))

Elsewhere in the code, a function called displayLinkFired(link: CADisplayLink) extracts the pixel buffer from the AVPlayerItem.

Starting the Player

Apple notes that loading a video file takes a measurable amount of time. It can take even longer when the video file is streaming across a network. AVFoundation allows your program to continue to execute while the setup steps and initial buffering are occurring in the background. You can run into issues if you load a video into AVFoundation and then immediately attempt to work with it. AVFoundation may not show an error or status message, but it will not perform as expected either.

The most important step in this entire process is using an observer to wait until the AVPlayerItem has a .readyToPlay status before attaching the output and displaying link objects. In the example below statusOberserver, player, playerItemVideoOutput and displayLink are all declared at the class level. They need to persist the entire time the video is being used.

//create a player
let videoItem = AVPlayerItem(url: streamURL!)
self.player = AVPlayer(playerItem: videoItem) //1
//*important* add the display link and the output only after it is ready to play
self.statusObserver = videoItem.observe(\.status, //2
                       options: [.new, .old],
                 changeHandler: { playerItem, change in
    if playerItem.status == .readyToPlay { //3
      playerItem.add(self.playerItemVideoOutput)
      self.displayLink.add(to: .main, forMode: .common)
      self.player?.play() //4
    }
})

Here is what to notice about the code above

Creating the player will take a large amount of time, but the program execution will not stop and wait
NSKeyValueObservation is how swift defines key value observers the code will execute every time the videoItem.status property changes values
Checks to see if the .status property is .readyToPlay
After adding the display link and the video output objects start playing the video

Extracting the Pixel Buffer

Once the video begins to play, the display link function gets called for each screen refresh. Here is an example of how to extract the pixel buffer and render it to a UIImageView.

@objc func displayLinkFired(link: CADisplayLink) { //1
  let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
  if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) { //2
    if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
      let frameImage = CIImage(cvImageBuffer: buffer) //3
      //4
      let pixelate = CIFilter(name: "CIPixellate")!
      pixelate.setValue(frameImage, forKey: kCIInputImageKey)
      pixelate.setValue(self.filterSlider.value, forKey: kCIInputScaleKey)
      pixelate.setValue(CIVector(x: frameImage.extent.midX, y: frameImage.extent.midY), forKey: kCIInputCenterKey)
     let newFrame = pixelate.outputImage!.cropped(to: frameImage.extent)
     self.videoView.image = UIImage(ciImage: newFrame) //5
    }
  }
}

Here is what the code is doing:

Marks the func with @objc so that it works with the display link selector
Asks if there is a new pixel buffer to display. Depending on the refresh rate of the screen and the frame rate of the video, there will not always be a new buffer.
After extracting the pixel buffer, convert it to a CoreImage object
Apply any filters. This example applies the CIPixellate filter. It uses a slider to let the user change the intensity as the video plays
Convert the filtered image to a UIImage and assign it to the UIImageView

Now as the pixellate scale changes the image will be filtered but the video will continue to play smoothly.

Going Further

Using the strategy above you should be able to apply different filters to the video stream in real-time. Remember that you only have a few milliseconds though, then the video will begin to stutter. CoreImage filters are optimized to make use of the GPU and should be fast enough in most cases. For better performance, you can render the filtered image to an MTKView and use Metal. Apple has also begun to create Metal Performance Shaders which are even more efficient than CoreImage filters when used with a MetalKit view. As of now, there are many more CoreImage filters, so that may give you the most flexibility.
The strategy in this tutorial is adequate for filtering video streams. If you want to give your users more flexibility with filters and other video tweaks, an editor such as VideoEditorSDK can let you focus on your application’s core functions and let a team of professionals worry about the video. Using VideoEditorSDK your users can apply filters to video as well as text annotations and more.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions or suggestions.

You like what we do? Check our careers page. Even if you can’t find your role listed, we might be interested since you are already here!

How To Record Screen Video Using ReplayKit for iOS

Walter — Tue, 21 Sep 2021 15:05:08 GMT

In this article, you will see how to use Swift and ReplayKit to add screen recording to your application. You will also see how to use the new rolling clips feature, introduced in iOS 15. The code in this article uses Swift 5 and Xcode 13. Clone this repository for a sample project and example code.

ReplayKit Basics

Apple introduced the ReplayKit framework in 2015 with iOS 9. They have refined and expanded it since then, but it still has a small API. ReplayKit uses the same shared components as AirPlay and the device screen recorder (accessible from the device Control Center). For this reason, ReplayKit functions will not work when using AirPlay or playing video. Also, the screen recording functions do not work on the Xcode simulators. You should test using a physical device.

Your app makes all its calls to the RPScreenRecorder.shared() object. This is a singleton object for your app that ReplayKit provides. There is no need to store it in a variable or pass it around in your code.

Rolling Clip Recording

In iOS 15, Apple introduced rolling clip recording. ReplayKit will maintain an updated video buffer of the most recent fifteen seconds. Your app can request the video at any time. ReplayKit comes from GameCenter. Apple’s idea for this feature is for creating a short, sharable video of the moments before an exciting event in a game. Another use can be for a user to record the steps leading up to a defect during testing. When the user is unlikely to know in advance that they will want a screen recording, rolling clips may be a good solution.

To start rolling clip buffering, include a method such as this one

RPScreenRecorder.shared().startClipBuffering { err in
  if let err = err {
    //either the user denied permission or something like
    //AVPlayer or AirPlay is using the shared recorder resources.
    print("Error attempting to start buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering started.")
  }
}

Screen recording has privacy implications. ReplayKit presents the user with a modal, requesting to record before it starts. If the user denies the request or if the buffering is unable to start, ReplayKit passes an Error object to the closure. Unlike other privacy modals, you do not have an opportunity to add your own language to the message. The modal appears as soon as you execute the .startClipBuffering command, so be sure that the user is not surprised by it. Execute the command at a logical transition in your application. After the user grants permissions, the modal will not appear again unless eight minutes pass before the next call to .startClipBuffering.

While the clip buffering runs, at any time, you can save the buffer to a URL on the device. The clip you save can be any duration up to fifteen seconds. If the recorder is unable to write the clip to disk, it will send an error.

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().exportClip(to: clipURL, duration: TimeInterval(15)) { err in
  if let err = err {
    print("An error? \(err.localizedDescription )")
  } else {
    print("Clip saved to url \(clipURL)")
}

The code above stores a fifteen-second clip at a URL in the user’s temporary directory. The temporary directory gets cleared by the system, but not while the app is running. To ensure that the files are not deleted except by the user, store them in the user’s documents directory instead.
When your app is done collecting video, use .stopClipBuffering to allow the system to clean up resources and stop the recording.

RPScreenRecorder.shared().stopClipBuffering { err in
  if let err = err {
    print("Error attempting to stop buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering stopped.")
  }
}

Recording Screen and App Audio

Rolling clip buffering runs in the background. You can also allow the user to control when to start and stop screen recording. The API looks similar and is available since iOS 10.

RPScreenRecorder.shared().startRecording { err in
  guard err == nil else { print(err.debugDescription); return }
  //code to display recording indicator
}

As with the rolling clip buffering, the system will display a permissions dialog. If the user denies permission you will get an error and the recording will not start.

The same eight-minute rule applies as for rolling clip recording. But, since the user has just taken some action to start recording, they should expect to see this modal.
Apple provides a basic view controller for editing and sharing the video.

ReplayKit passes the RPPreviewViewController to the .stopRecording handler.

RPScreenRecorder.shared().stopRecording { preview, err in
  guard let preview = preview else { print("no preview window"); return }
  //update recording controls
  preview.modalPresentationStyle = .overFullScreen
  preview.previewControllerDelegate = self
  self.present(preview, animated: true)
}

The preview object above is the RPPreviewViewController editor. You can display it as a modal or as a popover depending on your needs. The Apple provided editor will allow the user to save to their camera roll or share the clip after editing.

Notice in the code above that there is a previewControllerDelegate. You will need to add some code to the delegate to cleanup and dismiss the editor when the user is done sharing or saving. An example is below.

func previewControllerDidFinish(_ previewController: RPPreviewViewController) {
  previewController.dismiss(animated: true) { [weak self] in
  //reset the UI and show the recording controls
  }
}

Using Many `UIWindow` Objects

The ReplayKit recording will only capture the main window of your application. When designing for screen recording, you can place any UI elements in a separate UIWindow object. Although applications typically only have one UIWindow, iOS allows many windows. A UIWindow requires a .rootViewController to provide content. Then use it like any other view in your application.
The UI for this application consists of three UIWindow objects.

Apple places the status bar in a separate window from your application. The recording controls are in a separate window from the main application and will not appear in the screen recording.

To add some recording controls or a recording indicator to your app. Create a new UIWindow and add it to the same windowScene as your view.

controlsWindow = UIWindow(frame: CGRect(x: 0, y: 0, width: view.frame.width, height: 45 + view.safeAreaInsets.top)) //1
controlsWindow?.windowScene = view.window?.windowScene //2
controlsWindow?.makeKeyAndVisible() //3
let recordingIndicator = UIButton.systemButton(with: UIImage(systemName: "record.circle")!, target: self, action: #selector(recordingToggled(_:))) //4
let vc = UIViewController()
controlsWindow?.rootViewController = vc //5
vc.view.addSubview(recordingIndicator) //6
recordingIndicator.center = CGPoint(x: vc.view.center.x, y: vc.view.center.y + 20)

The code above creates the red/blue recording button and banner at the top of the view. It relies on a controlsWindow variable created at the class level. Here is what it is doing:

Creates a new UIWindow that is the width of the view and tall enough for the button and the safe area (the area around the notch and status bar)
Adds the window to the same windowScene as the current view
Makes the window visible (UIWindow objects are hidden when created) and brings it to the front of the stack of windows
Creates a button and links it to a method in your class recordingToggled(_:)
Adds a view controller to the window so that there will be some content
Adds the button to the view controller

Because the recording indicator and button are in a separate window, they will not appear in any screen recording. Be mindful when showing modal views or alert views, as they may appear behind the extra window. A good strategy is to hide the extra window when showing these views. You can use a similar strategy to hide any other UI elements that should not appear in screen recordings.

Going Further

Though Apple provides a basic editor, you can provide your own editing tools. The examples above showed the rolling clip editor writes the video to the disk. The regular recorder can also write the video to disk. You can stop recording using this code:

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().stopRecording(withOutput: clipURL) { err in
  //check for an error and process the url
}

This method would replace the .stopRecording { preview, err in method shown above.
With the video saved, you can use an editor such as the VideoEditor SDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to capture a rolling screen recording as well as how to capture the screen manually. Further, you saw how using an SDK such as VideoEditor SDK allows you to annotate and enhance your clips before sharing.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Almost Getting Sherlocked by Apple’s Core Image Team

Malte — Tue, 12 Jun 2018 00:00:00 GMT

During each World Wide Developer Conference keynote, app developers all over the world are fearing their app may get sherlocked by Apple. This has happened to flashlight apps when Apple introduced a simple toggle in the control center, to f.lux when Apple added exactly the same functionality to macOS and iOS, in parts to Dropbox with the launch of iCloud Drive and to many other developers. This year there was only one obvious of such things, the Workflow app which was acquired a few month ago and instantly turned into Siri Shortcuts by Apple. So no real ‘sherlocking’. But when we skimmed the newest APIs, release notes and session descriptions after the keynote, we found out that the technology behind our Portrait by img.ly app might have gotten sherlocked by the Core Image team. The reason is perfectly summarized in this tweet:

“Portrait Segmentation API

A new API for third-party developers allows for the separation of layers in a photo, such as separating the background from the foreground.”

Hi @halidecamera
— Preshit Deorukhkar (@preshit) June 4, 2018

How we found out about the new Portrait Segmentation API.

Quick spoiler: While Core Images new API is impressive, our technology is still ahead in terms of general availability, required hardware and cross-platform compatibility. But let’s start from the beginning:

We started working on automated image segmentation in 2016 and decided to focus on portraits in 2017. After spending most of the year on building a custom deep learning model, running hundreds of experiments and tweaking our post processing pipeline, we finally released the Portrait app in fall 2017. The app is able to generate portrait segmentations in real-time and allows the user to take a nice selfie, which is then automatically stylized using the segmentation mask and some post processing. This allows for sophisticated effects and got great reception all over the world. The app even got featured multiple times in many different countries. Recently we started looking into inferring depth, as we wanted to bring depth based features to our PhotoEditor SDK, but needed support on more iPhone devices, Android and especially required depth data for existing images.

Reading the tweet above, it looked like Apple just added the Portrait apps core functionality to the Core Image framework, making it available to all developers and users running or targeting iOS 12. A little worried about the new competition, we immediately looked into the docs and eagerly waited for the session on last Thursday to see if we’d soon need to find a new unique way of using technology to empower creativity.

We found out that all you need to do in order to get a segmentation mask along with your image is toggle the following flags when requesting a photo capture from your AVCapturePhotoOutput:

let settings: AVCapturePhotoSettings = AVCapturePhotoSettings()
settings.isDepthDataDeliveryEnabled = true
settings.isPortraitEffectsMatteDeliveryEnabled = true
...

After the photo is captured, you’re then able to extract the matte in the didFinishProcessingPhoto callback. And voila, you now have the captured image, a depth map and the mask separating fore- and background available:

Of course, we quickly spun up our own models and algorithms and compared the portrait matte, as well as the depth map to Apples results. As there is no way to rerun Apples matting algorithm for an existing image, we took one of Apples samples and compared their mask with results from our model:

While it may not have been the greatest idea to pick Apples shiny developer example, Apples mask is a little more detailed and for this particular image our model missed parts of the neck on the left. But overall we were still very happy with our results, especially when considering, that everything was generated from the plain image and didn’t require any dedicated hardware like dual cameras or a TrueDepth sensor. We did of course expect a superior depth map from Apple, as we’re clearly lacking data and are still actively working on the depth model used to generate the image above. But Apples depthmap interestingly has some issues around the neck as well, despite their use of a dual camera system. And keep in mind, that our results were all processed on the mobile device, entirely based on the RGB data contained in the image, and could be repeated on an Android phone, older iOS devices and even your browser.

When we wanted to try more samples, we quickly noticed the major limitations of Apples API: As the portrait matte capture is only available in combination with depth data, it’s limited to the iPhone 7 Plus, 8 Plus and iPhone X. And, most important to us, portrait mattes taken using the front camera are exclusively limited to the iPhone X and it’s TrueDepth sensor array. So for our Portrait app, switching to the new API would require us to ditch our live preview and go iPhone X and iOS 12 only. This was enough to calm our minds and we started thinking deeper about Apples technology:

While the depth requirement is annoying, it also explains the higher quality of Apples predictions. Our model is trained to solve the individual problems of portrait matting and depth map inference as a whole, but Apple is able to focus on improving the edges, while the ‘rough’ foreground/background segmentation is handled by masking based on the depth of the face detected within the image. We might be able to combine both models as well, but that would make the segmentation model currently used in the Portrait app obsolete and would most certainly kill the real-time functionality.

Overall, Apples technology is, as almost always, pretty impressive and the portrait matte generation works flawless, but we can now say, that for the use within our app, we reach a good quality with our current algorithms and think that a real-time preview is more important to our users. For our PhotoEditor SDK we’d happily use the high fidelity maps generated by iOS 12, but the hardware requirements are not yet suitable for SDK deployment. Our algorithms on the other hand, neither require a depth map, nor are we limited to processing after the image was captured, but can perform real-time inference on devices as old as the iPhone 6S. And the best of all: Once a TrueDepth camera is more common and everyone is running iOS 12, we can still use the Portrait Segmentation API and enjoy it’s simplicity.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Apple – IMG.LY Blog

How to Add a Filter to a Video Stream in iOS

Streams or Files

Setting up AVPlayer

Using AVPlayerItemVideoOutput

Creating the Display Link

Starting the Player

Extracting the Pixel Buffer

Going Further

How To Record Screen Video Using ReplayKit for iOS

ReplayKit Basics

Rolling Clip Recording

Recording Screen and App Audio

Using Many UIWindow Objects

Going Further

Wrapping Up

Almost Getting Sherlocked by Apple’s Core Image Team

Using Many `UIWindow` Objects