iOS App Development – IMG.LY Blog

CE.SDK v1.73 Release Notes

Neslihan — Tue, 21 Apr 2026 13:29:35 GMT

v1.73 is a mobile-focused release. The main act is our new architectural foundation for iOS and Android: Starter Kits give you full ownership of your editor configuration. On the engine side, CE.SDK now supports SVG export across all platforms.

Own Your Mobile Editor — Not Just the Config

The old model was simple: drop in a solution component (PhotoEditor, VideoEditor, DesignEditor), pass a license key, ship. For most teams, that was exactly the right starting point. Fast to integrate, zero configuration overhead, a full editor running in your app within hours.

As products matured, though, teams needed more. Deeper customization. Granular control over behaviors. An editor that felt like theirs.

That’s what Starter Kits for Mobile addresses.

The prebuilt solution components are replaced by a single Editor entry point and a set of Starter Kits: complete, working editor projects for each use case that live in your codebase, not ours. Set it up in minutes.

For your product, this means: you can adapt the editor to fit your product, not the other way around.

Everything Is In Your Code Now
Each Starter Kit is a real editor project — not a thin config wrapper. Every toolbar item, every callback, every loading phase is written out explicitly in files you own.

That means your team can read the integration, review changes to it, and understand exactly what the editor does and why. No more hunting through SDK internals to figure out why something behaves the way it does.

SDK Updates Stop Touching Your UI
Previously, updating CE.SDK could introduce new buttons, changed defaults, or different menus without you asking for them. Your QA had to catch it.

That dynamic is reversed now. Engine improvements, bug fixes, and performance gains come through updates as before. But your editor UI stays exactly where you left it. New features arrive as optional building blocks, documented and ready when you want them.

No More Customization Ceiling
The old configuration APIs had a line between what you could change and what you couldn’t. Starter Kits remove that line entirely. You can restructure initialization, inject custom logic between loading phases, and build workflows specific to your product.

If you can see it in the code, you can change it.

If none of the five kits match your needs, the Editor view accepts raw configuration directly.

Migration

This is a breaking change on both iOS and Android. Migration guides cover every change with before-and-after examples.

→ Android Migration Guide
→ iOS Migration Guide

Starter Kit Repositories

Editor	iOS (Swift)	Android (Kotlin)
Photo	GitHub	GitHub
Design	GitHub	GitHub
Video	GitHub	GitHub
Postcard	GitHub	GitHub
Apparel	GitHub	GitHub

Export Vector Designs as SVG

Until now, users working with vector content in CE.SDK had no path to get that content out as a vector file. CE.SDK v1.73 adds SVG export via the image/svg+xml MIME type.

Text is exported as vector paths for consistent rendering across environments regardless of font availability. Drop shadows, blur, effects, and embedded raster images are rasterized and included as embedded images within the SVG output.

→ Exporting Blocks Engine Guide

Changelog

See the full changelog for the complete list.

Thanks for building with IMG.LY.

Join 3,000+ creative professionals who get early access to new features and updates—subscribe.

CE.SDK v1.46 Release Notes

Neslihan — Tue, 11 Mar 2025 15:58:30 GMT

Welcome to a new release! CE.SDK v1.46 introduces a more flexible and intuitive user interface for iOS, allowing you to optimize your workflow with a customizable dock and inspector. Read on for upcoming features.

Shape Your iOS Interface Around Your Workflow

We unlock full dock and inspector bar customization, giving you the flexibility to optimize your workflow. With an open API, you can now tailor these UI elements to better suit your editing needs—whether by reordering actions, adding custom functionality, or simplifying the interface.

Take control of your interface by modifying both the dock and inspector bar:

Rearrange & Customize Actions – Prioritize your most-used tools by reordering or removing actions to create a cleaner, more efficient workspace.
Add Custom Buttons & Views – Extend functionality by integrating custom buttons or views that trigger specific actions, such as opening an asset library or displaying additional information.

These enhancements provide a seamless, intuitive editing experience that adapts to your unique workflow needs.

Start customizing your dock and inspector bar today with our documentation.

Upcoming

Boost Engagement and Accessibility with Video Captions
Captions are essential in today’s social media landscape, where videos often autoplay on mute and clear messaging is key. Our upcoming Video Captions for Web will allow you to:

Manually add subtitles to videos or upload an SRT file.
Adjust timing and formatting in an intuitive interface to match your brand’s style.
Embed captions directly into videos, ensuring they remain visible across all platforms.

Stay tuned—Captions are coming soon!

View all bug fixes and changes in the Changelog. Thanks for reading!

3,000+ creative professionals gain early access to new features and updates—don’t miss out, and subscribe to our newsletter.

CE.SDK v1.33 Release Notes

Neslihan — Thu, 29 Aug 2024 12:08:57 GMT

Welcome to the latest update of CE.SDK! We are bringing you new features that enhance your app’s creative capabilities and user experience. This release includes Configurations for a Design Editor or Photo Editor on iOS and Android, Plugins, Voiceover Capabilities, and more.

With this release, you can:

Integrate a Design Editor on iOS and Android

Get started with our new Design Editor configuration for iOS and Android. The out-of-the-box package is designed for easy creation and management of multipage designs.

Users can switch between editing and the page overview mode to review and adjust their work.

Keep the design process straightforward and efficient: the editor features a dock at the bottom, offering convenient access to essential tools like adding text, images, stickers, and uploading new assets. Additionally, you can insert a photo directly from your camera.

Key Features

Page Management
Create and manage your design pages within a template. It’s easy to create fun carousel posts, as seen on Instagram and LinkedIn, or create a series. Head to the ‘Pages’ icon on the top right bar. In the page overview, you can create, duplicate, and edit a page. Lastly, you can change the order of pages by moving them up and down.

Image Settings
Selecting an image lets you fine-tune adjustment settings, and add filters, effects, or blur. Easily crop, straighten, rotate, or flip images.

Try the Design Editor Configuration on iOS and Android.

Use Plugins to Customize UI and Add Quick Actions

We’re excited to introduce the first part of our plugin release for CE.SDK on the web: with this release, you can tailor the editor’s UI elements, reorder components, and add quick actions like background removal.

We’ve enabled extension points within key areas of the UI: the Canvas, Navigation bar, Inspector Bar, Dock, and Canvas Menu. Modify the appearance of existing features, register new components, and integrate quick actions. For example, you can add a “Background Removal” button directly in the Canvas Menu or create complex interactions like custom cut-out lines.

For more details on this first batch release and the upcoming plugins, please read our dedicated article.

Add Voiceovers to Videos on iOS

Add voiceover tracks directly within your mobile app, making your projects more dynamic and engaging. Whether creating a tutorial, narrating a slideshow, or adding commentary to a video, this feature allows you to seamlessly record, edit, and integrate voiceovers.

The popular feature, as enjoyed on TikTok and Instagram, is a staple for content creation and adds a personal touch to video creations.

To get started, navigate to “Add Audio”, select “Voiceover” section and start recording. You can preview the recording, re-record it, and add more audio, such as music. Adjust the audio levels for a balanced background track.

Try recording a voiceover with our Demo App for iOS.

Integrate a Photo Editor for iOS and Android

Our new Photo Editor Configuration is designed to offer an intuitive, out-of-the-box solution that bundles a range of key photo editing capabilities. Enhance individual photos, add text and shapes, or apply filters—this configuration is a kick-start to provide high-quality photo editing for iOS and Android with CE.SDK.

Key Features

Crop, Resize
Crop, straighten, flip, resize, and rotate your photos.

Adjustments
Fine-tune your photos by adjusting brightness, contrast, and other key settings, ensuring your images look their best.

Text, Shapes, and Stickers
Enhance your photos by adding and customizing text, shapes, and stickers. Whether you’re creating eye-catching titles, intricate shapes, or playful stickers—our configuration gives you the flexibility to adjust fonts, font colors, sizes, and positions to perfectly match your vision.

Effects, Filters, Blur, Blending Modes
Transform your images with fun effects, blur, and filters. Apply overlays to add depth and texture to your photos by adjusting blend modes and opacity to achieve the perfect effect.

Additionally, use our range of photo filters to set the mood, from warm and nostalgic tones to vibrant, modern effects, giving your images a distinctive and professional look.

Check our documentation to get started with our photo editor configuration, or try the photo editor on iOS and Android.

Improvements

Preview Audio on iOS and Android

We’ve made it easier to select the perfect audio clip for your projects. Listen to audio tracks directly within the interface before adding them to your video. Tap the play button of an audio clip in the asset library to preview it, and adjust your audio clip within the timeline.

Resize Pages

Resize pages in your web editor by simply grabbing corners and side handles, moving beyond the limitations of input fields or preset format sizes. This intuitive resizing integrates seamlessly with other adjustments, such as image cropping, ensuring a smooth workflow. Enable or disable this behavior as needed.

Create Rounded Corners

You can now easily create rounded corners for images on the web, iOS and Android. Within your designs, you can add a sleek and modern touch to your projects.

Drag and Select Multiple Elements

You can now click and drag with your mouse to draw a selection rectangle, instantly selecting all blocks within the covered area. This feature saves you time and effort, working with complex templates.

Thanks for reading! Don’t hesitate to reach out if you have any questions or need assistance.

Over 3,000 creative professionals gain early access to our new features, demos, and updates—don’t miss out, and subscribe to our newsletter.

Please note that these release notes also include improvements from previous versions.

CE.SDK v1.23 Release Notes

Neslihan — Thu, 28 Mar 2024 09:47:34 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. In the past month, we have received valuable insight from you in our exclusive Beta testing. Now we are excited to announce CE.SDK v1.23!

Record and Edit Short-Form Video Content on iOS

Video Content Creation is here! Your users can record videos, blend them with audio, text, and graphics on a timeline. This includes features like countdown timer, and the popular split screen modes for reactions and duets. You may recognize these from Instagram Reels or TikTok. Making adjustments is straightforward using filters, effects, and adjustment settings.

Record with a New Camera

Record a video and utilize the Dual Camera feature. This allows you to record simultaneously with your front and back camera, producing a split-screen video. Determine when the recording begins using a countdown timer.

Edit on a Timeline

Users can reorder, arrange and split clips on a timeline, and use filters, visual effects, along with other adjustments to modify the appearance of clips and images. Add audio tracks, image or video overlays, format text, integrate stickers, and use tools for cropping, filtering, blurring, and adjusting images and videos. Try recording and editing Video Content with our Demo App for iOS.

Get started with our documentation.

Quickly Add Editors to Mobile Apps with UI Packages

In the past, integrating our editors into your app required developers to manually collect code from our showcases and piece together the UI. This not only demanded effort but also necessitated frequent updates to ensure compatibility with each of our new releases.

To free your time and resources, we are now introducing ready-to-use UI packages for iOS and Android, that bring advanced editors directly to your app. This eliminates the previous labor-intensive process. These packages include a wide range of options, from a fully equipped Video Editor (iOS) and Camera UI (iOS) to specialized examples like an Apparel Editor or Postcard Editor.

We maintain UI packages for you, ensuring compatibility across devices and development environments. Now, you can immediately start with fully featured editors, staying up-to-date with just a click, and without manual intervention. Get started with our documentation.

Integrate a User-Friendly Editor

We’ve made enhancements for a more user-friendly editor setup of CE.SDK.

To enhance legibility and perception of our UI elements, we have reworked our UI colors, making sure they meet the minimal contrast requirements for an improved experience. Additionally, the active state of settings is now more apparent than it was previously.

Next, libraries and all inspectors open on the same side of our default editor now, instead of on both sides. This reduces mouse travel, saves space, and streamlines workflows.

Additionally, you can now resize images more easily by dragging from any side, not just using the handles. This same feature is also applicable when cropping your image.

Lastly, elements stay selected after grouping or ungrouping. In the past, changing group settings would deselect your elements.

Integrate A Smooth Video Editor into your App

We recently launched our Video Editor for the Web. Since the release, we significantly improved the playback performance, leading to smoother editing and playback.

The timeline lets you arrange and edit videos, photos, text, audio, shapes, and stickers. Enhance projects with filters or adjustments, and add a personal touch with customizable audio. Precisely split, trim, and replace elements. Add audio files to make your project stand out. Try editing yourself with our Showcase for Sales Outreach Videos.

Please note that these release notes also include improvements from v1.22.

Thanks for reading. Join over 3000 specialists with powerful apps, and subscribe to our newsletter. We keep you in the loop with brand-new features, early access, and updates.

CE.SDK v1.21 Release Notes

Neslihan — Thu, 22 Feb 2024 16:22:28 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we are happy to introduce CE.SDK v1.21!

Create Videos With A New Timeline for Web

Create eye-catching videos within your web app for any platform, including Instagram, TikTok, or YouTube! Here’s what’s new: the integrated timeline supports the arrangement of elements like videos, photos, text, audio, shapes and stickers. Users can fine-tune their projects with adjustments or filters, and personalize their creations with audio from a customizable library.

To simplify your content creation, use our formats that automatically set your canvas size for social media and other popular video resolutions.

Effectively arrange and edit multiple elements within your videos, offering precise control over the editing process. Easily split, trim, and replace. Lastly, make it pop by adding audio files to the composition.

Creating templates with placeholders for your users is just as easy. Simply select an element, click ‘Placeholder’, and decide which elements can be edited.

To integrate, browse our Guide. Try the video editor live demo yourself. This showcase contains templates, audio and video examples for you to try.

Keep Designs in Focus on iOS

Our new Camera Zoom Limits make editing a design easier. Your page stays in focus to avoid accidental scrolling outside the design. The zoom level is set within a minimum and maximum range, ensuring a practical, usable view at all times. When you select an element in your design, it stays in focus when opening adjustments or other inspectors. It allows you to immediately see changes made to the selected element. This improvement is available for iOS and Android, and can be tested in our Design Editor Demo on the App Store and Google Play.

Ensure Color Purity in Print

Our newly introduced Underlayer API guarantees the preservation of color purity whenever you export designs using our CE.SDK for print. This solution specifically addresses the challenge of maintaining the richness and vibrancy of colors, irrespective of the color of the canvas, which could range from apparel to paper.

When you’re ready to export your design, this feature adds a special ‘underbase’ layer, underneath your design. This layer, typically a white or specific primer coat, makes sure the final printed colors match your original vision perfectly.

Very soon: Integrate Short-Form Video Creation

Video remains key for attracting, engaging and retaining audiences, especially on mobile. Creating a video editor can be tough, thus we’re thrilled to soon release our turnkey solution for mobile video content creation.

Combined with our new Camera SDK, users can seamlessly blend video, audio, text, and graphics on a sleek timeline. It includes features like voiceover, zoom, tap to record, and the popular split screen modes for reactions and duets. You may recognize these from Instagram or TikTok!

Last chance: Secure your waitlist spot to our Video Content Creation Release. You will also receive access to our TestFlight app.
Registration is only open until March 6, 2024.

? Gain Access to our Video Content Creation Release
? Download Feature Exposé instantly (PDF)

We can’t wait to hear your feedback!

Thanks for reading. Join over 3000 specialists with powerful apps, and subscribe to our newsletter. We keep you in the loop with brand-new features and updates.

CE.SDK v1.20 Release Notes

Neslihan — Mon, 29 Jan 2024 10:37:42 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we are happy to introduce CE.SDK v1.20!

With this release, you can:

Create Unique Element Combinations

Platforms: iOS, Android, Web, API

Forge limitless combinations of photos, videos, texts, and shapes with our new boolean operations feature. This dynamic suite includes operations such as Combine, Subtract, Intersect, and Exclude.

Subtract text from a shape, or seamlessly merge a distinctive shape with a lively video clip. With these powerful tools, your creativity knows no bounds, offering the freedom to craft truly unique design elements.

Refine Images with iOS Appearance Settings

Breathe life into your content and set the ambiance! We’ve recently updated the user interface on iOS to include Appearance Settings. Now, users can enhance their visual content with advanced improvements. This includes precise adjustments, creative filters, effects, and blur options.
Experience our new features by using the free Demo Design Editor, now live on the App Store.

Optimize High-Quality Asset Imports

Importing high-quality assets, like photos, can strain your device and impact your editor performance. Imagine editing on your phone and just importing a 5k pixels stock photo. Our smart engine steps in, creating an image source set with different resolutions: The intelligent engine selects the optimal target resolution for your asset and seamlessly switches to it.

The technical scoop? This update minimizes network bandwidth, loading times, and runtime memory usage.

Keep Designs in Focus on Android

With our new Camera Zoom Limits, editing a design is simpler. Your page remains in focus to prevent accidental scrolling outside the design. The zoom level is set within a minimum and maximum range for a sensible, usable view at all times. Lastly, when you select an element in your design, it will remain in focus when opening adjustments or any other inspectors. This allows you to instantly view changes made to the selected element.
Try our live Demo Design Editor for free, on Google Play.

Exclusive Access: Integrate Short-Form Video Creation

Video continues to be the most effective method for attracting and retaining your audience. However, creating a video editor from scratch can be challenging. We’re excited to announce the upcoming release: your out-of-the-box solution for video content creation!

Download Feature Exposé (PDF)

Give your users the ability to arrange video, audio, text, and graphics seamlessly on a sleek video timeline with our new Camera. This includes popular features like Voiceover, Zoom, Tap to Record, and more.

Additionally, we include the beloved Split Screen Modes for Reactions and Duets, similar to those on TikTok and Instagram.

Gain exclusive early access to our Video Content Creation Release.

Thanks for reading! Join over 3000 app specialists, and subscribe to our newsletter. We keep you in the loop with brand-new features and updates.

Time-Based Sprites for VE.SDK on iOS and Android

Neslihan — Mon, 05 Sep 2022 07:06:05 GMT

We are happy to extend VideoEditor SDK with a highly-requested feature: Time-Based Sprites. This new feature sets the duration of text and stickers in your video timeline. Users may now place fun stickers and text at the right moment, and give their videos a special touch.

Time-Based Sprites

The popular feature known from TikTok and Instagram Reels is now available in VE.SDK: set the starting and end point of your text or sticker. Tap your text or sticker and select Duration to determine the duration of your asset.

Unless you have specified a custom set of sticker or text actions, this feature is enabled by default since VE.SDK v10.3.0 for Android and v11.3.0 for iOS. See the official documentation for iOS or Android on Time-Based Sprites.

Why is Video Important?

Video content has become an essential medium for social media, marketing, sales, and support teams. According to marketing statistics, people are watching an average of 19 hours of online video per week in 2022. 88% of people say that they felt convinced to buy a product or service by watching a brand’s video. The growing trend and preference of consumers are why businesses are shifting their attention toward including videos in their strategies and applications.

We commit to extending our video editing features to help developers meet the demand, save resources and streamline the process of building great applications. Let us have a look at our previous VE.SDK releases:

Video Composition
Users may seamlessly edit their footage by trimming and adjusting video files with advanced filters. Finally, they can set the correct order of their video sequences to create a single composition.

Audio Support
Replace or add sound in videos by loading audio files. Users can trim their audio according to their footage. Developers can provide media libraries for audio and video files. That way, users may access media by choosing from labeled folders, such as genre, theme, or artist names.

Force Trim
The VE.SDK trim tool allows users to determine the start and end frame of a video clip and change the duration of their footage. Now you can enforce a minimum and maximum length of videos. Force Trim will come in handy for use cases, such as social media stories and posts popularly limited to bite size 15 or 60 seconds by widely loved apps – see TikTok or Instagram. Adopting a ready-to-use solution like VE.SDK will set you on your path to creating equally beautiful apps.

Thanks for reading! Let us know what you think on Twitter – or check out our Newsletter for more accelerating updates.

How to Remove Backgrounds Using Core ML

Walter — Tue, 28 Jun 2022 15:26:53 GMT

In this tutorial, you will learn how to use machine learning to identify an image background and mask it out. This is particularly useful for making stickers or avatars or adding a fake background to a video call. The process of assigning the pixels in an image to a specific object is called “segmentation”. Apple provides an optimized method with pictures of people in its Vision framework. For performing the same tasks with non-human subjects, you can use the DeepLabV3 machine learning model with Core ML. The code examples in this tutorial have been tested using Xcode 13 and Swift 5. Because of the use of Core ML, Vision, and CoreImage in this tutorial, you should run the demo code on a device, not on the Simulator. An iOS project with the demo code is on GitHub.

Image segmentation is a different process and requires other machine learning models than image recognition. With recognition, the model produces bounding rectangles that the system believes to contain the entire object. With segmentation, the model identifies the actual pixels of the object.

In this tutorial, you’ll start with an image of your subject. Then you’ll generate a mask for the background using Vision. Finally, you’ll use CoreImage filters to blend the original image, image mask, and the new image background.

Whether using the Vision framework alone or supplementing Vision with another Core ML model – the process will be the same:

Create a Vision request object with some parameters.
Create a Vision request handler with the image to be processed.
Process the image with the handler and object.
Process the results into the final image using CoreImage.

When working with still images, Apple’s CoreImage framework is usually the best option, mainly because of the large number of available filters and the ability to create reusable pipelines that you can run on the CPU or the GPU.

Using Vision to Segment People

Without an external model, Vision can only segment documents or people in an image. It is vital to understand that Vision will only identify a pixel in the image as “this pixel is part of a person” or “this pixel is not part of a person.” If an image contains a group of people, Vision will not be able to separate individuals.

To segment the image, start by creating an instance of a VNGeneratePersonSegmentationRequest.

var segmentationRequest = VNGeneratePersonSegmentationRequest()
segmentationRequest.qualityLevel = .balanced

This type of request has a few options you can set. .qualityLevel can be .fast, .balanced or .accurate. The level of .accurate is the default. This will determine how closely the mask conforms to the boundaries of the original image. The different levels process at different speeds. The .fast setting is intended for use in a video so that frames don’t get dropped. Using .balanced or .accurate causes noticeable delay in an app processing the image on most devices. Experiment with different settings depending on your needs.

The output of the segmentation request will be a CVPixelBuffer. This is a structure that contains information for each pixel of the image. Most of Apple’s video frameworks as well as CoreImage can work with pixel buffers. The default for VNGeneratePersonSegmentationRequest is a buffer where the color of each pixel is represented by an 8-bit number. Any pixel that Vision thinks contains part of a person will be white and any not-a-person will be black and represented by zero. This will be exactly what you want for generating a mask to work with CIBlendWithMask.

Next, create a VNImageRequestHandler with the image to be processed. The handler class has a number of initializers for different types of data. In this example we will use CGImage, but you could also start with CIImage, CVPixelbuffer, Data, or others. You can also specify options for the handler. In this tutorial we will not specify any, but one of the options is to pass in a CIContext. This can help with performance as you can tell your app to do all of the processing with a single context on the GPU using Metal.

guard let originalCG = originalImage?.cgImage else { abort() }
let handler = VNImageRequestHandler(cgImage: originalCG)
try? handler.perform([segmentationRequest])
guard let maskPixelBuffer =
  segmentationRequest.results?.first?.pixelBuffer else { return }
let maskImage = CGImage.create(pixelBuffer: maskPixelBuffer)

In the code above, the originalImage (which happens to be a UIImage) gets converted to a CGImage. Then we use the image to initialize a request handler. The VNImageRequestHandler has a method .perform which takes an array of all of the Vision requests you want to use to process the image. The .perform method will not return until all of the requests in the array have been completed. Depending on how you want to structure your code, you can either provide a completion handler to use with each of the requests or just process the results in-line. In this example, we’ll process in-line.

If the segmentationRequest found any people in the image, the results array will contain a .pixelBuffer. Create the mask we need by converting the pixelBuffer into a CGImage. To convert to a CGImage, the example uses some helper methods that Matthijs Hollemans published to GitHub specifically for working with Core ML inputs and outputs.

Now that we have the mask image, use CIFilter to compose the final image. It will take a few steps. First, resize the new background, mask, and original image to the same size. Then, blend the three images.

//Convert main image to a CIImage and get the size
let mainImage = CIImage(cgImage: self.originalImage!.cgImage!)
let originalSize = mainImage.extent.size
//Convert the maskimage to CIImage and set the size
//to be the same as the original
var maskCI = CIImage(cgImage: maskImage!)
let scaleX = originalSize.width / maskCI.extent.width
let scaleY = originalSize.height / maskCI.extent.height
maskCI = maskCI.transformed(by: .init(scaleX: scaleX, y: scaleY))
//Convert the new background to a CIImage and set the size
//to be the same as the original
let backgroundUIImage = UIImage(named: "starfield")!.resized(to: originalSize)
let background = CIImage(cgImage: backgroundUIImage.cgImage!)
//Use CIBlendWithMask to combine the three images
let filter = CIFilter(name: "CIBlendWithMask")
filter?.setValue(background, forKey: kCIInputBackgroundImageKey)
filter?.setValue(mainImage, forKey: kCIInputImageKey)
filter?.setValue(maskCI, forKey: kCIInputMaskImageKey)
//Update the UI
self.filteredImageView.image = UIImage(ciImage: filter!.outputImage!)

The above code resizes the images to match the size of the original image and converts them to CIImage. Many machine learning models commonly resize inputs and outputs. For example, the output of the VNGeneratePersonSegmentationRequest using the demo image is 384x512.

Both CIImage and UIImage formats describe an image but do not always have a bitmap representation. That is why each image gets converted to a CGImage first. Though this guarantees that the demo code will work, converting formats makes the code run slower. In your app, you should experiment with different methods to get your images into CIImage format for filtering. Also, the above code uses Matthijs’ resized helper method to do some of the resizing.

With everything resized, the CIBlendWithMask filter stitches the images together. In place of the black pixels in the mask, the final image will show the background – for white pixels, the foreground. Wherever the mask image has a gray pixel, the final image will blend background and foreground.

Now let’s see how to use an additional CoreML model to process images that don’t have a person as the main subject.

Choosing a Model

Apple provides a number of pre-built models for text and image processing. The DeepLab v3 Machine Learning model can perform segmentation requests on images that have subjects like dogs. You can download it from Apple directly or also find it at the DeepLab repo on GitHub. Once you have downloaded the model, add it to your Xcode project the same as any other file. The DeepLab model will recognize people, the same as the VNPersonSegmentationRequest, but also other objects. This does come at a cost in an increased filesize for your application. You may notice that Apple provides multiple versions of the model of different file sizes. They all perform the same task, but differ in how they represent the output and a few other things.

Models on Apple’s website are packaged to work with Core ML and Xcode, so you can easily try a different model. The input and output method names will be the same. It is beyond the scope of this tutorial, yet Apple provides tutorials and example scripts on how to convert TensorFlow and other machine learning models to work with Core ML.

Core ML and Xcode

When you are working with a Core ML model, Xcode provides some convenient tools. Access them by highlighting the name of the model in the File Navigator pane of Xcode. For instance, clicking “Preview” will let you test the model with your data.

You can also use the “Predictions” tab to determine how the input needs to be formatted and what to expect for the output.

Here you can see that the DeepLabV3 model expects images to be 513 x 513 and will output a multidimensional array of integers that is 513 x 513 in size.

Finally, on this screen, you can see an entry for the “Model Class” in the headers. Double-click on the name to jump into the Swift wrapper class for the model to see how to call the model in your code and work with the inputs and outputs.

The DeepLabV3 Model

The DeepLab model input will be a color image that is 513 x 513 pixels. The Vision framework will handle resizing the input, but you can provide options on how that resize should work. The DeepLabV3 model has been trained to recognize and segment these items:

aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
dining table
dog
horse
motorbike
person
potted plant
sheep
sofa
train
tv or monitor

Anything that the model does not recognize, it will consider a background. After performing the recognition, the model returns a two-dimensional 513 x 513 array. Each entry in the array corresponds to one pixel in the original 513 x 513 input image. For instance, every pixel in the original image that shows a dog will be represented in the output by a 12 in the corresponding array entry. Any pixel that is not a recognized object will be given a 0 in the output array to represent a background.

If the model recognizes multiple objects, there will be multiple numbers in the array. If that is not what you want, you need to make changes to the array before creating the mask. For example, using an image of a dog riding a horse, some entries in the array will be 12, others will be 13, and the rest will be 0. To filter out the horse, you need to loop through the array and change any 13s to 0s.

Segmentation with DeepLab

To use the DeepLab model in your code, you again use a request to the Vision framework but this time you can specify a model.

let config = MLModelConfiguration()
var segmentationModel = try! DeepLabV3(configuration: config)
if let visionModel = try? VNCoreMLModel(for: segmentationModel.model) {
  self.request = VNCoreMLRequest(model: visionModel)
  self.request?.imageCropAndScaleOption = .scaleFill
}

In the code above, we create an instance of the DeepLab Core ML object and then initialize a generic VNCoreMLRequest with its model. The only option we set on the request is to tell it how to modify the image when it resizes it for input.

Now the code is very similar to the first example.

let cgImage = originalImage?.cgImage
let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
try? handler.perform([request])
if let observations = request.results as? [VNCoreMLFeatureValueObservation],
  let segmentationmap = observations.first?.featureValue.multiArrayValue {
  guard let maskUIImage = segmentationmap.image(min: 0.0, max: 1.0) else { return }
  applyBackgroundMask(maskUIImage)
}

Because we are using a generic VNCoreMLRequest we have to cast the results as VNCoreMLFeatureValueObservation. The DeepLab model returns a .multiArrayValue instead of a .pixelBuffer so we will again rely on the helper methods to convert it to an image. Now that the mask image has been created, applying the background is the same as in the original example.

Going Further

This tutorial focused on still images, yet both the standard Vision segmentation requests and DeepLabV3 allow processing video input as they can work with CVPixelBuffer and VNSequenceRequestHandler.
The rest of the process will be almost identical to our example. You will need to finetune the performance, or else you will experience frame drops.

Apple provides another method for identifying the background and primary subject in a still image: Portrait mode. When a device renders Portrait mode, it takes pictures with all cameras on the device and stitches them together. This way, the depth of field can change, and you can adjust what items are in focus or blurred.

If the DeepLab model does not cover your subject matter, you can train the DeepLabV3 model to recognize other objects using your own data.

However, you may consider using SDKs such as PE.SDK and VE.SDK. Enabling background removal for your users is easy — all it takes is a few lines in your configuration as described in the official PE.SDK documentation for iOS and Android, which adds a control to the photo editor to toggle the setting.

Integrate a fully customizable photo editor with Background Removal into your app with PE.SDK.

The latest VE.SDK release introduced Background Removal for stickers. This feature recognizes people in pictures and removes the background with one tap – no need for manual outlines or masks.

Let your users create beautiful visuals with Sticker Background Removal.

This feature is available on Android and iOS 15.0 and higher only. See the official documentation for Sticker Background Removal on Android or iOS.

Using an SDK will simplify and accelerate your app development, as you can save time and resources, and focus on the growth and innovation of your application instead.

Wrapping Up

In this tutorial, you saw how to use Vision and Core ML to segment an image of a person and remove the background. You also saw how to use DeepLab to work with images of non-persons.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.

Working with Large Video and Image Files on iOS with Swift

Walter — Tue, 10 May 2022 13:27:44 GMT

Video files and image files are among the largest files. The AVFoundation and CoreImage libraries that Apple supplies widely store data on the disk. They try to bring into memory only as much of a file as is needed to complete a task. As you are working on an application that uses files, there are times you will want to compress or resize them. Usually, this is to upload to a server or because you have noticed performance issues, such as stuttering when scrolling or long render times.

This article will cover strategies for compressing and resizing images, videos, and other files. We will also be looking at using URLSession for file uploads and downloads.

As iPhone cameras can take higher-quality pictures, the file sizes have grown quite large. For the last few years, Apple has been using a 12MP camera that takes pictures at 3024 x 4032 resolution. Apple uses a special HEIC compression to make each image about 1.7MB. The same image will generate a 15MB file when saved as a .png and a 3MB file when saved as a .jpeg. Though this will be a high-quality image, services like Instagram or Facebook will often only display images at less than half of that size. The extra pixel data will be compressed away. Perhaps the first and most important part of working with large image files in your app is planning around what size or quality you need. Social Media services similarly resize video files. For instance, TikTok displays images and video using 1080 x 1920, but any iPhone after X will capture video at a larger size.

Writing an Image to Disk

When working with a UIImage or CoreImage object, normally you can save it as an NSData object. These files are lossless (the image quality will not degrade), but they are only useful inside of your application. The standards on the Internet are jpeg and png files.

Fortunately, Apple provides methods to convert UIImage to .jpegData or .pngData. There are also ways to convert CIImage and CGImage, but they are slightly more complicated to configure.

let imageToSave = UIImage(ciImage: <someImage>)! \\1
let jpegData = imageToSave.jpegData(compressionQuality: 1.0) \\2
let file = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first \\3
do {
  try jpegData?.write(to: file!.appendingPathComponent("sleeping_dog.jpg")) //4
} catch(let err) {
  print(err.localizedDescription) //5
}

Here is what’s going on in the code above:

First we need to convert our image to a UIImage. The UIImage class has several different initializers. Use UIImage(named: "some image name") to read an image from the app bundle. Use UIImage(contentsOfFile: "some filepath") to read from the disk. Use UIImage(cgImage: "some cgImage") and UIImage(ciImage: "some ciImage") when you have images already in memory. If you are storing images in a CoreData store or somewhere as Data you can use UIImage(data: "some data").
However you get a UIImage the next task is to convert it to either a .jpg or .png file. Generally, .jpg is a better option when working with photographs and .png produces higher quality for line art. The example above creates a .jpg at the highest quality possible.
Ask the FileManager for a URL to the user’s documents directory. That is generally a safe place to write files. Because of the way the function is formed, it returns an array of URL items. In this case, there is only one, but you still need to remember to specify the .first item.
Writing Data object to disk can throw errors, so it is good practice to place it in a try…catch block of code. The URL to the user’s documents directory needs to have the actual filename appended.
Any errors that the system throws can be handled here.

In order to create a .png using the imageToSave above, the line would be

let pngData = imageToSave.pngData()

The .png filetype does not have any compression settings. The filetype is “lossless”, so it is always compressed as much as possible without losing any data.

Compression Options for JPEG

When generating jpeg data above, we supplied a quality value. The value is a Float between 0.0 and 1.0. The .jpg filetype uses a “lossy” compression algorithm that is specifically designed for photographs.

In the image above saving the jpeg with a quality of 1.0 produced the dog on the right and the file is 1.3MB. Saving the jpeg with a quality of 0.0 produced the image on the left and the middle image is produced saving with a quality of 0.6. The file on the left is only about 6% of the size of the original.

Considering what the images are being used for can help you decide on an optimal compression setting for a particular app.

Compressing Data with `zlib`

Though png and jpeg files are already compressed sometimes you will want to compress other types of files before uploading them (compressing a png, mov or jpeg will often not affect the file size or will make the file larger). A web server will often be able to accept a POST that compresses its body data using gzip. Additionally, some web servers will require that binary data is transformed into a base64EncodedString before uploading. Base64 encoding will make data size grow by about 33%. Using gzip will help mitigate that somewhat. Apple’s implementation of gzip is called .zlib when compressing data. Swift provides some other compression algorithms such as .lzfse which will result in smaller files, but they are not widely adopted.

For example, to encode something as base64 and then compress it with with gzip so you can upload it, we can use code like below

let jsonToUpload = "{"books":[{"title":"The Three Musketeers","author":"Dumas"}]}"
let jsonAsData = jsonToUpload.data(using: .utf8)
let encodedString = jsonAsData?.base64EncodedString()
let compressedString = try? NSData(data: (encodedString?.data(using: .utf8))!).compressed(using: .zlib) as Data

The code above creates a string of data in JSON format. It then encodes the String as a Data object using .utf8 encoding. Using .utf8 is a standard. If the web server you are uploading to requires a different encoding, you can change it. In the final step, the encoded string is compressed using gzip. Notice that the .compressed(using:) method is on NSData not on Data. NSData is an older library, so you convert the compressed NSData object back to Data by casting using as Data.

The above example was just to show syntax. The compressedString will be slightly larger than the encodedString. Based on the kind of data your app works with and the capabilities of the web server, you will need to experiment with how to encode and compress to get the best results.

Transferring Data With a Server

Depending on the original format of the data, your web server may want the .httpBody compressed. You can do that with some code like this:

let someJSONData = Data() //1
var request = URLRequest(url: URL(string: "https://example.com/uploads/")!) //2
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
if let compressedJSONData = try? NSData(data: someJSONData).compressed(using: .zlib) as Data { //3
  request.httpBody = compressedJSONData
}
let task = URLSession.shared.dataTask(with: request) {data, response, error in
  //check for errors and response data when the task is done
  //response will contain the status and other header messages
  //data will contain any payload the server returns
}
  task.resume() //4

Here is what the code above is doing:

Get the payload into a Data object.
Create a URLRequest as a var so that you can configure it. If you create it as a let it will be a GET request with default header fields.
Use gzip to compress the data and if successful, assign it to the .httpBody of the request.
Create a data task for the upload. Upon completion, the data, response and error variables will have any response from the server.
Actually start the task.

Uploading a Large File

An issue with the code above is that all of the initial data will be in memory. So for a large image or video upload, this may not work. However, Swift does provide a different method that will upload a file directly from the disk, reading into memory only what is needed at the time.

Code to implement that would follow this form.

var request = URLRequest(url: URL(string: "https://example.com/uploads")!)
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first
let fileToUpload = documentsDirectory?.appendingPathComponent("the-big-giant-file.mp4")
let uploadTask = URLSession.shared.uploadTask(with: request, fromFile: fileToUpload) {data, response, error in
//get some information from the server when the file has been uploaded
}
uploadTask.resume()

This looks very similar to the previous example but notice that you do not set an .httpBody. There will be header values to set though. Most web servers will want to know how large the file is, authentication credentials, and whether the data is multi-part, or raw or other metadata. These will all be set using .setValue(forHTTPHeaderField:) in the request object.

For larger files, it is often better to use a URLSessionDataDelegate instead of the single closure with data, response, error. When using the delegate, you can write code to monitor the progress of the file transfer as well as respond to authentication challenges and make decisions about caching. You can also implement URLSessionDelegate and URLSessionTaskDelegate methods. All of the delegate methods are options, so you only need to implement ones that are important to your application.

Configuring Background Transfers

Another benefit of using the uploadTask (and its related downloadTask) instead of the dataTask is that you can configure the transfer to continue for a time even if the app goes to the background. In that case, instead of using the URLSession.shared object, you will want to configure a session that has some special properties to allow it to work in the background.

let configuration = URLSessionConfiguration.background(withIdentifier: "some.unique.identifier")
let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
let uploadTask = session.uploadTask(with: request, fromFile: fileToUpload)

Completely configuring and supporting the downloads and backgrounding is covered in this article by Apple. It will require implementing some delegate methods for the session as well as updating some of the methods in the main application delegate. The most important is the handleEventsForBackgroundURLSession which is how the system will wake up your app with information about the upload or download.

Additionally, download tasks can be paused and restarted. The cancel(byProducingResumeData:) method on URLSessionDownloadTask will cancel a download and optionally provide some resume data you can use to resume the download at a later time. You can call this method on the task when the user taps a button or when the app goes to the background or similar. The web server must support byte-range requests and ETag or Last-Modified headers. The documentation for pausing and restarting explains how to pause the download and how to resume using downloadTaks(withResumeData:). There is not a similar structure for pausing and resuming an upload.

Resizing an image

Perhaps the simplest way to make a file smaller is to reduce its dimensions. Using CoreImage you can use the Lanczos resampling algorithm to resize an image while keeping high quality.

let url = Bundle.main.url(forResource: "sleeping-dog", withExtension: "jpeg")
let image = CIImage(contentsOf: url!) //1
let filter = CIFilter(name: "CILanczosScaleTransform") //2
filter?.setValue(image, forKey: kCIInputImageKey)
filter?.setValue(0.5, forKey: kCIInputScaleKey) //3
let result = filter?.outputImage
let converter = UIImage(ciImage: result!) //4

Here is what this code is doing:

Create a CIImage from a URL.
Create a CILanczosScaleTransform filter.
Configure the filter to reduce the image dimensions by 50%.
Convert the scaled image to a UIImage.

Recently, Apple has provided a UIGraphicsImageRenderer that will work directly on UIImage objects. The code below will resize a UIImage to 50%. Notice that the UIGraphicsImageRenderer can also use an explicit value for size (unlike the CILanczosScaleTransform)

let imageToSave = UIImage(named: "sleeping-dog.jpeg")! //1
let newSize = imageToSave.size.applying(CGAffineTransform(scaleX: 0.5, y: 0.5)) //2
let renderer = UIGraphicsImageRenderer(size: newSize) //3
let scaledImage = renderer.image { (context) in
  let rect = CGRect(origin: CGPoint.zero, size: newSize)
  imageToSave.draw(in: rect) //4
}

Here is what this code is doing:

Load the image as a UIImage.
Scale the size down by half.
Create a UIGraphicsImageRenderer that will render in this new size.
Actually draw the image into the renderer at the new size.

In addition to rendering the image back as a UIImage the renderer can create jpegData and pngData which would save a few steps if you are resizing and converting. The full documentation for UIGraphicsImageRenderer explains the details.

Resizing a Video

Though the UIGraphicsImageRenderer is faster, using the older CoreImage resizing has a benefit. CoreImage filters can be applied to video files as a composition. Large video files can be scaled down. The code below will create a composition that will scale a video asset by 50%. Remember that you will need to import AVFoundation to use these tools.

let newAsset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
var newSize = <some size that you've calculated> //2
let resizeComposition = AVMutableVideoComposition(asset: newAsset, applyingCIFiltersWithHandler: { request in
  let filter = CIFilter(name: "CILanczosScaleTransform") //3
  filter?.setValue(request.sourceImage, forKey: kCIInputImageKey)
  filter?.setValue(<some scale factor>, forKey: kCIInputScaleKey) //4
  let resultImage = filter?.outputImage
  request.finish(with: resultImage, context: nil)
})
resizeComposition.renderSize = newSize //5

Here’s what the code above is doing to create a new composition:

Create an AVAsset from a URL.
Create a variable newSize to hold the final size.
In the composition, configure the CIFilter that will be applied to each frame.
Calculate the scale factor based on the newSize variable and the actual size of the request.sourceImage.extent.size.
Set the renderSize property of the composition to the new size.

If you don’t set the renderSize then there will be a black letterbox around the video.

With the resizing composition, you can now export the AVAsset as a .mov or .m4v file using an AVExportSession.

let asset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("exported.mov") //2
//create exporter
let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)
//configure exporter
exporter?.videoComposition = <the composition you created above> //3
exporter?.outputURL = outputMovieURL!
exporter?.outputFileType = .mov
//export!
exporter?.exportAsynchronously(completionHandler: { [weak exporter] in //4
  DispatchQueue.main.async {
    if let error = exporter?.error {
      print("failed \(error.localizedDescription)")
    } else {
      print("file saved at \(outputMovieURL)") //5
    }
  }
})

Here’s what the code above is doing:

Load an AVAsset from a URL.
Create a URL to save the resized movie.
Apply the resizing composition to an AVAssetExportSession.
Asynchronously export the asset to disk.
Print the destination URL of the new movie when it is saved.

Another method to resize a video is to use one of the AVAssetExportSession presets. Apple offers several size and quality presets for export sessions. In the line above that creates the exporter, replace AVAssetExportPresetHighestQuality with one of the other values. There are generic quality presets: AVAssetExportPresetLowQuality, AVAssetExportPresetMediumQuality and there are size presets including: AVAssetExportPreset1280x720 and AVAssetExportPreset640x480. When using one of the presets you do not need to use a composition unless you want to do further manipulation or unless you need a size or quality combination that is not provided by any preset. As with images, experiment with different settings until you get a balance between quality and size that works for you.

Going Further

Some other strategies for working with large files are to chunk data file or split up video files. However, to use this strategy, you will need to coordinate with someone to stitch them back together after they are uploaded or downloaded.

Using the AVAssetExportSession above, you could call it multiple times and pass in a time range perhaps using code like this

let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)
//some code to create the exporter
exporter?.timeRange = timeRange

Depending on the video and audio of the AVAsset the splits in the video may be noticable. Because of modern frame rates, the audio would probably be more likely to be noticed. Time ranges can be sub-second, so you would need to experiment. However, iOS would manage the file for you so that the entire AVAsset is never loaded into memory.

It is also possible to split Data objects using .subData(in:) and looping through the bytes of the object. Again, you would need to also write code to stitch them back together later. Additionally, you would likely need to bring the entire Data object into a memory buffer, which might not be desirable.

Another way to handle large video files is to use Apple’s HTTP Live Streaming tools. These create files that you can upload to any web server that is recognized by iOS and Mac devices. The tools will segment the video as well as create multiple versions so that your users with different bandwidth capabilities can see different quality streams. Most Android devices and web browsers will at least be able to see the video, even if they cannot dynamically switch between streams.

Wrapping Up

In this tutorial, you saw some different strategies for shrinking and compressing the large image and video files that an iPhone camera can produce. You also saw some strategies for transferring larger data payloads with a web server. As mentioned in the beginning, consider what sizes the images and videos will display and use that to determine what sizes of images to store. Also, consider storing multiple versions of images or using thumbnails when displaying a gallery or list of images.

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

PE.SDK and VE.SDK 10 for Android and 11 for iOS Release

Sascha — Wed, 04 May 2022 12:46:21 GMT

We are thrilled to announce the release of PhotoEditor SDK and VideoEditor SDK 10 for Android and 11 for iOS. This new major version is packed with features and improvements.

This release is adding:

Custom Watermarks
Background Removal for Photos
Background Removal for Stickers
Custom Sticker Libraries
Giphy Integration
And More Updates for Android and iOS

Custom Watermarks

With our new support for custom watermarks, you can add your company logo or any other image to photo and video exports. You can place your watermark in either corner or the center of your photo and video. Additionally, you can specify the size of the watermark and determine its distance from the outline if you choose to place it near a corner. Your users will not be able to remove nor modify the watermark.

For more details on Custom Watermarks, take a look at our documentation for Android or iOS.

Background Removal for Photos

You have unwaveringly requested this feature: we are excited to offer the automatic removal of backgrounds for static content! Background Removal is fully running on-device and currently specializes in images containing a person only. PE.SDK enables this feature exclusively when it detects a person in the photo.

The removed background will turn transparent if your selected output supports transparency, such as PNG. In other cases, such as JPG, your background removal will be tinted black.

This feature is available on Android and iOS 15.0 and higher only. Easily enable Background Removal for Photos with our Android guide or iOS guide.

Background Removal for Stickers

A more common use case for automatic background removal is with stickers, so we’ve also added this functionality. Select a sticker including a person and tap the Remove BG button. Simple!

This feature is available on Android and iOS 15.0 and higher only. Easily enable Background Removal for Stickers with our Android documentation or iOS documentation.

Custom Sticker Libraries

We wanted to make it easier for you to provide your users with more sticker tool content. So far, you were able to create multiple sticker categories with different stickers that could either be part of your app or stored on a server. Additionally, you could enable our personal stickers feature on iOS or Android), giving your users the option to select stickers from their camera roll.

We have now added more ways to provide your users with stickers.

Android

You can now create sticker categories that do not present our sticker selection interface, but will instead load any arbitrary Fragment. That gives you complete control over the sticker selection screen, and you can use it to load and display any content. When the user selects any of your stickers, all you have to do is pass it back to the SDK with a callback. Get started with your Custom Sticker Library with our Android documentation.

iOS

You can now create sticker categories that do not present our sticker selection interface, but will instead load any arbitrary view controller that conforms to the StickerCollection protocol. That gives you complete control over the sticker selection screen, and you can use it to load and display any content. When the user selects any of your stickers, simply pass it back to the SDK using a delegate method.
Additionally, we’ve created a new StickerProviderCategory that you can pass any object conforming to the StickerProvider protocol. This category provides most of the features required to interact with sticker content providers, such as search and pagination. You can use this category to very quickly integrate any sticker service.

Find more information on Custom Sticker Libraries in the iOS documentation.

GIPHY Integration

This online database and search engine for funny GIFs and beautiful animations is a staple in many social applications. Easily integrate GIPHY into your app:

Android

Using our new Custom Sticker Library, we’ve also created an integration for GIPHY on Android. All you have to do is head over to GIPHY, create your API key, pass it to your GiphySettings and create a GiphyStickerCategoryItem. Your users will then immediately have access to all the content GIPHY has to offer.

For more details, visit our documentation on adding GIPHY for Android.

iOS

Using the new StickerProviderCategory, we added GIPHY support on iOS. All you have to do is head over to GIPHY, create your API key, and pass this to the GIPHY sticker category. Your users will then immediately have access to all the content GIPHY has to offer.

For more details, visit our documentation here.

More Updates for Android

Additionally, we have:

Increased the minimum API level for PhotoEditor SDK from 16 (Android 4) to 21 (Android 5), which ensures support for 98% of all phones in use.
Added support for headless video exporting.
Increased video playback stability.
Fixed a number of issues with degraded audio quality in videos.
And added the ability to choose a theme at runtime.

More Updates for iOS

Additionally, we have:

Increased our deployment target from iOS 9 to iOS 13.
Migrated all usage of OpenGL to Metal.
Added a better result API.
Added support for SwiftUI.
Enhanced our video export to display a progress indicator and the option to cancel the export.

How to Merge Videos in iOS with Swift

Walter — Wed, 12 Jan 2022 12:12:55 GMT

Combining video clips or parts of video clips into a single video may appear complicated, but the things that add complexity also provide flexibility. In this tutorial, you will see how to combine a few short clips into a single video. You will also gain an understanding of building on this basic pattern to make more advanced creations. This tutorial was created and tested with Xcode 13 and Swift 5.

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with code that supports this tutorial is available on Github.

Making a Composition

Whenever you want to edit media or add effects, the first place to look is AVComposition. This class in AVFoundation has the purpose of arranging different assets and types of assets into a single asset for playback or processing. Asset types can include audio and video subtitles, metadata, and text. When working with AVComposition it is helpful to think about AVAsset and AVAssetTrack objects. Unfortunately, because these items are so similar (AVComposition is a subclass of AVAsset) it is easy to get confused. Try to remember that the AVTrack is a wrapper around the actual pixel, sound-wave, or other data and that AVAsset is a collection of AVTrack objects. When we want to combine and manipulate AVAsset objects, an AVComposition helps us do that. Changes made to any AVAsset by an AVComposition will not impact the original media file.

As an extra layer, Apple keeps the editing features of an AVComposition separate from the playback features by creating an AVMutableComposition object. You will work with an AVMutableComposition object in this tutorial.

Each track in our composition will have a start time and a duration. The composition will ensure that all of the data it holds displays at the right time and that the tracks stay in sync.

At the simplest, a video clip is a single AVTrack of audio data and a single AVTrack of video data.

So, to set up an empty AVComposition for combining clips, use code such as this:

    let movie = AVMutableComposition()
    let videoTrack = movie.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
    let audioTrack = movie.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)

The code above creates a new, empty AVMutableComposition and then adds two tracks. One track will be able to hold .video assets, and the other will hold .audio assets. AVFoundation supports many different kinds of audio and video formats, but the .video track cannot hold .audio formats. The kCMPersistendTrackID_Invalid signals that you want a new, unique track id generated.

AVFoundation can manipulate any of the ISO media formats. The Quicktime (.mov) and MPEG (.mp4) are the most common, however, .heif, .3gp and other formats are all supported.

Adding Video Clips

After creation, the AVMutableComposition has a start time of CMTime.zero and a duration of CMTime.zero. Use the insertTimeRange(_ timeRange: CMTimeRange, of track: AVAssetTrack, at startTime: CMTime) throws function on the audio and again on the video track to add data from the clip. In order to add data to the tracks, you load an AVAsset, determine what range of time will go into the new composition, and then call the insert function. Your code might look like this:

  let beachMovie = AVURLAsset(url: Bundle.main.url(forResource: "beach", withExtension: "mov")!) //1

  let beachAudioTrack = beachMovie.tracks(withMediaType: .audio).first! //2
  let beachVideoTrack = beachMovie.tracks(withMediaType: .video).first!
  let beachRange = CMTimeRangeMake(start: CMTime.zero, duration: beachMovie.duration) //3
  try videoTrack?.insertTimeRange(beachRange, of: beachVideoTrack, at: CMTime.zero) //4
  try audioTrack?.insertTimeRange(beachRange, of: beachAudioTrack, at: CMTime.zero)

Here is what this code does:

Load a video file from a local URL. In the example, the video is in the application bundle, but it could also be a URL that points to a file somewhere else. This will not work with a streaming URL. It needs to be a video file.
Extract the main videos and audio tracks from the video file. There may be multiple video and audio tracks in a file, but this code just grabs the first one.
Create a time range. In this example, the range is the same duration as the clip we loaded.
Insert the video track from the clip at the beginning of the video track of the composition and insert the audio track from the clip at the beginning of the audio track of the composition. A common mistake in this step is to use the .duration property of the composition to insert the audio and video at the end. However, as soon as media is inserted into any track, the duration of the entire composition will change. So the audio and video would be played one after the other and you would see a blank screen when the audio is playing, and have no audio when the video is playing.

Repeat the steps for each clip that you want to add to the final composition.

Using the Composition

After all of the clips have been combined into the composition, it can be played in an AVPlayer or exported using an AVAssetExportSession. The code for these is the exact same as for any other AVAsset.

  self.player = AVPlayer(playerItem: AVPlayerItem(asset: movie)) //1
  let playerLayer = AVPlayerLayer(player: player) //2
  playerLayer.frame = playerView.layer.bounds
  playerLayer.videoGravity = .resizeAspect
  playerView.layer.addSublayer(playerLayer) //3
  player?.play() //4

Here is what the code above is doing:

Create an AVPlayer using a movie object, which is the AVMutableComposition that was created earlier.
Create a playerLayer to display the video and set its aspect ratio
Add the playerLayer to playerView which is a regular UIView in a UIViewController
Play the video clip

Alternatively you may want to export the new composition as a new movie file. Again, because the composition is an AVAsset using a standard AVAssetExportSession will write the movie to disk.

  //create exporter
  let exporter = AVAssetExportSession(asset: movie,
                                 presetName: AVAssetExportPresetHighestQuality) //1
  //configure exporter
  exporter?.outputURL = outputMovieURL //2
  exporter?.outputFileType = .mov
  //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error { //3
        print("failed \(error.localizedDescription)")
      } else {
        print("movie has been exported to \(outputMovieURL)")
      }
    }
  })

Here is what this code is doing:

Create an export session using the movie composition you made earlier.
Set the save location to some file URL on your device.
Wait for the exporter to finish and display the result.

Don’t forget that exporting an AVAsset can take a long time and is asynchronous. You will want to display a spinner or a message to your user.

Going Further

You can now combine clips into a longer clip for playback and export. However, trimming the original clips, adding filters, and rearranging the order will require more work before your application is ready for the store. Using an SDK like the IMG.LY’s VideoEditor SDK can help you provide an app that has the features users will expect in addition to stitching clips together.

The VideoEditorSDK offers a Video Composition Controller which will allow the users to stitch clips together. Additionally, it will allow the user to add new clips and edit clips using the other editing tools. Because the Video Composition Controller is a subclass of UIViewController it can just be configured and dropped into your application. Instead of the code in our example, you pass the videos to the controller:

let videoClips = [
  Bundle.main.url(forResource: "stream", withExtension: "mov"),
  Bundle.main.url(forResource: "beach", withExtension: "mov"),
  Bundle.main.url(forResource: "mountain", withExtension: "mov"),
].compactMap { $0.map{ AVURLAsset(url: $0) } }
let video = Video(assets: videoClips)
let videoEditViewController = VideoEditViewController(videoAsset: video, configuration: Configuration())
present(videoEditViewController, animated: true, completion: nil)

The code above loads the same video clips as our demo application and then uses the VideoEditViewController to present them. From there the user can make further edits preview and export the composition.

Wrapping Up

In this example, you saw how to combine several video clips, each with one video and one audio track into a single movie file with one video and one audio track. To make more complex movie files consider building on this tutorial by:

Add a second audio track, AVFoundation will play sounds from both tracks at times specified at equal volume. Use AVAudioMix to adjust the volume of the tracks at different times (the sample project has an example of adding a second audio track).
Add more video tracks and use AVMutableCompositionLayerInstruction to determine which track is visible at any time in the final video and to make fancy transitions between tracks.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How to Add a Filter to a Video Stream in iOS

Walter — Mon, 08 Nov 2021 15:33:55 GMT

Using filters and effects with video streams requires different strategies than applying them to files. In this tutorial, you will see how to apply effects and filters to video streams in real-time. The code in this tutorial compiles with Xcode 13.

As with most AVFoundation code, the Xcode simulator is not the best platform for running the code. Test on an actual device. A project with code that supports this tutorial can be found on GitHub.

Streams or Files

For video files stored on the device, you can use an AVMutableVideoComposition to apply filters. When streaming video from a remote location, this strategy will not work. The AVMutuableVideoComposition classes are not designed to work in real-time with streams. Apple provides a way to extract the pixels from the stream at regular intervals. You can manipulate the pixels and then render them to the screen. An AVPlayerItemVideoOutput object gives access to the pixels that make up a video frame. Apple also supplies a CADisplayLink object to ensure you are extracting the right pixels at the right time. The display link is a specialized timer that will fire in sync with the redraw rate of the current display.

So, to filter a video stream, the strategy becomes:

Set up an AVPlayer as normal and assign it the URL of the stream
Attach an AVPlayerItemVideoOutput object to the stream
Attach a CADisplayLink object to the run loop
Extract the current pixel buffer from the AVPlayerItem every time the screen redraws
Convert the pixel buffer to a CoreImage object
Apply filters
Display the image

Setting up AVPlayer

You won’t use the AVPlayerLayer or AVPlayerViewController to display the video, but the AVPlayer plays a central role. The AVPlayer keeps the video in sync with the audio, handles decoding whatever format the stream uses, controls buffering, and more. The basic setup is the same as with any other project:

//create a player
let videoItem = AVPlayerItem(url: streamURL)
self.player = AVPlayer(playerItem: videoItem)

The URL can either point to a local resource or a stream.

Using AVPlayerItemVideoOutput

The AVPlayerItemVideoOutput allows you to query the AVPlayerItem for the pixel buffer (one screen worth of pixels) at any given time. You can specify different pixel formats and other options for the output. For this tutorial, you will you can create the object with a simple instantiation using the default formats.

let playerItemVideoOutput = AVPlayerItemVideoOutput()

Later you will add this video output to your player item. Then you can ask it to generate the pixel buffers as needed. In this tutorial, you will ask for the pixel buffer of the current frame. You could ask for the frame for any timestamp though.

Creating the Display Link

The CADisplayLink class is a specialized NSTimer that synchronizes itself to the screen refresh rate of the display. This ensures that the pixel buffer you extract will be from the correct time of your video stream. It will get rendered at the correct time in the screen refresh. Syncing a standard NSTimer to the screen refresh has always been problematic. With the new variable refresh rate screens that Apple makes, it becomes impossible. However, CADisplayLink adjusts its rate as the screen refresh rate changes. Apple explains how this works in the 2021 WWDC Session Optimize for Variable Refresh Rate Displays. When you create the display link, you give it the name of a function to call each time it executes. A sample for creating a link could look like this:

lazy var displayLink: CADisplayLink = CADisplayLink(target: self,
                                                  selector: #selector(displayLinkFired(link:)))

Elsewhere in the code, a function called displayLinkFired(link: CADisplayLink) extracts the pixel buffer from the AVPlayerItem.

Starting the Player

Apple notes that loading a video file takes a measurable amount of time. It can take even longer when the video file is streaming across a network. AVFoundation allows your program to continue to execute while the setup steps and initial buffering are occurring in the background. You can run into issues if you load a video into AVFoundation and then immediately attempt to work with it. AVFoundation may not show an error or status message, but it will not perform as expected either.

The most important step in this entire process is using an observer to wait until the AVPlayerItem has a .readyToPlay status before attaching the output and displaying link objects. In the example below statusOberserver, player, playerItemVideoOutput and displayLink are all declared at the class level. They need to persist the entire time the video is being used.

//create a player
let videoItem = AVPlayerItem(url: streamURL!)
self.player = AVPlayer(playerItem: videoItem) //1
//*important* add the display link and the output only after it is ready to play
self.statusObserver = videoItem.observe(\.status, //2
                       options: [.new, .old],
                 changeHandler: { playerItem, change in
    if playerItem.status == .readyToPlay { //3
      playerItem.add(self.playerItemVideoOutput)
      self.displayLink.add(to: .main, forMode: .common)
      self.player?.play() //4
    }
})

Here is what to notice about the code above

Creating the player will take a large amount of time, but the program execution will not stop and wait
NSKeyValueObservation is how swift defines key value observers the code will execute every time the videoItem.status property changes values
Checks to see if the .status property is .readyToPlay
After adding the display link and the video output objects start playing the video

Extracting the Pixel Buffer

Once the video begins to play, the display link function gets called for each screen refresh. Here is an example of how to extract the pixel buffer and render it to a UIImageView.

@objc func displayLinkFired(link: CADisplayLink) { //1
  let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
  if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) { //2
    if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
      let frameImage = CIImage(cvImageBuffer: buffer) //3
      //4
      let pixelate = CIFilter(name: "CIPixellate")!
      pixelate.setValue(frameImage, forKey: kCIInputImageKey)
      pixelate.setValue(self.filterSlider.value, forKey: kCIInputScaleKey)
      pixelate.setValue(CIVector(x: frameImage.extent.midX, y: frameImage.extent.midY), forKey: kCIInputCenterKey)
     let newFrame = pixelate.outputImage!.cropped(to: frameImage.extent)
     self.videoView.image = UIImage(ciImage: newFrame) //5
    }
  }
}

Here is what the code is doing:

Marks the func with @objc so that it works with the display link selector
Asks if there is a new pixel buffer to display. Depending on the refresh rate of the screen and the frame rate of the video, there will not always be a new buffer.
After extracting the pixel buffer, convert it to a CoreImage object
Apply any filters. This example applies the CIPixellate filter. It uses a slider to let the user change the intensity as the video plays
Convert the filtered image to a UIImage and assign it to the UIImageView

Now as the pixellate scale changes the image will be filtered but the video will continue to play smoothly.

Going Further

Using the strategy above you should be able to apply different filters to the video stream in real-time. Remember that you only have a few milliseconds though, then the video will begin to stutter. CoreImage filters are optimized to make use of the GPU and should be fast enough in most cases. For better performance, you can render the filtered image to an MTKView and use Metal. Apple has also begun to create Metal Performance Shaders which are even more efficient than CoreImage filters when used with a MetalKit view. As of now, there are many more CoreImage filters, so that may give you the most flexibility.
The strategy in this tutorial is adequate for filtering video streams. If you want to give your users more flexibility with filters and other video tweaks, an editor such as VideoEditorSDK can let you focus on your application’s core functions and let a team of professionals worry about the video. Using VideoEditorSDK your users can apply filters to video as well as text annotations and more.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions or suggestions.

You like what we do? Check our careers page. Even if you can’t find your role listed, we might be interested since you are already here!

How To Record Screen Video Using ReplayKit for iOS

Walter — Tue, 21 Sep 2021 15:05:08 GMT

In this article, you will see how to use Swift and ReplayKit to add screen recording to your application. You will also see how to use the new rolling clips feature, introduced in iOS 15. The code in this article uses Swift 5 and Xcode 13. Clone this repository for a sample project and example code.

ReplayKit Basics

Apple introduced the ReplayKit framework in 2015 with iOS 9. They have refined and expanded it since then, but it still has a small API. ReplayKit uses the same shared components as AirPlay and the device screen recorder (accessible from the device Control Center). For this reason, ReplayKit functions will not work when using AirPlay or playing video. Also, the screen recording functions do not work on the Xcode simulators. You should test using a physical device.

Your app makes all its calls to the RPScreenRecorder.shared() object. This is a singleton object for your app that ReplayKit provides. There is no need to store it in a variable or pass it around in your code.

Rolling Clip Recording

In iOS 15, Apple introduced rolling clip recording. ReplayKit will maintain an updated video buffer of the most recent fifteen seconds. Your app can request the video at any time. ReplayKit comes from GameCenter. Apple’s idea for this feature is for creating a short, sharable video of the moments before an exciting event in a game. Another use can be for a user to record the steps leading up to a defect during testing. When the user is unlikely to know in advance that they will want a screen recording, rolling clips may be a good solution.

To start rolling clip buffering, include a method such as this one

RPScreenRecorder.shared().startClipBuffering { err in
  if let err = err {
    //either the user denied permission or something like
    //AVPlayer or AirPlay is using the shared recorder resources.
    print("Error attempting to start buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering started.")
  }
}

Screen recording has privacy implications. ReplayKit presents the user with a modal, requesting to record before it starts. If the user denies the request or if the buffering is unable to start, ReplayKit passes an Error object to the closure. Unlike other privacy modals, you do not have an opportunity to add your own language to the message. The modal appears as soon as you execute the .startClipBuffering command, so be sure that the user is not surprised by it. Execute the command at a logical transition in your application. After the user grants permissions, the modal will not appear again unless eight minutes pass before the next call to .startClipBuffering.

While the clip buffering runs, at any time, you can save the buffer to a URL on the device. The clip you save can be any duration up to fifteen seconds. If the recorder is unable to write the clip to disk, it will send an error.

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().exportClip(to: clipURL, duration: TimeInterval(15)) { err in
  if let err = err {
    print("An error? \(err.localizedDescription )")
  } else {
    print("Clip saved to url \(clipURL)")
}

The code above stores a fifteen-second clip at a URL in the user’s temporary directory. The temporary directory gets cleared by the system, but not while the app is running. To ensure that the files are not deleted except by the user, store them in the user’s documents directory instead.
When your app is done collecting video, use .stopClipBuffering to allow the system to clean up resources and stop the recording.

RPScreenRecorder.shared().stopClipBuffering { err in
  if let err = err {
    print("Error attempting to stop buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering stopped.")
  }
}

Recording Screen and App Audio

Rolling clip buffering runs in the background. You can also allow the user to control when to start and stop screen recording. The API looks similar and is available since iOS 10.

RPScreenRecorder.shared().startRecording { err in
  guard err == nil else { print(err.debugDescription); return }
  //code to display recording indicator
}

As with the rolling clip buffering, the system will display a permissions dialog. If the user denies permission you will get an error and the recording will not start.

The same eight-minute rule applies as for rolling clip recording. But, since the user has just taken some action to start recording, they should expect to see this modal.
Apple provides a basic view controller for editing and sharing the video.

ReplayKit passes the RPPreviewViewController to the .stopRecording handler.

RPScreenRecorder.shared().stopRecording { preview, err in
  guard let preview = preview else { print("no preview window"); return }
  //update recording controls
  preview.modalPresentationStyle = .overFullScreen
  preview.previewControllerDelegate = self
  self.present(preview, animated: true)
}

The preview object above is the RPPreviewViewController editor. You can display it as a modal or as a popover depending on your needs. The Apple provided editor will allow the user to save to their camera roll or share the clip after editing.

Notice in the code above that there is a previewControllerDelegate. You will need to add some code to the delegate to cleanup and dismiss the editor when the user is done sharing or saving. An example is below.

func previewControllerDidFinish(_ previewController: RPPreviewViewController) {
  previewController.dismiss(animated: true) { [weak self] in
  //reset the UI and show the recording controls
  }
}

Using Many `UIWindow` Objects

The ReplayKit recording will only capture the main window of your application. When designing for screen recording, you can place any UI elements in a separate UIWindow object. Although applications typically only have one UIWindow, iOS allows many windows. A UIWindow requires a .rootViewController to provide content. Then use it like any other view in your application.
The UI for this application consists of three UIWindow objects.

Apple places the status bar in a separate window from your application. The recording controls are in a separate window from the main application and will not appear in the screen recording.

To add some recording controls or a recording indicator to your app. Create a new UIWindow and add it to the same windowScene as your view.

controlsWindow = UIWindow(frame: CGRect(x: 0, y: 0, width: view.frame.width, height: 45 + view.safeAreaInsets.top)) //1
controlsWindow?.windowScene = view.window?.windowScene //2
controlsWindow?.makeKeyAndVisible() //3
let recordingIndicator = UIButton.systemButton(with: UIImage(systemName: "record.circle")!, target: self, action: #selector(recordingToggled(_:))) //4
let vc = UIViewController()
controlsWindow?.rootViewController = vc //5
vc.view.addSubview(recordingIndicator) //6
recordingIndicator.center = CGPoint(x: vc.view.center.x, y: vc.view.center.y + 20)

The code above creates the red/blue recording button and banner at the top of the view. It relies on a controlsWindow variable created at the class level. Here is what it is doing:

Creates a new UIWindow that is the width of the view and tall enough for the button and the safe area (the area around the notch and status bar)
Adds the window to the same windowScene as the current view
Makes the window visible (UIWindow objects are hidden when created) and brings it to the front of the stack of windows
Creates a button and links it to a method in your class recordingToggled(_:)
Adds a view controller to the window so that there will be some content
Adds the button to the view controller

Because the recording indicator and button are in a separate window, they will not appear in any screen recording. Be mindful when showing modal views or alert views, as they may appear behind the extra window. A good strategy is to hide the extra window when showing these views. You can use a similar strategy to hide any other UI elements that should not appear in screen recordings.

Going Further

Though Apple provides a basic editor, you can provide your own editing tools. The examples above showed the rolling clip editor writes the video to the disk. The regular recorder can also write the video to disk. You can stop recording using this code:

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().stopRecording(withOutput: clipURL) { err in
  //check for an error and process the url
}

This method would replace the .stopRecording { preview, err in method shown above.
With the video saved, you can use an editor such as the VideoEditor SDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to capture a rolling screen recording as well as how to capture the screen manually. Further, you saw how using an SDK such as VideoEditor SDK allows you to annotate and enhance your clips before sharing.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

How to Create Image Filters for iOS

Walter — Tue, 24 Aug 2021 08:09:37 GMT

In this tutorial, you will learn how to write a custom CIFilter in Metal and Swift. You can use this filter in any CoreImage pipeline.

Apple offers over 200 image filters in the CoreImage framework, but sometimes you need a little extra to make your images perfect. Apple provides two ways to create customize image filters. You can chain CIFilters together in a CIFilter subclass or wrap a CIKernel in a CIFilter subclass. Either method creates a filter that CoreImage can execute on the GPU. That makes the filter fast. CoreImage filters can filter live video without impacting the frame rate. Before iOS 11 the only way to write a custom CIKernel was to pass a string containing the code to the GPU at runtime using the OpenGL Shading Language. This had two drawbacks: it was difficult to find errors in the string of commands and the kernel was not compiled until runtime. This tutorial will show you how to add a custom filter with a Metal-based CIKernel to any Swift project in Xcode. The code samples in this tutorial use Xcode 12.5 and Swift 5.

Clone this repository for a sample project and example code that supports this tutorial. The repo also contains some other custom filters, demonstrating other aspects of Metal filters.

Adding Metal Support to an Xcode Project

The first step is to add a Metal source code file to your project.

Find the “Metal File” template, select it and click Next. Name the file and save it. At compile time, Xcode combines all of the .metal files in your project into a single .metallib file. So, organize your Metal code into lots of files or just one file as you prefer. The last step before you start writing code is to add compiler flags for CIKernel objects.

In the Build Settings, find the other Metal Compiler Flags and add an entry of -fcikernel.

Then find the Other Metal Linker Flags and add an entry of -cikernel.

Important note: any project without .metal files hides the Metal Compiler and Linker sections from build Settings.

Setting Up a Metal File for CoreImage Kernels

Metal is a general-purpose technology for writing code that will execute on the GPU. The compiler flags tell Metal that you will be writing code to work with CoreImage. With a Metal source file in your project and the compiler flags set, you’re finally ready to write some code! Open the Metal file you created above. It should be empty except for

#include <metal_stdlib>
using namespace metal;

Below these lines, import the CoreImage headers by adding:

#include <CoreImage/CoreImage.h>

and add a stub for your filter code with:

extern "C" {
  namespace coreimage {
    // KERNEL GOES HERE
  }
}

Prefix your kernel code with extern "C" and the CoreImage namespace. You write kernel code in the Metal Shader Language, which is a variation of C. If you only know Swift, you should be able to rely on Xcode’s code-completion and symbol lookup to help you as you’re learning to write Metal code. In addition to the general language reference, Apple provides a Metal Shading Language for Core Image Kernels document.

The example below creates a CIColorKernel. This kernel is optimized for changing the color value of each pixel in an image. It will change each pixel to a grayscale version of itself. A filter applying his kernel will send the color value of each pixel of the image to the kernel.

float4 grayscaleFilterKernel(sample_t s) { //1
  float gray = (s.r + s.g + s.b) / 3;      //2
  return float4(gray, gray, gray, s.a);    //3
}

Here is how this kernel works:

The filter provides the color of the current pixel to the kernel as a sample_t type. The kernel will return the new color for that pixel as a float4 type. The sample_t type is equivalent to a float4 type and is only named differently because of historical convention. A float4 type contains four float values. In the case of color, they are the red, green, blue and alpha values for the pixel. Each float has a value between zero and 1.
Calculate a grayscale version of the pixel by averaging the three color channels.
Create a new float4 with the averaged value for the three color channels and the original alpha value for the fourth.

CoreImage also provides a CIWarpKernel class that is optimized for changing the position of each pixel and a CIBlendKernel for blending two images. For more complex tasks, CoreImage also provides a general CIKernel. Apple suggests that chaining together optimized kernels in a pipeline is usually more efficient than trying to write a larger, general kernel. A color-optimized kernel only has access to the color of the current pixel. A transform-optimized kernel can only affect the position of the current pixel. A general kernel can impact color and position as well as read other values from the source image.

Wrapping a `CIKernel` with a `CIFilter`

Now you’ll make a CIFilter subclass as the interface between the Metal code and your Swift code. Every CIFilter has an outputImage variable that returns a CIImage. Generally, input variables for a filter begin with input. The grayscale filter you are making has a single input and a single output. Create a new Swift file in your project and title it “GrayscaleFilter.swift”. Then be sure to import CoreImage by adding

import CoreImage

to the top of the file. Then create the filter as a subclass of CIFilter.

class GrayscaleFilter: CIFilter {

Next, create a kernel variable as either static or lazy, so that it only gets created one time, regardless of how often it’s called.

static var kernel: CIColorKernel = { () -> CIColorKernel in
    let url = Bundle.main.url(forResource: "default",
                            withExtension: "metallib")! //1
    let data = try! Data(contentsOf: url)
    return try! CIColorKernel(functionName: "grayscaleFilterKernel",
                      fromMetalLibraryData: data) //2
}()

Look in the bundle for the compiled metal code. The metal code will have a filename of “default”.
The .metalib may contain many kernels so use the one called “grayscaleFilterKernel”

Now add a variable to hold the input image.

var inputImage: CIImage?

Finally override the outputImage variable.

override var outputImage: CIImage? {
    guard let inputImage = inputImage else { return .none }
    return GrayscaleFilter.kernel.apply(extent: inputImage.extent,
                                   roiCallback: { (index, rect) -> CGRect in
                                                  return rect
                                  }, arguments: [inputImage])
}

The apply function of the kernel takes an extent argument. For a CIImage the extent property is the width and height of the image. The function has an roiCallback that allows you to specify what part of the image the kernel uses for each calculation. For a color filter, you generally just pass back the rect. Finally, pass in any arguments as an array. Notice that there is not any type checking here. It is up to you to pass in the correct types in the correct order.

Using the Filter

With the Metal code wrapped in a CIFilter you can now apply the filter images the same as using one of the built-in filters. Create an instance of the filter, then assign the inputImage variable a CIImage and display the outputImage. Your code might look something like this:

let filter = GrayscaleFilter()
filter.inputImage = image

displayView.image = UIImage(ciImage: (filter.outputImage ?? image) ?? CIImage())

and it can render a grayscale version of an image, like in this example.

Going Further

You can write custom filters and chain them to the built-in filters to invent new effects. Many of the math formulas for effects are available on the Internet. Most are easy to translate into the Metal Shader Language to make them perfect for your application. Writing an entire image or video editing application to showcase your filters is a much larger task. Using an SDK like the IMG.LY PhotoEditorSDK or VideoEditorSDK can help you provide an app that has the features that users will expect in addition to your cool filters.

To add your filter to the PhotoEditorSDK you first wrap your CIFilter with an Effect. For a basic filter, you only need to override the newEffectFilter variable.

import PhotoEditorSDK

class GrayscaleEffect: Effect {
  override var newEffectFilter: CIFilter? {
      GrayscaleFilter()
  }
}

Then add the filter to the Effect.all array when configuring the SDK, using something like

Effect.all = [NoEffect(), GrayscaleEffect(identifier: "grayscaleFilter", displayName: "Best Gray")]

Now your filter appears the same as the other 60+ filters that ship with the SDK.

It applies to live camera previews as well as still images.

Wrapping Up

In this tutorial, you learned how to configure Xcode for custom Metal code. You also learned how to create a color filter kernel in Metal and wrap it in a CIFilter. By combining your filters with Apple’s filters in pipelines, you can create unique effects for your next app. Further, using an SDK such as PhotoEditorSDK or VideoEditorSDK allows you to showcase your filters in a full-featured image or video editing application. The PhotoEditor SDK for iOS documentation will give you deeper look into all other image adjustments including, ofcourse, the image filtering we discussed above.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

Case Study: Vapiano & PhotoEditor SDK

Felix — Thu, 24 Aug 2017 00:00:00 GMT

The German casual dining restaurant chain Vapiano facilitates and streamlines their customer’s visits with a feature-rich app that lets its users: check in at the restaurant, order beverages and desserts from their seat, pay after their visit, collect loyalty points, browse the menu, save favorite dishes, locate the nearest subsidiary and even create images for social media postings.

With 180 restaurants in 31 countries, Vapiano is one of the biggest fast-casual dining chains around. Especially at lunchtime and early evening hours each restaurant wines and dines plenty of hungry customers, with Vapiano’s current approach (ordering at the counter, receiving the meal, finding a table and after that checking out on a regular register) that sometimes resulted in long lines and consequently frustration among their guests. With the Vapiano app, that is a thing of the past as the visit can almost solely be handled by the app from check-in to check-out. As Vapiano’s target audience is very photography and social media affine, their app holds another gem for their users to play around with. When taking a picture of themselves and their food in a Vapiano restaurant, the users can directly add Vapiano stickers to their pictures and then post them online. Said feature was realized by MobiLab Solutions utilizing the PhotoEditor SDK. We sat down with Max Afflerbach and Robert Rabe of MobiLab Solutions GmbH to talk about their experiences with PhotoEditor SDK.

“Vapiano’s vision regarding the photo feature is to spread their brand, enhance brand recognition and broaden the brand’s visibility,” Max Afflerbach, Chief Customer Officer at MobiLab Solutions explains. “Basically, they wanted to provide their guests with a tool that facilitates what they’re already doing when dining at Vapiano, that is taking pictures of their food and then uploading them to Snapchat, Instagram, Facebook, you name it. And so, we came up with a built-in branding feature that uses custom stickers.”

“…on StackOverflow or Quora, PhotoEditor SDK is listed and praised as the only real alternative to Aviary”

As they were on a schedule and didn’t have the time to develop the feature themselves the folks at MobiLab started searching for a third-party solution. “We’ve been looking for an SDK that does exactly what we imagined the feature to be capable of, namely taking a picture, adding assets and then exporting it,” says Robert Rabe, iOS developer at MobiLab Solutions. “We first considered Adobe Creative SDK because it is the best-known SDK in that field. However, it lacked a crucial feature for us that PhotoEditor SDK luckily has and that is the possibility to upload custom stickers,” says Max Afflerbach. “Also,” Rabe adds, “we weren’t 100% sure whether this whole thing would work without the user having to log into the Adobe Cloud.” “When you’re searching for other image processing SDKs on StackOverflow or Quora, PhotoEditor SDK is listed and praised as the only real alternative to Aviary. And besides, there aren’t exactly a lot of SDKs out there that cover both iOS and Android. So, that wasn’t a tough decision at all,” Afflerbach explains.

From the initial decision to work with PhotoEditor SDK until the feature was finished and ready for rollout, only a short period passed as Rabe explains: “Working with the SDK is really straightforward. The implementation is fast, and it’s all well documented. The SDK is a powerful tool and holds a lot of features with a great many possibilities. Setting up the feature exactly as we wanted it to be didn’t take longer than three working days tops. And if we had any issues or questions, we had no problems whatsoever getting in touch with the folks at PhotoEditor SDK which is really nice.”

“I really like the product, it’s working very well, and especially the communication with the PhotoEditor SDK team was highly pleasant,” Robert Rabe concludes. “Working together with the team of PhotoEditor SDK was super easygoing, the few requests and inquiries we had were answered quickly, and the technical licensing was a cinch as well. We are delighted with the product, the integration was very simple, and our developers are happy, I guess you couldn’t ask for more from an agency’s perspective,” Max Afflerbach adds.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

iOS App Development – IMG.LY Blog

CE.SDK v1.73 Release Notes

Own Your Mobile Editor — Not Just the Config

Migration

Starter Kit Repositories

Export Vector Designs as SVG

Changelog

CE.SDK v1.46 Release Notes

Shape Your iOS Interface Around Your Workflow

Upcoming

CE.SDK v1.33 Release Notes

Integrate a Design Editor on iOS and Android

Key Features

Use Plugins to Customize UI and Add Quick Actions

Add Voiceovers to Videos on iOS

Integrate a Photo Editor for iOS and Android

Key Features

Improvements

CE.SDK v1.23 Release Notes

Record and Edit Short-Form Video Content on iOS

Record with a New Camera

Edit on a Timeline

Quickly Add Editors to Mobile Apps with UI Packages

Integrate a User-Friendly Editor

Integrate A Smooth Video Editor into your App

CE.SDK v1.21 Release Notes

Create Videos With A New Timeline for Web

Keep Designs in Focus on iOS

Ensure Color Purity in Print

Very soon: Integrate Short-Form Video Creation

CE.SDK v1.20 Release Notes

Create Unique Element Combinations

Refine Images with iOS Appearance Settings

Optimize High-Quality Asset Imports

Keep Designs in Focus on Android

Exclusive Access: Integrate Short-Form Video Creation

Time-Based Sprites for VE.SDK on iOS and Android

Time-Based Sprites

Why is Video Important?

How to Remove Backgrounds Using Core ML

Using Vision to Segment People

Choosing a Model

Core ML and Xcode

The DeepLabV3 Model

Segmentation with DeepLab

Going Further

Wrapping Up

Working with Large Video and Image Files on iOS with Swift

Writing an Image to Disk

Compression Options for JPEG

Compressing Data with zlib

Transferring Data With a Server

Uploading a Large File

Configuring Background Transfers

Resizing an image

Resizing a Video

Going Further

Wrapping Up

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

PE.SDK and VE.SDK 10 for Android and 11 for iOS Release

Custom Watermarks

Background Removal for Photos

Background Removal for Stickers

Custom Sticker Libraries

Android

iOS

GIPHY Integration

Android

iOS

More Updates for Android

More Updates for iOS

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

How to Merge Videos in iOS with Swift

Making a Composition

Adding Video Clips

Using the Composition

Going Further

Wrapping Up

How to Add a Filter to a Video Stream in iOS

Streams or Files

Compressing Data with `zlib`

Using Many `UIWindow` Objects