Swift – IMG.LY Blog

Building Custom Variable SF Symbols in Figma for our CreativeEditor SDK for iOS

Dustin — Tue, 13 May 2025 07:00:06 GMT

If you’ve ever wrestled with vector paths in Figma just to get a custom variable SF Symbol out the door, you know the frustration: one missed node here, one stray smoothing there, and suddenly your interpolation breaks. Plus, you would design your icons within an SVG file you exported beforehand from the SF Symbols App.

When Apple introduced SF Symbols in 2019, they changed the game for iOS design. Suddenly, we had access to thousands of scalable, weight-variable icons that integrate seamlessly with system type. Our CreativeEditor SDK (CE.SDK) is a customizable design and video editor you can integrate into your product. Yet, as it grew, we needed icons as unique as its use cases—icons that simply didn’t exist in Apple’s library.

In this article, I’ll share our journey from relying on two community Figma plugins—SF Symbols Organizer and Vector Path Editor—to building our own tool, SF Symbol Generator. Along the way, I’ll show you how we wrestled with aspect ratios, node counts, and path order, and how we eventually streamlined the entire process into a few simple clicks.

What Are Variable SF Symbols?

At their core, variable SF Symbols behave much like variable fonts: they allow you to interpolate smoothly between multiple “master” designs, rather than shipping separate static assets for each weight. To create one, you supply three distinct versions of your icon—typically Ultra Light, Regular, and Black—each sharing the exact same path structure and node count. When the SF Symbols app or iOS runtime renders your symbol, it calculates intermediate weights on the fly by linearly interpolating between those three masters. This approach not only shrinks your asset footprint but also gives you rich typographic control, letting you fine-tune how heavy or light an icon appears in any context.

The Early Struggle

From the moment we discovered the Figma plugin SF Symbols Organizer, it became an indispensable part of our workflow. Suddenly, what had been a tedious, manual export process turned into a single click: select your three masters, batch-export them as SF Symbols-ready SVGs, and you were halfway home. The plugin’s enforcement of a 1:1 aspect-ratio rule was actually a blessing when you’re just getting started, ensuring that every icon fit out of the gate.

However, as our icon library expanded beyond a handful of simple glyphs, we began to feel the boundaries of that magic square. Some of our more ambitious designs needed wider canvases or asymmetrical compositions, and shoehorning them into perfect squares led to awkward scaling and extra cropping steps. Meanwhile, although SF Symbols Organizer reliably generated valid SVGs, it didn’t warn us about deeper issues—like mismatched node counts or flipped path directions—until we imported into the SF Symbols app and discovered the interpolation errors firsthand.

That’s where Vector Path Editor pitched in. It couldn’t automate our entire workflow, but it did let us quickly correct path order and direction when Figma’s flattening shuffled our nodes. Even with that help, though, we still found ourselves jumping between tools to catch errors after the fact—proof that, while both plugins were fantastic at what they did, we needed something more proactive.

Yet, as powerful as these two plugins were together, their reactive nature meant we still spent long sessions hopping between Figma and Apple’s toolchain, correcting mistakes after they happened instead of catching them up front. It was a great start—and without these plugins, we wouldn’t have made it this far—but we knew there had to be a way to bake most of the missing quality checks into our design process itself.

Why We Wanted “SF Symbol Generator”

After enough trial and error, we realized we needed more than just export and pray. We needed flexibility and foresight.

First, we didn’t want our icons locked into a square. Some of our designs needed wider canvases. Second, we craved live feedback. We wanted to know if our three icons had matching node counts, aligned first nodes, and consistent path winding before exporting. Lastly, we longed for a single, unified workflow that could handle scanning, validating, and exporting without leaving Figma.

It quickly became clear that the only way to achieve this was to build our own plugin. And so SF Symbol Generator was born.

Inside SF Symbol Generator

SF Symbol Generator transforms the way we build custom symbols. Rather than flattening paths prematurely, it encourages us to keep boolean groups intact, flattening only at export time. When you run the plugin, it scans all three masters in situ, then displays a detailed readout of node counts, first-node anchors, and path directions. Any discrepancies are highlighted in red, so you can go back to your designs and fix them.

*Pro Tip: Sometimes Figma’s automatic flattening doesn’t handle rounded corners cleanly and can introduce extra nodes. If the plugin’s scan flags a weight with mismatched node counts or errant anchors, you can manually flatten that master ahead of time, clean up extra nodes using Figma’s vector editing tools, and then re-run the scan to ensure perfect parity.*

Once everything checks out, a single click exports a ready-to-use variable SVG bundle—no detours into external editors required.

Our Step-By-Step Workflow

Our process now is refreshingly simple. We start in Figma by creating a 32-pixel-tall frame that can have any width you need.

Inside that frame, we draw our Regular-weight icon using a 2.45-pixel stroke and a 3-pixel corner radius. We then duplicate the frame twice and adjust the strokes for Ultra Light (0.8 px, 2.5 px radius) and Black (4.95 px, 3.5 px radius). iOS commonly incorporates softer corners in their icon design, and you can achieve them with Figma’s corner smoothing. Though, it’s important to keep Figma’s corner smoothing at 0%, because even a sliver of smoothing can throw off interpolation. We also center-align our strokes wherever possible so that the shapes grow uniformly when the weight changes.

	Stroke size	Border radius
Ultra Thin	0.8	2.5
Regular	2.45	3
Black	4.95–3.5	3.5

With your three master frames selected, launch SF Symbol Generator. In the plugin’s Variable panel, assign each frame to its corresponding weight—Ultra Light, Regular, and Black. If your frames are named correctly (for example, “Icon Ultra Light,” “Icon Regular,” “Icon Black”), the plugin will detect and map them automatically. As soon as each weight is assigned, the previews immediately show node counts, first-node positions, and path directions. Any mismatches appear in red, at which point you might need to go back and manually flatten that master, clean up extra nodes (for example, Figma sometimes adds unwanted anchors on rounded corners), and re-scan them again. When everything looks great, just hit Export to generate your ready-to-use variable SVG bundle.

Importing into the SF Symbols App

Once your variable SVG bundle is exported, the final import into Apple’s SF Symbols app is straightforward. Open the SF Symbols application (available on Apple’s developer site), switch to the Custom Symbols tab, and drag your SVG file into the panel. Your new icons will appear alongside Apple’s built-in set, ready to be organized into collections or shared with teammates. When you return to Xcode, your custom symbols show up just like any system glyph, complete with full support for dynamic weights.

Optional Extra Step: Preparing More Complex Symbols

If your icon consists of multiple layers or requires special styling per layer, take this extra step to ensure it renders perfectly in all weights. After dragging your SVG into the SF Symbols app:

Select your newly imported symbol and click the Rendering Inspector (the segmented button with the brush icon) on the right side.
You’ll see each layer exposed as a separate sub-path. You can create more layers and even group them to change them individually as well.
Tweak opacity, colors, or layer order at different render modes to achieve the exact look you want.

Once you’re satisfied, select and export the symbol again from the SF Symbols app (File > Export Symbol), and it’s truly ready for production.

Lessons Learned

Building SF Symbol Generator taught us several key lessons.
First, node parity is king: the number and order of path segments must match across masters. Early validation saves hours, since catching mismatches before export means fewer cycles in Apple’s SF Symbols app.
Second, non-destructive workflows pay dividends: boolean groups let you iterate quickly without losing stroke data.
And finally, automation is your friend: what once took dozens of manual steps now happens in a single click.

Conclusion

Variable SF Symbols unlock a level of typographic and iconographic dynamism that feels truly native to iOS. While community tools like SF Symbols Organizer and Vector Path Editor can get you pretty far already, building our own SF Symbol Generator transformed our workflow from reactive to proactive. Now, we design freely with any aspect ratio, validate instantly, and export with confidence—no more praying, just seamless symbol-making.

For our CE.SDK for iOS, we’ll be gradually rolling out updated Custom Symbols created with SF Symbol Generator, ensuring every icon feels right at home on Apple devices. Stay tuned for those releases!

If you’re ready to streamline your own custom SF Symbol workflow, check out SF Symbol Generator on the Figma Community and take it for a spin. Happy symbol-making!

3,000+ creative professionals gain early access to new features and updates—don’t miss out, and subscribe to our newsletter.

Working with Large Video and Image Files on iOS with Swift

Walter — Tue, 10 May 2022 13:27:44 GMT

Video files and image files are among the largest files. The AVFoundation and CoreImage libraries that Apple supplies widely store data on the disk. They try to bring into memory only as much of a file as is needed to complete a task. As you are working on an application that uses files, there are times you will want to compress or resize them. Usually, this is to upload to a server or because you have noticed performance issues, such as stuttering when scrolling or long render times.

This article will cover strategies for compressing and resizing images, videos, and other files. We will also be looking at using URLSession for file uploads and downloads.

As iPhone cameras can take higher-quality pictures, the file sizes have grown quite large. For the last few years, Apple has been using a 12MP camera that takes pictures at 3024 x 4032 resolution. Apple uses a special HEIC compression to make each image about 1.7MB. The same image will generate a 15MB file when saved as a .png and a 3MB file when saved as a .jpeg. Though this will be a high-quality image, services like Instagram or Facebook will often only display images at less than half of that size. The extra pixel data will be compressed away. Perhaps the first and most important part of working with large image files in your app is planning around what size or quality you need. Social Media services similarly resize video files. For instance, TikTok displays images and video using 1080 x 1920, but any iPhone after X will capture video at a larger size.

Writing an Image to Disk

When working with a UIImage or CoreImage object, normally you can save it as an NSData object. These files are lossless (the image quality will not degrade), but they are only useful inside of your application. The standards on the Internet are jpeg and png files.

Fortunately, Apple provides methods to convert UIImage to .jpegData or .pngData. There are also ways to convert CIImage and CGImage, but they are slightly more complicated to configure.

let imageToSave = UIImage(ciImage: <someImage>)! \\1
let jpegData = imageToSave.jpegData(compressionQuality: 1.0) \\2
let file = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first \\3
do {
  try jpegData?.write(to: file!.appendingPathComponent("sleeping_dog.jpg")) //4
} catch(let err) {
  print(err.localizedDescription) //5
}

Here is what’s going on in the code above:

First we need to convert our image to a UIImage. The UIImage class has several different initializers. Use UIImage(named: "some image name") to read an image from the app bundle. Use UIImage(contentsOfFile: "some filepath") to read from the disk. Use UIImage(cgImage: "some cgImage") and UIImage(ciImage: "some ciImage") when you have images already in memory. If you are storing images in a CoreData store or somewhere as Data you can use UIImage(data: "some data").
However you get a UIImage the next task is to convert it to either a .jpg or .png file. Generally, .jpg is a better option when working with photographs and .png produces higher quality for line art. The example above creates a .jpg at the highest quality possible.
Ask the FileManager for a URL to the user’s documents directory. That is generally a safe place to write files. Because of the way the function is formed, it returns an array of URL items. In this case, there is only one, but you still need to remember to specify the .first item.
Writing Data object to disk can throw errors, so it is good practice to place it in a try…catch block of code. The URL to the user’s documents directory needs to have the actual filename appended.
Any errors that the system throws can be handled here.

In order to create a .png using the imageToSave above, the line would be

let pngData = imageToSave.pngData()

The .png filetype does not have any compression settings. The filetype is “lossless”, so it is always compressed as much as possible without losing any data.

Compression Options for JPEG

When generating jpeg data above, we supplied a quality value. The value is a Float between 0.0 and 1.0. The .jpg filetype uses a “lossy” compression algorithm that is specifically designed for photographs.

In the image above saving the jpeg with a quality of 1.0 produced the dog on the right and the file is 1.3MB. Saving the jpeg with a quality of 0.0 produced the image on the left and the middle image is produced saving with a quality of 0.6. The file on the left is only about 6% of the size of the original.

Considering what the images are being used for can help you decide on an optimal compression setting for a particular app.

Compressing Data with `zlib`

Though png and jpeg files are already compressed sometimes you will want to compress other types of files before uploading them (compressing a png, mov or jpeg will often not affect the file size or will make the file larger). A web server will often be able to accept a POST that compresses its body data using gzip. Additionally, some web servers will require that binary data is transformed into a base64EncodedString before uploading. Base64 encoding will make data size grow by about 33%. Using gzip will help mitigate that somewhat. Apple’s implementation of gzip is called .zlib when compressing data. Swift provides some other compression algorithms such as .lzfse which will result in smaller files, but they are not widely adopted.

For example, to encode something as base64 and then compress it with with gzip so you can upload it, we can use code like below

let jsonToUpload = "{"books":[{"title":"The Three Musketeers","author":"Dumas"}]}"
let jsonAsData = jsonToUpload.data(using: .utf8)
let encodedString = jsonAsData?.base64EncodedString()
let compressedString = try? NSData(data: (encodedString?.data(using: .utf8))!).compressed(using: .zlib) as Data

The code above creates a string of data in JSON format. It then encodes the String as a Data object using .utf8 encoding. Using .utf8 is a standard. If the web server you are uploading to requires a different encoding, you can change it. In the final step, the encoded string is compressed using gzip. Notice that the .compressed(using:) method is on NSData not on Data. NSData is an older library, so you convert the compressed NSData object back to Data by casting using as Data.

The above example was just to show syntax. The compressedString will be slightly larger than the encodedString. Based on the kind of data your app works with and the capabilities of the web server, you will need to experiment with how to encode and compress to get the best results.

Transferring Data With a Server

Depending on the original format of the data, your web server may want the .httpBody compressed. You can do that with some code like this:

let someJSONData = Data() //1
var request = URLRequest(url: URL(string: "https://example.com/uploads/")!) //2
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
if let compressedJSONData = try? NSData(data: someJSONData).compressed(using: .zlib) as Data { //3
  request.httpBody = compressedJSONData
}
let task = URLSession.shared.dataTask(with: request) {data, response, error in
  //check for errors and response data when the task is done
  //response will contain the status and other header messages
  //data will contain any payload the server returns
}
  task.resume() //4

Here is what the code above is doing:

Get the payload into a Data object.
Create a URLRequest as a var so that you can configure it. If you create it as a let it will be a GET request with default header fields.
Use gzip to compress the data and if successful, assign it to the .httpBody of the request.
Create a data task for the upload. Upon completion, the data, response and error variables will have any response from the server.
Actually start the task.

Uploading a Large File

An issue with the code above is that all of the initial data will be in memory. So for a large image or video upload, this may not work. However, Swift does provide a different method that will upload a file directly from the disk, reading into memory only what is needed at the time.

Code to implement that would follow this form.

var request = URLRequest(url: URL(string: "https://example.com/uploads")!)
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first
let fileToUpload = documentsDirectory?.appendingPathComponent("the-big-giant-file.mp4")
let uploadTask = URLSession.shared.uploadTask(with: request, fromFile: fileToUpload) {data, response, error in
//get some information from the server when the file has been uploaded
}
uploadTask.resume()

This looks very similar to the previous example but notice that you do not set an .httpBody. There will be header values to set though. Most web servers will want to know how large the file is, authentication credentials, and whether the data is multi-part, or raw or other metadata. These will all be set using .setValue(forHTTPHeaderField:) in the request object.

For larger files, it is often better to use a URLSessionDataDelegate instead of the single closure with data, response, error. When using the delegate, you can write code to monitor the progress of the file transfer as well as respond to authentication challenges and make decisions about caching. You can also implement URLSessionDelegate and URLSessionTaskDelegate methods. All of the delegate methods are options, so you only need to implement ones that are important to your application.

Configuring Background Transfers

Another benefit of using the uploadTask (and its related downloadTask) instead of the dataTask is that you can configure the transfer to continue for a time even if the app goes to the background. In that case, instead of using the URLSession.shared object, you will want to configure a session that has some special properties to allow it to work in the background.

let configuration = URLSessionConfiguration.background(withIdentifier: "some.unique.identifier")
let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
let uploadTask = session.uploadTask(with: request, fromFile: fileToUpload)

Completely configuring and supporting the downloads and backgrounding is covered in this article by Apple. It will require implementing some delegate methods for the session as well as updating some of the methods in the main application delegate. The most important is the handleEventsForBackgroundURLSession which is how the system will wake up your app with information about the upload or download.

Additionally, download tasks can be paused and restarted. The cancel(byProducingResumeData:) method on URLSessionDownloadTask will cancel a download and optionally provide some resume data you can use to resume the download at a later time. You can call this method on the task when the user taps a button or when the app goes to the background or similar. The web server must support byte-range requests and ETag or Last-Modified headers. The documentation for pausing and restarting explains how to pause the download and how to resume using downloadTaks(withResumeData:). There is not a similar structure for pausing and resuming an upload.

Resizing an image

Perhaps the simplest way to make a file smaller is to reduce its dimensions. Using CoreImage you can use the Lanczos resampling algorithm to resize an image while keeping high quality.

let url = Bundle.main.url(forResource: "sleeping-dog", withExtension: "jpeg")
let image = CIImage(contentsOf: url!) //1
let filter = CIFilter(name: "CILanczosScaleTransform") //2
filter?.setValue(image, forKey: kCIInputImageKey)
filter?.setValue(0.5, forKey: kCIInputScaleKey) //3
let result = filter?.outputImage
let converter = UIImage(ciImage: result!) //4

Here is what this code is doing:

Create a CIImage from a URL.
Create a CILanczosScaleTransform filter.
Configure the filter to reduce the image dimensions by 50%.
Convert the scaled image to a UIImage.

Recently, Apple has provided a UIGraphicsImageRenderer that will work directly on UIImage objects. The code below will resize a UIImage to 50%. Notice that the UIGraphicsImageRenderer can also use an explicit value for size (unlike the CILanczosScaleTransform)

let imageToSave = UIImage(named: "sleeping-dog.jpeg")! //1
let newSize = imageToSave.size.applying(CGAffineTransform(scaleX: 0.5, y: 0.5)) //2
let renderer = UIGraphicsImageRenderer(size: newSize) //3
let scaledImage = renderer.image { (context) in
  let rect = CGRect(origin: CGPoint.zero, size: newSize)
  imageToSave.draw(in: rect) //4
}

Here is what this code is doing:

Load the image as a UIImage.
Scale the size down by half.
Create a UIGraphicsImageRenderer that will render in this new size.
Actually draw the image into the renderer at the new size.

In addition to rendering the image back as a UIImage the renderer can create jpegData and pngData which would save a few steps if you are resizing and converting. The full documentation for UIGraphicsImageRenderer explains the details.

Resizing a Video

Though the UIGraphicsImageRenderer is faster, using the older CoreImage resizing has a benefit. CoreImage filters can be applied to video files as a composition. Large video files can be scaled down. The code below will create a composition that will scale a video asset by 50%. Remember that you will need to import AVFoundation to use these tools.

let newAsset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
var newSize = <some size that you've calculated> //2
let resizeComposition = AVMutableVideoComposition(asset: newAsset, applyingCIFiltersWithHandler: { request in
  let filter = CIFilter(name: "CILanczosScaleTransform") //3
  filter?.setValue(request.sourceImage, forKey: kCIInputImageKey)
  filter?.setValue(<some scale factor>, forKey: kCIInputScaleKey) //4
  let resultImage = filter?.outputImage
  request.finish(with: resultImage, context: nil)
})
resizeComposition.renderSize = newSize //5

Here’s what the code above is doing to create a new composition:

Create an AVAsset from a URL.
Create a variable newSize to hold the final size.
In the composition, configure the CIFilter that will be applied to each frame.
Calculate the scale factor based on the newSize variable and the actual size of the request.sourceImage.extent.size.
Set the renderSize property of the composition to the new size.

If you don’t set the renderSize then there will be a black letterbox around the video.

With the resizing composition, you can now export the AVAsset as a .mov or .m4v file using an AVExportSession.

let asset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("exported.mov") //2
//create exporter
let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)
//configure exporter
exporter?.videoComposition = <the composition you created above> //3
exporter?.outputURL = outputMovieURL!
exporter?.outputFileType = .mov
//export!
exporter?.exportAsynchronously(completionHandler: { [weak exporter] in //4
  DispatchQueue.main.async {
    if let error = exporter?.error {
      print("failed \(error.localizedDescription)")
    } else {
      print("file saved at \(outputMovieURL)") //5
    }
  }
})

Here’s what the code above is doing:

Load an AVAsset from a URL.
Create a URL to save the resized movie.
Apply the resizing composition to an AVAssetExportSession.
Asynchronously export the asset to disk.
Print the destination URL of the new movie when it is saved.

Another method to resize a video is to use one of the AVAssetExportSession presets. Apple offers several size and quality presets for export sessions. In the line above that creates the exporter, replace AVAssetExportPresetHighestQuality with one of the other values. There are generic quality presets: AVAssetExportPresetLowQuality, AVAssetExportPresetMediumQuality and there are size presets including: AVAssetExportPreset1280x720 and AVAssetExportPreset640x480. When using one of the presets you do not need to use a composition unless you want to do further manipulation or unless you need a size or quality combination that is not provided by any preset. As with images, experiment with different settings until you get a balance between quality and size that works for you.

Going Further

Some other strategies for working with large files are to chunk data file or split up video files. However, to use this strategy, you will need to coordinate with someone to stitch them back together after they are uploaded or downloaded.

Using the AVAssetExportSession above, you could call it multiple times and pass in a time range perhaps using code like this

let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)
//some code to create the exporter
exporter?.timeRange = timeRange

Depending on the video and audio of the AVAsset the splits in the video may be noticable. Because of modern frame rates, the audio would probably be more likely to be noticed. Time ranges can be sub-second, so you would need to experiment. However, iOS would manage the file for you so that the entire AVAsset is never loaded into memory.

It is also possible to split Data objects using .subData(in:) and looping through the bytes of the object. Again, you would need to also write code to stitch them back together later. Additionally, you would likely need to bring the entire Data object into a memory buffer, which might not be desirable.

Another way to handle large video files is to use Apple’s HTTP Live Streaming tools. These create files that you can upload to any web server that is recognized by iOS and Mac devices. The tools will segment the video as well as create multiple versions so that your users with different bandwidth capabilities can see different quality streams. Most Android devices and web browsers will at least be able to see the video, even if they cannot dynamically switch between streams.

Wrapping Up

In this tutorial, you saw some different strategies for shrinking and compressing the large image and video files that an iPhone camera can produce. You also saw some strategies for transferring larger data payloads with a web server. As mentioned in the beginning, consider what sizes the images and videos will display and use that to determine what sizes of images to store. Also, consider storing multiple versions of images or using thumbnails when displaying a gallery or list of images.

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How to Make an Animated GIF Using Swift

Walter — Tue, 10 May 2022 12:07:58 GMT

In this article, you will see how to use Swift to create an animated GIF file. You will also see how to extract individual frames from a video file to convert the whole video or just clips from a video into an animated GIF. The code in this article uses Swift 5 and Xcode 13. The tutorial uses AVFoundation and CoreGraphics. Clone this repository for a sample project and example code.

Animated GIF Basics

The GIF is one of the older file types on the Internet. But, every platform supports it (not every platform supports every video format). Unlike some other image formats, GIF files can contain one image or a series of images. When a GIF file has a series of images and some metadata to describe how long to show each one, it can substitute for animation and video. Because of this, a platform that doesn’t necessarily support video files can display moving images and animations.

GIF files are easy to make. However, animated GIF files are not perfect: they are large and inefficient compared to modern video files. They have a limited color palette, and none of the images in the file have compression. For example, the video file we use in the sample project is about 5MB in mp4 format but over 100MB after it becomes an animated GIF. Finally, GIF files don’t have a soundtrack. Still, they are popular and do certain tasks well, so let’s make some using AVFoundation and Swift.

The Basic Strategy

In order to create animated GIF files you need a series of images and some metadata to describe how long each image will show. This tutorial will start with how to put these images into a GIF. Then you will see some different strategies for gathering the images. A GIF file contains bitmap images, not vector graphics. When working with bitmap images, a powerful framework to explore is CoreGraphics.

Creating an Empty File

The first step is to create an empty file on the disk that will become our animated GIF.

//create empty file to hold gif
let destinationFilename = String(NSUUID().uuidString + ".gif")
let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(destinationFilename)
//metadata for gif file to describe it as an animated gif
let fileDictionary = [kCGImagePropertyGIFDictionary : [
  kCGImagePropertyGIFLoopCount : 0]]
//create the file and set the file properties
guard let animatedGifFile = CGImageDestinationCreateWithURL(destinationURL as CFURL, UTType.gif.identifier as CFString, totalFrames, nil) else {
  print("error creating gif file")
  return
}
CGImageDestinationSetProperties(animatedGifFile, fileDictionary as CFDictionary)

The code above creates a URL to hold the animated GIF in the user’s Temporary directory. The location is not crucial – it just needs to be where you have write permission. Next it creates a Dictionary that will describe the file as an animated GIF file that will loop forever. The Apple Documentation for GIF Image Properties contains the full list of options for the dictionary. Next, you create the empty file using CGImageDestinatonCreateWithURL. After creating the file, set the properties dictionary. Now there is an empty file ready to accept the frames that will make up the animated GIF.

When creating the file with CGImageDestinationCreateWithURL, you give parameters for the URL to the file location, the UTType identifier to show it will be a .gif file, and the totalFrames that it should expect. totalFrames is just a variable that holds how many actual frames the file will eventually hold. You will need to work out the actual number of frames for your file manually. For example, if you want to display 20 images each second and your GIF will be four seconds long, the total number of frames will be 20 * 4 or 80. The last parameter is always nil. The documentation indicates it “is reserved for future use” (but this function has been available to the Mac since 2005 and iOS since 2010, so we will see). The documentation also indicates you are responsible for releasing the memory allocated by the creation command, but Swift will handle that for you.

As you work with CoreGraphics classes and types, you will notice that they have a slightly different format than other frameworks. That is a hint that CoreGraphics is one of the older frameworks and that it is all C based. Swift has bridging code so that the system will worry about translating your Dictionary to a CFDictionary and your String to a CFString and so on.

Adding Images

To add images to the empty file, you will need an image for each frame of the animation, and you will need a small Dictionary to tell the system how long to display the image during the animation. As mentioned above, there are other options you can apply, such as color mapping or frame dimensions.

A Dictionary to display each frame for 1/20 second could look like this:

    let frameDictionary = [kCGImagePropertyGIFDictionary : [
      kCGImagePropertyGIFDelayTime: 1.0 / 20.0]]

Notice that the values above are Double and are not Int. Using 0.05 would have also been fine as it is equal to 1.0/20.0.

Compressed video files and movies are usually 30, 60, or even 120 frames per second. Recall that animated GIF images do not have compression, so you want to have as low of a frame rate (or as small of an image frame) as you can. Experiment with different frame rates depending on the speed of the action in an animation. Often times 20 or even 10 frames per second will produce high enough quality.

Now, loop through your images and append each image to the .gif file on disk using code like this:

CGImageDestinationAddImage(animatedGifFile, frameImage, frameDictionary as CFDictionary)

The animatedGifFile is the URL to the empty file on the disk that you created above. The frameDictionary is the metadata to add to each frame and the frameImage is the actual image for the frame. Your image for the frame needs to be a CGImage. Depending on the source of images you may have UIImage or CIImage or something else. Most image formats in Swift have a conversion method. For example:

let cgImage = UIImage().cgImage
let anotherCGImage = CIImage().cgImage

There are also CGImage initializers to read .jpeg and .png data.

Finalizing the File

Once you finish writing every frame image to the GIF file, close the file and instruct the system to clean up using:

CGImageDestinationFinalize(animatedGifFile)

This function will return true if it is successful. You must call this function after appending the frames or the resulting GIF will be unusable. Also, after you call this function, you cannot add more frames.

At this point the animatedGifFile URL points to an animated GIF file you can use as needed.

Collecting Frame Images

To create the animated GIF, it is necessary to have an image in CGImage for each frame. It will be easier if they are all rotated the same way and have the same dimensions. Where those images come from is not crucial. They might be a series of files on disk. It could be that your user snaps a series of photos that your app converts to a GIF in real-time. Your app could even intersperse photographs and bitmap drawings. Something to consider regardless of the source of the frame images is how much memory they consume. Loading the entire set of images into an array before making the animated GIF could cause your app to ask the system for gigabytes of memory.

Extracting Images from Video

A common use of animated GIF files is to display videos. So, let’s look at how to extract frames from a video. In some of the other tutorials, you have seen how we can use CoreImage and AVFoundation to manipulate each frame of a video. In those cases, you can’t easily change the frames per second. A 60 frames-per-second animated GIF would quickly become an unusably large file.

AVFoundation provides the AVAssetImageGenerator class specifically to extract frames from AVAssets at specific times. To extract a single image from an AVAsset you can use:

copyCGImage(at requestedTime: CMTime, actualTime: UnsafeMutablePointer<CMTime>?)

This will create a CGImage at or near the requestedTime in the video. Sometimes, the requested time is between two frames, so the function will extract the closest frame it can and return the actualTime of that frame. Your code can then decide if it’s an acceptable substitution and what to do about it. Usually, though, you don’t care and can pass nil for the actualTime parameter. Notice that the actualTime is a pointer to a CMTime variable. This means it is an “inout” parameter. Using “inout” parameters was a common practice when CoreGraphics and AVFoundation were originally written. To use an “inout” parameter, you declare the variable before calling the function and then check its value after. Additionally, when you pass the variable into the function, place an ampersand before the variable name.

For extracting a series of images, copyCGImage is not efficient. It runs on the calling thread and may block your application. Apple provides a different function when you want to extract many frames from a video asset.

generateCGImagesAsynchronously(forTimes requestedTimes: [NSValue], completionHandler handler: @escaping AVAssetImageGeneratorCompletionHandler)

This function takes an array of requestedTimes and attempts to extract the image at each of those timestamps. It will pass the extracted image to the completionHandler. The handler will execute its code one time for every extraction attempt and will not block the calling thread.

Before calling the function to extract images, you need to prepare the inputs. First, you create an AVAssetImageGenerator and configure it.

let movie = AVAsset(url: Bundle.main.url(forResource: "swish", withExtension: "mp4")!)
let frameGenerator = AVAssetImageGenerator(asset: movie)
frameGenerator.requestedTimeToleranceBefore = CMTime(seconds: 0, preferredTimescale: 600)
frameGenerator.requestedTimeToleranceAfter = CMTime(seconds: 0, preferredTimescale: 600)

This code loads a movie file from the app bundle into an AVAsset and then creates a generator for that asset. The .requestedTimeToleranceBefore and .requestedTimeToleranceAfter parameters say you want the exact frame for any requested time. This will make the extraction of frames slower. If you are trying to tune performance, consider modifying these parameters.

Now you will need to calculate the timestamp of each frame of the animated GIF. Determine the start time and the duration of the clip you want to create. Then decide what frame rate (frames per second) to use. You can see in the image below that the entire video consists of about 480 frames at 60 frames per second.

An animated GIF of just the player making the basketball shot would start at 2 seconds and end at 6 seconds of the original video. In the original video, that segment is 240 frames. An animated GIF of the same length at 20 frames per second only needs about 80 frames. The first frame is the one showing at the 2-second timestamp. The next frame should be the one shown at 2 + 1/20 or 2.05 seconds. The next frame would be the one for 2.10 seconds.

The generator requires the array of timestamps as CMTime values. You can create the array using code like this:

var startTime: Double = 2.0 //seconds
var endTime: Double =  6.0 //seconds
var frameRate = 20 //how many frames per second
let totalFrames = Int((endTime - startTime) * Double(frameRate))
var timeStamps = [NSValue]()
for currentFrame in 0..<totalFrames {
  let frameTime = Double(currentFrame) / Double(frameRate)
  let timeStamp = CMTime(seconds: startTime + Double(frameTime), preferredTimescale: 600)
  timeStamps.append(NSValue(time: timeStamp))
}

This code calculates a value for totalFrames which must be an Int so that you can use it to loop. But, startTime and endTime need to be Double since the timestamps will be sub-second times. The frameTime of the first frame is equal to zero in the animated GIF but 2.0 seconds in the original video. Convert that to the video timeStamp by adding the startTime. When creating the CMTime using “600” as the preferredTimescale is common because most frame rates are divisible into 600 evenly so the math for the timestamp will be more accurate. Finally, wrap the timeStamp in an NSValue object and add it to the array. The AVAssetImageGenerator uses timestamps wrapped in NSValue objects. NSValue is just a generic type that holds values.

With the generator configured and the array of timestamps created, you can extract the images using code like this:

frameGenerator.generateCGImagesAsynchronously(forTimes: timeStamps) { requestedTime, frameImage, actualTime, result, error in
  guard let frameImage = frameImage else {
    print("no image")
    return
  }
  if error != nil {
    print("an error")
    return
  }
  //do someting interesting with frameImage
}

The frameImage in the completion handler will be a CGImage type if the extraction was successful. As with the function to extract a single image, if the system had to choose a different time than requested, you will be informed of the actual time. You can read about the other parameters in the documentation for AVAssetImageGeneratorCompletionHandler.

Now, you can add the frame to an open CGImageDestination, you could write it to a disk to use later, and you could save it to an array. If you don’t write it to an animated GIF file right away, you will need to devise a way to keep the frames in sequence. You may notice that there is not an explicit way for the function to signal that it is finished. Because the completion handler is called for every extraction attempt, a common strategy is to make a simple counter and increment the counter in the completion handler. When the counter variable is equal to the number of frames requested, then you are finished. Another way is to match the requestedTime to the last timeStamp.

Wrapping Up

In this article, you saw how to create an animated GIF from a series of images. You also saw how to extract still images from a video at pre-determined timestamps. Clone this repository for a simple project that contains the sample code and a video file you can experiment with.

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

How to Make Videos from Still Images with AVFoundation and Swift

Walter — Thu, 10 Feb 2022 15:35:45 GMT

Whether making a slide show or breaking up video tracks with title scenes, inserting still images into a video file can be tricky. This tutorial will show you how to convert still images into standard Quicktime video files so you can work with them further in your favorite video editor. This tutorial was created and tested with Xcode 13 and Swift 5. This tutorial will discuss two strategies:

convert each image to a separate movie and stitch them together later (this is best suited for times you want to interleave the images and other video files using some other editing workflow)
create a “blank” video and use the applyingCIFiltersWithHandler variation to overlay the images (this is best suited for times you want to make a quick slide show with no further editing)

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with this code (and some extra goodies) can be found on GitHub.

The Problem with an Image

The basic models that are used in AVFoundation for making video files are AVAsset and AVAssetTrack. You can compose multiple tracks and assets together to make a single video file. All tracks must have some media data and a duration. Static image files don’t have a duration. This is the primary problem you have to overcome when you want to add static images to video.
Fortunately, you can use many of the same AVFoundation tools that are used for video capture and frame manipulation. However instead of using a CADisplayLink and AVPlayerItemVideoOutput or an AVCaptureDevice to provide frame buffers to save, you will create the frame buffers manually. Then you can use AVAssetWriter to convert the frame buffers into a video file.

Some items to consider before you actually generate the video file are:

What framerate to use? Because the image doesn’t have any motion, you can use a very low frame rate to create smaller file sizes. However, this might cause issues if you are interleaving it with regular video, which usually has a frame rate of 30 or 60 fps (or higher, for slow motion video)
What dimensions will the final video have? It is easy to resize an image and add padding using CoreImage or some image editing software before it becomes a video. As the image becomes a video and moves through different steps in an AVFoundation pipeline, the different classes in AVFoundation will resize or crop the images in different ways when the input dimensions and output dimension don’t match.

Writing Buffers using `AVAssetWriter`

The AVAssetWriter exists to encode media to a file on disk. Though this tutorial will use a single video input AVAssetWriter supports multiple inputs of different kinds (audio, video, metadata). It is quite similar to AVAssetExportSession. The difference is that AVAssetWriter uses multiple inputs and each input is a single track where the AVAssetExportSession takes a single AVAsset as an input. The result of both is a single file.

As indicated above, our basic strategy will be:

Create a pixel buffer that is the correct color space and size to hold the image
Render the image into the pixel buffer
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop, and each time through the loop, append the pixel buffer
Clean up

Create the Pixel Buffer

The code below creates a CIImage from an image file. Then it creates a pixel buffer the same size as the image and uses a standard color space. Finally, a CIContext renders the image into the pixel buffer.

//create a CIImage
guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
}
//create a variable to hold the pixelBuffer
var pixelBuffer: CVPixelBuffer?
//set some standard attributes
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
     kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
//create the width and height of the buffer to match the image
let width:Int = Int(staticImage.extent.size.width)
let height:Int = Int(staticImage.extent.size.height)
//create a buffer (notice it uses an in/out parameter for the pixelBuffer variable)
CVPixelBufferCreate(kCFAllocatorDefault,
                    width,
                    height,
                    kCVPixelFormatType_32BGRA,
                    attrs,
                    &pixelBuffer)
//create a CIContext
let context = CIContext()
//use the context to render the image into the pixelBuffer
context.render(staticImage, to: pixelBuffer!)

Though we generally think of CIImage as an image, Apple is clear that it is not an image by itself. CIImage needs a context for rendering. This is where a lot of the power of CoreImage and filters and GPU rendering come from, the fact that CIImage is just the instructions for creating an image. Note: You can also use CGImage and the CoreGraphics framework to create pixel buffers. I find it easier to use the CoreImage framework.

Configure AVAssetWriter

Now with a buffer created, you can configure the AVAssetWriter. It will take several parameters as a dictionary to determine the dimensions and format of the outputs. You can either set them individually or else Apple provides presets.
One of the most important settings is the output dimension. If the output dimension matches the original image, the video file will show no distortion or letterboxing. However, if the output is smaller, it will compress the image until it fits. If the output is larger, the image will expand until its width or height matches the output size and then will letterbox. Note: if an image expands too much, it will appear grainy. The image below shows different output sizes for a 640 x 480 input image.

To create your settings for the AVAssetWriter first create a dictionary containing the different values. The example here sets an output dimension of 400 x 400 and provides for .h264 encoding.

let assetWriterSettings = [AVVideoCodecKey: AVVideoCodecType.h264, AVVideoWidthKey : 400, AVVideoHeightKey: 400] as [String : Any]

To use one of the presets, use the AVOutputSettingsAssistant. You can read about the different settings in the Apple documentation. An example to create 1080p video output would look like this:

let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings

Write the pixel buffer to the video file

With the settings configured, the asset writer can loop through and append the contents of the pixel buffer to create each frame of the video. Something important to notice in the code below is that AVFoundation has a philosophy to preserve data. So, it will make copies, and it will generally not overwrite files. That is why you have to delete any old files before you can create the new AVAssetWriter.

//generate a file url to store the video. some_image.jpg becomes some_image.mov
guard let imageNameRoot = imageName.split(separator: ".").first, let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("\(imageNameRoot).mov") else {
  throw ConstructionError.invalidURL //an error i made up
}
//delete any old file
do {
  try FileManager.default.removeItem(at: outputMovieURL)
} catch {
  print("Could not remove file \(error.localizedDescription)")
}
//create an assetwriter instance
guard let assetwriter = try? AVAssetWriter(outputURL: outputMovieURL, fileType: .mov) else {
  abort()
}
//generate 1080p settings
let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings
//create a single video input
let assetWriterInput = AVAssetWriterInput(mediaType: .video, outputSettings: settingsAssistant)
//create an adaptor for the pixel buffer
let assetWriterAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: assetWriterInput, sourcePixelBufferAttributes: nil)
//add the input to the asset writer
assetwriter.add(assetWriterInput)
//begin the session
assetwriter.startWriting()
assetwriter.startSession(atSourceTime: CMTime.zero)
//determine how many frames we need to generate
let framesPerSecond = 30
//duration is the number of seconds for the final video
let totalFrames = duration * framesPerSecond
var frameCount = 0
while frameCount < totalFrames {
  if assetWriterInput.isReadyForMoreMediaData {
    let frameTime = CMTimeMake(value: Int64(frameCount), timescale: Int32(framesPerSecond))
    //append the contents of the pixelBuffer at the correct time
    assetWriterAdaptor.append(pixelBuffer!, withPresentationTime: frameTime)
    frameCount+=1
  }
}
//close everything
assetWriterInput.markAsFinished()
assetwriter.finishWriting {
  pixelBuffer = nil
  //outputMovieURL now has the video
  Logger().info("Finished video location: \(outputMovieURL)")
}
}

Now, your image has become a Quicktime video of whatever duration you specified and is ready for use as any other video file! You can now continue with the like of VideoEditor SDK to trim videos, and add filters or overlays. You might also want to combine the video with other video files to make a new creation.

Say hi to my dog! A plain video from still images

Creating a Blank Video

The previous example created a single .mov file for each of your images. The most robust way to get these images into a video with sound, transitions, etc., is to use AVMutableComposition with multiple tracks and layer instructions. However, for a quick slideshow with just a few elementary transitions, a strategy is to create a blank movie and then use AVVideoComposition with (asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void). That will let you use CIImage and CIFilter to paint each frame of the blank movie with images.

Our basic strategy for this method will be:

Create a pixel buffer that is the right color space and size and fill it with a solid color
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop and each time through the loop append the pixel buffer
Create an AVVideoComposition to add the images to the individual frames of the video
Let AVPlayer combine the video and the composition

In the first strategy, you started by making a pixel buffer and using an AVAssetWriter to create the video file using this code:

guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
    }

However, this time, instead of using a source image, you create a CIImage that is a single color.

let staticImage = CIImage(color: bgColor).cropped(to: CGRect(x: 0, y: 0, width: 960, height: 540))

The CIImage(color:) initializer creates an image that is of infinite size and is just a single color. Using the .cropped(to:) modifier lets you make it the same size as the AVAssetWriter output so that there won’t be any letterboxing. The rest of the code is the same as before and the end result will be a video file that is just the single color.

Adding a Composition

Once the video file has been created and saved to disk, load it as an asset, then create an AVVideoComposition. This allows you to generate individual frames. The function below uses the request property of the composition. This has a CIImage representation of the current frame as well as the timestamp of when this frame will appear. The .fetchSlide(forTime:) is a helper function in the demo app that returns the appropriate CIImage for the slide to display. Then the .sourceOverCompositing filter combines the original frame image with the slide. Using a CGAffineTransform moves the slide across the screen as the video plays.

func createComposition(_ asset: AVAsset) {
  let slideshowComposition = AVVideoComposition(asset: asset) {[weak self] request in
    guard let self = self else { return }
    let slide = self.fetchSlide(forTime: request.compositionTime)
    let compose = CIFilter.sourceOverCompositing() //filter to join two images
    compose.backgroundImage = request.sourceImage
    compose.inputImage = slide?.transformed(by: CGAffineTransform(translationX: request.compositionTime.seconds * 50, y: 0))
    //always finish with the last output of the pipeline
    request.finish(with: compose.outputImage!, context: nil)
  }
  self.outputSlideshow = slideshowComposition
}

Because the creation of the video is done asynchronously, you can composite images over the original frame and add filters without the danger of impacting the final frame rate. Recall that CIFilter works as a pipeline, each CIFilter has an .outputImage and most have .inputImage and some configuration settings. Feeding the output from one filter to the input of the next will let you build complex compositions. Once a composition has been created, AVPlayer can join the original video with the composition for playback or export.

let item = AVPlayerItem(asset: self.outputMovie!)
item.videoComposition = outputSlideshow
self.player = AVPlayer(playerItem: item)

The code above might generate a video like this one.

As with the first video, the final video is a plain .mov file.

Going Further

In this tutorial, you saw two ways to convert static images into .mov files that you can then edit with any video editor. The sample project also has some code for exporting your creation as a .mov file and code for adding sound to the creation.

These are not the only ways to go about solving this problem. For example, instead of creating a blank video and overlaying the images, using a stock video of waves at the beach and overlaying the images might work better for your project. Conversely, because the initial pixel buffer is created from a CIImage, there is no reason that the CIImage cannot be the end of a long pipeline of CIFilters. This way the composition is done at the beginning, and the AVAssetWriter is the only tool needed. The best strategy is the one that makes sense to you and that works with the kinds of media you have.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Merge Videos in iOS with Swift

Walter — Wed, 12 Jan 2022 12:12:55 GMT

Combining video clips or parts of video clips into a single video may appear complicated, but the things that add complexity also provide flexibility. In this tutorial, you will see how to combine a few short clips into a single video. You will also gain an understanding of building on this basic pattern to make more advanced creations. This tutorial was created and tested with Xcode 13 and Swift 5.

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with code that supports this tutorial is available on Github.

Making a Composition

Whenever you want to edit media or add effects, the first place to look is AVComposition. This class in AVFoundation has the purpose of arranging different assets and types of assets into a single asset for playback or processing. Asset types can include audio and video subtitles, metadata, and text. When working with AVComposition it is helpful to think about AVAsset and AVAssetTrack objects. Unfortunately, because these items are so similar (AVComposition is a subclass of AVAsset) it is easy to get confused. Try to remember that the AVTrack is a wrapper around the actual pixel, sound-wave, or other data and that AVAsset is a collection of AVTrack objects. When we want to combine and manipulate AVAsset objects, an AVComposition helps us do that. Changes made to any AVAsset by an AVComposition will not impact the original media file.

As an extra layer, Apple keeps the editing features of an AVComposition separate from the playback features by creating an AVMutableComposition object. You will work with an AVMutableComposition object in this tutorial.

Each track in our composition will have a start time and a duration. The composition will ensure that all of the data it holds displays at the right time and that the tracks stay in sync.

At the simplest, a video clip is a single AVTrack of audio data and a single AVTrack of video data.

So, to set up an empty AVComposition for combining clips, use code such as this:

    let movie = AVMutableComposition()
    let videoTrack = movie.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
    let audioTrack = movie.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)

The code above creates a new, empty AVMutableComposition and then adds two tracks. One track will be able to hold .video assets, and the other will hold .audio assets. AVFoundation supports many different kinds of audio and video formats, but the .video track cannot hold .audio formats. The kCMPersistendTrackID_Invalid signals that you want a new, unique track id generated.

AVFoundation can manipulate any of the ISO media formats. The Quicktime (.mov) and MPEG (.mp4) are the most common, however, .heif, .3gp and other formats are all supported.

Adding Video Clips

After creation, the AVMutableComposition has a start time of CMTime.zero and a duration of CMTime.zero. Use the insertTimeRange(_ timeRange: CMTimeRange, of track: AVAssetTrack, at startTime: CMTime) throws function on the audio and again on the video track to add data from the clip. In order to add data to the tracks, you load an AVAsset, determine what range of time will go into the new composition, and then call the insert function. Your code might look like this:

  let beachMovie = AVURLAsset(url: Bundle.main.url(forResource: "beach", withExtension: "mov")!) //1

  let beachAudioTrack = beachMovie.tracks(withMediaType: .audio).first! //2
  let beachVideoTrack = beachMovie.tracks(withMediaType: .video).first!
  let beachRange = CMTimeRangeMake(start: CMTime.zero, duration: beachMovie.duration) //3
  try videoTrack?.insertTimeRange(beachRange, of: beachVideoTrack, at: CMTime.zero) //4
  try audioTrack?.insertTimeRange(beachRange, of: beachAudioTrack, at: CMTime.zero)

Here is what this code does:

Load a video file from a local URL. In the example, the video is in the application bundle, but it could also be a URL that points to a file somewhere else. This will not work with a streaming URL. It needs to be a video file.
Extract the main videos and audio tracks from the video file. There may be multiple video and audio tracks in a file, but this code just grabs the first one.
Create a time range. In this example, the range is the same duration as the clip we loaded.
Insert the video track from the clip at the beginning of the video track of the composition and insert the audio track from the clip at the beginning of the audio track of the composition. A common mistake in this step is to use the .duration property of the composition to insert the audio and video at the end. However, as soon as media is inserted into any track, the duration of the entire composition will change. So the audio and video would be played one after the other and you would see a blank screen when the audio is playing, and have no audio when the video is playing.

Repeat the steps for each clip that you want to add to the final composition.

Using the Composition

After all of the clips have been combined into the composition, it can be played in an AVPlayer or exported using an AVAssetExportSession. The code for these is the exact same as for any other AVAsset.

  self.player = AVPlayer(playerItem: AVPlayerItem(asset: movie)) //1
  let playerLayer = AVPlayerLayer(player: player) //2
  playerLayer.frame = playerView.layer.bounds
  playerLayer.videoGravity = .resizeAspect
  playerView.layer.addSublayer(playerLayer) //3
  player?.play() //4

Here is what the code above is doing:

Create an AVPlayer using a movie object, which is the AVMutableComposition that was created earlier.
Create a playerLayer to display the video and set its aspect ratio
Add the playerLayer to playerView which is a regular UIView in a UIViewController
Play the video clip

Alternatively you may want to export the new composition as a new movie file. Again, because the composition is an AVAsset using a standard AVAssetExportSession will write the movie to disk.

  //create exporter
  let exporter = AVAssetExportSession(asset: movie,
                                 presetName: AVAssetExportPresetHighestQuality) //1
  //configure exporter
  exporter?.outputURL = outputMovieURL //2
  exporter?.outputFileType = .mov
  //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error { //3
        print("failed \(error.localizedDescription)")
      } else {
        print("movie has been exported to \(outputMovieURL)")
      }
    }
  })

Here is what this code is doing:

Create an export session using the movie composition you made earlier.
Set the save location to some file URL on your device.
Wait for the exporter to finish and display the result.

Don’t forget that exporting an AVAsset can take a long time and is asynchronous. You will want to display a spinner or a message to your user.

Going Further

You can now combine clips into a longer clip for playback and export. However, trimming the original clips, adding filters, and rearranging the order will require more work before your application is ready for the store. Using an SDK like the IMG.LY’s VideoEditor SDK can help you provide an app that has the features users will expect in addition to stitching clips together.

The VideoEditorSDK offers a Video Composition Controller which will allow the users to stitch clips together. Additionally, it will allow the user to add new clips and edit clips using the other editing tools. Because the Video Composition Controller is a subclass of UIViewController it can just be configured and dropped into your application. Instead of the code in our example, you pass the videos to the controller:

let videoClips = [
  Bundle.main.url(forResource: "stream", withExtension: "mov"),
  Bundle.main.url(forResource: "beach", withExtension: "mov"),
  Bundle.main.url(forResource: "mountain", withExtension: "mov"),
].compactMap { $0.map{ AVURLAsset(url: $0) } }
let video = Video(assets: videoClips)
let videoEditViewController = VideoEditViewController(videoAsset: video, configuration: Configuration())
present(videoEditViewController, animated: true, completion: nil)

The code above loads the same video clips as our demo application and then uses the VideoEditViewController to present them. From there the user can make further edits preview and export the composition.

Wrapping Up

In this example, you saw how to combine several video clips, each with one video and one audio track into a single movie file with one video and one audio track. To make more complex movie files consider building on this tutorial by:

Add a second audio track, AVFoundation will play sounds from both tracks at times specified at equal volume. Use AVAudioMix to adjust the volume of the tracks at different times (the sample project has an example of adding a second audio track).
Add more video tracks and use AVMutableCompositionLayerInstruction to determine which track is visible at any time in the final video and to make fancy transitions between tracks.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How To Add Text to Video in Swift

Walter — Fri, 15 Oct 2021 15:05:59 GMT

In this article, you will see how to use Swift and AVMutableVideoComposition to add a text overlay to a video clip. The code in this article uses Swift 5. Clone this repository for a Swift Playground with the example code.

Setting up

To access the AVFoundation and CoreImage objects, be sure to add these imports to your code

import AVFoundation
import CoreImage.CIFilterBuiltins

Importing the CIFilterBuiltins lets you use autocompletion when working with CIFilters. Otherwise, just import CoreImage. But then you will need to access the filter properties using strings.

When working with video clips and movie files, the basic starting point is the AVAsset class. This class combines all of the timed video and audio tracks that make up a movie. Additionally there may be subtitles and timed metadata or captions.

Creating a Video Composition

After you create a video composition for an asset, you can apply it to an AVPlayerItem to display on screen or to an AVAssetExportSession to write to a file. The composition for this tutorial will use the init(asset:applyingCIFiltersWithHandler:) initializer. Then you can apply CIFilters to each frame of the video.

First, load a movie file as an AVAsset using its URL.

//Fetch a URL for the movie from the bundle
let waterfallURL = Bundle.main.url(forResource: "waterfall", withExtension: "mov")
//Create an AVAsset with the url
let waterfallAsset = AVAsset(url: waterfallURL!)

Now you can create a AVMutableVideoComposition with the asset

let titleComposition = AVMutableVideoComposition(asset: waterfallAsset) {request in
//apply filters here
request.finish(with: request.sourceImage, context: nil)
}

The code above passes the video frame unaltered. The request is the current video frame. It is an AVAsynchronousCIImageFilteringRequest object. The request object has three properties:

the sourceImage which is the video frame as a CIImage
the renderSize which is the size of the video frame
compositionTime which is the timestamp of the current frame

The code in the handler will run once for each frame of the video clip.

Generating a Text Image

Because the sourceImage is a CIImage, you can use any of the over 200 CIFilter objects that Core Image provides. There are two text generator filters: CIAttributedTextImageGenerator and CITextImageGenerator. Either of these will render a string as a CIImage. This tutorial will use the Attributed Text version so you can modify the color and add a shadow.

Start by creating a shadow and then creating a dictionary of attributes to apply to the string.

let whiteShadow = NSShadow()
whiteShadow.shadowBlurRadius = 5
whiteShadow.shadowColor = UIColor.white
let attributes = [
  NSAttributedString.Key.foregroundColor : UIColor.blue,
  NSAttributedString.Key.font : UIFont(name: "Marker Felt", size: 36.0)!,
  NSAttributedString.Key.shadow : whiteShadow
]

Create the attributed string by combining the string and the attributes

let waterfallText = NSAttributedString(string: "Waterfall!", attributes: attributes)

Provide the attributed string and a scale factor to the filter to generate an image like the one shown above.

let textFilter = CIFilter.attributedTextImageGenerator()
textFilter.text = waterfallText
textFilter.scaleFactor = 4.0

The textFilter.outputImage will be the image of the rendered text with the attributes applied. The extent of the outputImage will be a rectangle that is large enough to encompass the text. The text will render on a single line, it doesn’t wrap using this method.

If you were to place the text on the video image at this point, the bottom-left of the text would be at the bottom-left of the video. Unlike UIView objects, the origin point (0,0) is at the bottom-left, not the top-left. To center the text and move it off of the bottom, you can apply a standard CGAffineTransform

let centerHorizontal = (request.renderSize.width - textFilter.outputImage!.extent.width)/2
let moveTextTransform = CGAffineTransform(translationX: centerHorizontal, y: 200)
let positionedText = textFilter.outputImage!.transformed(by: moveTextTransform)

Now finish the pipeline by composing the new text image over the original source image.

positionedText.composited(over: request.sourceImage)

All together, the original composition now becomes

let titleComposition = AVMutableVideoComposition(asset: waterfallAsset) { request in
//Create a white shadow for the text
let whiteShadow = NSShadow()
whiteShadow.shadowBlurRadius = 5
whiteShadow.shadowColor = UIColor.white
let attributes = [
  NSAttributedString.Key.foregroundColor : UIColor.blue,
  NSAttributedString.Key.font : UIFont(name: "Marker Felt", size: 36.0)!,
  NSAttributedString.Key.shadow : whiteShadow
]
//Create an Attributed String
let waterfallText = NSAttributedString(string: "Waterfall!", attributes: attributes)
//Convert attributed string to a CIImage
let textFilter = CIFilter.attributedTextImageGenerator()
textFilter.text = waterfallText
textFilter.scaleFactor = 4.0
//Center text and move 200 px from the origin
//source image is 720 x 1280
let positionedText = textFilter.outputImage!.transformed(by: CGAffineTransform(translationX: (request.renderSize.width - textFilter.outputImage!.extent.width)/2, y: 200))
//Compose text over video image
request.finish(with: positionedText.composited(over: request.sourceImage), context: nil)
}

Displaying The Finished Video

With an AVAsset and an AVMutableVideoComposition you can now combine the two into an AVPlayer to display in an AVPlayerViewController or in your own UIViewController

let waterFallItem = AVPlayerItem(asset: waterfallAsset)
waterFallItem.videoComposition = titleComposition
let player = AVPlayer(playerItem: waterFallItem)

As stated above, you can also combine the asset and the composition and write them to a new movie file using an AVAssetExportSession.

Going Further

The method in this tutorial is suitable for adding watermark or title text to videos. If you want to give users the ability to add custom text, there is more work to do. You will need to create font and color pickers as well as code to position the text in the frame.

You can use an editor such as VideoEditorSDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to use AVMutableVideoComposition to add text to a video clip. Further, you saw how using an SDK such as VideoEditorSDK allows you to annotate and enhance your clips before sharing. Including typography, audio support and video composition, IMG.LY provides a comprehensive solution for mobile video editing – find the documentation here.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

How to Trim and Crop Video in Swift

Walter — Tue, 31 Aug 2021 16:03:25 GMT

In this article, you will see how to use Swift and AVKit to crop a video clip and how to trim a video’s timeline. Then you will learn how to use AVAssetExportSession to write your edited video to disk. The code in this article uses Swift 5 and Xcode 12.5. Clone this repository for a sample project and example code.

Anatomy of a Video File

The AVFoundation and AVKit frameworks are what Swift and iOS use to manage audio and video. When starting out, you may use an AVPlayerViewController for playback with the same controls and features as the native players. You can also use an AVPlayer object and provide your own playback and editing controls. Whichever you choose, you begin by loading a media file (.mp3, .mov, etc.) into an AVPlayerItem. Inside the AVPlayerItem, the media becomes an AVAsset which may have many tracks of video, audio, text, closed captions etc.

To manipulate an AVAsset, Apple provides the AVComposition class. The AVComposition can act on a single track or many tracks to filter and mix underlying media and produces a single output. An AVPlayer shows the output of an AVComposition on a device. When it is time to export, you can use an AVAssetExportSession with the same AVComposition to write to disk or upload to a server.

Cropping Video to a Rectangle

Apple provides some different AVComposition classes optimized for common tasks. Apple recommends using one of these classes instead of writing custom AVComposition classes whenever possible. The reason for this is that when Apple introduces new technologies (like HDR Video), they will ensure it works with their classes. If you’ve written your own, you will have to update it to work with the new technologies. In this article you will crop the video to a rectangle and let the audio pass through as recorded. A good class to use for this task will be the AVMutableVideoComposition with the init(asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void) initializer. This will allow you to apply a CIFilter to each frame of the video. Apple provides an efficient cropping filter called CICrop.

func transformVideo(item: AVPlayerItem, cropRect: CGRect) {

  let cropScaleComposition = AVMutableVideoComposition(asset: item.asset, applyingCIFiltersWithHandler: {request in

    let cropFilter = CIFilter(name: "CICrop")! //1
    cropFilter.setValue(request.sourceImage, forKey: kCIInputImageKey) //2
    cropFilter.setValue(CIVector(cgRect: cropRect), forKey: "inputRectangle")


    let imageAtOrigin = cropFilter.outputImage!.transformed(by: CGAffineTransform(translationX: -cropRect.origin.x, y: -cropRect.origin.y)) //3

    request.finish(with: imageAtOrigin, context: nil) //4
    })

  cropScaleComposition.renderSize = cropRect.size //5
  item.videoComposition = cropScaleComposition  //6
}

The applyingCIFiltersWithHandler will execute for every frame in the asset of the AVPlayerItem. Here is what the code above will do:

Create a CICrop filter (note that this demo uses ! to force unwrap, production code should handle failure)
Add the .sourceImage (a CIImage) from the request to the filter.
Move the cropped image to the origin of the video frame. When you resize the frame (step 4) it will resize from the origin.
Output the transformed frame image
Set the size of the video frame to the cropped size
Attach the composition to the videoComposition property of the asset

Notice that the transformation does not alter the underlying asset. It creates a videoComposition filter to attach to the item during playback. After the transformation, AVPlayer will display the new creation when executing its .play() function.

Sharp eyed readers will have noticed that the init method mentions “applying CIFilters” plural. Instead of one filter, you can create an entire pipeline of CIFilter objects to manipulate the visual properties of the frames. The request object also contains the compositionTime so filters can change at different parts of the video.

Trimming the Time of a Video

When an AVPlayer loads an AVItem the start time of the video will be at CMTime.zero. The end will be the duration property of the AVItem. To adjust the playback times, you call the .seek function to move to the new start time and set the forwardPlaybackEndTime to the new end time. Now the player will only play the part of the clip between those times.

//The player object is already created and configured to play video in the ViewController

//load a video .mov file
self.player = AVPlayer(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)

//Set the start time to 5 seconds.
let startTime = CMTimeMakeWithSeconds(5, preferredTimescale: 600)

//Convert the duration of the video to seconds
let videoDurationInSeconds = self.player!.currentItem!.duration.seconds

//Subtract 5 seconds from the end time
let endTime = CMTimeMakeWithSeconds(videoDurationInSeconds - 5, preferredTimescale: 600)

//Assign the new values to the start and end time
self.player?.seek(to: startTime)
self.player?.currentItem?.forwardPlaybackEndTime = endTime

//Play the video
self.player?.play()

The CMTime (Core Media Time) object uses timescales and Int values to map to the individual frames of tracks. Video tracks commonly come to your app with 24, 30, 60 or 120 frames per second. Converting between these different formats using Floats or Doubles would be imprecise. When converting seconds to CMTime with video, 600 is the commonly used timescale because it is a common multiple of all of the standard frames per second.

Exporting the Video

As with the videoComposition property, setting the start and end times of the player only modifies the playback of the video clip. The underlying asset remains unchanged. When you use the AVAssetExportSession, the new video file will have the change. The export session applies time range and video composition objects as it writes the file. A function to export might look like:

func export(_ asset: AVAsset, to outputMovieURL: URL, startTime: CMTime, endTime: CMTime, composition: AVVideoComposition) {

    //Create trim range
  let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)

    //delete any old file
  do {
    try FileManager.default.removeItem(at: outputMovieURL)
  } catch {
    print("Could not remove file \(error.localizedDescription)")
  }

    //create exporter
  let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)

    //configure exporter
  exporter?.videoComposition = composition
  exporter?.outputURL = outputMovieURL
  exporter?.outputFileType = .mov
  exporter?.timeRange = timeRange

    //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error {
        print("failed \(error.localizedDescription)")
      } else {
        print("Video saved to \(outputMovieURL)")
      }
    }
  })
}

Going Further

AVKit and AVFoundation provide simple objects for manipulating video files. The difficulty when working with video and audio tracks usually comes while providing editing controls for a user. The image below shows how the original video dimensions might change from creation to display.

The original 2160x3840 video appears on an iPhone in a 357x635 frame. Additionally the origin point (green dot) of the video file and the origin point of the UIViews are not equal. Passing the frame of the red, cropping rectangle to an AVComposition would not work as expected. Inside of the AVComposition the video resumes it’s 720x1280 dimensions while the 250x250 cropping rectangle would remain 250x250.

Before using a UIView rectangle with an AVComposition it needs to resize and the origin point needs to align to the video’s origin. Your app must apply a CGAffineTransform to reorient the origin points.

var cropRect = self.croppingView.frame //1
let originFlipTransform = CGAffineTransform(scaleX: 1, y: -1)
let frameTranslateTransform = CGAffineTransform(translationX: 0, y: renderingSize.height)
cropRect = cropRect.applying(originFlipTransform) //2
cropRect = cropRect.applying(frameTranslateTransform) //3

Now you apply regular ratio math to the dimensions of the cropping rectangle to match the video dimensions.

let renderingSize = playerItem.presentationSize

let xFactor = renderingSize.width / playerView.bounds.size.width
let yFactor = renderingSize.height / playerView.bounds.size.height

let newX = croppingView.frame.origin.x * xFactor
let newW = croppingView.frame.width * xFactor
let newY = croppingView.frame.origin.y * yFactor
let newH = croppingView.frame.height * yFactor
var cropRect = CGRect(x: newX, y: newY, width: newW, height: newH)

The transformations above are standard when working with AVKit and UIKit or SwiftUI in an app. The math is not complex, but it is tedious. Unless your app is a custom video editing application, you may find that using a commercial solution such as VideoEditorSDK is a better approach. By adding the VideoEditorSDK, your app can have professional appearing trim and crop controls as well as filtering, text and more.

Wrapping Up

In this article, you saw how to crop a video image and how to trim time from a track, all in Swift. Further, you saw how using an SDK such as VideoEditorSDK allows you to provide full-featured video editing for any application.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How to build Instagram’s Story Editor in a Day

Malte — Wed, 13 Dec 2017 00:00:00 GMT

A common question we get asked from our customers, is whether they’ll be able to create an entirely different user interface using our PhotoEditor SDK. The SDK comes with it’s own customizable UI, but customization are of course limited to a certain extent. As our SDK is used in many different use cases and contexts, we like to explore what’s possible with the PhotoEditor SDK and its included components. Here, we decided to build a UI similar to Instagram’s Stories or Snapchat using our SDK. By now, this ‘Story UI’ has become a popular way of quickly designing with different elements (stickers, brush and text) rather than enhancing and styling the image only.

So we grabbed our own docs and headed out to create a demo in which we recreate the Instagram Stories using components from the PhotoEditor SDK. This article presents our approach by starting with a general overview and then diving into the different view controllers to look at specific implementation details. You can follow along by downloading the accompanying code.

Architecture and Overview

Any image editing interface is naturally centered around some sort of canvas or preview. This is where the image with all operations applied is rendered and any changes are shown to the user. All tools add their own interface elements to allow modifications like adding stickers, text or brush strokes to the image. To create such a hierarchy of tools, the PhotoEditor SDK makes heavy use of an iOS pattern called view controller containment. The root view controller, an EditViewController in this case, manages a series of child view controllers:

A PhotoEditPreviewController that handles the internal model and all rendering, as well as the rendering canvas itself
A TextSpriteEditController above the PhotoEditPreviewController that allows selection and manipulation of text sprites
A StickerSpriteEditController above the PhotoEditPreviewControllermanages selection and manipulation of stickers

The child view controllers may manage additional view controllers themselves, but we only need to manage the topmost objects and wire their interfaces together. This is done within the EditViewController who is also responsible for presenting the view controllers that implement the different tools, StickerViewController, BrushViewController and TextViewController.

These are mainly responsible for managing the interface and adjusting the model accordingly, while the ‘real’ work is being done in the background by the PhotoEditPreviewController whenever the model gets updated. Thanks to the SDK, creating our BrushViewController boils down to creating and wiring a BrushEditController and adding its view, color and size controls to the BrushViewControllers view. This creates a fully fledged brush tool with OpenGL rendering, color and brush size adjustments, and great performance in just 89 lines of code.

EditViewController

As described in the previous section, the EditViewController is the root view controller of our demo application and is responsible for showing a preview, presenting tools and rendering the final output. To do so, we add a PreviewEditViewController and SpriteEditControllers to the root view and register the EditViewController as their delegate. Registering as a delegate, allows the EditViewController to wire all child view controllers and pass data between them. In order to do so, we just need to return objects of other view controllers or react to model changes. All controllers then use the EditViewController to ask for objects or data they need, for example the current size of the preview view, in order to rearrange interface elements or calculate model updates. As an example, the spriteEditControllerPreviewView(_ spriteEditController:) method of the SpriteEditControllerDelegate protocol, asks for the current preview view. All we need to do in our EditViewController is to get this view from our PreviewEditViewController and return it:

Sprite Handling

Stickers and text sprites are both sprites, but there are different SpriteEditController subclasses for each sprite type, because of differing gestures and UI elements. The Instagram UI allows the editing of both sticker and text sprites within a single view, so we need to add both specific controllers, StickerSprite- and TextSpriteEditController to our EditViewController and dynamically enable the appropriate one depending on the currently selected sprite.

StickerViewController

For adding stickers, the Instagram Stories UI offers a single button that presents a grid of all available stickers. Replicating this using the PhotoEditor SDK is easy to do, as the SDK uses a collection view for presentation as well. This collection view is embedded in a StickerSelectionController which we add to our StickerViewController. Once again, after registering as the StickerSelectionControllers delegate, we get notified upon selection of any sticker and just need to change the PhotoEditModel accordingly. To do so, we create a link between the root and the currently presented view controller using the StickerViewControllerDelegate protocol. Once a sticker has been selected, we create or update a StickerSpriteModel, pass it to the EditViewController, which updates the PreviewEditViewController model:

This triggers a new rendering pass and the sticker appears on screen. All that’s left to do now is closing the StickerViewController. Again, we use the delegate pattern to ask the EditViewController about the referenceSize that’s needed to calculate the sticker’s normalized size.

BrushViewController

This view controller, just as described above, is just a wrapper around the BrushEditController that adds a color and size selection. Once again, all interaction is handled through delegates and upon closing of the brush tool, the internal model is updated. To support undo, we need to pass the PreviewEditViewController’s undoController and begin a new group, whenever we fire up the tool.

TextViewController

The text editor Instagram created within their Stories UI is essentially a fullscreen textfield with additional controls for color and size selection. Ignoring the customized design, this can easily be recreated using stock iOS components and essentially only requires listening for keyboard notifications to handle the layout and adjusting the textfield’s properties according to user interactions. Once finished, we create a corresponding TextSpriteModel that is positioned below the UITextField and notify the EditViewController using the delegate. After dismissing the TextViewController the text rendering is handled by the PhotoEditPreviewController. When reselecting an existing label, we just prefill the UITextField with it’s contents and update the model when the label textfield changes. To ensure the UITextField is the only place the text is currently visible, we hide the spriteView we previously selected and show it again, once editing has finished.

The Result

Recreating the Instagram Stories UI seemed rather challenging at first, but using our PhotoEditor SDK, it quickly turned into a very rewarding project. Being able to create something like a brush tool without the need to actually implement any drawing or gesture handling is amazing and helped to create a working prototype in no time. Tricky details like sprite transformations, remote loading of stickers or undo management were entirely handled by the SDK and we could completely focus on the interface itself.

For more details and some actual code, take a look at the repository. We documented all tricky parts and you should be able to start your own implementation using the code. Please keep in mind, that the code may target a slightly older PhotoEditor SDK version, so check for a new version before you start your journey.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Bringing Wide Color to PhotoEditor SDK

Sascha — Tue, 10 Jan 2017 00:00:00 GMT

Inspired by Mike Krieger’s great post on supporting wide color images in Instagram, I decided to challenge myself by introducing the same wide color support into our PhotoEditor SDK within a day.

Mike did an excellent job describing everything that is needed to support wide color images so I am not going to reiterate his explanation. Instead, here I’m going to share two findings I made in the process.

Image Export

Mike suggests that in order to preserve the color space while converting an UIImage into a JPEG, the UIImageJPEGRepresentation(_:_:) method has to be replaced with the new UIGraphicsImageRenderer.jpegData(withCompressionQuality:actions:) method. Checking the color profile of the generated JPEG by UIImageJPEGRepresentation(_:) I noticed that it seemed unnecessary and resulting in more work than needed. To verify this, I started Hopper and took a look at the internals of the new method. Here’s what it basically does:

It passes actions to UIGraphicsRenderer.runDrawingActions(_:completionActions:).
It obtains the resulting image using UIGraphicsImageRendererContext.currentImage.
It passes that image to UIImageJPEGRepresentation(_:_:).

As you can see, it makes use of UIImageJPEGRepresentation(_:_:) internally too, so there really is no need to go the extra mile. If you are actually doing any drawing within the actions block using wide color UIColors it absolutely makes sense to use Mike’s apoproach. But if you only want to convert an UIImage into a JPEG image, using the old method is far easier.

Core Image

We are using CIImage, CIFilter and CIContext for most of our processing. In our live preview we use a CIContext to directly render a CIImage into a GLKView and I’ve run into exactly the same problem as Mike has — namely not being able to get that view to be color space-aware. Using the offscreen buffer workaround that he suggested was going to be too much work though and also I didn’t want to sacrifice any performance.

After doing some research and digging further into the GLKit and Core Image assembly I found a nice solution. It looks like in order to render wide color images into OpenGL, OpenGLES 3 and a GLKViewDrawableColorFormat (that is currently not defined as an enum case) has to be used. The following code works like a charm for me:

let api: EAGLRenderingAPI
let colorFormat: GLKViewDrawableColorFormat

if #available(iOS 10.0, *) {
  if UIScreen.main.traitCollection.displayGamut == .P3 {
    api = .openGLES3
    colorFormat = GLKViewDrawableColorFormat(rawValue: 10)!
  } else {
    api = .openGLES2
    colorFormat = .RGBA8888
  }
} else {
  api = .openGLES2
  colorFormat = .RGBA8888
}

let context = EAGLContext(api: api)
let previewView = GLKView(frame: CGRect.zero, context: context!)
previewView.delegate = self
previewView.drawableColorFormat = colorFormat

This code creates a GLKView that is backed by a CAEAGLLayer with the following drawableProperties set:

[
 "EAGLDrawablePropertyRetained": 0,
 "EAGLDrawablePropertyColorFormat": EAGLColorFormatRGBA_XR10_64BPP
]

Unfortunately EAGLColorFormatRGBA_XR10_64BPP also doesn’t seem to be documented at this point, but my guess is that setting this color format on Instagram’s EAGLView would also solve their problem of making the view color space-aware.

Thanks to Core Image we can just switch to OpenGLES 3 without having to do any other modifications to our code. However, I am currently not sure if this is considered a private API use. I would greatly appreciate if anyone could shed some light on this.

I have not yet been able to test this on any device other than the iPhone 7 Plus, but I will update this post if I run into any problems with future tests.

Final Thoughts

Getting an app to support wide color images is definitely worth it and not as difficult as I had imagined. Unfortunately, it currently looks like iOS is still missing (public) support for wide color in some parts of its frameworks. I hope that Apple addresses these problems soon. Nevertheless we’re optimistic that we’ll be able to include wide color image support in the next release of our SDK. Feel free to check out our demo app in the App Store which we will update as soon as possible.

Swift – IMG.LY Blog

Building Custom Variable SF Symbols in Figma for our CreativeEditor SDK for iOS

What Are Variable SF Symbols?

The Early Struggle

Why We Wanted “SF Symbol Generator”

Inside SF Symbol Generator

Our Step-By-Step Workflow

Importing into the SF Symbols App

Optional Extra Step: Preparing More Complex Symbols

Lessons Learned

Conclusion

Working with Large Video and Image Files on iOS with Swift

Writing an Image to Disk

Compression Options for JPEG

Compressing Data with zlib

Transferring Data With a Server

Uploading a Large File

Configuring Background Transfers

Resizing an image

Resizing a Video

Going Further

Wrapping Up

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

How to Make an Animated GIF Using Swift

Animated GIF Basics

The Basic Strategy

Creating an Empty File

Adding Images

Finalizing the File

Collecting Frame Images

Extracting Images from Video

Wrapping Up

How to Make Videos from Still Images with AVFoundation and Swift

The Problem with an Image

Writing Buffers using AVAssetWriter

Create the Pixel Buffer

Configure AVAssetWriter

Write the pixel buffer to the video file

Creating a Blank Video

Adding a Composition

Going Further

How to Merge Videos in iOS with Swift

Making a Composition

Adding Video Clips

Using the Composition

Going Further

Wrapping Up

How To Add Text to Video in Swift

Setting up

Creating a Video Composition

Generating a Text Image

Displaying The Finished Video

Going Further

Wrapping Up

How to Trim and Crop Video in Swift

Anatomy of a Video File

Cropping Video to a Rectangle

Trimming the Time of a Video

Exporting the Video

Going Further

Wrapping Up

How to build Instagram’s Story Editor in a Day

Architecture and Overview

EditViewController

Sprite Handling

StickerViewController

BrushViewController

TextViewController

The Result

Bringing Wide Color to PhotoEditor SDK

Image Export

Core Image

Final Thoughts

Compressing Data with `zlib`

Writing Buffers using `AVAssetWriter`