Walter – IMG.LY Blog

Cutting Through The Jungle: An In-depth Review of Cloud GPU Providers to Train Your AI Models in 2024

Walter — Mon, 22 Apr 2024 06:35:54 GMT

Navigating the World of AI Models Hosting

Here at IMG.LY, we recently dug into finding the best place to host AI models to support apps we’re dreaming up. We wanted to figure out if using cloud GPUs or going serverless would work better for us. As we were looking specifically for service providers to run Image Generation Workloads on, we focused on those that could be the best fit for that. Along the way, we picked up some cool insights and ran into a few hiccups. We think sharing our journey and the things we figured out could help you when you’re looking to deploy your own AI models.

First off, we’ll explain what cloud GPU and serverless hosting really mean. Then, we’ll chat about their good and not-so-good sides when it comes to hosting AI models. It’s super important to make sure whatever hosting you choose fits your model like a glove. We’ll talk about some tools we stumbled upon that could help with that. Next up, we’ll give you a peek at some of the providers we checked out and our thoughts on how they might fit with what we’re working on. We decided to skip over the big names like IBM, Google, and Amazon this time. We were curious about what the newer, smaller companies have to offer.

To wrap things up, we’ll share some final thoughts on all our research. Plus, we’ll throw in some tips and ideas you might want to think about when you’re doing your own digging. Whether you’re developing AI models or planning to host some of the well-known ones, we hope our adventure helps you nail down the perfect hosting solution for what you need. Ready to jump in?

Kinds of Cloud Hosting for AI Models

Cloud hosting has been around for as long as there has been a cloud. Though the server hardware is not at your location, earlier versions of cloud hosting required that your team learnt lots about server infrastructure. As things have evolved, providers now manage the infrastructure so that you can focus on your work. You can now host even just a single function in the cloud, if that’s what you need. In our research, we looked at general serverless hosting and at Cloud GPU AI providers.

Serverless Hosting

Serverless hosting can be defined as an architecture model that lets developers build and run applications and services without managing the servers they run on. The cloud provider manages things like security, provisioning, scaling, and connectivity.

In a serverless CPU-loads hosting the host provisions your services to the most appropriate and available hardware. However, with most of the providers of GPU loads you get to choose.

Serverless Pros:

Pay-per-compute model: you only pay for the compute time you consume.
Autoscaling: the provider will automatically scale up or down depending on load, from a few requests a day to thousands per second.
No server management: eliminates the need for developers to also understand server infrastructure. Often, just a Docker image holding an application is sufficient.

Serverless Cons:

Cold starts: instance deallocates after a certain idle time (enabling the great pay-per-compute model) so initial request after this can be noticeably slow.
Limited control over specifics: certain GPU hardware or even server hardware may be unavailable at times which can impact performance.
Limitations on time - there may be limitations on the execution time of functions, which can impact long-running processes.

Cloud GPU Hosting

Cloud GPU hosting provides access to GPU and TPU (Tensor Processing Unit) hardware that can perform the parallel operations essential for AI model training and inference. The provider allows users to configure specific hardware for their jobs.

With cloud GPU each service or model gets its own GPU while running. Your other services communicate with the model through an API.

Cloud GPU Pros:

High performance: GPUs are specifically designed to run AI models and other tasks like deep learning and complex simulations.
Full control of hardware: users can specify specific hardware configurations for their projects.
Persistent availability: resources are not deallocated, so there is no latency for provisioning for the first request.
Cost-effective experiments: the upfront cost of purchasing GPU hardware to experiment with different configurations is eliminated. Services are priced with a pay-as-you-go model.

Cloud GPU Cons:

Costs over time: costs do not go down during periods of low demand. Over time, costs can potentially surpass the cost of investing in local hardware.
Management overhead - managing and optimizing hardware configurations is not automatically part of the hosting. You’ve got to learn some server administration and manage security and upgrades.

Providers

It’s important to understand that this isn’t a ranking of the best providers or an endorsement. It’s what we discovered with some web searching, reviewing the available documentation, and tinkering with any demo or free tools and models the provider makes available. The list could easily have been different providers and we think some of the pros and cons and qualities would be the same. Hopefully, some of the questions we raise and the pros or cons we noticed in our research can help you to guide your research.

Our goal was to find potential hosts for various workflows with different models in a scalable manner. We want to be able to build applications around the workflows. Some of our, specific, requirements include:

Autoscaling, ideally out-of-the-box without the need for custom Kubernetes setup or similar technologies.
Minimal vendor lock-in.
Compatibility with various technologies (REST API, WebSocket, Webhooks, etc.).
Support for Windows Server.

With those disclaimers and caveats, here is a short summary of our research.

Provider	Best For
Runpod IO (Serverless)	Deploy AI models with GPU support and require customizable API interfaces.
Vast AI (Serverless)	Affordable GPU resources and a variety of GPU options for AI model training.
Paperspace (Serverless)	Flexible workflows and support for different stages of AI model development.
CoreWeave (Serverless)	Strong knowledge of Kubernetes and need autoscaling capabilities for AI workloads.
Modal (Serverless)	Comprehensive documentation and examples for deploying AI models in containers.
ComfyICU (Serverless)	Serverless infrastructure tailored for hosting ComfyUI applications.
Replicate (Serverless)	Easy-to-use API for executing AI tasks without managing infrastructure.
Genesis Cloud (Cloud GPU)	Sustainability and need scalable GPU instances for AI model training.
Fly IO (Cloud GPU)	To deploy complete applications with GPU support in a scalable environment.
Runpod IO (Cloud GPU)	GPU resources in various regions and require customizable Docker-based deployments.
Lamda Labs (Cloud GPU)	On-demand GPU resources for model training and inference tasks.
Together AI (Cloud GPU)	A platform for testing serverless models and occasional access to GPU clusters.

If you want to skip ahead to a specific part, here are the providers we will be diving into:

Serverless Providers
Runpod IO (Serverless)
Vast AI
Paperspace
Banana Dev
CoreWeave
Modal
ComfyICU
Replicate

GPU Cloud Providers
Genesis Cloud
Fly IO
Runpod IO (Cloud GPU)
Lamda Labs
Together AI

Serverless Providers

Runpod IO (Serverless)

Runpod IO

Concept:

A Docker image that includes the installation of Python + GPU packages, models, and ComfyUI.
Python/Go handlers act as an API interface to ComfyUI, which is vendor-specific, but can be wrapped in a more general API for reuse. For more information, see this article on hosting a ComfyUI workflow via API.

Pros:

Good documentation, including public GitHub repositories with examples.
Relatively large community for a new provider.
Compatibility with Windows Server.
Handlers allow for webhook and WebSocket-like communication for API feedback.
Network volume to store models/data and reduce cold start times.
Control over the number of workers and the ability to define persistently active workers.

Cons:

Availability of GPUs, especially in Europe, needs to be validated.
Handlers can only be written in Python and Go.

Open Questions:

General open questions regarding serverless infrastructure and AI inference tasks.

Conclusion:

The overall package seems very mature. The setup can largely be adopted from the GitHub examples. Good documentation and community support (notably on Reddit). The open questions regarding pricing and cold starts are typical for serverless infrastructure.

Vast AI

Concept:

Peer-to-Peer Sharing. Companies/organizations can rent out their unused GPUs.
A GPU Marketplace approach.

Pros:

Affordable prices through their peer-to-peer GPU sharing model.
A wide selection of different GPUs.
Good global availability of GPUs.
Ability to define autoscaler groups, allowing different workflows to scale differently.

Cons:

The autoscaler is currently only in beta mode.
Data privacy/security concerns when renting GPUs from anonymous providers.

Open Questions:

How will the autoscaler beta evolve?
Control over GPU providers: Can one allow only certain trusted providers (e.g., those based in the EU)?

Conclusion:

Even though the pricing is more affordable, there may be significant issues, in terms of security and data protection, as well as the fact that the autoscaler is still in the beta phase.

Paperspace

Concept:

The serverless approach (Workflows or Gradient) is still in beta Paperspace Gradient Workflows is based on Argo Workflows which utilizes Kubernetes.
A predefined API is available for communicating with workflows, as detailed in DigitalOcean’s documentation for Paperspace commands.

Pros:

The ability to use different machines (GPUs) at different stages of a workflow.
Provided by Digital Ocean, allows for general hosting customers to expand into GPU hosting without finding a new vendor.
Possible Windows support as outlined in DigitalOcean’s documentation on running Windows apps.

Cons:

Complex documentation: offers many features for various use cases (AI learning, data preparation, validation, and inference).
Vendor lock-in through a proprietary system: Gradient Workflows and YAML config are specific to Paperspace.
No real-time feedback over the API.

Open Questions:

Since it’s still in beta, how will the ecosystem continue to develop?
How extensive is the knowledge of Kubernetes required to implement autoscaling?

Conclusion:

It’s positive that it’s offered by Digital Ocean as they are a more mature company with general hosting experience. The approach seems very specific to Digital Ocean. Furthermore, it may require experience with Kubernetes.

Banana Dev

It has been excluded: Recently, they announced the termination of their serverless model as it was not cost-effective.

Learning from this: Currently, there are many new providers entering the market aiming to establish themselves as cloud GPU or serverless GPU providers. This highlights the importance of minimizing vendor lock-in.

CoreWeave

Concept:

Heavily based on Kubernetes.
- A Kubernetes file is created for setup; scaling and additional infrastructure are managed by Core Weave.

Pros:

Autoscaling by default with the possibility of scaling to zero.
Supports Windows.
Minimal vendor lock-in due to Kubernetes configuration.

Cons:

Strong dependency on Kubernetes, with the serverless setup based on KNative documentation.
Does not offer a handler API, etc., to communicate directly with ComfyUI.

Open Questions:

How complicated would it be to implement an API interface and resulting scaling to address the correct instances, etc.

Conclusion:

Good documentation and a close interface to Kubernetes. For a team with strong knowledge of Kubernetes, this could be a prime candidate.

Concept:

Container Setup: Containers are defined through Modal’s own container setup Modal custom container documentation.
- Docker images can also be used.
Modal-specific handlers to communicate with ComfyUI and other models.

Pros:

Supports webhooks and custom endpoints Modal webhooks documentation.
Focus on fast startups/cold starts.
Emphasis on AI inference tasks.
Comprehensive documentation with many examples.

Cons:

Vendor lock-in if Modal’s container setup is used.
Autoscaling and scaling configuration are not directly described.

Open Questions:

How exactly does the autoscaling work?

Assessment:

For us, this is a candidate for closer consideration. The container setup can be managed through Dockerfiles, and the API defined by Modal’s own interface.

ComfyICU

Concept:

Pure focus on ComfyUI, serverless infrastructure.
API interface for communication.

Pros:

Minimal setup effort.

Cons:

Limited control over the API.
Limited GPU resources.

Open Questions:

How does the autoscaling work, if it exists at all?
Community-based open source. What is the long-term support for this project?

Conclusion:

Potentially useful for testing or building a demo site, but probably not suitable for developing our commercial applications.

Replicate

Concept:

Execution of AI tasks/models in the cloud via an API.
No access to infrastructure, etc.

Pros:

Supports various languages: Node, Python, Swift.

Cons:

No control over the infrastructure, number of GPUs, or workers.
API rate limits.

Open Questions:

How can autoscaling be enabled?
Is it possible to create custom API endpoints, webhooks, websockets?

Conclusion:

For testing or as a demo for one’s own model, this can be a very good platform. However, as a standalone application interface, it doesn’t meet some of our core requirements.

GPU Cloud Providers

Genesis Cloud

Concept:

Focus on sustainability and renewable energy.
Scaling through instances as detailed in here.

Pros:

A REST API is available for managing instances.

Cons:

The availability of GPUs varies significantly by region.
Limited selection of GPUs.

Open Questions:

How quickly can new instances be scaled up or down?

Conclusion:

The use case for Genesis Cloud appears to be more suited for model training or tasks that require a significant amount of computing power for extended periods.

Fly IO

Concept:

Focus on the deployment of complete applications.
Also offers its own GPU servers.

Pros:

Docker File support with additional configuration via a TOML file.
Quick scaling of GPUs up or down facilitated by the launch process.

Cons:

Limited selection of GPUs, with only very large GPUs available.
Specifically tailored for Linux.

Open Questions:

How well does the launch system perform for relatively fast inference tasks?

Conclusion:

Since primarily large GPUs are available, the focus here also appears to be more on model training or other long-duration tasks. However, the launch system might also potentially be used for inference.

Runpod IO (Cloud GPU)

Runpod IO

Concept:

A wide range of GPUs available across various regions.
Base Docker images for popular tasks or support for custom Docker images.

Pros:

Many different data center regions.
A variety of CPUs available.
Simple setup via Docker images.

Cons:

No direct autoscaling (would need to use Runpod Serverless for that).
Despite a large selection of GPUs and many different data center locations, the availability of GPUs is not very high.

Open Questions:

Can autoscaling be implemented without using serverless?

Conclusion:

The setup can largely be adopted from the GitHub examples. There is good documentation and a community (much of it on Reddit). The availability of GPUs could become a problem, especially for smaller GPUs.

Lamda Labs

Concept:

On-demand cloud with a focus on model training and inference.
Similar concept to Runpod, offering a variety of GPUs.
- GPU availability is very limited.

Conclusion:

Runpod and Lambda Labs seem to have a similar approach and similar offerings. Runpod appears to have greater availability.

Together AI

Concept:

Offers an API and playground for testing serverless models.
Also offers GPU clusters but only upon request.

Conclusion:

We didn’t dig into the GPU clusters since information is available only upon request. Otherwise, in the API/serverless area, it appears to be similar to Replicate.

Established Providers

As we said in the introduction we did not examine the old, large providers like Google Cloud, AWS, Azure, Nvidia, etc., in detail. Rather, we focused on the new providers aiming specifically at the market segment of AI GPUs. With the older providers, we are more in the realm of cloud GPUs and less in serverless. Given the size of these providers and the wide range of market segments they cover, it can make sense to opt for them if one is already familiar with their architecture and documentation.

Google Cloud Platform (GCP)
AWS
Microsoft Azure
IBM Cloud
NVIDIA GPU Cloud (NGC)

Conclusion

Just as we saw that performance can vary wildly for different models, pricing can be similarly complex. When evaluating costs, consider factors like response times, the number of required workers, and potential charges for features like caching. Many providers offer detailed pricing guidelines on their websites, which can be crucial for ensuring you only pay for the computing power you truly need. Experimenting with performance of your model and applications during development will be helpful to make sure your hardware and pricing are both optimized for your application.

Another thing to consider is what kind of experience does your team already have? Most cloud GPU services provide tools like CLI or REST APIs to manage resources, which can be a steep learning curve if your team is not familiar with these technologies. Additionally, while serverless platforms may support multiple programming languages, compatibility with your team’s preferred language—be it JavaScript, Python, or Go—is essential. As exciting as it can be to learn new languages, it’s probably not the best use of your team’s time.

The size of files you’ll be moving between your model and the other parts of your project may also be a factor. Your users may not notice latency for models that communicate using text only. Text moves quickly from point to point in a network. However, if your model takes large image files as input or output, you may find that moving data between data centers is too slow. Then you’d want to focus on providers who can offer more general hosting in addition to cloud GPU hosting.

As we continue to research this for our own projects, we are thinking the best configuration for us is to use a cloud GPU exclusively for generation tasks and communicate with it via an API from our existing back end. We will have to experiment to see if we can have those functions geographically separate, or if we need to find one hosting company and one data center for both. As we learn more we may change our ideas, but that’s part of the fun of working in technology, things change. By using the higher-cost cloud GPU for as few tasks as possible, we’ll know we aren’t wasting compute power for things easily handled by a general CPU.

We hope this has given you some useful background and ideas as you research hosting options for your AI projects. Understanding the subtle differences between serverless and cloud GPU hosting can spark innovative ideas tailored to your needs. Perhaps some of the lesser-known providers we’ve explored might just be the perfect fit for your next project. As always, the dynamic nature of technology keeps us on our toes—ready to adapt and evolve. Happy hosting!

Thanks for reading. Join over 3000 specialists with powerful apps and subscribe to our newsletter. We keep you in the loop with brand-new features, early access, and updates.

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Walter — Thu, 30 Mar 2023 11:20:26 GMT

Now that TikTok is facing a ban in the US any day now, we better wait in the wings with an alternative ready to go and scoop up those millions of hobby dancers, micro bloggers, and would-be influencers.

In this article, you’ll learn how to use Swift, SwiftUI, and the IMG.LY CreativeEditor SDK to build a simple iOS app for recording, editing, and viewing short videos. This app features views providing core functionalities similar to making a post in a video-sharing platform like TikTok.

The editing controller is the central hub. It allows users to capture video clips and enhance their videos by adding filters, stickers, music, and other effects. CreativeEditor SDK calls the overall project a “scene” and the parts (clips, stickers, text, etc.) “blocks”. Once the scene is ready, the user can compose it into a standard video file and export it. Finally, the playback controller showcases the finished projects using Apple’s standard VideoPlayer struct. For recording and editing, the CreativeEditor SDK’s Camera and Video Editor handle the heavy lifting, enabling swift development of a robust app. You can easily extend its features, filters, and creative options as your users demand.

Follow along to see how to capture video, enhance it, and export the finished product as a polished movie file.

Setting Up Your Project

You can try out the TikTok clone by cloning the GitHub repository supporting the article. Otherwise, follow this step-by-step tutorial and learn how to build the TikTok clone with CreativeEditor SDK by yourself.

To add the CreativeEditor SDK to an Xcode project, include it with your other package dependencies. Use the URL below to add the package, either to your Package.swift file or using File > Add Packages...

https://github.com/imgly/IMGLYUI-swift

The package includes a number of different frameworks:

IMGLYUI, an umbrella framework of all of the different editors and a camera.
IMGLYCamera, a standalone camera View. This is the same camera as the editors use.
IMGLYApparelEditor, IMGLYDesignEditor, IMGLYPostcardEditor, and IMGLYVideoEditor which are four different configurations to support different use cases.

Each of these frameworks uses the IMG.LY Engine for image processing. To include them in your project, navigate to the ‘Frameworks, Libraries, and Embedded Content’ section in your app’s General settings. Click the ’+’ button and select the desired frameworks. Initially, consider adding the IMGLYUI package, as it encompasses all other packages. Before releasing your app, review and include only the necessary components to minimize app size and avoid unused code.

Once Xcode downloads and resolves the packages, you just need to include IMGLYVideoEditor in any of the files that will reference the editor in your app. When you’re working with the underlying engine directly you’ll also want to include IMGLYEngine. You’ll only work with the engine as you start to customize the workflow.

Asking for Permissions

Before any app can access the phone’s microphone, camera, documents or photo library, the user must specifically give permission to record audio or video as well as access the user’s Photo library. If your app doesn’t secure these permissions properly before trying to use the camera the app will crash and also probably won’t get through Apple’s app review. In earlier versions of iOS you could develop an app on the Simulator without asking permissions, but always on the device the app will crash. In current versions of iOS the app crashes regardless of platform. The dialogue requesting permissions is a system dialog and looks like this:

You do not have the ability to change this dialog. You can supply the reason you are asking for the permission using a key in the info.plist file in the Xcode project. In the example above “Lets you record videos” is in the info.plist. For the video camera you will have to ask for video and audio permission. The first step is to add a short message to the user in the info.plist.

For video the key to add is NSCameraUsageDescription and for the microphone you need to add NSMicrophoneUsageDescription. Whenever your app attempts to use the camera, iOS will first check to see if the user has already granted access. If not, it will then display the dialogs using your entries in the info.plist or crash if you have not provided entries. The user might be surprised by these dialog boxes and may accidentally tap Don't Allow if they are trying to quickly launch the camera. It is better practice to set up some kind of onboarding view and secure the permissions before you need them. You might have a view that explains what you are going to ask for and then displays the permission dialog.

switch AVCaptureDevice.authorizationStatus(for: .video) {
  case .authorized:
    //the user has authroized permission!
    break
  case .denied:
    //the user has previously denied permission!
    //perhaps we should ask them again
    break
  case .restricted:
    //the user cannot grant permission!
    break
  case .notDetermined:
    //the user has never been asked for permission
    //so let's ask them now
    AVCaptureDevice.requestAccess(for: .video) { accessGranted in
      if accessGranted {
        //the user has authorized permission!
      } else {
        //the user has denied permission!
      }
    }
  @unknown default:
    break
}

The snippet above lets your app read the permission status for the video camera and ask for permission if it has not been granted. To request access to the microphone pass .audio to the AVCaptureDevice.authorizationStatus(for:) and AVCaptureDevice.requestAccess(for:) functions. The AVCaptureDevice.requestAccess(for:) command is what displays the system dialog actually requesting access. The accessGranted variable reports back to your app what the user chose. You can take care of getting the permissions any time but when the CreativeEditor SDK first launches it will attempt to access the camera, so the dialog will appear if the user has not already granted permission.

The .restricted case is fairly new. In this case, there are policies on the phone that prohibit the user from granting permission to your app. In addition to asking permissions during onboarding, it is good practice to check for permission every time before you attempt to run any code that uses the camera. The user can change permissions using the Settings app on their phone at any time. If the user has denied permissions and you present the video camera anyway, your app will record black video frames and silence on audio tracks.

In addition to asking for camera and microphone permissions, your app will probably want to access the photos on the user’s phone. You will need to add an entry to the info.plist for NSPhotoLibraryUsageDescription. As with the camera and the microphone, the dialog will appear when the Video Editor first attempts access, but you can give your user some ability to grant permission during onboarding. For the user’s photos, you can check the authorization status using

let status = PHPhotoLibrary.authroizationStatus(for: .readWrite)

As with the camera, the photo library has PHPhotoLibrary.requestAuthroization(for: .readWrite) but instead of just returning a “granted” or “denied” status, this command returns the actual new status. In addition to the status values for the camera, the photo library may return a .limited status meaning that the user has granted permission to only some of the photos. If the user has chosen to share only some of their photos, Apple provides some views you can present so that the user can manage the photos. Any videos that your app saves to the user’s photo library will always be available when the user has chosen .limited. You can read more about how to work with permissions and the user’s photo library by reading this Apple article Delivering an Enhanced Privacy Experience in Your Photos App.

Making Video Recordings

The start of a great TikTok is the camera. When our user wants to make a new video clip with our app, they tap on the button at the bottom of the initial screen to start the creation workflow.

Here there is a decision to make. TikTok and the CreativeEditor SDK can each start with the creation controls and a video preview. When using the CreativeEditor SDK though, you can also start with some video that came from somewhere else. So if you already have some camera code you like, or if you want to start with some video clip from somewhere else, you can insert that into the editor when it launches. You can do that by leveraging the onCreate modifier of the editor.

let engine = EngineSettings(license: "<your license code>",
                              userID: "<your unique user id>")

VideoEditor(engine)
  .imgly.onCreate({ engine in
    try await engine.scene.create(fromVideo: Bundle.main.url(forResource: "dog_water", withExtension: "mov")!)

    //set other defaults
  }

In the code above, the app initializes an instance of the Video Editor view and then reads the dog_water.mov file from the app bundle. launches the editor with that.

Whether starting with a video or starting with a blank canvas a user can add video clips to their creation using the embedded camera.

The image above compares two different camera controllers: the left one from the TikTok app and the right showing the default settings of the MobileCamera, either standalone or within a Video Editor scene. Key controls in each are marked with blue numbers:

1 - Flip camera button
2 - Flash button
3 - Recording time indicator
4 - Add video and finish buttons
5 - Cancel button

While the TikTok camera integrates several editing functions during video capture, the Video Editor separates these tasks, focusing the camera purely on recording. Both systems allow users to start and stop video recording to compile a series of clips before moving to the editing phase.

Configuring the Camera

The camera used inside of the Video Editor UI isn’t customizable. You have limited options to customize when you use the standalone camera though. You can set the colors of the buttons, helpful if you have a preferred color scheme, and you can set a maximum duration for the video recording allowed.

let config = CameraConfiguration(
  recordingColor: .orange,
  highlightColor: .blue,
  maxTotalDuration: 30,
  allowExceedingMaxDuration: false)

Camera(engine, config: config) { result in
 //handle the recorded video collection
}

The code above sets the record button to orange while recording and the clip view to orange. There is a maximum duration of 30 seconds for all videos in the collection and the button to save the clips highlighted in blue.

If your app requires further customization, you’ll probably want to make your own camera view and work with the IMG.LY Creative Engine directly.

The documentation website is a great resource for understanding how much you can add to and customize the IMGLYUI resources and when you’ll need to drop down to the level of the engine.

Presenting the Controller

As demonstrated above, both the Video Editor and the Camera are View structures that can be presented directly or through overlays such as overlay or fullScreenCover. However, they are typically enclosed in a NavigationView to utilize the navigation bar for displaying export and other buttons. Therefore, when triggering the Video Editor from a button, it’s crucial to embed it within a NavigationView.

Button("Edit the Video") {
  isPresented = true
}
 .fullScreenCover(isPresented: $isPresented) {
   NavigationView {
     VideoEditor(engine)
      //any setup modifiers
   }

Building Your Creation

After capturing a video, the app displays an editor to complete the creation. The editing tools are in the red circles. TikTok provides about ten different tools on the editing screen. Additionally, TikTok provided some editing tools on the capture screen. The CreativeEditor SDK apps provide a scrolling set of asset libraries along the bottom for items to add to the scene. But when any of the elements of the scene, like your video or an audio clip, are highlighted, the bottom row becomes tools to manipulate that element. The basic Asset Library types are uploads, videos, audio, images, text, shapes, and stickers.

How to configure and customize each of these tools is beyond the scope of this tutorial. But, a lot of the customization of the assets, fonts, blurs and even filters can be done without code. By design, the CreativeEditor SDK downloads assets from a central server. This is a great way for you to be able to push updated filters, stickers, and other assets to your users without going through the app update process. In the next section, we’ll walk through that and provide some links for further research.

Configuring the Editors Assets

The IMG.LY Engine looks for assets for stickers, vector shapes, LUT filters, color filters, color palettes, blurs, and fonts using a URL by default. The design of the system is that you would have your own server supporting your app. However, assets in the app bundle are also accessible by URL, so as long as the assets are in the folder format that the engine expects, it doesn’t matter whether they are local or remote.

In the documentation for this process you can find a URL to the default asset bundle. Once you download that bundle you can add it to your app. The bundle is structured much like a regular iOS Assets.catalog with some JSON containing metadata about the asset and a file that provides the actual asset. For some of the assets like filters and vectors, no file is needed as everything can be defined in the JSON.

In the image above, you can see the JSON to describe the ape sticker and the emoji_ape image file after the IMGLYAssets bundle has been added to our app. You can edit these assets and add your own.

Once you’re done editing the asset bundle, you can use it instead of the default bundle during the onCreate modifier like this:

VideoEditor(engine)
  .imgly.onCreate({ engine in
     try await engine.scene.load(from: VideoEditor.defaultScene)
              // Add asset sources
     let bundleURL = Bundle.main.url(forResource: "IMGLYAssets", withExtension: "bundle")!
     try await engine.addDefaultAssetSources(baseURL: bundleURL)
     try await engine.addDemoAssetSources(sceneMode: engine.scene.getMode(),
       withUploadAssetSources: true)
     try await engine.asset.addSource(TextAssetSource(engine: engine))
  })

In the code above, we create a blank scene using VideoEditor.defaultScene static method. Then set a URL to point to the bundle in the app instead of a remote bundle and make that the source of addDefaultAssetSources. Now our app will use the assets from the bundle.

Exporting the Finished Video

Once the user has finished with their creation, they tap the share button and the Video Editor composes all of the video, audio, filters, and static items into a single .mp4 file. Then it displays a share sheet. If you’d rather do something different, like save the creation locally so you can play it back, you can override this default behavior.

In your app, after the .onCreate modifier, you can define a .onExport modifier. When the user taps on the share button you can export the video and then save it to the documents directory to playback later.

In the documentation for modifiers you can find the source code for the Video Editor’s default behavior. Instead of duplicating it in detail here, we can just summarize and provide a little snippet for our change.

First, the .onCreate handler takes an engine and a eventHandler parameter. The engine is the underlying IMG.LY Engine object we’ve been using. The eventHandler lets us send information back to the Video Editor UI about the progress or any errors.

In the documentation, there is a helper function to export the video. After it runs it returns the data of the exported video and a mimeType, which will be .mp4.

let data: Date, mimeType: MIMEType
let mode = try mainEngine.scene.getMode()
guard mode == .video else { throw CallbackError.unknownSceneMode; return }
(data, mimeType) = try await exportVideo()

The code in the documentation handles static files from the DesignEditor as well as videos from the VideoEditor. The above code makes it so that only video can be exported, since our app only works with video.

In the exportVideo function, the code first checks to make sure it has a properly formed scene object to work with. Then it starts the export using mainEngine.block.exportVideo(_, mimeType:). This begins an async throwing stream during the export. The VideoExport objects in the stream are either .progress or .finished. When a .progress object arrives, the exportVideo function sends an update to the eventHandler to display in the UI. When the .finished object arrives, it has an associated object that is the data for the video.

Now instead of opening the share sheet, we can write the video to the documents directory for the app.

do {
      guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
        else {
          logger.error("Documents directory not found"); return
        }

  let filePath = documentDirectory.appending(component: UUID().uuidString)
  try data.write(to: filePath)
} catch (let error as NSError) {
  logger.error("could not write finished file: \(error.localizedDescription)")
}

eventHandler.send(.exportCompleted {})

The code above takes the data from the exportVideo function and writes it to the app’s documents directory, giving it a UUID string value as a filename. Then it updates the UI to let it know the export is done but sends an empty action block.

Setting Up a Playback Controller

For this example app, the playback controller will play any clips in the app’s Documents directory. Each clip plays on a loop. The user can swipe up to get the next clip and tap to pause or restart the clip. If there are no clips, the user will be encouraged to make a new one.

When the video player view appears, the first task is to see if there are any videos to load. Any video files in the Documents directory are assumed to be ready to play.

func loadVideos() {
  //get a handle to the documents directory for this app
  guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true) else { logger.error("Documents directory not found"); return}

  if let files = try? FileManager.default.contentsOfDirectory(at: documentDirectory, includingPropertiesForKeys: nil) {
    videos = files
  }
}

The code above reads all of the filenames from the documents directory into a files array and then assigns that array to a videos variable.

Each time the app is launched or the VideoEditor view is dismissed, the video player view will get the onAppear message. This is a good place to check for videos using the loadVideos function. After the videos are loaded, the app can start to play the first one.

func setupVideoPlayer() {
  guard let currentVideo = videos.first else {logger.error("No videos to play"); return}

  self.shouldHideEmptyDirectoryText = true

       let endMonitor = NotificationCenter.default.publisher(for: NSNotification.Name.AVPlayerItemDidPlayToEndTime)

  self.player = AVPlayerItem(url: currentVideo)
  self.currentVideo = currentVideo

}

In the code above, we first check to see if there are any video files. If there are, then hide the message about videos and create a new AVPlayerItem using the URL of the first video. The endMonitor, shouldHideEmptyDirectoryText, and player are both being watched by the View. The AVPlayer gets rendered in a regular SwiftUI View.

VideoPlayer(player: player)
  .frame(width: 640, height: 360)
  .onAppear {
    player.play()
  }
  .onReceive(endMonitor) { _ in
    player.seek(to: .zero)
    player.play()
}

When the video reaches the end, it will seek(to: .zero) which rewinds the video and loops it.

In the TikTok app, you can advance to the next video by swiping up. Additionally, you can start and stop any video by tapping on the screen. We can add both of those features using gesture recognizers.

The code for play and pause is attached to a Tap Gesture modifier we can append to the player.

.onTapGesture {
  if player.rate > 0.0 {
    player.rate = 0.0
  } else {
    player.rate = 1.0
  }
}

We can just adjust the playback rate. A value of 1.0 is a normal speed and 0.0 is paused. The player also has a .pause and .play methods, but sometimes these seem buggy.

Swipe is a little more difficult since SwiftUI only has swiping on List items. So we can use a drag gesture and then evaluate the translations when it’s complete.

.gesture(
    DragGesture(minimumDistance: 3.0, coordinateSpace: .local)
        .onEnded { value in
            // Check for a vertical, upward swipe with minimal horizontal deviation
          if abs(value.translation.width) < 100 &&
             value.translation.height < 0 {
                // It was an up swipe!
		viewModel.advanceVideo()
            }
        }
)

Then advancing the video is a simple matter of refreshing the player

func advanceVideo() {
  guard let currentVideo = self.currentVideo else { logger.error("No current video."); return}

  let currentIndex = videos.firstIndex(of: currentVideo)
  var nextVideo = videos.index(after: currentIndex!)
  if nextVideo == videos.count {
    nextVideo = 0
  }

  self.currentvideo = videos[nextVideo]
  self.player = AVPlayer(url: self.currentVideo!)

}

This code will get the next file URL from the array of videos. When it reaches the end, it will loop back and get the file at index 0. Then it creates a new player. Because the view is observing that player variable, it will update with the new video.

With this view, the user can create new videos by tapping the creation button and swipe to view all of their created videos.

Where to Go From Here

This tutorial has focused on how the CreativeEditor SDK can help you quickly make a video creation app like TikTok. Good next steps would be to further customize the editing tools and build out the network for sharing and tagging videos. Something that is important to consider is the data structure for each video. The iOS system is optimized to read only as much of a video file off of disk as is needed at any moment. Your app should use those optimizations to run faster. So, don’t load the whole video into memory as a Data object. Your data structures should keep the large video files somehow separate from the much smaller metadata (likes, comments, etc.). Consider storing the URL or filename of the video in the same object as the likes and comments. This will also allow you to cache video files after they have been downloaded so that you don’t need to redownload them when other data such as comments or number of likes changes.

Thanks for reading! We hope that you’ve gotten a better understanding for how a tool like the CreativeEditor SDK can bring your ideas to market faster. Feel free to reach out to us with any questions, comments, or suggestions.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

To stay in the loop, subscribe to our Newsletter or follow us on LinkedIn and Twitter.

Build a Simple Real-Time Video Editor with Metal for iOS

Walter — Tue, 30 Aug 2022 20:06:49 GMT

In this tutorial you’ll see how to extract frames from live camera streams, regular movie files and streamed movie files and display them on a MetalKit View. Using Metal allows you to have a great deal of control over how the pixels are rendered and ensures that the GPU is used for rendering, which keeps from slowing down the CPU.

The tutorial code will always convert the video frames into CIImage objects before display. This is to help you adapt the code to your own applications. Applying filters, resizing and other effects are fast and easy when working with CIImage. Additionally, as long as you set your CIContext to the GPU, you can be sure that your code uses the GPU and the CPU efficiently.

When working with Video, Apple provides a number of frameworks that operate at different levels. For the higher level frameworks such as AVKit or CoreImage, the system determines when to use the GPU and when to use the CPU for rendering and computation. The higher-level frameworks also offer a tradeoff between ease of use and granularity of control. If you want to have direct access to tell the GPU how to render every pixel, then you will want to use Metal. Remember, though, that you are now responsible for keeping video and audio in sync, encoding and decoding data and determining a reasonable UI for your user. For many use cases, using a higher level framework is probably a better choice. Apple’s engineers have worked to ensure the graphics frameworks use the GPU and CPU in reasonable ways. However, when you need to use Metal, you you need it. So, let’s get started.

Setting Up a MTKView

An MTKView is a subclass of a UIView so it has a frame and bounds and other properties. Its drawing and rendering is backed directly by drawables and textures rendered on the GPU, so drawing to it can be quite fast. In addition to displaying video, you can render 3D model objects and graphics shaders, so for a graphics rich application it can be a powerful tool.

Before you can send data to the view it needs to be configured. MetalKit Views are still heavily influenced by UIKit, so if you are working in a SwiftUI project, you will need to wrap them in ViewRepresentable code.

//MetalKit Variables
@IBOutlet var displayView: MTKView!
var metalDevice : MTLDevice!
var metalCommandQueue : MTLCommandQueue!

//CoreImage Variables
var ciContext : CIContext!
var filteredImage: CIImage?
var cleanImage: CIImage?

//get a reference to the GPU
metalDevice = MTLCreateSystemDefaultDevice()

//link the GPU to our MTKView
displayView.device = metalDevice

//link our command queue variable to the GPU
metalCommandQueue = metalDevice.makeCommandQueue()

//associate our CIContext with the metal stack
ciContext = CIContext(mtlCommandQueue: metalCommandQueue)

You will need to get a reference to the GPU of the iOS device. At least for now, iOS devices only have one GPU. Then you will need to link the displayView and the metalCommandQueue to the metalDevice. Finally we will link the ciContext to the GPU so that all of our image manipulation code will run in the same place.

//tell our MTKView that we want to call .draw() to make updates
displayView.isPaused = true
displayView.enableSetNeedsDisplay = false
//let it's drawable texture be writen to dynamically
displayView.framebufferOnly = false

//set our code to be our MTKView's delegate
displayView.delegate = self

In the code above, we set the MTKView to be in a isPaused state and do not enable setNeedsDisplay. This will ensure that it will only redraw when we explicitly tell it to redraw using a .draw() method in our delegate. By setting framebufferOnly to false you are telling the view that you will be writing to it multiple times and may also read from it.

Drawing in an MTKView

Now the MTKView is configured. The next step is to update the delegate methods. The MTKViewDelegate has two methods. If your code needs to respond to the view dimensions changing (to support device rotation, or if you want the user to be able to resize the window) make your adjustments in func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize). For this example we are only interested in the delegate method: func draw(in view: MTKView). The code below draws a CIImage to the MTKView. When showing video, we can extract each frame of the video as a CIImage

//create a buffer to hold this round of draw commands
guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
  return
  }

//grab the filtered or a clean image to display
guard let ciImage = filteredImage ?? cleanImage else {
  return
  }

//get a drawable if the GPU is not busy
guard let currentDrawable = view.currentDrawable else {
  return
  }

//make sure frame is centered on screen
let heightOfImage = ciImage.extent.height
let heightOfDrawable = view.drawableSize.height
let yOffset = (heightOfDrawable - heightOfImage)/2

//render into the metal texture
self.ciContext.render(ciImage,
             to: currentDrawable.texture,
  commandBuffer: commandBuffer,
         bounds: CGRect(origin: CGPoint(x: 0, y: -yOffset),
                          size: view.drawableSize),
     colorSpace: CGColorSpaceCreateDeviceRGB())

//present the drawable and buffer
commandBuffer.present(currentDrawable)

//send the commands to the GPU
commandBuffer.commit()

For each pass of the draw method, we will create a new MTLCommandBuffer then we will get the MTKView’s .currentDrawable which contains a texture we can send pixel data to. The texture will have a size and a color space. Though we are using Metal to render images to the screen, Metal can be used to send any valid commands to the GPU for things like computations. Once the buffer has been filled with commands it can be assigned to the drawable and committed. When the buffer is committed, the GPU will execute all of the commands. If another .draw() gets called while the buffer is being executed, the GPU will not interrupt the current buffer in order to start the new one.

The bounds of the render allow you to move where the CIImage gets displayed in the MTKView and resize the CIImage within the view. This can be valuable when compositing multiple images or video streams onto the same MTKView. Remember that unlike a UIView the origin point is the bottom left of the texture and CIImage.

Since iOS 11 Apple provides a new lighter weight API for sending renders to the buffer. Use whichever form makes sense to you.

//render into the metal texture
let destination = CIRenderDestination(mtlTexture: currentDrawable.texture,
                                   commandBuffer: commandBuffer)
do {
 try self.ciContext.startTask(toRender: ciImage, to: destination)
} catch {
 print(error)
}

The CIRenderDestination has some other optional parameters such as height and width but will always display with an origin of 0,0. So, unlike the earlier call, if you need to rotate or change the dimensions of the image, you will need to do it to the CIImage earlier in the code. However, if your app always displays the video at full size in the MTKView this may be easier. Also the startTask call will return immediately, instead of waiting for other tasks on the CPU to complete.

It is important to remember that you can have any number of renders in the commandBuffer before the calls to .present and .commit. This means that the same MTKView can display video from multiple streams as well as static images or animations.

Working with Local Files

In order to extract individual frames from a local file, we can use an AVAssetReader to get pixel data to render in the MTKView

When using higher-level frameworks, a quick way to display a video file for playback is to load it into an AVPlayer. However, an alternative is to use an AVAssetReader to read the tracks and the individual frames from the video tracks.

let asset = AVAsset(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)
let reader = try! AVAssetReader(asset: asset)

guard let track = asset.tracks(withMediaType: .video).last else {
  return
  }

let outputSettings: [String: Any] = [
        kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32ARGB
    ]
let trackOutput = AVAssetReaderTrackOutput(track: track, outputSettings: outputSettings)
    reader.add(trackOutput)
    reader.startReading()

It is important that the outputSettings of the reader match the image format of the video track. If there is a mismatch the color may be off or the reader may be unable to extract a usable buffer. If your code is generating empty or black buffers, the outputSettings are the first place to troubleshoot. Once the reader starts reading then it can extract pixel buffers. You can use a while loop to get all of the buffers from the track and send them to the MTKView for display.

var sampleBuffer = trackOutput.copyNextSampleBuffer()
while sampleBuffer != nil {
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer!) else {
   return
  }
  print(track.preferredTransform)
  print(sampleBuffer?.outputPresentationTimeStamp)

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()

  sampleBuffer = trackOutput.copyNextSampleBuffer()
  }
}

In the code above, use CMSampleBufferGetImageBuffer to ensure that we have a valid image. Then assign the image to a CoreImage and execute the .draw method of the MTKView. Afterwards, get the next sample buffer. This will continue until the end of the track, when .copyNextSampleBuffer() will return nil.

In the code above, there are two print statements to note some valuable information that your app may want to store for later use. Remember, when working with buffers and the GPU directly, much of the higher-level metadata about the video is lost, so you’ll need to keep track of it in your code if you want to use it later. An image may be rotated because of the way it was originally generated in the video. The preferredTransform will provide data that your app can use to transform the image to the “proper” orientation before display. Additionally the outputPresentationTimeStamp tells when the image represented by the buffer appeared in the original video. This can be helpful when you are trying to sync individual frames back to audio tracks or if your app only wants to modify specific frames and then reinsert them into the original clip.

Working with Streams

In addition to local files, an app may want to use a video stream as the input. The process is largely the same, as there is a method to extract a CVPixelBuffer from an AVPlayer that is streaming. You can find the complete example code for pushing pixel buffers from a stream to an MTKView in this blog post about streaming. However, the important part of the code perhaps looks familiar by now:

let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) {
  if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
     let frameImage = CIImage(cvImageBuffer: buffer)
     self.currentFrame = frameImage //a CIImage var
     self.videoView.draw() //our MTKView
   }
}

The code above uses a CADisplayLink to query the stream on a regular basis for new video frames. It then extracts a buffer and sends it over the MTKView for display.

Working with the Cameras

Much like a video stream with an AVPlayerItemVideoOutput to output pixel buffer data, a standard object to use with the camera is an AVCaptureVideoDataOutput object. First, you need to set up a standard capture session for either the front or back camera. Then attach a video data output object to the session.

videoOutput = AVCaptureVideoDataOutput()
let videoQueue = DispatchQueue(label: "captureQueue", qos: .userInteractive)
videoOutput.setSampleBufferDelegate(self, queue: videoQueue)

if captureSession.canAddOutput(videoOutput) {
  captureSession.addOutput(videoOutput)
} else {
  fatalError("configuration failed")
}

Whenever the camera has collected enough data to generate a pixel buffer it will send that to its delegate. The delegate can then convert that data to a CIImage and send it to the MTKView to get rendered. The method in a AVCaptureVideoDataOutputSampleBufferDelegate would look like this.

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
  //get a CVImageBuffer from the camera
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    return
  }

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()
}

Going Further

In this tutorial, you saw how to extract pixel buffers from the camera, local file and remote streams and convert them to CIImage Then you saw how to render a CIImage to fill all or part of an MTKView. If your application needs to resize or apply filters to the images, you can do that before calling the .draw method of the MTKView. If the only reason you are considering using MKTViews is because of a Metal or OpenGL kernel you want to use, consider our tutorial on how to wrap kernels into Core Image filters.

If your application needs to gather streams of video, filter and then render them to the screen with precision, then drawing to an MTKView may be sufficient. However, if you want to let your users dictate how and where to filter the streams, then you may want to consider an SDK like VideoEditor SDK or CreativeEditor SDK.

How to Remove Backgrounds Using Core ML

Walter — Tue, 28 Jun 2022 15:26:53 GMT

In this tutorial, you will learn how to use machine learning to identify an image background and mask it out. This is particularly useful for making stickers or avatars or adding a fake background to a video call. The process of assigning the pixels in an image to a specific object is called “segmentation”. Apple provides an optimized method with pictures of people in its Vision framework. For performing the same tasks with non-human subjects, you can use the DeepLabV3 machine learning model with Core ML. The code examples in this tutorial have been tested using Xcode 13 and Swift 5. Because of the use of Core ML, Vision, and CoreImage in this tutorial, you should run the demo code on a device, not on the Simulator. An iOS project with the demo code is on GitHub.

Image segmentation is a different process and requires other machine learning models than image recognition. With recognition, the model produces bounding rectangles that the system believes to contain the entire object. With segmentation, the model identifies the actual pixels of the object.

In this tutorial, you’ll start with an image of your subject. Then you’ll generate a mask for the background using Vision. Finally, you’ll use CoreImage filters to blend the original image, image mask, and the new image background.

Whether using the Vision framework alone or supplementing Vision with another Core ML model – the process will be the same:

Create a Vision request object with some parameters.
Create a Vision request handler with the image to be processed.
Process the image with the handler and object.
Process the results into the final image using CoreImage.

When working with still images, Apple’s CoreImage framework is usually the best option, mainly because of the large number of available filters and the ability to create reusable pipelines that you can run on the CPU or the GPU.

Using Vision to Segment People

Without an external model, Vision can only segment documents or people in an image. It is vital to understand that Vision will only identify a pixel in the image as “this pixel is part of a person” or “this pixel is not part of a person.” If an image contains a group of people, Vision will not be able to separate individuals.

To segment the image, start by creating an instance of a VNGeneratePersonSegmentationRequest.

var segmentationRequest = VNGeneratePersonSegmentationRequest()
segmentationRequest.qualityLevel = .balanced

This type of request has a few options you can set. .qualityLevel can be .fast, .balanced or .accurate. The level of .accurate is the default. This will determine how closely the mask conforms to the boundaries of the original image. The different levels process at different speeds. The .fast setting is intended for use in a video so that frames don’t get dropped. Using .balanced or .accurate causes noticeable delay in an app processing the image on most devices. Experiment with different settings depending on your needs.

The output of the segmentation request will be a CVPixelBuffer. This is a structure that contains information for each pixel of the image. Most of Apple’s video frameworks as well as CoreImage can work with pixel buffers. The default for VNGeneratePersonSegmentationRequest is a buffer where the color of each pixel is represented by an 8-bit number. Any pixel that Vision thinks contains part of a person will be white and any not-a-person will be black and represented by zero. This will be exactly what you want for generating a mask to work with CIBlendWithMask.

Next, create a VNImageRequestHandler with the image to be processed. The handler class has a number of initializers for different types of data. In this example we will use CGImage, but you could also start with CIImage, CVPixelbuffer, Data, or others. You can also specify options for the handler. In this tutorial we will not specify any, but one of the options is to pass in a CIContext. This can help with performance as you can tell your app to do all of the processing with a single context on the GPU using Metal.

guard let originalCG = originalImage?.cgImage else { abort() }
let handler = VNImageRequestHandler(cgImage: originalCG)
try? handler.perform([segmentationRequest])
guard let maskPixelBuffer =
  segmentationRequest.results?.first?.pixelBuffer else { return }
let maskImage = CGImage.create(pixelBuffer: maskPixelBuffer)

In the code above, the originalImage (which happens to be a UIImage) gets converted to a CGImage. Then we use the image to initialize a request handler. The VNImageRequestHandler has a method .perform which takes an array of all of the Vision requests you want to use to process the image. The .perform method will not return until all of the requests in the array have been completed. Depending on how you want to structure your code, you can either provide a completion handler to use with each of the requests or just process the results in-line. In this example, we’ll process in-line.

If the segmentationRequest found any people in the image, the results array will contain a .pixelBuffer. Create the mask we need by converting the pixelBuffer into a CGImage. To convert to a CGImage, the example uses some helper methods that Matthijs Hollemans published to GitHub specifically for working with Core ML inputs and outputs.

Now that we have the mask image, use CIFilter to compose the final image. It will take a few steps. First, resize the new background, mask, and original image to the same size. Then, blend the three images.

//Convert main image to a CIImage and get the size
let mainImage = CIImage(cgImage: self.originalImage!.cgImage!)
let originalSize = mainImage.extent.size
//Convert the maskimage to CIImage and set the size
//to be the same as the original
var maskCI = CIImage(cgImage: maskImage!)
let scaleX = originalSize.width / maskCI.extent.width
let scaleY = originalSize.height / maskCI.extent.height
maskCI = maskCI.transformed(by: .init(scaleX: scaleX, y: scaleY))
//Convert the new background to a CIImage and set the size
//to be the same as the original
let backgroundUIImage = UIImage(named: "starfield")!.resized(to: originalSize)
let background = CIImage(cgImage: backgroundUIImage.cgImage!)
//Use CIBlendWithMask to combine the three images
let filter = CIFilter(name: "CIBlendWithMask")
filter?.setValue(background, forKey: kCIInputBackgroundImageKey)
filter?.setValue(mainImage, forKey: kCIInputImageKey)
filter?.setValue(maskCI, forKey: kCIInputMaskImageKey)
//Update the UI
self.filteredImageView.image = UIImage(ciImage: filter!.outputImage!)

The above code resizes the images to match the size of the original image and converts them to CIImage. Many machine learning models commonly resize inputs and outputs. For example, the output of the VNGeneratePersonSegmentationRequest using the demo image is 384x512.

Both CIImage and UIImage formats describe an image but do not always have a bitmap representation. That is why each image gets converted to a CGImage first. Though this guarantees that the demo code will work, converting formats makes the code run slower. In your app, you should experiment with different methods to get your images into CIImage format for filtering. Also, the above code uses Matthijs’ resized helper method to do some of the resizing.

With everything resized, the CIBlendWithMask filter stitches the images together. In place of the black pixels in the mask, the final image will show the background – for white pixels, the foreground. Wherever the mask image has a gray pixel, the final image will blend background and foreground.

Now let’s see how to use an additional CoreML model to process images that don’t have a person as the main subject.

Choosing a Model

Apple provides a number of pre-built models for text and image processing. The DeepLab v3 Machine Learning model can perform segmentation requests on images that have subjects like dogs. You can download it from Apple directly or also find it at the DeepLab repo on GitHub. Once you have downloaded the model, add it to your Xcode project the same as any other file. The DeepLab model will recognize people, the same as the VNPersonSegmentationRequest, but also other objects. This does come at a cost in an increased filesize for your application. You may notice that Apple provides multiple versions of the model of different file sizes. They all perform the same task, but differ in how they represent the output and a few other things.

Models on Apple’s website are packaged to work with Core ML and Xcode, so you can easily try a different model. The input and output method names will be the same. It is beyond the scope of this tutorial, yet Apple provides tutorials and example scripts on how to convert TensorFlow and other machine learning models to work with Core ML.

Core ML and Xcode

When you are working with a Core ML model, Xcode provides some convenient tools. Access them by highlighting the name of the model in the File Navigator pane of Xcode. For instance, clicking “Preview” will let you test the model with your data.

You can also use the “Predictions” tab to determine how the input needs to be formatted and what to expect for the output.

Here you can see that the DeepLabV3 model expects images to be 513 x 513 and will output a multidimensional array of integers that is 513 x 513 in size.

Finally, on this screen, you can see an entry for the “Model Class” in the headers. Double-click on the name to jump into the Swift wrapper class for the model to see how to call the model in your code and work with the inputs and outputs.

The DeepLabV3 Model

The DeepLab model input will be a color image that is 513 x 513 pixels. The Vision framework will handle resizing the input, but you can provide options on how that resize should work. The DeepLabV3 model has been trained to recognize and segment these items:

aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
dining table
dog
horse
motorbike
person
potted plant
sheep
sofa
train
tv or monitor

Anything that the model does not recognize, it will consider a background. After performing the recognition, the model returns a two-dimensional 513 x 513 array. Each entry in the array corresponds to one pixel in the original 513 x 513 input image. For instance, every pixel in the original image that shows a dog will be represented in the output by a 12 in the corresponding array entry. Any pixel that is not a recognized object will be given a 0 in the output array to represent a background.

If the model recognizes multiple objects, there will be multiple numbers in the array. If that is not what you want, you need to make changes to the array before creating the mask. For example, using an image of a dog riding a horse, some entries in the array will be 12, others will be 13, and the rest will be 0. To filter out the horse, you need to loop through the array and change any 13s to 0s.

Segmentation with DeepLab

To use the DeepLab model in your code, you again use a request to the Vision framework but this time you can specify a model.

let config = MLModelConfiguration()
var segmentationModel = try! DeepLabV3(configuration: config)
if let visionModel = try? VNCoreMLModel(for: segmentationModel.model) {
  self.request = VNCoreMLRequest(model: visionModel)
  self.request?.imageCropAndScaleOption = .scaleFill
}

In the code above, we create an instance of the DeepLab Core ML object and then initialize a generic VNCoreMLRequest with its model. The only option we set on the request is to tell it how to modify the image when it resizes it for input.

Now the code is very similar to the first example.

let cgImage = originalImage?.cgImage
let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
try? handler.perform([request])
if let observations = request.results as? [VNCoreMLFeatureValueObservation],
  let segmentationmap = observations.first?.featureValue.multiArrayValue {
  guard let maskUIImage = segmentationmap.image(min: 0.0, max: 1.0) else { return }
  applyBackgroundMask(maskUIImage)
}

Because we are using a generic VNCoreMLRequest we have to cast the results as VNCoreMLFeatureValueObservation. The DeepLab model returns a .multiArrayValue instead of a .pixelBuffer so we will again rely on the helper methods to convert it to an image. Now that the mask image has been created, applying the background is the same as in the original example.

Going Further

This tutorial focused on still images, yet both the standard Vision segmentation requests and DeepLabV3 allow processing video input as they can work with CVPixelBuffer and VNSequenceRequestHandler.
The rest of the process will be almost identical to our example. You will need to finetune the performance, or else you will experience frame drops.

Apple provides another method for identifying the background and primary subject in a still image: Portrait mode. When a device renders Portrait mode, it takes pictures with all cameras on the device and stitches them together. This way, the depth of field can change, and you can adjust what items are in focus or blurred.

If the DeepLab model does not cover your subject matter, you can train the DeepLabV3 model to recognize other objects using your own data.

However, you may consider using SDKs such as PE.SDK and VE.SDK. Enabling background removal for your users is easy — all it takes is a few lines in your configuration as described in the official PE.SDK documentation for iOS and Android, which adds a control to the photo editor to toggle the setting.

Integrate a fully customizable photo editor with Background Removal into your app with PE.SDK.

The latest VE.SDK release introduced Background Removal for stickers. This feature recognizes people in pictures and removes the background with one tap – no need for manual outlines or masks.

Let your users create beautiful visuals with Sticker Background Removal.

This feature is available on Android and iOS 15.0 and higher only. See the official documentation for Sticker Background Removal on Android or iOS.

Using an SDK will simplify and accelerate your app development, as you can save time and resources, and focus on the growth and innovation of your application instead.

Wrapping Up

In this tutorial, you saw how to use Vision and Core ML to segment an image of a person and remove the background. You also saw how to use DeepLab to work with images of non-persons.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.

Working with Large Video and Image Files on iOS with Swift

Walter — Tue, 10 May 2022 13:27:44 GMT

Video files and image files are among the largest files. The AVFoundation and CoreImage libraries that Apple supplies widely store data on the disk. They try to bring into memory only as much of a file as is needed to complete a task. As you are working on an application that uses files, there are times you will want to compress or resize them. Usually, this is to upload to a server or because you have noticed performance issues, such as stuttering when scrolling or long render times.

This article will cover strategies for compressing and resizing images, videos, and other files. We will also be looking at using URLSession for file uploads and downloads.

As iPhone cameras can take higher-quality pictures, the file sizes have grown quite large. For the last few years, Apple has been using a 12MP camera that takes pictures at 3024 x 4032 resolution. Apple uses a special HEIC compression to make each image about 1.7MB. The same image will generate a 15MB file when saved as a .png and a 3MB file when saved as a .jpeg. Though this will be a high-quality image, services like Instagram or Facebook will often only display images at less than half of that size. The extra pixel data will be compressed away. Perhaps the first and most important part of working with large image files in your app is planning around what size or quality you need. Social Media services similarly resize video files. For instance, TikTok displays images and video using 1080 x 1920, but any iPhone after X will capture video at a larger size.

Writing an Image to Disk

When working with a UIImage or CoreImage object, normally you can save it as an NSData object. These files are lossless (the image quality will not degrade), but they are only useful inside of your application. The standards on the Internet are jpeg and png files.

Fortunately, Apple provides methods to convert UIImage to .jpegData or .pngData. There are also ways to convert CIImage and CGImage, but they are slightly more complicated to configure.

let imageToSave = UIImage(ciImage: <someImage>)! \\1
let jpegData = imageToSave.jpegData(compressionQuality: 1.0) \\2
let file = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first \\3
do {
  try jpegData?.write(to: file!.appendingPathComponent("sleeping_dog.jpg")) //4
} catch(let err) {
  print(err.localizedDescription) //5
}

Here is what’s going on in the code above:

First we need to convert our image to a UIImage. The UIImage class has several different initializers. Use UIImage(named: "some image name") to read an image from the app bundle. Use UIImage(contentsOfFile: "some filepath") to read from the disk. Use UIImage(cgImage: "some cgImage") and UIImage(ciImage: "some ciImage") when you have images already in memory. If you are storing images in a CoreData store or somewhere as Data you can use UIImage(data: "some data").
However you get a UIImage the next task is to convert it to either a .jpg or .png file. Generally, .jpg is a better option when working with photographs and .png produces higher quality for line art. The example above creates a .jpg at the highest quality possible.
Ask the FileManager for a URL to the user’s documents directory. That is generally a safe place to write files. Because of the way the function is formed, it returns an array of URL items. In this case, there is only one, but you still need to remember to specify the .first item.
Writing Data object to disk can throw errors, so it is good practice to place it in a try…catch block of code. The URL to the user’s documents directory needs to have the actual filename appended.
Any errors that the system throws can be handled here.

In order to create a .png using the imageToSave above, the line would be

let pngData = imageToSave.pngData()

The .png filetype does not have any compression settings. The filetype is “lossless”, so it is always compressed as much as possible without losing any data.

Compression Options for JPEG

When generating jpeg data above, we supplied a quality value. The value is a Float between 0.0 and 1.0. The .jpg filetype uses a “lossy” compression algorithm that is specifically designed for photographs.

In the image above saving the jpeg with a quality of 1.0 produced the dog on the right and the file is 1.3MB. Saving the jpeg with a quality of 0.0 produced the image on the left and the middle image is produced saving with a quality of 0.6. The file on the left is only about 6% of the size of the original.

Considering what the images are being used for can help you decide on an optimal compression setting for a particular app.

Compressing Data with `zlib`

Though png and jpeg files are already compressed sometimes you will want to compress other types of files before uploading them (compressing a png, mov or jpeg will often not affect the file size or will make the file larger). A web server will often be able to accept a POST that compresses its body data using gzip. Additionally, some web servers will require that binary data is transformed into a base64EncodedString before uploading. Base64 encoding will make data size grow by about 33%. Using gzip will help mitigate that somewhat. Apple’s implementation of gzip is called .zlib when compressing data. Swift provides some other compression algorithms such as .lzfse which will result in smaller files, but they are not widely adopted.

For example, to encode something as base64 and then compress it with with gzip so you can upload it, we can use code like below

let jsonToUpload = "{"books":[{"title":"The Three Musketeers","author":"Dumas"}]}"
let jsonAsData = jsonToUpload.data(using: .utf8)
let encodedString = jsonAsData?.base64EncodedString()
let compressedString = try? NSData(data: (encodedString?.data(using: .utf8))!).compressed(using: .zlib) as Data

The code above creates a string of data in JSON format. It then encodes the String as a Data object using .utf8 encoding. Using .utf8 is a standard. If the web server you are uploading to requires a different encoding, you can change it. In the final step, the encoded string is compressed using gzip. Notice that the .compressed(using:) method is on NSData not on Data. NSData is an older library, so you convert the compressed NSData object back to Data by casting using as Data.

The above example was just to show syntax. The compressedString will be slightly larger than the encodedString. Based on the kind of data your app works with and the capabilities of the web server, you will need to experiment with how to encode and compress to get the best results.

Transferring Data With a Server

Depending on the original format of the data, your web server may want the .httpBody compressed. You can do that with some code like this:

let someJSONData = Data() //1
var request = URLRequest(url: URL(string: "https://example.com/uploads/")!) //2
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
if let compressedJSONData = try? NSData(data: someJSONData).compressed(using: .zlib) as Data { //3
  request.httpBody = compressedJSONData
}
let task = URLSession.shared.dataTask(with: request) {data, response, error in
  //check for errors and response data when the task is done
  //response will contain the status and other header messages
  //data will contain any payload the server returns
}
  task.resume() //4

Here is what the code above is doing:

Get the payload into a Data object.
Create a URLRequest as a var so that you can configure it. If you create it as a let it will be a GET request with default header fields.
Use gzip to compress the data and if successful, assign it to the .httpBody of the request.
Create a data task for the upload. Upon completion, the data, response and error variables will have any response from the server.
Actually start the task.

Uploading a Large File

An issue with the code above is that all of the initial data will be in memory. So for a large image or video upload, this may not work. However, Swift does provide a different method that will upload a file directly from the disk, reading into memory only what is needed at the time.

Code to implement that would follow this form.

var request = URLRequest(url: URL(string: "https://example.com/uploads")!)
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first
let fileToUpload = documentsDirectory?.appendingPathComponent("the-big-giant-file.mp4")
let uploadTask = URLSession.shared.uploadTask(with: request, fromFile: fileToUpload) {data, response, error in
//get some information from the server when the file has been uploaded
}
uploadTask.resume()

This looks very similar to the previous example but notice that you do not set an .httpBody. There will be header values to set though. Most web servers will want to know how large the file is, authentication credentials, and whether the data is multi-part, or raw or other metadata. These will all be set using .setValue(forHTTPHeaderField:) in the request object.

For larger files, it is often better to use a URLSessionDataDelegate instead of the single closure with data, response, error. When using the delegate, you can write code to monitor the progress of the file transfer as well as respond to authentication challenges and make decisions about caching. You can also implement URLSessionDelegate and URLSessionTaskDelegate methods. All of the delegate methods are options, so you only need to implement ones that are important to your application.

Configuring Background Transfers

Another benefit of using the uploadTask (and its related downloadTask) instead of the dataTask is that you can configure the transfer to continue for a time even if the app goes to the background. In that case, instead of using the URLSession.shared object, you will want to configure a session that has some special properties to allow it to work in the background.

let configuration = URLSessionConfiguration.background(withIdentifier: "some.unique.identifier")
let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
let uploadTask = session.uploadTask(with: request, fromFile: fileToUpload)

Completely configuring and supporting the downloads and backgrounding is covered in this article by Apple. It will require implementing some delegate methods for the session as well as updating some of the methods in the main application delegate. The most important is the handleEventsForBackgroundURLSession which is how the system will wake up your app with information about the upload or download.

Additionally, download tasks can be paused and restarted. The cancel(byProducingResumeData:) method on URLSessionDownloadTask will cancel a download and optionally provide some resume data you can use to resume the download at a later time. You can call this method on the task when the user taps a button or when the app goes to the background or similar. The web server must support byte-range requests and ETag or Last-Modified headers. The documentation for pausing and restarting explains how to pause the download and how to resume using downloadTaks(withResumeData:). There is not a similar structure for pausing and resuming an upload.

Resizing an image

Perhaps the simplest way to make a file smaller is to reduce its dimensions. Using CoreImage you can use the Lanczos resampling algorithm to resize an image while keeping high quality.

let url = Bundle.main.url(forResource: "sleeping-dog", withExtension: "jpeg")
let image = CIImage(contentsOf: url!) //1
let filter = CIFilter(name: "CILanczosScaleTransform") //2
filter?.setValue(image, forKey: kCIInputImageKey)
filter?.setValue(0.5, forKey: kCIInputScaleKey) //3
let result = filter?.outputImage
let converter = UIImage(ciImage: result!) //4

Here is what this code is doing:

Create a CIImage from a URL.
Create a CILanczosScaleTransform filter.
Configure the filter to reduce the image dimensions by 50%.
Convert the scaled image to a UIImage.

Recently, Apple has provided a UIGraphicsImageRenderer that will work directly on UIImage objects. The code below will resize a UIImage to 50%. Notice that the UIGraphicsImageRenderer can also use an explicit value for size (unlike the CILanczosScaleTransform)

let imageToSave = UIImage(named: "sleeping-dog.jpeg")! //1
let newSize = imageToSave.size.applying(CGAffineTransform(scaleX: 0.5, y: 0.5)) //2
let renderer = UIGraphicsImageRenderer(size: newSize) //3
let scaledImage = renderer.image { (context) in
  let rect = CGRect(origin: CGPoint.zero, size: newSize)
  imageToSave.draw(in: rect) //4
}

Here is what this code is doing:

Load the image as a UIImage.
Scale the size down by half.
Create a UIGraphicsImageRenderer that will render in this new size.
Actually draw the image into the renderer at the new size.

In addition to rendering the image back as a UIImage the renderer can create jpegData and pngData which would save a few steps if you are resizing and converting. The full documentation for UIGraphicsImageRenderer explains the details.

Resizing a Video

Though the UIGraphicsImageRenderer is faster, using the older CoreImage resizing has a benefit. CoreImage filters can be applied to video files as a composition. Large video files can be scaled down. The code below will create a composition that will scale a video asset by 50%. Remember that you will need to import AVFoundation to use these tools.

let newAsset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
var newSize = <some size that you've calculated> //2
let resizeComposition = AVMutableVideoComposition(asset: newAsset, applyingCIFiltersWithHandler: { request in
  let filter = CIFilter(name: "CILanczosScaleTransform") //3
  filter?.setValue(request.sourceImage, forKey: kCIInputImageKey)
  filter?.setValue(<some scale factor>, forKey: kCIInputScaleKey) //4
  let resultImage = filter?.outputImage
  request.finish(with: resultImage, context: nil)
})
resizeComposition.renderSize = newSize //5

Here’s what the code above is doing to create a new composition:

Create an AVAsset from a URL.
Create a variable newSize to hold the final size.
In the composition, configure the CIFilter that will be applied to each frame.
Calculate the scale factor based on the newSize variable and the actual size of the request.sourceImage.extent.size.
Set the renderSize property of the composition to the new size.

If you don’t set the renderSize then there will be a black letterbox around the video.

With the resizing composition, you can now export the AVAsset as a .mov or .m4v file using an AVExportSession.

let asset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("exported.mov") //2
//create exporter
let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)
//configure exporter
exporter?.videoComposition = <the composition you created above> //3
exporter?.outputURL = outputMovieURL!
exporter?.outputFileType = .mov
//export!
exporter?.exportAsynchronously(completionHandler: { [weak exporter] in //4
  DispatchQueue.main.async {
    if let error = exporter?.error {
      print("failed \(error.localizedDescription)")
    } else {
      print("file saved at \(outputMovieURL)") //5
    }
  }
})

Here’s what the code above is doing:

Load an AVAsset from a URL.
Create a URL to save the resized movie.
Apply the resizing composition to an AVAssetExportSession.
Asynchronously export the asset to disk.
Print the destination URL of the new movie when it is saved.

Another method to resize a video is to use one of the AVAssetExportSession presets. Apple offers several size and quality presets for export sessions. In the line above that creates the exporter, replace AVAssetExportPresetHighestQuality with one of the other values. There are generic quality presets: AVAssetExportPresetLowQuality, AVAssetExportPresetMediumQuality and there are size presets including: AVAssetExportPreset1280x720 and AVAssetExportPreset640x480. When using one of the presets you do not need to use a composition unless you want to do further manipulation or unless you need a size or quality combination that is not provided by any preset. As with images, experiment with different settings until you get a balance between quality and size that works for you.

Going Further

Some other strategies for working with large files are to chunk data file or split up video files. However, to use this strategy, you will need to coordinate with someone to stitch them back together after they are uploaded or downloaded.

Using the AVAssetExportSession above, you could call it multiple times and pass in a time range perhaps using code like this

let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)
//some code to create the exporter
exporter?.timeRange = timeRange

Depending on the video and audio of the AVAsset the splits in the video may be noticable. Because of modern frame rates, the audio would probably be more likely to be noticed. Time ranges can be sub-second, so you would need to experiment. However, iOS would manage the file for you so that the entire AVAsset is never loaded into memory.

It is also possible to split Data objects using .subData(in:) and looping through the bytes of the object. Again, you would need to also write code to stitch them back together later. Additionally, you would likely need to bring the entire Data object into a memory buffer, which might not be desirable.

Another way to handle large video files is to use Apple’s HTTP Live Streaming tools. These create files that you can upload to any web server that is recognized by iOS and Mac devices. The tools will segment the video as well as create multiple versions so that your users with different bandwidth capabilities can see different quality streams. Most Android devices and web browsers will at least be able to see the video, even if they cannot dynamically switch between streams.

Wrapping Up

In this tutorial, you saw some different strategies for shrinking and compressing the large image and video files that an iPhone camera can produce. You also saw some strategies for transferring larger data payloads with a web server. As mentioned in the beginning, consider what sizes the images and videos will display and use that to determine what sizes of images to store. Also, consider storing multiple versions of images or using thumbnails when displaying a gallery or list of images.

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How to Make an Animated GIF Using Swift

Walter — Tue, 10 May 2022 12:07:58 GMT

In this article, you will see how to use Swift to create an animated GIF file. You will also see how to extract individual frames from a video file to convert the whole video or just clips from a video into an animated GIF. The code in this article uses Swift 5 and Xcode 13. The tutorial uses AVFoundation and CoreGraphics. Clone this repository for a sample project and example code.

Animated GIF Basics

The GIF is one of the older file types on the Internet. But, every platform supports it (not every platform supports every video format). Unlike some other image formats, GIF files can contain one image or a series of images. When a GIF file has a series of images and some metadata to describe how long to show each one, it can substitute for animation and video. Because of this, a platform that doesn’t necessarily support video files can display moving images and animations.

GIF files are easy to make. However, animated GIF files are not perfect: they are large and inefficient compared to modern video files. They have a limited color palette, and none of the images in the file have compression. For example, the video file we use in the sample project is about 5MB in mp4 format but over 100MB after it becomes an animated GIF. Finally, GIF files don’t have a soundtrack. Still, they are popular and do certain tasks well, so let’s make some using AVFoundation and Swift.

The Basic Strategy

In order to create animated GIF files you need a series of images and some metadata to describe how long each image will show. This tutorial will start with how to put these images into a GIF. Then you will see some different strategies for gathering the images. A GIF file contains bitmap images, not vector graphics. When working with bitmap images, a powerful framework to explore is CoreGraphics.

Creating an Empty File

The first step is to create an empty file on the disk that will become our animated GIF.

//create empty file to hold gif
let destinationFilename = String(NSUUID().uuidString + ".gif")
let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(destinationFilename)
//metadata for gif file to describe it as an animated gif
let fileDictionary = [kCGImagePropertyGIFDictionary : [
  kCGImagePropertyGIFLoopCount : 0]]
//create the file and set the file properties
guard let animatedGifFile = CGImageDestinationCreateWithURL(destinationURL as CFURL, UTType.gif.identifier as CFString, totalFrames, nil) else {
  print("error creating gif file")
  return
}
CGImageDestinationSetProperties(animatedGifFile, fileDictionary as CFDictionary)

The code above creates a URL to hold the animated GIF in the user’s Temporary directory. The location is not crucial – it just needs to be where you have write permission. Next it creates a Dictionary that will describe the file as an animated GIF file that will loop forever. The Apple Documentation for GIF Image Properties contains the full list of options for the dictionary. Next, you create the empty file using CGImageDestinatonCreateWithURL. After creating the file, set the properties dictionary. Now there is an empty file ready to accept the frames that will make up the animated GIF.

When creating the file with CGImageDestinationCreateWithURL, you give parameters for the URL to the file location, the UTType identifier to show it will be a .gif file, and the totalFrames that it should expect. totalFrames is just a variable that holds how many actual frames the file will eventually hold. You will need to work out the actual number of frames for your file manually. For example, if you want to display 20 images each second and your GIF will be four seconds long, the total number of frames will be 20 * 4 or 80. The last parameter is always nil. The documentation indicates it “is reserved for future use” (but this function has been available to the Mac since 2005 and iOS since 2010, so we will see). The documentation also indicates you are responsible for releasing the memory allocated by the creation command, but Swift will handle that for you.

As you work with CoreGraphics classes and types, you will notice that they have a slightly different format than other frameworks. That is a hint that CoreGraphics is one of the older frameworks and that it is all C based. Swift has bridging code so that the system will worry about translating your Dictionary to a CFDictionary and your String to a CFString and so on.

Adding Images

To add images to the empty file, you will need an image for each frame of the animation, and you will need a small Dictionary to tell the system how long to display the image during the animation. As mentioned above, there are other options you can apply, such as color mapping or frame dimensions.

A Dictionary to display each frame for 1/20 second could look like this:

    let frameDictionary = [kCGImagePropertyGIFDictionary : [
      kCGImagePropertyGIFDelayTime: 1.0 / 20.0]]

Notice that the values above are Double and are not Int. Using 0.05 would have also been fine as it is equal to 1.0/20.0.

Compressed video files and movies are usually 30, 60, or even 120 frames per second. Recall that animated GIF images do not have compression, so you want to have as low of a frame rate (or as small of an image frame) as you can. Experiment with different frame rates depending on the speed of the action in an animation. Often times 20 or even 10 frames per second will produce high enough quality.

Now, loop through your images and append each image to the .gif file on disk using code like this:

CGImageDestinationAddImage(animatedGifFile, frameImage, frameDictionary as CFDictionary)

The animatedGifFile is the URL to the empty file on the disk that you created above. The frameDictionary is the metadata to add to each frame and the frameImage is the actual image for the frame. Your image for the frame needs to be a CGImage. Depending on the source of images you may have UIImage or CIImage or something else. Most image formats in Swift have a conversion method. For example:

let cgImage = UIImage().cgImage
let anotherCGImage = CIImage().cgImage

There are also CGImage initializers to read .jpeg and .png data.

Finalizing the File

Once you finish writing every frame image to the GIF file, close the file and instruct the system to clean up using:

CGImageDestinationFinalize(animatedGifFile)

This function will return true if it is successful. You must call this function after appending the frames or the resulting GIF will be unusable. Also, after you call this function, you cannot add more frames.

At this point the animatedGifFile URL points to an animated GIF file you can use as needed.

Collecting Frame Images

To create the animated GIF, it is necessary to have an image in CGImage for each frame. It will be easier if they are all rotated the same way and have the same dimensions. Where those images come from is not crucial. They might be a series of files on disk. It could be that your user snaps a series of photos that your app converts to a GIF in real-time. Your app could even intersperse photographs and bitmap drawings. Something to consider regardless of the source of the frame images is how much memory they consume. Loading the entire set of images into an array before making the animated GIF could cause your app to ask the system for gigabytes of memory.

Extracting Images from Video

A common use of animated GIF files is to display videos. So, let’s look at how to extract frames from a video. In some of the other tutorials, you have seen how we can use CoreImage and AVFoundation to manipulate each frame of a video. In those cases, you can’t easily change the frames per second. A 60 frames-per-second animated GIF would quickly become an unusably large file.

AVFoundation provides the AVAssetImageGenerator class specifically to extract frames from AVAssets at specific times. To extract a single image from an AVAsset you can use:

copyCGImage(at requestedTime: CMTime, actualTime: UnsafeMutablePointer<CMTime>?)

This will create a CGImage at or near the requestedTime in the video. Sometimes, the requested time is between two frames, so the function will extract the closest frame it can and return the actualTime of that frame. Your code can then decide if it’s an acceptable substitution and what to do about it. Usually, though, you don’t care and can pass nil for the actualTime parameter. Notice that the actualTime is a pointer to a CMTime variable. This means it is an “inout” parameter. Using “inout” parameters was a common practice when CoreGraphics and AVFoundation were originally written. To use an “inout” parameter, you declare the variable before calling the function and then check its value after. Additionally, when you pass the variable into the function, place an ampersand before the variable name.

For extracting a series of images, copyCGImage is not efficient. It runs on the calling thread and may block your application. Apple provides a different function when you want to extract many frames from a video asset.

generateCGImagesAsynchronously(forTimes requestedTimes: [NSValue], completionHandler handler: @escaping AVAssetImageGeneratorCompletionHandler)

This function takes an array of requestedTimes and attempts to extract the image at each of those timestamps. It will pass the extracted image to the completionHandler. The handler will execute its code one time for every extraction attempt and will not block the calling thread.

Before calling the function to extract images, you need to prepare the inputs. First, you create an AVAssetImageGenerator and configure it.

let movie = AVAsset(url: Bundle.main.url(forResource: "swish", withExtension: "mp4")!)
let frameGenerator = AVAssetImageGenerator(asset: movie)
frameGenerator.requestedTimeToleranceBefore = CMTime(seconds: 0, preferredTimescale: 600)
frameGenerator.requestedTimeToleranceAfter = CMTime(seconds: 0, preferredTimescale: 600)

This code loads a movie file from the app bundle into an AVAsset and then creates a generator for that asset. The .requestedTimeToleranceBefore and .requestedTimeToleranceAfter parameters say you want the exact frame for any requested time. This will make the extraction of frames slower. If you are trying to tune performance, consider modifying these parameters.

Now you will need to calculate the timestamp of each frame of the animated GIF. Determine the start time and the duration of the clip you want to create. Then decide what frame rate (frames per second) to use. You can see in the image below that the entire video consists of about 480 frames at 60 frames per second.

An animated GIF of just the player making the basketball shot would start at 2 seconds and end at 6 seconds of the original video. In the original video, that segment is 240 frames. An animated GIF of the same length at 20 frames per second only needs about 80 frames. The first frame is the one showing at the 2-second timestamp. The next frame should be the one shown at 2 + 1/20 or 2.05 seconds. The next frame would be the one for 2.10 seconds.

The generator requires the array of timestamps as CMTime values. You can create the array using code like this:

var startTime: Double = 2.0 //seconds
var endTime: Double =  6.0 //seconds
var frameRate = 20 //how many frames per second
let totalFrames = Int((endTime - startTime) * Double(frameRate))
var timeStamps = [NSValue]()
for currentFrame in 0..<totalFrames {
  let frameTime = Double(currentFrame) / Double(frameRate)
  let timeStamp = CMTime(seconds: startTime + Double(frameTime), preferredTimescale: 600)
  timeStamps.append(NSValue(time: timeStamp))
}

This code calculates a value for totalFrames which must be an Int so that you can use it to loop. But, startTime and endTime need to be Double since the timestamps will be sub-second times. The frameTime of the first frame is equal to zero in the animated GIF but 2.0 seconds in the original video. Convert that to the video timeStamp by adding the startTime. When creating the CMTime using “600” as the preferredTimescale is common because most frame rates are divisible into 600 evenly so the math for the timestamp will be more accurate. Finally, wrap the timeStamp in an NSValue object and add it to the array. The AVAssetImageGenerator uses timestamps wrapped in NSValue objects. NSValue is just a generic type that holds values.

With the generator configured and the array of timestamps created, you can extract the images using code like this:

frameGenerator.generateCGImagesAsynchronously(forTimes: timeStamps) { requestedTime, frameImage, actualTime, result, error in
  guard let frameImage = frameImage else {
    print("no image")
    return
  }
  if error != nil {
    print("an error")
    return
  }
  //do someting interesting with frameImage
}

The frameImage in the completion handler will be a CGImage type if the extraction was successful. As with the function to extract a single image, if the system had to choose a different time than requested, you will be informed of the actual time. You can read about the other parameters in the documentation for AVAssetImageGeneratorCompletionHandler.

Now, you can add the frame to an open CGImageDestination, you could write it to a disk to use later, and you could save it to an array. If you don’t write it to an animated GIF file right away, you will need to devise a way to keep the frames in sequence. You may notice that there is not an explicit way for the function to signal that it is finished. Because the completion handler is called for every extraction attempt, a common strategy is to make a simple counter and increment the counter in the completion handler. When the counter variable is equal to the number of frames requested, then you are finished. Another way is to match the requestedTime to the last timeStamp.

Wrapping Up

In this article, you saw how to create an animated GIF from a series of images. You also saw how to extract still images from a video at pre-determined timestamps. Clone this repository for a simple project that contains the sample code and a video file you can experiment with.

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

How to Make Videos from Still Images with AVFoundation and Swift

Walter — Thu, 10 Feb 2022 15:35:45 GMT

Whether making a slide show or breaking up video tracks with title scenes, inserting still images into a video file can be tricky. This tutorial will show you how to convert still images into standard Quicktime video files so you can work with them further in your favorite video editor. This tutorial was created and tested with Xcode 13 and Swift 5. This tutorial will discuss two strategies:

convert each image to a separate movie and stitch them together later (this is best suited for times you want to interleave the images and other video files using some other editing workflow)
create a “blank” video and use the applyingCIFiltersWithHandler variation to overlay the images (this is best suited for times you want to make a quick slide show with no further editing)

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with this code (and some extra goodies) can be found on GitHub.

The Problem with an Image

The basic models that are used in AVFoundation for making video files are AVAsset and AVAssetTrack. You can compose multiple tracks and assets together to make a single video file. All tracks must have some media data and a duration. Static image files don’t have a duration. This is the primary problem you have to overcome when you want to add static images to video.
Fortunately, you can use many of the same AVFoundation tools that are used for video capture and frame manipulation. However instead of using a CADisplayLink and AVPlayerItemVideoOutput or an AVCaptureDevice to provide frame buffers to save, you will create the frame buffers manually. Then you can use AVAssetWriter to convert the frame buffers into a video file.

Some items to consider before you actually generate the video file are:

What framerate to use? Because the image doesn’t have any motion, you can use a very low frame rate to create smaller file sizes. However, this might cause issues if you are interleaving it with regular video, which usually has a frame rate of 30 or 60 fps (or higher, for slow motion video)
What dimensions will the final video have? It is easy to resize an image and add padding using CoreImage or some image editing software before it becomes a video. As the image becomes a video and moves through different steps in an AVFoundation pipeline, the different classes in AVFoundation will resize or crop the images in different ways when the input dimensions and output dimension don’t match.

Writing Buffers using `AVAssetWriter`

The AVAssetWriter exists to encode media to a file on disk. Though this tutorial will use a single video input AVAssetWriter supports multiple inputs of different kinds (audio, video, metadata). It is quite similar to AVAssetExportSession. The difference is that AVAssetWriter uses multiple inputs and each input is a single track where the AVAssetExportSession takes a single AVAsset as an input. The result of both is a single file.

As indicated above, our basic strategy will be:

Create a pixel buffer that is the correct color space and size to hold the image
Render the image into the pixel buffer
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop, and each time through the loop, append the pixel buffer
Clean up

Create the Pixel Buffer

The code below creates a CIImage from an image file. Then it creates a pixel buffer the same size as the image and uses a standard color space. Finally, a CIContext renders the image into the pixel buffer.

//create a CIImage
guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
}
//create a variable to hold the pixelBuffer
var pixelBuffer: CVPixelBuffer?
//set some standard attributes
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
     kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
//create the width and height of the buffer to match the image
let width:Int = Int(staticImage.extent.size.width)
let height:Int = Int(staticImage.extent.size.height)
//create a buffer (notice it uses an in/out parameter for the pixelBuffer variable)
CVPixelBufferCreate(kCFAllocatorDefault,
                    width,
                    height,
                    kCVPixelFormatType_32BGRA,
                    attrs,
                    &pixelBuffer)
//create a CIContext
let context = CIContext()
//use the context to render the image into the pixelBuffer
context.render(staticImage, to: pixelBuffer!)

Though we generally think of CIImage as an image, Apple is clear that it is not an image by itself. CIImage needs a context for rendering. This is where a lot of the power of CoreImage and filters and GPU rendering come from, the fact that CIImage is just the instructions for creating an image. Note: You can also use CGImage and the CoreGraphics framework to create pixel buffers. I find it easier to use the CoreImage framework.

Configure AVAssetWriter

Now with a buffer created, you can configure the AVAssetWriter. It will take several parameters as a dictionary to determine the dimensions and format of the outputs. You can either set them individually or else Apple provides presets.
One of the most important settings is the output dimension. If the output dimension matches the original image, the video file will show no distortion or letterboxing. However, if the output is smaller, it will compress the image until it fits. If the output is larger, the image will expand until its width or height matches the output size and then will letterbox. Note: if an image expands too much, it will appear grainy. The image below shows different output sizes for a 640 x 480 input image.

To create your settings for the AVAssetWriter first create a dictionary containing the different values. The example here sets an output dimension of 400 x 400 and provides for .h264 encoding.

let assetWriterSettings = [AVVideoCodecKey: AVVideoCodecType.h264, AVVideoWidthKey : 400, AVVideoHeightKey: 400] as [String : Any]

To use one of the presets, use the AVOutputSettingsAssistant. You can read about the different settings in the Apple documentation. An example to create 1080p video output would look like this:

let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings

Write the pixel buffer to the video file

With the settings configured, the asset writer can loop through and append the contents of the pixel buffer to create each frame of the video. Something important to notice in the code below is that AVFoundation has a philosophy to preserve data. So, it will make copies, and it will generally not overwrite files. That is why you have to delete any old files before you can create the new AVAssetWriter.

//generate a file url to store the video. some_image.jpg becomes some_image.mov
guard let imageNameRoot = imageName.split(separator: ".").first, let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("\(imageNameRoot).mov") else {
  throw ConstructionError.invalidURL //an error i made up
}
//delete any old file
do {
  try FileManager.default.removeItem(at: outputMovieURL)
} catch {
  print("Could not remove file \(error.localizedDescription)")
}
//create an assetwriter instance
guard let assetwriter = try? AVAssetWriter(outputURL: outputMovieURL, fileType: .mov) else {
  abort()
}
//generate 1080p settings
let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings
//create a single video input
let assetWriterInput = AVAssetWriterInput(mediaType: .video, outputSettings: settingsAssistant)
//create an adaptor for the pixel buffer
let assetWriterAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: assetWriterInput, sourcePixelBufferAttributes: nil)
//add the input to the asset writer
assetwriter.add(assetWriterInput)
//begin the session
assetwriter.startWriting()
assetwriter.startSession(atSourceTime: CMTime.zero)
//determine how many frames we need to generate
let framesPerSecond = 30
//duration is the number of seconds for the final video
let totalFrames = duration * framesPerSecond
var frameCount = 0
while frameCount < totalFrames {
  if assetWriterInput.isReadyForMoreMediaData {
    let frameTime = CMTimeMake(value: Int64(frameCount), timescale: Int32(framesPerSecond))
    //append the contents of the pixelBuffer at the correct time
    assetWriterAdaptor.append(pixelBuffer!, withPresentationTime: frameTime)
    frameCount+=1
  }
}
//close everything
assetWriterInput.markAsFinished()
assetwriter.finishWriting {
  pixelBuffer = nil
  //outputMovieURL now has the video
  Logger().info("Finished video location: \(outputMovieURL)")
}
}

Now, your image has become a Quicktime video of whatever duration you specified and is ready for use as any other video file! You can now continue with the like of VideoEditor SDK to trim videos, and add filters or overlays. You might also want to combine the video with other video files to make a new creation.

Say hi to my dog! A plain video from still images

Creating a Blank Video

The previous example created a single .mov file for each of your images. The most robust way to get these images into a video with sound, transitions, etc., is to use AVMutableComposition with multiple tracks and layer instructions. However, for a quick slideshow with just a few elementary transitions, a strategy is to create a blank movie and then use AVVideoComposition with (asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void). That will let you use CIImage and CIFilter to paint each frame of the blank movie with images.

Our basic strategy for this method will be:

Create a pixel buffer that is the right color space and size and fill it with a solid color
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop and each time through the loop append the pixel buffer
Create an AVVideoComposition to add the images to the individual frames of the video
Let AVPlayer combine the video and the composition

In the first strategy, you started by making a pixel buffer and using an AVAssetWriter to create the video file using this code:

guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
    }

However, this time, instead of using a source image, you create a CIImage that is a single color.

let staticImage = CIImage(color: bgColor).cropped(to: CGRect(x: 0, y: 0, width: 960, height: 540))

The CIImage(color:) initializer creates an image that is of infinite size and is just a single color. Using the .cropped(to:) modifier lets you make it the same size as the AVAssetWriter output so that there won’t be any letterboxing. The rest of the code is the same as before and the end result will be a video file that is just the single color.

Adding a Composition

Once the video file has been created and saved to disk, load it as an asset, then create an AVVideoComposition. This allows you to generate individual frames. The function below uses the request property of the composition. This has a CIImage representation of the current frame as well as the timestamp of when this frame will appear. The .fetchSlide(forTime:) is a helper function in the demo app that returns the appropriate CIImage for the slide to display. Then the .sourceOverCompositing filter combines the original frame image with the slide. Using a CGAffineTransform moves the slide across the screen as the video plays.

func createComposition(_ asset: AVAsset) {
  let slideshowComposition = AVVideoComposition(asset: asset) {[weak self] request in
    guard let self = self else { return }
    let slide = self.fetchSlide(forTime: request.compositionTime)
    let compose = CIFilter.sourceOverCompositing() //filter to join two images
    compose.backgroundImage = request.sourceImage
    compose.inputImage = slide?.transformed(by: CGAffineTransform(translationX: request.compositionTime.seconds * 50, y: 0))
    //always finish with the last output of the pipeline
    request.finish(with: compose.outputImage!, context: nil)
  }
  self.outputSlideshow = slideshowComposition
}

Because the creation of the video is done asynchronously, you can composite images over the original frame and add filters without the danger of impacting the final frame rate. Recall that CIFilter works as a pipeline, each CIFilter has an .outputImage and most have .inputImage and some configuration settings. Feeding the output from one filter to the input of the next will let you build complex compositions. Once a composition has been created, AVPlayer can join the original video with the composition for playback or export.

let item = AVPlayerItem(asset: self.outputMovie!)
item.videoComposition = outputSlideshow
self.player = AVPlayer(playerItem: item)

The code above might generate a video like this one.

As with the first video, the final video is a plain .mov file.

Going Further

In this tutorial, you saw two ways to convert static images into .mov files that you can then edit with any video editor. The sample project also has some code for exporting your creation as a .mov file and code for adding sound to the creation.

These are not the only ways to go about solving this problem. For example, instead of creating a blank video and overlaying the images, using a stock video of waves at the beach and overlaying the images might work better for your project. Conversely, because the initial pixel buffer is created from a CIImage, there is no reason that the CIImage cannot be the end of a long pipeline of CIFilters. This way the composition is done at the beginning, and the AVAssetWriter is the only tool needed. The best strategy is the one that makes sense to you and that works with the kinds of media you have.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Merge Videos in iOS with Swift

Walter — Wed, 12 Jan 2022 12:12:55 GMT

Combining video clips or parts of video clips into a single video may appear complicated, but the things that add complexity also provide flexibility. In this tutorial, you will see how to combine a few short clips into a single video. You will also gain an understanding of building on this basic pattern to make more advanced creations. This tutorial was created and tested with Xcode 13 and Swift 5.

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with code that supports this tutorial is available on Github.

Making a Composition

Whenever you want to edit media or add effects, the first place to look is AVComposition. This class in AVFoundation has the purpose of arranging different assets and types of assets into a single asset for playback or processing. Asset types can include audio and video subtitles, metadata, and text. When working with AVComposition it is helpful to think about AVAsset and AVAssetTrack objects. Unfortunately, because these items are so similar (AVComposition is a subclass of AVAsset) it is easy to get confused. Try to remember that the AVTrack is a wrapper around the actual pixel, sound-wave, or other data and that AVAsset is a collection of AVTrack objects. When we want to combine and manipulate AVAsset objects, an AVComposition helps us do that. Changes made to any AVAsset by an AVComposition will not impact the original media file.

As an extra layer, Apple keeps the editing features of an AVComposition separate from the playback features by creating an AVMutableComposition object. You will work with an AVMutableComposition object in this tutorial.

Each track in our composition will have a start time and a duration. The composition will ensure that all of the data it holds displays at the right time and that the tracks stay in sync.

At the simplest, a video clip is a single AVTrack of audio data and a single AVTrack of video data.

So, to set up an empty AVComposition for combining clips, use code such as this:

    let movie = AVMutableComposition()
    let videoTrack = movie.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
    let audioTrack = movie.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)

The code above creates a new, empty AVMutableComposition and then adds two tracks. One track will be able to hold .video assets, and the other will hold .audio assets. AVFoundation supports many different kinds of audio and video formats, but the .video track cannot hold .audio formats. The kCMPersistendTrackID_Invalid signals that you want a new, unique track id generated.

AVFoundation can manipulate any of the ISO media formats. The Quicktime (.mov) and MPEG (.mp4) are the most common, however, .heif, .3gp and other formats are all supported.

Adding Video Clips

After creation, the AVMutableComposition has a start time of CMTime.zero and a duration of CMTime.zero. Use the insertTimeRange(_ timeRange: CMTimeRange, of track: AVAssetTrack, at startTime: CMTime) throws function on the audio and again on the video track to add data from the clip. In order to add data to the tracks, you load an AVAsset, determine what range of time will go into the new composition, and then call the insert function. Your code might look like this:

  let beachMovie = AVURLAsset(url: Bundle.main.url(forResource: "beach", withExtension: "mov")!) //1

  let beachAudioTrack = beachMovie.tracks(withMediaType: .audio).first! //2
  let beachVideoTrack = beachMovie.tracks(withMediaType: .video).first!
  let beachRange = CMTimeRangeMake(start: CMTime.zero, duration: beachMovie.duration) //3
  try videoTrack?.insertTimeRange(beachRange, of: beachVideoTrack, at: CMTime.zero) //4
  try audioTrack?.insertTimeRange(beachRange, of: beachAudioTrack, at: CMTime.zero)

Here is what this code does:

Load a video file from a local URL. In the example, the video is in the application bundle, but it could also be a URL that points to a file somewhere else. This will not work with a streaming URL. It needs to be a video file.
Extract the main videos and audio tracks from the video file. There may be multiple video and audio tracks in a file, but this code just grabs the first one.
Create a time range. In this example, the range is the same duration as the clip we loaded.
Insert the video track from the clip at the beginning of the video track of the composition and insert the audio track from the clip at the beginning of the audio track of the composition. A common mistake in this step is to use the .duration property of the composition to insert the audio and video at the end. However, as soon as media is inserted into any track, the duration of the entire composition will change. So the audio and video would be played one after the other and you would see a blank screen when the audio is playing, and have no audio when the video is playing.

Repeat the steps for each clip that you want to add to the final composition.

Using the Composition

After all of the clips have been combined into the composition, it can be played in an AVPlayer or exported using an AVAssetExportSession. The code for these is the exact same as for any other AVAsset.

  self.player = AVPlayer(playerItem: AVPlayerItem(asset: movie)) //1
  let playerLayer = AVPlayerLayer(player: player) //2
  playerLayer.frame = playerView.layer.bounds
  playerLayer.videoGravity = .resizeAspect
  playerView.layer.addSublayer(playerLayer) //3
  player?.play() //4

Here is what the code above is doing:

Create an AVPlayer using a movie object, which is the AVMutableComposition that was created earlier.
Create a playerLayer to display the video and set its aspect ratio
Add the playerLayer to playerView which is a regular UIView in a UIViewController
Play the video clip

Alternatively you may want to export the new composition as a new movie file. Again, because the composition is an AVAsset using a standard AVAssetExportSession will write the movie to disk.

  //create exporter
  let exporter = AVAssetExportSession(asset: movie,
                                 presetName: AVAssetExportPresetHighestQuality) //1
  //configure exporter
  exporter?.outputURL = outputMovieURL //2
  exporter?.outputFileType = .mov
  //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error { //3
        print("failed \(error.localizedDescription)")
      } else {
        print("movie has been exported to \(outputMovieURL)")
      }
    }
  })

Here is what this code is doing:

Create an export session using the movie composition you made earlier.
Set the save location to some file URL on your device.
Wait for the exporter to finish and display the result.

Don’t forget that exporting an AVAsset can take a long time and is asynchronous. You will want to display a spinner or a message to your user.

Going Further

You can now combine clips into a longer clip for playback and export. However, trimming the original clips, adding filters, and rearranging the order will require more work before your application is ready for the store. Using an SDK like the IMG.LY’s VideoEditor SDK can help you provide an app that has the features users will expect in addition to stitching clips together.

The VideoEditorSDK offers a Video Composition Controller which will allow the users to stitch clips together. Additionally, it will allow the user to add new clips and edit clips using the other editing tools. Because the Video Composition Controller is a subclass of UIViewController it can just be configured and dropped into your application. Instead of the code in our example, you pass the videos to the controller:

let videoClips = [
  Bundle.main.url(forResource: "stream", withExtension: "mov"),
  Bundle.main.url(forResource: "beach", withExtension: "mov"),
  Bundle.main.url(forResource: "mountain", withExtension: "mov"),
].compactMap { $0.map{ AVURLAsset(url: $0) } }
let video = Video(assets: videoClips)
let videoEditViewController = VideoEditViewController(videoAsset: video, configuration: Configuration())
present(videoEditViewController, animated: true, completion: nil)

The code above loads the same video clips as our demo application and then uses the VideoEditViewController to present them. From there the user can make further edits preview and export the composition.

Wrapping Up

In this example, you saw how to combine several video clips, each with one video and one audio track into a single movie file with one video and one audio track. To make more complex movie files consider building on this tutorial by:

Add a second audio track, AVFoundation will play sounds from both tracks at times specified at equal volume. Use AVAudioMix to adjust the volume of the tracks at different times (the sample project has an example of adding a second audio track).
Add more video tracks and use AVMutableCompositionLayerInstruction to determine which track is visible at any time in the final video and to make fancy transitions between tracks.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How to Add a Filter to a Video Stream in iOS

Walter — Mon, 08 Nov 2021 15:33:55 GMT

Using filters and effects with video streams requires different strategies than applying them to files. In this tutorial, you will see how to apply effects and filters to video streams in real-time. The code in this tutorial compiles with Xcode 13.

As with most AVFoundation code, the Xcode simulator is not the best platform for running the code. Test on an actual device. A project with code that supports this tutorial can be found on GitHub.

Streams or Files

For video files stored on the device, you can use an AVMutableVideoComposition to apply filters. When streaming video from a remote location, this strategy will not work. The AVMutuableVideoComposition classes are not designed to work in real-time with streams. Apple provides a way to extract the pixels from the stream at regular intervals. You can manipulate the pixels and then render them to the screen. An AVPlayerItemVideoOutput object gives access to the pixels that make up a video frame. Apple also supplies a CADisplayLink object to ensure you are extracting the right pixels at the right time. The display link is a specialized timer that will fire in sync with the redraw rate of the current display.

So, to filter a video stream, the strategy becomes:

Set up an AVPlayer as normal and assign it the URL of the stream
Attach an AVPlayerItemVideoOutput object to the stream
Attach a CADisplayLink object to the run loop
Extract the current pixel buffer from the AVPlayerItem every time the screen redraws
Convert the pixel buffer to a CoreImage object
Apply filters
Display the image

Setting up AVPlayer

You won’t use the AVPlayerLayer or AVPlayerViewController to display the video, but the AVPlayer plays a central role. The AVPlayer keeps the video in sync with the audio, handles decoding whatever format the stream uses, controls buffering, and more. The basic setup is the same as with any other project:

//create a player
let videoItem = AVPlayerItem(url: streamURL)
self.player = AVPlayer(playerItem: videoItem)

The URL can either point to a local resource or a stream.

Using AVPlayerItemVideoOutput

The AVPlayerItemVideoOutput allows you to query the AVPlayerItem for the pixel buffer (one screen worth of pixels) at any given time. You can specify different pixel formats and other options for the output. For this tutorial, you will you can create the object with a simple instantiation using the default formats.

let playerItemVideoOutput = AVPlayerItemVideoOutput()

Later you will add this video output to your player item. Then you can ask it to generate the pixel buffers as needed. In this tutorial, you will ask for the pixel buffer of the current frame. You could ask for the frame for any timestamp though.

Creating the Display Link

The CADisplayLink class is a specialized NSTimer that synchronizes itself to the screen refresh rate of the display. This ensures that the pixel buffer you extract will be from the correct time of your video stream. It will get rendered at the correct time in the screen refresh. Syncing a standard NSTimer to the screen refresh has always been problematic. With the new variable refresh rate screens that Apple makes, it becomes impossible. However, CADisplayLink adjusts its rate as the screen refresh rate changes. Apple explains how this works in the 2021 WWDC Session Optimize for Variable Refresh Rate Displays. When you create the display link, you give it the name of a function to call each time it executes. A sample for creating a link could look like this:

lazy var displayLink: CADisplayLink = CADisplayLink(target: self,
                                                  selector: #selector(displayLinkFired(link:)))

Elsewhere in the code, a function called displayLinkFired(link: CADisplayLink) extracts the pixel buffer from the AVPlayerItem.

Starting the Player

Apple notes that loading a video file takes a measurable amount of time. It can take even longer when the video file is streaming across a network. AVFoundation allows your program to continue to execute while the setup steps and initial buffering are occurring in the background. You can run into issues if you load a video into AVFoundation and then immediately attempt to work with it. AVFoundation may not show an error or status message, but it will not perform as expected either.

The most important step in this entire process is using an observer to wait until the AVPlayerItem has a .readyToPlay status before attaching the output and displaying link objects. In the example below statusOberserver, player, playerItemVideoOutput and displayLink are all declared at the class level. They need to persist the entire time the video is being used.

//create a player
let videoItem = AVPlayerItem(url: streamURL!)
self.player = AVPlayer(playerItem: videoItem) //1
//*important* add the display link and the output only after it is ready to play
self.statusObserver = videoItem.observe(\.status, //2
                       options: [.new, .old],
                 changeHandler: { playerItem, change in
    if playerItem.status == .readyToPlay { //3
      playerItem.add(self.playerItemVideoOutput)
      self.displayLink.add(to: .main, forMode: .common)
      self.player?.play() //4
    }
})

Here is what to notice about the code above

Creating the player will take a large amount of time, but the program execution will not stop and wait
NSKeyValueObservation is how swift defines key value observers the code will execute every time the videoItem.status property changes values
Checks to see if the .status property is .readyToPlay
After adding the display link and the video output objects start playing the video

Extracting the Pixel Buffer

Once the video begins to play, the display link function gets called for each screen refresh. Here is an example of how to extract the pixel buffer and render it to a UIImageView.

@objc func displayLinkFired(link: CADisplayLink) { //1
  let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
  if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) { //2
    if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
      let frameImage = CIImage(cvImageBuffer: buffer) //3
      //4
      let pixelate = CIFilter(name: "CIPixellate")!
      pixelate.setValue(frameImage, forKey: kCIInputImageKey)
      pixelate.setValue(self.filterSlider.value, forKey: kCIInputScaleKey)
      pixelate.setValue(CIVector(x: frameImage.extent.midX, y: frameImage.extent.midY), forKey: kCIInputCenterKey)
     let newFrame = pixelate.outputImage!.cropped(to: frameImage.extent)
     self.videoView.image = UIImage(ciImage: newFrame) //5
    }
  }
}

Here is what the code is doing:

Marks the func with @objc so that it works with the display link selector
Asks if there is a new pixel buffer to display. Depending on the refresh rate of the screen and the frame rate of the video, there will not always be a new buffer.
After extracting the pixel buffer, convert it to a CoreImage object
Apply any filters. This example applies the CIPixellate filter. It uses a slider to let the user change the intensity as the video plays
Convert the filtered image to a UIImage and assign it to the UIImageView

Now as the pixellate scale changes the image will be filtered but the video will continue to play smoothly.

Going Further

Using the strategy above you should be able to apply different filters to the video stream in real-time. Remember that you only have a few milliseconds though, then the video will begin to stutter. CoreImage filters are optimized to make use of the GPU and should be fast enough in most cases. For better performance, you can render the filtered image to an MTKView and use Metal. Apple has also begun to create Metal Performance Shaders which are even more efficient than CoreImage filters when used with a MetalKit view. As of now, there are many more CoreImage filters, so that may give you the most flexibility.
The strategy in this tutorial is adequate for filtering video streams. If you want to give your users more flexibility with filters and other video tweaks, an editor such as VideoEditorSDK can let you focus on your application’s core functions and let a team of professionals worry about the video. Using VideoEditorSDK your users can apply filters to video as well as text annotations and more.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions or suggestions.

You like what we do? Check our careers page. Even if you can’t find your role listed, we might be interested since you are already here!

How To Add Text to Video in Swift

Walter — Fri, 15 Oct 2021 15:05:59 GMT

In this article, you will see how to use Swift and AVMutableVideoComposition to add a text overlay to a video clip. The code in this article uses Swift 5. Clone this repository for a Swift Playground with the example code.

Setting up

To access the AVFoundation and CoreImage objects, be sure to add these imports to your code

import AVFoundation
import CoreImage.CIFilterBuiltins

Importing the CIFilterBuiltins lets you use autocompletion when working with CIFilters. Otherwise, just import CoreImage. But then you will need to access the filter properties using strings.

When working with video clips and movie files, the basic starting point is the AVAsset class. This class combines all of the timed video and audio tracks that make up a movie. Additionally there may be subtitles and timed metadata or captions.

Creating a Video Composition

After you create a video composition for an asset, you can apply it to an AVPlayerItem to display on screen or to an AVAssetExportSession to write to a file. The composition for this tutorial will use the init(asset:applyingCIFiltersWithHandler:) initializer. Then you can apply CIFilters to each frame of the video.

First, load a movie file as an AVAsset using its URL.

//Fetch a URL for the movie from the bundle
let waterfallURL = Bundle.main.url(forResource: "waterfall", withExtension: "mov")
//Create an AVAsset with the url
let waterfallAsset = AVAsset(url: waterfallURL!)

Now you can create a AVMutableVideoComposition with the asset

let titleComposition = AVMutableVideoComposition(asset: waterfallAsset) {request in
//apply filters here
request.finish(with: request.sourceImage, context: nil)
}

The code above passes the video frame unaltered. The request is the current video frame. It is an AVAsynchronousCIImageFilteringRequest object. The request object has three properties:

the sourceImage which is the video frame as a CIImage
the renderSize which is the size of the video frame
compositionTime which is the timestamp of the current frame

The code in the handler will run once for each frame of the video clip.

Generating a Text Image

Because the sourceImage is a CIImage, you can use any of the over 200 CIFilter objects that Core Image provides. There are two text generator filters: CIAttributedTextImageGenerator and CITextImageGenerator. Either of these will render a string as a CIImage. This tutorial will use the Attributed Text version so you can modify the color and add a shadow.

Start by creating a shadow and then creating a dictionary of attributes to apply to the string.

let whiteShadow = NSShadow()
whiteShadow.shadowBlurRadius = 5
whiteShadow.shadowColor = UIColor.white
let attributes = [
  NSAttributedString.Key.foregroundColor : UIColor.blue,
  NSAttributedString.Key.font : UIFont(name: "Marker Felt", size: 36.0)!,
  NSAttributedString.Key.shadow : whiteShadow
]

Create the attributed string by combining the string and the attributes

let waterfallText = NSAttributedString(string: "Waterfall!", attributes: attributes)

Provide the attributed string and a scale factor to the filter to generate an image like the one shown above.

let textFilter = CIFilter.attributedTextImageGenerator()
textFilter.text = waterfallText
textFilter.scaleFactor = 4.0

The textFilter.outputImage will be the image of the rendered text with the attributes applied. The extent of the outputImage will be a rectangle that is large enough to encompass the text. The text will render on a single line, it doesn’t wrap using this method.

If you were to place the text on the video image at this point, the bottom-left of the text would be at the bottom-left of the video. Unlike UIView objects, the origin point (0,0) is at the bottom-left, not the top-left. To center the text and move it off of the bottom, you can apply a standard CGAffineTransform

let centerHorizontal = (request.renderSize.width - textFilter.outputImage!.extent.width)/2
let moveTextTransform = CGAffineTransform(translationX: centerHorizontal, y: 200)
let positionedText = textFilter.outputImage!.transformed(by: moveTextTransform)

Now finish the pipeline by composing the new text image over the original source image.

positionedText.composited(over: request.sourceImage)

All together, the original composition now becomes

let titleComposition = AVMutableVideoComposition(asset: waterfallAsset) { request in
//Create a white shadow for the text
let whiteShadow = NSShadow()
whiteShadow.shadowBlurRadius = 5
whiteShadow.shadowColor = UIColor.white
let attributes = [
  NSAttributedString.Key.foregroundColor : UIColor.blue,
  NSAttributedString.Key.font : UIFont(name: "Marker Felt", size: 36.0)!,
  NSAttributedString.Key.shadow : whiteShadow
]
//Create an Attributed String
let waterfallText = NSAttributedString(string: "Waterfall!", attributes: attributes)
//Convert attributed string to a CIImage
let textFilter = CIFilter.attributedTextImageGenerator()
textFilter.text = waterfallText
textFilter.scaleFactor = 4.0
//Center text and move 200 px from the origin
//source image is 720 x 1280
let positionedText = textFilter.outputImage!.transformed(by: CGAffineTransform(translationX: (request.renderSize.width - textFilter.outputImage!.extent.width)/2, y: 200))
//Compose text over video image
request.finish(with: positionedText.composited(over: request.sourceImage), context: nil)
}

Displaying The Finished Video

With an AVAsset and an AVMutableVideoComposition you can now combine the two into an AVPlayer to display in an AVPlayerViewController or in your own UIViewController

let waterFallItem = AVPlayerItem(asset: waterfallAsset)
waterFallItem.videoComposition = titleComposition
let player = AVPlayer(playerItem: waterFallItem)

As stated above, you can also combine the asset and the composition and write them to a new movie file using an AVAssetExportSession.

Going Further

The method in this tutorial is suitable for adding watermark or title text to videos. If you want to give users the ability to add custom text, there is more work to do. You will need to create font and color pickers as well as code to position the text in the frame.

You can use an editor such as VideoEditorSDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to use AVMutableVideoComposition to add text to a video clip. Further, you saw how using an SDK such as VideoEditorSDK allows you to annotate and enhance your clips before sharing. Including typography, audio support and video composition, IMG.LY provides a comprehensive solution for mobile video editing – find the documentation here.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

How To Record Screen Video Using ReplayKit for iOS

Walter — Tue, 21 Sep 2021 15:05:08 GMT

In this article, you will see how to use Swift and ReplayKit to add screen recording to your application. You will also see how to use the new rolling clips feature, introduced in iOS 15. The code in this article uses Swift 5 and Xcode 13. Clone this repository for a sample project and example code.

ReplayKit Basics

Apple introduced the ReplayKit framework in 2015 with iOS 9. They have refined and expanded it since then, but it still has a small API. ReplayKit uses the same shared components as AirPlay and the device screen recorder (accessible from the device Control Center). For this reason, ReplayKit functions will not work when using AirPlay or playing video. Also, the screen recording functions do not work on the Xcode simulators. You should test using a physical device.

Your app makes all its calls to the RPScreenRecorder.shared() object. This is a singleton object for your app that ReplayKit provides. There is no need to store it in a variable or pass it around in your code.

Rolling Clip Recording

In iOS 15, Apple introduced rolling clip recording. ReplayKit will maintain an updated video buffer of the most recent fifteen seconds. Your app can request the video at any time. ReplayKit comes from GameCenter. Apple’s idea for this feature is for creating a short, sharable video of the moments before an exciting event in a game. Another use can be for a user to record the steps leading up to a defect during testing. When the user is unlikely to know in advance that they will want a screen recording, rolling clips may be a good solution.

To start rolling clip buffering, include a method such as this one

RPScreenRecorder.shared().startClipBuffering { err in
  if let err = err {
    //either the user denied permission or something like
    //AVPlayer or AirPlay is using the shared recorder resources.
    print("Error attempting to start buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering started.")
  }
}

Screen recording has privacy implications. ReplayKit presents the user with a modal, requesting to record before it starts. If the user denies the request or if the buffering is unable to start, ReplayKit passes an Error object to the closure. Unlike other privacy modals, you do not have an opportunity to add your own language to the message. The modal appears as soon as you execute the .startClipBuffering command, so be sure that the user is not surprised by it. Execute the command at a logical transition in your application. After the user grants permissions, the modal will not appear again unless eight minutes pass before the next call to .startClipBuffering.

While the clip buffering runs, at any time, you can save the buffer to a URL on the device. The clip you save can be any duration up to fifteen seconds. If the recorder is unable to write the clip to disk, it will send an error.

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().exportClip(to: clipURL, duration: TimeInterval(15)) { err in
  if let err = err {
    print("An error? \(err.localizedDescription )")
  } else {
    print("Clip saved to url \(clipURL)")
}

The code above stores a fifteen-second clip at a URL in the user’s temporary directory. The temporary directory gets cleared by the system, but not while the app is running. To ensure that the files are not deleted except by the user, store them in the user’s documents directory instead.
When your app is done collecting video, use .stopClipBuffering to allow the system to clean up resources and stop the recording.

RPScreenRecorder.shared().stopClipBuffering { err in
  if let err = err {
    print("Error attempting to stop buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering stopped.")
  }
}

Recording Screen and App Audio

Rolling clip buffering runs in the background. You can also allow the user to control when to start and stop screen recording. The API looks similar and is available since iOS 10.

RPScreenRecorder.shared().startRecording { err in
  guard err == nil else { print(err.debugDescription); return }
  //code to display recording indicator
}

As with the rolling clip buffering, the system will display a permissions dialog. If the user denies permission you will get an error and the recording will not start.

The same eight-minute rule applies as for rolling clip recording. But, since the user has just taken some action to start recording, they should expect to see this modal.
Apple provides a basic view controller for editing and sharing the video.

ReplayKit passes the RPPreviewViewController to the .stopRecording handler.

RPScreenRecorder.shared().stopRecording { preview, err in
  guard let preview = preview else { print("no preview window"); return }
  //update recording controls
  preview.modalPresentationStyle = .overFullScreen
  preview.previewControllerDelegate = self
  self.present(preview, animated: true)
}

The preview object above is the RPPreviewViewController editor. You can display it as a modal or as a popover depending on your needs. The Apple provided editor will allow the user to save to their camera roll or share the clip after editing.

Notice in the code above that there is a previewControllerDelegate. You will need to add some code to the delegate to cleanup and dismiss the editor when the user is done sharing or saving. An example is below.

func previewControllerDidFinish(_ previewController: RPPreviewViewController) {
  previewController.dismiss(animated: true) { [weak self] in
  //reset the UI and show the recording controls
  }
}

Using Many `UIWindow` Objects

The ReplayKit recording will only capture the main window of your application. When designing for screen recording, you can place any UI elements in a separate UIWindow object. Although applications typically only have one UIWindow, iOS allows many windows. A UIWindow requires a .rootViewController to provide content. Then use it like any other view in your application.
The UI for this application consists of three UIWindow objects.

Apple places the status bar in a separate window from your application. The recording controls are in a separate window from the main application and will not appear in the screen recording.

To add some recording controls or a recording indicator to your app. Create a new UIWindow and add it to the same windowScene as your view.

controlsWindow = UIWindow(frame: CGRect(x: 0, y: 0, width: view.frame.width, height: 45 + view.safeAreaInsets.top)) //1
controlsWindow?.windowScene = view.window?.windowScene //2
controlsWindow?.makeKeyAndVisible() //3
let recordingIndicator = UIButton.systemButton(with: UIImage(systemName: "record.circle")!, target: self, action: #selector(recordingToggled(_:))) //4
let vc = UIViewController()
controlsWindow?.rootViewController = vc //5
vc.view.addSubview(recordingIndicator) //6
recordingIndicator.center = CGPoint(x: vc.view.center.x, y: vc.view.center.y + 20)

The code above creates the red/blue recording button and banner at the top of the view. It relies on a controlsWindow variable created at the class level. Here is what it is doing:

Creates a new UIWindow that is the width of the view and tall enough for the button and the safe area (the area around the notch and status bar)
Adds the window to the same windowScene as the current view
Makes the window visible (UIWindow objects are hidden when created) and brings it to the front of the stack of windows
Creates a button and links it to a method in your class recordingToggled(_:)
Adds a view controller to the window so that there will be some content
Adds the button to the view controller

Because the recording indicator and button are in a separate window, they will not appear in any screen recording. Be mindful when showing modal views or alert views, as they may appear behind the extra window. A good strategy is to hide the extra window when showing these views. You can use a similar strategy to hide any other UI elements that should not appear in screen recordings.

Going Further

Though Apple provides a basic editor, you can provide your own editing tools. The examples above showed the rolling clip editor writes the video to the disk. The regular recorder can also write the video to disk. You can stop recording using this code:

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().stopRecording(withOutput: clipURL) { err in
  //check for an error and process the url
}

This method would replace the .stopRecording { preview, err in method shown above.
With the video saved, you can use an editor such as the VideoEditor SDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to capture a rolling screen recording as well as how to capture the screen manually. Further, you saw how using an SDK such as VideoEditor SDK allows you to annotate and enhance your clips before sharing.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

How to Trim and Crop Video in Swift

Walter — Tue, 31 Aug 2021 16:03:25 GMT

In this article, you will see how to use Swift and AVKit to crop a video clip and how to trim a video’s timeline. Then you will learn how to use AVAssetExportSession to write your edited video to disk. The code in this article uses Swift 5 and Xcode 12.5. Clone this repository for a sample project and example code.

Anatomy of a Video File

The AVFoundation and AVKit frameworks are what Swift and iOS use to manage audio and video. When starting out, you may use an AVPlayerViewController for playback with the same controls and features as the native players. You can also use an AVPlayer object and provide your own playback and editing controls. Whichever you choose, you begin by loading a media file (.mp3, .mov, etc.) into an AVPlayerItem. Inside the AVPlayerItem, the media becomes an AVAsset which may have many tracks of video, audio, text, closed captions etc.

To manipulate an AVAsset, Apple provides the AVComposition class. The AVComposition can act on a single track or many tracks to filter and mix underlying media and produces a single output. An AVPlayer shows the output of an AVComposition on a device. When it is time to export, you can use an AVAssetExportSession with the same AVComposition to write to disk or upload to a server.

Cropping Video to a Rectangle

Apple provides some different AVComposition classes optimized for common tasks. Apple recommends using one of these classes instead of writing custom AVComposition classes whenever possible. The reason for this is that when Apple introduces new technologies (like HDR Video), they will ensure it works with their classes. If you’ve written your own, you will have to update it to work with the new technologies. In this article you will crop the video to a rectangle and let the audio pass through as recorded. A good class to use for this task will be the AVMutableVideoComposition with the init(asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void) initializer. This will allow you to apply a CIFilter to each frame of the video. Apple provides an efficient cropping filter called CICrop.

func transformVideo(item: AVPlayerItem, cropRect: CGRect) {

  let cropScaleComposition = AVMutableVideoComposition(asset: item.asset, applyingCIFiltersWithHandler: {request in

    let cropFilter = CIFilter(name: "CICrop")! //1
    cropFilter.setValue(request.sourceImage, forKey: kCIInputImageKey) //2
    cropFilter.setValue(CIVector(cgRect: cropRect), forKey: "inputRectangle")


    let imageAtOrigin = cropFilter.outputImage!.transformed(by: CGAffineTransform(translationX: -cropRect.origin.x, y: -cropRect.origin.y)) //3

    request.finish(with: imageAtOrigin, context: nil) //4
    })

  cropScaleComposition.renderSize = cropRect.size //5
  item.videoComposition = cropScaleComposition  //6
}

The applyingCIFiltersWithHandler will execute for every frame in the asset of the AVPlayerItem. Here is what the code above will do:

Create a CICrop filter (note that this demo uses ! to force unwrap, production code should handle failure)
Add the .sourceImage (a CIImage) from the request to the filter.
Move the cropped image to the origin of the video frame. When you resize the frame (step 4) it will resize from the origin.
Output the transformed frame image
Set the size of the video frame to the cropped size
Attach the composition to the videoComposition property of the asset

Notice that the transformation does not alter the underlying asset. It creates a videoComposition filter to attach to the item during playback. After the transformation, AVPlayer will display the new creation when executing its .play() function.

Sharp eyed readers will have noticed that the init method mentions “applying CIFilters” plural. Instead of one filter, you can create an entire pipeline of CIFilter objects to manipulate the visual properties of the frames. The request object also contains the compositionTime so filters can change at different parts of the video.

Trimming the Time of a Video

When an AVPlayer loads an AVItem the start time of the video will be at CMTime.zero. The end will be the duration property of the AVItem. To adjust the playback times, you call the .seek function to move to the new start time and set the forwardPlaybackEndTime to the new end time. Now the player will only play the part of the clip between those times.

//The player object is already created and configured to play video in the ViewController

//load a video .mov file
self.player = AVPlayer(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)

//Set the start time to 5 seconds.
let startTime = CMTimeMakeWithSeconds(5, preferredTimescale: 600)

//Convert the duration of the video to seconds
let videoDurationInSeconds = self.player!.currentItem!.duration.seconds

//Subtract 5 seconds from the end time
let endTime = CMTimeMakeWithSeconds(videoDurationInSeconds - 5, preferredTimescale: 600)

//Assign the new values to the start and end time
self.player?.seek(to: startTime)
self.player?.currentItem?.forwardPlaybackEndTime = endTime

//Play the video
self.player?.play()

The CMTime (Core Media Time) object uses timescales and Int values to map to the individual frames of tracks. Video tracks commonly come to your app with 24, 30, 60 or 120 frames per second. Converting between these different formats using Floats or Doubles would be imprecise. When converting seconds to CMTime with video, 600 is the commonly used timescale because it is a common multiple of all of the standard frames per second.

Exporting the Video

As with the videoComposition property, setting the start and end times of the player only modifies the playback of the video clip. The underlying asset remains unchanged. When you use the AVAssetExportSession, the new video file will have the change. The export session applies time range and video composition objects as it writes the file. A function to export might look like:

func export(_ asset: AVAsset, to outputMovieURL: URL, startTime: CMTime, endTime: CMTime, composition: AVVideoComposition) {

    //Create trim range
  let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)

    //delete any old file
  do {
    try FileManager.default.removeItem(at: outputMovieURL)
  } catch {
    print("Could not remove file \(error.localizedDescription)")
  }

    //create exporter
  let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)

    //configure exporter
  exporter?.videoComposition = composition
  exporter?.outputURL = outputMovieURL
  exporter?.outputFileType = .mov
  exporter?.timeRange = timeRange

    //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error {
        print("failed \(error.localizedDescription)")
      } else {
        print("Video saved to \(outputMovieURL)")
      }
    }
  })
}

Going Further

AVKit and AVFoundation provide simple objects for manipulating video files. The difficulty when working with video and audio tracks usually comes while providing editing controls for a user. The image below shows how the original video dimensions might change from creation to display.

The original 2160x3840 video appears on an iPhone in a 357x635 frame. Additionally the origin point (green dot) of the video file and the origin point of the UIViews are not equal. Passing the frame of the red, cropping rectangle to an AVComposition would not work as expected. Inside of the AVComposition the video resumes it’s 720x1280 dimensions while the 250x250 cropping rectangle would remain 250x250.

Before using a UIView rectangle with an AVComposition it needs to resize and the origin point needs to align to the video’s origin. Your app must apply a CGAffineTransform to reorient the origin points.

var cropRect = self.croppingView.frame //1
let originFlipTransform = CGAffineTransform(scaleX: 1, y: -1)
let frameTranslateTransform = CGAffineTransform(translationX: 0, y: renderingSize.height)
cropRect = cropRect.applying(originFlipTransform) //2
cropRect = cropRect.applying(frameTranslateTransform) //3

Now you apply regular ratio math to the dimensions of the cropping rectangle to match the video dimensions.

let renderingSize = playerItem.presentationSize

let xFactor = renderingSize.width / playerView.bounds.size.width
let yFactor = renderingSize.height / playerView.bounds.size.height

let newX = croppingView.frame.origin.x * xFactor
let newW = croppingView.frame.width * xFactor
let newY = croppingView.frame.origin.y * yFactor
let newH = croppingView.frame.height * yFactor
var cropRect = CGRect(x: newX, y: newY, width: newW, height: newH)

The transformations above are standard when working with AVKit and UIKit or SwiftUI in an app. The math is not complex, but it is tedious. Unless your app is a custom video editing application, you may find that using a commercial solution such as VideoEditorSDK is a better approach. By adding the VideoEditorSDK, your app can have professional appearing trim and crop controls as well as filtering, text and more.

Wrapping Up

In this article, you saw how to crop a video image and how to trim time from a track, all in Swift. Further, you saw how using an SDK such as VideoEditorSDK allows you to provide full-featured video editing for any application.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How to Create Image Filters for iOS

Walter — Tue, 24 Aug 2021 08:09:37 GMT

In this tutorial, you will learn how to write a custom CIFilter in Metal and Swift. You can use this filter in any CoreImage pipeline.

Apple offers over 200 image filters in the CoreImage framework, but sometimes you need a little extra to make your images perfect. Apple provides two ways to create customize image filters. You can chain CIFilters together in a CIFilter subclass or wrap a CIKernel in a CIFilter subclass. Either method creates a filter that CoreImage can execute on the GPU. That makes the filter fast. CoreImage filters can filter live video without impacting the frame rate. Before iOS 11 the only way to write a custom CIKernel was to pass a string containing the code to the GPU at runtime using the OpenGL Shading Language. This had two drawbacks: it was difficult to find errors in the string of commands and the kernel was not compiled until runtime. This tutorial will show you how to add a custom filter with a Metal-based CIKernel to any Swift project in Xcode. The code samples in this tutorial use Xcode 12.5 and Swift 5.

Clone this repository for a sample project and example code that supports this tutorial. The repo also contains some other custom filters, demonstrating other aspects of Metal filters.

Adding Metal Support to an Xcode Project

The first step is to add a Metal source code file to your project.

Find the “Metal File” template, select it and click Next. Name the file and save it. At compile time, Xcode combines all of the .metal files in your project into a single .metallib file. So, organize your Metal code into lots of files or just one file as you prefer. The last step before you start writing code is to add compiler flags for CIKernel objects.

In the Build Settings, find the other Metal Compiler Flags and add an entry of -fcikernel.

Then find the Other Metal Linker Flags and add an entry of -cikernel.

Important note: any project without .metal files hides the Metal Compiler and Linker sections from build Settings.

Setting Up a Metal File for CoreImage Kernels

Metal is a general-purpose technology for writing code that will execute on the GPU. The compiler flags tell Metal that you will be writing code to work with CoreImage. With a Metal source file in your project and the compiler flags set, you’re finally ready to write some code! Open the Metal file you created above. It should be empty except for

#include <metal_stdlib>
using namespace metal;

Below these lines, import the CoreImage headers by adding:

#include <CoreImage/CoreImage.h>

and add a stub for your filter code with:

extern "C" {
  namespace coreimage {
    // KERNEL GOES HERE
  }
}

Prefix your kernel code with extern "C" and the CoreImage namespace. You write kernel code in the Metal Shader Language, which is a variation of C. If you only know Swift, you should be able to rely on Xcode’s code-completion and symbol lookup to help you as you’re learning to write Metal code. In addition to the general language reference, Apple provides a Metal Shading Language for Core Image Kernels document.

The example below creates a CIColorKernel. This kernel is optimized for changing the color value of each pixel in an image. It will change each pixel to a grayscale version of itself. A filter applying his kernel will send the color value of each pixel of the image to the kernel.

float4 grayscaleFilterKernel(sample_t s) { //1
  float gray = (s.r + s.g + s.b) / 3;      //2
  return float4(gray, gray, gray, s.a);    //3
}

Here is how this kernel works:

The filter provides the color of the current pixel to the kernel as a sample_t type. The kernel will return the new color for that pixel as a float4 type. The sample_t type is equivalent to a float4 type and is only named differently because of historical convention. A float4 type contains four float values. In the case of color, they are the red, green, blue and alpha values for the pixel. Each float has a value between zero and 1.
Calculate a grayscale version of the pixel by averaging the three color channels.
Create a new float4 with the averaged value for the three color channels and the original alpha value for the fourth.

CoreImage also provides a CIWarpKernel class that is optimized for changing the position of each pixel and a CIBlendKernel for blending two images. For more complex tasks, CoreImage also provides a general CIKernel. Apple suggests that chaining together optimized kernels in a pipeline is usually more efficient than trying to write a larger, general kernel. A color-optimized kernel only has access to the color of the current pixel. A transform-optimized kernel can only affect the position of the current pixel. A general kernel can impact color and position as well as read other values from the source image.

Wrapping a `CIKernel` with a `CIFilter`

Now you’ll make a CIFilter subclass as the interface between the Metal code and your Swift code. Every CIFilter has an outputImage variable that returns a CIImage. Generally, input variables for a filter begin with input. The grayscale filter you are making has a single input and a single output. Create a new Swift file in your project and title it “GrayscaleFilter.swift”. Then be sure to import CoreImage by adding

import CoreImage

to the top of the file. Then create the filter as a subclass of CIFilter.

class GrayscaleFilter: CIFilter {

Next, create a kernel variable as either static or lazy, so that it only gets created one time, regardless of how often it’s called.

static var kernel: CIColorKernel = { () -> CIColorKernel in
    let url = Bundle.main.url(forResource: "default",
                            withExtension: "metallib")! //1
    let data = try! Data(contentsOf: url)
    return try! CIColorKernel(functionName: "grayscaleFilterKernel",
                      fromMetalLibraryData: data) //2
}()

Look in the bundle for the compiled metal code. The metal code will have a filename of “default”.
The .metalib may contain many kernels so use the one called “grayscaleFilterKernel”

Now add a variable to hold the input image.

var inputImage: CIImage?

Finally override the outputImage variable.

override var outputImage: CIImage? {
    guard let inputImage = inputImage else { return .none }
    return GrayscaleFilter.kernel.apply(extent: inputImage.extent,
                                   roiCallback: { (index, rect) -> CGRect in
                                                  return rect
                                  }, arguments: [inputImage])
}

The apply function of the kernel takes an extent argument. For a CIImage the extent property is the width and height of the image. The function has an roiCallback that allows you to specify what part of the image the kernel uses for each calculation. For a color filter, you generally just pass back the rect. Finally, pass in any arguments as an array. Notice that there is not any type checking here. It is up to you to pass in the correct types in the correct order.

Using the Filter

With the Metal code wrapped in a CIFilter you can now apply the filter images the same as using one of the built-in filters. Create an instance of the filter, then assign the inputImage variable a CIImage and display the outputImage. Your code might look something like this:

let filter = GrayscaleFilter()
filter.inputImage = image

displayView.image = UIImage(ciImage: (filter.outputImage ?? image) ?? CIImage())

and it can render a grayscale version of an image, like in this example.

Going Further

You can write custom filters and chain them to the built-in filters to invent new effects. Many of the math formulas for effects are available on the Internet. Most are easy to translate into the Metal Shader Language to make them perfect for your application. Writing an entire image or video editing application to showcase your filters is a much larger task. Using an SDK like the IMG.LY PhotoEditorSDK or VideoEditorSDK can help you provide an app that has the features that users will expect in addition to your cool filters.

To add your filter to the PhotoEditorSDK you first wrap your CIFilter with an Effect. For a basic filter, you only need to override the newEffectFilter variable.

import PhotoEditorSDK

class GrayscaleEffect: Effect {
  override var newEffectFilter: CIFilter? {
      GrayscaleFilter()
  }
}

Then add the filter to the Effect.all array when configuring the SDK, using something like

Effect.all = [NoEffect(), GrayscaleEffect(identifier: "grayscaleFilter", displayName: "Best Gray")]

Now your filter appears the same as the other 60+ filters that ship with the SDK.

It applies to live camera previews as well as still images.

Wrapping Up

In this tutorial, you learned how to configure Xcode for custom Metal code. You also learned how to create a color filter kernel in Metal and wrap it in a CIFilter. By combining your filters with Apple’s filters in pipelines, you can create unique effects for your next app. Further, using an SDK such as PhotoEditorSDK or VideoEditorSDK allows you to showcase your filters in a full-featured image or video editing application. The PhotoEditor SDK for iOS documentation will give you deeper look into all other image adjustments including, ofcourse, the image filtering we discussed above.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

Walter – IMG.LY Blog

Cutting Through The Jungle: An In-depth Review of Cloud GPU Providers to Train Your AI Models in 2024

Navigating the World of AI Models Hosting

Kinds of Cloud Hosting for AI Models

Serverless Hosting

Cloud GPU Hosting

Providers

Serverless Providers

Runpod IO (Serverless)

Vast AI

Paperspace

Banana Dev

CoreWeave

Modal

ComfyICU

Replicate

GPU Cloud Providers

Genesis Cloud

Fly IO

Runpod IO (Cloud GPU)

Lamda Labs

Together AI

Established Providers

Conclusion

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Setting Up Your Project

Asking for Permissions

Making Video Recordings

Configuring the Camera

Presenting the Controller

Building Your Creation

Configuring the Editors Assets

Exporting the Finished Video

Setting Up a Playback Controller

Where to Go From Here

Build a Simple Real-Time Video Editor with Metal for iOS

Setting Up a MTKView

Drawing in an MTKView

Working with Local Files

Working with Streams

Working with the Cameras

Going Further

How to Remove Backgrounds Using Core ML

Using Vision to Segment People

Choosing a Model

Core ML and Xcode

The DeepLabV3 Model

Segmentation with DeepLab

Going Further

Wrapping Up

Working with Large Video and Image Files on iOS with Swift

Writing an Image to Disk

Compression Options for JPEG

Compressing Data with zlib

Transferring Data With a Server

Uploading a Large File

Configuring Background Transfers

Resizing an image

Resizing a Video

Going Further

Wrapping Up

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

How to Make an Animated GIF Using Swift

Animated GIF Basics

The Basic Strategy

Creating an Empty File

Adding Images

Finalizing the File

Collecting Frame Images

Extracting Images from Video

Wrapping Up

How to Make Videos from Still Images with AVFoundation and Swift

The Problem with an Image

Writing Buffers using AVAssetWriter

Create the Pixel Buffer

Configure AVAssetWriter

Write the pixel buffer to the video file

Creating a Blank Video

Adding a Composition

Going Further

Compressing Data with `zlib`

Writing Buffers using `AVAssetWriter`

Using Many `UIWindow` Objects