iOS – IMG.LY Blog

Building Custom Variable SF Symbols in Figma for our CreativeEditor SDK for iOS

Dustin — Tue, 13 May 2025 07:00:06 GMT

If you’ve ever wrestled with vector paths in Figma just to get a custom variable SF Symbol out the door, you know the frustration: one missed node here, one stray smoothing there, and suddenly your interpolation breaks. Plus, you would design your icons within an SVG file you exported beforehand from the SF Symbols App.

When Apple introduced SF Symbols in 2019, they changed the game for iOS design. Suddenly, we had access to thousands of scalable, weight-variable icons that integrate seamlessly with system type. Our CreativeEditor SDK (CE.SDK) is a customizable design and video editor you can integrate into your product. Yet, as it grew, we needed icons as unique as its use cases—icons that simply didn’t exist in Apple’s library.

In this article, I’ll share our journey from relying on two community Figma plugins—SF Symbols Organizer and Vector Path Editor—to building our own tool, SF Symbol Generator. Along the way, I’ll show you how we wrestled with aspect ratios, node counts, and path order, and how we eventually streamlined the entire process into a few simple clicks.

What Are Variable SF Symbols?

At their core, variable SF Symbols behave much like variable fonts: they allow you to interpolate smoothly between multiple “master” designs, rather than shipping separate static assets for each weight. To create one, you supply three distinct versions of your icon—typically Ultra Light, Regular, and Black—each sharing the exact same path structure and node count. When the SF Symbols app or iOS runtime renders your symbol, it calculates intermediate weights on the fly by linearly interpolating between those three masters. This approach not only shrinks your asset footprint but also gives you rich typographic control, letting you fine-tune how heavy or light an icon appears in any context.

The Early Struggle

From the moment we discovered the Figma plugin SF Symbols Organizer, it became an indispensable part of our workflow. Suddenly, what had been a tedious, manual export process turned into a single click: select your three masters, batch-export them as SF Symbols-ready SVGs, and you were halfway home. The plugin’s enforcement of a 1:1 aspect-ratio rule was actually a blessing when you’re just getting started, ensuring that every icon fit out of the gate.

However, as our icon library expanded beyond a handful of simple glyphs, we began to feel the boundaries of that magic square. Some of our more ambitious designs needed wider canvases or asymmetrical compositions, and shoehorning them into perfect squares led to awkward scaling and extra cropping steps. Meanwhile, although SF Symbols Organizer reliably generated valid SVGs, it didn’t warn us about deeper issues—like mismatched node counts or flipped path directions—until we imported into the SF Symbols app and discovered the interpolation errors firsthand.

That’s where Vector Path Editor pitched in. It couldn’t automate our entire workflow, but it did let us quickly correct path order and direction when Figma’s flattening shuffled our nodes. Even with that help, though, we still found ourselves jumping between tools to catch errors after the fact—proof that, while both plugins were fantastic at what they did, we needed something more proactive.

Yet, as powerful as these two plugins were together, their reactive nature meant we still spent long sessions hopping between Figma and Apple’s toolchain, correcting mistakes after they happened instead of catching them up front. It was a great start—and without these plugins, we wouldn’t have made it this far—but we knew there had to be a way to bake most of the missing quality checks into our design process itself.

Why We Wanted “SF Symbol Generator”

After enough trial and error, we realized we needed more than just export and pray. We needed flexibility and foresight.

First, we didn’t want our icons locked into a square. Some of our designs needed wider canvases. Second, we craved live feedback. We wanted to know if our three icons had matching node counts, aligned first nodes, and consistent path winding before exporting. Lastly, we longed for a single, unified workflow that could handle scanning, validating, and exporting without leaving Figma.

It quickly became clear that the only way to achieve this was to build our own plugin. And so SF Symbol Generator was born.

Inside SF Symbol Generator

SF Symbol Generator transforms the way we build custom symbols. Rather than flattening paths prematurely, it encourages us to keep boolean groups intact, flattening only at export time. When you run the plugin, it scans all three masters in situ, then displays a detailed readout of node counts, first-node anchors, and path directions. Any discrepancies are highlighted in red, so you can go back to your designs and fix them.

*Pro Tip: Sometimes Figma’s automatic flattening doesn’t handle rounded corners cleanly and can introduce extra nodes. If the plugin’s scan flags a weight with mismatched node counts or errant anchors, you can manually flatten that master ahead of time, clean up extra nodes using Figma’s vector editing tools, and then re-run the scan to ensure perfect parity.*

Once everything checks out, a single click exports a ready-to-use variable SVG bundle—no detours into external editors required.

Our Step-By-Step Workflow

Our process now is refreshingly simple. We start in Figma by creating a 32-pixel-tall frame that can have any width you need.

Inside that frame, we draw our Regular-weight icon using a 2.45-pixel stroke and a 3-pixel corner radius. We then duplicate the frame twice and adjust the strokes for Ultra Light (0.8 px, 2.5 px radius) and Black (4.95 px, 3.5 px radius). iOS commonly incorporates softer corners in their icon design, and you can achieve them with Figma’s corner smoothing. Though, it’s important to keep Figma’s corner smoothing at 0%, because even a sliver of smoothing can throw off interpolation. We also center-align our strokes wherever possible so that the shapes grow uniformly when the weight changes.

	Stroke size	Border radius
Ultra Thin	0.8	2.5
Regular	2.45	3
Black	4.95–3.5	3.5

With your three master frames selected, launch SF Symbol Generator. In the plugin’s Variable panel, assign each frame to its corresponding weight—Ultra Light, Regular, and Black. If your frames are named correctly (for example, “Icon Ultra Light,” “Icon Regular,” “Icon Black”), the plugin will detect and map them automatically. As soon as each weight is assigned, the previews immediately show node counts, first-node positions, and path directions. Any mismatches appear in red, at which point you might need to go back and manually flatten that master, clean up extra nodes (for example, Figma sometimes adds unwanted anchors on rounded corners), and re-scan them again. When everything looks great, just hit Export to generate your ready-to-use variable SVG bundle.

Importing into the SF Symbols App

Once your variable SVG bundle is exported, the final import into Apple’s SF Symbols app is straightforward. Open the SF Symbols application (available on Apple’s developer site), switch to the Custom Symbols tab, and drag your SVG file into the panel. Your new icons will appear alongside Apple’s built-in set, ready to be organized into collections or shared with teammates. When you return to Xcode, your custom symbols show up just like any system glyph, complete with full support for dynamic weights.

Optional Extra Step: Preparing More Complex Symbols

If your icon consists of multiple layers or requires special styling per layer, take this extra step to ensure it renders perfectly in all weights. After dragging your SVG into the SF Symbols app:

Select your newly imported symbol and click the Rendering Inspector (the segmented button with the brush icon) on the right side.
You’ll see each layer exposed as a separate sub-path. You can create more layers and even group them to change them individually as well.
Tweak opacity, colors, or layer order at different render modes to achieve the exact look you want.

Once you’re satisfied, select and export the symbol again from the SF Symbols app (File > Export Symbol), and it’s truly ready for production.

Lessons Learned

Building SF Symbol Generator taught us several key lessons.
First, node parity is king: the number and order of path segments must match across masters. Early validation saves hours, since catching mismatches before export means fewer cycles in Apple’s SF Symbols app.
Second, non-destructive workflows pay dividends: boolean groups let you iterate quickly without losing stroke data.
And finally, automation is your friend: what once took dozens of manual steps now happens in a single click.

Conclusion

Variable SF Symbols unlock a level of typographic and iconographic dynamism that feels truly native to iOS. While community tools like SF Symbols Organizer and Vector Path Editor can get you pretty far already, building our own SF Symbol Generator transformed our workflow from reactive to proactive. Now, we design freely with any aspect ratio, validate instantly, and export with confidence—no more praying, just seamless symbol-making.

For our CE.SDK for iOS, we’ll be gradually rolling out updated Custom Symbols created with SF Symbol Generator, ensuring every icon feels right at home on Apple devices. Stay tuned for those releases!

If you’re ready to streamline your own custom SF Symbol workflow, check out SF Symbol Generator on the Figma Community and take it for a spin. Happy symbol-making!

3,000+ creative professionals gain early access to new features and updates—don’t miss out, and subscribe to our newsletter.

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Walter — Thu, 30 Mar 2023 11:20:26 GMT

Now that TikTok is facing a ban in the US any day now, we better wait in the wings with an alternative ready to go and scoop up those millions of hobby dancers, micro bloggers, and would-be influencers.

In this article, you’ll learn how to use Swift, SwiftUI, and the IMG.LY CreativeEditor SDK to build a simple iOS app for recording, editing, and viewing short videos. This app features views providing core functionalities similar to making a post in a video-sharing platform like TikTok.

The editing controller is the central hub. It allows users to capture video clips and enhance their videos by adding filters, stickers, music, and other effects. CreativeEditor SDK calls the overall project a “scene” and the parts (clips, stickers, text, etc.) “blocks”. Once the scene is ready, the user can compose it into a standard video file and export it. Finally, the playback controller showcases the finished projects using Apple’s standard VideoPlayer struct. For recording and editing, the CreativeEditor SDK’s Camera and Video Editor handle the heavy lifting, enabling swift development of a robust app. You can easily extend its features, filters, and creative options as your users demand.

Follow along to see how to capture video, enhance it, and export the finished product as a polished movie file.

Setting Up Your Project

You can try out the TikTok clone by cloning the GitHub repository supporting the article. Otherwise, follow this step-by-step tutorial and learn how to build the TikTok clone with CreativeEditor SDK by yourself.

To add the CreativeEditor SDK to an Xcode project, include it with your other package dependencies. Use the URL below to add the package, either to your Package.swift file or using File > Add Packages...

https://github.com/imgly/IMGLYUI-swift

The package includes a number of different frameworks:

IMGLYUI, an umbrella framework of all of the different editors and a camera.
IMGLYCamera, a standalone camera View. This is the same camera as the editors use.
IMGLYApparelEditor, IMGLYDesignEditor, IMGLYPostcardEditor, and IMGLYVideoEditor which are four different configurations to support different use cases.

Each of these frameworks uses the IMG.LY Engine for image processing. To include them in your project, navigate to the ‘Frameworks, Libraries, and Embedded Content’ section in your app’s General settings. Click the ’+’ button and select the desired frameworks. Initially, consider adding the IMGLYUI package, as it encompasses all other packages. Before releasing your app, review and include only the necessary components to minimize app size and avoid unused code.

Once Xcode downloads and resolves the packages, you just need to include IMGLYVideoEditor in any of the files that will reference the editor in your app. When you’re working with the underlying engine directly you’ll also want to include IMGLYEngine. You’ll only work with the engine as you start to customize the workflow.

Asking for Permissions

Before any app can access the phone’s microphone, camera, documents or photo library, the user must specifically give permission to record audio or video as well as access the user’s Photo library. If your app doesn’t secure these permissions properly before trying to use the camera the app will crash and also probably won’t get through Apple’s app review. In earlier versions of iOS you could develop an app on the Simulator without asking permissions, but always on the device the app will crash. In current versions of iOS the app crashes regardless of platform. The dialogue requesting permissions is a system dialog and looks like this:

You do not have the ability to change this dialog. You can supply the reason you are asking for the permission using a key in the info.plist file in the Xcode project. In the example above “Lets you record videos” is in the info.plist. For the video camera you will have to ask for video and audio permission. The first step is to add a short message to the user in the info.plist.

For video the key to add is NSCameraUsageDescription and for the microphone you need to add NSMicrophoneUsageDescription. Whenever your app attempts to use the camera, iOS will first check to see if the user has already granted access. If not, it will then display the dialogs using your entries in the info.plist or crash if you have not provided entries. The user might be surprised by these dialog boxes and may accidentally tap Don't Allow if they are trying to quickly launch the camera. It is better practice to set up some kind of onboarding view and secure the permissions before you need them. You might have a view that explains what you are going to ask for and then displays the permission dialog.

switch AVCaptureDevice.authorizationStatus(for: .video) {
  case .authorized:
    //the user has authroized permission!
    break
  case .denied:
    //the user has previously denied permission!
    //perhaps we should ask them again
    break
  case .restricted:
    //the user cannot grant permission!
    break
  case .notDetermined:
    //the user has never been asked for permission
    //so let's ask them now
    AVCaptureDevice.requestAccess(for: .video) { accessGranted in
      if accessGranted {
        //the user has authorized permission!
      } else {
        //the user has denied permission!
      }
    }
  @unknown default:
    break
}

The snippet above lets your app read the permission status for the video camera and ask for permission if it has not been granted. To request access to the microphone pass .audio to the AVCaptureDevice.authorizationStatus(for:) and AVCaptureDevice.requestAccess(for:) functions. The AVCaptureDevice.requestAccess(for:) command is what displays the system dialog actually requesting access. The accessGranted variable reports back to your app what the user chose. You can take care of getting the permissions any time but when the CreativeEditor SDK first launches it will attempt to access the camera, so the dialog will appear if the user has not already granted permission.

The .restricted case is fairly new. In this case, there are policies on the phone that prohibit the user from granting permission to your app. In addition to asking permissions during onboarding, it is good practice to check for permission every time before you attempt to run any code that uses the camera. The user can change permissions using the Settings app on their phone at any time. If the user has denied permissions and you present the video camera anyway, your app will record black video frames and silence on audio tracks.

In addition to asking for camera and microphone permissions, your app will probably want to access the photos on the user’s phone. You will need to add an entry to the info.plist for NSPhotoLibraryUsageDescription. As with the camera and the microphone, the dialog will appear when the Video Editor first attempts access, but you can give your user some ability to grant permission during onboarding. For the user’s photos, you can check the authorization status using

let status = PHPhotoLibrary.authroizationStatus(for: .readWrite)

As with the camera, the photo library has PHPhotoLibrary.requestAuthroization(for: .readWrite) but instead of just returning a “granted” or “denied” status, this command returns the actual new status. In addition to the status values for the camera, the photo library may return a .limited status meaning that the user has granted permission to only some of the photos. If the user has chosen to share only some of their photos, Apple provides some views you can present so that the user can manage the photos. Any videos that your app saves to the user’s photo library will always be available when the user has chosen .limited. You can read more about how to work with permissions and the user’s photo library by reading this Apple article Delivering an Enhanced Privacy Experience in Your Photos App.

Making Video Recordings

The start of a great TikTok is the camera. When our user wants to make a new video clip with our app, they tap on the button at the bottom of the initial screen to start the creation workflow.

Here there is a decision to make. TikTok and the CreativeEditor SDK can each start with the creation controls and a video preview. When using the CreativeEditor SDK though, you can also start with some video that came from somewhere else. So if you already have some camera code you like, or if you want to start with some video clip from somewhere else, you can insert that into the editor when it launches. You can do that by leveraging the onCreate modifier of the editor.

let engine = EngineSettings(license: "<your license code>",
                              userID: "<your unique user id>")

VideoEditor(engine)
  .imgly.onCreate({ engine in
    try await engine.scene.create(fromVideo: Bundle.main.url(forResource: "dog_water", withExtension: "mov")!)

    //set other defaults
  }

In the code above, the app initializes an instance of the Video Editor view and then reads the dog_water.mov file from the app bundle. launches the editor with that.

Whether starting with a video or starting with a blank canvas a user can add video clips to their creation using the embedded camera.

The image above compares two different camera controllers: the left one from the TikTok app and the right showing the default settings of the MobileCamera, either standalone or within a Video Editor scene. Key controls in each are marked with blue numbers:

1 - Flip camera button
2 - Flash button
3 - Recording time indicator
4 - Add video and finish buttons
5 - Cancel button

While the TikTok camera integrates several editing functions during video capture, the Video Editor separates these tasks, focusing the camera purely on recording. Both systems allow users to start and stop video recording to compile a series of clips before moving to the editing phase.

Configuring the Camera

The camera used inside of the Video Editor UI isn’t customizable. You have limited options to customize when you use the standalone camera though. You can set the colors of the buttons, helpful if you have a preferred color scheme, and you can set a maximum duration for the video recording allowed.

let config = CameraConfiguration(
  recordingColor: .orange,
  highlightColor: .blue,
  maxTotalDuration: 30,
  allowExceedingMaxDuration: false)

Camera(engine, config: config) { result in
 //handle the recorded video collection
}

The code above sets the record button to orange while recording and the clip view to orange. There is a maximum duration of 30 seconds for all videos in the collection and the button to save the clips highlighted in blue.

If your app requires further customization, you’ll probably want to make your own camera view and work with the IMG.LY Creative Engine directly.

The documentation website is a great resource for understanding how much you can add to and customize the IMGLYUI resources and when you’ll need to drop down to the level of the engine.

Presenting the Controller

As demonstrated above, both the Video Editor and the Camera are View structures that can be presented directly or through overlays such as overlay or fullScreenCover. However, they are typically enclosed in a NavigationView to utilize the navigation bar for displaying export and other buttons. Therefore, when triggering the Video Editor from a button, it’s crucial to embed it within a NavigationView.

Button("Edit the Video") {
  isPresented = true
}
 .fullScreenCover(isPresented: $isPresented) {
   NavigationView {
     VideoEditor(engine)
      //any setup modifiers
   }

Building Your Creation

After capturing a video, the app displays an editor to complete the creation. The editing tools are in the red circles. TikTok provides about ten different tools on the editing screen. Additionally, TikTok provided some editing tools on the capture screen. The CreativeEditor SDK apps provide a scrolling set of asset libraries along the bottom for items to add to the scene. But when any of the elements of the scene, like your video or an audio clip, are highlighted, the bottom row becomes tools to manipulate that element. The basic Asset Library types are uploads, videos, audio, images, text, shapes, and stickers.

How to configure and customize each of these tools is beyond the scope of this tutorial. But, a lot of the customization of the assets, fonts, blurs and even filters can be done without code. By design, the CreativeEditor SDK downloads assets from a central server. This is a great way for you to be able to push updated filters, stickers, and other assets to your users without going through the app update process. In the next section, we’ll walk through that and provide some links for further research.

Configuring the Editors Assets

The IMG.LY Engine looks for assets for stickers, vector shapes, LUT filters, color filters, color palettes, blurs, and fonts using a URL by default. The design of the system is that you would have your own server supporting your app. However, assets in the app bundle are also accessible by URL, so as long as the assets are in the folder format that the engine expects, it doesn’t matter whether they are local or remote.

In the documentation for this process you can find a URL to the default asset bundle. Once you download that bundle you can add it to your app. The bundle is structured much like a regular iOS Assets.catalog with some JSON containing metadata about the asset and a file that provides the actual asset. For some of the assets like filters and vectors, no file is needed as everything can be defined in the JSON.

In the image above, you can see the JSON to describe the ape sticker and the emoji_ape image file after the IMGLYAssets bundle has been added to our app. You can edit these assets and add your own.

Once you’re done editing the asset bundle, you can use it instead of the default bundle during the onCreate modifier like this:

VideoEditor(engine)
  .imgly.onCreate({ engine in
     try await engine.scene.load(from: VideoEditor.defaultScene)
              // Add asset sources
     let bundleURL = Bundle.main.url(forResource: "IMGLYAssets", withExtension: "bundle")!
     try await engine.addDefaultAssetSources(baseURL: bundleURL)
     try await engine.addDemoAssetSources(sceneMode: engine.scene.getMode(),
       withUploadAssetSources: true)
     try await engine.asset.addSource(TextAssetSource(engine: engine))
  })

In the code above, we create a blank scene using VideoEditor.defaultScene static method. Then set a URL to point to the bundle in the app instead of a remote bundle and make that the source of addDefaultAssetSources. Now our app will use the assets from the bundle.

Exporting the Finished Video

Once the user has finished with their creation, they tap the share button and the Video Editor composes all of the video, audio, filters, and static items into a single .mp4 file. Then it displays a share sheet. If you’d rather do something different, like save the creation locally so you can play it back, you can override this default behavior.

In your app, after the .onCreate modifier, you can define a .onExport modifier. When the user taps on the share button you can export the video and then save it to the documents directory to playback later.

In the documentation for modifiers you can find the source code for the Video Editor’s default behavior. Instead of duplicating it in detail here, we can just summarize and provide a little snippet for our change.

First, the .onCreate handler takes an engine and a eventHandler parameter. The engine is the underlying IMG.LY Engine object we’ve been using. The eventHandler lets us send information back to the Video Editor UI about the progress or any errors.

In the documentation, there is a helper function to export the video. After it runs it returns the data of the exported video and a mimeType, which will be .mp4.

let data: Date, mimeType: MIMEType
let mode = try mainEngine.scene.getMode()
guard mode == .video else { throw CallbackError.unknownSceneMode; return }
(data, mimeType) = try await exportVideo()

The code in the documentation handles static files from the DesignEditor as well as videos from the VideoEditor. The above code makes it so that only video can be exported, since our app only works with video.

In the exportVideo function, the code first checks to make sure it has a properly formed scene object to work with. Then it starts the export using mainEngine.block.exportVideo(_, mimeType:). This begins an async throwing stream during the export. The VideoExport objects in the stream are either .progress or .finished. When a .progress object arrives, the exportVideo function sends an update to the eventHandler to display in the UI. When the .finished object arrives, it has an associated object that is the data for the video.

Now instead of opening the share sheet, we can write the video to the documents directory for the app.

do {
      guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
        else {
          logger.error("Documents directory not found"); return
        }

  let filePath = documentDirectory.appending(component: UUID().uuidString)
  try data.write(to: filePath)
} catch (let error as NSError) {
  logger.error("could not write finished file: \(error.localizedDescription)")
}

eventHandler.send(.exportCompleted {})

The code above takes the data from the exportVideo function and writes it to the app’s documents directory, giving it a UUID string value as a filename. Then it updates the UI to let it know the export is done but sends an empty action block.

Setting Up a Playback Controller

For this example app, the playback controller will play any clips in the app’s Documents directory. Each clip plays on a loop. The user can swipe up to get the next clip and tap to pause or restart the clip. If there are no clips, the user will be encouraged to make a new one.

When the video player view appears, the first task is to see if there are any videos to load. Any video files in the Documents directory are assumed to be ready to play.

func loadVideos() {
  //get a handle to the documents directory for this app
  guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true) else { logger.error("Documents directory not found"); return}

  if let files = try? FileManager.default.contentsOfDirectory(at: documentDirectory, includingPropertiesForKeys: nil) {
    videos = files
  }
}

The code above reads all of the filenames from the documents directory into a files array and then assigns that array to a videos variable.

Each time the app is launched or the VideoEditor view is dismissed, the video player view will get the onAppear message. This is a good place to check for videos using the loadVideos function. After the videos are loaded, the app can start to play the first one.

func setupVideoPlayer() {
  guard let currentVideo = videos.first else {logger.error("No videos to play"); return}

  self.shouldHideEmptyDirectoryText = true

       let endMonitor = NotificationCenter.default.publisher(for: NSNotification.Name.AVPlayerItemDidPlayToEndTime)

  self.player = AVPlayerItem(url: currentVideo)
  self.currentVideo = currentVideo

}

In the code above, we first check to see if there are any video files. If there are, then hide the message about videos and create a new AVPlayerItem using the URL of the first video. The endMonitor, shouldHideEmptyDirectoryText, and player are both being watched by the View. The AVPlayer gets rendered in a regular SwiftUI View.

VideoPlayer(player: player)
  .frame(width: 640, height: 360)
  .onAppear {
    player.play()
  }
  .onReceive(endMonitor) { _ in
    player.seek(to: .zero)
    player.play()
}

When the video reaches the end, it will seek(to: .zero) which rewinds the video and loops it.

In the TikTok app, you can advance to the next video by swiping up. Additionally, you can start and stop any video by tapping on the screen. We can add both of those features using gesture recognizers.

The code for play and pause is attached to a Tap Gesture modifier we can append to the player.

.onTapGesture {
  if player.rate > 0.0 {
    player.rate = 0.0
  } else {
    player.rate = 1.0
  }
}

We can just adjust the playback rate. A value of 1.0 is a normal speed and 0.0 is paused. The player also has a .pause and .play methods, but sometimes these seem buggy.

Swipe is a little more difficult since SwiftUI only has swiping on List items. So we can use a drag gesture and then evaluate the translations when it’s complete.

.gesture(
    DragGesture(minimumDistance: 3.0, coordinateSpace: .local)
        .onEnded { value in
            // Check for a vertical, upward swipe with minimal horizontal deviation
          if abs(value.translation.width) < 100 &&
             value.translation.height < 0 {
                // It was an up swipe!
		viewModel.advanceVideo()
            }
        }
)

Then advancing the video is a simple matter of refreshing the player

func advanceVideo() {
  guard let currentVideo = self.currentVideo else { logger.error("No current video."); return}

  let currentIndex = videos.firstIndex(of: currentVideo)
  var nextVideo = videos.index(after: currentIndex!)
  if nextVideo == videos.count {
    nextVideo = 0
  }

  self.currentvideo = videos[nextVideo]
  self.player = AVPlayer(url: self.currentVideo!)

}

This code will get the next file URL from the array of videos. When it reaches the end, it will loop back and get the file at index 0. Then it creates a new player. Because the view is observing that player variable, it will update with the new video.

With this view, the user can create new videos by tapping the creation button and swipe to view all of their created videos.

Where to Go From Here

This tutorial has focused on how the CreativeEditor SDK can help you quickly make a video creation app like TikTok. Good next steps would be to further customize the editing tools and build out the network for sharing and tagging videos. Something that is important to consider is the data structure for each video. The iOS system is optimized to read only as much of a video file off of disk as is needed at any moment. Your app should use those optimizations to run faster. So, don’t load the whole video into memory as a Data object. Your data structures should keep the large video files somehow separate from the much smaller metadata (likes, comments, etc.). Consider storing the URL or filename of the video in the same object as the likes and comments. This will also allow you to cache video files after they have been downloaded so that you don’t need to redownload them when other data such as comments or number of likes changes.

Thanks for reading! We hope that you’ve gotten a better understanding for how a tool like the CreativeEditor SDK can bring your ideas to market faster. Feel free to reach out to us with any questions, comments, or suggestions.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

To stay in the loop, subscribe to our Newsletter or follow us on LinkedIn and Twitter.

CE.SDK v1.8.0 Release

Neslihan — Tue, 18 Oct 2022 14:48:05 GMT

Since our CE.SDK v1.7.0 Release, we implemented more highly requested features. Today, we shine light on them. As always, we are excited to hear your feedback and suggestions to build powerful features for your application. Our Product Roadmap is the best place to make your case and be the first one to know of future releases.

CE.SDK v1.8.0

This release includes:

Node.js Platform Support
Common Touch Gesture Support for iOS
Emoji Support ?
Image Straightening in Advanced & Default UI
Multi Selection Rotation
Outlook

Node.js Platform Support

Since the release of our Node Package, you can run a headless version of CE.SDK on the server. Now with this release, you have the full power of our Creative Engine at your fingertips, enabling you to programmatically generate images on the server or command line, render scenes created with the web SDK, and implement any creative automation workflow.

Common Touch Gesture Support for iOS

We know that a mobile presence and functionality is crucial for your application. If users find it difficult to complete a task or are frustrated by poor usability, they could quickly bid adieu to your app and ditch your hard work for a competitor. So with this update, we add common touch gesture to improve the editing experience on touchpads and mobile devices. This includes gestures like pinching, spreading, dragging, rotating, tap and double tapping.

Emoji Support

Sometimes, a party face says more than a thousand words ?: Emojis have become an essential part of modern communication and are omnipresent, especially on mobile platforms, but on desktop as well. We added Emoji rendering support for more fun edits and colorful messaging.

Image Straightening in Advanced & Default UI

While cropping your image, you can now also straighten, flip and rotate it! This feature gives you full control over your image in one place – available in both Advanced and Default User Interfaces.

Multi Selection Rotation

For an accelerated design and editing process, the engine and user interface now allow rotating several blocks at once. To multi-rotate, simply hold your Shift key while selecting the blocks you would like to move simultaneously.

Outlook

Native iOS Platform Support: With this feature, we will add support for developing native iOS applications with the SDK. It will come with full Swift API support and many integration examples.

Video for Web with CreativeEditor SDK: Users will be able to edit and create fantastic videos based on templates in their browser with our SDK in your application soon! Sign up for early access.

Thanks for reading! To stay in the loop, feel free to subscribe to our Newsletter.

Time-Based Sprites for VE.SDK on iOS and Android

Neslihan — Mon, 05 Sep 2022 07:06:05 GMT

We are happy to extend VideoEditor SDK with a highly-requested feature: Time-Based Sprites. This new feature sets the duration of text and stickers in your video timeline. Users may now place fun stickers and text at the right moment, and give their videos a special touch.

Time-Based Sprites

The popular feature known from TikTok and Instagram Reels is now available in VE.SDK: set the starting and end point of your text or sticker. Tap your text or sticker and select Duration to determine the duration of your asset.

Unless you have specified a custom set of sticker or text actions, this feature is enabled by default since VE.SDK v10.3.0 for Android and v11.3.0 for iOS. See the official documentation for iOS or Android on Time-Based Sprites.

Why is Video Important?

Video content has become an essential medium for social media, marketing, sales, and support teams. According to marketing statistics, people are watching an average of 19 hours of online video per week in 2022. 88% of people say that they felt convinced to buy a product or service by watching a brand’s video. The growing trend and preference of consumers are why businesses are shifting their attention toward including videos in their strategies and applications.

We commit to extending our video editing features to help developers meet the demand, save resources and streamline the process of building great applications. Let us have a look at our previous VE.SDK releases:

Video Composition
Users may seamlessly edit their footage by trimming and adjusting video files with advanced filters. Finally, they can set the correct order of their video sequences to create a single composition.

Audio Support
Replace or add sound in videos by loading audio files. Users can trim their audio according to their footage. Developers can provide media libraries for audio and video files. That way, users may access media by choosing from labeled folders, such as genre, theme, or artist names.

Force Trim
The VE.SDK trim tool allows users to determine the start and end frame of a video clip and change the duration of their footage. Now you can enforce a minimum and maximum length of videos. Force Trim will come in handy for use cases, such as social media stories and posts popularly limited to bite size 15 or 60 seconds by widely loved apps – see TikTok or Instagram. Adopting a ready-to-use solution like VE.SDK will set you on your path to creating equally beautiful apps.

Thanks for reading! Let us know what you think on Twitter – or check out our Newsletter for more accelerating updates.

Build a Simple Real-Time Video Editor with Metal for iOS

Walter — Tue, 30 Aug 2022 20:06:49 GMT

In this tutorial you’ll see how to extract frames from live camera streams, regular movie files and streamed movie files and display them on a MetalKit View. Using Metal allows you to have a great deal of control over how the pixels are rendered and ensures that the GPU is used for rendering, which keeps from slowing down the CPU.

The tutorial code will always convert the video frames into CIImage objects before display. This is to help you adapt the code to your own applications. Applying filters, resizing and other effects are fast and easy when working with CIImage. Additionally, as long as you set your CIContext to the GPU, you can be sure that your code uses the GPU and the CPU efficiently.

When working with Video, Apple provides a number of frameworks that operate at different levels. For the higher level frameworks such as AVKit or CoreImage, the system determines when to use the GPU and when to use the CPU for rendering and computation. The higher-level frameworks also offer a tradeoff between ease of use and granularity of control. If you want to have direct access to tell the GPU how to render every pixel, then you will want to use Metal. Remember, though, that you are now responsible for keeping video and audio in sync, encoding and decoding data and determining a reasonable UI for your user. For many use cases, using a higher level framework is probably a better choice. Apple’s engineers have worked to ensure the graphics frameworks use the GPU and CPU in reasonable ways. However, when you need to use Metal, you you need it. So, let’s get started.

Setting Up a MTKView

An MTKView is a subclass of a UIView so it has a frame and bounds and other properties. Its drawing and rendering is backed directly by drawables and textures rendered on the GPU, so drawing to it can be quite fast. In addition to displaying video, you can render 3D model objects and graphics shaders, so for a graphics rich application it can be a powerful tool.

Before you can send data to the view it needs to be configured. MetalKit Views are still heavily influenced by UIKit, so if you are working in a SwiftUI project, you will need to wrap them in ViewRepresentable code.

//MetalKit Variables
@IBOutlet var displayView: MTKView!
var metalDevice : MTLDevice!
var metalCommandQueue : MTLCommandQueue!

//CoreImage Variables
var ciContext : CIContext!
var filteredImage: CIImage?
var cleanImage: CIImage?

//get a reference to the GPU
metalDevice = MTLCreateSystemDefaultDevice()

//link the GPU to our MTKView
displayView.device = metalDevice

//link our command queue variable to the GPU
metalCommandQueue = metalDevice.makeCommandQueue()

//associate our CIContext with the metal stack
ciContext = CIContext(mtlCommandQueue: metalCommandQueue)

You will need to get a reference to the GPU of the iOS device. At least for now, iOS devices only have one GPU. Then you will need to link the displayView and the metalCommandQueue to the metalDevice. Finally we will link the ciContext to the GPU so that all of our image manipulation code will run in the same place.

//tell our MTKView that we want to call .draw() to make updates
displayView.isPaused = true
displayView.enableSetNeedsDisplay = false
//let it's drawable texture be writen to dynamically
displayView.framebufferOnly = false

//set our code to be our MTKView's delegate
displayView.delegate = self

In the code above, we set the MTKView to be in a isPaused state and do not enable setNeedsDisplay. This will ensure that it will only redraw when we explicitly tell it to redraw using a .draw() method in our delegate. By setting framebufferOnly to false you are telling the view that you will be writing to it multiple times and may also read from it.

Drawing in an MTKView

Now the MTKView is configured. The next step is to update the delegate methods. The MTKViewDelegate has two methods. If your code needs to respond to the view dimensions changing (to support device rotation, or if you want the user to be able to resize the window) make your adjustments in func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize). For this example we are only interested in the delegate method: func draw(in view: MTKView). The code below draws a CIImage to the MTKView. When showing video, we can extract each frame of the video as a CIImage

//create a buffer to hold this round of draw commands
guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
  return
  }

//grab the filtered or a clean image to display
guard let ciImage = filteredImage ?? cleanImage else {
  return
  }

//get a drawable if the GPU is not busy
guard let currentDrawable = view.currentDrawable else {
  return
  }

//make sure frame is centered on screen
let heightOfImage = ciImage.extent.height
let heightOfDrawable = view.drawableSize.height
let yOffset = (heightOfDrawable - heightOfImage)/2

//render into the metal texture
self.ciContext.render(ciImage,
             to: currentDrawable.texture,
  commandBuffer: commandBuffer,
         bounds: CGRect(origin: CGPoint(x: 0, y: -yOffset),
                          size: view.drawableSize),
     colorSpace: CGColorSpaceCreateDeviceRGB())

//present the drawable and buffer
commandBuffer.present(currentDrawable)

//send the commands to the GPU
commandBuffer.commit()

For each pass of the draw method, we will create a new MTLCommandBuffer then we will get the MTKView’s .currentDrawable which contains a texture we can send pixel data to. The texture will have a size and a color space. Though we are using Metal to render images to the screen, Metal can be used to send any valid commands to the GPU for things like computations. Once the buffer has been filled with commands it can be assigned to the drawable and committed. When the buffer is committed, the GPU will execute all of the commands. If another .draw() gets called while the buffer is being executed, the GPU will not interrupt the current buffer in order to start the new one.

The bounds of the render allow you to move where the CIImage gets displayed in the MTKView and resize the CIImage within the view. This can be valuable when compositing multiple images or video streams onto the same MTKView. Remember that unlike a UIView the origin point is the bottom left of the texture and CIImage.

Since iOS 11 Apple provides a new lighter weight API for sending renders to the buffer. Use whichever form makes sense to you.

//render into the metal texture
let destination = CIRenderDestination(mtlTexture: currentDrawable.texture,
                                   commandBuffer: commandBuffer)
do {
 try self.ciContext.startTask(toRender: ciImage, to: destination)
} catch {
 print(error)
}

The CIRenderDestination has some other optional parameters such as height and width but will always display with an origin of 0,0. So, unlike the earlier call, if you need to rotate or change the dimensions of the image, you will need to do it to the CIImage earlier in the code. However, if your app always displays the video at full size in the MTKView this may be easier. Also the startTask call will return immediately, instead of waiting for other tasks on the CPU to complete.

It is important to remember that you can have any number of renders in the commandBuffer before the calls to .present and .commit. This means that the same MTKView can display video from multiple streams as well as static images or animations.

Working with Local Files

In order to extract individual frames from a local file, we can use an AVAssetReader to get pixel data to render in the MTKView

When using higher-level frameworks, a quick way to display a video file for playback is to load it into an AVPlayer. However, an alternative is to use an AVAssetReader to read the tracks and the individual frames from the video tracks.

let asset = AVAsset(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)
let reader = try! AVAssetReader(asset: asset)

guard let track = asset.tracks(withMediaType: .video).last else {
  return
  }

let outputSettings: [String: Any] = [
        kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32ARGB
    ]
let trackOutput = AVAssetReaderTrackOutput(track: track, outputSettings: outputSettings)
    reader.add(trackOutput)
    reader.startReading()

It is important that the outputSettings of the reader match the image format of the video track. If there is a mismatch the color may be off or the reader may be unable to extract a usable buffer. If your code is generating empty or black buffers, the outputSettings are the first place to troubleshoot. Once the reader starts reading then it can extract pixel buffers. You can use a while loop to get all of the buffers from the track and send them to the MTKView for display.

var sampleBuffer = trackOutput.copyNextSampleBuffer()
while sampleBuffer != nil {
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer!) else {
   return
  }
  print(track.preferredTransform)
  print(sampleBuffer?.outputPresentationTimeStamp)

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()

  sampleBuffer = trackOutput.copyNextSampleBuffer()
  }
}

In the code above, use CMSampleBufferGetImageBuffer to ensure that we have a valid image. Then assign the image to a CoreImage and execute the .draw method of the MTKView. Afterwards, get the next sample buffer. This will continue until the end of the track, when .copyNextSampleBuffer() will return nil.

In the code above, there are two print statements to note some valuable information that your app may want to store for later use. Remember, when working with buffers and the GPU directly, much of the higher-level metadata about the video is lost, so you’ll need to keep track of it in your code if you want to use it later. An image may be rotated because of the way it was originally generated in the video. The preferredTransform will provide data that your app can use to transform the image to the “proper” orientation before display. Additionally the outputPresentationTimeStamp tells when the image represented by the buffer appeared in the original video. This can be helpful when you are trying to sync individual frames back to audio tracks or if your app only wants to modify specific frames and then reinsert them into the original clip.

Working with Streams

In addition to local files, an app may want to use a video stream as the input. The process is largely the same, as there is a method to extract a CVPixelBuffer from an AVPlayer that is streaming. You can find the complete example code for pushing pixel buffers from a stream to an MTKView in this blog post about streaming. However, the important part of the code perhaps looks familiar by now:

let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) {
  if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
     let frameImage = CIImage(cvImageBuffer: buffer)
     self.currentFrame = frameImage //a CIImage var
     self.videoView.draw() //our MTKView
   }
}

The code above uses a CADisplayLink to query the stream on a regular basis for new video frames. It then extracts a buffer and sends it over the MTKView for display.

Working with the Cameras

Much like a video stream with an AVPlayerItemVideoOutput to output pixel buffer data, a standard object to use with the camera is an AVCaptureVideoDataOutput object. First, you need to set up a standard capture session for either the front or back camera. Then attach a video data output object to the session.

videoOutput = AVCaptureVideoDataOutput()
let videoQueue = DispatchQueue(label: "captureQueue", qos: .userInteractive)
videoOutput.setSampleBufferDelegate(self, queue: videoQueue)

if captureSession.canAddOutput(videoOutput) {
  captureSession.addOutput(videoOutput)
} else {
  fatalError("configuration failed")
}

Whenever the camera has collected enough data to generate a pixel buffer it will send that to its delegate. The delegate can then convert that data to a CIImage and send it to the MTKView to get rendered. The method in a AVCaptureVideoDataOutputSampleBufferDelegate would look like this.

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
  //get a CVImageBuffer from the camera
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    return
  }

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()
}

Going Further

In this tutorial, you saw how to extract pixel buffers from the camera, local file and remote streams and convert them to CIImage Then you saw how to render a CIImage to fill all or part of an MTKView. If your application needs to resize or apply filters to the images, you can do that before calling the .draw method of the MTKView. If the only reason you are considering using MKTViews is because of a Metal or OpenGL kernel you want to use, consider our tutorial on how to wrap kernels into Core Image filters.

If your application needs to gather streams of video, filter and then render them to the screen with precision, then drawing to an MTKView may be sufficient. However, if you want to let your users dictate how and where to filter the streams, then you may want to consider an SDK like VideoEditor SDK or CreativeEditor SDK.

How to Merge Videos in iOS with Swift

Walter — Wed, 12 Jan 2022 12:12:55 GMT

Combining video clips or parts of video clips into a single video may appear complicated, but the things that add complexity also provide flexibility. In this tutorial, you will see how to combine a few short clips into a single video. You will also gain an understanding of building on this basic pattern to make more advanced creations. This tutorial was created and tested with Xcode 13 and Swift 5.

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with code that supports this tutorial is available on Github.

Making a Composition

Whenever you want to edit media or add effects, the first place to look is AVComposition. This class in AVFoundation has the purpose of arranging different assets and types of assets into a single asset for playback or processing. Asset types can include audio and video subtitles, metadata, and text. When working with AVComposition it is helpful to think about AVAsset and AVAssetTrack objects. Unfortunately, because these items are so similar (AVComposition is a subclass of AVAsset) it is easy to get confused. Try to remember that the AVTrack is a wrapper around the actual pixel, sound-wave, or other data and that AVAsset is a collection of AVTrack objects. When we want to combine and manipulate AVAsset objects, an AVComposition helps us do that. Changes made to any AVAsset by an AVComposition will not impact the original media file.

As an extra layer, Apple keeps the editing features of an AVComposition separate from the playback features by creating an AVMutableComposition object. You will work with an AVMutableComposition object in this tutorial.

Each track in our composition will have a start time and a duration. The composition will ensure that all of the data it holds displays at the right time and that the tracks stay in sync.

At the simplest, a video clip is a single AVTrack of audio data and a single AVTrack of video data.

So, to set up an empty AVComposition for combining clips, use code such as this:

    let movie = AVMutableComposition()
    let videoTrack = movie.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid)
    let audioTrack = movie.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)

The code above creates a new, empty AVMutableComposition and then adds two tracks. One track will be able to hold .video assets, and the other will hold .audio assets. AVFoundation supports many different kinds of audio and video formats, but the .video track cannot hold .audio formats. The kCMPersistendTrackID_Invalid signals that you want a new, unique track id generated.

AVFoundation can manipulate any of the ISO media formats. The Quicktime (.mov) and MPEG (.mp4) are the most common, however, .heif, .3gp and other formats are all supported.

Adding Video Clips

After creation, the AVMutableComposition has a start time of CMTime.zero and a duration of CMTime.zero. Use the insertTimeRange(_ timeRange: CMTimeRange, of track: AVAssetTrack, at startTime: CMTime) throws function on the audio and again on the video track to add data from the clip. In order to add data to the tracks, you load an AVAsset, determine what range of time will go into the new composition, and then call the insert function. Your code might look like this:

  let beachMovie = AVURLAsset(url: Bundle.main.url(forResource: "beach", withExtension: "mov")!) //1

  let beachAudioTrack = beachMovie.tracks(withMediaType: .audio).first! //2
  let beachVideoTrack = beachMovie.tracks(withMediaType: .video).first!
  let beachRange = CMTimeRangeMake(start: CMTime.zero, duration: beachMovie.duration) //3
  try videoTrack?.insertTimeRange(beachRange, of: beachVideoTrack, at: CMTime.zero) //4
  try audioTrack?.insertTimeRange(beachRange, of: beachAudioTrack, at: CMTime.zero)

Here is what this code does:

Load a video file from a local URL. In the example, the video is in the application bundle, but it could also be a URL that points to a file somewhere else. This will not work with a streaming URL. It needs to be a video file.
Extract the main videos and audio tracks from the video file. There may be multiple video and audio tracks in a file, but this code just grabs the first one.
Create a time range. In this example, the range is the same duration as the clip we loaded.
Insert the video track from the clip at the beginning of the video track of the composition and insert the audio track from the clip at the beginning of the audio track of the composition. A common mistake in this step is to use the .duration property of the composition to insert the audio and video at the end. However, as soon as media is inserted into any track, the duration of the entire composition will change. So the audio and video would be played one after the other and you would see a blank screen when the audio is playing, and have no audio when the video is playing.

Repeat the steps for each clip that you want to add to the final composition.

Using the Composition

After all of the clips have been combined into the composition, it can be played in an AVPlayer or exported using an AVAssetExportSession. The code for these is the exact same as for any other AVAsset.

  self.player = AVPlayer(playerItem: AVPlayerItem(asset: movie)) //1
  let playerLayer = AVPlayerLayer(player: player) //2
  playerLayer.frame = playerView.layer.bounds
  playerLayer.videoGravity = .resizeAspect
  playerView.layer.addSublayer(playerLayer) //3
  player?.play() //4

Here is what the code above is doing:

Create an AVPlayer using a movie object, which is the AVMutableComposition that was created earlier.
Create a playerLayer to display the video and set its aspect ratio
Add the playerLayer to playerView which is a regular UIView in a UIViewController
Play the video clip

Alternatively you may want to export the new composition as a new movie file. Again, because the composition is an AVAsset using a standard AVAssetExportSession will write the movie to disk.

  //create exporter
  let exporter = AVAssetExportSession(asset: movie,
                                 presetName: AVAssetExportPresetHighestQuality) //1
  //configure exporter
  exporter?.outputURL = outputMovieURL //2
  exporter?.outputFileType = .mov
  //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error { //3
        print("failed \(error.localizedDescription)")
      } else {
        print("movie has been exported to \(outputMovieURL)")
      }
    }
  })

Here is what this code is doing:

Create an export session using the movie composition you made earlier.
Set the save location to some file URL on your device.
Wait for the exporter to finish and display the result.

Don’t forget that exporting an AVAsset can take a long time and is asynchronous. You will want to display a spinner or a message to your user.

Going Further

You can now combine clips into a longer clip for playback and export. However, trimming the original clips, adding filters, and rearranging the order will require more work before your application is ready for the store. Using an SDK like the IMG.LY’s VideoEditor SDK can help you provide an app that has the features users will expect in addition to stitching clips together.

The VideoEditorSDK offers a Video Composition Controller which will allow the users to stitch clips together. Additionally, it will allow the user to add new clips and edit clips using the other editing tools. Because the Video Composition Controller is a subclass of UIViewController it can just be configured and dropped into your application. Instead of the code in our example, you pass the videos to the controller:

let videoClips = [
  Bundle.main.url(forResource: "stream", withExtension: "mov"),
  Bundle.main.url(forResource: "beach", withExtension: "mov"),
  Bundle.main.url(forResource: "mountain", withExtension: "mov"),
].compactMap { $0.map{ AVURLAsset(url: $0) } }
let video = Video(assets: videoClips)
let videoEditViewController = VideoEditViewController(videoAsset: video, configuration: Configuration())
present(videoEditViewController, animated: true, completion: nil)

The code above loads the same video clips as our demo application and then uses the VideoEditViewController to present them. From there the user can make further edits preview and export the composition.

Wrapping Up

In this example, you saw how to combine several video clips, each with one video and one audio track into a single movie file with one video and one audio track. To make more complex movie files consider building on this tutorial by:

Add a second audio track, AVFoundation will play sounds from both tracks at times specified at equal volume. Use AVAudioMix to adjust the volume of the tracks at different times (the sample project has an example of adding a second audio track).
Add more video tracks and use AVMutableCompositionLayerInstruction to determine which track is visible at any time in the final video and to make fancy transitions between tracks.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How to Add a Filter to a Video Stream in iOS

Walter — Mon, 08 Nov 2021 15:33:55 GMT

Using filters and effects with video streams requires different strategies than applying them to files. In this tutorial, you will see how to apply effects and filters to video streams in real-time. The code in this tutorial compiles with Xcode 13.

As with most AVFoundation code, the Xcode simulator is not the best platform for running the code. Test on an actual device. A project with code that supports this tutorial can be found on GitHub.

Streams or Files

For video files stored on the device, you can use an AVMutableVideoComposition to apply filters. When streaming video from a remote location, this strategy will not work. The AVMutuableVideoComposition classes are not designed to work in real-time with streams. Apple provides a way to extract the pixels from the stream at regular intervals. You can manipulate the pixels and then render them to the screen. An AVPlayerItemVideoOutput object gives access to the pixels that make up a video frame. Apple also supplies a CADisplayLink object to ensure you are extracting the right pixels at the right time. The display link is a specialized timer that will fire in sync with the redraw rate of the current display.

So, to filter a video stream, the strategy becomes:

Set up an AVPlayer as normal and assign it the URL of the stream
Attach an AVPlayerItemVideoOutput object to the stream
Attach a CADisplayLink object to the run loop
Extract the current pixel buffer from the AVPlayerItem every time the screen redraws
Convert the pixel buffer to a CoreImage object
Apply filters
Display the image

Setting up AVPlayer

You won’t use the AVPlayerLayer or AVPlayerViewController to display the video, but the AVPlayer plays a central role. The AVPlayer keeps the video in sync with the audio, handles decoding whatever format the stream uses, controls buffering, and more. The basic setup is the same as with any other project:

//create a player
let videoItem = AVPlayerItem(url: streamURL)
self.player = AVPlayer(playerItem: videoItem)

The URL can either point to a local resource or a stream.

Using AVPlayerItemVideoOutput

The AVPlayerItemVideoOutput allows you to query the AVPlayerItem for the pixel buffer (one screen worth of pixels) at any given time. You can specify different pixel formats and other options for the output. For this tutorial, you will you can create the object with a simple instantiation using the default formats.

let playerItemVideoOutput = AVPlayerItemVideoOutput()

Later you will add this video output to your player item. Then you can ask it to generate the pixel buffers as needed. In this tutorial, you will ask for the pixel buffer of the current frame. You could ask for the frame for any timestamp though.

Creating the Display Link

The CADisplayLink class is a specialized NSTimer that synchronizes itself to the screen refresh rate of the display. This ensures that the pixel buffer you extract will be from the correct time of your video stream. It will get rendered at the correct time in the screen refresh. Syncing a standard NSTimer to the screen refresh has always been problematic. With the new variable refresh rate screens that Apple makes, it becomes impossible. However, CADisplayLink adjusts its rate as the screen refresh rate changes. Apple explains how this works in the 2021 WWDC Session Optimize for Variable Refresh Rate Displays. When you create the display link, you give it the name of a function to call each time it executes. A sample for creating a link could look like this:

lazy var displayLink: CADisplayLink = CADisplayLink(target: self,
                                                  selector: #selector(displayLinkFired(link:)))

Elsewhere in the code, a function called displayLinkFired(link: CADisplayLink) extracts the pixel buffer from the AVPlayerItem.

Starting the Player

Apple notes that loading a video file takes a measurable amount of time. It can take even longer when the video file is streaming across a network. AVFoundation allows your program to continue to execute while the setup steps and initial buffering are occurring in the background. You can run into issues if you load a video into AVFoundation and then immediately attempt to work with it. AVFoundation may not show an error or status message, but it will not perform as expected either.

The most important step in this entire process is using an observer to wait until the AVPlayerItem has a .readyToPlay status before attaching the output and displaying link objects. In the example below statusOberserver, player, playerItemVideoOutput and displayLink are all declared at the class level. They need to persist the entire time the video is being used.

//create a player
let videoItem = AVPlayerItem(url: streamURL!)
self.player = AVPlayer(playerItem: videoItem) //1
//*important* add the display link and the output only after it is ready to play
self.statusObserver = videoItem.observe(\.status, //2
                       options: [.new, .old],
                 changeHandler: { playerItem, change in
    if playerItem.status == .readyToPlay { //3
      playerItem.add(self.playerItemVideoOutput)
      self.displayLink.add(to: .main, forMode: .common)
      self.player?.play() //4
    }
})

Here is what to notice about the code above

Creating the player will take a large amount of time, but the program execution will not stop and wait
NSKeyValueObservation is how swift defines key value observers the code will execute every time the videoItem.status property changes values
Checks to see if the .status property is .readyToPlay
After adding the display link and the video output objects start playing the video

Extracting the Pixel Buffer

Once the video begins to play, the display link function gets called for each screen refresh. Here is an example of how to extract the pixel buffer and render it to a UIImageView.

@objc func displayLinkFired(link: CADisplayLink) { //1
  let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
  if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) { //2
    if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
      let frameImage = CIImage(cvImageBuffer: buffer) //3
      //4
      let pixelate = CIFilter(name: "CIPixellate")!
      pixelate.setValue(frameImage, forKey: kCIInputImageKey)
      pixelate.setValue(self.filterSlider.value, forKey: kCIInputScaleKey)
      pixelate.setValue(CIVector(x: frameImage.extent.midX, y: frameImage.extent.midY), forKey: kCIInputCenterKey)
     let newFrame = pixelate.outputImage!.cropped(to: frameImage.extent)
     self.videoView.image = UIImage(ciImage: newFrame) //5
    }
  }
}

Here is what the code is doing:

Marks the func with @objc so that it works with the display link selector
Asks if there is a new pixel buffer to display. Depending on the refresh rate of the screen and the frame rate of the video, there will not always be a new buffer.
After extracting the pixel buffer, convert it to a CoreImage object
Apply any filters. This example applies the CIPixellate filter. It uses a slider to let the user change the intensity as the video plays
Convert the filtered image to a UIImage and assign it to the UIImageView

Now as the pixellate scale changes the image will be filtered but the video will continue to play smoothly.

Going Further

Using the strategy above you should be able to apply different filters to the video stream in real-time. Remember that you only have a few milliseconds though, then the video will begin to stutter. CoreImage filters are optimized to make use of the GPU and should be fast enough in most cases. For better performance, you can render the filtered image to an MTKView and use Metal. Apple has also begun to create Metal Performance Shaders which are even more efficient than CoreImage filters when used with a MetalKit view. As of now, there are many more CoreImage filters, so that may give you the most flexibility.
The strategy in this tutorial is adequate for filtering video streams. If you want to give your users more flexibility with filters and other video tweaks, an editor such as VideoEditorSDK can let you focus on your application’s core functions and let a team of professionals worry about the video. Using VideoEditorSDK your users can apply filters to video as well as text annotations and more.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions or suggestions.

You like what we do? Check our careers page. Even if you can’t find your role listed, we might be interested since you are already here!

How To Record Screen Video Using ReplayKit for iOS

Walter — Tue, 21 Sep 2021 15:05:08 GMT

In this article, you will see how to use Swift and ReplayKit to add screen recording to your application. You will also see how to use the new rolling clips feature, introduced in iOS 15. The code in this article uses Swift 5 and Xcode 13. Clone this repository for a sample project and example code.

ReplayKit Basics

Apple introduced the ReplayKit framework in 2015 with iOS 9. They have refined and expanded it since then, but it still has a small API. ReplayKit uses the same shared components as AirPlay and the device screen recorder (accessible from the device Control Center). For this reason, ReplayKit functions will not work when using AirPlay or playing video. Also, the screen recording functions do not work on the Xcode simulators. You should test using a physical device.

Your app makes all its calls to the RPScreenRecorder.shared() object. This is a singleton object for your app that ReplayKit provides. There is no need to store it in a variable or pass it around in your code.

Rolling Clip Recording

In iOS 15, Apple introduced rolling clip recording. ReplayKit will maintain an updated video buffer of the most recent fifteen seconds. Your app can request the video at any time. ReplayKit comes from GameCenter. Apple’s idea for this feature is for creating a short, sharable video of the moments before an exciting event in a game. Another use can be for a user to record the steps leading up to a defect during testing. When the user is unlikely to know in advance that they will want a screen recording, rolling clips may be a good solution.

To start rolling clip buffering, include a method such as this one

RPScreenRecorder.shared().startClipBuffering { err in
  if let err = err {
    //either the user denied permission or something like
    //AVPlayer or AirPlay is using the shared recorder resources.
    print("Error attempting to start buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering started.")
  }
}

Screen recording has privacy implications. ReplayKit presents the user with a modal, requesting to record before it starts. If the user denies the request or if the buffering is unable to start, ReplayKit passes an Error object to the closure. Unlike other privacy modals, you do not have an opportunity to add your own language to the message. The modal appears as soon as you execute the .startClipBuffering command, so be sure that the user is not surprised by it. Execute the command at a logical transition in your application. After the user grants permissions, the modal will not appear again unless eight minutes pass before the next call to .startClipBuffering.

While the clip buffering runs, at any time, you can save the buffer to a URL on the device. The clip you save can be any duration up to fifteen seconds. If the recorder is unable to write the clip to disk, it will send an error.

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().exportClip(to: clipURL, duration: TimeInterval(15)) { err in
  if let err = err {
    print("An error? \(err.localizedDescription )")
  } else {
    print("Clip saved to url \(clipURL)")
}

The code above stores a fifteen-second clip at a URL in the user’s temporary directory. The temporary directory gets cleared by the system, but not while the app is running. To ensure that the files are not deleted except by the user, store them in the user’s documents directory instead.
When your app is done collecting video, use .stopClipBuffering to allow the system to clean up resources and stop the recording.

RPScreenRecorder.shared().stopClipBuffering { err in
  if let err = err {
    print("Error attempting to stop buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering stopped.")
  }
}

Recording Screen and App Audio

Rolling clip buffering runs in the background. You can also allow the user to control when to start and stop screen recording. The API looks similar and is available since iOS 10.

RPScreenRecorder.shared().startRecording { err in
  guard err == nil else { print(err.debugDescription); return }
  //code to display recording indicator
}

As with the rolling clip buffering, the system will display a permissions dialog. If the user denies permission you will get an error and the recording will not start.

The same eight-minute rule applies as for rolling clip recording. But, since the user has just taken some action to start recording, they should expect to see this modal.
Apple provides a basic view controller for editing and sharing the video.

ReplayKit passes the RPPreviewViewController to the .stopRecording handler.

RPScreenRecorder.shared().stopRecording { preview, err in
  guard let preview = preview else { print("no preview window"); return }
  //update recording controls
  preview.modalPresentationStyle = .overFullScreen
  preview.previewControllerDelegate = self
  self.present(preview, animated: true)
}

The preview object above is the RPPreviewViewController editor. You can display it as a modal or as a popover depending on your needs. The Apple provided editor will allow the user to save to their camera roll or share the clip after editing.

Notice in the code above that there is a previewControllerDelegate. You will need to add some code to the delegate to cleanup and dismiss the editor when the user is done sharing or saving. An example is below.

func previewControllerDidFinish(_ previewController: RPPreviewViewController) {
  previewController.dismiss(animated: true) { [weak self] in
  //reset the UI and show the recording controls
  }
}

Using Many `UIWindow` Objects

The ReplayKit recording will only capture the main window of your application. When designing for screen recording, you can place any UI elements in a separate UIWindow object. Although applications typically only have one UIWindow, iOS allows many windows. A UIWindow requires a .rootViewController to provide content. Then use it like any other view in your application.
The UI for this application consists of three UIWindow objects.

Apple places the status bar in a separate window from your application. The recording controls are in a separate window from the main application and will not appear in the screen recording.

To add some recording controls or a recording indicator to your app. Create a new UIWindow and add it to the same windowScene as your view.

controlsWindow = UIWindow(frame: CGRect(x: 0, y: 0, width: view.frame.width, height: 45 + view.safeAreaInsets.top)) //1
controlsWindow?.windowScene = view.window?.windowScene //2
controlsWindow?.makeKeyAndVisible() //3
let recordingIndicator = UIButton.systemButton(with: UIImage(systemName: "record.circle")!, target: self, action: #selector(recordingToggled(_:))) //4
let vc = UIViewController()
controlsWindow?.rootViewController = vc //5
vc.view.addSubview(recordingIndicator) //6
recordingIndicator.center = CGPoint(x: vc.view.center.x, y: vc.view.center.y + 20)

The code above creates the red/blue recording button and banner at the top of the view. It relies on a controlsWindow variable created at the class level. Here is what it is doing:

Creates a new UIWindow that is the width of the view and tall enough for the button and the safe area (the area around the notch and status bar)
Adds the window to the same windowScene as the current view
Makes the window visible (UIWindow objects are hidden when created) and brings it to the front of the stack of windows
Creates a button and links it to a method in your class recordingToggled(_:)
Adds a view controller to the window so that there will be some content
Adds the button to the view controller

Because the recording indicator and button are in a separate window, they will not appear in any screen recording. Be mindful when showing modal views or alert views, as they may appear behind the extra window. A good strategy is to hide the extra window when showing these views. You can use a similar strategy to hide any other UI elements that should not appear in screen recordings.

Going Further

Though Apple provides a basic editor, you can provide your own editing tools. The examples above showed the rolling clip editor writes the video to the disk. The regular recorder can also write the video to disk. You can stop recording using this code:

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().stopRecording(withOutput: clipURL) { err in
  //check for an error and process the url
}

This method would replace the .stopRecording { preview, err in method shown above.
With the video saved, you can use an editor such as the VideoEditor SDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to capture a rolling screen recording as well as how to capture the screen manually. Further, you saw how using an SDK such as VideoEditor SDK allows you to annotate and enhance your clips before sharing.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

How to Create Image Filters for iOS

Walter — Tue, 24 Aug 2021 08:09:37 GMT

In this tutorial, you will learn how to write a custom CIFilter in Metal and Swift. You can use this filter in any CoreImage pipeline.

Apple offers over 200 image filters in the CoreImage framework, but sometimes you need a little extra to make your images perfect. Apple provides two ways to create customize image filters. You can chain CIFilters together in a CIFilter subclass or wrap a CIKernel in a CIFilter subclass. Either method creates a filter that CoreImage can execute on the GPU. That makes the filter fast. CoreImage filters can filter live video without impacting the frame rate. Before iOS 11 the only way to write a custom CIKernel was to pass a string containing the code to the GPU at runtime using the OpenGL Shading Language. This had two drawbacks: it was difficult to find errors in the string of commands and the kernel was not compiled until runtime. This tutorial will show you how to add a custom filter with a Metal-based CIKernel to any Swift project in Xcode. The code samples in this tutorial use Xcode 12.5 and Swift 5.

Clone this repository for a sample project and example code that supports this tutorial. The repo also contains some other custom filters, demonstrating other aspects of Metal filters.

Adding Metal Support to an Xcode Project

The first step is to add a Metal source code file to your project.

Find the “Metal File” template, select it and click Next. Name the file and save it. At compile time, Xcode combines all of the .metal files in your project into a single .metallib file. So, organize your Metal code into lots of files or just one file as you prefer. The last step before you start writing code is to add compiler flags for CIKernel objects.

In the Build Settings, find the other Metal Compiler Flags and add an entry of -fcikernel.

Then find the Other Metal Linker Flags and add an entry of -cikernel.

Important note: any project without .metal files hides the Metal Compiler and Linker sections from build Settings.

Setting Up a Metal File for CoreImage Kernels

Metal is a general-purpose technology for writing code that will execute on the GPU. The compiler flags tell Metal that you will be writing code to work with CoreImage. With a Metal source file in your project and the compiler flags set, you’re finally ready to write some code! Open the Metal file you created above. It should be empty except for

#include <metal_stdlib>
using namespace metal;

Below these lines, import the CoreImage headers by adding:

#include <CoreImage/CoreImage.h>

and add a stub for your filter code with:

extern "C" {
  namespace coreimage {
    // KERNEL GOES HERE
  }
}

Prefix your kernel code with extern "C" and the CoreImage namespace. You write kernel code in the Metal Shader Language, which is a variation of C. If you only know Swift, you should be able to rely on Xcode’s code-completion and symbol lookup to help you as you’re learning to write Metal code. In addition to the general language reference, Apple provides a Metal Shading Language for Core Image Kernels document.

The example below creates a CIColorKernel. This kernel is optimized for changing the color value of each pixel in an image. It will change each pixel to a grayscale version of itself. A filter applying his kernel will send the color value of each pixel of the image to the kernel.

float4 grayscaleFilterKernel(sample_t s) { //1
  float gray = (s.r + s.g + s.b) / 3;      //2
  return float4(gray, gray, gray, s.a);    //3
}

Here is how this kernel works:

The filter provides the color of the current pixel to the kernel as a sample_t type. The kernel will return the new color for that pixel as a float4 type. The sample_t type is equivalent to a float4 type and is only named differently because of historical convention. A float4 type contains four float values. In the case of color, they are the red, green, blue and alpha values for the pixel. Each float has a value between zero and 1.
Calculate a grayscale version of the pixel by averaging the three color channels.
Create a new float4 with the averaged value for the three color channels and the original alpha value for the fourth.

CoreImage also provides a CIWarpKernel class that is optimized for changing the position of each pixel and a CIBlendKernel for blending two images. For more complex tasks, CoreImage also provides a general CIKernel. Apple suggests that chaining together optimized kernels in a pipeline is usually more efficient than trying to write a larger, general kernel. A color-optimized kernel only has access to the color of the current pixel. A transform-optimized kernel can only affect the position of the current pixel. A general kernel can impact color and position as well as read other values from the source image.

Wrapping a `CIKernel` with a `CIFilter`

Now you’ll make a CIFilter subclass as the interface between the Metal code and your Swift code. Every CIFilter has an outputImage variable that returns a CIImage. Generally, input variables for a filter begin with input. The grayscale filter you are making has a single input and a single output. Create a new Swift file in your project and title it “GrayscaleFilter.swift”. Then be sure to import CoreImage by adding

import CoreImage

to the top of the file. Then create the filter as a subclass of CIFilter.

class GrayscaleFilter: CIFilter {

Next, create a kernel variable as either static or lazy, so that it only gets created one time, regardless of how often it’s called.

static var kernel: CIColorKernel = { () -> CIColorKernel in
    let url = Bundle.main.url(forResource: "default",
                            withExtension: "metallib")! //1
    let data = try! Data(contentsOf: url)
    return try! CIColorKernel(functionName: "grayscaleFilterKernel",
                      fromMetalLibraryData: data) //2
}()

Look in the bundle for the compiled metal code. The metal code will have a filename of “default”.
The .metalib may contain many kernels so use the one called “grayscaleFilterKernel”

Now add a variable to hold the input image.

var inputImage: CIImage?

Finally override the outputImage variable.

override var outputImage: CIImage? {
    guard let inputImage = inputImage else { return .none }
    return GrayscaleFilter.kernel.apply(extent: inputImage.extent,
                                   roiCallback: { (index, rect) -> CGRect in
                                                  return rect
                                  }, arguments: [inputImage])
}

The apply function of the kernel takes an extent argument. For a CIImage the extent property is the width and height of the image. The function has an roiCallback that allows you to specify what part of the image the kernel uses for each calculation. For a color filter, you generally just pass back the rect. Finally, pass in any arguments as an array. Notice that there is not any type checking here. It is up to you to pass in the correct types in the correct order.

Using the Filter

With the Metal code wrapped in a CIFilter you can now apply the filter images the same as using one of the built-in filters. Create an instance of the filter, then assign the inputImage variable a CIImage and display the outputImage. Your code might look something like this:

let filter = GrayscaleFilter()
filter.inputImage = image

displayView.image = UIImage(ciImage: (filter.outputImage ?? image) ?? CIImage())

and it can render a grayscale version of an image, like in this example.

Going Further

You can write custom filters and chain them to the built-in filters to invent new effects. Many of the math formulas for effects are available on the Internet. Most are easy to translate into the Metal Shader Language to make them perfect for your application. Writing an entire image or video editing application to showcase your filters is a much larger task. Using an SDK like the IMG.LY PhotoEditorSDK or VideoEditorSDK can help you provide an app that has the features that users will expect in addition to your cool filters.

To add your filter to the PhotoEditorSDK you first wrap your CIFilter with an Effect. For a basic filter, you only need to override the newEffectFilter variable.

import PhotoEditorSDK

class GrayscaleEffect: Effect {
  override var newEffectFilter: CIFilter? {
      GrayscaleFilter()
  }
}

Then add the filter to the Effect.all array when configuring the SDK, using something like

Effect.all = [NoEffect(), GrayscaleEffect(identifier: "grayscaleFilter", displayName: "Best Gray")]

Now your filter appears the same as the other 60+ filters that ship with the SDK.

It applies to live camera previews as well as still images.

Wrapping Up

In this tutorial, you learned how to configure Xcode for custom Metal code. You also learned how to create a color filter kernel in Metal and wrap it in a CIFilter. By combining your filters with Apple’s filters in pipelines, you can create unique effects for your next app. Further, using an SDK such as PhotoEditorSDK or VideoEditorSDK allows you to showcase your filters in a full-featured image or video editing application. The PhotoEditor SDK for iOS documentation will give you deeper look into all other image adjustments including, ofcourse, the image filtering we discussed above.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

Flutter: Our new plugins made for Dart developers

Leon — Thu, 11 Mar 2021 14:12:09 GMT

Flutter enables developers to build native iOS and Android applications with a single Dart codebase whilst the application’s performance is kept at a very high native-like level. This is even more ensured by the huge amount of Flutter plugins, which wrap native code for the use in Dart.

With the rising demand for photo and video editing solutions, we were committed to provide plugins that enable developers to use our native PhotoEditor SDK and our native VideoEditor SDK within their Flutter applications. This said we are proud to announce that the Flutter plugins photo_editor_sdk and video_editor_sdk are now extending our list of cross-platform modules. The combination of both products allows you to add comprehensive image and video editing tools to your Flutter application for iOS and Android - within minutes.

Usage

While building the plugins, we focused on making the use and integration of our existing SDKs as easy as possible and minimizing the required platform-specific knowledge and skillset. Therefore, using the plugins is as easy as writing a single line of Dart code:

Customization

Furthermore, we wanted to keep the high level of customization options like cropping or resizing that our customers are used to in our products. In this sense, we created a dedicated Configuration Dart class which serves the option to customize the SDK to your needs without ever leaving the Flutter world.
Using the configuration is fast and easy:

Where to go from here?

Feel free to explore our packages, usage examples as well as the API documentation on pub.dev and/ or GitHub.
We released our new Flutter plugins under open-source licenses, so feedback and pull requests are welcome.

Make it pop - introducing Animated and Smart Stickers

Michael K. — Thu, 13 Aug 2020 18:31:18 GMT

Our latest release: brand-new sticker types for iOS and Android

We are always on the lookout to improve our editor tools to engage users to be creative and add beautifully designed content.

Our latest release for both mobile PhotoEditor SDK and VideoEditor SDK takes our stickers to the next level: With animated and smart stickers, we added popular features best known from apps like Instagram and TikTok – making editing more dynamic, expressive, and engaging. You can now capture your favorite moments by adding more context to your photos and videos. Maybe you finally landed that kickflip on a warm Monday morning, so you add the date and weather information. You can now also add your animated logo.

Our animated stickers not only provide you with a set of premade assets that come with the latest version of the editor but also let you integrate your custom ones tailored to your company and brand.

Smart Stickers on the other hand open up a whole range of new possibilities, visualizing dynamic information like weather data, location, or dates. Like our text design feature, it’s the ease of use that makes these stickers special – simply cycle through different variations with a tap.

Take a look at all updates and assets on our Mobile SDK page.

PS: Do you have feedback or suggestions for new smart stickers you want to see from us? Let us know in our Quick-Survey. We’re happy to hear from you!

React Native: Native Modules made for React developers

Alexander — Thu, 20 Feb 2020 23:00:00 GMT

*TL;DR******: How much of “native” does a React Native developer need? “Nativ”, “nati”, “nat”, “na”, “n”, or none? Version 0.60 drastically shifts this requirement towards none, even when native modules depend on external native libraries!*

Ideally, React Native (RN) should enable web developers to write React code once and thus magically create truly native Android and iOS apps without ever having to touch a single line of platform-specific native code. In reality, developers that use RN to build cross-platform apps need to be quite experienced with the respective development tool stacks and programming languages of the native platforms. This unfortunate truth is particularly the case when integrating native third-party libraries from scratch as we detailed in a previous blog post.

React Native 0.60+ — A relief for iOS

There are two ingredients to the 0.60 release that will save non-native developers countless hours of integration trouble and also make experienced native developers very happy.

First, RN 0.60 reconsidered CocoaPods as the primary build and dependency management system for the iOS platform. Using CocoaPods is now the default for every new React Native app. This choice greatly simplifies the installation and integration process of native third-party frameworks.

Second, autolinking native modules per default finally removes the need to remember the react-native link command after installing any React Native library with npm or yarn in favor of manually issuing another command, namely (cd ios && pod install) to trigger CocoaPods. This change highlights the new integral role of CocoaPods for RN. The utmost benefit of this transition is that all native dependencies of your project are installed in a fully automatic manner. Even nested native dependencies are properly resolved and installed as one would expect.

No more juggling with .framework bundles for third-party libraries with CocoaPods support!

Eventually, even the “burden” of manually pulling the CocoaPods trigger might be gone in future RN versions…

Technically, if writing headlines with Unicode characters would be a good idea, the title of this blog post should have been:

N͟a͟t͟i͟v͟e͟ M͟o͟d͟u͟l͟e͟s͟ made for React N̶a̶t̶i̶v̶e̶ developers

to underline the vanishingly required attribute “native” for developers that are capable of creating stunning React Native apps by using powerful external native libraries without the native tooling hassle.

Certificate of happiness

Building on this strong and developer-friendly foundation of RN 0.60, we are proud to introduce the all-new React Native integrations [react-native-photoeditorsdk](https://www.npmjs.com/package/react-native-photoeditorsdk) and [react-native-videoeditorsdk](https://www.npmjs.com/package/react-native-videoeditorsdk) for our lineup of native SDKs — PhotoEditor SDK and VideoEditor SDK. The interplay of both products allows you to add comprehensive image and video editing tools to your React Native app for iOS and Android — within minutes.

While crafting our new modules, the primary objective was and always will be to foster their adoption by reducing the required platform-specific knowledge and skill set to a bare minimum. We are thrilled to put a wealth of configuration and customization options at the developers’ finger tips without having them to leave the JavaScript tooling and programming environment. The complete API is typed and thoroughly documented so that any decent source code editor will auto-import missing types and display context-sensitive quick help pop-ups alongside your code.

Bye-bye native platform-specific asset management

Hello require! Our new JavaScript API heavily relies on this well-known pseudo-keyword-like function which makes handling static images as well as static non-image resources a breeze. By using it, e.g., in your App.js file, RN will do the heavy lifting and bundle all static assets with the native apps for you. The following snippet demonstrates this convenience for customizing the sticker tool (line 8, 11, 14) and for passing an input image to the editor (line 24). It presumes that the corresponding images are available in your app’s project folder.

Further examples

We created two separate repositories for a photo editor example project and a video editor example project, so that our sample assets won’t increase the size of your node_modules folder when installing our modules.

If you are looking for a step-by-step guide that explains in detail how to supercharge your RN app with image and video editing capabilities, then stay tuned for upcoming additions to our screencasts. You will also learn how to use advanced SDK features, such as serializing and reusing previously applied editing operations directly in your RN app.

Open-source community

We release our new RN modules that expose our native SDKs to the RN ecosystem under open-source licenses. Feedback and pull requests are welcome. The sources are also a valuable basis for bridging other cross-platform frameworks that we are currently not supporting with official integrations.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

The future is Video — the future is now!

Felix — Tue, 13 Aug 2019 00:00:00 GMT

VideoEditor SDK expands the powerful tools and features of PhotoEditor SDK into the realm of mobile video creation, empowering you to seamlessly integrate a cinematic experience into your mobile applications.With an intuitive and elegant UI, an extensive filter gallery, advanced adjustment tools, and crops for social aspect ratios you’ll gift your users with the ability to create engaging and professional-looking videos on the fly.

https://www.youtube.com/embed/FDuwEtBf8N8?feature=oembed

Video is one of the most effective means of communication in our era. According to Hubspot, the usage and consumption of video content continue to grow and still haven’t reached their saturation point. Video is changing the way we interact on a daily basis, and it’s here to stay.

Although most videos are shot spontaneously and then posted instantly, even slight editing can have a tremendous effect on the content’s impact. VideoEditor SDK lets your users create professional-looking footage without having to rely on external apps while using your solution. The SDK is packed with intuitive yet powerful tools that allow for the creation of an endless variety of stunning visual effects. All tools come with an instant preview and can be tailored to fit your app perfectly.

Trimming & Transformation: Keeping it relevant

The Trimming tool helps your users to keep their content on point and to get rid of unnecessary, boring, or unwanted parts with ease. Furthermore, the SDK is equipped with a Transform tool kit that unifies cropping, flipping, and rotating operations. Featuring various preset video aspect ratios most popular for Instagram (1:1), YouTube (16:9) or other platforms like Snapchat (9:16) — the tool lets your users present their content in the most appropriate format.

Video grading: The Hollywood look and feel

VideoEditor SDK ships with over 60 video filters that let your users achieve a cinematic look for their videos with a single tap. Additionally, the Adjustment section holds a variety of handy tools to tweak and fine-tune video content ranging from basic operations like brightness and contrast to more sophisticated options like exposure and gamma. Through instant preview, your users can directly see how their changes affect the final video.

Captioning and Assets: Compelling storytelling

Video is a narrational medium and VideoEditor SDK provides all necessary functions for your users to quickly turn their footage into engaging and attention-grabbing stories. They can customize their videos with captions using the Text tool or add a stunning typography design with the Text Design tool that automatically merges input text with designer-grade typography layouts.

On top of that, the SDK ships with various asset libraries for your users to personalize their content with Stickers, Frames, or Overlays to create something they’ll love to share.

Learn more about VideoEditor SDK and delight your users with powerful video editing capabilities.

How to integrate a Photo Editor into your iOS App

Felix — Wed, 10 Jul 2019 00:00:00 GMT

This tutorial is going to walk you through the integration process of the PhotoEditor SDK for iOS. You’ll learn how to setup the editor in seven minutes using CocoaPods. We created similar tutorials for Android and HTML5, so make sure to check those out as well.(Please make sure to get a trial license for the PhotoEditor SDK before integrating it.)

https://www.youtube.com/embed/AHzD4Mnm-NA?feature=oembed

Transcript

In this tutorial, we’re going to show you how to integrate the PhotoEditor SDK into your iOS app. We’re going to use Xcode and the PhotoEditor SDK’s documentation. So, first of all, we navigate to the “Getting Started” guide for iOS. We’re going to install the SDK using CocoaPods, because it is the recommended way to initialize this dependency.

We create a new Xcode project and select the “Single View App” template. Then, we assign a name to our project. For this tutorial, we’ll go with “PhotoEditorApp.” Here, it is essential that the given bundle identifier is identical to the one you used to request your license with. We decide to save our project files to the documents folder and create the new project.

Now, we change the iOS Deployment Target to version 9.0 as the SDK supports all iOS versions from 9.0 upwards, and thus, our app will be available for the highest possible number of devices. After that, we insert our previously downloaded license file into our app by simply dragging and dropping the file into our app’s folder. Here, it’s important that “Copy items if needed” is checked so that our license file will be copied to the appropriate place in our project instead of just being referenced.

Now, we can close our Xcode project. We open our folder structure and see that our project has been created in the documents folder. Now, we need the terminal to install the PhotoEditor SDK via CocoaPods. We type cd which is short for “change directory”, to change the directory of our source files. We can copy the project’s path via drag and drop. We now instruct CocoaPods to prepare Xcode for the use with CocoaPods by typing pod init. After that, CocoaPods creates the file “podfile.” We open the podfile in a text editor by double-clicking.

Here, we uncomment the second line and thus define that our project shall run from iOS version 9.0 upwards. Next, we copy this line (pod 'PhotoEditorSDK') from the PhotoEditor SDK’s documentation and paste it under # Pods for PhotoEditorApp to create a dependency between our project and the PhotoEditor SDK. We save the podfile and close the text editor. Now, we return to the terminal and instruct CocoaPods to install all dependencies by typing pod install and executing our command with the return key.

Here we can see that CocoaPods advises us to open our Xcode project with the xcworkspace file only. Furthermore, we see that there’s now a new folder called “pods.” Ideally, you wouldn’t check that folder into your source control system as it potentially contains large files, which can result in some issues. Github, for example, has a limit for the maximum file size, which the PhotoEditor SDK would exceed. So, it would be best if every user that works on the project activates pod install to install the dependencies for each processing machine.

Now, we can open the project file to incorporate our license into the file AppDelegate.swift. We copy this line from our documentation and paste it under the method UIApplication, didfinishLaunchingWithOptions.

if let licenseURL = Bundle.main.url(forResource: "license", withExtension: "") {
    PESDK.unlockWithLicense(at: licenseURL)
  }

It is important that we name our license file exactly like we did when we incorporated it into our project. In our case, that would be “ios_license” instead of “license.” Also, we can see that there is an error message, because we haven’t imported the PhotoEditor SDK yet. We simply type **import** PhotoEditorSDK, and the issue is resolved. Now our project is prepared so that we can use the PhotoEditor SDK with an active license.

Next, we’re going to prepare our app, so that it actually uses the PESDK. We configure our app with the Main.storyboard by opening the assistantview and creating a new button by just dragging and dropping it on the canvas. We’re going to use Alignment Constraints so that the button will always be displayed in the middle of the screen no matter the form factor of the device in use.

We’re going to name our button after our app and link it from our storyboard with our source code on the right-hand side via drag and drop. Now, we’re going to set a name for our method that is going to be called once the button is pressed. For this tutorial, we’ll go with startPhotoEditor.

We’ve decided that we want to start the PhotoEditor SDK with a CameraViewController. So, we copy the respective code from our documentation and paste it in the body of our method under startPhotoEditor.

let cameraViewController = CameraViewController()
present(cameraViewController, animated: true, completion: nil)

When we save the file, we get yet another error message that the camera can’t be found because we haven’t imported the PhotoEditor SDK for this source file either. Once we’ve done that, the error message disappears.

As we want our editor to start in camera mode, we have to create some entries in the Info.plist file that will allow us to use our device’s camera and grant access to our device’s image library. That is a restriction by Apple. If we didn’t create these entries, we wouldn’t be able to use our app as it would crash as soon as we open the camera. All necessary entries can be found under “Privacy.” First, we need “Camera Usage Description.” Here, we have to give a reason why we want access to our user’s camera. Said reason should be as meaningful as possible. For this tutorial, we’ll go with “We use your camera to take photos for editing.” We also want to allow the import of photos from the device’s library, so we add the identifier “Photo Library Usage Description” and type “We use your photos so that you can edit them” as description.

Now, we can run the app in the simulator. If we wanted to run the app on a physical device, we’d have to provide an Apple iOS developer account under “Team” so that the app can be signed because only registered developers are allowed to install apps on physical devices. However, we’ll go with the simulator.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

Updates to the PhotoEditor SDK

Felix — Tue, 28 May 2019 00:00:00 GMT

Over the course of the last weeks, we’ve integrated a couple of handy features into the PhotoEditor SDK. It’s now easier than ever for you to equip your service with the tools that make it the one-stop creative solution for your users.

Color Pipette

Colors are important for virtually every aspect of the digital world and a powerful tool when it comes to design, branding, social media and marketing. Color is one of the most important things to consider when communicating online. If used right these visual cues harmonize the impression and emphasize the message. But if used wrong, colors may also wreck design and content just as thoroughly as Comic Sans.

Whether it is for creating content, ads, special offers or a brand logo, finding the right color is just too important a decision to be taken lightly. The PhotoEditor SDK’s color pipette helps your users to discover and marry their content with the right colors for any purpose.

The Color Pipette is available for iOS and Android.

DuoTone Filters

Duotone is amongst the big trends in modern web design. What once served as a way of saving money on printed materials, today, adds boldness to designs and can be used to many ends. Duotone keeps things minimalistic yet captivating. In a world of full-color imagery, duotone images are a refreshing novelty as they add visual appeal by subtracting colors.

We added 8 filters to the PhotoEditor SDK’s gallery that map contrasting colors to an image based on its luminance values. Via a slider, your users can control which color the creative should lean to.

DuoTone filters are available for iOS, Android, and HTML5.

Folders

To streamline your users’ workflow and facilitate the exploration of the PhotoEditor SDK’s vast filter library, we categorized them into 7 folders. We also exchanged all filter names for more descriptive ones.

Folders are available for all platforms and completely customizable, you can rename the folders, add your own and rearrange the filters.

Snapping

To facilitate the precise layout of creatives, we added a snapping feature to our Text, Text Design, and Sticker tool. The visual elements snap to the center and borders as well as to certain angles.

You can customize the snapping properties to your wishes. Snapping is available for HTML5 iOS and Android.

Light Theme

Rumor has it that Apple is going to introduce light and dark theming in iOS 13. With our latest update, we complemented our standard dark themed UI with a light counterpart. Without any customization efforts, you can switch from dark to light theming with a single line of code.

Light theme is available for iOS and Android. For more on that, please check out our documentation.

If you think that the PhotoEditor SDK would be the right fit for your project, let us know and we’ll be in touch.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

Text ❤ Design [Pt. II]

Felix — Wed, 28 Nov 2018 00:00:00 GMT

After giving you a sneak peek at the Text Design tool in a previous blog post, we can now finally announce that it is available as a part of the PhotoEditor SDK for all platforms. Over the last weeks, we finalized the Text Design tool and added functionalities that make it a handy creative tool for many industries and use cases.

One of our main goals is to make good design more accessible to everyone by developing technology that facilitates and streamlines creative processes so that even people without previous knowledge of sophisticated editing tools can yield appealing results. The Text Design tool enables you to provide your users with the power and features necessary to give speech to their ideas and create a narrative for their visual communication.

Attention-grabbing and precise visual communication is key to engaging one’s target audience. Our novel tool automates typesetting and lets your users combine photos and text fast and easy to create stunning imagery for any tone and task. The Text Design tool features complex text layouts based on recipes crafted by professional designers. Their balanced combination of fonts, sizes, alignments, and decorations allow for the fast creation of assets with a designer look for various purposes like social media, online ads, short-term offers, flyers, product images, posters, promo material and so on. Especially for small companies and teams, without the budget for a professional designer or a dedicated design department, the Text Design Tool can save a lot of time and stress by reducing what would usually take hours to a single tap.

All it takes is an idea. The Text Design Tool tends to 90% of the execution.

The tool makes design just as fast as your users’ ideas. Combined with the PhotoEditor SDK’s extensive feature suite, branding opportunities through sticker upload and the photo roll with endless source material, your users can achieve a harmonious and nuanced visual communication off the bat to stand out against their competitors. The designs help convey your users’ message and can be used to create a textural hierarchy to guide the spectators view to the essential infos for your users’ goals. Furthermore, the shuffle functionality randomizes fonts, alignments, and decorations thus offering endless possibilities for creative expression.

A recent change we made to the mobile versions of the Text Design tool is lifting the limitation of one design per creative to allow further artistic expression and get the most out of the interplay and combination of the various designs. On top of that, we optimized the inverting feature of the tool to allow for the adjustment of the inverted designs’ padding and size.

If you want to empower your users to give a voice to their creatives head over to IMG.LY and get in touch with our sales team.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

Almost Getting Sherlocked by Apple’s Core Image Team

Malte — Tue, 12 Jun 2018 00:00:00 GMT

During each World Wide Developer Conference keynote, app developers all over the world are fearing their app may get sherlocked by Apple. This has happened to flashlight apps when Apple introduced a simple toggle in the control center, to f.lux when Apple added exactly the same functionality to macOS and iOS, in parts to Dropbox with the launch of iCloud Drive and to many other developers. This year there was only one obvious of such things, the Workflow app which was acquired a few month ago and instantly turned into Siri Shortcuts by Apple. So no real ‘sherlocking’. But when we skimmed the newest APIs, release notes and session descriptions after the keynote, we found out that the technology behind our Portrait by img.ly app might have gotten sherlocked by the Core Image team. The reason is perfectly summarized in this tweet:

“Portrait Segmentation API

A new API for third-party developers allows for the separation of layers in a photo, such as separating the background from the foreground.”

Hi @halidecamera
— Preshit Deorukhkar (@preshit) June 4, 2018

How we found out about the new Portrait Segmentation API.

Quick spoiler: While Core Images new API is impressive, our technology is still ahead in terms of general availability, required hardware and cross-platform compatibility. But let’s start from the beginning:

We started working on automated image segmentation in 2016 and decided to focus on portraits in 2017. After spending most of the year on building a custom deep learning model, running hundreds of experiments and tweaking our post processing pipeline, we finally released the Portrait app in fall 2017. The app is able to generate portrait segmentations in real-time and allows the user to take a nice selfie, which is then automatically stylized using the segmentation mask and some post processing. This allows for sophisticated effects and got great reception all over the world. The app even got featured multiple times in many different countries. Recently we started looking into inferring depth, as we wanted to bring depth based features to our PhotoEditor SDK, but needed support on more iPhone devices, Android and especially required depth data for existing images.

Reading the tweet above, it looked like Apple just added the Portrait apps core functionality to the Core Image framework, making it available to all developers and users running or targeting iOS 12. A little worried about the new competition, we immediately looked into the docs and eagerly waited for the session on last Thursday to see if we’d soon need to find a new unique way of using technology to empower creativity.

We found out that all you need to do in order to get a segmentation mask along with your image is toggle the following flags when requesting a photo capture from your AVCapturePhotoOutput:

let settings: AVCapturePhotoSettings = AVCapturePhotoSettings()
settings.isDepthDataDeliveryEnabled = true
settings.isPortraitEffectsMatteDeliveryEnabled = true
...

After the photo is captured, you’re then able to extract the matte in the didFinishProcessingPhoto callback. And voila, you now have the captured image, a depth map and the mask separating fore- and background available:

Of course, we quickly spun up our own models and algorithms and compared the portrait matte, as well as the depth map to Apples results. As there is no way to rerun Apples matting algorithm for an existing image, we took one of Apples samples and compared their mask with results from our model:

While it may not have been the greatest idea to pick Apples shiny developer example, Apples mask is a little more detailed and for this particular image our model missed parts of the neck on the left. But overall we were still very happy with our results, especially when considering, that everything was generated from the plain image and didn’t require any dedicated hardware like dual cameras or a TrueDepth sensor. We did of course expect a superior depth map from Apple, as we’re clearly lacking data and are still actively working on the depth model used to generate the image above. But Apples depthmap interestingly has some issues around the neck as well, despite their use of a dual camera system. And keep in mind, that our results were all processed on the mobile device, entirely based on the RGB data contained in the image, and could be repeated on an Android phone, older iOS devices and even your browser.

When we wanted to try more samples, we quickly noticed the major limitations of Apples API: As the portrait matte capture is only available in combination with depth data, it’s limited to the iPhone 7 Plus, 8 Plus and iPhone X. And, most important to us, portrait mattes taken using the front camera are exclusively limited to the iPhone X and it’s TrueDepth sensor array. So for our Portrait app, switching to the new API would require us to ditch our live preview and go iPhone X and iOS 12 only. This was enough to calm our minds and we started thinking deeper about Apples technology:

While the depth requirement is annoying, it also explains the higher quality of Apples predictions. Our model is trained to solve the individual problems of portrait matting and depth map inference as a whole, but Apple is able to focus on improving the edges, while the ‘rough’ foreground/background segmentation is handled by masking based on the depth of the face detected within the image. We might be able to combine both models as well, but that would make the segmentation model currently used in the Portrait app obsolete and would most certainly kill the real-time functionality.

Overall, Apples technology is, as almost always, pretty impressive and the portrait matte generation works flawless, but we can now say, that for the use within our app, we reach a good quality with our current algorithms and think that a real-time preview is more important to our users. For our PhotoEditor SDK we’d happily use the high fidelity maps generated by iOS 12, but the hardware requirements are not yet suitable for SDK deployment. Our algorithms on the other hand, neither require a depth map, nor are we limited to processing after the image was captured, but can perform real-time inference on devices as old as the iPhone 6S. And the best of all: Once a TrueDepth camera is more common and everyone is running iOS 12, we can still use the Portrait Segmentation API and enjoy it’s simplicity.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Text ❤ Design

Felix — Thu, 12 Apr 2018 00:00:00 GMT

Introducing the Text Design Tool

There is power in words just as there is in design. In another advance in making good design more accessible to everyone, today, we are proud to give you a sneak peek at an upcoming product we created in addition to the PhotoEditor SDK: The Text Design Tool merges input text with typography, creating stunning designs for a multitude of use-cases.

While most of us are accustomed to the use of photo editing and enhancement tools, results often lack the desired appeal when it comes to typography and layout. Simply because most people don’t know about character or line spacing and other rules of typesetting and let’s be honest, why should they? But as well-designed text layout can add a lot to its expressiveness, we wanted to equip everyone with a tool that automates typesetting and layout, similar to apps like Typorama.

To transform our vision into a user-friendly tool, we first created recipes and set layout rules for different combinations of fonts with decoration elements. The Text Design Tool currently holds 16 layout designs for various use-cases. We worked close with the designers Tommi Gutscher and Ramona Schratt to conceptualize the designs and their different flavors. Through simple keyboard entries, you can populate the designs with your own words. The tool then lays out your text according to our recipes upon a single tap. You can then fine-tune your creative by choosing from 15 different text colors or by using the randomize functionality that shuffles the fonts, alignments and decorations until the creative lives up to your vision. You can even create a mask from your text that lets the background image shine through.

But that is not the end of the road, as, for the future, we will work with a wide range of artists and designers to expand our tool with a broad spectrum of new and exciting designs that can be applied to a variety of use-cases.

The Text Design Tool is currently available for iOS only. Versions for Android and HTML5 are going to be released in the next weeks.

Download our Photo Editor App for iOS to get a hands-on experience with the Text Design Tool.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

How to recreate Snapchat’s UI within a single day with the PhotoEditor SDK for Android

Niklas — Sun, 18 Feb 2018 23:00:00 GMT

Is it possible to completely rejig the UI of the PhotoEditor SDK within a day? Principally, everything is possible, however, for me as a prospective Java developer this whole undertaking seemed rather tough at first. The task at hand was to rebuild the SDK’s UI to match popular apps like Snapchat or Instagram Stories in only a few steps. A few weeks ago, we already did something similar with the iOS SDK, however, as there are some differences between the platforms the whole process had to be repeated for Android.

As my colleague Malte Baumann already pointed out in his blog post, the main idea behind this endeavor is to explore the possibilities and limitations of the SDK’s customizable UI. But above that, it is also a great way of making oneself familiar with the SDK’s architecture and functionalities. Below, I’m going to explain some of the technical aspects as well as the setup of the story UI in a little more detail.

The Architecture

The interface is composed of several layers. The first layer is the EditorRootView that contains all other views and is, therefore, their parent object. Beneath that lies the EditorPreview that is responsible for the display of the selected image. Following comes the BrushLayer that can be drawn upon. In this specific layer array, the brush cannot be placed in front of a sticker. The stickers, however, are all generated in dedicated layers which makes it easy to adjust their order. Once selected, a sticker will automatically be brought to the front. The UI elements can be switched visible and invisible at will; they lie at the very top so that no sticker can cover them.

At the heart of the UI lies the StateHandler that functions as the communication interface with the SDK. The StateHandler includes the EditorShowState and the HistoryState. The HistoryState allows for the retrieval of methods that are necessary for the function of the “undo” button. For example, every new brush stroke adds a new memory state that can be retrieved by the HistoryState and hence reversed. The EditorShowState manages the EditMode that allows for the activation of the different SDK tools. To draw with the brush tool, the EditMode simply has to be set to brush. Color and stroke width are then controlled by the BrushLayerSettings.

There is a dedicated button to add emojis that displays a RecyclerView filled with every sticker available in the config upon press. Once a sticker is selected a new layer with matching StickerLayerSettings is added to the image.

Although text is realized as a sticker as well within the SDK, it is a little more complicated to generate a text sticker as the required TextStickerConfig has to be generated first. The text tool can create text stickers through keyboard entries. The text and background color, as well as the text alignment, can be adjusted separately. Those functions can be controlled via different buttons. However, as the SDK has no canned methods for these operations, I had to write widgets that control the buttons. The different button states are interpreted and executed in the TextPanel.

To export an image, you simply have to launch the saveImage method.

The Result

Thanks to the documentation, the execution of the UI revamp was pretty straightforward even without previous knowledge or source code access. As main functionalities like the brush tool or the undo function are already built into the SDK, it was a manageable task to integrate those into the new UI. Against that, the styling of the SeekBar that is needed to adjust the brush size was a little more time-consuming.

The design was mainly realized through layouts with fairly simple code. Even though there were some minor hurdles to overcome, the allotted time of a working day should be adequate for a more experienced developer.

You can find the code for this example UI in our repository.

Thanks to Felix Rau and Malte Baumann!

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

How to build Instagram’s Story Editor in a Day

Malte — Wed, 13 Dec 2017 00:00:00 GMT

A common question we get asked from our customers, is whether they’ll be able to create an entirely different user interface using our PhotoEditor SDK. The SDK comes with it’s own customizable UI, but customization are of course limited to a certain extent. As our SDK is used in many different use cases and contexts, we like to explore what’s possible with the PhotoEditor SDK and its included components. Here, we decided to build a UI similar to Instagram’s Stories or Snapchat using our SDK. By now, this ‘Story UI’ has become a popular way of quickly designing with different elements (stickers, brush and text) rather than enhancing and styling the image only.

So we grabbed our own docs and headed out to create a demo in which we recreate the Instagram Stories using components from the PhotoEditor SDK. This article presents our approach by starting with a general overview and then diving into the different view controllers to look at specific implementation details. You can follow along by downloading the accompanying code.

Architecture and Overview

Any image editing interface is naturally centered around some sort of canvas or preview. This is where the image with all operations applied is rendered and any changes are shown to the user. All tools add their own interface elements to allow modifications like adding stickers, text or brush strokes to the image. To create such a hierarchy of tools, the PhotoEditor SDK makes heavy use of an iOS pattern called view controller containment. The root view controller, an EditViewController in this case, manages a series of child view controllers:

A PhotoEditPreviewController that handles the internal model and all rendering, as well as the rendering canvas itself
A TextSpriteEditController above the PhotoEditPreviewController that allows selection and manipulation of text sprites
A StickerSpriteEditController above the PhotoEditPreviewControllermanages selection and manipulation of stickers

The child view controllers may manage additional view controllers themselves, but we only need to manage the topmost objects and wire their interfaces together. This is done within the EditViewController who is also responsible for presenting the view controllers that implement the different tools, StickerViewController, BrushViewController and TextViewController.

These are mainly responsible for managing the interface and adjusting the model accordingly, while the ‘real’ work is being done in the background by the PhotoEditPreviewController whenever the model gets updated. Thanks to the SDK, creating our BrushViewController boils down to creating and wiring a BrushEditController and adding its view, color and size controls to the BrushViewControllers view. This creates a fully fledged brush tool with OpenGL rendering, color and brush size adjustments, and great performance in just 89 lines of code.

EditViewController

As described in the previous section, the EditViewController is the root view controller of our demo application and is responsible for showing a preview, presenting tools and rendering the final output. To do so, we add a PreviewEditViewController and SpriteEditControllers to the root view and register the EditViewController as their delegate. Registering as a delegate, allows the EditViewController to wire all child view controllers and pass data between them. In order to do so, we just need to return objects of other view controllers or react to model changes. All controllers then use the EditViewController to ask for objects or data they need, for example the current size of the preview view, in order to rearrange interface elements or calculate model updates. As an example, the spriteEditControllerPreviewView(_ spriteEditController:) method of the SpriteEditControllerDelegate protocol, asks for the current preview view. All we need to do in our EditViewController is to get this view from our PreviewEditViewController and return it:

Sprite Handling

Stickers and text sprites are both sprites, but there are different SpriteEditController subclasses for each sprite type, because of differing gestures and UI elements. The Instagram UI allows the editing of both sticker and text sprites within a single view, so we need to add both specific controllers, StickerSprite- and TextSpriteEditController to our EditViewController and dynamically enable the appropriate one depending on the currently selected sprite.

StickerViewController

For adding stickers, the Instagram Stories UI offers a single button that presents a grid of all available stickers. Replicating this using the PhotoEditor SDK is easy to do, as the SDK uses a collection view for presentation as well. This collection view is embedded in a StickerSelectionController which we add to our StickerViewController. Once again, after registering as the StickerSelectionControllers delegate, we get notified upon selection of any sticker and just need to change the PhotoEditModel accordingly. To do so, we create a link between the root and the currently presented view controller using the StickerViewControllerDelegate protocol. Once a sticker has been selected, we create or update a StickerSpriteModel, pass it to the EditViewController, which updates the PreviewEditViewController model:

This triggers a new rendering pass and the sticker appears on screen. All that’s left to do now is closing the StickerViewController. Again, we use the delegate pattern to ask the EditViewController about the referenceSize that’s needed to calculate the sticker’s normalized size.

BrushViewController

This view controller, just as described above, is just a wrapper around the BrushEditController that adds a color and size selection. Once again, all interaction is handled through delegates and upon closing of the brush tool, the internal model is updated. To support undo, we need to pass the PreviewEditViewController’s undoController and begin a new group, whenever we fire up the tool.

TextViewController

The text editor Instagram created within their Stories UI is essentially a fullscreen textfield with additional controls for color and size selection. Ignoring the customized design, this can easily be recreated using stock iOS components and essentially only requires listening for keyboard notifications to handle the layout and adjusting the textfield’s properties according to user interactions. Once finished, we create a corresponding TextSpriteModel that is positioned below the UITextField and notify the EditViewController using the delegate. After dismissing the TextViewController the text rendering is handled by the PhotoEditPreviewController. When reselecting an existing label, we just prefill the UITextField with it’s contents and update the model when the label textfield changes. To ensure the UITextField is the only place the text is currently visible, we hide the spriteView we previously selected and show it again, once editing has finished.

The Result

Recreating the Instagram Stories UI seemed rather challenging at first, but using our PhotoEditor SDK, it quickly turned into a very rewarding project. Being able to create something like a brush tool without the need to actually implement any drawing or gesture handling is amazing and helped to create a working prototype in no time. Tricky details like sprite transformations, remote loading of stickers or undo management were entirely handled by the SDK and we could completely focus on the interface itself.

For more details and some actual code, take a look at the repository. We documented all tricky parts and you should be able to start your own implementation using the code. Please keep in mind, that the code may target a slightly older PhotoEditor SDK version, so check for a new version before you start your journey.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

iOS – IMG.LY Blog

Building Custom Variable SF Symbols in Figma for our CreativeEditor SDK for iOS

What Are Variable SF Symbols?

The Early Struggle

Why We Wanted “SF Symbol Generator”

Inside SF Symbol Generator

Our Step-By-Step Workflow

Importing into the SF Symbols App

Optional Extra Step: Preparing More Complex Symbols

Lessons Learned

Conclusion

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Setting Up Your Project

Asking for Permissions

Making Video Recordings

Configuring the Camera

Presenting the Controller

Building Your Creation

Configuring the Editors Assets

Exporting the Finished Video

Setting Up a Playback Controller

Where to Go From Here

CE.SDK v1.8.0 Release

CE.SDK v1.8.0

Node.js Platform Support

Common Touch Gesture Support for iOS

Emoji Support

Image Straightening in Advanced & Default UI

Multi Selection Rotation

Outlook

Time-Based Sprites for VE.SDK on iOS and Android

Time-Based Sprites

Why is Video Important?

Build a Simple Real-Time Video Editor with Metal for iOS

Setting Up a MTKView

Drawing in an MTKView

Working with Local Files

Working with Streams

Working with the Cameras

Going Further

How to Merge Videos in iOS with Swift

Making a Composition

Adding Video Clips

Using the Composition

Going Further

Wrapping Up

How to Add a Filter to a Video Stream in iOS

Streams or Files

Setting up AVPlayer

Using AVPlayerItemVideoOutput

Creating the Display Link

Starting the Player

Extracting the Pixel Buffer

Going Further

How To Record Screen Video Using ReplayKit for iOS

ReplayKit Basics

Rolling Clip Recording

Recording Screen and App Audio

Using Many UIWindow Objects

Going Further

Wrapping Up

How to Create Image Filters for iOS

Adding Metal Support to an Xcode Project

Setting Up a Metal File for CoreImage Kernels

Wrapping a `CIKernel` with a `CIFilter`

Using the Filter

Going Further

Wrapping Up

Flutter: Our new plugins made for Dart developers

Usage

Customization

Where to go from here?

Make it pop - introducing Animated and Smart Stickers

React Native: Native Modules made for React developers

React Native 0.60+ — A relief for iOS

Certificate of happiness

Bye-bye native platform-specific asset management

Further examples

Open-source community

The future is Video — the future is now!

Using Many `UIWindow` Objects