Tutorials – IMG.LY Blog

A Modern Video Editor SDK for your Flutter App

Jan — Fri, 13 Sep 2024 11:26:15 GMT

Video has been the only type of content with steadily growing demand, and it will remain a staple and an expected medium for users to create and consume.

With platforms such as TikTok, Instagram Reels, and YouTube Shorts dominating and shaping user habits, the ability to create and edit videos directly within your app can significantly increase engagement.

In this post, we’ll show how to integrate a video editor into a Flutter app using CreativeEditor SDK and customize it for specific use cases, like creating TikTok-style short videos.

You can check out the code for the default and custom Flutter video editor on GitHub

Why add a Video Editor to Your Flutter App?

The last decade has made one thing clear: video content is king. Particularly, short-form dominates digital content. At the forefront of this trend are apps like TikTok, which made people more comfortable creating, editing, and sharing bite-sized clips.

These apps have capitalized on their ability to let users apply filters, and add audio overlays, text, and more, empowering users at all skill levels to make professional-looking videos with minimal effort.

A host of different use cases stand to profit from this trend by enhancing their user experience with video editing features, particularly marketing tech tooling and apps with a messaging or social media component.

In general, video creation lowers the threshold for user-generated content and can positively impact distribution and product engagement.

Flutter, Google’s UI toolkit for building apps across platforms from a single code base, is a perfect fit for IMG.LY’s Video Editor. What often motivates the choice to use Flutter is that It ensures performance, flexibility, and a seamless user experience on iOS and Android devices. Similarly, our SDK is designed to offer a native cross-platform experience, while relying on the same underlying graphics processing engine, ensuring consistency and interoperability.

Getting Started: Integrating the Video Editor in Flutter

Let’s dive into how to set up a video editor in your Flutter app using CreativeEditor SDK.

Requirements

Before we get into the details, here’s what you’ll need:

Flutter 3.16.0+
Dart 2.12.0+
iOS 16 or later (as the video editor is currently only available for iOS, Android coming soon)
Swift 5.10 and Xcode 15.4

First, ensure your pubspec.yml file includes the necessary dependencies:

dependencies:
  flutter:
    sdk: flutter
  imgly_editor: 1.34.0

Once your environment is set up, you’re ready to start coding.

Setting Up the Editor

After adding the imgly_editor dependency, import the package in your code:

import "package:imgly_editor/imgly_editor.dart";

The next step is to initialize the editor by providing an instance of EditorSettings. This setup requires a license key, which you can obtain from CreativeEditor SDK, and an optional userId that helps track monthly active users (MAUs) across different devices.

Here’s a basic example of how to open the video editor:

class VideoEditorSolution {
  /// Opens the editor.
  void openEditor() async {
    final settings = EditorSettings(
        license: "YOUR_LICENSE",
        userId: "YOUR_USER_ID"
    );

    const preset = EditorPreset.video;

		/// Open the editor and retrieve the editing result.
    final result = await IMGLYEditor.openEditor(
        preset: preset,
        settings: settings
    );
  }
}

This launches the editor with the video preset, enabling users to trim, cut, and add filters, text overlays, music, and more to their videos.

Use Cases: Building a TikTok-Like Experience

Now that we have integrated the basic editor configuration into your Flutter app, let’s revisit some prominent use cases and how to configure the editor to support them.

Short-Form Video Creation

For a short-form video, TikTok-like user experience, the most important factor is the ease of use of any video editor and content such as stickers or audio overlays.

The following SDK capabilities can help you achieve that goal:

Control Video Duration and Position in Time: Users should be able to trim video clips on a timeline and position them relative to other tracks, such as audio clips.
Filters & Effects: Filters are essential for setting the tone of the video. Whether users want a retro, high contrast, or soft pastel look, filters provide creative control over the visual style.
Text & Stickers: Allow users to add text captions—especially important for content often watched on mute. Stickers and emojis can add a playful element to videos.
Music & Audio: Like TikTok, you can allow users to add background music or sound effects. Providing a library of popular tracks or sound snippets can enhance the content creation experience.
Video Templates: Give your users a head start with customizable video templates for different themes, topics, and occasions. IMG.LY’s CE.SDK comes with a web video editor to help build and deliver these templates to your Flutter app. You can explore a range of different templates in our video editor demo.

By integrating these features, you’ll empower users to create engaging, shareable content, elevating your app’s video creation experience to the level of consumer-grade social media apps.

Influencers and Marketing

If your app is targeting influencers or businesses, the ability to quickly create branded video content is key. You can offer:

Branded Templates: Pre-made templates that align with a brand’s style, allowing users to easily drop in their video footage and text while maintaining brand consistency. Again, check out the web editor demo to see how to build video templates.
Watermarks: Adding a watermark or logo as a brand signifier to videos can help with brand visibility and make sure content always points back to the creator.

This setup allows influencers and brands to create professional-looking videos that they can share across social media platforms with ease.

eCommerce and User-Generated Content

Video is a powerful medium to showcase products, especially on marketplaces that can leverage user-generated content and enable vendors as well as customers to create product or review videos.

For example, users could create unboxing videos, product reviews, or tutorials using the same trimming, filter, and music tools offered in a TikTok-style editor. These videos can then be shared on social media or used within the app to enhance the shopping experience.

Likewise, showcasing products using authentic demo videos tap into familiar patterns of video consumption and can boost sales for vendors.

Customization Options with CreativeEditor SDK

We have designed the Flutter plugin to be adaptable to match your brand and use case-specific requirements. Soon the plugin will also support more advanced customization options such as the ability to activate or deactivate features by default.

UI Customization: Modify the look and feel of the editor to match your app’s branding. We provide options for theming, styling color palettes, and callbacks to hook into editor events.
Video Presets: You can create custom video presets based on your users’ needs. For instance, you may want to limit video length or optimize for certain resolutions, depending on the target platform.
Video Templates: You can use the CE.SDK web UI to generate any number of video templates including collages, text designs, and animations to give your users professional-looking starting points for their video designs.
Assets: Provide custom filters, fonts, and stickers to enhance the user experience. For niche apps, this can be an excellent way to engage users by offering content that feels unique to your community.

Conclusion

Integrating a video editor into your Flutter app will improve your UX, and help boost engagement, retention, and potential distribution of your product, whether you’re building a social media platform, an influencer tool, or an e-commerce app. With CreativeEditor SDK, you can create a TikTok-like video editing experience or offer specialized tools for businesses and creators.

By following the steps outlined in this post, you can bring professional-level video editing features to your users. Explore CE.SDK’s video capabilities and dive into the docs.

Stay tuned for more updates, and please reach out if you have any questions. Thank you for reading.

Over 3,000 creative professionals gain early access to our new features, demos, and updates—don’t miss out, and subscribe to our newsletter.

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Walter — Thu, 30 Mar 2023 11:20:26 GMT

Now that TikTok is facing a ban in the US any day now, we better wait in the wings with an alternative ready to go and scoop up those millions of hobby dancers, micro bloggers, and would-be influencers.

In this article, you’ll learn how to use Swift, SwiftUI, and the IMG.LY CreativeEditor SDK to build a simple iOS app for recording, editing, and viewing short videos. This app features views providing core functionalities similar to making a post in a video-sharing platform like TikTok.

The editing controller is the central hub. It allows users to capture video clips and enhance their videos by adding filters, stickers, music, and other effects. CreativeEditor SDK calls the overall project a “scene” and the parts (clips, stickers, text, etc.) “blocks”. Once the scene is ready, the user can compose it into a standard video file and export it. Finally, the playback controller showcases the finished projects using Apple’s standard VideoPlayer struct. For recording and editing, the CreativeEditor SDK’s Camera and Video Editor handle the heavy lifting, enabling swift development of a robust app. You can easily extend its features, filters, and creative options as your users demand.

Follow along to see how to capture video, enhance it, and export the finished product as a polished movie file.

Setting Up Your Project

You can try out the TikTok clone by cloning the GitHub repository supporting the article. Otherwise, follow this step-by-step tutorial and learn how to build the TikTok clone with CreativeEditor SDK by yourself.

To add the CreativeEditor SDK to an Xcode project, include it with your other package dependencies. Use the URL below to add the package, either to your Package.swift file or using File > Add Packages...

https://github.com/imgly/IMGLYUI-swift

The package includes a number of different frameworks:

IMGLYUI, an umbrella framework of all of the different editors and a camera.
IMGLYCamera, a standalone camera View. This is the same camera as the editors use.
IMGLYApparelEditor, IMGLYDesignEditor, IMGLYPostcardEditor, and IMGLYVideoEditor which are four different configurations to support different use cases.

Each of these frameworks uses the IMG.LY Engine for image processing. To include them in your project, navigate to the ‘Frameworks, Libraries, and Embedded Content’ section in your app’s General settings. Click the ’+’ button and select the desired frameworks. Initially, consider adding the IMGLYUI package, as it encompasses all other packages. Before releasing your app, review and include only the necessary components to minimize app size and avoid unused code.

Once Xcode downloads and resolves the packages, you just need to include IMGLYVideoEditor in any of the files that will reference the editor in your app. When you’re working with the underlying engine directly you’ll also want to include IMGLYEngine. You’ll only work with the engine as you start to customize the workflow.

Asking for Permissions

Before any app can access the phone’s microphone, camera, documents or photo library, the user must specifically give permission to record audio or video as well as access the user’s Photo library. If your app doesn’t secure these permissions properly before trying to use the camera the app will crash and also probably won’t get through Apple’s app review. In earlier versions of iOS you could develop an app on the Simulator without asking permissions, but always on the device the app will crash. In current versions of iOS the app crashes regardless of platform. The dialogue requesting permissions is a system dialog and looks like this:

You do not have the ability to change this dialog. You can supply the reason you are asking for the permission using a key in the info.plist file in the Xcode project. In the example above “Lets you record videos” is in the info.plist. For the video camera you will have to ask for video and audio permission. The first step is to add a short message to the user in the info.plist.

For video the key to add is NSCameraUsageDescription and for the microphone you need to add NSMicrophoneUsageDescription. Whenever your app attempts to use the camera, iOS will first check to see if the user has already granted access. If not, it will then display the dialogs using your entries in the info.plist or crash if you have not provided entries. The user might be surprised by these dialog boxes and may accidentally tap Don't Allow if they are trying to quickly launch the camera. It is better practice to set up some kind of onboarding view and secure the permissions before you need them. You might have a view that explains what you are going to ask for and then displays the permission dialog.

switch AVCaptureDevice.authorizationStatus(for: .video) {
  case .authorized:
    //the user has authroized permission!
    break
  case .denied:
    //the user has previously denied permission!
    //perhaps we should ask them again
    break
  case .restricted:
    //the user cannot grant permission!
    break
  case .notDetermined:
    //the user has never been asked for permission
    //so let's ask them now
    AVCaptureDevice.requestAccess(for: .video) { accessGranted in
      if accessGranted {
        //the user has authorized permission!
      } else {
        //the user has denied permission!
      }
    }
  @unknown default:
    break
}

The snippet above lets your app read the permission status for the video camera and ask for permission if it has not been granted. To request access to the microphone pass .audio to the AVCaptureDevice.authorizationStatus(for:) and AVCaptureDevice.requestAccess(for:) functions. The AVCaptureDevice.requestAccess(for:) command is what displays the system dialog actually requesting access. The accessGranted variable reports back to your app what the user chose. You can take care of getting the permissions any time but when the CreativeEditor SDK first launches it will attempt to access the camera, so the dialog will appear if the user has not already granted permission.

The .restricted case is fairly new. In this case, there are policies on the phone that prohibit the user from granting permission to your app. In addition to asking permissions during onboarding, it is good practice to check for permission every time before you attempt to run any code that uses the camera. The user can change permissions using the Settings app on their phone at any time. If the user has denied permissions and you present the video camera anyway, your app will record black video frames and silence on audio tracks.

In addition to asking for camera and microphone permissions, your app will probably want to access the photos on the user’s phone. You will need to add an entry to the info.plist for NSPhotoLibraryUsageDescription. As with the camera and the microphone, the dialog will appear when the Video Editor first attempts access, but you can give your user some ability to grant permission during onboarding. For the user’s photos, you can check the authorization status using

let status = PHPhotoLibrary.authroizationStatus(for: .readWrite)

As with the camera, the photo library has PHPhotoLibrary.requestAuthroization(for: .readWrite) but instead of just returning a “granted” or “denied” status, this command returns the actual new status. In addition to the status values for the camera, the photo library may return a .limited status meaning that the user has granted permission to only some of the photos. If the user has chosen to share only some of their photos, Apple provides some views you can present so that the user can manage the photos. Any videos that your app saves to the user’s photo library will always be available when the user has chosen .limited. You can read more about how to work with permissions and the user’s photo library by reading this Apple article Delivering an Enhanced Privacy Experience in Your Photos App.

Making Video Recordings

The start of a great TikTok is the camera. When our user wants to make a new video clip with our app, they tap on the button at the bottom of the initial screen to start the creation workflow.

Here there is a decision to make. TikTok and the CreativeEditor SDK can each start with the creation controls and a video preview. When using the CreativeEditor SDK though, you can also start with some video that came from somewhere else. So if you already have some camera code you like, or if you want to start with some video clip from somewhere else, you can insert that into the editor when it launches. You can do that by leveraging the onCreate modifier of the editor.

let engine = EngineSettings(license: "<your license code>",
                              userID: "<your unique user id>")

VideoEditor(engine)
  .imgly.onCreate({ engine in
    try await engine.scene.create(fromVideo: Bundle.main.url(forResource: "dog_water", withExtension: "mov")!)

    //set other defaults
  }

In the code above, the app initializes an instance of the Video Editor view and then reads the dog_water.mov file from the app bundle. launches the editor with that.

Whether starting with a video or starting with a blank canvas a user can add video clips to their creation using the embedded camera.

The image above compares two different camera controllers: the left one from the TikTok app and the right showing the default settings of the MobileCamera, either standalone or within a Video Editor scene. Key controls in each are marked with blue numbers:

1 - Flip camera button
2 - Flash button
3 - Recording time indicator
4 - Add video and finish buttons
5 - Cancel button

While the TikTok camera integrates several editing functions during video capture, the Video Editor separates these tasks, focusing the camera purely on recording. Both systems allow users to start and stop video recording to compile a series of clips before moving to the editing phase.

Configuring the Camera

The camera used inside of the Video Editor UI isn’t customizable. You have limited options to customize when you use the standalone camera though. You can set the colors of the buttons, helpful if you have a preferred color scheme, and you can set a maximum duration for the video recording allowed.

let config = CameraConfiguration(
  recordingColor: .orange,
  highlightColor: .blue,
  maxTotalDuration: 30,
  allowExceedingMaxDuration: false)

Camera(engine, config: config) { result in
 //handle the recorded video collection
}

The code above sets the record button to orange while recording and the clip view to orange. There is a maximum duration of 30 seconds for all videos in the collection and the button to save the clips highlighted in blue.

If your app requires further customization, you’ll probably want to make your own camera view and work with the IMG.LY Creative Engine directly.

The documentation website is a great resource for understanding how much you can add to and customize the IMGLYUI resources and when you’ll need to drop down to the level of the engine.

Presenting the Controller

As demonstrated above, both the Video Editor and the Camera are View structures that can be presented directly or through overlays such as overlay or fullScreenCover. However, they are typically enclosed in a NavigationView to utilize the navigation bar for displaying export and other buttons. Therefore, when triggering the Video Editor from a button, it’s crucial to embed it within a NavigationView.

Button("Edit the Video") {
  isPresented = true
}
 .fullScreenCover(isPresented: $isPresented) {
   NavigationView {
     VideoEditor(engine)
      //any setup modifiers
   }

Building Your Creation

After capturing a video, the app displays an editor to complete the creation. The editing tools are in the red circles. TikTok provides about ten different tools on the editing screen. Additionally, TikTok provided some editing tools on the capture screen. The CreativeEditor SDK apps provide a scrolling set of asset libraries along the bottom for items to add to the scene. But when any of the elements of the scene, like your video or an audio clip, are highlighted, the bottom row becomes tools to manipulate that element. The basic Asset Library types are uploads, videos, audio, images, text, shapes, and stickers.

How to configure and customize each of these tools is beyond the scope of this tutorial. But, a lot of the customization of the assets, fonts, blurs and even filters can be done without code. By design, the CreativeEditor SDK downloads assets from a central server. This is a great way for you to be able to push updated filters, stickers, and other assets to your users without going through the app update process. In the next section, we’ll walk through that and provide some links for further research.

Configuring the Editors Assets

The IMG.LY Engine looks for assets for stickers, vector shapes, LUT filters, color filters, color palettes, blurs, and fonts using a URL by default. The design of the system is that you would have your own server supporting your app. However, assets in the app bundle are also accessible by URL, so as long as the assets are in the folder format that the engine expects, it doesn’t matter whether they are local or remote.

In the documentation for this process you can find a URL to the default asset bundle. Once you download that bundle you can add it to your app. The bundle is structured much like a regular iOS Assets.catalog with some JSON containing metadata about the asset and a file that provides the actual asset. For some of the assets like filters and vectors, no file is needed as everything can be defined in the JSON.

In the image above, you can see the JSON to describe the ape sticker and the emoji_ape image file after the IMGLYAssets bundle has been added to our app. You can edit these assets and add your own.

Once you’re done editing the asset bundle, you can use it instead of the default bundle during the onCreate modifier like this:

VideoEditor(engine)
  .imgly.onCreate({ engine in
     try await engine.scene.load(from: VideoEditor.defaultScene)
              // Add asset sources
     let bundleURL = Bundle.main.url(forResource: "IMGLYAssets", withExtension: "bundle")!
     try await engine.addDefaultAssetSources(baseURL: bundleURL)
     try await engine.addDemoAssetSources(sceneMode: engine.scene.getMode(),
       withUploadAssetSources: true)
     try await engine.asset.addSource(TextAssetSource(engine: engine))
  })

In the code above, we create a blank scene using VideoEditor.defaultScene static method. Then set a URL to point to the bundle in the app instead of a remote bundle and make that the source of addDefaultAssetSources. Now our app will use the assets from the bundle.

Exporting the Finished Video

Once the user has finished with their creation, they tap the share button and the Video Editor composes all of the video, audio, filters, and static items into a single .mp4 file. Then it displays a share sheet. If you’d rather do something different, like save the creation locally so you can play it back, you can override this default behavior.

In your app, after the .onCreate modifier, you can define a .onExport modifier. When the user taps on the share button you can export the video and then save it to the documents directory to playback later.

In the documentation for modifiers you can find the source code for the Video Editor’s default behavior. Instead of duplicating it in detail here, we can just summarize and provide a little snippet for our change.

First, the .onCreate handler takes an engine and a eventHandler parameter. The engine is the underlying IMG.LY Engine object we’ve been using. The eventHandler lets us send information back to the Video Editor UI about the progress or any errors.

In the documentation, there is a helper function to export the video. After it runs it returns the data of the exported video and a mimeType, which will be .mp4.

let data: Date, mimeType: MIMEType
let mode = try mainEngine.scene.getMode()
guard mode == .video else { throw CallbackError.unknownSceneMode; return }
(data, mimeType) = try await exportVideo()

The code in the documentation handles static files from the DesignEditor as well as videos from the VideoEditor. The above code makes it so that only video can be exported, since our app only works with video.

In the exportVideo function, the code first checks to make sure it has a properly formed scene object to work with. Then it starts the export using mainEngine.block.exportVideo(_, mimeType:). This begins an async throwing stream during the export. The VideoExport objects in the stream are either .progress or .finished. When a .progress object arrives, the exportVideo function sends an update to the eventHandler to display in the UI. When the .finished object arrives, it has an associated object that is the data for the video.

Now instead of opening the share sheet, we can write the video to the documents directory for the app.

do {
      guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
        else {
          logger.error("Documents directory not found"); return
        }

  let filePath = documentDirectory.appending(component: UUID().uuidString)
  try data.write(to: filePath)
} catch (let error as NSError) {
  logger.error("could not write finished file: \(error.localizedDescription)")
}

eventHandler.send(.exportCompleted {})

The code above takes the data from the exportVideo function and writes it to the app’s documents directory, giving it a UUID string value as a filename. Then it updates the UI to let it know the export is done but sends an empty action block.

Setting Up a Playback Controller

For this example app, the playback controller will play any clips in the app’s Documents directory. Each clip plays on a loop. The user can swipe up to get the next clip and tap to pause or restart the clip. If there are no clips, the user will be encouraged to make a new one.

When the video player view appears, the first task is to see if there are any videos to load. Any video files in the Documents directory are assumed to be ready to play.

func loadVideos() {
  //get a handle to the documents directory for this app
  guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true) else { logger.error("Documents directory not found"); return}

  if let files = try? FileManager.default.contentsOfDirectory(at: documentDirectory, includingPropertiesForKeys: nil) {
    videos = files
  }
}

The code above reads all of the filenames from the documents directory into a files array and then assigns that array to a videos variable.

Each time the app is launched or the VideoEditor view is dismissed, the video player view will get the onAppear message. This is a good place to check for videos using the loadVideos function. After the videos are loaded, the app can start to play the first one.

func setupVideoPlayer() {
  guard let currentVideo = videos.first else {logger.error("No videos to play"); return}

  self.shouldHideEmptyDirectoryText = true

       let endMonitor = NotificationCenter.default.publisher(for: NSNotification.Name.AVPlayerItemDidPlayToEndTime)

  self.player = AVPlayerItem(url: currentVideo)
  self.currentVideo = currentVideo

}

In the code above, we first check to see if there are any video files. If there are, then hide the message about videos and create a new AVPlayerItem using the URL of the first video. The endMonitor, shouldHideEmptyDirectoryText, and player are both being watched by the View. The AVPlayer gets rendered in a regular SwiftUI View.

VideoPlayer(player: player)
  .frame(width: 640, height: 360)
  .onAppear {
    player.play()
  }
  .onReceive(endMonitor) { _ in
    player.seek(to: .zero)
    player.play()
}

When the video reaches the end, it will seek(to: .zero) which rewinds the video and loops it.

In the TikTok app, you can advance to the next video by swiping up. Additionally, you can start and stop any video by tapping on the screen. We can add both of those features using gesture recognizers.

The code for play and pause is attached to a Tap Gesture modifier we can append to the player.

.onTapGesture {
  if player.rate > 0.0 {
    player.rate = 0.0
  } else {
    player.rate = 1.0
  }
}

We can just adjust the playback rate. A value of 1.0 is a normal speed and 0.0 is paused. The player also has a .pause and .play methods, but sometimes these seem buggy.

Swipe is a little more difficult since SwiftUI only has swiping on List items. So we can use a drag gesture and then evaluate the translations when it’s complete.

.gesture(
    DragGesture(minimumDistance: 3.0, coordinateSpace: .local)
        .onEnded { value in
            // Check for a vertical, upward swipe with minimal horizontal deviation
          if abs(value.translation.width) < 100 &&
             value.translation.height < 0 {
                // It was an up swipe!
		viewModel.advanceVideo()
            }
        }
)

Then advancing the video is a simple matter of refreshing the player

func advanceVideo() {
  guard let currentVideo = self.currentVideo else { logger.error("No current video."); return}

  let currentIndex = videos.firstIndex(of: currentVideo)
  var nextVideo = videos.index(after: currentIndex!)
  if nextVideo == videos.count {
    nextVideo = 0
  }

  self.currentvideo = videos[nextVideo]
  self.player = AVPlayer(url: self.currentVideo!)

}

This code will get the next file URL from the array of videos. When it reaches the end, it will loop back and get the file at index 0. Then it creates a new player. Because the view is observing that player variable, it will update with the new video.

With this view, the user can create new videos by tapping the creation button and swipe to view all of their created videos.

Where to Go From Here

This tutorial has focused on how the CreativeEditor SDK can help you quickly make a video creation app like TikTok. Good next steps would be to further customize the editing tools and build out the network for sharing and tagging videos. Something that is important to consider is the data structure for each video. The iOS system is optimized to read only as much of a video file off of disk as is needed at any moment. Your app should use those optimizations to run faster. So, don’t load the whole video into memory as a Data object. Your data structures should keep the large video files somehow separate from the much smaller metadata (likes, comments, etc.). Consider storing the URL or filename of the video in the same object as the likes and comments. This will also allow you to cache video files after they have been downloaded so that you don’t need to redownload them when other data such as comments or number of likes changes.

Thanks for reading! We hope that you’ve gotten a better understanding for how a tool like the CreativeEditor SDK can bring your ideas to market faster. Feel free to reach out to us with any questions, comments, or suggestions.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

To stay in the loop, subscribe to our Newsletter or follow us on LinkedIn and Twitter.

How to Add Stickers and Overlays to a Video in Flutter

Natalia — Fri, 24 Mar 2023 09:32:33 GMT

When working with interactive content, it may happen that you will need to add remote or local assets on top of your video file. The assets varying from images and stickers to videos and fonts are usually grouped and deployed within your application and are available at runtime. This tutorial will teach you how to apply stickers and overlays to a video in Flutter and make your video content more personalizable and user-oriented. We start by considering the prerequisites for successful video integration, then talk about how Flutter manages overlays in order to personalize our video file.

How does Flutter handle video assets?

Unlike images, Flutter displays a video file by employing a special video_player plugin that provides playback, stores a file, and manages speed and sound control. For iOS-based applications, the video is integrated via AVPlayer, whereas the Android system employs ExoPlayer. Crucially, to create a simple video player, one should follow the steps below:

First, to ensure that pubspec.yaml file contains the video_player dependency;
Then permit access to videos for the application;
Make a VideoPlayerController and set it up;
Finally, don’t forget to display and play the video player;

So, let’s closely look at the setup process.

Prerequisites:

We start by adding the following dependencies of video_player plugin to pubspec.yaml file.

dependencies:
  flutter:
    sdk: flutter
  video_player: ^2.4.7

Afterwards, change configurations of both android and ios systems to guarantee that an application has the proper authorization for video streaming: e.g. for Android-oriented programs, proceed to the <project root>/android/app/src/main/AndroidManifest.xml directory – where it’s necessary to add the next line to the AndroidManifest.xml.

<manifest xmlns:android="<http://schemas.android.com/apk/res/android>">
    <application ...>

    </application>

    <uses-permission android:name="android.permission.INTERNET"/>
</manifest>

For iOS-based apps the Info.plist file is stored at <project root>/ios/Runner/Info.plist. Once again, don’t forget to add the following specification to the original code:

<key>NSAppTransportSecurity</key>
<dict>
  <key>NSAllowsArbitraryLoads</key>
  <true/>
</dict>

Successful video integration in Flutter is managed through the VideoPlayer widget and a VideoPlayerController, which sets the connection to the asset and initialize the controller for playback.

/// Stateful widget to fetch and then display video content.
class VideoApp extends StatefulWidget {
  const VideoApp({Key? key}) : super(key: key);

  @override
  _VideoAppState createState() => _VideoAppState();
}

class _VideoAppState extends State<VideoApp> {
  late VideoPlayerController _controller;

  @override
  void initState() {
    super.initState();
    _controller = VideoPlayerController.asset('assets/bee.mp4')

      /// Specify the path to your video asset here
      ..initialize().then((_) {
        /// Ensure the first frame is shown after the video is initialized,
        /// even before the play button has been pressed.
        setState(() {});
      });
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }
}

Note that video tends to take up as much space on the screen as possible by default, which can significantly deteriorate the video’s quality. Therefore, the Flutter team suggests employing the AspectRatiowidget to adjust the video proportions.

? AspectRatio(
    aspectRatio: _controller.value.aspectRatio)

Adding Overlay Stickers and Text to a Video in Flutter

To see the complete code of our demo application, you can clone the GitHub repository or put the following command to your Terminal:

git clone git@github.com:nataliakzm/Adding_Stickers_and_Overlays_to_video_in_Flutter.git

Similarly to how we handled resizing images in Flutter, the same method is used when overlaying an object on a video. The above example consists basically of three containers, one with a sticker, another with the textual watermark and the last with a local video asset. However, the critical difference here is that a Container filled with a sticker and a Container filled with a textual watermark cannot be simply listed one by one in your code but rather must be both placed within a Stack widget.

The Stack container is designed as a “mother”-widget which retains multiple layers of widgets on the screen. Stack structures these branches of children into a hierarchical system from bottom to top. Thus, the uppermost widget goes to the background, and the bottommost item is displayed in the foreground.

Stack(
  children: <Widget>[
    BottomWidget(),
    MiddleWidget(),
    TopWidget(),
  ]),

Since the size of the Stack widget is aimed to be the largest size among layers, we will place our video asset on this layer. Then, to implement the overlays, it’s essential to position and/or align each child’s container.


child: Stack(children: [
  VideoPlayer(_controller),

  /// Don't forget to align the position of the Сontainer
  Positioned( bottom: 5, left: 10,
      child: Container( width: 80, height: 40,

        /// In case you want to check the position/size of the Container uncomment the next line
        //color: Color(0xff0360da),
        child: Align(
            alignment: Alignment.center,

            /// Integrate Sticker overlay into a video
            child: Image.asset('assets/sticker.png')))),

  Positioned( top: 5, right: 10,
      child: Container( width: 80, height: 40,
        child: Align(
            alignment: Alignment.center,

            /// Integrate Text overlay into a video
            child: Text("IMG.LY",
                style: TextStyle(
                    color: Colors.white,
                    fontSize: 20,
                    fontWeight: FontWeight.bold))))),
                        ]))

  /// If there is no video, a blank page will be return   : Container()),

Summing up

Thus, you can see from the code below that integrating multiple layers over interactive assets can be a tricky task which requires bearing in mind the position and order of each layer, as well as managing how the children are aligned to a video file and the screen size of the user. Conversely, the discussed method allows personalizing an application without significant modifications to the original file.

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';

/// Import video_player package to integrate a video asset
import 'package:video_player/video_player.dart';

void main() => runApp(const VideoApp());

/// Stateful widget to fetch and then display video content.
class VideoApp extends StatefulWidget {
  const VideoApp({Key? key}) : super(key: key);

  @override
  _VideoAppState createState() => _VideoAppState();
}

class _VideoAppState extends State<VideoApp> {
  late VideoPlayerController _controller;

  @override
  void initState() {
    super.initState();
    _controller = VideoPlayerController.asset('assets/bee.mp4')

      /// Specify the path to your video asset here
      ..initialize().then((_) {
        /// Ensure the first frame is shown after the video is initialized,
        /// even before the play button has been pressed.
        setState(() {});
      });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
        debugShowCheckedModeBanner: false,
        home: Scaffold(
            appBar: AppBar(
                title: Text('How to add Stickers and Overlays'),
                backgroundColor: Color(0xff0360da)),
            body: Center(
                child: _controller.value.isInitialized

                    /// First, we specify the AspectRatio of a video frame
                    ? AspectRatio(
                        aspectRatio: _controller.value.aspectRatio,

                        /// Then, we initialize a Stack widget
                        child: Stack(children: [
                          VideoPlayer(_controller),

                          /// Don't forget to align the position of the Сontainer
                          Positioned( bottom: 5, left: 10,
                              child: Container( width: 80, height: 40,

                                  /// Uncomment the following line to check the position/size of the Container
                                  //color: Color(0xff0360da),
                                  child: Align(
                                      alignment: Alignment.center,

                                      /// Integrate Image overlay into a video
                                      child: Image.asset('assets/sticker.png')))),

                          Positioned( top: 5, right: 10,
                              child: Container( width: 80, height: 40,
                                  child: Align(
                                      alignment: Alignment.center,

																			/// Integrate some Text overlay into a video
                                      child: Text("IMG.LY",
                                          style: TextStyle(
                                              color: Colors.white,
                                              fontSize: 20,
                                              fontWeight: FontWeight.bold))))),
                        ]))

                    /// If there is no video, a blank page will be return
                    : Container()),
            floatingActionButton: FloatingActionButton.extended(
                backgroundColor: const Color(0xff0360da),
                foregroundColor: Colors.white,
                onPressed: () {
                  setState(() {
                    _controller.value.isPlaying
                        ? _controller.pause()
                        : _controller.play();
                  });
                },
                icon: Icon(_controller.value.isPlaying
                    ? Icons.pause
                    : Icons.play_arrow),
                label: Text('Play for IMG.LY'))));
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }
}

Adding Overlays to a Video with Flutter package for VideoEditor SDK

However, if you would like to offer more advanced sticker and overlay functionality to your user and you might need a more complex UI structure. Since this can be an extremely time consuming undertaking you might want to opt for a ready-to-use solution such as VideoEditor SDK which among a host of other essential video editing features allows users to easily add stickers and different overlays to their videos.

To replicate the example above, go to the official documentation to discover how to get started with VideoEditor SDK or follow our guide on how to integrate video editor for Flutter into your app. Try to upload your video file and modify it with various stickers, text and other overlays. Alternatively, you can download our free mobile demo app from the App Store or Google Play, and test a comprehensive set video editing tools.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

How To Build a Video Editor With Wasm in React

Antonello — Mon, 16 Jan 2023 16:31:57 GMT

You may think that video processing can only be performed on a server, but this is not true. Thanks to WebAssembly (Wasm), you can run high-performance code written in C or C++ in your browser. This opens up a lot of possibilities and gives you the ability to build a client-side video editor. Let’s now learn how to create a Wasm-based video editor in React.

Follow this tutorial and learn how to build the application below:

This React video editor allows users to upload a video, select a portion of it, convert it to GIF, and download the resulting image file – all of this in your browser.

What is WebAssembly?

WebAssembly (also called Wasm) is a new type of code that modern web browsers can run and understand. Specifically, Wasm provides new functionality to web development and brings significant performance benefits. This applies to both frontend and backend, since Wasm can be used by both clients and servers.

Wasm code should not be written by hand. This is because Wasm is a binary instruction format. So, Wasm is designed to be a compilation target for source languages such as C, C++, Rust, and others. Learn more about Wasm on the WebAssembly MDN Web Docs page.

Wasm provides a way to run code written in several languages on the Web at near-native speed. This means that WebAssembly forms can be easily imported and executed by a Web client or server application. In other words, you can use WebAssembly functions via JavaScript and achieve results that were not previously possible on a browser, especially in terms of performance.

What is `ffmpeg.wasm`?

As stated on the GitHub page of the project, ffmpeg.wasm is a WebAssembly and JavaScript port of FFmpeg. If you are not familiar with it, FFmpeg is an open-source software suite that includes several libraries and programs for managing audio, video, streams, and other media files. All of these tools can be invoked and executed via the ffmpeg command-line instruction.

Therefore, ffmpeg.wasm enables video and audio processing in JavaScript-based web applications. In detail, it enables video and audio recording, format transcoding, video and audio editing, and video scaling. Even though it is a WebAssembly-based library, you can use it in your JavaScript code just like any other npm library. This is the power of Wasm.

Since it transpiles to Wasm, you can take advantage of ffmpeg.wasm directly in your browser without performance concerns. This means that ffmpeg.wasm opens the door to client-side audio and video processing. Let’s now learn how to use ffmpeg.wasm to build a client-side video editor in React!

Building a Video Editor in React with WASM

In this step-by-step tutorial, you will learn how to build a Wasm-based video editor in React with ffmpeg.wasm. This React video editor application will allow you to upload a video, trim it, and convert it to GIF in the browser, without using any backend functionality or external API.

Clone the GitHub repository that supports the tutorial and try the video editor application by launching the following commands:

git clone https://github.com/Tonel/video-editor-wasm-react
cd video-editor-wasm-react
npm install
npm run start

Let’s waste no more time and see how to build a video editor application in React.

Prerequisites

This is the list of libraries the video editor application you are about to build depends on:

Node.js and npm 8+ and higher
React 18.2+
[ffmpeg.wasm](https://github.com/ffmpegwasm/ffmpeg.wasm)
[antd](https://www.npmjs.com/package/antd) >= 4.3
[video-react](https://www.npmjs.com/package/video-react) >= 0.15

antd is one of the most popular UI libraries for React. In this tutorial, its [Slider](https://ant.design/components/slider/) component will be used to implement the video cutting feature. So, any other UI library including a slider component with a range option will do.

video-react is one of the most advanced and reliable HTML5 video players for React. You can replace it with any other React video player library.

Initializing a React Project

Let’s initialize a new Create React App React project called video-editor-wasm-react with the command below:

npx create-react-app video-editor-wasm-react

Your video-editor-wasm-react directory should now contain the following files:

video-editor-wasm-react
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Enter the video-editor-wasm-react folder in your terminal and launch a React local server with the commands below:

cd video-editor-wasm-react
npm start

Now, visit the [http://localhost:3000/](http://localhost:3000/) page in your browser, and you should be seeing the default Create React App screen.

Before installing ffmpeg.wasm, you need to get your server ready. As stated in the official documentation, fmmpeg.wasm depends on [SharedArrayBuffer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer). This means that only browsers that support SharedArrayBuffer can run ffmpeg.wasm. You can find a complete list of all browsers here.

In the most popular browsers, SharedArrayBuffer is only available to cross-origin isolated webpages. To enable this, you need to set the following headers in your server hosting the React application:

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

In a Create-React-App application, you achieve this by creating a src/setupProxy.js file and making sure it contains the following code:

module.exports = function (app) {
    app.use(function (req, res, next) {
        res.setHeader("Cross-Origin-Opener-Policy", "same-origin")
        res.setHeader("Cross-Origin-Embedder-Policy", "require-corp")
        next()
    })
}

Your local development server is now ready to use ffmpeg.wasm!

Installing the Project’s Dependencies

It is now time to add the aforementioned required dependencies to your project.

First, install ffmpeg.wasm with the following command:

npm install @ffmpeg/ffmpeg @ffmpeg/core

Now, let’s install Antd Design:

npm install antd

Finally, it is time to install Video-React:

npm install video-react redux

Note that video-react also requires [redux](https://www.npmjs.com/package/redux) to work.

You have now everything you need to start developing your React video editor!

Uploading a Video to the Video Editor

Create a VideoUpload component as below:

// src/components/VideoUpload.js

import { Button, Upload } from 'antd';

function VideoUpload({ disabled, onChange = () => {}, onRemove = () => {} }) {
  return (
    <>
      <Upload
        disabled={disabled}
        beforeUpload={() => {
          return false;
        }}
        accept="video/*"
        onChange={(info) => {
          if (info.fileList && info.fileList.length > 0) {
            onChange(info.fileList[0].originFileObj);
          }
        }}
        showUploadList={false}
      >
        <Button>Upload Video</Button>
      </Upload>
      <Button
        danger={true}
        disabled={!disabled}
        onClick={() => {
          onRemove(undefined);
        }}
      >
        Remove
      </Button>
    </>
  );
}

export default VideoUpload;

As you can see, this component uses the Antd Design Upload component to allow users to upload a video file. Then, a “Remove” button gives users the ability to remove the uploaded video. Note that when the Upload component is enabled, the “Remove” button is disabled and vice versa. This way, users can deal with only one video at a time.

Playing a Video in React

You now need a VideoPlayer component to play the uploaded video. You can build it as below:

// src/components/VideoPlayer.js

import {
  BigPlayButton,
  ControlBar,
  LoadingSpinner,
  Player,
  PlayToggle,
} from 'video-react';
import 'video-react/dist/video-react.css';
import { useEffect, useState } from 'react';

export function VideoPlayer({
  src,
  onPlayerChange = () => {},
  onChange = () => {},
  startTime = undefined,
}) {
  const [player, setPlayer] = useState(undefined);
  const [playerState, setPlayerState] = useState(undefined);

  useEffect(() => {
    if (playerState) {
      onChange(playerState);
    }
  }, [playerState]);

  useEffect(() => {
    onPlayerChange(player);

    if (player) {
      player.subscribeToStateChange(setPlayerState);
    }
  }, [player]);

  return (
    <div className={'video-player'}>
      <Player
        ref={(player) => {
          setPlayer(player);
        }}
        startTime={startTime}
      >
        <source src={src} />
        <BigPlayButton position="center" />
        <LoadingSpinner />
        <ControlBar autoHide={false} disableDefaultControls={true}>
          <PlayToggle />
        </ControlBar>
      </Player>
    </div>
  );
}

This is a simple video-react-based video player component, which only allows playing or stop the video file received with the src prop.

Cutting a Video in React

Your goal is to allow users to select a portion of the video and watch it before converting it into GIF. Implement this with the logic below:

import { Slider } from "antd"
import { sliderValueToVideoTime } from "../utils/utils"

function VideoEditor() {
  const [sliderValues, setSliderValues] = useState([0, 100])

  // logic to handle videoFile, videoPlayer, and videoPlayerState...

  useEffect(() => {
      const min = sliderValues[0]
      // when the slider values are updated, updating the
      // video time
      if (min !== undefined && videoPlayerState && videoPlayer) {
          videoPlayer.seek(sliderValueToVideoTime(videoPlayerState.duration, min))
      }
  }, [sliderValues])

  useEffect(() => {
      if (videoPlayer && videoPlayerState) {
          // allowing users to watch only the portion of
          // the video selected by the slider
          const [min, max] = sliderValues

          const minTime = sliderValueToVideoTime(videoPlayerState.duration, min)
          const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max)

          if (videoPlayerState.currentTime < minTime) {
              videoPlayer.seek(minTime)
          }
          if (videoPlayerState.currentTime > maxTime) {
              // looping logic
              videoPlayer.seek(minTime)
          }
      }
  }, [videoPlayerState])

  return (
          // other video editor components...
          <Slider
            disabled={!videoPlayerState}
            value={sliderValues}
            range={true}
            onChange={(values) => {
              setSliderValues(values)
            }}
            tooltip={{
              formatter: null,
            }}
          />
        // ...
}

This component equips users with the ability to select a portion of the uploaded video with the Antd Design Slider component and watch it in loop, before converting it into a GIF file.

Specifically, the logic here uses the sliderValueToVideoTime() function below to transform the slider values into time coordinates:

// src/utils/utils.js

export function sliderValueToVideoTime(duration, sliderValue) {
    return Math.round(duration * sliderValue / 100)
}

Note that the actual video processing slider will be performed by the next component through ffmpeg.

Converting a Video to GIF in React

To convert a video to GIF, create a VideoConversionButton component as follows:

// src/components/VideoConversionButton.js

import { Button } from 'antd';
import { fetchFile } from '@ffmpeg/ffmpeg';
import { sliderValueToVideoTime } from '../utils/utils';

function VideoConversionButton({
  videoPlayerState,
  sliderValues,
  videoFile,
  ffmpeg,
  onConversionStart = () => {},
  onConversionEnd = () => {},
  onGifCreated = () => {},
}) {
  const convertToGif = async () => {
    // starting the conversion process
    onConversionStart(true);

    const inputFileName = 'gif.mp4';
    const outputFileName = 'output.gif';

    // writing the video file to memory
    ffmpeg.FS('writeFile', inputFileName, await fetchFile(videoFile));

    const [min, max] = sliderValues;
    const minTime = sliderValueToVideoTime(videoPlayerState.duration, min);
    const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max);

    // cutting the video and converting it to GIF with an FFMpeg command
    await ffmpeg.run(
      '-i',
      inputFileName,
      '-ss',
      `${minTime}`,
      '-to',
      `${maxTime}`,
      '-f',
      'gif',
      outputFileName
    );

    // reading the resulting file
    const data = ffmpeg.FS('readFile', outputFileName);

    // converting the GIF file created by FFmpeg to a valid image URL
    const gifUrl = URL.createObjectURL(
      new Blob([data.buffer], { type: 'image/gif' })
    );
    onGifCreated(gifUrl);

    // ending the conversion process
    onConversionEnd(false);
  };

  return <Button onClick={() => convertToGif()}>Convert to GIF</Button>;
}

export default VideoConversionButton;

This component uses ffmpeg.wasm to launch an ffmpeg command. This command takes care of clipping the video to the time limits defined by sliderValues and converting it to GIF. Note that all these operations are performed client-side in the browser.

In detail, the ffmpeg -ss flag defines the time from which to start reading the video file, while -to defines the end time. Then, the gif option specifies that you want to produce a GIF file. Learn more about the flags and options offered by ffmpeg here.

The resulting GIF image is stored in the output.gif memory file, loaded as a valid "image/gif" Blob, and finally converted to a valid image URL with the [URL.createObjectURL()](https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL_static) native function.

Putting It All Together

Now, let’s see the full code of the VideoEditor component:

// src/components/VideoEditor.js

import { createFFmpeg } from '@ffmpeg/ffmpeg';
import { useEffect, useState } from 'react';
import { Slider, Spin } from 'antd';
import { VideoPlayer } from './VideoPlayer';
import { sliderValueToVideoTime } from '../utils/utils';
import VideoUpload from './VideoUpload';
import VideoConversionButton from './VideoConversionButton';
const ffmpeg = createFFmpeg({ log: true });

function VideoEditor() {
  const [ffmpegLoaded, setFFmpegLoaded] = useState(false);
  const [videoFile, setVideoFile] = useState();
  const [videoPlayerState, setVideoPlayerState] = useState();
  const [videoPlayer, setVideoPlayer] = useState();
  const [gifUrl, setGifUrl] = useState();
  const [sliderValues, setSliderValues] = useState([0, 100]);
  const [processing, setProcessing] = useState(false);

  useEffect(() => {
    // loading ffmpeg on startup
    ffmpeg.load().then(() => {
      setFFmpegLoaded(true);
    });
  }, []);

  useEffect(() => {
    const min = sliderValues[0];
    // when the slider values are updated, updating the
    // video time
    if (min !== undefined && videoPlayerState && videoPlayer) {
      videoPlayer.seek(sliderValueToVideoTime(videoPlayerState.duration, min));
    }
  }, [sliderValues]);

  useEffect(() => {
    if (videoPlayer && videoPlayerState) {
      // allowing users to watch only the portion of
      // the video selected by the slider
      const [min, max] = sliderValues;

      const minTime = sliderValueToVideoTime(videoPlayerState.duration, min);
      const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max);

      if (videoPlayerState.currentTime < minTime) {
        videoPlayer.seek(minTime);
      }
      if (videoPlayerState.currentTime > maxTime) {
        // looping logic
        videoPlayer.seek(minTime);
      }
    }
  }, [videoPlayerState]);

  useEffect(() => {
    // when the current videoFile is removed,
    // restoring the default state
    if (!videoFile) {
      setVideoPlayerState(undefined);
      setSliderValues([0, 100]);
      setVideoPlayerState(undefined);
      setGifUrl(undefined);
    }
  }, [videoFile]);

  return (
    <div>
      <Spin
        spinning={processing || !ffmpegLoaded}
        tip={!ffmpegLoaded ? 'Waiting for FFmpeg to load...' : 'Processing...'}
      >
        <div>
          {videoFile ? (
            <VideoPlayer
              src={URL.createObjectURL(videoFile)}
              onPlayerChange={(videoPlayer) => {
                setVideoPlayer(videoPlayer);
              }}
              onChange={(videoPlayerState) => {
                setVideoPlayerState(videoPlayerState);
              }}
            />
          ) : (
            <h1>Upload a video</h1>
          )}
        </div>
        <div className={'upload-div'}>
          <VideoUpload
            disabled={!!videoFile}
            onChange={(videoFile) => {
              setVideoFile(videoFile);
            }}
          />
        </div>
        <div className={'slider-div'}>
          <h3>Cut Video</h3>
          <Slider
            disabled={!videoPlayerState}
            value={sliderValues}
            range={true}
            onChange={(values) => {
              setSliderValues(values);
            }}
            tooltip={{
              formatter: null,
            }}
          />
        </div>
        <div className={'conversion-div'}>
          <VideoConversionButton
            onConversionStart={() => {
              setProcessing(true);
            }}
            onConversionEnd={() => {
              setProcessing(false);
            }}
            ffmpeg={ffmpeg}
            videoPlayerState={videoPlayerState}
            sliderValues={sliderValues}
            videoFile={videoFile}
            onGifCreated={(girUrl) => {
              setGifUrl(girUrl);
            }}
          />
        </div>
        {gifUrl && (
          <div className={'gif-div'}>
            <h3>Resulting GIF</h3>
            <img
              src={gifUrl}
              className={'gif'}
              alt={'GIF file generated in the client side'}
            />
            <a
              href={gifUrl}
              download={'test.gif'}
              className={'ant-btn ant-btn-default'}
            >
              Download
            </a>
          </div>
        )}
      </Spin>
    </div>
  );
}

export default VideoEditor;

Since ffmpeg takes time to load, you need to let users know that they need to wait a while before they can use the video editor component. Also, ffmpeg operations take time to execute. This wait logic is implemented by using the Antd Design [Spin](https://ant.design/components/spin/) component.

Also, note that by passing the log: true option to the createFFmpeg() function, ffmpeg will log useful debug info in the console. For example, after ffmpeg gets loaded, you should be able to see the following lines in the browser console:

[info] use ffmpeg.wasm v0.11.5
createFFmpeg.js:43 [info] load ffmpeg-core
createFFmpeg.js:43 [info] loading ffmpeg-core
createFFmpeg.js:43 [info] ffmpeg-core loaded

What the VideoComponent does is use all the components presented earlier to allow users to upload a video, select a portion through a slider, convert it to GIF, and download the resulting GIF file.

Et voilà! You just learned how to build a client-side video editor in React without using any backend functionality!

Commercial Alternative

VideoEditor SDK by IMG.LY provides powerful video editing features, including cropping and trimming videos in your project. You will receive staples of video editing, including straightening videos, filters, brightness, color adjustments, and more.

Conclusion

Building a client-side video processing and editing application that uses only the browser possible. With WebAssembly, you can run high-performance code written in C or C++ in JavaScript. This allows you to perform complex operations at lightning speed directly in the browser, without having to delegate the work to a server.

In this tutorial, you learned how to use ffmpeg.wasm to build a video editor in React. ffmpeg.wasm is the Wasm port of ffmpeg and enables video and audio processing and editing directly in the browser. Using it to build a web application to upload a video, cut it, and export it to GIF requires only a few lines of code, and here we have seen how to implement such a video editing application.

If your app goes beyond merely displaying video, and you want to allow your users to also edit video or create video based templates in the browser, explore our VideoEditor SDK for the Web – a performant video editing solution based on WASM and WebCodecs. Try the editor for yourself in action with our showcases.

Thanks for reading! Let us know what you think on Twitter! To stay in the loop, subscribe to our Newsletter.

FFmpeg - The Ultimate Guide

Csaba — Mon, 21 Nov 2022 12:16:19 GMT

In this guide, we’ll go through the hot topics of FFmpeg. But before that, we’ll cover some base ground to help you understand basic media concepts and FFmpeg. Feel free to skip the parts that are already trivial for you!

Introduction to FFmpeg

FFmpeg.org’s definition is the following: “FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge. No matter if they were designed by some standards committee, the community or a corporation.”

I think of FFmpeg as the go-to application for audio/video manipulation in an automated or scripted manner.

When you need to implement a service that manipulates video, or just have 300 media files that need to be converted into a different format, FFmpeg is your - nerdy - friend.

FFmpeg can do large chunks of the basic functionalities of a modern Non-linear (NLE) video editors, e.g., Davinci Resolve Studio or Premiere Pro. But, it does not have a graphical interface in that sense as those behemoths do, and unarguably it is way less friendly.

In a general NLE, you might do things like these:

Click to import a file
Drop it into the timeline
Trim and Cut
Add an overlay image
Crop that overlay
Add vignette
Add some color changing effects, e.g. change the hue
Add an extra audio track to the mix
Change the volume
Add some effects, e.g.: echo
Export into various formats
Export into a deployable video format
Export the master audio in wav

Learn how to crop and trim videos in Flutter. Or, to achieve the exact same thing, you could also execute this command:

ffmpeg -y  \
    -ss 20 -t 60 -i bbb_sunflower_1080p_60fps_normal.mp4 \
    -i train.jpg \
    -ss 4 -i voice_recording.wav \
    -filter_complex "[0:v]hue=h=80:s=1[main] ; [1:v]crop=w=382:h=304:x=289:y=227[train] ; [main][train]overlay=x=200:y=200,vignette=PI/4[video] ; [2:a]volume=1.5,aecho=0.8:0.9:100:0.3[speech] ; [0:a][speech]amix=duration=shortest,asplit[audio1][audio2]" \
    -map '[video]' -map '[audio1]' -metadata title="Editor's cut" bbb_edited.mp4 \
    -map '[audio2]' bbb_edited_audio_only.wav

Yes, it isn’t friendly at all, but it is very, very powerful once you become friends with FFmpeg.

Check out this comparison of the original and the edited one:

If you want to try this command out, get the example files and see it for yourself!

Installing FFmpeg

FFmpeg is available for most common and even uncommon platforms and architectures. You can be on Linux, Mac OS X or Microsoft Windows, and you’ll be able to run or link to FFmpeg.

Installing FFmpeg is easy on most platforms! There is no installer, usually just a compressed archive you need to get for your platform and architecture.

In the case of Linux, most distributions include a pre-built FFmpeg in their software repositories. Therefore, you can install FFmpeg from those even more quickly.

FFmpeg history

The project was started in 2000 by the awesome Fabrice Bellard. The name is a concatenation of “FF” meaning “fast-forward” and MPEG, the name of a video standards group. It has been very well, active and alive since then, releasing a new release about every three months.

FFmpeg supported codecs and formats

The default FFmpeg shipped with my Ubuntu Linux distribution supports about 460 codecs and 370 formats.

See it for yourself:

ffmpeg -codecs
ffmpeg -formats

Compilation of FFmpeg

Keep in mind that the supported codecs and formats (and filters, demuxers, muxers, input and output methods, etc.) are highly dependent on the so-called compilation flags.

This means that the above number only represents the fact that it supports at least this many codecs and formats. Still, there are even more that the package builders excluded for various reasons, e.g.: licensing, architecture, size considerations, etc.

Since FFmpeg is open source, you can compile FFmpeg for yourself at any time.

Suppose for example, that you care about your layer’s size (therefore the bootstrap speed) in AWS Lambda. In this case, you can compile an FFmpeg binary that only contains the mp3 encoder for example, and nothing else. For a full tutorial on running FFmpeg on AWS Spot Instances, see our cloud guide. Prefer Google Cloud? Our guide on running FFmpeg on Google Cloud Platform shows you how.

Also, you might not want to run into licensing issues and leave out stuff that would cause problems for your use case. Therefore you choose to leave out particular codecs/formats. I highly recommend checking out the “—enable-gpl”, “—enable-nonfree” and “—enable-version3” compilation flags in this case, as well as this.

Or you might want to have a standalone FFmpeg binary in your project (e.g.: embedded, or some cloud instance), that does not depend on any operating system libraries. Then you want to make a so-called static build, that compiles in all the libraries into a single binary file, and does not depend on your OS’ libraries and the runtime loading of other FFmpeg libraries. Search around for “—enable-static” in this case.

Finally, you can find pre-built static FFmpeg builds right here too. Alternatively, you can package FFmpeg in a Docker container for consistent environments - our Docker guide covers this approach.

FFmpeg’s strengths

FFmpeg reads and writes most video and audio formats that matter for most of us. It is a very capable and high-performance tool for converting and manipulating these formats.

But FFmpeg can do even more! For examples of these operations integrated into an automated pipeline, read our article on a batch video processing server.

Filtering

FFmpeg has vast amounts of filters for audio and video. Therefore, video manipulation is also a key feature of FFmpeg.

Hardware acceleration

It does support many kinds of hardware accelerations! Video encoding is a very resource-intensive operation, and you might come across quite a few hardware devices or features that might speed up your process!

Most notably, if you have an NVIDIA card, you can increase your H.264 or H.265 encoding and decoding throughput by multipliers compared to your CPU. But other things, such as VDPAU, VAAPI, or OpenCL, can be leveraged to boost your pipeline’s throughput.

Learn more about the supported hardware acceleration methods here.

Versatile input/output methods

FFmpeg is also very capable when it comes to accessing input and output data.

Just to name a few: it can use your webcam, record from your microphone, grab your screen, or capture from your Blackmagic DeckLink. But FFmpeg can download directly from a web address, open all kinds of streams, read from a pipe, a socket, and of course, from files.

The same holds true for outputting the data. It can write to your webcam, play audio on your microphone… Just kidding:) It can output to files, streams, pipes, sockets and so on.

Running example commands

This article is full of FFmpeg commands that are working examples. The reason for that is that you could test these out for yourself! But the command line interfaces of different operating systems are slightly different, so the commands in this article are meant to be executed in a Linux bash shell.

To adopt these command lines to Microsoft Windows, you might need to:

Change (cd) into the directory where you extracted the ffmpeg.exe. Alternatively, add that directory to the path to make it callable from anywhere.
You might need to replace “ffmpeg” to “ffmpeg.exe”
You will need to replace ”\“-s (backslashes) at the end of the lines with ”^“-s (hats)
You’ll need to replace the fontfile argument’s value to something like this: fontfile=/Windows/Fonts/arial.ttf to get commands with the drawtext filter working.

MacOS users will need steps #1 and #4.

Introduction to media concepts

Now let’s have a quick overview of media concepts. These concepts will be vital for us if we want to understand the latter sections of this article and FFmpeg’s workings. To keep this section brief, it is a higher-level, simplified explanation of these concepts.

Audio

We’ll briefly cover the following terms:

Sampling rate
Bitrate
Channels

Sampling Rate

The sampling rate is the factor that shows how many times we measure/scan/sample the input data stream.

The image below shows the measurement windows (quantization) as gray bars.

Why does this matter? Because it is a balancing act. If we measure the signal less often, we’ll lose more details (bad). Also, by having fewer samples, we’ll have less data in the end. Therefore the file size will be smaller (good).

Here are some ballpark values:

8 kHz (GSM - Low quality)
44.1 kHz (CD - High quality)
48 kHz (Very high quality)
88.2 kHz (Insane - usually for production only)
96 kHz (Insane - usually for production only)

There are no definite “right answers” here. The question is what is “good enough” for your use case? GSM focuses on speech, and not even quality but understandability and the least possible amount of data. Therefore, they found that 8 kHz is enough (there are quite a few more tricks), for their purposes.

The “CD quality” aimed for high quality. Therefore they chose 44.1 kHz, that number has some history in it, but the main reason for aiming above 40 kHz lies in physics and how the human ear works.

There were two very smart guys whose theorem basically says that if you want a quite good signal representation, you have to sample it at twice the speed as its original frequency. Human hearing generally works up until about 20 kHz, so if you want “good quality”, you should aim for at least 40 kHz. And 40 kHz + some headroom + some more physics + historical reasons = 44.1 kHz! :)

As for the higher rates, those are only used when very high-quality audio editing is needed.

Bitrate

Bitrate represents the amount of data per second that results from our transcoding/quantization process. If it is 1411 kbit/s, that means that for every second of audio data, about 1411 kbit of output data will be produced.

Therefore, you can say that 1 minute of audio with 1411 kbit/sec will require:

(1411 kbit / 8) kbyte * 60 second = 10582 kbyte = 10.33 mbyte

Now, it is only easy like that with raw audio data and with a few simple codecs, e.g. PCM in WAVs.

Codecs compressing hard might throw your numbers around a little, as input data might be compressible with different rates. Variable bitrate is usually happening to save space. The encoder might output a lower bitrate if the data is “simple” and does not require high precision.

Here are some ballpark values:

13 kbits/s (GSM quality)
320 kbit/s (High-quality MP3)
1411 kbit/s (16bit WAV, CD quality, PCM)

Channels

Inside of most audio formats, you can have more audio channels. This means multiple, separated audio streams can be in the same file.

Many times, multiple channels have their own name:

If you have a single microphone, you will most probably record it into a single channel called Mono.
General music from the FM radio or streaming services usually has two channels in a so-called “Stereo” configuration.

With stereo, there could be several methods how the audio “image” can be made richer by leveraging audio panning, time and phase-shifting and much more. There is a special recording technique too, called Binaural recording, which is super awesome. Wear headphones for this, and don’t be scared:)

For example, here are Big Buck Bunny’s audio waveforms in Audacity:

You can see that there are two lines of waveforms and also that they are pretty similar. That is normal, as you usually hear the same thing with your two ears, but the matter is in the subtle differences between the two. That’s where directionality, richness, and all kinds of other effects lie.

But why stop at two? The list continues:

2.1, as it is often called, means three channels: 2 for stereo and one for the LFE (“low-frequency effects” a.k.a.: “bass”).
5.1 is similar, with five directional channels (2 front, 1 center, 2 rear) and the LFE.

So channels are just separate “recordings” or “streams” of audio signals.

Image properties

For images, there are quite a few parameters, but we’ll check out only these:

Resolution
Bit-depth
Transparency

Resolution

An image consists of pixels, single points that have a single color. The resolution of an image determines how many columns and rows of pixels are in an image. In other words: an image has a width and a height.

This image shows the first 10 pixels in the first row.

Here are some ballpark values for resolution:

“HD” or “Full HD” or “1K” or “1080p” means 1920x1080 pixels.
“4K” could mean a few values, but it should be about 3840x2160 pixels.
A regular 16mp photo you make of your cat is about 4608x3456 pixels.
General social media image posts are about 1080x1080 pixels.

Bit-depth

Bit-depth represents the number of bits used for storing a single pixel’s color value. This is the same balancing game, and you need to decide between quality or file size.

General ballpark values for bit-depth:

Bits	Colors	Notes
1	2	Black & White
8	256	B/W or Limited color palette
24	16.7m	3x8 bit for R-G-B “True color”
30	1073m	3x10 bit for R-G-B “Deep color”

These last two sometimes are referred to as “8 bit” or “10 bit” respectively, especially when talking about videos. That means 8/10 bits per single color channel.

Transparency

Some image formats support an additional channel together with the red, green, and blue components: the alpha channel. The alpha channel determines how transparent a single pixel is, and it can have different bit-depths, it is usually either 1, 8 or 16 bits.

If the alpha channel is 1 bit, then the format can encode a pixel to be either transparent or non-transparent. If it is 8 or more bits, then the format can encode 256 or more steps of transparency.

Video properties

Video data is built by single images shown right after each other. This brings in most attributes of images and a few more!

So a video has a resolution that is its width and height.

Then the first obvious parameter of a video is the framerate, which defines how many images are shown in a second. Common values for this are 24, 25, 30, or 60.

A video file also has a codec assigned to it, which is the format describing how all those images were compressed into this video file. There are many more attributes of videos, but this is a good start.

Video codecs

Compression is a super important thing when it comes to video because you have thousands of images to keep together. If you aren’t doing it in a smart way, then the resulting video will be very, very large.

Just imagine a 2-minute video, with 30 fps. That means it will have 60 s * 2 * 30 fps = 3600 frames! I have just taken a screenshot of an HD video, which was 730 kbyte in JPEG format. Now 3600 frame * 730 kbyte equals 2.5 gigabytes!

Can you imagine that? I hope not, and that’s because compression brings that way, way down, to the level of tens of megabytes. These days a video of that size is quite high quality and about 2 hours long. Also, don’t forget, that JPEG is already compressed, a single frame would be 6 mbyte when uncompressed. Now that 2-minute video would be 21 gigabytes if we’d store it uncompressed.

Standard codecs such as H.264 and H.265 are doing very clever and complex operations to achieve high compression ratios with good quality.

Just think about that, most frames in a video are quite similar, only containing small differences. So if we could only store that little difference between frames, we’d won a huge bonus! And that’s just one of the many tricks codecs do.

Codec designers are also exploiting the weaknesses and features of the human eye. Such as the fact that we are more sensitive to light intensity changes than color changes (say hello to YUV). And they can get away with lower quality details for parts that are moving fast, and so on.

Because why lose precious bits for things that you can’t even notice?!

There are many codecs out there, with different goals in mind, although the majority focus on keeping the file size low.

H.264, H.265: These are the most common ones, with the widest support in browsers, phones, players, etc. It focuses on small file sizes with good quality. (At the cost of resource intensiveness.)
Apple ProRes, DNxHD: These are common formats for production. They focus on quality and ease of processing and not on file size.

Audio codecs

The goal of audio codecs is the same as what we saw with the video codecs. It is just harder to demonstrate it as audio does not consist of single image frames but audio frames/packets. So an analog audio signal is of an almost infinite, or at least very high quality if you think of it.

At the lowest level, the speed and amplitude resolution is very high. We could say “atomic”, as we need to measure and store the speed and direction of atoms. So if you want to store that exactly, that will require a super high-quality measurement, which will also result in a very high bitrate data stream.

Thankfully, the sound is at least not propagating with light speed so we can save quite a lot just by that fact. (There’s no need for an extreme sampling rate.) Then our hearing is very limited if we take the previous paragraph as a scale, so we win there again. We don’t need most of that high precision that is there.

But still, if we take our hearing capability and want to store raw audio data with about 44.1 kHz of sample rate with about 1 Mbit/sec bitrate, we’d still get quite a lot of data. Check the calculations in the audio bitrate section above.

So raw audio can be compressed further, which is what many popular codecs do. They also exploit the human senses, but this time the human ear. We started with the basics that the human ear has a limit on the frequencies it can detect. Therefore, we can save a lot by cutting out the range of frequencies outside our hearing range. Unless you are a bat, you are fine between 20-20khz! :)

But there are other tricks, for example, auditory masking. That means that the presence of one frequency can affect your capability to detect a different frequency. From the codec’s viewpoint, it can skip encoding a few frequencies if it is smart enough to know which ones you’ll not notice. I’m sure there are a lot more tricks, let me know if you know about a few more interesting ones!

Here is a list of common codecs:

MP3, AAC, OGG: These are common lossy audio formats.
PCM (e.g. in a WAV container), FLAC: These are lossless formats.
MIDI: It is a funny format. It is like a music sheet that might sound different on different players or settings. It is usually not made from real audio data, but from recording a digital keyboard or as an output from an audio composing software.

Containers

Now we got through the fundamental building blocks, the image, the video, the video codecs, and the audio codecs, and we reached the top of this iceberg: the containers.

A container is a format specification, that combines all these streams into a single file format. It defines how to put all these data together, how to attach metadata (e.g. author, description, etc), how to synchronize these streams, and sometimes a container even contains indexes to aid seeking.

So, for example, a MOV container can contain an H.264 video stream and an AAC audio stream together.

Common containers:

MOV
MP4
MKV
WebM
WAV (audio only)

Example Material

I will use these example materials as inputs in the following parts of this article. If you’d like to follow along, save these files for yourself!

Name	Resource
Big Buck Bunny	http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4
Train	train.jpg
Smiley	smiley.png
Voice recording	voice_recording.wav
Big Buck Bunny’s audio	ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 bbb_audio.wav

And we will make our own audio file by extracting the audio from the Big Buck Bunny movie! We’ll use this file as an example, so after downloading the video file, please execute this:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 bbb_audio.wav

By the middle of this article, you’ll understand this command, but for now, just make sure to have the WAV file next to your video file to test out the commands later in the article.

We’ll use these files in the following parts of this article. Therefore make sure to get them!

FFplay and FFprobe

FFmpeg is the name of the main binary and the project itself, but it is shipped together with two other binaries, ffplay and ffprobe.

Let’s check them out quickly, right in the command line!

FFplay

FFplay is a basic video player, that can be used for playing media. It’s not a friendly video player, but it is a good testing ground for various things.

To execute it, just simply supply a media file:

ffplay bbb_sunflower_1080p_60fps_normal.mp4

If you want to test this exact command, you’ll need to get the example files.

For example, it can be used to preview filters (we’ll discuss those later), but let’s see an example:

ffplay -vf "drawtext=text='HELLO THERE':y=h-text_h-10:x=(w/2-text_w/2):fontsize=200:f

FFprobe

FFprobe, as its name implies, is a tool for getting information about media files.

This command:

ffprobe bbb_sunflower_1080p_60fps_normal.mp4

Will return us some general information about the video file:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':
  Metadata:
[...]
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
[...]
  Stream #0:0[0x1](und): Video: h264 [...]
[...]
  Stream #0:1[0x2](und): Audio: mp3 [...]
[...]
  Stream #0:2[0x3](und): Audio: ac3 [...]

I have abbreviated it heavily, as we’ll check this out later.

But FFprobe is way more powerful than just this!

With the following command, we can get the same listing in JSON format, which is machine-readable!

ffprobe -v error -hide_banner -print_format json -show_streams bbb_sunflower_1080p_60fps_normal.mp4

The explanation of this command is the following:

“-v error -hide_banner”: This part hides extra output, such as headers and the default build information.
“-print_format json”: Obviously, this causes ffprobe to output a JSON.
“-show_streams” is the main switch that requests the stream information.

{
  "streams": [
    {
      "index": 0,
      "codec_name": "h264",
      "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
      "width": 1920,
      "height": 1080,
      "bit_rate": "4001453",
      "duration": "634.533333",
      "############################": "[~50 lines removed]"
    },
    {
      "index": 1,
      "codec_name": "mp3",
      "channels": 2,
      "bit_rate": "160000",
      "############################": "[~40 lines removed]"
    },
    {
      "index": 2,
      "codec_name": "ac3",
      "channels": 6,
      "############################": "[~20 lines removed]"
    }
  ]
}

In this output, you can see three streams of data in this video file. The first (index: 0) is a video stream, that is an HD video with an H.264 codec. Then we have two audio streams, the first (index: 1) is a simple mp3 stream with stereo audio, and the second (index: 2) is an ac3 stream with 6 channels, most likely in an 5.1 configuration.

I have removed quite a lot of output for brevity, but you can get way more information out of these streams, e.g. fps for the video stream and so on.

Other than -show_streams, there are 3 more: -show_format, -show_packets and -show_frames. Unless you are really deep in the rabbit hole, you’ll not need the last two, but -show_format could be useful:

ffprobe -v error -hide_banner -print_format json -show_format bbb_sunflower_1080p_60fps_normal.mp4

{
  "format": {
    "filename": "bbb_sunflower_1080p_60fps_normal.mp4",
    "nb_streams": 3,
    "nb_programs": 0,
    "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
    "format_long_name": "QuickTime / MOV",
    "start_time": "0.000000",
    "duration": "634.533333",
    "size": "355856562",
    "bit_rate": "4486529",
    "probe_score": 100,
    "tags": {
      "major_brand": "isom",
      "minor_version": "1",
      "compatible_brands": "isomavc1",
      "creation_time": "2013-12-16T17:59:32.000000Z",
      "title": "Big Buck Bunny, Sunflower version",
      "artist": "Blender Foundation 2008, Janus Bager Kristensen 2013",
      "comment": "Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net",
      "genre": "Animation",
      "composer": "Sacha Goedegebure"
    }
  }
}

This is an overview of “what is this file”. As we see, it is a MOV file (format_name), with three streams (nb_streams), and it is 634 seconds long. Also, there are some tags where we can see the title, the artist, and other information.

FFmpeg concepts

Here is a quick intro to how FFmpeg actually works!

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

FFmpeg opens the file, decodes it into memory, then encodes the in-memory packets back and puts them into some container: some output file. The term “codec” is a mix of the words “coder & encoder”. Those are the magic parts before and after the “decoded frames”.

The decoded frames are uncompressed images in-memory, e.g. the most basic pixel format for video frames is called “rgb24”. This just stores red, green, and blue values right after each other in 3x8 bits, or 3x1 byte, which could hold 16m colors.

The importance of this is that other than a few exceptions, you can only manipulate or encode the decoded frames. So when we get to different audio/video filters or transcoding, you’ll need the decoded frames for all that. But don’t worry, FFmpeg does this automatically for you.

Inputs

So you see and probably guessed, that FFmpeg must access the input data somehow. FFmpeg knows how to handle most media files, as the awesome people who develop FFmpeg and the related libraries made encoders and decoders for most formats available!

Don’t think that it is a trivial thing. Many formats are reverse engineered, a hard task requiring brilliant people.

So although we often refer to input files, the input could come from many sources, such as the network, a hardware device and so on. We’ll learn more about that later on in this article.

Many media files are containers for different streams, meaning that a single file might contain multiple streams of content.

For example, a .mov file might contain one or more streams:

video tracks
audio tracks (e.g. for the different languages or audio formats such as stereo or 5.1)
subtitle tracks
thumbnails
…

All these are streams of data from the viewpoint of FFmpeg. Input files and their streams are numerically differentiated with a 0-based index. So, for example, 1:0 means the first(0) stream of the second(1) input file. We’ll learn more about that later too!

Important to note that FFmpeg can open any number of input files simultaneously, and the filtering and mapping will decide what it will do with those. Again more on that later!

Streams

As we have seen in the previous section, streams are the fundamental building blocks of containers. So every input file must have at least one stream. And that’s what you can list by the simple ffmpeg -i command for example.

A stream might contain an audio format such as MP3, or a video format such as an H.264 stream.

Also, a stream, depending on the codec, might contain multiple “things”. For example, an mp3 or a WAV stream might include various audio channels.

So the building block hierarchy, in this case is: File → Stream → Channels.

Outputs

Of course, an output could be a local file, but it doesn’t need to be. It could be a socket, a stream and so on. In the same way as with inputs, you could have multiple outputs, and the mapping determines what goes into which output file.

The output also must have some format or container. Most of the time FFmpeg can and will guess that for us, mostly from the extension, but we can specify it too.

Mapping

Mapping refers to the act of connecting input file streams with output file streams. So if you give 3 input files and 4 output files to FFmpeg, you must also define what should go to where.

If you give a single input and a single output, then FFmpeg will guess it for you without specifying any mapping, but make sure you know how exactly that happens, to avoid surprises. More on all that later!

Filtering

Filtering stands for the feature of FFmpeg to modify the decoded frames (audio or video). Other applications might call them effects, but i’m sure there is a reason why FFmpeg calls them filters.

There are two kinds of filtering supported by FFmpeg, simple and complex. In this article we’ll only discuss the complex filters, as it is a superset of the simple filters, and this way, we avoid confusion and redundant content.

Simple filters are a single chain of filters between a single input and output. Complex filters can have more chains of filters, with any number of inputs and outputs.

The following figure extends the previous overview image with the filtering module:

A complex filter graph is built from filter chains, which are built from filters.

So a single filter does a single thing, for example, changes the volume. This filter is quite trivial, it has a single input, changes the volume, and it has a single output.

For video, we could check out the scale filter, which is also quite straightforward: it has a single input, scales the incoming frames, and it has a single output too.

You can chain these filters, meaning that you connect the output of one to the input of the next one! So you can have a volume filter after an echo filter, for example, and this way, you’ll add echo, and then you change the volume.

This way, your chain will have a single input, and it will do several things with it and will output something at the end.

Now, the “complex” comes in when you have multiple chains of these filters!

But before we go there, you should also know that some single filters might have multiple inputs or outputs!

For example:

The overlay filter puts 2 video streams above each other and will output a single video stream.
The split filter splits a single video stream into 2+ video streams (by copying).

So let’s discuss a complex example from a bird’s eye view! I have two video files, I want to put them above each other, and I want the output in two files/sizes, 720p and 1080p.

Now, that’s where complex filtering will be faithful to its name: to achieve this, you’ll need several filter chains!

Chain 1: [input1.mp4] [input2.mp4] → overlay → split → [overlaid1] [overlaid2]
Chain 2: [overlaid1] → scale → [720p_output]
Chain 3: [overlaid2] → scale → [1080p_output]

As you see, you can connect chains, and you can connect chains to output files. There is a rule that you can only consume a chain once, and that’s why we used split instead of the same input for chains 2 and 3.

The takeaway is this: with complex filter graphs (and mapping), you can:

build individual chains of filters
connect input files to filter chains
connect filter chains to filter chains
connect filter chains to output files

FFmpeg’s command line system

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

FFmpeg CLI

Finally, we arrived at FFmpeg, and trust me, we’ll execute it quite a lot of times! Let’s see how FFmpeg’s command line options are organized, as that is the first tricky part we need to understand!

FFmpeg mostly thinks about input and output files and their options together with global options. You specify input files with the “-i” flag followed by a file name. For the output file, specify it as-is without any preceding CLI (command line interface) flag.

Specifying an input file

Let’s specify just an input file:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4

The following image helps to understand the output:

First, you get the “banner”, where you see the build information and lib versions. If you watch closely, you’ll see the compilation flags, starting with —, e.g. —enable-shared.
Then you get the same output as we have seen with ffprobe earlier.
And then you get a complaint that there is no output file(s) specified. That’s fine for now.

You can remove the banner here with “-hide_banner”, but for brevity’s sake I’ll not include that anymore in the commands here, and I will leave it out from the outputs too.

Now, let’s get brave, and specify an output file!

Specifying an output

As I’ve said earlier, the output file is understood by FFmpeg as it is just a filename. But more specifically, it is after the input(s) specifications, and it is not a value of any other switches.

Don’t be confused for now, but yes, FFmpeg can have as many inputs and outputs as you’d like. We’ll cover that in more detail soon!

This command line specifies a single output file:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

Before taking a look at the output, let me congratulate you! You have just converted a video file into an audio file, by keeping just the audio content!

This is how you transcode! Of course, you’ll want to specify more parameters later on.

So, here is the output:

Let’s analyze it!

(1) First, we have our input metadata printing, which we saw many times already.

(2) Then we have something called “stream mapping”. We forced FFmpeg into a decision situation, as we specified an input file with 1 video and 2 audio streams. We said we wanted an audio output (guessed from the .wav extension). But we didn’t specify which audio stream we wanted, so let’s see what FFmpeg decided:

“Stream #0:2” means “The first input file’s third stream” or “input file index 0’s stream with index 2.” This is the input.
”-> #0:0” means the first output file’s first stream. This is the output.
Here you can learn more about how FFmpeg decide this.
Later on, we’ll manually override the mapping.
Summary: FFmpeg decided to convert the third stream in the input file (the ac3 5.1 audio) into the first stream of the output file.

(3) Then we have our output metadata information. This reveals what FFmpeg will output. It usually copies most of the metadata, and here you also see the container/format information too.

(4) And then we see the output summary. For example, the transcoding was 181x faster than the playback speed. Nice!

Understanding the command line order

Before going further, let’s understand FFmpeg’s command line arguments from a bird’s eye view!

In the manual, you’ll see this:

ffmpeg [global_options] {[input_file_options] -i input_url} ... {[output_file_options] output_url} ...

(Parts in […] are meant to be optional, and parts in {…} are meant to be specified 1 or more times.)

This is the general outline of how to specify inputs, outputs, input options, output options, and global options. The order matters, but it is easy to remember: global options, inputs and outputs. Also, i/o options come BEFORE the i/o specification.

Let’s put these into pseudo command line options, to understand it better:

# One inputs, one output, nothing fancy
ffmpeg -i input1.mp4 output1.wav

# Two inputs, one output
ffmpeg -i input1.mp4 -i input2.mp4 output1.wav

# Two inputs, two outputs
ffmpeg -i input1.mp4 -i input2.mp4 output1.wav output2.mp3

# One input, one output, with options
ffmpeg [input1 options] -i input1.mp4 [output2 options] output1.wav

# Two inputs, two outputs with options
ffmpeg [input1 options] -i input1.mp4 \
       [input2 options] -i input2.mp4 \
       [output1 options] output1.wav \
       [output2 options] output2.mp3

As for the global options, these are the ones you might care about:

-hide_banner: To skip printing the banner.
-y: To overwrite the output even if it exists.

For example, you can run this as many times as you want:

ffmpeg -y -hide_banner -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

And it will overwrite the output and be less verbose than earlier.

Without explaining the options themselves, let’s just see some real-world examples with options:

And here it is with two inputs and two outputs:

Mapping files

We saw above that this command:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

… will result in an audio file that contains one of the audio streams from the input video chosen by FFmpeg. This automatic stream selection is usually handy when it is trivial. For example, when you have one stream as input and one output file, you don’t need to specify any mapping manually.

But in cases where it is not so trivial, you are usually better off manually specifying what you really want to do.

The following image summarises what our current situation is:

The video stream was not matched, as the output format was an audio file (.wav). But then FFmpeg chose Stream #2, because it has more channels.

So what if we’d like to get the stereo track instead? That is where mapping comes in! The mapping is a parameter of the OUTPUT file. Therefore the mapping arguments should come right before our output file definition!

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav

The argument -map 0:1 means, that in the output (since we specify it as an output option) we’d like to have Input #0’s (the first input file) Stream #1!

Let’s see the relevant parts from the output!

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':

[...]

Stream mapping:
  Stream #0:1 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))

[...]

Output #0, wav, to 'stereo_audio_only.wav':
  Metadata:
[...]
    Stream #0:0(und): [...] stereo [...]

The “Stream #0:1 -> #0:0” part means that we have successfully overridden the mapping, to get the mp3 stream (0:1) into our output! Also, the output metadata reveals that we’ll get a stereo result instead of the 5.1 earlier.

Multiple outputs

You can have multiple outputs from a single input, let’s see when that might be useful!

Let’s say, we want to extract BOTH audio streams into two separate WAV files! It is super easy:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav -map 0:2 ac3_audio_only.wav

See? I have just specified two output files with two mapping specifications! Also, I have sneaked in the “-y” to have it overwrite our previous file!

Let’s check out the relevant parts of the output!

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':

[...]

Stream mapping:
  Stream #0:1 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))
  Stream #0:2 -> #1:0 (ac3 (native) -> pcm_s16le (native))

[...]

Output #0, wav, to 'stereo_audio_only.wav':
    Stream #0:0(und): [...] stereo

[...]

Output #1, wav, to 'ac3_audio_only.wav':
    Stream #1:0(und): Audio: [...] 5.1(side)

Now the mapping reveals two lines, as we have two outputs! And indeed, you’ll get two .wav files as the output, one is stereo, and one is 5.1!

There might be several other reasons why you’d want to get multiple outputs. Let’s briefly check out a few!

Different formats:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 stereo_audio_only.wav  stereo_audio_only.mp3

Wow, did you catch that? We just created a WAV and an mp3 in a single command line! I’ve reverted to the automatic stream selection for brevity’s sake.

A bit closer to real-life needs, you might want different output qualities:

ffmpeg -y  -i bbb_sunflower_1080p_60fps_normal.mp4  \
-map 0:1 -b:a 320k stereo_audio_only_high_quality.mp3 \
-map 0:1 -b:a 64k  stereo_audio_only_low_quality.mp3

Here -b:a 320k means “bitrate of audio should be around 320 kbit/sec”. So I have requested FFmpeg to make two mp3s for me, from the stereo stream of the input.

Checking on the files, this is what we got:

 25Mb stereo_audio_only_high_quality.mp3
4,9Mb stereo_audio_only_low_quality.mp3

One more common reason for having multiple outputs or using mapping is when we introduce filters into our pipeline, but that will be discussed later!

Now you understand the foundations of how to communicate your basic requirements to FFmpeg via its command line! Great job! Now we can dive even deepert.

Hands-on with FFmpeg

In this section, we will discover and even try out some common features of FFmpeg!

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

Inputs

Let’s see the common ways FFmpeg is fed with different data!

File

Of course, you have already seen that if you have a local file on your filesystem, FFmpeg is happy to read it!

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav

This command which is exactly the same as one of our previous ones just reads a local file. Really, that’s it.

Network

Did you know, that FFmpeg can open a file directly on the network?!

ffmpeg -t 5 -i http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4 bbb_first_5_seconds.mp4

The command above opens the file directly from the network and saves the first 5 seconds into a local file!

I wanted to spare bandwidth for these awesome guys over renderfarming.net, so I added the duration flag: -t 5. FFmpeg doesn’t even download the full video for this operation. Isn’t that wonderful?!

Webcam

FFmpeg can also open your webcam!

This is an example command for Linux:

ffmpeg -f v4l2 -framerate 25 -video_size 640x480 -t 10 -i /dev/video0 10seconds_of_webcam.webm

This would record 10 seconds of your webcam!

Accessing the webcam happens differently on different platforms. Also specifying parameters is different for each platform, so for this reason, if you’d like to access your webcam with FFmpeg, please refer to the documentation:

Microphone

Let’s record some audio directly from your microphone!

List microphones:

arecord -l

Start 10 seconds of recording:

ffmpeg -f alsa -i hw:0,0 -t 10 out.wav

This command was meant to work on Linux, but you can check out how to do that on Microsoft Windows or macOS.

Pipe

Finally, FFmpeg can read from a pipe, and also output to a pipe.

On Linux, you could do something like this:

cat bbb_sunflower_1080p_60fps_normal.mp4 | ffmpeg -i - -f wav pipe:1 | pv > output.wav

# Alternative, without pv:
cat bbb_sunflower_1080p_60fps_normal.mp4 | ffmpeg -i - -f wav pipe:1 > output.wav

This command would use the cat program to simply read in the video file and output it to its standard output. Then this output is piped INTO FFmpeg, through its standard input. The combination “-i -” means “read from standard input”. By the way, standard input would be your keyboard otherwise, if we wouldn’t use any redirection here.

Then we specify the required output format for FFmpeg, with “-f wav”. This is needed because now we’ll have no output file name, and FFmpeg will not be able to guess the format. Then we specify “pipe:1” as an output, meaning we’d like FFmpeg to output to its standard output.

From then, we pipe the data into a program called “pv”, it is just a metering tool, that dumps information on the throughput (from its stdin to its stdout). Finally, we redirect pv’s output into a WAV file.

You might ask why we’d want to do that, why we talk about this. Piping can be useful if you build a complex pipeline from different programs or if you want to spare reading and writing to a local file.

For example, the node package fluent-ffmpeg can leverage this functionality by supplying input and output streams. For example, you can read from an S3 bucket and write to one directly.

But be warned, hell is awaiting you on that road. No kidding. You need to research the limitations of this technique. For example, many formats can not be streamed in this manner, as they need random access to the output data to write the indices at the beginning of the file after processing.

Outputs

FFmpeg can output into many protocols, from local file storage and ftp to message queue protocols all the way to streaming protocols.

For more information, check out the documentation here.

Transcoding audio with FFmpeg

In this chapter, we’ll be going to see how to transcode into audio with FFmpeg!

The general formula is:

ffmpeg -i {input audio or video file with audio} [output options] output_audio.ext

Choosing a format

FFmpeg is quite smart, and by the extension, it can determine which codec to use. If you specify “audio.wav” or “audio.mp3” for example, FFmpeg will use the appropriate codec to do the encoding.

It is perfectly guessing most of the time. But if you want to specify the format manually, then the “-f” flag is your friend.

For this, you might want to consult the list of formats:

ffmpeg -formats

So, these three commands will do exactly the same, but the last two requires the -f flag.

# Output codec is determined from the extension
ffmpeg -i bbb_audio.wav bbb_audio.mp3

# No extension in the filename
ffmpeg -i bbb_audio.wav -f mp3 bbb_audio

# Piped output therefore no filename, so no extension to use for guessing
ffmpeg -i bbb_audio.wav -f mp3 pipe:1 > bbb_audio

Setting the bitrate

In most cases. you want to specify the target bitrate you expect from your codec to output. If you are unsure what bitrate is, please read this article’s audio bitrate section.

To specify the audio bitrate, use the “-b:a” option with a corresponding value, e.g.:

-b:a 320k: For the mp3 codec this is considered high quality.
-b:a 128k: Lower quality.
-b:a 64k: Low quality.

For example:

ffmpeg -i bbb_audio.wav -b:a 320k bbb_audio_320k.mp3

Setting the sample rate

You may want to specify the sample rate to ensure quality or low output file size. Half the sample rate could mean half the output file size. If you are unsure what the sample rate is, please read the “audio sample rate” section of this article.

To specify the audio sample rate, use the “-ar” option with a corresponding value, e.g.:

-ar 48000: For high quality.
-ar 44100: For CD quality (still high).
-ar 22500: A bit of a compromise, not recommended for music, but for speech, it might be enough.
-ar 8000: Low quality, e.g. if you only want “understandable” speech.

For example:

ffmpeg -i bbb_audio.wav -ar 44100 bbb_audio_44100khz.mp3

Setting the channel count

Setting the channel count can be useful, for example, if you have a stereo recording of a single person’s speech. In that case, you might be content with just a mono output half the size of the original recording.

If you are unsure what an audio channel is, please read the “audio channels” section of this article.

To specify the channel count use the “-ac” option with a corresponding value, e.g.:

-ac 1: For mono
-ac 2: For stereo
-ac 6: For 5.1

For example:

ffmpeg -i bbb_audio.wav -ac 1 bbb_audio_mono.mp3

Complete command line for converting audio with FFmpeg

This is how you produce a high-quality output:

# Convert wav to mp3
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.mp3

# Convert wav to m4a (aac)
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.m4a

# Convert wav to ogg (vorbis)
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.ogg

Check out this documentation about good quality audio transcoding too!.

Lossless formats

If you want to convert audio into a lossless format, here are a few choices for you:

# Convert to flac (Free Lossless Audio Codec)
ffmpeg -i bbb_audio.wav -compression_level 12 bbb_audio_lossless_12.flac # Best compression, slowest
ffmpeg -i bbb_audio.wav -compression_level 5 bbb_audio_lossless_5.flac   # Default
ffmpeg -i bbb_audio.wav -compression_level 0 bbb_audio_lossless_0.flac   # Least compression, fastest


# Convert to wav
cp bbb_audio.wav bbb_audio_lossless.wav # Just kidding:)

# Convert to wav
ffmpeg -i any_audio.ext bbb_audio_lossless.wav

It’s good if you know that flac results in a smaller file than WAV, as WAV doesn’t actually compress by default:

117M bbb_audio.wav
52M  bbb_audio_lossless_0.flac
45M  bbb_audio_lossless_5.flac
43M  bbb_audio_lossless_12.flac

WAV is generally thought of as a lossless format, but keep in mind that the WAV container can contain lossy content too, but by default FFmpeg uses the pcm_s16le format, which is the 16 bit PCM, that could be understood as lossless.

Learn more here and here.

Transcoding video with FFmpeg

In this chapter, we’ll be going to see how to transcode a video file into the two most common formats!

Converting to H.264

H264 is one of the most popular video codecs. Most devices, browsers and video players understand how to play it. It is efficient in storing video content, but as with most advanced video codecs, it is a resource intensive-process to encode and decode.

A complete command line for a high-quality H.264 transcoding with high-quality AAC audio is the following:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 \
-c:v libx264 -preset slow -crf 22 \
-profile:v main -g 250 -pix_fmt yuv420p \
-map 0:0 -map 0:1 \
-acodec aac -ar 44100 -b:a 320k bbb_transcoded_h264_HQ.mov

Make sure to understand this command and to customize it to match your needs.

To help you do that, let’s dissect this command!

Global options:

-y: Overwrite the output.

Input options:

-i bbb_sunflower_1080p_60fps_normal.mp4: The input file.

Output options:

-c:v libx264: Set the codec to libx264.

-preset slow: libx264 has a lot of variables that you can be tune, and most of them balance the coding speed and the resulting file size. To make your life easier, there are presets by which you can easily declare what you need: small size or speed.

-crf 22: This is the constant rate factor, the main option for setting image quality. It is a number between 0-51, where 0 is lossless, and 51 is the worst quality. Generally, you want something between 17 and 28. This is the option to tune the balance between image quality and file size. Check my comparison video here.

-profile:v main -g 250 -pix_fmt yuv420p: These are advanced options, guaranteeing you a quite backward compatible result. (See this, this, and this.)

-map 0:0 -map 0:1: You might not need this: these options are selecting the correct video and audio streams. In our case, we have two audio streams, and we need the stereo one to avoid some issues with our aac stream.

-acodec aac: Select the AAC (Advanced Audio Coding) codec for the audio in the output. We need to be more specific than just -f for the format. We need to specify the audio codec here manually.

-ar 44100: Set the audio sampling rate (learn more about that in previous chapters of this article).

-b:a 320k: Set the audio bitrate (learn more about that in previous chapters of this article).

30seconds_of_bb.mkv: The output file name. All the options since the last -i (or the last output file) considered to be a modifier for this output.

Let’s see the output:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':

[...]

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (mp3 (mp3float) -> aac (native))

[...]

Output #0, mov, to 'bbb_transcoded_h264_HQ.mov':
    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 60 fps, 15360 tbn, 60 tbc (default)
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, 5.1(side), fltp, 320 kb/s (default)

[...]

frame=38074 fps= 35 q=-1.0 Lsize=  324855kB time=00:10:34.51 bitrate=4194.1kbits/s dup=2 drop=0 speed=0.58x

From this, we understand that FFmpeg chose the mp3 stream from the input file because we told it to do so. (Remember, it has two audio streams in it, a stereo mp3 and a 5.1 ac3.) We also see that my machine could transcode with 35fps (0.58 times the playback speed), and our settings resulted in an average video bitrate of 4200 kbit/s.

The video bitrate is an interesting question in this mode. With the CRF option, we specify the “constant visual quality” we want. To reach a constant visual quality, the encoder works hard to guess how much it can compress certain parts of every frame, and the result of that guess defines the final average video bitrate.

If you want even better results with H.264, and you can afford a bit more processing time and a bit more complicated process, check out the 2-pass encoding instead of the constant rate factor method introduced above.

To learn more about these two different rate control methods, read the awesome Understanding Rate Control Modes article. And to learn more about the intricacies of H.264 encoding, check out the H264 encoding guide.

Finally, later on, I will show you a comparison video that shows how different CRF values perform!

Converting to H.265

H.265 is the successor of H.264, according to the official FFmpeg manual, it offers 25-50% bitrate savings while retaining the same visual quality.

A complete command line for a high-quality H.265 transcoding with high-quality AAC audio is the following:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 \
-c:v libx265 -preset slow -crf 27 \
-profile:v main -g 250 -pix_fmt yuv420p \
-map 0:0 -map 0:1 \
-acodec aac -ar 44100 -b:a 320k bbb_transcoded_h265_HQ.mov

And the result is:

...
encoded 38074 frames in 3384.84s (11.25 fps), 1720.32 kb/s, Avg QP:35.29

H.265 also has multiple rate control algorithms, I used the CRF method here. If you want to use a different rate control algorithm, then you may check out the H.265 encoding guide. Also, check out the next section, where I’ll reveal how different CRF values perform!

This command is almost the same as what we used in the H.264 example above, so please refer to that section to understand the arguments.

If we compare H.264 and H.265 with our commands above, taking into account this 10-minute long video on my system, these are the results:

H.264 is 3 times faster (35 fps vs 11 fps)
H.264 produces a 2 times larger file (318 mb vs 156 mb)

Comparing CRF values with H.264 and H.265

I have created a video for your convenience, that shows the different crf values in action. The selected frame had some movement on it with the leaves in the bunny’s hand. Movement is important with video codecs, as usually that’s where quality losses are first visible.

This video shows how the different CRF values perform, from 0-51 with the H.264 and H.265 formats!

H.264 & H.265 CRF comparison video

(Can you guess which program I was using to make this?:))

Basic editing with FFmpeg

In this section, we’ll achieve basic editing tasks by using FFmpeg only!

We’ll just get a basic mp4 with default settings in these examples to keep things simple. But to encode the result in a proper, high quality way, please check the earlier sections where we learned how to encode into H.264 and H.265!

Trimming from the beginning of the clip

It is possible to specify an in-point for a media file. By doing that, you essentially cut off the specified amount from the beginning of the input file. Therefore, FFmpeg will skip the first part of the file and only transcode the remainder!

For this, you need the “-ss” flag! The value can be specified in seconds (5 or 5.2) or as a timestamp (HOURS:MM:SS.MILLISECONDS).

To get the outro only, we could seek all the way to the end of the video! (It is 00:10:34.53 or 635 seconds long!)

# Get
# 635 - 4 = 631
ffmpeg -y -ss 631 -i bbb_sunflower_1080p_60fps_normal.mp4 last_4_seconds.mp4


# 00:10:34.53 - 4 = 00:10:30.53
ffmpeg -y -ss 00:10:30.53 -i bbb_sunflower_1080p_60fps_normal.mp4 last_4_seconds.mp4

Seeking can be a bit tricky, so you may want to learn more about seeking here.

Trimming from the end of the clip

You can also set an out-point for an input file, therefore shortening it. There are two options for this:

-t: This sets the duration.
-to: This sets the timestamp where the input video should stop.

These two are mutually exclusive, and also they do the same if no -ss is specified. The value can be specified in seconds (5 or 5.2) or as a timestamp (HOURS:MM:SS.MILLISECONDS).

Let’s experiment with them!

# "Get 30 seconds of the input."
ffmpeg -y -t 30 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -t 00:00:30.0 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4

# "Get everything until the content's 30th second."
ffmpeg -y -to 30 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -to 00:00:30.0 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4

All four above commands result in exactly the same video. (For nerds: even the md5sum is the same.)

But let’s see how they perform when we introduce seeking!

# "Seek to the 10th second and get me 30 seconds of the input."
ffmpeg -y -ss 10 -t 30 -i bbb_sunflower_1080p_60fps_normal.mp4 part_between_10_and_40.mp4

# "Seek to the 10th second and get the content until the 30th second."
ffmpeg -y -ss 10 -to 30 -i bbb_sunflower_1080p_60fps_normal.mp4 part_between_10_and_30.mp4

The first command will result in a 30 second long video, while the second command will be 20 seconds long only!

The figure below shows the difference:

Editing without reencoding

FFmpeg can do something I’m not aware of in any other popular NLE: it can edit videos without reencoding them!

The usual workflow is to decode the data frames (a/v) into memory, modify them as much as we like and then encode them into a new video file. The problem with this is that unless you work with raw or lossless codecs, you’ll lose some quality in the process. Another issue with this approach is that it is computationally intensive.

For certain operations, you can configure FFmpeg, to keep the data frames intact, and this way, you can avoid decoding and encoding them! This is incredibly faster than regular transcoding, usually hundreds of times faster.

The “certain operations” are those that don’t need to modify the data frames themselves. For example, you can cut and trim this way. Also, you can manipulate streams while keeping others, like you can replace the audio track without touching the video frames.

All this is a bit of magic, and there are caveats you need to prepare for, but it is good if you know about this, as it is often handy!

The trick lies in two options:

-c:v copy: The “copy” video codec
-c:a copy: The “copy” audio codec

Let’s see a few examples!

Remove audio while keeping the video without reencoding

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -c:v copy -an copied_video_only.mp4

Here, we used the “-an” option, which removes all audio streams. I remembered it as “audio no”, but that is just my mnemonic:)

Let’s see how fast it was:

frame=38072 fps=20950 q=-1.0 Lsize=  310340kB time=00:10:34.51 bitrate=4006.7kbits/s speed= 349x

So It processed the whole 10 minutes of video in 2 seconds, 349x faster than playback, with 20950 fps!

Remove video while keeping the audio without reencoding

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -c:a copy -vn copied_audio_only.wav

Here, we used the “-vn” option, which removes all video streams. I remembered it as “video no”.

Let’s see how fast it was:

size=   24772kB time=00:10:34.14 bitrate= 320.0kbits/s speed= 776x

776x faster than playback, finished in about a second, not bad!

Cut and trim without reencoding

ffmpeg -ss 10 -t 10  -i bbb_sunflower_1080p_60fps_normal.mp4 -c:a copy -c:v copy part_from_10_to_20_copied.mp4

There could be precision issues with seeking while you do this, so you may want to learn more about seeking and copying here.

Replace audio on video file without reencoding

We have removed audio and video already, but what if we want to swap them?

ffmpeg -y \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-i voice_recording.wav \
-map "0:v" -map "1:a" \
-c:v copy -c:a copy \
bbb_with_replaced_audio.mov

There is quite a lot going on in here, so let’s explain the parts!

First, we have two inputs (-i), meaning we are better off manually specifying the mapping. The command would work without the “-map” options, but it would ignore our second input.

-map "0:v" -map "1:a" means that please use the first file’s (first) video stream and the second file’s (first) audio stream.

With -c:v copy -c:a copy, we require FFmpeg to copy the already encoded data packets without touching them. Therefore FFmpeg’s work is mostly really just copying bytes, no decoding, no encoding.

Not surprisingly, that’s what we see in the stream mapping too:

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=38072 fps=9750 q=-1.0 Lsize=  320645kB time=00:10:34.51 bitrate=4139.7kbits/s speed= 162x

And since it is just copying, it was crazy fast, 162x of the playback speed, or almost 10k frames per second!

But!

Execute the exact same command, but with “bbb_with_replaced_audio.mp4” (.mp4 container instead of .mov) as an output file! You’ll get this:

Could not find tag for codec pcm_s16le in stream #1, codec not currently supported in container

The message is quite clear. You can not have a pcm_s16le (raw WAV, say that 10 times:)) stream in an MP4 container. I’m not sure if it is FFmpeg’s or the container’s lack of support, but we need to solve this. If you run into this situation, you might consider two solutions:

Change the container: I’ve just tried MOV, and it worked.
Encode the audio: We still copy the video data, and encoding audio isn’t that painful.

I just showed you option #1, so let’s see option #2:

ffmpeg -y \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-i voice_recording.wav \
-map "0:v" -map "1:a" \
-c:v copy \
-c:a aac -b:a 320k -ar 44100 \
bbb_with_replaced_audio_aac.mp4

This copies the video frames and encodes our WAV into a supported codec to be held in the mp4 container. You can refer back to the audio encoding section if you want to learn more about that.

Here is the output:

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
...
frame=38072 fps=2176 q=-1.0 Lsize=  313058kB time=00:10:34.51 bitrate=4041.8kbits/s speed=36.3x

“Only” 36x faster than playback, 2176 fps, still not that bad!

Filtering overview

FFmpeg supports many audio and video filters. Currently, there are 116 audio and 286 video filters, but there are a bit more if we count the hardware accelerated ones too.

So how do we leverage them?

There are two ways to define filters, but I’m going to explain the complex filter, as the difference is not much, but it is more versatile. So there is a global option for FFmpeg, called: -filter_complex. With quite a weird syntax, you can specify all your filters and their parameters right after this option.

You can imagine the process with the following image:

Basically, your filter graph can access all the inputs (-i a.mp4 -i b.mp4 -i c.mp4), and it can produce as many outputs as you like (-map might be needed).

Basic syntax

Let’s take a look at a simple, basic example:

ffmpeg -y  -t 5 \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex1.mp4

Although -filter_complex is a global option, I like to put it after the inputs and before the outputs as it is a bit easier to overlook the whole command that way. Thankfully the command line parser of FFmpeg is smart enough, and it works.

The command above produces a 5-second-long video, where the text “HELLO THERE” is overlaid on the intro of Big Buck Bunny.

Let’s understand the weird format for specifying filters!

We’ll go bottom-up, and we’ll build it from there. So the most basic format is this:

FILTER_NAME=ARGUMENT1=VALUE1:ARGUMENT2=VALUE2

For example:

drawtext=text='HELLO THERE':y=20:x=30

The first thing before the first equal (=) sign is the filter’s name, which is the drawtext filter in this case. Then we have our first argument, “text” and its value “‘HELLO THERE’”. Right after that, separated with a colon (:) comes the next argument, “y” with a value of “20”.

You can guess what each of the text, y, x, fontsize and fontfile arguments do, as it is quite self-explaining. But especially for the first time, you’ll heavily rely on the filtering documentation to understand every filter and every argument.

Also, several characters are reserved, such as: , : = and a few others depending on your environment, so sooner or later you need to learn about escaping too.

To recap, our pipeline looks like this now:

Multiple filters in a chain

This previous command is a single filter chain that consists of a single filter only, but you could have more filters put right after each other! It means that the output of one filter will be the input for the next! The way to do this is by separating them with a comma!

Let’s draw two boxes with the drawbox filter!

ffmpeg -y  -t 5 \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "  drawbox=x=10:y=10:w=100:h=100:color=red  ,  drawbox=x=200:y=200:w=100:h=100:color=blue  " \
filter_complex2.mp4

See? The output of the first filter is passed to the output of the second filter!

Let’s visualize our pipeline again:

Input and output pads

Now, we have skipped something this far, because for simple uses FFmpeg is smart enough to do it for us. And this is the specification of a chain’s input and output pads!

Let’s draw just a single rectangle for now:

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex "drawbox=x=10:y=10:w=100:h=100:color=red" filter_complex3.mp4

FFmpeg sees that the input for our filter chain is a single video file, and the output is a single output video file. Therefore, it safely assumes that we want that single input as the input of our single filter chain. And that single output should be the single output of our single output chain.

That’s really nice, as, in simple situations like this, we don’t need to assign and map inputs and outputs manually! But when we get more inputs, filter chains, or outputs, it is no longer possible. Therefore, we need to understand how to assign inputs and outputs!

First of all, let’s compare the following two command lines. They result in exactly the same result, but the second one represents what FFmpeg does internally (roughly):

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex "drawbox=x=10:y=10:w=100:h=100:color=red" filter_complex3.mp4

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex "[0:v]drawbox=x=10:y=10:w=100:h=100:color=red[out_link_0]" -map "[out_link_0]" filter_complex3.mp4

Do you see the difference? Before our filter chain, an “input pad” is defined: [0:v]. The expected format between the square brackets is documented in the stream selection section of the official documentation, and this article already covered it.

But, a quick summary:

0:v: This means the first video stream of the first input file.
0:v:0: Means exactly the same thing but in a long form.
0:0: Means the first stream of the first input file (not recommended, as it could be anything in theory. It could be a subtitle stream, a thumbnail, a video or an audio stream…)
0:a: This means the first audio stream of the first input file.
0:a:0: Means exactly the same thing but in a long form.
0:a:1: Means the second (index #1) audio stream of the first input file.

So we can specify which input file should be connected to which input of the filter graph!

Also, something similar is going on at the end! Do you see, the [out_link_0] output pad definition at the end of our filter chain?

The naming here is easier, as basically you can specify any arbitrary name in here. It roughly means, “please store the output data under this name”.

And when you specify your output file, you can or need to map it by selecting one of your filter graph outputs! Therefore, we must add the -map “[out_link_0]” option before our output file.

This map option means this: “Please save the data stream with this name into the following output file.”

This is how you can visualize this input/output mapping:

Multiple chains

Coming from the previous sections, you are now ready to see and understand an even more complicated configuration, which has multiple input files, output files, and filter chains!

ffmpeg -y  \
-i train.jpg \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "[0:v]drawbox=x=10:y=10:w=100:h=100:color=red[train_box] ; [1:v]drawbox=x=10:y=10:w=100:h=100:color=red[bbb_box]" \
-map "[train_box]" filter_complex4_train.jpg \
-map "[bbb_box]" filter_complex4_bbb.mp4

Let’s see the output (two files next to each other):

We had two inputs, and we got two output files, an image, and a video, with a red rectangle on them, with a single command!

Are you still here? I hope! Let’s understand what happened in that crazy command! We have two input files:

-i train.jpg: A simple image file
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4: Our video file, but to make it quick, just the first five seconds of it

Then the first thing to note is that we have two filter chains! They are separated with a ”;”.

Our first filter graph is this: [0:v]...[train_box]

This requests the first input file as an input
Draws a red box
Saves the output into the “train_box” output pad

Our second filter graph is this: [1:v]...[bbb_box]

This requests the second input file as an input
Draws a red box
Saves the output into the “bbb_box” output pad

And finally, we got two outputs, each mapping to one of the outputs of the filter graph:

-map “[train_box]” filter_complex4_train.jpg
-map “[bbb_box]” filter_complex4_bbb.mp4

Here is the same thing visually:

If you are thinking about making it even more complex and making filter graphs that combine multiple inputs into one for example, you are on the right track! It is possible, and we will get to that!

This was the introduction to the filtering system and its syntax.

Editing video

Now let’s get to know a few filters and make some interesting stuff!

Resizing or scaling

The scale filter is a simple one, yet it is quite powerful!

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "scale=width=600:height=-1:force_original_aspect_ratio=decrease" \
filter_complex5_scaled1.mp4

The arguments speak for themselves, but a few things:

Specifying -1 to either width or height means rescaling while keeping the aspect ratio.
“force_original_aspect_ratio” can be increase, decrease. Meaning it will increase or decrease the image to fit the specified bounding box while keeping the aspect ratio.

Adding text

We have already covered this a little, so let’s dive deeper!

This is what we used earlier:

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex1.mp4

Now let’s discover how to align the text!

Many filters, including drawtext, support variables in some of its argument’s values. If you scroll down in the documentation of drawtext, you’ll find this:

“The parameters for x and y are expressions containing the following constants and functions: ”

And after this part, you’ll see many variables which you can include in your x and y variables!

Let’s see:

# Align the text to the center
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=h/2-text_h/2:x=w/2-text_w/2:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex6_center.mp4
# y=h/2-text_h/2 means: y position = (image height / 2) - (text height / 2)

# Align the text to the right:
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=30:x=w-text_w-20:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex6_right.mp4
# x=w-text_w-20 means: x position = image width - text width - 20pixel padding


# Align the text to the bottom:
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=h-text_h-20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex6_bottom.mp4
# y=h-text_h-20 means: y position = image height - text height - 20pixel padding

And this is what we’ll get in the end:

I need to mention one good trick that might not be obvious at first. So the text_h variable is a tricky one, because different text will be of different height! E.g.: ”____” and “WWW” will result in a different height.

For this reason, you do not always want to use text_h or even just a constant y=value expression but rather, you need to align text by its baseline. So just remember to use the “ascent” variable whenever you need to align text vertically!

Check out these two examples! Each has two drawtext filters printing ”_” and “_H”:

# This one uses y=200 for both, still the text isn't aligned properly!
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='_':y=200:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf,drawtext=text='_H':y=200:x=500:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex7_bad_text.mp4

# This one uses y=200-ascent for both and the text is aligned as expected!
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='_':y=200-ascent:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf,drawtext=text='_H':y=200-ascent:x=500:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex7_good_text.mp4

Now let’s compare the difference:

See? This is the difference between aligning the “top left” or the “baseline” of the text!

Adding an overlay

Overlaying is a very interesting thing to do with FFmpeg. Let’s jump right in!

Basic

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "overlay" \
filter_complex8_overlay1.mp4

Easy as that!

Of course, the overlay filter has a ton of options, but I wanted to demonstrate the easiest possible command line. We don’t even need to mess with input/output pads, as FFmpeg automatically understands the situation: two inputs for the overlay filter and its single output into a single output.

But just to exercise, we could have executed it like this:

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "[0:v][1:v]overlay[output]" \
-map "[output]" filter_complex8_overlay2.mp4

And this would result in the same output! Check it out, now I have specified the two inputs for the overlay: [0:v][1:v]!

Aligned

Let’s align the smiley into the center!

As we have seen with the drawtext, the overlay filter’s arguments also support a few dynamic variables. We’ll use those to achieve what we want!

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "overlay=x=main_w/2-overlay_w/2:y=main_h/2-overlay_h/2" \
filter_complex8_overlay3.mp4

Preprocessing the input for overlay

Let’s get a bit creative!

I want to make it smaller, and I also want to blur it!

Now pause for a minute, and think about it, how you’d do that?!

…

Ready?

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "[1:v]scale=w=200:h=-1,gblur=sigma=3[smiley] ; [0:v][smiley]overlay=x=100:y=100" \
filter_complex8_overlay4.mp4

For this we needed to have two filter graphs!

The first one is this: [1:v]scale=w=200:h=-1,gblur=sigma=3[smiley]

Scales the input image (the smiley).
Then the scaled output is also blurred.
Then the output is saved into the output pad named “smiley”.

Then, we have our second filter graph: [0:v][smiley]overlay=x=100:y=100

This takes as input the first input file (the video).
This also takes as input the output pad named “smiley”. (We are connecting two chains this time!)
Then the overlay filter does its overlaying thing, and we trust FFmpeg to pair the unnamed output with the single output file we specified.

Reusing content

Let’s do one more, a really complicated one!

Let’s have the outro overlaid over the intro!

ffmpeg -y \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-t 5 -ss 00:09:40 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-filter_complex " [1:v]scale=w=1920/2:h=-1[outro]; [0:v][outro]overlay" \
filter_complex8_overlay5.mp4

We could have achieved it in several ways, e.g. we could use the trim filter, but to keep it easy, we just open the same file twice and seek/trim them.

-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4: Open the video, and keep the first five seconds of it.
-t 5 -ss 00:09:40 -i bbb_sunflower_1080p_60fps_normal.mp4: Open the same video again, but seek to the end and keep five seconds from there.

Then we have two filter graphs again, one scales down the outro, and the second is just an overlay.

Are you excited?:) I hope these made-up examples opened up your eye for the possibilities, and I hope you’ll create very creative stuff with this knowledge!

Chroma keying, green screen, blue screen

In this section, we’ll use chroma keying to remove the background from Big Buck Bunny’s intro, and then we will put the transparent logo over the original video, as if it would be some kind of a logo overlay!

ffmpeg -y \
-ss 0.5 -t 2 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-ss 10 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-filter_complex " [0:v]chromakey=color=0xfdfdfd:similarity=0.1:blend=0.2 , scale=w=-1:h=300 , loop=loop=-1:start=0:size=120[intro] ; [1:v][intro]overlay=x=-40:y=-40" \
-t 10 filter_complex9.mp4

So just to recap, Big Buck Bunny’s first few seconds are like this:

And this is the result:

Also, the butterfly moves its wings repeatedly!

Let’s examine the command!

-ss 0.5 -t 2 -i bbb_sunflower_1080p_60fps_normal.mp4: We read in the intro from 0.5 to 2.5 seconds.
-ss 10 -i bbb_sunflower_1080p_60fps_normal.mp4: We read in the video, starting from the 10th second.

Then we have two filter graphs, the first being this:

[0:v]chromakey=color=0xfdfdfd:similarity=0.1:blend=0.2 , scale=w=-1:h=300 , loop=loop=-1:start=0:size=120[intro]

As we see, we have three filters in here!

chromakey: This one takes a color and a few parameters as input, and outputs transparent frames. The specified color + the blended areas will be the transparent sections. In our case we replaced the white-ish (#fdfdfd) background color with transparency.
scale: We resize the full 1080p image into something around 300px high.
loop: With the loop filter, we repeat all the 2 seconds worth of 120 frames (60*2) over and over again, to have the butterfly move its wings continuously.

And then, finally we have the second filter graph:

[1:v][intro]overlay=x=-40:y=-40

Nothing fancy, just an overlay of the original video and our chrome keyed intro.

What else?

You might want to check out a few more filters, that I didn’t cover here.

Here are just a few interesting ones:

Audio manipulation

In this chapter, we’ll be going to check out some audio manipulation techniques with FFmpeg!

First of all, let’s see our example file:

It is a voice recording, and it is intentionally… well, quite bad.

From the waveform, it is obvious that there are very different volume ranges in it. This is an example recording where each sentence was read in different strengths: “normal”, “whisper” or “powerful”, this is why you see repeating patterns of amplitude ranges on the image.

It isn’t visible, but it has some noise too, and of course, it is not normalized or enhanced in any way. Yet.

Please note that there are different scenarios, requirements, and ways to enhance audio. This is a simplified method to show the outline of the process in this article. I’m not an audio engineer, although I have some experience in the area. So if you know it better, feel free to fine-tune it for yourself even more, or contact me and recommend improvements!

I’m showing an example here with a very rough input, one that you’d just reject in real life as it would be useless due to its quality. But it is an excellent example to show the different steps of the enhancing process and to see what can be done to it!

The following steps are built upon each other, and we’ll reach the complete command at the end!

Don’t forget that these settings are specific to this voice recording. Sadly this can not be generalized too much.

Gate

Let’s start with the gate filter!

A gate is like a switch that opens only if the signal is stronger than the threshold. So if the signal level is lower than the threshold, it cuts to complete silence. Although you might soften or delay this cut with the knee, attack, and release arguments.

We’ll use this filter as a basic noise reduction method now! This helps us remove the noise between words and sentences by cutting it to silence. It doesn’t remove noise in any other way, e.g. it doesn’t touch the static on the voice itself.

Check this out!

ffmpeg -y \
-i voice_recording.wav \
-filter_complex "agate=threshold=0.01:attack=80:release=840:makeup=1:ratio=3:knee=8" \
gate.wav

Let’s hear it: gate.wav

And let’s see it:

As you can see, the “silent” parts were attenuated heavily, while the above-the-threshold parts remained similar. Those parts were still affected by the knee, attack, and release arguments determining how hard (knee) and quick (attack/release) the cut is.

I’ve left a quite high release timeout here to avoid sudden dips in the amplitude.

This is where we are right now:

The silent parts are more silent than before, but still, the amplitude range or the dynamic range is quite high. You must change your volume levels to hear everything and void blowing your speakers/brain out.

Equalization

Before fixing that, let’s do a bit more housekeeping. Let’s do some equalization and frequency filtering!

We’ll use these filters:

ffmpeg -y \
-i gate.wav  \
-filter_complex "highpass=f=100:width_type=q:width=0.5 , lowpass=f=10000 , anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1" \
gate_eq.wav

Let’s hear it: gate_eq.wav

This command gradually attenuates frequencies below 100hz, as there are not much valuable content in there, but it can really lower the clarity of the speech.

Then we do the same, but for frequencies above 10 kHz. This is mostly needed because we have a lot of high-frequency noise, so this is a workaround for those. Also, a male voice is generally deeper than a woman’s, so you might want to pay attention to how low you can put the bar.

Then comes anequalizer, which has a crazy an exceptional way of setting its arguments:

This: anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1 means:

at 250hz with a width of 100hz boost by 2 db, with Chebyshev type 1 filter on channel 0.
at 700hz with a width of 500hz attenuate by 5 db, with Chebyshev type 1 filter on channel 0.
at 2000hz with a width of 1000hz attenuate by 2 db, with Chebyshev type 1 filter on channel 0.

I agree. You might have used a friendlier equalizer in your life than this one:)

Those values are based on experimentation and common recommendations for voice. Feel free to tune it for your own needs!

Let’s compare the frequency plots before and after:

Tip: To see the frequency plot in Audacity, open a file, select all, and choose Analyze → Plot spectrum!

Compression

The compressor filter applies dynamic range compression on the incoming audio data. To simplify this, the compressor varies the attenuation based on the incoming signal level. Basically, when you watch a badly mastered movie, this is what you are doing. When it is way too loud in some action scene, you reach for the remote control or mouse to lower the volume, but in the next moment, you will not hear what your heroes are saying, so you increase it back again.

Dynamic range compression roughly does the same. You may set it up in a way so that it would attenuate louder parts, therefore keeping the overall volume range relatively small.

It often happens that performers on the stage use a high dynamic range. Many performers will shout at one moment and then whisper in the next to increase drama or keep the attention. If you want to avoid manually adjusting the volume in real-time (while blowing off your speakers and pulling your hair out), then a compressor will save you in these situations!

This is why our example audio consists of different speaking strengths, so that we could see the dramatic effect of this filter.

ffmpeg -y \
-i gate_eq.wav \
-filter_complex "acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6" \
gate_eq_comp.wav

Let’s hear it: gate_eq_comp.wav

And let’s compare the result of this with the original waveform!

Original:

Result:

Quite dramatic, isn’t it?:)

Let’s analyze this: acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6

First, level_in=6 sets the input gain. It is 1 by default, but since our example, audio is extremely silent at places, we boost up the whole thing before processing.

Then threshold=0.025 defines that everything above 0.025 should be attenuated.

Based on the image below, I’ve decided to cut at this point, as this is above most of the whispering, which cuts hard pops and “s”-es even in the “whisper zone”.

Then ratio=20 means 1:20 in attenuation ratio, which means that if the level rises 20 dB above the threshold, it will be only 1 dB above the line after the attenuation. Basically, this is a very strong compression ratio, it is almost a limiter.

This far, we boosted the signal, then turned down everything that was above our “whisper line” with a quite strong ratio, and now, everything is basically at the whisper level, even the parts that are shouting.

Finally, with the makeup=6 we just bring back everything to the level where the “normal” parts were before.

Let’s take a look back now, to understand why we used the gate and did the equalization before the compressor.

Generally, you want to remove unneeded parts and frequencies before compression, as the compressor will likely increase those too! So by removing most of the noise in the gaps, we avoided level_in=6 to increase them too! And the same goes for the high- and lowpass filtering.

Changing the volume

Now, if we want to make the result a bit louder, we could increase the previous step’s makeup argument, or leverage the volume filter.

While we are at it, let’s cut the first 4 seconds too with -ss 4.

ffmpeg -y \
-ss 4 -i gate_eq_comp.wav \
-filter_complex "volume=1.1" \
gate_eq_volume_comp.wav

Let’s hear it: gate_eq_volume_comp.wav

Let’s make audio gate again

Excuse me for that title:)

So as I’ve described earlier, compression can amplify the noises, so you might want to run the result through a gate again:

ffmpeg -y \
-i gate_eq_volume_comp.wav \
-filter_complex "agate=threshold=0.1:attack=50:release=50:ratio=1.5:knee=4" \
gate_eq_volume_comp_gate.wav

Let’s hear it: gate_eq_volume_comp_gate.wav

In this case, I’ve used a softer gate, with ratio=1.5. Because of this, I could use shorter attack and release delays too, as the attenuation is not that strong, it isn’t causing hard dips in the audio.

Putting it all together

Just a single command could have achieved all the steps above:

ffmpeg -y \
-i voice_recording.wav \
-filter_complex "agate=threshold=0.01:attack=80:release=840:makeup=1:ratio=3:knee=8 , highpass=f=100:width_type=q:width=0.5 , lowpass=f=10000 , anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1 , acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6 , volume=1.1 , agate=threshold=0.1:attack=50:release=50:ratio=1.5:knee=4" \
gate_eq_volume_comp_gate_together.wav

I just copy-pasted all the filters right after each other with a comma between them.

Isn’t it beautiful? Yeah, it isn’t, but it is very practical:)

For the last time, check out the difference:

Original: voice_recording.wav
Final: gate_eq_volume_comp_gate.wav

It has less noise, more clear voice, and a small volume range. Therefore it is easy on your ears!

What else?

You might want to check out a few more filters that I didn’t cover here.

Here are just a few interesting ones:

Documentation

For your convenience, let me list the most important documentations that might be important for you! Most of these were already linked many times in this article.

If you got this far from top to bottom, then you are a true hero! I hope you enjoyed this, and I also hope that it inspired you to create something awesome with FFmpeg! Please consider donating to FFmpeg – they are fantastic.

If you’re looking to take your creative projects to the next level, check out our products - Creative Editor SDK, Video Editor SDK, and Photo Editor SDK. These versatile tools empower you to bring your vision to life, whether you’re editing images, crafting stunning videos, or unleashing your artistic talents.

Thanks for reading! Let us know what you think on Twitter! To stay in the loop, subscribe to our Newsletter.

How to Apply Filters and Effects to an Image in Flutter

Natalia — Tue, 15 Nov 2022 10:51:25 GMT

Introduction

Having previously discussed how Flutter deals with resizing and various overlays, in this tutorial, we will talk about a more popular type of image manipulation; applying filters and effects. At least since the ubiquity of social media platforms such as Instagram, Snapchat or more recently TikTok, applying various filters and effects has become an expected feature of any application dealing with image manipulation.

Applying Filters and Effects in Flutter demo application

In this tutorial, we go through the process of applying filters and effects to images in Flutter. Like the example given in the video, we will examine the options for developing an image editor and the effects available in the Flutter framework. The Git repository supporting this article can be cloned with the command provided below:

git clone git@github.com:nataliakzm/Applying_Filters_and_Effects_to_Images_Flutter.git

One of the leading solutions proposed by the Flutter team on this issue was the ImageFiltered class. Overall, the widget deals with major image manipulations (from blurring to matrix transformation, rotation etc.), transforming and rearranging image pixels. To create an ImageFilter, you need to follow a particular structure and apply the ImageFilter class to its child elements:

Moreover, in order to start working with ImageFiltered within your Flutter application, first, don’t forget to import this library on the top of your .dart file.

const ImageFiltered({
  super.key,
  required this.imageFilter,
  super.child,
  this.enabled = true,
}) : assert(imageFilter != null);

import 'dart:ui';

Then, ImageFiltered offers several filter choices for image processing, depending on the specification of your app or visual asset.

So, for example, with the help of ImageFilter.blur or the so-called Gaussian blur, the pixels of the image can be blurred, which can, for instance, be used to create a graphic element that can serve as a background image.
On the other hand, two other filters such as ImageFilter.dilate and ImageFilter.erode affect the pixels’ value by either enlarging them to the max significance within the specified range along the x and y axes or by shrinking pixels to their minimum values;
You can also easily rotate, scale and change your asset in other ways by transforming its matrix with ImageFilter.matrix;
Finally, ImageFilter.compose is handy for merging two filters and combining their effects.

At this point, the purpose of each effect seems much clearer, but we still miss some crucial steps to implement these filters in your Flutter application. Let’s see some real examples:

ImageFilter.blur constructor

To apply a blurring effect on your image, you have to call an ****ImageFiltered.blur(), and adjust the values of sigmaX and sigmaY arguments for blurring the image. Note that the asset must be called as a child element of the filter to achieve the desired effect.

Container(
		child: Center(
				child: Column(
						mainAxisAlignment: MainAxisAlignment.center,
						children: [
      // Blur an Image
      ImageFiltered(
					imageFilter: ImageFilter.blur(sigmaY: 5, sigmaX: 5),
					child: Image.asset("assets/logo.png"))

ImageFilter.matrix constructor

The situation is slightly different with ImageFilter.matrix filter since the effect can be initialized by one of two methods – Float64List.fromList or Matrix4.rotationZ . Thus, for the first method, the user needs to directly provide the matrix as a Float64List as it is done below:

// Matrix Transformation
ImageFiltered(
		imageFilter: ImageFilter.matrix(Float64List.fromList([
			1, 0.5, 0.0, 0.0,
			0.0, 1, 0.0, 0.0,
			0.0, 0.0, 1.0, 0.0,
			0.0, 0.0, 0.0, 1.0,
    ])),
		child: Image.asset("assets/logo.png", scale: 3.5)) // Note that scale is optional and depends on the image size

While the Matrix4 class simply provides rotation in Z directions:

// Matrix4 Rotation
ImageFiltered(
		imageFilter: ImageFilter.matrix(Matrix4.rotationZ(0.2).storage),
		child: Image.asset("assets/logo.png", scale: 6)) //Note that scale is optional and depends on the image size

ColorFiltered Effects

Alternatively, Flutter has several options to integrate some filters without calling any extra library. One such example is the ColorFilter function executed within the widget of the same name. ColorFiltered processes two colors and outputs only one of them, which is eventually displayed at the top of the layer.

 // Black & White Effect
ColorFiltered(
		colorFilter: const ColorFilter.mode(Colors.grey, BlendMode.saturation),
		child: Image.asset("assets/logo.png"))

Users can also engage with several construction methods: from directly calling the sRGB gamma curve with ColorFilter.linearToSrgbGamma() widget or its inverse prototype with ColorFilter.srgbToLinearGamma() to transforming the color by a 5x5 matrix within ColorFilter.matrix() function or simply applying the blend mode with a given color specified in ColorFilter.mode().

Final Thoughts

In conclusion, Flutter’s image processing capabilities are limited by a small variety of effects and often depend on the developer’s ability to handle complex concepts like matrix or sRGB curves. In this article, we reviewed some of the main image processing techniques proposed by the Flutter team and examined how to integrate them into photo editing apps. However, it is not always possible to allocate several days to calculate the correct matrix for transforming just one image when it comes to processing a large number of graphic assets. In such cases, you may want to think about using the PhotoEditor SDK for your next Flutter app. Just like in the interactive example below, PhotoEditor SDK has many different effects and filters that will suit any of your requests.

Applying Filters via PhotoEditor SDK

To learn more, check out our article on PhotoEditor SDK and how to set it up in your app and official PE.SDK guides. You can also follow our guide on how to integrate video editor for Flutter into your app. For any of your questions that might occur in the process, you can contact our Support, who will be glad to help you.

How to Draw on Images in React Native

Natalia — Thu, 20 Oct 2022 13:09:38 GMT

It has become a commonplace requirement for many applications to manipulate images in one way or another before users are ready to share them. Therefore, it is crucial to foresee end-user needs while working on an app involving any interaction with graphical content. Image annotation can be necessary in various cases, such as simple inspection apps, bug reporting software, or machine learning applications relying on manual user input.

In this article, we are shedding light on how to draw on images in React Native with the help of a simple HTML5

Enable the characteristics of the
Set the thickness and color of the brush instrument;
Define at what moment your brush needs to be activated and when the drawing process will finish;
Start drawing!

Let’s first clear up the purpose of

Canvas Integration for Images

<canvas> is an HTML component commonly used for creating and manipulating graphics, animation, and other visualization features. This element provides JavaScript APIs enabling image overlays and handling user input to make drawing possible. Moreover, while we have several methods for image manipulation with

<! -- HTML example of using canvas: -->
<canvas id="myCanvas" width="200" height="100"></canvas>

You might wonder why you cannot easily substitute <canvas> with some lines of HTML and CSS in your code. The answer lies on the surface: <canvas> creates a single flattened graphic rather than multiple components lying on top of each other (i.e., typically the output of HTML and CSS).

Now, let’s see how to implement this powerful element in your React Native application:

Drawing with Canvas

import React, { useEffect } from 'react';

// CanvasContext here is used for drawing onto the canvas
import { useCanvas } from './CanvasContext';

export function Canvas() {
  const { canvasRef, prepareCanvas, startDrawing, finishDrawing, draw } =
    useCanvas();

  useEffect(() => {
    prepareCanvas();
  }, []);

  return (
    <canvas
      onMouseDown={startDrawing}
      onMouseUp={finishDrawing}
      onMouseMove={draw}
      ref={canvasRef}
    />
  );
}

We start by enabling the characteristics of our <canvas> container, which are meant to render the CanvasContext component presented below.

The size of our future canvas is determined with prepareCanvas, which includes both the height (canvas.height) and width (canvas.width) of the HTML element. For the drawing process itself, it is necessary to assign not only the thickness and color of the brush instrument (context) but also take into account its activation (startDrawing with onMouseDown) and how the user can finish a stroke (finishDrawing with onMouseUp).

import React, { useContext, useRef, useState } from 'react';

// enabling drawing on the blank canvas
const CanvasContext = React.createContext();

export const CanvasProvider = ({ children }) => {
  const [isDrawing, setIsDrawing] = useState(false);
  const canvasRef = useRef(null);
  const contextRef = useRef(null);

  // defining width & height of the canvas
  const prepareCanvas = () => {
    const canvas = canvasRef.current;
    canvas.width = window.innerWidth * 2;
    canvas.height = window.innerHeight * 2;
    canvas.style.width = `${window.innerWidth}px`;
    canvas.style.height = `${window.innerHeight}px`;

    // defining the thickness and colour of our brush
    const context = canvas.getContext('2d');
    context.scale(2, 2);
    context.lineCap = 'round';
    context.strokeStyle = 'black';
    context.lineWidth = 5;
    contextRef.current = context;
  };

  const startDrawing = ({ nativeEvent }) => {
    const { offsetX, offsetY } = nativeEvent;
    contextRef.current.beginPath();
    contextRef.current.moveTo(offsetX, offsetY);
    setIsDrawing(true);
  };

  const finishDrawing = () => {
    contextRef.current.closePath();
    setIsDrawing(false);
  };

  const draw = ({ nativeEvent }) => {
    if (!isDrawing) {
      return;
    }
    const { offsetX, offsetY } = nativeEvent;
    contextRef.current.lineTo(offsetX, offsetY);
    contextRef.current.stroke();
  };

  // Once the canvas is cleared it return to the default colour
  const clearCanvas = () => {
    const canvas = canvasRef.current;
    const context = canvas.getContext('2d');
    context.fillStyle = 'white';
    context.fillRect(0, 0, canvas.width, canvas.height);
  };

  return (
    <CanvasContext.Provider
      value={{
        canvasRef,
        contextRef,
        prepareCanvas,
        startDrawing,
        finishDrawing,
        clearCanvas,
        draw,
      }}
    >
      {children}
    </CanvasContext.Provider>
  );
};

export const useCanvas = () => useContext(CanvasContext);

Drawbacks of the Canvas Method

Despite the canvas component offering powerful features, it is also quite demanding to use in its original form. For example, it may be problematic to adjust the properties of the brush to ensure that you can draw with smooth movements. Thus, implementing any solution for this issue would further increase the coding time and reduce the compatibility between different frameworks. Also, separately defining canvas and brush properties overcomplicates the code and requires advanced sources for successful integration into your React Native app. Although some temporary solutions aim to facilitate this process (such as react-native-image-draw or react-native-sketch-draw), none provides a permanent solution with clear documentation and guidance.

If you wonder about a magic door to avoid this coding nightmare, you can always choose a commercial, all-in-one solution like PhotoEditor SDK. Implementing this SDK only takes a few lines of code, saves time and expenses in building an app, and you can expect great support from the developers. This way, you can skirt the hassle of coding an editor and instead focus on creating a great product.

PhotoEditor SDK Integration for Drawing on Your Images

To get started integrating the PhotoEditor SDK in your React Native app, refer to this guide from the official documentation. You can then ensure comfortable image interaction for your users with the optimized brush tool, as in the example below.

Conclusion

This article aimed to examine image editing instruments suitable for React Native framework. The HTML <canvas> element could have big potential as it provides fundamental tools for creating graphics or animation. However, its application is time-consuming and requires comprehensive knowledge in the field. Therefore, if you want to guarantee a smooth and easy integration of image drawing tools for your app, consider using advanced software like PhotoEditor SDK.

How To Build a Video Player in JavaScript

Paul — Tue, 27 Sep 2022 17:02:49 GMT

Probably a decade ago, it was impossible to play video or audio inside your browser without any third-party services such as Flash or Silverlight. You needed to install a plugin and only play your media while using it, so as you can see, it was very uncomfortable, with low speed and high delays. Nowadays, we have JavaScript with the new version of HTML5. With these new technologies and tools, we can stream our video much quicker, easier, and without any latency. To do it, you will need only a simple tag and give a link to your video stored on your computer. The simple attribute controls will give you a default video player built into the browser. It’s elementary and doesn’t have many features, so if you want to stream a video on your website in a more professional way using your video player, you’ll need to use JavaScript. And we’ll teach you how to do it in this article!

By the end of this guide, you’ll have something similar to this, so if you’re excited, keep reading and follow this tutorial step-by-step!

See the Pen How to build a video player in Javascript by Paul Knulst (@paulknulst) on CodePen.

Setting Up the Project

Assuming you are working with UNIX system (or have Git BASH on Windows) you can create all three files that are necessary to build a video player in JavaScript with this command:

mkdir video-player
cd video-player
touch index.html script.js style.css

To add a simple video player to our application, we have to add the following code to our index.html:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title>How to build a video player in Javascript</title>
        <link rel="stylesheet" href="style.css" />
    </head>
    <body>
        <div class="player">
            <video class="video" controls>
                <source
                    src="<https://ftp.f1nalboss.de/data/imgly/videoplayer/testvideo.mp4>"
                    type="video/mp4"
                />
                <source
                    src="<https://ftp.f1nalboss.de/data/imgly/videoplayer/testvideo.mp4>"
                    type="video/webm"
                />
                <p>No HTML5 video supported</p>
            </video>
        </div>
        <script src="script.js"></script>
    </body>
</html>

Within the above code, the <video> element uses a remote video from my FTP. You can either use my default video or add any video from your local computer by adjusting the src attribute. HTML5 specification supports three different video formats, and the snippet used multiple <source> tags to make the videos available in MP4 and WebM. Furthermore, the <p> tag is used to display pre-defined content to user agents that do not support the video element.

The HTML5 <video> tag accepts several native attributes. For example, the controls attribute displays the standard player controls when added or set to true. You can find out more about all video attributes here.

Before continuing, you should apply all styles that are needed within this tutorial by populating your style.css with all styles from this CodePen. Save and open your index.html and load it within the browser to see the embedded video player as seen below:

Customize the Video Player With JavaScript

To customize the video player, you first have to remove the controls attribute that displays Play, Pause, Volume, etc. because you will implement your own custom controls within this tutorial. Now, check your browser, you will recognize that the controls are gone, and you cannot play the video anymore.

Add Play and Pause

To enable play and pause the video, you have to add a new button to the index.html:

<div class="controls">
    <button
            class="controls__btn playPauseBtn"
            title="Toggle Play"
            >
        ►
    </button>
</div>

Afterward, open your script.js and enable functionality by adding this code:

const videoContainer = document.querySelector(".video-container");
const playPauseBtn = document.querySelector(".playPauseBtn");
function togglePlay() {
  if (videoContainer.paused || videoContainer.ended) {
    videoContainer.play();
  } else {
    videoContainer.pause();
  }
}
function updatePlayBtn() {
  playPauseBtn.innerHTML = videoContainer.paused ? "►" : "❚❚";
}
playPauseBtn.addEventListener("click", togglePlay);
videoContainer.addEventListener("click", togglePlay);
videoContainer.addEventListener("play", updatePlayBtn);
videoContainer.addEventListener("pause", updatePlayBtn);

Within this javascript code, first, the video-container element and the playPauseBtn is selected (Line 1 and 2). Then two functions are defined: togglePlay() and updatePlayBtn(). togglePlay() is used to stop and start the video based on its actual state. updatePlayBtn is used to switch between the Icon which is shown within the video player.

In the last part of the snippet, a click event listener is added to the playPauseBtn that executes the togglePlay() function. Next, three click event listeners are added to the videoContainer that executes togglePlay() on mouse click and also executes updatePlayBtn based on the video’s state.

Now you can reload your index.html and should be able to play and pause the video by either clicking the video or the button:

Add Progress Bar

Next, a progress bar will be implemented to show the current timestamp of the video when played. First, add a div tag to the index.html which will act as the progress bar:

<div class="controls">
    <div class="progress">
    	<div class="progress__filled"></div>
    </div>
    // ..
</div>

Then open the script.js and add the following snippet:

const progress = document.querySelector(".progress");
const progressBar = document.querySelector(".progress__filled");

function handleProgress() {
  const progressPercentage = (videoContainer.currentTime / videoContainer.duration) * 100;
  progressBar.style.flexBasis = `${progressPercentage}%`;
}

function jump(e) {
  const position = (e.offsetX / progress.offsetWidth) * videoContainer.duration;
  videoContainer.currentTime = position;
}

videoContainer.addEventListener("timeupdate", handleProgress);
progress.addEventListener("click", jump);
let mousedown = false;
progress.addEventListener("mousedown", () => (mousedown = true));
progress.addEventListener("mousemove", (e) => mousedown && jump(e));
progress.addEventListener("mouseup", () => (mousedown = false));

In this snippet, the progress container and the progress__filled element will be selected, and two functions will be added: handleProgress() and jump(e). handleProgress() will be responsible for updating the progress bar. The jump(e) function is used to enable clicking on the progress bar to jump to the position within the video.

The last part contains all event listeners that are needed for the progress bar. The handleProgress() will be called on every timeupdate event. Also, clicking anywhere on the progress bar will call the jump(e) method and the video will jump to the position. Additionally, mousedown, mousemove, and mouseup will be used to enable sliding through the video while holding the mouse button down on the progress bar.

Closing Notes

Congratulations! If you followed the tutorial, you learned how to implement your own video player and add custom controls using JavaScript. Now, you can use my CodePen and start implementing more controls like volume control, keyboard shortcuts, or skip controls to build your own customized video player.

If your app goes beyond merely displaying video, and you want to allow your users to also edit video or create video based templates in the browser, check out our Video Editor for Web!

How To Resize Images in Flutter

Natalia — Tue, 27 Sep 2022 15:03:53 GMT

Static images are a core part of mobile applications. Usually, you store them in the directory in their original sizes, which requires you to adjust the sizes now and then, depending on where they are displayed. This article discusses how to resize images in Flutter and adjust their width, height, and size with efficient lines of code.

Resizing Methods

For image editing tools, Flutter offers an excellent and easy-in-use Boxfit property that works within the fit parameter, and you can easily integrate it into your application.

child: Image.asset( 'images/pexels.jpg', fit: BoxFit.cover)

Depending on your needs, you can choose between multiple attributes. For example, .cover and .fitHeight properties are similar when called, and both result in maximum frame coverage. Yet, although the image is widened proportionally, these methods can significantly affect the quality of crop image borders if they overfill. On the contrary, with .fitWidth or .scaledown, the asset is resized to the container’s width boundaries, which could be helpful when multiple images are needed to be displayed simultaneously. If you are dealing with a static image of small size, you can also employ .fill attribute that helps stretch your assets without cropping any critical information.

Alternatively, by calling .scale 0.5 the fit parameter will return you a graphical asset based on the scale you define. Note that any value less than 1 would reduce the image size.

Adjusting Image Size with Flutter

Now, let’s look at how Flutter allows applying these image manipulation techniques. The crucial difference from the other frameworks is that in Flutter, images should be stored in a specific folder. In other words, once you upload graphic files, go to the pubspec.yaml file and add the path to the directory around under assets(Note: by default the dependencies corresponding to ****assetsare usually placed around 45-46th lines in the code and need to be uncommented first by eliminating # and one space).

# Don't forget to define the assets folder in your directory
# To do so, find lines 45 - 46 in the **pubspec.yaml** file
# and uncomment the assets section like this:
assets:
  - images/ #Note that this should correspond to the name of your folder

Alternatively, you can specify the path to each image in the list:

assets:
  - assets/images/your_image.jpg
  - assets/images/your_image2.jpg

When the image is uploaded to Flutter, it seeks to occupy as much size as possible. But let’s presume that we have a “frame” or container that differs in size from the visual asset. One way to see the differences in sizes of both canvases is to color one frame, like color: Colors.indigo. Then we provide Flutter with the size specification and render our image in a child node child: Image.asset(‘images/pexels.jpg’). Thus, now we can see the image is placed inside the frame, and we need only to assign one of the filling methods (for example, fit: BoxFit.cover) discussed above.

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';

void main() {
  runApp(const ImgApp());
}

class ImgApp extends StatelessWidget {
  const ImgApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      debugShowCheckedModeBanner: false,
      home: const ResizePage(),
    );
  }
}

class ResizePage extends StatelessWidget {
  const ResizePage({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text('How to Resize Images'),
          backgroundColor: Color(0xffe55586),
        ),

        body: Center(
            // Enabling the Image Frame
            child: Container(
                color: Colors.indigo, // To see the difference between the image's original size and the frame
                height: 550,
                width: 300,

                // Uploading the Image from Assets
                child: Image.asset(
                  'images/pexels.jpg',

                  // Resizing the Image to the Frame Size
                  fit: BoxFit.cover,
                ))));
  }
}

You can find the associated to this tutorial Git repository ― here. Otherwise, run the following command to clone the complete code to your system:

git clone <https://github.com/nataliakzm/Resizing_Images_with_Flutter>

Conclusion

This article covered Flutter’s approach to the image resizing problem and saw how to use it in practice. However, despite the diversity of suggested methods that could fulfill basic users’ needs, Flutter fails to provide a comfortable integration. Just imagine you are building an app similar to Instagram in its functionality and you need not only upload various images from different assets but also resize them differently. If you want to avoid spending your time writing and maintaining a ton of code, then you may consider using PhotoEditor SDK in your next project.

PhotoEditor SDK is available for various frameworks: first, read this article from the official documentation to set up dependencies for your Flutter-based project. You can also follow our guide on how to integrate video editor for Flutter into your app.

In case you encounter any difficulties with the installation process, don’t hesitate to contact our support who will be happy to help you. Then, as shown in the example below, you can efficiently resize, crop, rotate or flip your visual assets with the transform tool.

How to Crop and Trim Videos in Flutter

Michael H. — Tue, 27 Sep 2022 08:45:16 GMT

If you are looking for a package that crops and trims videos in Flutter, you must have already come across the video_trimmer Flutter package. This package can trim videos but does not provide video cropping (at least not out-of-the-box). In fact, none of the packages on pub.dev, as of today, allow cropping a video in Flutter. If you have dug deeper, you might have come across FFmpeg — a powerful video editing command line tool, that is not the easiest to get started with. See the Ultimate Guide to FFmpeg for help.In this article we will use the FFmpeg library to crop and trim a video in a Flutter project.

Here is a list of packages that we will be using; ffmpeg_kit_flutter package for cropping and trimming videos, the path_provider package to get the path to the application or external directory where the video files will be stored, and the video_player package to play the video preview.

This article will not cover building any UI for cropping and trimming. But, it will discuss in brief how to crop the video using the video_trimmer library as well. If you want to package this workflow for production, check out our guide on running FFmpeg inside a Docker container. Also, you can try out the code used in this article from this GitHub repository.

Get Started

The end result of this tutorial will be a simple app that would look like the following screenshot.

When you tap the Save Video, the preview will refresh with the cropped and trimmed video replacing the original video. Follow the instructions in this article, and you will be able to develop a similar Flutter app.

First, let us start by adding dependencies to a new Flutter project.

Add Dependencies

In your project’s pubspec.yaml file add the following 3 dependencies.

dependencies:
  ...
  ffmpeg_kit_flutter: ^4.5.1
  path_provider: ^2.0.11
  video_player: ^2.4.6

Then, execute flutter pub get command from the project folder root.

Set Minimum SDK Version and Platform Version

The ffmpeg_kit_flutter plugin runs on Android SDK API level 24+ and iOS SDK 12.1+. Therefore, modify the module level build.gradle file to declare the minSdkVersion.

android {
  defaultConfig {
    ...
    minSdkVersion 24
    ...
  }
}

Then, modify the Podfile to declare the minimum global platform.

# Uncomment this line to define a global platform for your project
platform :ios, '12.1'

Add a Video Asset

For simplicity, we will be using a video asset instead of implementing a file picker. Choose a video file you would like to work with or download this sample video file. Next, copy the video file to a new directory named assets at the root of your project folder.

And then reference the video file from the pubspec.yaml file.

flutter:
  assets:
    - assets/file1.mp4

Execute flutter pub get again.

Having added the required dependencies, let us move on to coding.

Implement Video Crop and Trim

Replace the code in your main.dart file with the following code in the code block. This will be our starter code.

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Crop and Trim Demo',
      home: const MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({Key? key}) : super(key: key);

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text("Flutter Crop and Trim"),
      ),
      body: Column(
        children: [

        ],
      ),
    );
  }
}

Initialize the Input Player

Add the following code to the _MyHomePageState class.

import 'dart:io';

...
import 'package:flutter/services.dart';
import 'package:path_provider/path_provider.dart';
import 'package:video_player/video_player.dart';

...

class _MyHomePageState extends State<MyHomePage> {
  late String inputPath;
  VideoPlayerController? controller;
  String outputPath = "";

  ...

  @override
  void initState() {
    super.initState();
    copyVideoToApplicationDirectory().then((path) async {
      inputPath = path;
      controller = VideoPlayerController.file(File(inputPath));
      await controller?.initialize();
      await controller?.play();
      setState(() {});
      outputPath = await getOutputPath();
    });
  }

  ///Copy input file to ApplicationStorage Directory
  ///returns path to copied video
  Future<String> copyVideoToApplicationDirectory() async {
    const filename = "file1.mp4";
    var bytes = await rootBundle.load("assets/file1.mp4");
    String dir = (await getApplicationDocumentsDirectory()).path;
    writeToFile(bytes, '$dir/$filename');

    return '$dir/$filename';
  }

  ///Write to Path.
  Future<void> writeToFile(ByteData data, String path) {
    final buffer = data.buffer;
    return File(path).writeAsBytes(
        buffer.asUint8List(data.offsetInBytes, data.lengthInBytes));
  }
}

The copyVideoToApplicationDirectory method is copying the video from Assets to the Application directory on the phone’s file system. The resulting path is then stored in the inputPath variable which is then supplied to the FFmpeg command. For large‑scale workloads, you can move this FFmpeg command to AWS Spot Instances; our AWS guide shows how to configure a cloud environment for scalable video processing.

Next, add the VideoPlayer widget to the Column widget.

body: Column(
  children: [
    (controller != null)
        ? AspectRatio(
        aspectRatio: controller!.value.aspectRatio,
        child: VideoPlayer(controller!))
        : const SizedBox(),
  ],
),

Run the app and the video should start playing.

In the next step, we will add the FFmpeg command, but before that add the following 2 more methods to the _MyHomePageState class.

...

import 'package:ffmpeg_kit_flutter/ffmpeg_kit.dart';
import 'package:ffmpeg_kit_flutter/log.dart';
import 'package:ffmpeg_kit_flutter/return_code.dart';

...

class _MyHomePageState extends State<MyHomePage> {
  ...

  /// Output path with a file name where the result will be stored.
  Future<String> getOutputPath() async {
    final appDirectory = Platform.isAndroid
        ? await getExternalStorageDirectory()
        : await getApplicationDocumentsDirectory();
    final externalPath = '${appDirectory?.path}/out_file.mp4';
    return externalPath;
  }

  ///Executes the FFMPEG [command]
  ///Note: Green bar on the right is a Flutter issue. <https://github.com/flutter/engine/pull/24888>
  ///Should get fixed in a 3.1.0+ stable release <https://github.com/flutter/engine/pull/24888#issuecomment-1212374010>
  Future<void> ffmpegExecute(String command) async {

    final session = await FFmpegKit.execute(command);

    final returnCode = await session.getReturnCode();
    if (ReturnCode.isSuccess(returnCode)) {

      print("Success");
      //Replace the preview video
      await controller?.pause();
      await controller?.dispose();
      controller = VideoPlayerController.file(File(outputPath));
      await controller?.initialize();
      await controller?.play();
      setState(() {});

    } else if (ReturnCode.isCancel(returnCode)) {

      print("Cancel");

    } else {

      print("Error");
      final failStackTrace = await session.getFailStackTrace();
      print(failStackTrace);
      List<Log> logs = await session.getLogs();
      for (var element in logs) {
        print(element.getMessage());
      }

    }
  }
}

The getOutputPath method provides the path where the resulting video will be saved. The outputPath will be passed to the FFmpeg command whereas the ffmpegExecute method is where the FFmpeg magic takes place.

The ffmpegExecute method expects a valid FFmpeg String command. The String command is then passed to the FFmpegKit.execute method which returns an instance of FFMpegSession . It will tell us whether our command was executed successfully or not. If it was not executed successfully, we can extract error logs from it.

FFmpeg Execute Command

Add a TextButton from where the FFmpeg command shall be sent.

TextButton(
  child: Text('Save Video'),
  onPressed: () async {
    //TODO: Call FFMPEG Execute
  },
),

Next, call the following ffmpegExecute method from the onPressed property.

ffmpegExecute('-ss 0:00:15 -to 0:00:45 -y -i $inputPath -filter:v "crop=320:150" -c:a copy $outputPath');

Run the app and tap the Save Video button. You will notice that the player is now playing the new cropped and trimmed video.

If you were able to successfully follow the instructions up till here, it is time to dig a little deeper into the FFmpeg command that we just ran earlier.

Understanding the FFmpeg Command

Because command FFmpeg is a command line tool, it expects string-only commands. For this reason, we do not have any dart Classes, Methods, or Parameters to work with that we usually get when working with a dart plugin. Let us dissect the above command line command that is helping us crop and trim a video.

Command	Description
-ss 0:00:15	Seeks to position on the input video. The trim starts from this position.
-to 0:00:45	Stops reading at the position in the input video. The trim stops at this position. The total length of the video is 0:02:05.
-y	Overwrite output files. Helpful when the Save Video button is tapped more than once.
-i $inputPath	Input file location. This is the file on which the crop and trim are applied.
-filter:v	Apply a filter to the video stream.
”crop=320:150”	Apply a crop filter from the center of the video that is 320 pixels wide and 150 pixels tall. The original dimensions of the video were 320 x 240.
-c:a copy $outputPath	Specifies the codec with which the output file must be encoded. Here, copy is a special value to indicate the stream is not to be re-encoded. The a after the colon is a stream specifier for the audio stream.

That will be either the application directory or the external directory of the application, depending on whether the app is running on iOS or Android.

You can check out the app’s code and play with it by downloading it from this GitHub Repository. This project allows you to tweak the duration of the trim and dimensions of the crop, so it has more code, but at the heart of it, it still uses the above FFmpeg command.

Cropping Using the video_trimmer Plugin

As a bonus, here is a tip on how to use the video_trimmer plugin for cropping videos.

The video_trimmer plugin also uses FFmpeg at its core and allows us to pass on an FFmpeg command while saving the video. For this reason, it is an easy task for us to apply the crop to a video using the video_trimmer plugin.

To do this, pass the following command to the saveTrimmedVideo method’s ffmpegCommand parameter as shown in the code block below. The video_trimmer package expects the customVideoFormat parameter argument when the ffmpegCommand parameter is used.

await _trimmer
    .saveTrimmedVideo(
        startValue: _startValue,
        endValue: _endValue,
        ffmpegCommand:
            '-filter:v "crop=320:150"',
        customVideoFormat: '.mp4')
    .then((value) {
  setState(() {
    _value = value;
  });
});

Limitations of this Approach

The crop that we are applying is from the center of the frame. The present command would also need starting coordinates to crop a non-center frame.

The project will need a refined UI for cropping and trimming videos to offer a complete app experience to users. The current approach is miles away from creating that experience.

But perhaps the most significant limitation is FFmpeg’s licensing. FFmpeg is available with both LGPL and GPL licenses, so you must ensure your project is compatible with those licenses. For many commercial projects, this is a non-starter. Alternatively, you can run FFmpeg on Google Cloud Platform; our GCP tutorial walks you through setup.

Commercial Alternative

VideoEditor SDK (VE.SDK) from IMG.LY provides powerful video editing features, including cropping and trimming videos in a Flutter project. You will receive staples of video editing, including straightening videos, filters, brightness, color adjustments, and more. Follow our guide to learn how to integrate IMG.LY’s video editor for Flutter into your app.

Conclusion

While still a complex topic, video manipulation is more attainable to implement on Flutter than on native Android. Nevertheless, FFmpeg is the only open-source, free-to-use option to edit videos on Flutter right now. To automate cropping and trimming across many files or build a transcoding server, see our article on building a production‑ready batch video processing server.

How To Crop and Trim Videos In Kotlin for Android

Paul — Fri, 02 Sep 2022 10:18:21 GMT

Cropping and trimming videos is a notoriously difficult task to achieve on Android. One way to implement this functionality is by using FFmpeg a free open-source suite of tools that can perform a wide range of tasks, from video converting to editing. Normally FFmpeg is used from the command line, to use it correctly in Android you have to understand its underlying APIs and how to use them.

In this tutorial, you will learn how to crop and trim videos in Android by using FFmpeg. Even if you are a beginner you should be able to follow the steps to achieve the desired results.

I try to summarise the most important basics that you need to know to manipulate videos with FFmpeg. After reading this article you should be able to use it in your own applications. Furthermore, I have developed a sample application and library that can be used to trim and crop videos with an Android device.

What is FFmpeg

FFmpeg is a great multimedia framework that is able to mux, demux, decode, encode, transcode, filter, stream, and play most media content that exists. FFmpeg supports different tools that can be used to develop an application to manipulate any kind of media to the desired output. You do not have any limitations on what to do with multimedia when using FFmpeg.

Unfortunately, to use FFmpeg in an Android App using Kotlin you have to compile and build the libraries to use them as a dependency in your project. This process is not straightforward because you have to manually compile a C/C++ library with help of the Android NDK.

Luckily, many people already have done this and we are able to use their compiled library in our project. However, if you want to compile FFmpeg from scratch you can do this. All necessary information should be available within the android developer documentation.

To show you how we can use FFmpeg in our app I will use a compiled FFmpeg library that can be found here. To find other FFmpeg libraries you can have a look at the official FFmpeg wiki where several pre-packaged sources are listed.

Setting up the project

In order to use the FFmpeg library of WritingMinds (or any other library), we have to follow some simple steps.

1. Add dependency for the FFmpeg library to your app-level build.gradle

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'com.writingminds:FFmpegAndroid:0.3.2'
}

2. Check if your device supports the current implementation of FFmpeg or not

fun initialize() {
        val ffmpeg = FFmpeg.getInstance(ctx.applicationContext)
        try {
            ffmpeg.loadBinary(object : LoadBinaryResponseHandler() {
                override fun onFinish() {
                    super.onFinish()
                }

                override fun onSuccess() {
                    super.onSuccess()
                }

                override fun onFailure() {
                    super.onFailure()
                }

                override fun onStart() {
                    super.onStart()
                }
            })
        } catch (e: FFmpegNotSupportedException) {
            Log.e("FFmpeg", "Your device does not support FFmpeg")
        }
    }

3. Initialize the FFmpeg module (leave command blank)

val ffmpeg = FFmpeg.getInstance(ctx)
       ffmpeg.loadBinary(object : FFmpegLoadBinaryResponseHandler {
            override fun onFinish() {
                Log.d("FFmpeg", "onFinish")
            }

            override fun onSuccess() {
                Log.d("FFmpeg", "onSuccess")
                val command = //TODO: the command will added here later
                try {
                    ffmpeg.execute(command, object : ExecuteBinaryResponseHandler() {
                        override fun onSuccess(message: String?) {
                            super.onSuccess(message)
                            Log.d(TAG, "onSuccess: " + message!!)
                        }

                        override fun onProgress(message: String?) {
                            super.onProgress(message)
                            Log.d(TAG, "onProgress: " + message!!)
                        }

                        override fun onFailure(message: String?) {
                            super.onFailure(message)
                            Log.e(TAG, "onFailure: " + message!!)
                        }

                        override fun onStart() {
                            super.onStart()
                            Log.d(TAG, "onStart")
                        }

                        override fun onFinish() {
                            super.onFinish()
                            Log.d(TAG, "onFinish")
                        }
                    })
                } catch (e: FFmpegCommandAlreadyRunningException) {
                            Log.e("FFmpeg", "FFmpeg runs already")
                }
            }

            override fun onFailure() {
                Log.e("FFmpeg", "onFailure")
            }

            override fun onStart() {
            }
        })

4. Implement the commands you want to use in your app

All commands that are supported by FFmpeg can be included in your app with help of an array. This is done by passing every command line argument as a single element within the array. The array will then be translated into an FFmpeg command using the execute method from FFmpeg:

ffmpeg.execute(command, object : ExecuteBinaryResponseHandler() { ... }

As we want to implement Trim and Crop I will show how this can be done using the arrayOf function.

Trim:

val command = arrayOf("-y", "-i", input, "-ss", startPos, "-to", endPos, "-c", "copy", output)

“-y”: overwrites output files without asking
“-i”: specifies an input file
input: the path of the source video to trim
“-ss”: specifies that the next value will be the starting point of the resulting video
startPos: the starting position in “%d:%02d:%02d” format
“-to”: specifies that the next value will be the end position of the resulting video
endPos: the end position in “%d:%02d:%02d” format
“-c” AND “copy”: defines that the stream will not be encoded. The resulting video will only be saved.
output: the path of the resulting video

To better understand every argument you can have a look at the official FFmpeg documentation.

Additionally, to provide arguments for startPos and endPos you normally would have to use a utility function that converts the timestamp of the start and end position into the desired “%d:%02d:%02d” format (a string):

fun convertTimestampToString(timeInMs: Float): String {
        val totalSeconds = (timeInMs / 1000).toInt()
        val seconds = totalSeconds % 60
        val minutes = totalSeconds / 60 % 60
        val hours = totalSeconds / 3600
        val formatter = Formatter()
        return if (hours > 0) {
            formatter.format("%d:%02d:%02d", hours, minutes, seconds).toString()
        } else {
            formatter.format("%02d:%02d", minutes, seconds).toString()
        }
    }

Crop:

val command = arrayOf("-i", input, "-filter:v", "crop=$w:$h:$x:$y", "-threads", "5", "-preset", "ultrafast", "-strict", "-2", "-c:a", "copy", output)

“-i”: specifies an input file
input: the path of the source video to trim
“-filter:v”: defines that a filtergraph is used
“crop=$w:$h:$x:$y”: use crop functionality to crop a part from the video start at point x:y and having a width(w) and height(h).
“-threads”: specifies that the next value will set the thread count
“5”: the number of threads to use
“-preset”: specifies that the next value will set the encoding preset
“ultrafast”: the encoding preset to use
“-strict”: specifies how strictly the standards should be followed
“-2”: the strict value
“-c:a” AND “copy” defines that the stream will not be encoded and ALL audio streams will also be used.
output: the path of the resulting video

Also, check the official FFmpeg documentation to better understand every argument.

5. Implement the UI

To create a fancy UI you should create a custom view that shows some of the frames available in the video (VideoPreviewView). Also, it should contain a SeekBar for cropping and a RangeSeekBar for trimming. The SeekBar will be used to switch to a certain timestamp within the video to see what is actually trimmed or cropped. The RangeSeekBar will only be used within the trimming UI to define the start and end position of the resulting video.

While SeekBar is an Android Widget the RangeSeekBar is more complicated but there exist several implementations that can be used: By Tiszideepan.

You can implement the VideoPreviewView with an easy approach. Divide the video into a specific number of frames that are based on the view’s width and display every frame sequentially within a single view to have a preview. To do this you need the width of the view and the duration of the video that can be found using the MediaMetadataRetriever. With the MediaMetadataRetriever, you can use getFrameAtTime() to fetch a single frame at a specific timestamp. If you now want to display a complete video preview you need to display viewWidth / frameWidth frames. Unfortunately, depending on the video length and video width, it could happen that only a few frames will be present within the VideoPreviewView. To fix this problem you have to maintain a threshold to ensure that a specific number of frames are displayed. This means that you have to crop the frames to a certain width until calculated frames equal the threshold.

The following code snippet will show how you can achieve this:

private fun createPreview(viewWidth: Int) {
        BackgroundExecutor.execute(object : BackgroundExecutor.Task("", 0L, "") {
            override fun execute() {
                try {
                    val threshold = 11
                    val thumbnails = LongSparseArray<Bitmap>()
                    val mediaMetadataRetriever = MediaMetadataRetriever()
                    mediaMetadataRetriever.setDataSource(context, videoUri)
                    val videoLengthInMs = (Integer.parseInt(
                        mediaMetadataRetriever.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION)
                    ) * 1000).toLong()
                    val frameHeight = viewHeight
                    val initialBitmap = mediaMetadataRetriever.getFrameAtTime(
                        0,
                        MediaMetadataRetriever.OPTION_CLOSEST_SYNC
                    )
                    val frameWidth =
                        ((initialBitmap.width.toFloat() / initialBitmap.height.toFloat()) * frameHeight.toFloat()).toInt()
                    var numThumbs = ceil((viewWidth.toFloat() / frameWidth)).toInt()
                    if (numThumbs < threshold) {
                        numThumbs = threshold
                    }
                    val cropWidth = viewWidth / threshold
                    val interval = videoLengthInMs / numThumbs
                    for (i in 0 until numThumbs) {
                        var bitmap = mediaMetadataRetriever.getFrameAtTime(
                            i * interval,
                            MediaMetadataRetriever.OPTION_CLOSEST_SYNC
                        )
                        bitmap?.let {
                            try {
                                bitmap = Bitmap.createScaledBitmap(
                                    bitmap,
                                    frameWidth,
                                    frameHeight,
                                    false
                                )
                                bitmap = Bitmap.createBitmap(bitmap, 0, 0, cropWidth, bitmap.height)
                            } catch (e: Exception) {
                                Log.e(TAG, "error while create bitmap: $e")
                            }
                            thumbnails.put(i.toLong(), bitmap)
                        }
                    }
                    mediaMetadataRetriever.release()
                    returnBitmaps(thumbnails)
                } catch (e: Throwable) {
                    Thread.getDefaultUncaughtExceptionHandler()
                        .uncaughtException(Thread.currentThread(), e)
                }
            }
        })
    }

Additionally, for cropping a video you need to display a crop rectangle within the UI to let users decide which part of the video they want to crop. Also, you have to give users the ability to reposition the crop area. To do this, you can use the ImageCropper library from ArthurHub which is initially designed to crop an image but can also be used for videos as you only need the values from the rectangle. Add the ImageCropper to your project:

dependencies {
    implementation 'com.theartofdev.edmodo:android-image-cropper:2.8.0'
}

Now, you can extend a CropperActivity which loads the current frame of the video (that you can extract with the VideoPreviewView) by inserting the CropImageView layout to get the cropping bounds. Once, you select an area in the UI, the rectangle values can be extracted and used to calculate the needed values for video cropping with this function:

val rect = cropFrame.cropRect
val w = abs(rect.left - rect.right)
val h = abs(rect.top - rect.bottom)
val x = rect.left
val y = rect.top

These values will then be used as input variables for the FFmpeg crop command.

To help you use all this information I created a project that uses all explained snippets to create a sample app that can crop and trim videos. The source code can be found on my personal GitHub.

Fixing error=13, Permission denied

Unfortunately, if you implement the described functionality and use the compiled FFmpeg library you will run into this problem if you compile your app for SDK 29 and onwards.

Permission denied error due to wrong target SDK The problem is caused because from Android Q onwards it is not allowed to execute binaries in your app’s private data directory. The error is described in the official issue tracker from google.

One solution will be that you compile your app for SDK 28 but this could lead to problems if you try to put your app into the Android Play Store.

Another solution will be that you compile your app for SDK 29 and stop putting binaries in any not supported directory. Unfortunately, the code that moves the binaries is within the 3rd party lib from WritingMinds and has to be removed by you (with a Pull Request) or by the maintainer of the library. A future-proof solution will be that you stop using external binaries and start compiling dependencies as an NDK project. This will be a lot of work but you can find help within a famous repository that compiles CPP to Java and also has FFmpeg

Add more commands

After you implemented crop and trim you maybe want to add more cool features to your Android app. To do this with FFmpeg and the previously described code you can do this by simply creating a new function within the VideoCommands() and translate the FFmpeg command into a VideoCommands() function.

The next sections will describe some fancy functions that would improve the usability of your app. Just add a new function with this command and fill the callbacks with a toast or anything else. Then call the function from anywhere within your app (because you do not need UI for this).

Removing audio from videos

You might run into a scenario where you’d only like to keep the visuals of a video and remove the audio track, for instance for voicing over certain footage or removing background noise:

val command = arrayOf("-i", input, "-an", output)

Compress a video

Make big videos smaller to save your valuable disk resources.

val command = arrayOf("-i", input, "-vf", "scale=$w:$h", "-c:v", "libx264", "-preset", "veryslow", "-crf", "24", output)

$w and $h are the new sizes (should be smaller). If one is -1 it will keep the ratio and is automatically adjusted in relation to the other one.

Change playback speed

Increase the playback speed to make long videos faster or decrease it for showing something really awesome very slowly.

 val command = arrayOf("-i", input, "-vf", "\"setpts=$scale*PTS\"", output)

$scale should be > 1 for slowing down a video or < 1 for fastening a video.

To find more cool commands search the official argument documentation and implement them by putting every argument into a value of the array function as described earlier.

For production-grade application you may be better served by using our VideoEditorSDK which in addition to professional trim and crop features allows your users to add filters, text and audio overlays. Recently, we released time-based sprites as well allowing users to time the appearance of text and stickers in videos.

Closing Notes

In this article, you learned that you can use FFmpeg to crop and trim videos on Android. Unfortunately, it does not work out of the box and you have to compile the library using complex techniques. Luckily, there exist some useful libraries that do the heavy lifting for you.

Using my app as a baseline you are able to crop and trim videos. Additionally, you can implement more commands easily if they do not need a GUI because creating a fancy-looking GUI was a complicated part of developing a cropping and trimming app.

Unfortunately, there are problems with the library that was used because Android Q introduces a security fix that forbids apps to execute binaries within their data folder. But, this is only a problem if you want to create a “Google Play Store” ready app because you can avoid this problem if compile your app for SDK 28 (and lower). Keep in mind that if you want to create a private app that you distribute on your website instead of using the Play Store you can do this. But, if you want to upload your app to the Google Play Store you have to search for another FFmpeg library within the documentation or build one from scratch so you can compile your app for the current needed SDK.

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

I hope you enjoyed reading this article and are able to create an Android app that can crop and trim videos. If you have any questions, need help, or want to give feedback @ us on Twitter. We are happy to help.

Build a Simple Real-Time Video Editor with Metal for iOS

Walter — Tue, 30 Aug 2022 20:06:49 GMT

In this tutorial you’ll see how to extract frames from live camera streams, regular movie files and streamed movie files and display them on a MetalKit View. Using Metal allows you to have a great deal of control over how the pixels are rendered and ensures that the GPU is used for rendering, which keeps from slowing down the CPU.

The tutorial code will always convert the video frames into CIImage objects before display. This is to help you adapt the code to your own applications. Applying filters, resizing and other effects are fast and easy when working with CIImage. Additionally, as long as you set your CIContext to the GPU, you can be sure that your code uses the GPU and the CPU efficiently.

When working with Video, Apple provides a number of frameworks that operate at different levels. For the higher level frameworks such as AVKit or CoreImage, the system determines when to use the GPU and when to use the CPU for rendering and computation. The higher-level frameworks also offer a tradeoff between ease of use and granularity of control. If you want to have direct access to tell the GPU how to render every pixel, then you will want to use Metal. Remember, though, that you are now responsible for keeping video and audio in sync, encoding and decoding data and determining a reasonable UI for your user. For many use cases, using a higher level framework is probably a better choice. Apple’s engineers have worked to ensure the graphics frameworks use the GPU and CPU in reasonable ways. However, when you need to use Metal, you you need it. So, let’s get started.

Setting Up a MTKView

An MTKView is a subclass of a UIView so it has a frame and bounds and other properties. Its drawing and rendering is backed directly by drawables and textures rendered on the GPU, so drawing to it can be quite fast. In addition to displaying video, you can render 3D model objects and graphics shaders, so for a graphics rich application it can be a powerful tool.

Before you can send data to the view it needs to be configured. MetalKit Views are still heavily influenced by UIKit, so if you are working in a SwiftUI project, you will need to wrap them in ViewRepresentable code.

//MetalKit Variables
@IBOutlet var displayView: MTKView!
var metalDevice : MTLDevice!
var metalCommandQueue : MTLCommandQueue!

//CoreImage Variables
var ciContext : CIContext!
var filteredImage: CIImage?
var cleanImage: CIImage?

//get a reference to the GPU
metalDevice = MTLCreateSystemDefaultDevice()

//link the GPU to our MTKView
displayView.device = metalDevice

//link our command queue variable to the GPU
metalCommandQueue = metalDevice.makeCommandQueue()

//associate our CIContext with the metal stack
ciContext = CIContext(mtlCommandQueue: metalCommandQueue)

You will need to get a reference to the GPU of the iOS device. At least for now, iOS devices only have one GPU. Then you will need to link the displayView and the metalCommandQueue to the metalDevice. Finally we will link the ciContext to the GPU so that all of our image manipulation code will run in the same place.

//tell our MTKView that we want to call .draw() to make updates
displayView.isPaused = true
displayView.enableSetNeedsDisplay = false
//let it's drawable texture be writen to dynamically
displayView.framebufferOnly = false

//set our code to be our MTKView's delegate
displayView.delegate = self

In the code above, we set the MTKView to be in a isPaused state and do not enable setNeedsDisplay. This will ensure that it will only redraw when we explicitly tell it to redraw using a .draw() method in our delegate. By setting framebufferOnly to false you are telling the view that you will be writing to it multiple times and may also read from it.

Drawing in an MTKView

Now the MTKView is configured. The next step is to update the delegate methods. The MTKViewDelegate has two methods. If your code needs to respond to the view dimensions changing (to support device rotation, or if you want the user to be able to resize the window) make your adjustments in func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize). For this example we are only interested in the delegate method: func draw(in view: MTKView). The code below draws a CIImage to the MTKView. When showing video, we can extract each frame of the video as a CIImage

//create a buffer to hold this round of draw commands
guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
  return
  }

//grab the filtered or a clean image to display
guard let ciImage = filteredImage ?? cleanImage else {
  return
  }

//get a drawable if the GPU is not busy
guard let currentDrawable = view.currentDrawable else {
  return
  }

//make sure frame is centered on screen
let heightOfImage = ciImage.extent.height
let heightOfDrawable = view.drawableSize.height
let yOffset = (heightOfDrawable - heightOfImage)/2

//render into the metal texture
self.ciContext.render(ciImage,
             to: currentDrawable.texture,
  commandBuffer: commandBuffer,
         bounds: CGRect(origin: CGPoint(x: 0, y: -yOffset),
                          size: view.drawableSize),
     colorSpace: CGColorSpaceCreateDeviceRGB())

//present the drawable and buffer
commandBuffer.present(currentDrawable)

//send the commands to the GPU
commandBuffer.commit()

For each pass of the draw method, we will create a new MTLCommandBuffer then we will get the MTKView’s .currentDrawable which contains a texture we can send pixel data to. The texture will have a size and a color space. Though we are using Metal to render images to the screen, Metal can be used to send any valid commands to the GPU for things like computations. Once the buffer has been filled with commands it can be assigned to the drawable and committed. When the buffer is committed, the GPU will execute all of the commands. If another .draw() gets called while the buffer is being executed, the GPU will not interrupt the current buffer in order to start the new one.

The bounds of the render allow you to move where the CIImage gets displayed in the MTKView and resize the CIImage within the view. This can be valuable when compositing multiple images or video streams onto the same MTKView. Remember that unlike a UIView the origin point is the bottom left of the texture and CIImage.

Since iOS 11 Apple provides a new lighter weight API for sending renders to the buffer. Use whichever form makes sense to you.

//render into the metal texture
let destination = CIRenderDestination(mtlTexture: currentDrawable.texture,
                                   commandBuffer: commandBuffer)
do {
 try self.ciContext.startTask(toRender: ciImage, to: destination)
} catch {
 print(error)
}

The CIRenderDestination has some other optional parameters such as height and width but will always display with an origin of 0,0. So, unlike the earlier call, if you need to rotate or change the dimensions of the image, you will need to do it to the CIImage earlier in the code. However, if your app always displays the video at full size in the MTKView this may be easier. Also the startTask call will return immediately, instead of waiting for other tasks on the CPU to complete.

It is important to remember that you can have any number of renders in the commandBuffer before the calls to .present and .commit. This means that the same MTKView can display video from multiple streams as well as static images or animations.

Working with Local Files

In order to extract individual frames from a local file, we can use an AVAssetReader to get pixel data to render in the MTKView

When using higher-level frameworks, a quick way to display a video file for playback is to load it into an AVPlayer. However, an alternative is to use an AVAssetReader to read the tracks and the individual frames from the video tracks.

let asset = AVAsset(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)
let reader = try! AVAssetReader(asset: asset)

guard let track = asset.tracks(withMediaType: .video).last else {
  return
  }

let outputSettings: [String: Any] = [
        kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32ARGB
    ]
let trackOutput = AVAssetReaderTrackOutput(track: track, outputSettings: outputSettings)
    reader.add(trackOutput)
    reader.startReading()

It is important that the outputSettings of the reader match the image format of the video track. If there is a mismatch the color may be off or the reader may be unable to extract a usable buffer. If your code is generating empty or black buffers, the outputSettings are the first place to troubleshoot. Once the reader starts reading then it can extract pixel buffers. You can use a while loop to get all of the buffers from the track and send them to the MTKView for display.

var sampleBuffer = trackOutput.copyNextSampleBuffer()
while sampleBuffer != nil {
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer!) else {
   return
  }
  print(track.preferredTransform)
  print(sampleBuffer?.outputPresentationTimeStamp)

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()

  sampleBuffer = trackOutput.copyNextSampleBuffer()
  }
}

In the code above, use CMSampleBufferGetImageBuffer to ensure that we have a valid image. Then assign the image to a CoreImage and execute the .draw method of the MTKView. Afterwards, get the next sample buffer. This will continue until the end of the track, when .copyNextSampleBuffer() will return nil.

In the code above, there are two print statements to note some valuable information that your app may want to store for later use. Remember, when working with buffers and the GPU directly, much of the higher-level metadata about the video is lost, so you’ll need to keep track of it in your code if you want to use it later. An image may be rotated because of the way it was originally generated in the video. The preferredTransform will provide data that your app can use to transform the image to the “proper” orientation before display. Additionally the outputPresentationTimeStamp tells when the image represented by the buffer appeared in the original video. This can be helpful when you are trying to sync individual frames back to audio tracks or if your app only wants to modify specific frames and then reinsert them into the original clip.

Working with Streams

In addition to local files, an app may want to use a video stream as the input. The process is largely the same, as there is a method to extract a CVPixelBuffer from an AVPlayer that is streaming. You can find the complete example code for pushing pixel buffers from a stream to an MTKView in this blog post about streaming. However, the important part of the code perhaps looks familiar by now:

let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) {
  if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
     let frameImage = CIImage(cvImageBuffer: buffer)
     self.currentFrame = frameImage //a CIImage var
     self.videoView.draw() //our MTKView
   }
}

The code above uses a CADisplayLink to query the stream on a regular basis for new video frames. It then extracts a buffer and sends it over the MTKView for display.

Working with the Cameras

Much like a video stream with an AVPlayerItemVideoOutput to output pixel buffer data, a standard object to use with the camera is an AVCaptureVideoDataOutput object. First, you need to set up a standard capture session for either the front or back camera. Then attach a video data output object to the session.

videoOutput = AVCaptureVideoDataOutput()
let videoQueue = DispatchQueue(label: "captureQueue", qos: .userInteractive)
videoOutput.setSampleBufferDelegate(self, queue: videoQueue)

if captureSession.canAddOutput(videoOutput) {
  captureSession.addOutput(videoOutput)
} else {
  fatalError("configuration failed")
}

Whenever the camera has collected enough data to generate a pixel buffer it will send that to its delegate. The delegate can then convert that data to a CIImage and send it to the MTKView to get rendered. The method in a AVCaptureVideoDataOutputSampleBufferDelegate would look like this.

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
  //get a CVImageBuffer from the camera
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    return
  }

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()
}

Going Further

In this tutorial, you saw how to extract pixel buffers from the camera, local file and remote streams and convert them to CIImage Then you saw how to render a CIImage to fill all or part of an MTKView. If your application needs to resize or apply filters to the images, you can do that before calling the .draw method of the MTKView. If the only reason you are considering using MKTViews is because of a Metal or OpenGL kernel you want to use, consider our tutorial on how to wrap kernels into Core Image filters.

If your application needs to gather streams of video, filter and then render them to the screen with precision, then drawing to an MTKView may be sufficient. However, if you want to let your users dictate how and where to filter the streams, then you may want to consider an SDK like VideoEditor SDK or CreativeEditor SDK.

How to Stream Videos Using Javascript and HTML5

Mark — Mon, 29 Aug 2022 20:36:00 GMT

If you would tell someone from the early 2000s what you’re about to do, they would be amazed and would give everything to understand how you did it. Because back then HTML was much simpler and didn’t even support video playback. So to stream your video inside of your browser you were forced to use some third-party services like Flash or Silverlight. Now we have HTML5 and new versions of JavaScript, and by combining these two new technologies you’ll end up with results that rival native applications. Streaming video inside of your web application nowadays is a breeze.

The `<video>` tag

You can easily start streaming your own video by using only one tag in HTML5. I’m talking about the <video> tag. It embeds a fully-fledged media player in the document and similar to the img tag accepts a path to the media we want to play inside the src attribute as well as accepting parameters for width and height.

In addition, there are video-specific properties such as whether the video should loop or autoplay and whether to display the default browser controls.

To handle the case of browsers not supporting the video tag you can provide fallback content inside the <video></video> tags.

Combining with JavaScript

You can also combine this simple video streaming in HTML with JavaScript and make it more professional and be able to control your streaming manually. When you get an element from the DOM of a video tag, the object that you get exposes various methods and attributes which can be used for manipulating media content. Firstly, we need to access the DOM and declare an object which will be used for our manipulations. Now we can use myVideo to programmatically pause, change the playback rate and current time of the video.

const myVideo = document.getElementById('#video1');

//pause the video
myVideo.pause();

//Change the speed of playback
myVideo.playbackRate = 3.0;

//Change current time
myVideo.currentTime = 4;

However, most videos that we see on the web today such as YouTube, Twitch, or any other social media display much more complex behaviors than just changing playback speed or pausing the video. On YouTube, you can change the quality of your video, add subtitles, or even add features like autoplay if you have several other videos queued up and don’t want to switch it manually.

All those services actually still use a video tag. But instead of streaming a video file through the “src” attribute inside of your tag, they use powerful tools such as Media Source Extensions or other adaptive streaming API that will help stream your video more efficiently.

What are Media Source Extensions (MSE)

Media Source Extensions is a W3C specification that allows JavaScript to send streams to media codecs within Web browsers that support HTML5 video and audio that most browsers implement today. In other words, it allows JavaScript to generate streams and facilitate a variety of use cases like adaptive streaming and time-shifting live streams. It’s quite an advanced solution if you want to make your video playback more customizable and professional. MSE gives us finer grained control over how much and how often content is fetched, and some control over memory usage details, such as when buffers are evicted.

Wrap your resource via the Media Source Extensions API instead of dealing with the URL directly (We’ll talk more closely about it a little bit later).

This specification was designed with the following goals:

Let JavaScript create a media stream regardless of how the media is fetched.
Leverage the browser cache as much as possible.
Provide requirements for byte stream format specifications.
Minimize the need for media parsing in JavaScript.

As I mentioned, we are still using HTML5 video tags, and even more surprising is that we’ll still use the “src” attribute. Only this time, instead of adding a link to the video, add a link to the MediaSource object.

If you’re a little bit confused, then don’t worry, I’ll explain everything. To enable this type of use case, the W3C has defined a static method URL.createObjectURL. You can use this API to create a URL that directly points to a JavaScript object created on the client, rather than actually pointing to an online resource.

You can see how it works in this example:

const videoTag = document.getElementById('my-video');

// creating the MediaSource, just with the "new" keyword, and the URL for it
const myMediaSource = new MediaSource();
const url = URL.createObjectURL(myMediaSource);

// attaching the MediaSource to the video tag
videoTag.src = url;

Now that you have a MediaSource, what should you do? That’s not the only MSE specification. It also defines another concept, Source Buffers.

Source Buffers

The SourceBuffer interface represents a chunk of media to be passed into an HTMLMediaElement and played via a MediaSource object. To put the video directly into the MediaSource for playback, you’ll need Source Buffers. MediaSource contains one or more instances of it. Each is associated with a content type, such as Audio, Video, or both of them at the same time. Source buffers are all associated with a single MediaSource and are used in JavaScript to add video data directly to HTML5 video tags. Separating video and audio allows managing them separately on the server. Doing so has several advantages such as control over your traffic and increasing performance. This is how it works:

// -- Create a MediaSource and attach it to the video (We already learned about that) --

const videoTag = document.getElementById('my-video');
const myMediaSource = new MediaSource();
const url = URL.createObjectURL(myMediaSource);
videoTag.src = url;

// 1. add source buffers

const audioSourceBuffer = myMediaSource.addSourceBuffer(
  'audio/mp4; codecs="mp4a.40.2"'
);
const videoSourceBuffer = myMediaSource.addSourceBuffer(
  'video/mp4; codecs="avc1.64001e"'
);

// 2. download and add our audio/video to the SourceBuffers

// for the audio SourceBuffer
fetch('<http://server.com/audio.mp4>')
  .then(function (response) {
    // The data has to be a JavaScript ArrayBuffer
    return response.arrayBuffer();
  })
  .then(function (audioData) {
    audioSourceBuffer.appendBuffer(audioData);
  });

// the same for the video SourceBuffer
fetch('<http://server.com/video.mp4>')
  .then(function (response) {
    // The data has to be a JavaScript ArrayBuffer
    return response.arrayBuffer();
  })
  .then(function (videoData) {
    videoSourceBuffer.appendBuffer(videoData);
  });

Adaptive Streaming

Many video players have an “auto quality” feature that automatically selects quality based on network speed. If you have a slow internet connection then you’ll end up watching a video in the lowest resolution (probably 360p or even lower), on the other hand, if the connection is good then the video will be in the highest resolution (probably 1080p or even higher). It depends on how many options the specific video has. In addition, your hardware capability is also taken into account.

On the server side, we create different segments for every quality we want to support. For example, we would put the following files on our server:

./video/
  ├── ./360p/
  |     ├── video_0.mp4
  |     ├── video_1.mp4
  |     └── video_2.mp4
	├── ./720p/
  |     ├── video_0.mp4
  |     ├── video_1.mp4
  |     └── video_2.mp4
  └── ./1080p/
        ├── video_0.mp4
        ├── video_1.mp4
        └── video_2.mp4

The web player then automatically selects the correct video file to load when the network or CPU state changes. This is entirely done in JavaScript. For video segments, it could, for example, look as follows:


/**
 * We first derive the desired quality from the bandwidth
 * Then pass it to the pushVideoSegment()
 * which fetches each segment with the desired quality in turn and returns a promise
**/

function pushVideoSegment(nb, wantedQuality) {
 const url = "<http://my-server/video/>" +
 wantedQuality + "/segment" + nb + ".mp4");
return fetch(url)
 .then((response) => response.arrayBuffer());
 .then(function(arrayBuffer) {
 videoSourceBuffer.appendBuffer(arrayBuffer);
 });
}

/**
 * Translate an estimated bandwidth to the right audio
 * quality as defined on server-side.
 * @param {number} bandwidth
 * @returns {string}
 */
function fromBandwidthToQuality(bandwidth) {
 return bandwidth > 320e3 ? "360p" : "720p";
}

// first estimate the bandwidth. Most often, this is based on
// the time it took to download the last segments
const bandwidth = estimateBandwidth();

const quality = fromBandwidthToQuality(bandwidth);

pushVideoSegment(0, quality)
 .then(() => pushVideoSegment(1, quality))
 .then(() => pushVideoSegment(2, quality));

Conclusion

Congratulations, you’ve learned multiple ways how to stream a video file into your browser. We have explored several tools that are helping you not just change some video parameters by using vanilla JS and HTML, and also Media Source Extensions (MSE) that are helping all developers around the world to build professional web players which you’re using yourself, such as YouTube, Twitch, TikTok and many others.

If you want to give your users the ability to also edit videos within your mobile application check out our Video Editor SDK offering features such as trimming, transforms, video composition and more. It’s fully customizable to match the look and feel of your app and the most performant solution on the market.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

If you have questions, feedback or comments reach out to us on Twitter.

How To Build a Canva Clone with CE.SDK

Antonello — Wed, 27 Jul 2022 06:43:59 GMT

Canva has popularized image editing, and user expectations of creative capabilities have increased accordingly. If your web or mobile application includes any design functionality - for book covers, t-shirt designs, or social media content - it has to be on par with the design experience offered by prosumer creation tools such as Canva.

Allowing users to edit images through an easy user interface, define high-quality templates, and give them the ability to share them with the community is a great way to boost engagement, add virality, and potentially new revenue streams.

You might think this requires a lot of time and effort, but that is not the case. CreativeEditor SDK makes it dead simple to build a Canva-like design editor in minutes. I’ll show you how!

Follow this step-by-step tutorial and learn how to implement a Canva clone in React with CE.SDK. At the end of this tutorial, you will achieve the following result:

Try the final result live on CodeSandbox.

What is Canva?

Canva is a free graphic design tool for image editing featuring a drag-and-drop interface for composing different elements on a canvas. You can easily create flyers, invitations, business cards, and more with professionally designed templates. You can think of it as a basic version of Photoshop that anyone can use.

You can create Canva templates in a web browser or on the official app for iOS or Android, share them and collaborate easily. Templates make Canva a powerful tool - that is also true for CE.SDK, which makes it a perfect harbor to set sail for new horizons and create an excellent editor. Before we learn how to replicate Canva, let us evaluate the SDK in question.

What is CE.SDK?

CreativeEditor SDK (CE.SDK) is a fully customizable, user-friendly design editor tool. You can integrate it easily into your application with just a few lines of code and benefit from the template-based editor in no time.

The role-specific editing UI focuses heavily on content adaptation. CE.SDK offers two modes: in Creator Mode, you can create a design from scratch or customize existing templates. Once ready, creators can share their templates and decide which elements others can change and to what degree. Then, users can import the template and customize it in Adopter Mode (Default UI).

Try CE.SDK live demos or start a free trial.

The Creator Mode is the most powerful and least restrictive role in CE.SDK. It empowers you to unleash your creativity and create custom templates for every imaginable use case. Add, move, modify, and delete elements as you wish and define constraints. Further elevate your designs with photo editings, such as filters, effects, and background removal.

The well-rounded text editor also allows you to define variables and programmatically give them a value using the CE.SDK Variable API. For example, you may introduce a {{Name}} Variable and have CESDK import names from a database. This automation is perfect for batch-processing designs, such as greeting cards. Based on this principle, we have built an example NFT Art Collection Generator in React with CE.SDK.

In the Creator Mode, you also can define placeholders. Turning an element into a placeholder will let you determine if it can be deleted, styled, or duplicated by the adopter.

This is what the CE.SDK Creator Mode looks like:

As you can see, you can use CE.SDK to easily create a wedding invitation card template. This is just an example, and you can find other cool sample templates for greeting cards, flyers, postcards, and business cards on showcase page.

What Adopter Mode Offers You

You can test the adopter view of the wedding invitation card template shown above here.

This is what the Adopter Mode looks like:

As shown above, CE.SDK’s Adopter Mode enables you to add and modify colors, text, and images based on the constraints enabled by the creator of the template. This mode provides the Adopter users with a simple interface, narrowed down to the properties that they are allowed to change. This prevents adopters from accidentally changing or deleting parts of a design that should not be modified.

Implement a Canva Clone with CE.SDK

Let’s now learn how to use CE.SDK to build a Canva clone in React.

Prerequisites

This is the list of all the prerequisites for the Canva clone application you are going to build:

Add @cesdk/cesdk-js to your project’s dependencies with the following command:

npm install @cesdk/cesdk-js

Initialize a React Project

You can try out the Canva clone by cloning the GitHub repository supporting the article and running the demo web application with the following commands:

git clone https://github.com/Tonel/canva-clone-cesdk-imgly
cd canva-clone-cesdk-imgly
npm i
npm start

Otherwise, follow this step-by-step tutorial and learn how to build the Canva clone with CE.SDK by yourself.

First, you need to initialize a canva-clone-cesdk-imgly React project. You can do it with Create React App by launching the command below:

npx create-react-app canva-clone-cesdk-imgly

Now, the canva-clone-cesdk-imgly folder will have the following file structure:

canva-clone-cesdk-imgly
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Move into the canva-clone-cesdk-imgly folder and start a local development server with:

cd canva-clone-cesdk-imgly
npm start

Reach http://localhost:3000/ in your browser and make sure you see the default Create React App screen below:

You are now ready to code!

Build the Canva Clone Component with CE.SDK

If you want your image editing component to provide a Canva-like experience, it must include key features, such as template and custom resource management. Let’s see how to implement them all with CE.SDK.

Configure CE.SDK to use templates

Managing templates in CE.SDK is easy. All you have to do is configure the set of predefined templates loaded by CE.SDK on initialization, as explained in the official documentation on adding templates. You can achieve this as below:

useEffect(() => {
  let cesdk;

  let config = {
    // initializing CE.SDK with a few templates
    presets: {
      templates: {
        postcard_1: {
          label: 'Postcard Design',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_1.png`,
        },
        postcard_2: {
          label: 'Postcard Tropical',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_2.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_2.png`,
        },
        business_card_1: {
          label: 'Business card',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.png`,
        },
        instagram_photo_1: {
          label: 'Instagram photo',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_photo_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_photo_1.png`,
        },
        instagram_story_1: {
          label: 'Instagram story',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_story_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_story_1.png`,
        },
        poster_1: {
          label: 'Poster',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_poster_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_poster_1.png`,
        },
        presentation_4: {
          label: 'Presentation',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_presentation_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_presentation_1.png`,
        },
        collage_1: {
          label: 'Collage',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_collage_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_collage_1.png`,
        },
      },
    },
  };

  if (cesdkContainer.current) {
    CreativeEditorSDK.init(cesdkContainer.current, config).then((instance) => {
      cesdk = instance;
    });
  }

  return () => {
    if (cesdk) {
      cesdk.dispose();
    }
  };
}, [cesdkContainer]);

Thanks to the presets.templates, you can specify all the CE.SDK templates to be made available to users.
Note that the cesdkContainer variable stores the React reference to the div where to mount CE.SDK.

This is what the CE.SDK template section will look like:

As you can see, all eight templates loaded with presets.templates.

Integrate External Asset Sources Into CE.SDK

CE.SDK supports external asset sources, meaning you can give your users a vast library of resources to supercharge their creative workflows. Instead of spending time building a repository of images manually, you can provide your users with image resources retrieved dynamically by connecting to services like Unsplash and Airtable. Learn how to integrate external asset sources in CE.SDK, or see a live demo based on the Unsplash and Airtable integration.

Add Custom Images to CE.SDK

You can load custom resources and make them available to users with CE.SDK. For example, you would like to add this Flaticon below to the available images in CE.SDK:

All you need to do is define a new asset source as explained in the documentation page on Integrating Custom Resources into CE.SDK.

You can achieve this as follows:

useEffect(() => {
  // path to the local image to load into CE.SDK
  const customImagePath = `${window.location.protocol + '//' + window.location.host}/resources/programming.png`;

  let cesdk;

  let config = {
    // loading the business card template as default template
    initialSceneURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.scene`,
    // loading the external asset sources
    assetSources: {
      // loading a custom image into CE.SDK
      custom: {
        findAssets: () => {
          return {
            assets: [
              {
                id: 'custom-image-1',
                type: 'ly.img.image',
                locale: 'en',
                label: 'Programming',
                thumbUri: customImagePath,
                size: {
                  width: 512,
                  height: 512,
                },
                meta: {
                  uri: customImagePath,
                },
                context: {
                  sourceId: 'custom',
                },
                credits: {
                  name: 'Freepik',
                  url: 'https://www.flaticon.com/free-icon/programming_1208884?related_id=1208782&origin=search',
                },
              },
            ],
            currentPage: 1,
            total: 1,
            nextPage: undefined,
          };
        },
      },
    },
    // translating the labels associates with the external asset sources
    i18n: {
      en: {
        'libraries.custom.label': 'Custom',
      },
    },
    // initializing CE.SDK with a few templates
    presets: {
      templates: {
        // ...
        business_card_1: {
          label: 'Business card',
          scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.scene`,
          thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.png`,
        },
        // ...
      },
    },
  };

  if (cesdkContainer.current) {
    CreativeEditorSDK.init(cesdkContainer.current, config).then((instance) => {
      cesdk = instance;
    });
  }

  return () => {
    if (cesdk) {
      cesdk.dispose();
    }
  };
}, [cesdkContainer]);

The customImagePath variable stores the path to the local image file programming.png located in the public/resources folder of the React project. This variable is used in the findAssets() function. This function is important because it defines the complete asset source. In other words, you only need one function to implement the custom asset retrieval logic. Note that providing images with labels is useful to make them searchable through the CE.SDK UI.

Now, users will be able to use the image in their template.

Tie it Together

This is what the final CanvaClone component looks like:

import './CanvaClone.css';
import CreativeEditorSDK from '@cesdk/cesdk-js';
import React, { useEffect, useRef } from 'react';
import { findAirtableAssets } from './airtableAssetLibrary';
import { findUnsplashAssets } from './unsplashAssetLibrary';

function CanvaClone({
  // initializing Airtable as default external asset library
  assetLibrary = 'airtable',
}) {
  const cesdkContainer = useRef(null);

  useEffect(() => {
    const externalAssetSources = {
      ...(assetLibrary === 'airtable' && {
        airtable: {
          findAssets: findAirtableAssets,
          credits: {
            name: 'Airtable',
            url: 'https://airtable.com/shr4x8s9jqaxiJxm5/tblSLR9GBwiVwFS8z?backgroundColor=orange',
          },
        },
      }),
      ...(assetLibrary === 'unsplash' && {
        unsplash: {
          findAssets: findUnsplashAssets,
          credits: {
            name: 'Unsplash',
            url: 'https://unsplash.com/',
          },
          license: {
            name: 'Unsplash license (free)',
            url: 'https://unsplash.com/license',
          },
        },
      }),
    };

    // path to the local image to load into CE.SDK
    const customImagePath = `${window.location.protocol + '//' + window.location.host}/resources/programming.png`;

    let cesdk;

    let config = {
      // loading the business card template as default template
      initialSceneURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.scene`,
      // loading the external asset sources
      assetSources: {
        // loading the AirTable or Unsplash asset library
        ...externalAssetSources,
        // loading a custom image into CE.SDK
        custom: {
          findAssets: () => {
            return {
              assets: [
                {
                  id: 'custom-image-1',
                  type: 'ly.img.image',
                  locale: 'en',
                  label: 'Programming',
                  thumbUri: customImagePath,
                  size: {
                    width: 512,
                    height: 512,
                  },
                  meta: {
                    uri: customImagePath,
                  },
                  context: {
                    sourceId: 'custom',
                  },
                  credits: {
                    name: 'Freepik',
                    url: 'https://www.flaticon.com/free-icon/programming_1208884?related_id=1208782&origin=search',
                  },
                },
              ],
              currentPage: 1,
              total: 1,
              nextPage: undefined,
            };
          },
        },
      },
      // translating the labels associates with the external asset sources
      i18n: {
        en: {
          'libraries.airtable.label': 'Airtable',
          'libraries.unsplash.label': 'Unsplash',
          'libraries.custom.label': 'Custom',
        },
      },
      // initializing CE.SDK with a few templates
      presets: {
        templates: {
          postcard_1: {
            label: 'Postcard Design',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_1.png`,
          },
          postcard_2: {
            label: 'Postcard Tropical',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_2.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_postcard_2.png`,
          },
          business_card_1: {
            label: 'Business card',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_business_card_1.png`,
          },
          instagram_photo_1: {
            label: 'Instagram photo',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_photo_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_photo_1.png`,
          },
          instagram_story_1: {
            label: 'Instagram story',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_story_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_instagram_story_1.png`,
          },
          poster_1: {
            label: 'Poster',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_poster_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_poster_1.png`,
          },
          presentation_4: {
            label: 'Presentation',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_presentation_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_presentation_1.png`,
          },
          collage_1: {
            label: 'Collage',
            scene: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_collage_1.scene`,
            thumbnailURL: `https://cdn.img.ly/packages/imgly/cesdk-js/latest/assets/templates/cesdk_collage_1.png`,
          },
        },
      },
    };
    if (cesdkContainer.current) {
      CreativeEditorSDK.init(cesdkContainer.current, config).then(
        (instance) => {
          cesdk = instance;
        }
      );
    }

    return () => {
      if (cesdk) {
        cesdk.dispose();
      }
    };
  }, [cesdkContainer, assetLibrary]);

  return (
    <div className="caseContainer">
      <div className="wrapper">
        <div ref={cesdkContainer} className="cesdk"></div>
      </div>
    </div>
  );
}

export default CanvaClone;

The findAirtableAssets() and findUnsplashAssets() functions come from the GitHub repo associated with the Unsplash and Airtable integration showcase page.

These functions allow CE.SDK to dynamically retrieve resources from online services. A search for images in the CE.SDK UI will perform a query on Airtable or Unsplash and provides them to your users. With just a few lines of code, you are making massive resources available to your users.

Conclusion

Canva has rapidly become one of the most popular graphic design tools. It is so prevalent that users expect similar features in your web and mobile applications. Building a Canva clone from scratch would take months, but with CreativeEditor SDK, it only takes minutes.

In this article, we used CE.SDK to initialize an advanced design editor in React. We adopted its API to configure and customize a design editor to build a Canva clone. This would not be possible without the intuitive UI offered by CE.SDK and its advanced features, such as placeholders and the possibility to define validation rules to guide your users’ creation process.

Here we only scratched the surface of what is possible with CE.SDK – you can use its powerful image processing API to implement many more features and make your Canva clone more and more complex. This is only the beginning!

See how IMG.LY compares to Canva Connect API and check out all the use cases of CreativeEditor SDK or contact sales to learn more!

How to Remove Backgrounds Using Core ML

Walter — Tue, 28 Jun 2022 15:26:53 GMT

In this tutorial, you will learn how to use machine learning to identify an image background and mask it out. This is particularly useful for making stickers or avatars or adding a fake background to a video call. The process of assigning the pixels in an image to a specific object is called “segmentation”. Apple provides an optimized method with pictures of people in its Vision framework. For performing the same tasks with non-human subjects, you can use the DeepLabV3 machine learning model with Core ML. The code examples in this tutorial have been tested using Xcode 13 and Swift 5. Because of the use of Core ML, Vision, and CoreImage in this tutorial, you should run the demo code on a device, not on the Simulator. An iOS project with the demo code is on GitHub.

Image segmentation is a different process and requires other machine learning models than image recognition. With recognition, the model produces bounding rectangles that the system believes to contain the entire object. With segmentation, the model identifies the actual pixels of the object.

In this tutorial, you’ll start with an image of your subject. Then you’ll generate a mask for the background using Vision. Finally, you’ll use CoreImage filters to blend the original image, image mask, and the new image background.

Whether using the Vision framework alone or supplementing Vision with another Core ML model – the process will be the same:

Create a Vision request object with some parameters.
Create a Vision request handler with the image to be processed.
Process the image with the handler and object.
Process the results into the final image using CoreImage.

When working with still images, Apple’s CoreImage framework is usually the best option, mainly because of the large number of available filters and the ability to create reusable pipelines that you can run on the CPU or the GPU.

Using Vision to Segment People

Without an external model, Vision can only segment documents or people in an image. It is vital to understand that Vision will only identify a pixel in the image as “this pixel is part of a person” or “this pixel is not part of a person.” If an image contains a group of people, Vision will not be able to separate individuals.

To segment the image, start by creating an instance of a VNGeneratePersonSegmentationRequest.

var segmentationRequest = VNGeneratePersonSegmentationRequest()
segmentationRequest.qualityLevel = .balanced

This type of request has a few options you can set. .qualityLevel can be .fast, .balanced or .accurate. The level of .accurate is the default. This will determine how closely the mask conforms to the boundaries of the original image. The different levels process at different speeds. The .fast setting is intended for use in a video so that frames don’t get dropped. Using .balanced or .accurate causes noticeable delay in an app processing the image on most devices. Experiment with different settings depending on your needs.

The output of the segmentation request will be a CVPixelBuffer. This is a structure that contains information for each pixel of the image. Most of Apple’s video frameworks as well as CoreImage can work with pixel buffers. The default for VNGeneratePersonSegmentationRequest is a buffer where the color of each pixel is represented by an 8-bit number. Any pixel that Vision thinks contains part of a person will be white and any not-a-person will be black and represented by zero. This will be exactly what you want for generating a mask to work with CIBlendWithMask.

Next, create a VNImageRequestHandler with the image to be processed. The handler class has a number of initializers for different types of data. In this example we will use CGImage, but you could also start with CIImage, CVPixelbuffer, Data, or others. You can also specify options for the handler. In this tutorial we will not specify any, but one of the options is to pass in a CIContext. This can help with performance as you can tell your app to do all of the processing with a single context on the GPU using Metal.

guard let originalCG = originalImage?.cgImage else { abort() }
let handler = VNImageRequestHandler(cgImage: originalCG)
try? handler.perform([segmentationRequest])
guard let maskPixelBuffer =
  segmentationRequest.results?.first?.pixelBuffer else { return }
let maskImage = CGImage.create(pixelBuffer: maskPixelBuffer)

In the code above, the originalImage (which happens to be a UIImage) gets converted to a CGImage. Then we use the image to initialize a request handler. The VNImageRequestHandler has a method .perform which takes an array of all of the Vision requests you want to use to process the image. The .perform method will not return until all of the requests in the array have been completed. Depending on how you want to structure your code, you can either provide a completion handler to use with each of the requests or just process the results in-line. In this example, we’ll process in-line.

If the segmentationRequest found any people in the image, the results array will contain a .pixelBuffer. Create the mask we need by converting the pixelBuffer into a CGImage. To convert to a CGImage, the example uses some helper methods that Matthijs Hollemans published to GitHub specifically for working with Core ML inputs and outputs.

Now that we have the mask image, use CIFilter to compose the final image. It will take a few steps. First, resize the new background, mask, and original image to the same size. Then, blend the three images.

//Convert main image to a CIImage and get the size
let mainImage = CIImage(cgImage: self.originalImage!.cgImage!)
let originalSize = mainImage.extent.size
//Convert the maskimage to CIImage and set the size
//to be the same as the original
var maskCI = CIImage(cgImage: maskImage!)
let scaleX = originalSize.width / maskCI.extent.width
let scaleY = originalSize.height / maskCI.extent.height
maskCI = maskCI.transformed(by: .init(scaleX: scaleX, y: scaleY))
//Convert the new background to a CIImage and set the size
//to be the same as the original
let backgroundUIImage = UIImage(named: "starfield")!.resized(to: originalSize)
let background = CIImage(cgImage: backgroundUIImage.cgImage!)
//Use CIBlendWithMask to combine the three images
let filter = CIFilter(name: "CIBlendWithMask")
filter?.setValue(background, forKey: kCIInputBackgroundImageKey)
filter?.setValue(mainImage, forKey: kCIInputImageKey)
filter?.setValue(maskCI, forKey: kCIInputMaskImageKey)
//Update the UI
self.filteredImageView.image = UIImage(ciImage: filter!.outputImage!)

The above code resizes the images to match the size of the original image and converts them to CIImage. Many machine learning models commonly resize inputs and outputs. For example, the output of the VNGeneratePersonSegmentationRequest using the demo image is 384x512.

Both CIImage and UIImage formats describe an image but do not always have a bitmap representation. That is why each image gets converted to a CGImage first. Though this guarantees that the demo code will work, converting formats makes the code run slower. In your app, you should experiment with different methods to get your images into CIImage format for filtering. Also, the above code uses Matthijs’ resized helper method to do some of the resizing.

With everything resized, the CIBlendWithMask filter stitches the images together. In place of the black pixels in the mask, the final image will show the background – for white pixels, the foreground. Wherever the mask image has a gray pixel, the final image will blend background and foreground.

Now let’s see how to use an additional CoreML model to process images that don’t have a person as the main subject.

Choosing a Model

Apple provides a number of pre-built models for text and image processing. The DeepLab v3 Machine Learning model can perform segmentation requests on images that have subjects like dogs. You can download it from Apple directly or also find it at the DeepLab repo on GitHub. Once you have downloaded the model, add it to your Xcode project the same as any other file. The DeepLab model will recognize people, the same as the VNPersonSegmentationRequest, but also other objects. This does come at a cost in an increased filesize for your application. You may notice that Apple provides multiple versions of the model of different file sizes. They all perform the same task, but differ in how they represent the output and a few other things.

Models on Apple’s website are packaged to work with Core ML and Xcode, so you can easily try a different model. The input and output method names will be the same. It is beyond the scope of this tutorial, yet Apple provides tutorials and example scripts on how to convert TensorFlow and other machine learning models to work with Core ML.

Core ML and Xcode

When you are working with a Core ML model, Xcode provides some convenient tools. Access them by highlighting the name of the model in the File Navigator pane of Xcode. For instance, clicking “Preview” will let you test the model with your data.

You can also use the “Predictions” tab to determine how the input needs to be formatted and what to expect for the output.

Here you can see that the DeepLabV3 model expects images to be 513 x 513 and will output a multidimensional array of integers that is 513 x 513 in size.

Finally, on this screen, you can see an entry for the “Model Class” in the headers. Double-click on the name to jump into the Swift wrapper class for the model to see how to call the model in your code and work with the inputs and outputs.

The DeepLabV3 Model

The DeepLab model input will be a color image that is 513 x 513 pixels. The Vision framework will handle resizing the input, but you can provide options on how that resize should work. The DeepLabV3 model has been trained to recognize and segment these items:

aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
dining table
dog
horse
motorbike
person
potted plant
sheep
sofa
train
tv or monitor

Anything that the model does not recognize, it will consider a background. After performing the recognition, the model returns a two-dimensional 513 x 513 array. Each entry in the array corresponds to one pixel in the original 513 x 513 input image. For instance, every pixel in the original image that shows a dog will be represented in the output by a 12 in the corresponding array entry. Any pixel that is not a recognized object will be given a 0 in the output array to represent a background.

If the model recognizes multiple objects, there will be multiple numbers in the array. If that is not what you want, you need to make changes to the array before creating the mask. For example, using an image of a dog riding a horse, some entries in the array will be 12, others will be 13, and the rest will be 0. To filter out the horse, you need to loop through the array and change any 13s to 0s.

Segmentation with DeepLab

To use the DeepLab model in your code, you again use a request to the Vision framework but this time you can specify a model.

let config = MLModelConfiguration()
var segmentationModel = try! DeepLabV3(configuration: config)
if let visionModel = try? VNCoreMLModel(for: segmentationModel.model) {
  self.request = VNCoreMLRequest(model: visionModel)
  self.request?.imageCropAndScaleOption = .scaleFill
}

In the code above, we create an instance of the DeepLab Core ML object and then initialize a generic VNCoreMLRequest with its model. The only option we set on the request is to tell it how to modify the image when it resizes it for input.

Now the code is very similar to the first example.

let cgImage = originalImage?.cgImage
let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
try? handler.perform([request])
if let observations = request.results as? [VNCoreMLFeatureValueObservation],
  let segmentationmap = observations.first?.featureValue.multiArrayValue {
  guard let maskUIImage = segmentationmap.image(min: 0.0, max: 1.0) else { return }
  applyBackgroundMask(maskUIImage)
}

Because we are using a generic VNCoreMLRequest we have to cast the results as VNCoreMLFeatureValueObservation. The DeepLab model returns a .multiArrayValue instead of a .pixelBuffer so we will again rely on the helper methods to convert it to an image. Now that the mask image has been created, applying the background is the same as in the original example.

Going Further

This tutorial focused on still images, yet both the standard Vision segmentation requests and DeepLabV3 allow processing video input as they can work with CVPixelBuffer and VNSequenceRequestHandler.
The rest of the process will be almost identical to our example. You will need to finetune the performance, or else you will experience frame drops.

Apple provides another method for identifying the background and primary subject in a still image: Portrait mode. When a device renders Portrait mode, it takes pictures with all cameras on the device and stitches them together. This way, the depth of field can change, and you can adjust what items are in focus or blurred.

If the DeepLab model does not cover your subject matter, you can train the DeepLabV3 model to recognize other objects using your own data.

However, you may consider using SDKs such as PE.SDK and VE.SDK. Enabling background removal for your users is easy — all it takes is a few lines in your configuration as described in the official PE.SDK documentation for iOS and Android, which adds a control to the photo editor to toggle the setting.

Integrate a fully customizable photo editor with Background Removal into your app with PE.SDK.

The latest VE.SDK release introduced Background Removal for stickers. This feature recognizes people in pictures and removes the background with one tap – no need for manual outlines or masks.

Let your users create beautiful visuals with Sticker Background Removal.

This feature is available on Android and iOS 15.0 and higher only. See the official documentation for Sticker Background Removal on Android or iOS.

Using an SDK will simplify and accelerate your app development, as you can save time and resources, and focus on the growth and innovation of your application instead.

Wrapping Up

In this tutorial, you saw how to use Vision and Core ML to segment an image of a person and remove the background. You also saw how to use DeepLab to work with images of non-persons.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.

How to Apply Filter Effects to Video using VidEffects library on Android

Neslihan — Tue, 24 May 2022 11:41:23 GMT

Video manipulation is a complex topic on Android because there is no out-of-the-box support from the Android SDK, and using FFmpeg requires not just a very elaborate setup but having to climb a steep learning curve to learn its CLI commands.

As an aside, we have also put together how-to blogs to elaborate on other video manipulation operations like merging still images to form videos, compressing and resizing videos, or background removal.

The VidEffects library is much faster to set up than FFmpeg, and it works by applying effects to GLSurfaceView. This blog post will demonstrate how to apply Filter effects to a video in an Android app using the VidEffects library.

VidEffects does support storing videos on the disk, but it has limitations in this regard which we will discuss in a moment.

Before we begin, keep in mind that the VidEffects library differentiates between Effects and Filters. As an example, we will apply both and discuss the crucial difference between the two.

Without much further ado, let’s start coding.

Create a VidEffects Example App

Create a New Project with an Empty Activity in Android Studio. Select Kotlin as the language (or Java if you wish), and set **Minimum SDK to API 21.

You can also download the final code from GitHub.

Add Dependency

In your module-level build.gradle (app/build.gradle) file add the latest VidEffects dependency.

dependencies {
    ...

    implementation "com.github.krazykira:videffects:1.1.1"

    ...
}

Next, enable viewBinding by adding the following lines within the android{} closure.

android {
    ...
    buildFeatures {
        viewBinding true
    }
}

By using View Binding, we can avoid the findViewById() method.

Click Sync Now to sync the project with gradle files.

Add Video Asset File

Place the sample video file in app/src/main/assets/ directory. Ideally, your app will load the video file from disk – but to avoid complexity, we are placing the video in the assets directory.

Create a Layout

Create the above layout with a VideoSurfaceView, three Filter buttons, and one Reset Button.

Replace your activity_main.xml`s code with the following.

<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">
    <com.sherazkhilji.videffects.view.VideoSurfaceView
        android:id="@+id/mVideoSurfaceView"
        android:layout_width="match_parent"
        android:layout_height="300dp"
        app:layout_constraintTop_toTopOf="parent" />
    <androidx.constraintlayout.widget.ConstraintLayout
        android:id="@+id/wrapper"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@id/mVideoSurfaceView">
        <androidx.appcompat.widget.AppCompatButton
            android:id="@+id/bwButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Black and White"
            app:layout_constraintStart_toStartOf="parent"
            app:layout_constraintTop_toTopOf="parent" />
        <androidx.appcompat.widget.AppCompatButton
            android:id="@+id/grainButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Grain"
            app:layout_constraintStart_toEndOf="@id/bwButton"
            app:layout_constraintTop_toTopOf="parent" />
        <androidx.appcompat.widget.AppCompatButton
            android:id="@+id/duotoneButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="DuoTone"
            app:layout_constraintStart_toEndOf="@id/grainButton"
            app:layout_constraintTop_toTopOf="parent" />
    </androidx.constraintlayout.widget.ConstraintLayout>
    <androidx.appcompat.widget.AppCompatButton
        android:id="@+id/resetButton"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Reset"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/wrapper" />
</androidx.constraintlayout.widget.ConstraintLayout>

Initialize the video

Before initializing the video, set up an instance of the Binding class to use it within the Activity.

Setup View Binding

Add the following code in MainActivity.kt before calling setContentView().

If you do not wish to use View Binding and instead prefer using findViewById(), skip to the next step Initializing the Sample Video.

private lateinit var binding: ActivityMainBinding
class MainActivity : AppCompatActivity() {
		private lateinit var binding: ActivityMainBinding
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        ...
    }
    companion object {
        private const val TAG = "MainActivity"
    }
}

The ActivityMainBinding class is auto-generated. If it did not get auto-generated, then sync the project with the gradle file from File > Sync Project with Gradle Files.

Next, pass the root view to the setContentView().

class MainActivity : AppCompatActivity() {
    ...
    override fun onCreate(savedInstanceState: Bundle?) {
        ...
        setContentView(binding.root)
    }
}

Initializing the Sample Video

Next, declare and initialize the MediaPlayer variable.

class MainActivity : AppCompatActivity() {
    ...
    private lateinit var mMediaPlayer: MediaPlayer
        override fun onCreate(savedInstanceState: Bundle?) {
            ...
            mMediaPlayer = MediaPlayer()
    }
}

Then open the video file from our assets directory and pass it as the data source to the Media Player.

override fun onCreate(savedInstanceState: Bundle?) {
    ...
    try {
        val afd = assets.openFd("sample.mp4")
        mMediaPlayer.setDataSource(
            afd.fileDescriptor,
            afd.startOffset, afd.length
        )
    } catch (e: Exception) {
        Log.e(TAG, e.message, e)
    }
}

Now, all that is remaining is to initialize the VideoSurfaceView.

override fun onCreate(savedInstanceState: Bundle?) {
        ...
        binding.mVideoSurfaceView.init(mMediaPlayer, NoEffectFilter())
    }

Your app should run. While the buttons still do not work, we will setOnClickListeners() next.

Apply VidEffects Effects

The VidEffects library offers multiple effects, but we will be applying three: Black and White, Grain, and Duotone.

In MainActivity’s onCreate(), copy the following code to setup all the setOnClickListeners.

override fun onCreate(savedInstanceState: Bundle?) {
    ...
    //Black and White Effect
    binding.bwButton.setOnClickListener {
        binding.mVideoSurfaceView.shader = BlackAndWhiteEffect()
    }
    //Grain Filter
    binding.grainButton.setOnClickListener {
        val grainFilter = GrainFilter(10, 10)
        grainFilter.setIntensity(0.5f)
        binding.mVideoSurfaceView.filter = grainFilter
    }
    //Duotone Effect
    binding.duotoneButton.setOnClickListener {
        binding.mVideoSurfaceView.shader =
            DuotoneEffect(Color.BLUE, Color.YELLOW)
    }
    //Reset with AutoFixFilter
    binding.resetButton.setOnClickListener {
        binding.mVideoSurfaceView.filter = NoEffectFilter()
    }
}

Run the app.

Difference Between Effects and Filters

As mentioned before, Effects and Filters are two separate functions in the VidEffects library.

Effects are temporary, view-only overlays that cannot be stored on the disk. Filters can be viewed and stored on the disk. Unfortunately, a majority of the overlays offered by VidEffects are Effects and therefore only useful for playback only. At the time of writing this article, only three Filters are available.

Limitations

One of the biggest limitations of this library, as mentioned above, is that the majority of the effects provided by the library are just view-only.

Moreover, to save videos with filters, the app must target minSdk 23 at least or use FFmpeg.

Many of these Effects do not have an intensity parameter and therefore we cannot fine-tune these effects.

Alternative

If you are looking for an easy-to-integrate solution that overcomes these limitations, take a look at IMG.LY’s VideoEditor SDK. It provides more than 60 high-quality adjustable filters out-of-the-box. Moreover, you can easily add custom filters using LUT files. VideoEditor SDK works for Android SDK version 21 and above and thus targets more devices.

If the purpose is to apply effects during the playback, the VidEffects library is a solid and free open-source solution. While it handles all the complex video editing, developers will have to spend some time developing the layout.

Working with Large Video and Image Files on iOS with Swift

Walter — Tue, 10 May 2022 13:27:44 GMT

Video files and image files are among the largest files. The AVFoundation and CoreImage libraries that Apple supplies widely store data on the disk. They try to bring into memory only as much of a file as is needed to complete a task. As you are working on an application that uses files, there are times you will want to compress or resize them. Usually, this is to upload to a server or because you have noticed performance issues, such as stuttering when scrolling or long render times.

This article will cover strategies for compressing and resizing images, videos, and other files. We will also be looking at using URLSession for file uploads and downloads.

As iPhone cameras can take higher-quality pictures, the file sizes have grown quite large. For the last few years, Apple has been using a 12MP camera that takes pictures at 3024 x 4032 resolution. Apple uses a special HEIC compression to make each image about 1.7MB. The same image will generate a 15MB file when saved as a .png and a 3MB file when saved as a .jpeg. Though this will be a high-quality image, services like Instagram or Facebook will often only display images at less than half of that size. The extra pixel data will be compressed away. Perhaps the first and most important part of working with large image files in your app is planning around what size or quality you need. Social Media services similarly resize video files. For instance, TikTok displays images and video using 1080 x 1920, but any iPhone after X will capture video at a larger size.

Writing an Image to Disk

When working with a UIImage or CoreImage object, normally you can save it as an NSData object. These files are lossless (the image quality will not degrade), but they are only useful inside of your application. The standards on the Internet are jpeg and png files.

Fortunately, Apple provides methods to convert UIImage to .jpegData or .pngData. There are also ways to convert CIImage and CGImage, but they are slightly more complicated to configure.

let imageToSave = UIImage(ciImage: <someImage>)! \\1
let jpegData = imageToSave.jpegData(compressionQuality: 1.0) \\2
let file = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first \\3
do {
  try jpegData?.write(to: file!.appendingPathComponent("sleeping_dog.jpg")) //4
} catch(let err) {
  print(err.localizedDescription) //5
}

Here is what’s going on in the code above:

First we need to convert our image to a UIImage. The UIImage class has several different initializers. Use UIImage(named: "some image name") to read an image from the app bundle. Use UIImage(contentsOfFile: "some filepath") to read from the disk. Use UIImage(cgImage: "some cgImage") and UIImage(ciImage: "some ciImage") when you have images already in memory. If you are storing images in a CoreData store or somewhere as Data you can use UIImage(data: "some data").
However you get a UIImage the next task is to convert it to either a .jpg or .png file. Generally, .jpg is a better option when working with photographs and .png produces higher quality for line art. The example above creates a .jpg at the highest quality possible.
Ask the FileManager for a URL to the user’s documents directory. That is generally a safe place to write files. Because of the way the function is formed, it returns an array of URL items. In this case, there is only one, but you still need to remember to specify the .first item.
Writing Data object to disk can throw errors, so it is good practice to place it in a try…catch block of code. The URL to the user’s documents directory needs to have the actual filename appended.
Any errors that the system throws can be handled here.

In order to create a .png using the imageToSave above, the line would be

let pngData = imageToSave.pngData()

The .png filetype does not have any compression settings. The filetype is “lossless”, so it is always compressed as much as possible without losing any data.

Compression Options for JPEG

When generating jpeg data above, we supplied a quality value. The value is a Float between 0.0 and 1.0. The .jpg filetype uses a “lossy” compression algorithm that is specifically designed for photographs.

In the image above saving the jpeg with a quality of 1.0 produced the dog on the right and the file is 1.3MB. Saving the jpeg with a quality of 0.0 produced the image on the left and the middle image is produced saving with a quality of 0.6. The file on the left is only about 6% of the size of the original.

Considering what the images are being used for can help you decide on an optimal compression setting for a particular app.

Compressing Data with `zlib`

Though png and jpeg files are already compressed sometimes you will want to compress other types of files before uploading them (compressing a png, mov or jpeg will often not affect the file size or will make the file larger). A web server will often be able to accept a POST that compresses its body data using gzip. Additionally, some web servers will require that binary data is transformed into a base64EncodedString before uploading. Base64 encoding will make data size grow by about 33%. Using gzip will help mitigate that somewhat. Apple’s implementation of gzip is called .zlib when compressing data. Swift provides some other compression algorithms such as .lzfse which will result in smaller files, but they are not widely adopted.

For example, to encode something as base64 and then compress it with with gzip so you can upload it, we can use code like below

let jsonToUpload = "{"books":[{"title":"The Three Musketeers","author":"Dumas"}]}"
let jsonAsData = jsonToUpload.data(using: .utf8)
let encodedString = jsonAsData?.base64EncodedString()
let compressedString = try? NSData(data: (encodedString?.data(using: .utf8))!).compressed(using: .zlib) as Data

The code above creates a string of data in JSON format. It then encodes the String as a Data object using .utf8 encoding. Using .utf8 is a standard. If the web server you are uploading to requires a different encoding, you can change it. In the final step, the encoded string is compressed using gzip. Notice that the .compressed(using:) method is on NSData not on Data. NSData is an older library, so you convert the compressed NSData object back to Data by casting using as Data.

The above example was just to show syntax. The compressedString will be slightly larger than the encodedString. Based on the kind of data your app works with and the capabilities of the web server, you will need to experiment with how to encode and compress to get the best results.

Transferring Data With a Server

Depending on the original format of the data, your web server may want the .httpBody compressed. You can do that with some code like this:

let someJSONData = Data() //1
var request = URLRequest(url: URL(string: "https://example.com/uploads/")!) //2
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
if let compressedJSONData = try? NSData(data: someJSONData).compressed(using: .zlib) as Data { //3
  request.httpBody = compressedJSONData
}
let task = URLSession.shared.dataTask(with: request) {data, response, error in
  //check for errors and response data when the task is done
  //response will contain the status and other header messages
  //data will contain any payload the server returns
}
  task.resume() //4

Here is what the code above is doing:

Get the payload into a Data object.
Create a URLRequest as a var so that you can configure it. If you create it as a let it will be a GET request with default header fields.
Use gzip to compress the data and if successful, assign it to the .httpBody of the request.
Create a data task for the upload. Upon completion, the data, response and error variables will have any response from the server.
Actually start the task.

Uploading a Large File

An issue with the code above is that all of the initial data will be in memory. So for a large image or video upload, this may not work. However, Swift does provide a different method that will upload a file directly from the disk, reading into memory only what is needed at the time.

Code to implement that would follow this form.

var request = URLRequest(url: URL(string: "https://example.com/uploads")!)
request.httpMethod = "POST"
//set the other header variables here using
//request.setValue("value to set", forHTTPHeaderField: "header field name")
let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first
let fileToUpload = documentsDirectory?.appendingPathComponent("the-big-giant-file.mp4")
let uploadTask = URLSession.shared.uploadTask(with: request, fromFile: fileToUpload) {data, response, error in
//get some information from the server when the file has been uploaded
}
uploadTask.resume()

This looks very similar to the previous example but notice that you do not set an .httpBody. There will be header values to set though. Most web servers will want to know how large the file is, authentication credentials, and whether the data is multi-part, or raw or other metadata. These will all be set using .setValue(forHTTPHeaderField:) in the request object.

For larger files, it is often better to use a URLSessionDataDelegate instead of the single closure with data, response, error. When using the delegate, you can write code to monitor the progress of the file transfer as well as respond to authentication challenges and make decisions about caching. You can also implement URLSessionDelegate and URLSessionTaskDelegate methods. All of the delegate methods are options, so you only need to implement ones that are important to your application.

Configuring Background Transfers

Another benefit of using the uploadTask (and its related downloadTask) instead of the dataTask is that you can configure the transfer to continue for a time even if the app goes to the background. In that case, instead of using the URLSession.shared object, you will want to configure a session that has some special properties to allow it to work in the background.

let configuration = URLSessionConfiguration.background(withIdentifier: "some.unique.identifier")
let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
let uploadTask = session.uploadTask(with: request, fromFile: fileToUpload)

Completely configuring and supporting the downloads and backgrounding is covered in this article by Apple. It will require implementing some delegate methods for the session as well as updating some of the methods in the main application delegate. The most important is the handleEventsForBackgroundURLSession which is how the system will wake up your app with information about the upload or download.

Additionally, download tasks can be paused and restarted. The cancel(byProducingResumeData:) method on URLSessionDownloadTask will cancel a download and optionally provide some resume data you can use to resume the download at a later time. You can call this method on the task when the user taps a button or when the app goes to the background or similar. The web server must support byte-range requests and ETag or Last-Modified headers. The documentation for pausing and restarting explains how to pause the download and how to resume using downloadTaks(withResumeData:). There is not a similar structure for pausing and resuming an upload.

Resizing an image

Perhaps the simplest way to make a file smaller is to reduce its dimensions. Using CoreImage you can use the Lanczos resampling algorithm to resize an image while keeping high quality.

let url = Bundle.main.url(forResource: "sleeping-dog", withExtension: "jpeg")
let image = CIImage(contentsOf: url!) //1
let filter = CIFilter(name: "CILanczosScaleTransform") //2
filter?.setValue(image, forKey: kCIInputImageKey)
filter?.setValue(0.5, forKey: kCIInputScaleKey) //3
let result = filter?.outputImage
let converter = UIImage(ciImage: result!) //4

Here is what this code is doing:

Create a CIImage from a URL.
Create a CILanczosScaleTransform filter.
Configure the filter to reduce the image dimensions by 50%.
Convert the scaled image to a UIImage.

Recently, Apple has provided a UIGraphicsImageRenderer that will work directly on UIImage objects. The code below will resize a UIImage to 50%. Notice that the UIGraphicsImageRenderer can also use an explicit value for size (unlike the CILanczosScaleTransform)

let imageToSave = UIImage(named: "sleeping-dog.jpeg")! //1
let newSize = imageToSave.size.applying(CGAffineTransform(scaleX: 0.5, y: 0.5)) //2
let renderer = UIGraphicsImageRenderer(size: newSize) //3
let scaledImage = renderer.image { (context) in
  let rect = CGRect(origin: CGPoint.zero, size: newSize)
  imageToSave.draw(in: rect) //4
}

Here is what this code is doing:

Load the image as a UIImage.
Scale the size down by half.
Create a UIGraphicsImageRenderer that will render in this new size.
Actually draw the image into the renderer at the new size.

In addition to rendering the image back as a UIImage the renderer can create jpegData and pngData which would save a few steps if you are resizing and converting. The full documentation for UIGraphicsImageRenderer explains the details.

Resizing a Video

Though the UIGraphicsImageRenderer is faster, using the older CoreImage resizing has a benefit. CoreImage filters can be applied to video files as a composition. Large video files can be scaled down. The code below will create a composition that will scale a video asset by 50%. Remember that you will need to import AVFoundation to use these tools.

let newAsset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
var newSize = <some size that you've calculated> //2
let resizeComposition = AVMutableVideoComposition(asset: newAsset, applyingCIFiltersWithHandler: { request in
  let filter = CIFilter(name: "CILanczosScaleTransform") //3
  filter?.setValue(request.sourceImage, forKey: kCIInputImageKey)
  filter?.setValue(<some scale factor>, forKey: kCIInputScaleKey) //4
  let resultImage = filter?.outputImage
  request.finish(with: resultImage, context: nil)
})
resizeComposition.renderSize = newSize //5

Here’s what the code above is doing to create a new composition:

Create an AVAsset from a URL.
Create a variable newSize to hold the final size.
In the composition, configure the CIFilter that will be applied to each frame.
Calculate the scale factor based on the newSize variable and the actual size of the request.sourceImage.extent.size.
Set the renderSize property of the composition to the new size.

If you don’t set the renderSize then there will be a black letterbox around the video.

With the resizing composition, you can now export the AVAsset as a .mov or .m4v file using an AVExportSession.

let asset = AVAsset(url:Bundle.main.url(forResource: "jumping-man", withExtension: "mov")!) //1
let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("exported.mov") //2
//create exporter
let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)
//configure exporter
exporter?.videoComposition = <the composition you created above> //3
exporter?.outputURL = outputMovieURL!
exporter?.outputFileType = .mov
//export!
exporter?.exportAsynchronously(completionHandler: { [weak exporter] in //4
  DispatchQueue.main.async {
    if let error = exporter?.error {
      print("failed \(error.localizedDescription)")
    } else {
      print("file saved at \(outputMovieURL)") //5
    }
  }
})

Here’s what the code above is doing:

Load an AVAsset from a URL.
Create a URL to save the resized movie.
Apply the resizing composition to an AVAssetExportSession.
Asynchronously export the asset to disk.
Print the destination URL of the new movie when it is saved.

Another method to resize a video is to use one of the AVAssetExportSession presets. Apple offers several size and quality presets for export sessions. In the line above that creates the exporter, replace AVAssetExportPresetHighestQuality with one of the other values. There are generic quality presets: AVAssetExportPresetLowQuality, AVAssetExportPresetMediumQuality and there are size presets including: AVAssetExportPreset1280x720 and AVAssetExportPreset640x480. When using one of the presets you do not need to use a composition unless you want to do further manipulation or unless you need a size or quality combination that is not provided by any preset. As with images, experiment with different settings until you get a balance between quality and size that works for you.

Going Further

Some other strategies for working with large files are to chunk data file or split up video files. However, to use this strategy, you will need to coordinate with someone to stitch them back together after they are uploaded or downloaded.

Using the AVAssetExportSession above, you could call it multiple times and pass in a time range perhaps using code like this

let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)
//some code to create the exporter
exporter?.timeRange = timeRange

Depending on the video and audio of the AVAsset the splits in the video may be noticable. Because of modern frame rates, the audio would probably be more likely to be noticed. Time ranges can be sub-second, so you would need to experiment. However, iOS would manage the file for you so that the entire AVAsset is never loaded into memory.

It is also possible to split Data objects using .subData(in:) and looping through the bytes of the object. Again, you would need to also write code to stitch them back together later. Additionally, you would likely need to bring the entire Data object into a memory buffer, which might not be desirable.

Another way to handle large video files is to use Apple’s HTTP Live Streaming tools. These create files that you can upload to any web server that is recognized by iOS and Mac devices. The tools will segment the video as well as create multiple versions so that your users with different bandwidth capabilities can see different quality streams. Most Android devices and web browsers will at least be able to see the video, even if they cannot dynamically switch between streams.

Wrapping Up

In this tutorial, you saw some different strategies for shrinking and compressing the large image and video files that an iPhone camera can produce. You also saw some strategies for transferring larger data payloads with a web server. As mentioned in the beginning, consider what sizes the images and videos will display and use that to determine what sizes of images to store. Also, consider storing multiple versions of images or using thumbnails when displaying a gallery or list of images.

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

How to Make an Animated GIF Using Swift

Walter — Tue, 10 May 2022 12:07:58 GMT

In this article, you will see how to use Swift to create an animated GIF file. You will also see how to extract individual frames from a video file to convert the whole video or just clips from a video into an animated GIF. The code in this article uses Swift 5 and Xcode 13. The tutorial uses AVFoundation and CoreGraphics. Clone this repository for a sample project and example code.

Animated GIF Basics

The GIF is one of the older file types on the Internet. But, every platform supports it (not every platform supports every video format). Unlike some other image formats, GIF files can contain one image or a series of images. When a GIF file has a series of images and some metadata to describe how long to show each one, it can substitute for animation and video. Because of this, a platform that doesn’t necessarily support video files can display moving images and animations.

GIF files are easy to make. However, animated GIF files are not perfect: they are large and inefficient compared to modern video files. They have a limited color palette, and none of the images in the file have compression. For example, the video file we use in the sample project is about 5MB in mp4 format but over 100MB after it becomes an animated GIF. Finally, GIF files don’t have a soundtrack. Still, they are popular and do certain tasks well, so let’s make some using AVFoundation and Swift.

The Basic Strategy

In order to create animated GIF files you need a series of images and some metadata to describe how long each image will show. This tutorial will start with how to put these images into a GIF. Then you will see some different strategies for gathering the images. A GIF file contains bitmap images, not vector graphics. When working with bitmap images, a powerful framework to explore is CoreGraphics.

Creating an Empty File

The first step is to create an empty file on the disk that will become our animated GIF.

//create empty file to hold gif
let destinationFilename = String(NSUUID().uuidString + ".gif")
let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(destinationFilename)
//metadata for gif file to describe it as an animated gif
let fileDictionary = [kCGImagePropertyGIFDictionary : [
  kCGImagePropertyGIFLoopCount : 0]]
//create the file and set the file properties
guard let animatedGifFile = CGImageDestinationCreateWithURL(destinationURL as CFURL, UTType.gif.identifier as CFString, totalFrames, nil) else {
  print("error creating gif file")
  return
}
CGImageDestinationSetProperties(animatedGifFile, fileDictionary as CFDictionary)

The code above creates a URL to hold the animated GIF in the user’s Temporary directory. The location is not crucial – it just needs to be where you have write permission. Next it creates a Dictionary that will describe the file as an animated GIF file that will loop forever. The Apple Documentation for GIF Image Properties contains the full list of options for the dictionary. Next, you create the empty file using CGImageDestinatonCreateWithURL. After creating the file, set the properties dictionary. Now there is an empty file ready to accept the frames that will make up the animated GIF.

When creating the file with CGImageDestinationCreateWithURL, you give parameters for the URL to the file location, the UTType identifier to show it will be a .gif file, and the totalFrames that it should expect. totalFrames is just a variable that holds how many actual frames the file will eventually hold. You will need to work out the actual number of frames for your file manually. For example, if you want to display 20 images each second and your GIF will be four seconds long, the total number of frames will be 20 * 4 or 80. The last parameter is always nil. The documentation indicates it “is reserved for future use” (but this function has been available to the Mac since 2005 and iOS since 2010, so we will see). The documentation also indicates you are responsible for releasing the memory allocated by the creation command, but Swift will handle that for you.

As you work with CoreGraphics classes and types, you will notice that they have a slightly different format than other frameworks. That is a hint that CoreGraphics is one of the older frameworks and that it is all C based. Swift has bridging code so that the system will worry about translating your Dictionary to a CFDictionary and your String to a CFString and so on.

Adding Images

To add images to the empty file, you will need an image for each frame of the animation, and you will need a small Dictionary to tell the system how long to display the image during the animation. As mentioned above, there are other options you can apply, such as color mapping or frame dimensions.

A Dictionary to display each frame for 1/20 second could look like this:

    let frameDictionary = [kCGImagePropertyGIFDictionary : [
      kCGImagePropertyGIFDelayTime: 1.0 / 20.0]]

Notice that the values above are Double and are not Int. Using 0.05 would have also been fine as it is equal to 1.0/20.0.

Compressed video files and movies are usually 30, 60, or even 120 frames per second. Recall that animated GIF images do not have compression, so you want to have as low of a frame rate (or as small of an image frame) as you can. Experiment with different frame rates depending on the speed of the action in an animation. Often times 20 or even 10 frames per second will produce high enough quality.

Now, loop through your images and append each image to the .gif file on disk using code like this:

CGImageDestinationAddImage(animatedGifFile, frameImage, frameDictionary as CFDictionary)

The animatedGifFile is the URL to the empty file on the disk that you created above. The frameDictionary is the metadata to add to each frame and the frameImage is the actual image for the frame. Your image for the frame needs to be a CGImage. Depending on the source of images you may have UIImage or CIImage or something else. Most image formats in Swift have a conversion method. For example:

let cgImage = UIImage().cgImage
let anotherCGImage = CIImage().cgImage

There are also CGImage initializers to read .jpeg and .png data.

Finalizing the File

Once you finish writing every frame image to the GIF file, close the file and instruct the system to clean up using:

CGImageDestinationFinalize(animatedGifFile)

This function will return true if it is successful. You must call this function after appending the frames or the resulting GIF will be unusable. Also, after you call this function, you cannot add more frames.

At this point the animatedGifFile URL points to an animated GIF file you can use as needed.

Collecting Frame Images

To create the animated GIF, it is necessary to have an image in CGImage for each frame. It will be easier if they are all rotated the same way and have the same dimensions. Where those images come from is not crucial. They might be a series of files on disk. It could be that your user snaps a series of photos that your app converts to a GIF in real-time. Your app could even intersperse photographs and bitmap drawings. Something to consider regardless of the source of the frame images is how much memory they consume. Loading the entire set of images into an array before making the animated GIF could cause your app to ask the system for gigabytes of memory.

Extracting Images from Video

A common use of animated GIF files is to display videos. So, let’s look at how to extract frames from a video. In some of the other tutorials, you have seen how we can use CoreImage and AVFoundation to manipulate each frame of a video. In those cases, you can’t easily change the frames per second. A 60 frames-per-second animated GIF would quickly become an unusably large file.

AVFoundation provides the AVAssetImageGenerator class specifically to extract frames from AVAssets at specific times. To extract a single image from an AVAsset you can use:

copyCGImage(at requestedTime: CMTime, actualTime: UnsafeMutablePointer<CMTime>?)

This will create a CGImage at or near the requestedTime in the video. Sometimes, the requested time is between two frames, so the function will extract the closest frame it can and return the actualTime of that frame. Your code can then decide if it’s an acceptable substitution and what to do about it. Usually, though, you don’t care and can pass nil for the actualTime parameter. Notice that the actualTime is a pointer to a CMTime variable. This means it is an “inout” parameter. Using “inout” parameters was a common practice when CoreGraphics and AVFoundation were originally written. To use an “inout” parameter, you declare the variable before calling the function and then check its value after. Additionally, when you pass the variable into the function, place an ampersand before the variable name.

For extracting a series of images, copyCGImage is not efficient. It runs on the calling thread and may block your application. Apple provides a different function when you want to extract many frames from a video asset.

generateCGImagesAsynchronously(forTimes requestedTimes: [NSValue], completionHandler handler: @escaping AVAssetImageGeneratorCompletionHandler)

This function takes an array of requestedTimes and attempts to extract the image at each of those timestamps. It will pass the extracted image to the completionHandler. The handler will execute its code one time for every extraction attempt and will not block the calling thread.

Before calling the function to extract images, you need to prepare the inputs. First, you create an AVAssetImageGenerator and configure it.

let movie = AVAsset(url: Bundle.main.url(forResource: "swish", withExtension: "mp4")!)
let frameGenerator = AVAssetImageGenerator(asset: movie)
frameGenerator.requestedTimeToleranceBefore = CMTime(seconds: 0, preferredTimescale: 600)
frameGenerator.requestedTimeToleranceAfter = CMTime(seconds: 0, preferredTimescale: 600)

This code loads a movie file from the app bundle into an AVAsset and then creates a generator for that asset. The .requestedTimeToleranceBefore and .requestedTimeToleranceAfter parameters say you want the exact frame for any requested time. This will make the extraction of frames slower. If you are trying to tune performance, consider modifying these parameters.

Now you will need to calculate the timestamp of each frame of the animated GIF. Determine the start time and the duration of the clip you want to create. Then decide what frame rate (frames per second) to use. You can see in the image below that the entire video consists of about 480 frames at 60 frames per second.

An animated GIF of just the player making the basketball shot would start at 2 seconds and end at 6 seconds of the original video. In the original video, that segment is 240 frames. An animated GIF of the same length at 20 frames per second only needs about 80 frames. The first frame is the one showing at the 2-second timestamp. The next frame should be the one shown at 2 + 1/20 or 2.05 seconds. The next frame would be the one for 2.10 seconds.

The generator requires the array of timestamps as CMTime values. You can create the array using code like this:

var startTime: Double = 2.0 //seconds
var endTime: Double =  6.0 //seconds
var frameRate = 20 //how many frames per second
let totalFrames = Int((endTime - startTime) * Double(frameRate))
var timeStamps = [NSValue]()
for currentFrame in 0..<totalFrames {
  let frameTime = Double(currentFrame) / Double(frameRate)
  let timeStamp = CMTime(seconds: startTime + Double(frameTime), preferredTimescale: 600)
  timeStamps.append(NSValue(time: timeStamp))
}

This code calculates a value for totalFrames which must be an Int so that you can use it to loop. But, startTime and endTime need to be Double since the timestamps will be sub-second times. The frameTime of the first frame is equal to zero in the animated GIF but 2.0 seconds in the original video. Convert that to the video timeStamp by adding the startTime. When creating the CMTime using “600” as the preferredTimescale is common because most frame rates are divisible into 600 evenly so the math for the timestamp will be more accurate. Finally, wrap the timeStamp in an NSValue object and add it to the array. The AVAssetImageGenerator uses timestamps wrapped in NSValue objects. NSValue is just a generic type that holds values.

With the generator configured and the array of timestamps created, you can extract the images using code like this:

frameGenerator.generateCGImagesAsynchronously(forTimes: timeStamps) { requestedTime, frameImage, actualTime, result, error in
  guard let frameImage = frameImage else {
    print("no image")
    return
  }
  if error != nil {
    print("an error")
    return
  }
  //do someting interesting with frameImage
}

The frameImage in the completion handler will be a CGImage type if the extraction was successful. As with the function to extract a single image, if the system had to choose a different time than requested, you will be informed of the actual time. You can read about the other parameters in the documentation for AVAssetImageGeneratorCompletionHandler.

Now, you can add the frame to an open CGImageDestination, you could write it to a disk to use later, and you could save it to an array. If you don’t write it to an animated GIF file right away, you will need to devise a way to keep the frames in sequence. You may notice that there is not an explicit way for the function to signal that it is finished. Because the completion handler is called for every extraction attempt, a common strategy is to make a simple counter and increment the counter in the completion handler. When the counter variable is equal to the number of frames requested, then you are finished. Another way is to match the requestedTime to the last timeStamp.

Wrapping Up

In this article, you saw how to create an animated GIF from a series of images. You also saw how to extract still images from a video at pre-determined timestamps. Clone this repository for a simple project that contains the sample code and a video file you can experiment with.

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

How To Apply Image Filters in WebGL

Antonello — Fri, 08 Apr 2022 13:06:16 GMT

In this article, you will learn how to apply custom filters to an image in WebGL. You can achieve this goal without using any external library. That is because the HTML5 canvas element natively supports WebGL and does not require the use of plug-ins.

As we have already covered in previous articles, <canvas> is a powerful tool that equips you with everything required to manipulate images. This is true regardless of the use of WebGL. So, you can filter images also without using WebGL.

Now, let’s see how to filter images in Vanilla JavaScript with WebGL in your browser. Follow this step-by-step tutorial and learn how to build the following demo:

Getting Started with WebGL

If you are not familiar with WebGL, you need to learn a few things before approaching it. To build an application in WebGL, start with the following three concepts.

1. Textures

Keep in mind that to draw an image in WebGL you have to use a texture. To use a texture, WebGL requires you to define the texture coordinates. These coordinates go from 0.0 to 1.0, regardless of the texture size.

2. Vertex shader

The vertex shader is a function you have to write in GLSL that is in charge of computing the vertex positions. Thanks to it, WebGL can rasterize the draw primitives, which include points, lines, and triangles. When rasterizing these primitives, WebGL calls another user-defined function called fragment shader. In other words, WebGL interpolates the values provided in the vertex shader function while it draws each pixel using the fragment shader function execution.

3. Fragment shader

The fragment shader is a function you have to write in GLSL whose goal is to generate a color for each pixel of the draw primitive currently being drawn. This function has little info per pixel, but you can provide it with everything required by using the varyings variables. These allow you to pass values from the vertex shader function to the fragment shader function.

Filtering Images with Kernels in WebGL with `<canvas>`

There are many ways to filter images, but the most common one involves the convolution operation. This is because when used on images, convolution applies a filter by taking the weighted sum of a square of pixels and assigning the resulting value to the current pixel. This logic is applied to every pixel the image consists of. Therefore, you can now imagine why convolution is one of the most relevant concepts when it comes to image processing.

The coefficients used to perform the weighted sum come from a matrix called kernel. The kernel represents the filter you want to apply through the convolution operation. So, by changing the kernel, the resulting image will change accordingly. Some kernels are more useful than others and can be used for blurring, sharpening, performing edge detection, and other operations. You can find a list of the most popular kernels on Wikipedia.

Now, let’s see how to implement kernel image filtering in WebGL.

Clone the GitHub repository that supports this article with the following command:

git clone https://github.com/Tonel/how-to-filter-an-image-in-webgl-imgly

Additionally, try the demo application by launching how-to-filter-an-image-in-webgl-imgly/index.html in your browser.

Otherwise, you can find the JavaScript function taking care of implementing the filter logic in WebGL below:

function filterImage(canvas, originalImage, kernel) {
  // assuming the kernel is a square matrix
  const kernelSize = Math.sqrt(kernel.length);
  const gl = canvas.getContext('webgl');

  // clearing the canvas
  gl.clearColor(1, 1, 1, 1);
  gl.clear(gl.COLOR_BUFFER_BIT);

  const vertexShaderSource = `
  attribute vec2 position;
  varying vec2 v_coordinate;
  void main() {
    gl_Position = vec4(position, 0, 1);
    v_coordinate = gl_Position.xy * 0.5 + 0.5;
  }
`;

  const fragmentShaderSource = `
  precision mediump float;
  // the varible defined in the vertex shader above
  varying vec2 v_coordinate;
  uniform vec2 imageSize;
  uniform sampler2D u_texture;
  
  void main() {
    vec2 position = vec2(v_coordinate.x, 1.0 - v_coordinate.y);
    vec2 onePixel = vec2(1, 1) / imageSize;
    vec4 color = vec4(0);
    mat3 kernel = mat3(
      ${kernel.join(',')}
    );
    // implementing the convolution operation
    for(int i = 0; i < ${kernelSize}; i++) {
      for(int j = 0; j < ${kernelSize}; j++) {
        // retrieving the sample position pixel
        vec2 samplePosition = position + vec2(i - 1 , j - 1) * onePixel;
        // retrieving the sample color
        vec4 sampleColor = texture2D(u_texture, samplePosition);
        sampleColor *= kernel\[i\][j];
        color += sampleColor;
      }
    }
    color.a = 1.0;
    gl_FragColor = color;
 }
`;

  const vertexShader = compileShader(gl, gl.VERTEX_SHADER, vertexShaderSource);
  const fragmentShader = compileShader(
    gl,
    gl.FRAGMENT_SHADER,
    fragmentShaderSource
  );

  // iniziailing the program
  const program = createProgram(gl, vertexShader, fragmentShader);

  const positionAttributeLocation = gl.getAttribLocation(program, 'position');

  const imageSizeLocation = gl.getUniformLocation(program, 'imageSize');

  // binding the position buffer to positionBuffer
  const positionBuffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, positionBuffer);

  // using the program defined above
  gl.useProgram(program);
  // enabling the texcoord attribute
  gl.enableVertexAttribArray(positionAttributeLocation);
  // setting up the size of the image
  gl.uniform2f(imageSizeLocation, canvas.width, canvas.height);
  // telling positionAttributeLocation how to retrieve data out of positionBuffer
  gl.vertexAttribPointer(positionAttributeLocation, 2, gl.FLOAT, false, 0, 0);
  // provide the texture coordinates
  gl.bufferData(
    gl.ARRAY_BUFFER,
    new Float32Array([-1, -1, -1, 1, 1, -1, 1, 1, 1, -1, -1, 1]),
    gl.STATIC_DRAW
  );

  // loading the original image as a texture
  const texture = gl.createTexture();
  texture.image = new Image();

  // setting the anonymous mode
  // Learn more about it here:
  // https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/crossOrigin
  texture.image.crossOrigin = '';
  texture.image.src = originalImage.src;
  texture.image.onload = function () {
    gl.bindTexture(gl.TEXTURE_2D, texture);
    // setting the parameters to be able to render any image,
    // regardless of its size
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
    // loading the original image as a texture
    gl.texImage2D(
      gl.TEXTURE_2D,
      0,
      gl.RGBA,
      gl.RGBA,
      gl.UNSIGNED_BYTE,
      texture.image
    );
    gl.drawArrays(gl.TRIANGLES, 0, 6);
  };
}

function compileShader(gl, type, shaderSource) {
  const shader = gl.createShader(type);
  gl.shaderSource(shader, shaderSource);
  gl.compileShader(shader);

  const outcome = gl.getShaderParameter(shader, gl.COMPILE_STATUS);

  if (outcome === false) {
    // logging the error message on failure
    console.error(gl.getShaderInfoLog(shader));
    gl.deleteShader(shader);
  }

  return shader;
}

function createProgram(gl, vertexShader, fragmentShader) {
  const program = gl.createProgram();
  gl.attachShader(program, vertexShader);
  gl.attachShader(program, fragmentShader);
  gl.linkProgram(program);

  const outcome = gl.getProgramParameter(program, gl.LINK_STATUS);

  if (outcome === false) {
    // logging the error message on failure
    console.error(gl.getProgramInfoLog(program));
    gl.deleteProgram(program);
  }

  return program;
}

The filterImage() function is where the magic happens. It takes the following three parameters:

canvas: an Element object representing an HTML canvas element
originalImage: an Element object representing an HTML img element
kernel: an array containing the values of the kernel to use in the convolution operation

The first part of the function takes care of extracting the WebGLRenderingContext object representing a three-dimensional rendering context from the canvas element. Then, the vertex shader and fragment shader functions are defined as strings written in GLSL. Next, they are compiled and finally used to create a WebGL program.

The last part of the function creates a texture starting from the original image passed as a parameter and passes it to the WebGLRenderingContext to produce the final result. This represents the filtered image and is finally displayed by the browser in the canvas element.

The results produced by the filterImage() function depend on the kernel chosen, as you can verify by playing with the live demo you can find at the beginning of the article.

Et voilà! You just learned how to filter images in WebGL!

Final Considerations

As shown above, you can filter images with WebGL in Vanilla JavaScript with a hundred lines. At the same time, this cannot be considered an easy task to achieve. The reason is that getting into WebGL takes time and effort. Plus, you have to learn how to use GLSL to write the vertex shader and the fragment shader functions. So, things can get more complicated than expected.

Plus, you learned how to perform non-complex filters, but several filtering techniques are complex and do not involve the convolution operation. That also means that implementing them can be challenging and result in inefficient algorithms.

To avoid a headache, you should consider a complete and all-in-one solution like PhotoEditor SDK. Commercial solutions make things easier and shield you from all difficulties, offering features that would be complex, time-consuming, and challenging to implement. That is particularly true when it comes to using WebGL, which is PhotoEditor SDK’s main renderer.

You will be able to harness WebGL’s power without writing a single line of GLSL or knowing about its existence. WebGL will be used behind the scene for you! Keep also in mind that you would not be alone, since developers at IMG.LY are happy to provide support.

Filtering Images With PhotoEditor SDK

Read the article from the official documentation to learn how to get started with PhotoEditor SDK in HTML and Vanilla JavaScript. In detail, the Filters feature gives you more than 60 high-quality filters to play with. Achieve the following result with a few clicks:

Check out this feature on the PhotoEditor SDK demo page.

Conclusion

In this article, we looked at how to filter an image in WebGL with the HTML canvas element. Implementing a feature allowing users to filter images through a kernel-based approach in WebGL involves only a dozen of lines of code. However, understanding how WebGL works and coding in GLSL cannot be considered easy tasks.

As a result, you might want to avoid dealing with WebGL entirely. In this case, consider a commercial and easy-to-adopt solution using WebGL behind the scene – such as PhotoEditor SDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.‌

How To Add Overlays to a Video in React Native

Antonello — Mon, 28 Mar 2022 16:23:32 GMT

In this article, you will learn to add a text and image overlay to a video with JavaScript in React Native. All you need is React Native Video, the correct styling rules, and just a few lines of code.

By following this guide, you will easily achieve a logo and text overlay:

First, let’s learn how to implement a React Native component that allows you to add an overlay image and overlay text to a video.

What is React Native Video?

Mobile apps come with several native interfaces and features. This is what makes React Native different from React, which is based on browser rendering. So, <div> becomes <View>, <p> becomes <Text>, and so on. In particular, React Native supports only a limited amount of components. You can find the entire list here.

To this day, React Native does not have a native <Video> component. That is why react-native-video was born - a widely used library without considering alternatives. This 500kB package equips you with everything you need to get started with videos in React Native.

Prerequisites

Here is the list of all the prerequisites for the demo application you are going to build:

You can add react-native-video to your project’s dependencies by launching the following npm command:

npm install react-native-video

Or if you are a yarn user:

yarn add react-native-video

Then, follow this guide to learn how to link react-native-video to your project. This step is not required for Android users with React Native 0.60 and above.

Adding Overlay Image and Text to a Video in React Native

You can clone the GitHub repository supporting this article and launch the following commands to try the demo application:

git clone https://github.com/Tonel/how-to-add-overlays-to-videos-in-react-native-imgly
cd how-to-add-overlays-to-videos-in-react-native-imgly
npm i
npm start

The project is a React Native Create App project, and you can learn more about how it works and how to run it here.

Otherwise, keep following this step-by-step tutorial and learn how to build a React Native component for adding overlays to a video.

1. Initializing a React Native Project

Create React Native App is the officially supported and easiest way to create an empty and working React Native application. Install it globally with the following command:

npm install -g create-react-native-app

Now, you can initialize a new project called react-native-video-overlays-demo with the following command:

create-react-native-app react-native-video-overlays-demo

The react-native-video-overlays-demo folder should now contain a demo project with the following file structure:

react-native-video-overlays-demo
├── .buckconfig
├── .gitattributes
├── .gitignore
├── App.js
├── app.json
├── babel.config.js
├── index.js
├── metro.config.js
├── package.json
├── package-lock.json
├── yarn.lock
├── android
│   └── ...
└── ios
    └── ...

Enter the react-native-video-overlays-demo folder:

cd react-native-video-overlays-demo

Then launch the development server with one of the following commands:

for launching an Expo app

npm start

for Android development on your phone or an emulator

npm run android

for iOS development on your phone or an emulator

npm run ios

for react-native-web development

npm run web

2. Defining a Component to Add Overlays on a Video

Let’s see how to build a VideoWithOverlays component that allows you to add overlays such as text and an image to a video. You can implement it as follows:

import React, { useEffect, useState } from 'react';
import { Image, StyleSheet, Text, View } from 'react-native';
import Video from 'react-native-video';

export default function VideoWithOverlays({
  videoComponentProps,
  text,
  imageSrc,
}) {
  const {
    video,
    overlayText,
    videoWithOverlays,
    overlayTextView,
    overlayImageView,
    overlayImage,
  } = styles;

  const [height, setHeight] = useState(0);
  const [overlayHeight, setOverlayHeight] = useState(0);

  useEffect(() => {
    // setting the overlay height to 10px starting
    // from the bottom of the video
    setOverlayHeight(height / 2 + height / 3 - 10);
  }, [height]);

  return (
    <View style={videoWithOverlays}>
      <Video
        {...videoComponentProps}
        style={video}
        onLayout={(event) => {
          const { height } = event.nativeEvent.layout;
          setHeight(height);
        }}
      />
      <View style={{ ...overlayTextView, top: overlayHeight }}>
        {text && <Text style={overlayText}>{text}</Text>}
      </View>
      <View style={{ ...overlayImageView, top: overlayHeight }}>
        {imageSrc && (
          <Image style={overlayImage} source={imageSrc} resizeMode="contain" />
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  overlayTextView: {
    position: 'relative',
    alignItems: 'flex-end',
    justifyContent: 'flex-end',
    right: 5,
  },
  overlayImageView: {
    position: 'relative',
    alignItems: 'flex-start',
    justifyContent: 'flex-end',
    left: 5,
  },
  overlayImage: {
    position: 'absolute',
    height: 40,
  },
  overlayText: {
    fontSize: 25,
    fontWeight: 'bold',
    color: 'white',
  },
  video: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
    marginTop: '50%',
  },
  videoWithOverlays: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
  },
});

The react-native-video Video component is used to play the video and its props are exposed thanks to the videoComponentProps prop. Then, after being placed into the layout, its height depending on the size of the device screen is stored in the height variable. This is used to calculate the height the overlays should be placed at in the useEffect hook. In detail, the overlays are placed at 10px from the bottom edge of the video.

The first internal View represents the optional text overlay. The overlay logic depends entirely on the custom overlayTextView and overlayText StyleSheet fields, representing the CSS classes responsible for displaying the text passed as a prop with text as an overlay. By default, the text overlay is placed in the bottom-right corner of the video.

The second internal View represents the optional image overlay. The overlay logic depends entirely on the custom overlayImageView and overlayImage StyleSheet fields, representing the CSS classes taking care of showing the image passed as a prop with imageSrc as an overlay. By default, the image overlay is placed in the bottom-left corner of the video. As you can see, both overlays are optional.

3. Putting It All Together

Let’s see the VideoWithOverlays component defined above in action. All you need to do is replace the App.js file with this:

import React from 'react';
import { StyleSheet, View } from 'react-native';
import VideoWithOverlays from './src/components/VideoWithOverlays/VideoWithOverlays';

export default function App() {
  // a static image stored into the assets folder
  const imgLy = require('./src/assets/images/IMG_LY.png');

  return (
    <View style={styles.container}>
      <VideoWithOverlays
        videoComponentProps={{
          // replace this free video source with your video
          source: {
            uri: 'https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4',
          },
        }}
        // overlay text
        text={'IMG.LY'}
        // overlay image
        imageSrc={imgLy}
      />
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#fff',
    alignItems: 'center',
    justifyContent: 'center',
  },
});

Here, <VideoWitOverlays> is initialized with a free video source retrieved from a public URL and two overlays. The text overlay states “IMG.LY” and it is placed in the bottom-right corner of the video. While the image overlay represents the IMG.LY logo and is placed in the bottom-left corner of the video. This is a screenshot of the VideoWitOverlays in action:

Final Considerations

Adding overlays elements to a video in React Native is not complex and takes just a few lines of code. On the other hand, the approach presented above is just a frontend trick and does not modify the video. You are just showing it behind text or an image, giving the illusion to the users that it is a single source of content.

If you wanted to let users modify their videos by adding overlays or other elements and then export them, this would involve complex logic and require an advanced UI. Building such a React Native component is a time-consuming task that represents a waste of energy. If that is your goal, you should take into consideration a complete and ready-to-use SDK solution, such as VideoEditorSDK. This would provide your users with the ability to add overlays and use many other cool features.

Add Overlays to a Video With React Native module for VideoEditor SDK

Visit this page from the official documentation to learn how to get started with VideoEditorSDK in React Native. Then, load your video and start editing it by adding overlays, text, and stickers. Follow the previous links to learn more about each feature.

Download our free mobile demo app from the App Store or Google Play, and try an extensive set of tools to edit videos.

Conclusion

In this tutorial, we learned how to define a React Native component that allows you to reproduce a video and add overlays. Specifically, we used react-native-video and defined everything required to place text or an image on your video. However, if you intend to export your edit, you will have to build a complex video editing application. In this scenario, you will want to consider a fully-featured, cutting-edge, and easy-to-adopt solution – such as VideoEditorSDK.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

Tutorials – IMG.LY Blog

A Modern Video Editor SDK for your Flutter App

Why add a Video Editor to Your Flutter App?

Getting Started: Integrating the Video Editor in Flutter

Requirements

Setting Up the Editor

Use Cases: Building a TikTok-Like Experience

Short-Form Video Creation

Influencers and Marketing

eCommerce and User-Generated Content

Customization Options with CreativeEditor SDK

Conclusion

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Setting Up Your Project

Asking for Permissions

Making Video Recordings

Configuring the Camera

Presenting the Controller

Building Your Creation

Configuring the Editors Assets

Exporting the Finished Video

Setting Up a Playback Controller

Where to Go From Here

How to Add Stickers and Overlays to a Video in Flutter

How does Flutter handle video assets?

Prerequisites:

Adding Overlay Stickers and Text to a Video in Flutter

Summing up

Adding Overlays to a Video with Flutter package for VideoEditor SDK

How To Build a Video Editor With Wasm in React

What is WebAssembly?

What is ffmpeg.wasm?

Building a Video Editor in React with WASM

Prerequisites

Initializing a React Project

Installing the Project’s Dependencies

Uploading a Video to the Video Editor

Playing a Video in React

Cutting a Video in React

Converting a Video to GIF in React

Putting It All Together

Commercial Alternative

Conclusion

FFmpeg - The Ultimate Guide

Introduction to FFmpeg

Installing FFmpeg

FFmpeg history

FFmpeg supported codecs and formats

Compilation of FFmpeg

FFmpeg’s strengths

Filtering

Hardware acceleration

Versatile input/output methods

Running example commands

Introduction to media concepts

Audio

Sampling Rate

Bitrate

Channels

Image properties

Resolution

Bit-depth

Transparency

Video properties

Video codecs

Audio codecs

Containers

Example Material

FFplay and FFprobe

FFplay

FFprobe

FFmpeg concepts

Inputs

Streams

Outputs

Mapping

Filtering

FFmpeg’s command line system

FFmpeg CLI

Specifying an input file

What is `ffmpeg.wasm`?