Tech – IMG.LY Blog

How to Build a Cross-Platform Editing Experience for Web-to-Print

Jan — Wed, 19 Apr 2023 12:34:14 GMT

We have previously talked about the importance of taking your web-to-print app cross-platform. Whether you are operating a print-on-demand platform and you want to retain customers through a mobile app, or you are developing a physical print kiosk at your point of sale, you will not get around a native editing experience.

https://www.youtube.com/embed/cRqm9-c17-c?feature=oembed

Relegating the user experience on mobile to the back seat of your digital strategy or simply using it as a checkout front leaves real money on the table.

Customers looking for a shopping experience on mobile will not abandon the process in order to switch to the desktop to perform design tasks.

Unfortunately, it is not as easy as dropping in a web editor into your mobile app or on your website, the editing experience on the web will be massively degraded as compared with a native mobile design editor. Gesture support is limited, and interaction patterns just do not feel native, what is more, operating within the confines of the browser imposes memory constraints and the inability to take advantage of hardware acceleration.

Fortunately, IMG.LY just released full cross-platform support of the CreativeEditor SDK. You can now build on IMG.LY’s engine tech on the web, Android and iOS.

We are going to explore what goes into building a unified editing experience for a particular use case across these platforms while taking into account platform-specific patterns and usage habits.

But first, we need some background on the technology powering our cross-platform solutions.

The Foundation: Platform Agnostic Graphical Engine

The primary challenge of attempting to take a graphics processing SDK cross-platform is avoiding the need to rebuild the same technology on each platform. Instead, we aim to build on previous insights and experience and reuse as much technology as possible. After four years of research and development, IMG.LY’s Creative Engine forms one core piece of software built in C++ that implements the foundational building blocks of state-of-the-art graphics processing. Crucially, the engine can be easily ported to any platform.

This “build once, port anywhere” approach allows for the rapid rolling out of new platform support while ensuring feature parity and most essentially consistency.

Ensuring Consistent Cross-Platform Output

Consistency in this context means that design templates can be shared across platforms while always rendering the exact same output. This is challenging to accomplish if your rendering engine differs from platform to platform.

Something as seemingly innocuous as consistently rendering the same text regardless of platform turns out to be notoriously difficult.
In the context of print applications, this is the biggest technical hurdle to clear.

It is imperative for designs on mobile and the web to produce the same print-ready output file, such as a PDF. Furthermore, to be effective across platforms, users expect to save work in progress and resume it on another device. Therefore, we must support the ability to serialize and deserialize in-progress designs while ensuring consistency.

In addition to interoperability during the design process, it is often necessary to re-render user-generated designs. This is the case when designs are intended to be applied to much larger media than can be rendered on a device, such as high-resolution banners or posters. In this scenario, the serialized edits must be applied to the high-resolution medium on a server, adding yet another platform into the mix.

Designing UIs - Balancing Requirements and Constraints

To illustrate the various platform requirements and constraints involved in designing UIs, we developed starter kit editors using the CE.SDK engine for apparel design. Our apparel designer enables users to design the front of a t-shirt for deployment on web storefronts, mobile apps (Android and iOS), and print kiosks for in-store customization.

While we could have used our default Design UI and added a t-shirt backdrop, we opted for a more specific approach that prioritizes the features necessary for an optimal t-shirt design experience. Our UI places these features front and center, while also providing guardrails to prevent bad designs and hide unnecessary complexity.

Thankfully, the CE.SDK Engine allows for the creation of entirely custom UIs using any UI framework, thanks to its image and video manipulation API.

Our custom UI for editing t-shirts on the web can be used on both desktop and mobile web, as well as deployed to a print kiosk inside a web view. To create an optimal user experience, we identified the most important editing options for a t-shirt design in order of relevance: overlaying text, adding images, shapes, and stickers, and uploading new image assets.

It’s worth noting that users start from a template rather than a blank canvas. We strive to give users the best starting point for their creative journey, minimizing setup and menial work as much as possible. The same principle applies to discovering editing features. When a user selects an element, a context menu appears with essential editing options. For example, for a text element, options would include changing color, font, or alignment.

Template Creation on the Web

The most important difference between the web and mobile use cases is that templates will in almost all cases be created using the web interface, whereas users on mobile will simply adapt existing templates.

https://www.youtube.com/embed/VLBXMxQMKak?feature=oembed

This means that in addition to the user-facing apparel UI, we also need to ship a more complex UI allowing the creation of templates. Creating templates requires more granular control over an element’s design, as well as the ability to define placeholders and design constraints.

These templates can then be deployed to the simpler adoption UIs on mobile, where the users’ design is guided by those constraints.

Mobile UIs

Designing for mobile devices presents a unique set of challenges due to the limited screen size, reliance on gestures for interaction, and hardware and bandwidth constraints. To address these constraints, we take a strict template-first approach. Most users prefer to begin their work from a template or continue working on a design they started on their desktop. Thus, the first step in the user journey is browsing and selecting designs.

In addition, we need to consider mobile interaction patterns and be even more context-sensitive than we are on the web. For instance, in a print context, the asset size is typically larger than the device’s screen size. This size limitation affects the navigation pattern, and users need to zoom in by pinching to focus on smaller elements. However, we cannot use the pinch gesture to modify an element’s size because it conflicts with the zoom gesture. Instead, users must first select the element explicitly to activate it and then change its size via a pinch gesture.

Conclusion

Creating user interfaces for designing print products on the web and mobile that allow users to be productive is a complex task. We started out by considering the technical complexities involved in bringing any cross-platform editing solution to fruition, and went on to consider both the broad strokes workflow and the minute details of individual element manipulation.

You can adapt the cross-platform apparel editor that we built in this process and integrate it into your product. Hopefully, these example UIs form a useful foundation on which to build your use case.

Designing user interfaces that allow users to create print products on web and mobile platforms is a complex task that requires a thorough understanding of technical and UX complexities. Our team embarked on this journey with a focus on bringing a cross-platform editing solution to life. We delved into the intricate details of individual element manipulation and considered the broad strokes of template-based workflow design, so you don’t have to.

Through this process, we developed a cross-platform apparel editor that can be easily adapted and integrated into your product. Our aim is to provide you with a solid foundation that will enable you to build a UI that is tailored to your use case. With our example UIs as a starting point, you can develop a product that empowers your users to create and customize print products with ease.

In conclusion, designing a user interface that allows for print product creation on web and mobile platforms is a complex undertaking. However, with the right approach and attention to detail, it is possible to create a UI that is both intuitive and efficient. Our team has provided a starting point with our cross-platform apparel editor, and we hope that it serves as a useful foundation for your own UI development efforts.

Reach out to our solutions team to explore your particular use case.

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Walter — Thu, 30 Mar 2023 11:20:26 GMT

Now that TikTok is facing a ban in the US any day now, we better wait in the wings with an alternative ready to go and scoop up those millions of hobby dancers, micro bloggers, and would-be influencers.

In this article, you’ll learn how to use Swift, SwiftUI, and the IMG.LY CreativeEditor SDK to build a simple iOS app for recording, editing, and viewing short videos. This app features views providing core functionalities similar to making a post in a video-sharing platform like TikTok.

The editing controller is the central hub. It allows users to capture video clips and enhance their videos by adding filters, stickers, music, and other effects. CreativeEditor SDK calls the overall project a “scene” and the parts (clips, stickers, text, etc.) “blocks”. Once the scene is ready, the user can compose it into a standard video file and export it. Finally, the playback controller showcases the finished projects using Apple’s standard VideoPlayer struct. For recording and editing, the CreativeEditor SDK’s Camera and Video Editor handle the heavy lifting, enabling swift development of a robust app. You can easily extend its features, filters, and creative options as your users demand.

Follow along to see how to capture video, enhance it, and export the finished product as a polished movie file.

Setting Up Your Project

You can try out the TikTok clone by cloning the GitHub repository supporting the article. Otherwise, follow this step-by-step tutorial and learn how to build the TikTok clone with CreativeEditor SDK by yourself.

To add the CreativeEditor SDK to an Xcode project, include it with your other package dependencies. Use the URL below to add the package, either to your Package.swift file or using File > Add Packages...

https://github.com/imgly/IMGLYUI-swift

The package includes a number of different frameworks:

IMGLYUI, an umbrella framework of all of the different editors and a camera.
IMGLYCamera, a standalone camera View. This is the same camera as the editors use.
IMGLYApparelEditor, IMGLYDesignEditor, IMGLYPostcardEditor, and IMGLYVideoEditor which are four different configurations to support different use cases.

Each of these frameworks uses the IMG.LY Engine for image processing. To include them in your project, navigate to the ‘Frameworks, Libraries, and Embedded Content’ section in your app’s General settings. Click the ’+’ button and select the desired frameworks. Initially, consider adding the IMGLYUI package, as it encompasses all other packages. Before releasing your app, review and include only the necessary components to minimize app size and avoid unused code.

Once Xcode downloads and resolves the packages, you just need to include IMGLYVideoEditor in any of the files that will reference the editor in your app. When you’re working with the underlying engine directly you’ll also want to include IMGLYEngine. You’ll only work with the engine as you start to customize the workflow.

Asking for Permissions

Before any app can access the phone’s microphone, camera, documents or photo library, the user must specifically give permission to record audio or video as well as access the user’s Photo library. If your app doesn’t secure these permissions properly before trying to use the camera the app will crash and also probably won’t get through Apple’s app review. In earlier versions of iOS you could develop an app on the Simulator without asking permissions, but always on the device the app will crash. In current versions of iOS the app crashes regardless of platform. The dialogue requesting permissions is a system dialog and looks like this:

You do not have the ability to change this dialog. You can supply the reason you are asking for the permission using a key in the info.plist file in the Xcode project. In the example above “Lets you record videos” is in the info.plist. For the video camera you will have to ask for video and audio permission. The first step is to add a short message to the user in the info.plist.

For video the key to add is NSCameraUsageDescription and for the microphone you need to add NSMicrophoneUsageDescription. Whenever your app attempts to use the camera, iOS will first check to see if the user has already granted access. If not, it will then display the dialogs using your entries in the info.plist or crash if you have not provided entries. The user might be surprised by these dialog boxes and may accidentally tap Don't Allow if they are trying to quickly launch the camera. It is better practice to set up some kind of onboarding view and secure the permissions before you need them. You might have a view that explains what you are going to ask for and then displays the permission dialog.

switch AVCaptureDevice.authorizationStatus(for: .video) {
  case .authorized:
    //the user has authroized permission!
    break
  case .denied:
    //the user has previously denied permission!
    //perhaps we should ask them again
    break
  case .restricted:
    //the user cannot grant permission!
    break
  case .notDetermined:
    //the user has never been asked for permission
    //so let's ask them now
    AVCaptureDevice.requestAccess(for: .video) { accessGranted in
      if accessGranted {
        //the user has authorized permission!
      } else {
        //the user has denied permission!
      }
    }
  @unknown default:
    break
}

The snippet above lets your app read the permission status for the video camera and ask for permission if it has not been granted. To request access to the microphone pass .audio to the AVCaptureDevice.authorizationStatus(for:) and AVCaptureDevice.requestAccess(for:) functions. The AVCaptureDevice.requestAccess(for:) command is what displays the system dialog actually requesting access. The accessGranted variable reports back to your app what the user chose. You can take care of getting the permissions any time but when the CreativeEditor SDK first launches it will attempt to access the camera, so the dialog will appear if the user has not already granted permission.

The .restricted case is fairly new. In this case, there are policies on the phone that prohibit the user from granting permission to your app. In addition to asking permissions during onboarding, it is good practice to check for permission every time before you attempt to run any code that uses the camera. The user can change permissions using the Settings app on their phone at any time. If the user has denied permissions and you present the video camera anyway, your app will record black video frames and silence on audio tracks.

In addition to asking for camera and microphone permissions, your app will probably want to access the photos on the user’s phone. You will need to add an entry to the info.plist for NSPhotoLibraryUsageDescription. As with the camera and the microphone, the dialog will appear when the Video Editor first attempts access, but you can give your user some ability to grant permission during onboarding. For the user’s photos, you can check the authorization status using

let status = PHPhotoLibrary.authroizationStatus(for: .readWrite)

As with the camera, the photo library has PHPhotoLibrary.requestAuthroization(for: .readWrite) but instead of just returning a “granted” or “denied” status, this command returns the actual new status. In addition to the status values for the camera, the photo library may return a .limited status meaning that the user has granted permission to only some of the photos. If the user has chosen to share only some of their photos, Apple provides some views you can present so that the user can manage the photos. Any videos that your app saves to the user’s photo library will always be available when the user has chosen .limited. You can read more about how to work with permissions and the user’s photo library by reading this Apple article Delivering an Enhanced Privacy Experience in Your Photos App.

Making Video Recordings

The start of a great TikTok is the camera. When our user wants to make a new video clip with our app, they tap on the button at the bottom of the initial screen to start the creation workflow.

Here there is a decision to make. TikTok and the CreativeEditor SDK can each start with the creation controls and a video preview. When using the CreativeEditor SDK though, you can also start with some video that came from somewhere else. So if you already have some camera code you like, or if you want to start with some video clip from somewhere else, you can insert that into the editor when it launches. You can do that by leveraging the onCreate modifier of the editor.

let engine = EngineSettings(license: "<your license code>",
                              userID: "<your unique user id>")

VideoEditor(engine)
  .imgly.onCreate({ engine in
    try await engine.scene.create(fromVideo: Bundle.main.url(forResource: "dog_water", withExtension: "mov")!)

    //set other defaults
  }

In the code above, the app initializes an instance of the Video Editor view and then reads the dog_water.mov file from the app bundle. launches the editor with that.

Whether starting with a video or starting with a blank canvas a user can add video clips to their creation using the embedded camera.

The image above compares two different camera controllers: the left one from the TikTok app and the right showing the default settings of the MobileCamera, either standalone or within a Video Editor scene. Key controls in each are marked with blue numbers:

1 - Flip camera button
2 - Flash button
3 - Recording time indicator
4 - Add video and finish buttons
5 - Cancel button

While the TikTok camera integrates several editing functions during video capture, the Video Editor separates these tasks, focusing the camera purely on recording. Both systems allow users to start and stop video recording to compile a series of clips before moving to the editing phase.

Configuring the Camera

The camera used inside of the Video Editor UI isn’t customizable. You have limited options to customize when you use the standalone camera though. You can set the colors of the buttons, helpful if you have a preferred color scheme, and you can set a maximum duration for the video recording allowed.

let config = CameraConfiguration(
  recordingColor: .orange,
  highlightColor: .blue,
  maxTotalDuration: 30,
  allowExceedingMaxDuration: false)

Camera(engine, config: config) { result in
 //handle the recorded video collection
}

The code above sets the record button to orange while recording and the clip view to orange. There is a maximum duration of 30 seconds for all videos in the collection and the button to save the clips highlighted in blue.

If your app requires further customization, you’ll probably want to make your own camera view and work with the IMG.LY Creative Engine directly.

The documentation website is a great resource for understanding how much you can add to and customize the IMGLYUI resources and when you’ll need to drop down to the level of the engine.

Presenting the Controller

As demonstrated above, both the Video Editor and the Camera are View structures that can be presented directly or through overlays such as overlay or fullScreenCover. However, they are typically enclosed in a NavigationView to utilize the navigation bar for displaying export and other buttons. Therefore, when triggering the Video Editor from a button, it’s crucial to embed it within a NavigationView.

Button("Edit the Video") {
  isPresented = true
}
 .fullScreenCover(isPresented: $isPresented) {
   NavigationView {
     VideoEditor(engine)
      //any setup modifiers
   }

Building Your Creation

After capturing a video, the app displays an editor to complete the creation. The editing tools are in the red circles. TikTok provides about ten different tools on the editing screen. Additionally, TikTok provided some editing tools on the capture screen. The CreativeEditor SDK apps provide a scrolling set of asset libraries along the bottom for items to add to the scene. But when any of the elements of the scene, like your video or an audio clip, are highlighted, the bottom row becomes tools to manipulate that element. The basic Asset Library types are uploads, videos, audio, images, text, shapes, and stickers.

How to configure and customize each of these tools is beyond the scope of this tutorial. But, a lot of the customization of the assets, fonts, blurs and even filters can be done without code. By design, the CreativeEditor SDK downloads assets from a central server. This is a great way for you to be able to push updated filters, stickers, and other assets to your users without going through the app update process. In the next section, we’ll walk through that and provide some links for further research.

Configuring the Editors Assets

The IMG.LY Engine looks for assets for stickers, vector shapes, LUT filters, color filters, color palettes, blurs, and fonts using a URL by default. The design of the system is that you would have your own server supporting your app. However, assets in the app bundle are also accessible by URL, so as long as the assets are in the folder format that the engine expects, it doesn’t matter whether they are local or remote.

In the documentation for this process you can find a URL to the default asset bundle. Once you download that bundle you can add it to your app. The bundle is structured much like a regular iOS Assets.catalog with some JSON containing metadata about the asset and a file that provides the actual asset. For some of the assets like filters and vectors, no file is needed as everything can be defined in the JSON.

In the image above, you can see the JSON to describe the ape sticker and the emoji_ape image file after the IMGLYAssets bundle has been added to our app. You can edit these assets and add your own.

Once you’re done editing the asset bundle, you can use it instead of the default bundle during the onCreate modifier like this:

VideoEditor(engine)
  .imgly.onCreate({ engine in
     try await engine.scene.load(from: VideoEditor.defaultScene)
              // Add asset sources
     let bundleURL = Bundle.main.url(forResource: "IMGLYAssets", withExtension: "bundle")!
     try await engine.addDefaultAssetSources(baseURL: bundleURL)
     try await engine.addDemoAssetSources(sceneMode: engine.scene.getMode(),
       withUploadAssetSources: true)
     try await engine.asset.addSource(TextAssetSource(engine: engine))
  })

In the code above, we create a blank scene using VideoEditor.defaultScene static method. Then set a URL to point to the bundle in the app instead of a remote bundle and make that the source of addDefaultAssetSources. Now our app will use the assets from the bundle.

Exporting the Finished Video

Once the user has finished with their creation, they tap the share button and the Video Editor composes all of the video, audio, filters, and static items into a single .mp4 file. Then it displays a share sheet. If you’d rather do something different, like save the creation locally so you can play it back, you can override this default behavior.

In your app, after the .onCreate modifier, you can define a .onExport modifier. When the user taps on the share button you can export the video and then save it to the documents directory to playback later.

In the documentation for modifiers you can find the source code for the Video Editor’s default behavior. Instead of duplicating it in detail here, we can just summarize and provide a little snippet for our change.

First, the .onCreate handler takes an engine and a eventHandler parameter. The engine is the underlying IMG.LY Engine object we’ve been using. The eventHandler lets us send information back to the Video Editor UI about the progress or any errors.

In the documentation, there is a helper function to export the video. After it runs it returns the data of the exported video and a mimeType, which will be .mp4.

let data: Date, mimeType: MIMEType
let mode = try mainEngine.scene.getMode()
guard mode == .video else { throw CallbackError.unknownSceneMode; return }
(data, mimeType) = try await exportVideo()

The code in the documentation handles static files from the DesignEditor as well as videos from the VideoEditor. The above code makes it so that only video can be exported, since our app only works with video.

In the exportVideo function, the code first checks to make sure it has a properly formed scene object to work with. Then it starts the export using mainEngine.block.exportVideo(_, mimeType:). This begins an async throwing stream during the export. The VideoExport objects in the stream are either .progress or .finished. When a .progress object arrives, the exportVideo function sends an update to the eventHandler to display in the UI. When the .finished object arrives, it has an associated object that is the data for the video.

Now instead of opening the share sheet, we can write the video to the documents directory for the app.

do {
      guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
        else {
          logger.error("Documents directory not found"); return
        }

  let filePath = documentDirectory.appending(component: UUID().uuidString)
  try data.write(to: filePath)
} catch (let error as NSError) {
  logger.error("could not write finished file: \(error.localizedDescription)")
}

eventHandler.send(.exportCompleted {})

The code above takes the data from the exportVideo function and writes it to the app’s documents directory, giving it a UUID string value as a filename. Then it updates the UI to let it know the export is done but sends an empty action block.

Setting Up a Playback Controller

For this example app, the playback controller will play any clips in the app’s Documents directory. Each clip plays on a loop. The user can swipe up to get the next clip and tap to pause or restart the clip. If there are no clips, the user will be encouraged to make a new one.

When the video player view appears, the first task is to see if there are any videos to load. Any video files in the Documents directory are assumed to be ready to play.

func loadVideos() {
  //get a handle to the documents directory for this app
  guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true) else { logger.error("Documents directory not found"); return}

  if let files = try? FileManager.default.contentsOfDirectory(at: documentDirectory, includingPropertiesForKeys: nil) {
    videos = files
  }
}

The code above reads all of the filenames from the documents directory into a files array and then assigns that array to a videos variable.

Each time the app is launched or the VideoEditor view is dismissed, the video player view will get the onAppear message. This is a good place to check for videos using the loadVideos function. After the videos are loaded, the app can start to play the first one.

func setupVideoPlayer() {
  guard let currentVideo = videos.first else {logger.error("No videos to play"); return}

  self.shouldHideEmptyDirectoryText = true

       let endMonitor = NotificationCenter.default.publisher(for: NSNotification.Name.AVPlayerItemDidPlayToEndTime)

  self.player = AVPlayerItem(url: currentVideo)
  self.currentVideo = currentVideo

}

In the code above, we first check to see if there are any video files. If there are, then hide the message about videos and create a new AVPlayerItem using the URL of the first video. The endMonitor, shouldHideEmptyDirectoryText, and player are both being watched by the View. The AVPlayer gets rendered in a regular SwiftUI View.

VideoPlayer(player: player)
  .frame(width: 640, height: 360)
  .onAppear {
    player.play()
  }
  .onReceive(endMonitor) { _ in
    player.seek(to: .zero)
    player.play()
}

When the video reaches the end, it will seek(to: .zero) which rewinds the video and loops it.

In the TikTok app, you can advance to the next video by swiping up. Additionally, you can start and stop any video by tapping on the screen. We can add both of those features using gesture recognizers.

The code for play and pause is attached to a Tap Gesture modifier we can append to the player.

.onTapGesture {
  if player.rate > 0.0 {
    player.rate = 0.0
  } else {
    player.rate = 1.0
  }
}

We can just adjust the playback rate. A value of 1.0 is a normal speed and 0.0 is paused. The player also has a .pause and .play methods, but sometimes these seem buggy.

Swipe is a little more difficult since SwiftUI only has swiping on List items. So we can use a drag gesture and then evaluate the translations when it’s complete.

.gesture(
    DragGesture(minimumDistance: 3.0, coordinateSpace: .local)
        .onEnded { value in
            // Check for a vertical, upward swipe with minimal horizontal deviation
          if abs(value.translation.width) < 100 &&
             value.translation.height < 0 {
                // It was an up swipe!
		viewModel.advanceVideo()
            }
        }
)

Then advancing the video is a simple matter of refreshing the player

func advanceVideo() {
  guard let currentVideo = self.currentVideo else { logger.error("No current video."); return}

  let currentIndex = videos.firstIndex(of: currentVideo)
  var nextVideo = videos.index(after: currentIndex!)
  if nextVideo == videos.count {
    nextVideo = 0
  }

  self.currentvideo = videos[nextVideo]
  self.player = AVPlayer(url: self.currentVideo!)

}

This code will get the next file URL from the array of videos. When it reaches the end, it will loop back and get the file at index 0. Then it creates a new player. Because the view is observing that player variable, it will update with the new video.

With this view, the user can create new videos by tapping the creation button and swipe to view all of their created videos.

Where to Go From Here

This tutorial has focused on how the CreativeEditor SDK can help you quickly make a video creation app like TikTok. Good next steps would be to further customize the editing tools and build out the network for sharing and tagging videos. Something that is important to consider is the data structure for each video. The iOS system is optimized to read only as much of a video file off of disk as is needed at any moment. Your app should use those optimizations to run faster. So, don’t load the whole video into memory as a Data object. Your data structures should keep the large video files somehow separate from the much smaller metadata (likes, comments, etc.). Consider storing the URL or filename of the video in the same object as the likes and comments. This will also allow you to cache video files after they have been downloaded so that you don’t need to redownload them when other data such as comments or number of likes changes.

Thanks for reading! We hope that you’ve gotten a better understanding for how a tool like the CreativeEditor SDK can bring your ideas to market faster. Feel free to reach out to us with any questions, comments, or suggestions.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

To stay in the loop, subscribe to our Newsletter or follow us on LinkedIn and Twitter.

How to Add Stickers and Overlays to a Video in Flutter

Natalia — Fri, 24 Mar 2023 09:32:33 GMT

When working with interactive content, it may happen that you will need to add remote or local assets on top of your video file. The assets varying from images and stickers to videos and fonts are usually grouped and deployed within your application and are available at runtime. This tutorial will teach you how to apply stickers and overlays to a video in Flutter and make your video content more personalizable and user-oriented. We start by considering the prerequisites for successful video integration, then talk about how Flutter manages overlays in order to personalize our video file.

How does Flutter handle video assets?

Unlike images, Flutter displays a video file by employing a special video_player plugin that provides playback, stores a file, and manages speed and sound control. For iOS-based applications, the video is integrated via AVPlayer, whereas the Android system employs ExoPlayer. Crucially, to create a simple video player, one should follow the steps below:

First, to ensure that pubspec.yaml file contains the video_player dependency;
Then permit access to videos for the application;
Make a VideoPlayerController and set it up;
Finally, don’t forget to display and play the video player;

So, let’s closely look at the setup process.

Prerequisites:

We start by adding the following dependencies of video_player plugin to pubspec.yaml file.

dependencies:
  flutter:
    sdk: flutter
  video_player: ^2.4.7

Afterwards, change configurations of both android and ios systems to guarantee that an application has the proper authorization for video streaming: e.g. for Android-oriented programs, proceed to the <project root>/android/app/src/main/AndroidManifest.xml directory – where it’s necessary to add the next line to the AndroidManifest.xml.

<manifest xmlns:android="<http://schemas.android.com/apk/res/android>">
    <application ...>

    </application>

    <uses-permission android:name="android.permission.INTERNET"/>
</manifest>

For iOS-based apps the Info.plist file is stored at <project root>/ios/Runner/Info.plist. Once again, don’t forget to add the following specification to the original code:

<key>NSAppTransportSecurity</key>
<dict>
  <key>NSAllowsArbitraryLoads</key>
  <true/>
</dict>

Successful video integration in Flutter is managed through the VideoPlayer widget and a VideoPlayerController, which sets the connection to the asset and initialize the controller for playback.

/// Stateful widget to fetch and then display video content.
class VideoApp extends StatefulWidget {
  const VideoApp({Key? key}) : super(key: key);

  @override
  _VideoAppState createState() => _VideoAppState();
}

class _VideoAppState extends State<VideoApp> {
  late VideoPlayerController _controller;

  @override
  void initState() {
    super.initState();
    _controller = VideoPlayerController.asset('assets/bee.mp4')

      /// Specify the path to your video asset here
      ..initialize().then((_) {
        /// Ensure the first frame is shown after the video is initialized,
        /// even before the play button has been pressed.
        setState(() {});
      });
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }
}

Note that video tends to take up as much space on the screen as possible by default, which can significantly deteriorate the video’s quality. Therefore, the Flutter team suggests employing the AspectRatiowidget to adjust the video proportions.

? AspectRatio(
    aspectRatio: _controller.value.aspectRatio)

Adding Overlay Stickers and Text to a Video in Flutter

To see the complete code of our demo application, you can clone the GitHub repository or put the following command to your Terminal:

git clone git@github.com:nataliakzm/Adding_Stickers_and_Overlays_to_video_in_Flutter.git

Similarly to how we handled resizing images in Flutter, the same method is used when overlaying an object on a video. The above example consists basically of three containers, one with a sticker, another with the textual watermark and the last with a local video asset. However, the critical difference here is that a Container filled with a sticker and a Container filled with a textual watermark cannot be simply listed one by one in your code but rather must be both placed within a Stack widget.

The Stack container is designed as a “mother”-widget which retains multiple layers of widgets on the screen. Stack structures these branches of children into a hierarchical system from bottom to top. Thus, the uppermost widget goes to the background, and the bottommost item is displayed in the foreground.

Stack(
  children: <Widget>[
    BottomWidget(),
    MiddleWidget(),
    TopWidget(),
  ]),

Since the size of the Stack widget is aimed to be the largest size among layers, we will place our video asset on this layer. Then, to implement the overlays, it’s essential to position and/or align each child’s container.


child: Stack(children: [
  VideoPlayer(_controller),

  /// Don't forget to align the position of the Сontainer
  Positioned( bottom: 5, left: 10,
      child: Container( width: 80, height: 40,

        /// In case you want to check the position/size of the Container uncomment the next line
        //color: Color(0xff0360da),
        child: Align(
            alignment: Alignment.center,

            /// Integrate Sticker overlay into a video
            child: Image.asset('assets/sticker.png')))),

  Positioned( top: 5, right: 10,
      child: Container( width: 80, height: 40,
        child: Align(
            alignment: Alignment.center,

            /// Integrate Text overlay into a video
            child: Text("IMG.LY",
                style: TextStyle(
                    color: Colors.white,
                    fontSize: 20,
                    fontWeight: FontWeight.bold))))),
                        ]))

  /// If there is no video, a blank page will be return   : Container()),

Summing up

Thus, you can see from the code below that integrating multiple layers over interactive assets can be a tricky task which requires bearing in mind the position and order of each layer, as well as managing how the children are aligned to a video file and the screen size of the user. Conversely, the discussed method allows personalizing an application without significant modifications to the original file.

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';

/// Import video_player package to integrate a video asset
import 'package:video_player/video_player.dart';

void main() => runApp(const VideoApp());

/// Stateful widget to fetch and then display video content.
class VideoApp extends StatefulWidget {
  const VideoApp({Key? key}) : super(key: key);

  @override
  _VideoAppState createState() => _VideoAppState();
}

class _VideoAppState extends State<VideoApp> {
  late VideoPlayerController _controller;

  @override
  void initState() {
    super.initState();
    _controller = VideoPlayerController.asset('assets/bee.mp4')

      /// Specify the path to your video asset here
      ..initialize().then((_) {
        /// Ensure the first frame is shown after the video is initialized,
        /// even before the play button has been pressed.
        setState(() {});
      });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
        debugShowCheckedModeBanner: false,
        home: Scaffold(
            appBar: AppBar(
                title: Text('How to add Stickers and Overlays'),
                backgroundColor: Color(0xff0360da)),
            body: Center(
                child: _controller.value.isInitialized

                    /// First, we specify the AspectRatio of a video frame
                    ? AspectRatio(
                        aspectRatio: _controller.value.aspectRatio,

                        /// Then, we initialize a Stack widget
                        child: Stack(children: [
                          VideoPlayer(_controller),

                          /// Don't forget to align the position of the Сontainer
                          Positioned( bottom: 5, left: 10,
                              child: Container( width: 80, height: 40,

                                  /// Uncomment the following line to check the position/size of the Container
                                  //color: Color(0xff0360da),
                                  child: Align(
                                      alignment: Alignment.center,

                                      /// Integrate Image overlay into a video
                                      child: Image.asset('assets/sticker.png')))),

                          Positioned( top: 5, right: 10,
                              child: Container( width: 80, height: 40,
                                  child: Align(
                                      alignment: Alignment.center,

																			/// Integrate some Text overlay into a video
                                      child: Text("IMG.LY",
                                          style: TextStyle(
                                              color: Colors.white,
                                              fontSize: 20,
                                              fontWeight: FontWeight.bold))))),
                        ]))

                    /// If there is no video, a blank page will be return
                    : Container()),
            floatingActionButton: FloatingActionButton.extended(
                backgroundColor: const Color(0xff0360da),
                foregroundColor: Colors.white,
                onPressed: () {
                  setState(() {
                    _controller.value.isPlaying
                        ? _controller.pause()
                        : _controller.play();
                  });
                },
                icon: Icon(_controller.value.isPlaying
                    ? Icons.pause
                    : Icons.play_arrow),
                label: Text('Play for IMG.LY'))));
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }
}

Adding Overlays to a Video with Flutter package for VideoEditor SDK

However, if you would like to offer more advanced sticker and overlay functionality to your user and you might need a more complex UI structure. Since this can be an extremely time consuming undertaking you might want to opt for a ready-to-use solution such as VideoEditor SDK which among a host of other essential video editing features allows users to easily add stickers and different overlays to their videos.

To replicate the example above, go to the official documentation to discover how to get started with VideoEditor SDK or follow our guide on how to integrate video editor for Flutter into your app. Try to upload your video file and modify it with various stickers, text and other overlays. Alternatively, you can download our free mobile demo app from the App Store or Google Play, and test a comprehensive set video editing tools.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

How To Build a Video Editor With Wasm in React

Antonello — Mon, 16 Jan 2023 16:31:57 GMT

You may think that video processing can only be performed on a server, but this is not true. Thanks to WebAssembly (Wasm), you can run high-performance code written in C or C++ in your browser. This opens up a lot of possibilities and gives you the ability to build a client-side video editor. Let’s now learn how to create a Wasm-based video editor in React.

Follow this tutorial and learn how to build the application below:

This React video editor allows users to upload a video, select a portion of it, convert it to GIF, and download the resulting image file – all of this in your browser.

What is WebAssembly?

WebAssembly (also called Wasm) is a new type of code that modern web browsers can run and understand. Specifically, Wasm provides new functionality to web development and brings significant performance benefits. This applies to both frontend and backend, since Wasm can be used by both clients and servers.

Wasm code should not be written by hand. This is because Wasm is a binary instruction format. So, Wasm is designed to be a compilation target for source languages such as C, C++, Rust, and others. Learn more about Wasm on the WebAssembly MDN Web Docs page.

Wasm provides a way to run code written in several languages on the Web at near-native speed. This means that WebAssembly forms can be easily imported and executed by a Web client or server application. In other words, you can use WebAssembly functions via JavaScript and achieve results that were not previously possible on a browser, especially in terms of performance.

What is `ffmpeg.wasm`?

As stated on the GitHub page of the project, ffmpeg.wasm is a WebAssembly and JavaScript port of FFmpeg. If you are not familiar with it, FFmpeg is an open-source software suite that includes several libraries and programs for managing audio, video, streams, and other media files. All of these tools can be invoked and executed via the ffmpeg command-line instruction.

Therefore, ffmpeg.wasm enables video and audio processing in JavaScript-based web applications. In detail, it enables video and audio recording, format transcoding, video and audio editing, and video scaling. Even though it is a WebAssembly-based library, you can use it in your JavaScript code just like any other npm library. This is the power of Wasm.

Since it transpiles to Wasm, you can take advantage of ffmpeg.wasm directly in your browser without performance concerns. This means that ffmpeg.wasm opens the door to client-side audio and video processing. Let’s now learn how to use ffmpeg.wasm to build a client-side video editor in React!

Building a Video Editor in React with WASM

In this step-by-step tutorial, you will learn how to build a Wasm-based video editor in React with ffmpeg.wasm. This React video editor application will allow you to upload a video, trim it, and convert it to GIF in the browser, without using any backend functionality or external API.

Clone the GitHub repository that supports the tutorial and try the video editor application by launching the following commands:

git clone https://github.com/Tonel/video-editor-wasm-react
cd video-editor-wasm-react
npm install
npm run start

Let’s waste no more time and see how to build a video editor application in React.

Prerequisites

This is the list of libraries the video editor application you are about to build depends on:

Node.js and npm 8+ and higher
React 18.2+
[ffmpeg.wasm](https://github.com/ffmpegwasm/ffmpeg.wasm)
[antd](https://www.npmjs.com/package/antd) >= 4.3
[video-react](https://www.npmjs.com/package/video-react) >= 0.15

antd is one of the most popular UI libraries for React. In this tutorial, its [Slider](https://ant.design/components/slider/) component will be used to implement the video cutting feature. So, any other UI library including a slider component with a range option will do.

video-react is one of the most advanced and reliable HTML5 video players for React. You can replace it with any other React video player library.

Initializing a React Project

Let’s initialize a new Create React App React project called video-editor-wasm-react with the command below:

npx create-react-app video-editor-wasm-react

Your video-editor-wasm-react directory should now contain the following files:

video-editor-wasm-react
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Enter the video-editor-wasm-react folder in your terminal and launch a React local server with the commands below:

cd video-editor-wasm-react
npm start

Now, visit the [http://localhost:3000/](http://localhost:3000/) page in your browser, and you should be seeing the default Create React App screen.

Before installing ffmpeg.wasm, you need to get your server ready. As stated in the official documentation, fmmpeg.wasm depends on [SharedArrayBuffer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer). This means that only browsers that support SharedArrayBuffer can run ffmpeg.wasm. You can find a complete list of all browsers here.

In the most popular browsers, SharedArrayBuffer is only available to cross-origin isolated webpages. To enable this, you need to set the following headers in your server hosting the React application:

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

In a Create-React-App application, you achieve this by creating a src/setupProxy.js file and making sure it contains the following code:

module.exports = function (app) {
    app.use(function (req, res, next) {
        res.setHeader("Cross-Origin-Opener-Policy", "same-origin")
        res.setHeader("Cross-Origin-Embedder-Policy", "require-corp")
        next()
    })
}

Your local development server is now ready to use ffmpeg.wasm!

Installing the Project’s Dependencies

It is now time to add the aforementioned required dependencies to your project.

First, install ffmpeg.wasm with the following command:

npm install @ffmpeg/ffmpeg @ffmpeg/core

Now, let’s install Antd Design:

npm install antd

Finally, it is time to install Video-React:

npm install video-react redux

Note that video-react also requires [redux](https://www.npmjs.com/package/redux) to work.

You have now everything you need to start developing your React video editor!

Uploading a Video to the Video Editor

Create a VideoUpload component as below:

// src/components/VideoUpload.js

import { Button, Upload } from 'antd';

function VideoUpload({ disabled, onChange = () => {}, onRemove = () => {} }) {
  return (
    <>
      <Upload
        disabled={disabled}
        beforeUpload={() => {
          return false;
        }}
        accept="video/*"
        onChange={(info) => {
          if (info.fileList && info.fileList.length > 0) {
            onChange(info.fileList[0].originFileObj);
          }
        }}
        showUploadList={false}
      >
        <Button>Upload Video</Button>
      </Upload>
      <Button
        danger={true}
        disabled={!disabled}
        onClick={() => {
          onRemove(undefined);
        }}
      >
        Remove
      </Button>
    </>
  );
}

export default VideoUpload;

As you can see, this component uses the Antd Design Upload component to allow users to upload a video file. Then, a “Remove” button gives users the ability to remove the uploaded video. Note that when the Upload component is enabled, the “Remove” button is disabled and vice versa. This way, users can deal with only one video at a time.

Playing a Video in React

You now need a VideoPlayer component to play the uploaded video. You can build it as below:

// src/components/VideoPlayer.js

import {
  BigPlayButton,
  ControlBar,
  LoadingSpinner,
  Player,
  PlayToggle,
} from 'video-react';
import 'video-react/dist/video-react.css';
import { useEffect, useState } from 'react';

export function VideoPlayer({
  src,
  onPlayerChange = () => {},
  onChange = () => {},
  startTime = undefined,
}) {
  const [player, setPlayer] = useState(undefined);
  const [playerState, setPlayerState] = useState(undefined);

  useEffect(() => {
    if (playerState) {
      onChange(playerState);
    }
  }, [playerState]);

  useEffect(() => {
    onPlayerChange(player);

    if (player) {
      player.subscribeToStateChange(setPlayerState);
    }
  }, [player]);

  return (
    <div className={'video-player'}>
      <Player
        ref={(player) => {
          setPlayer(player);
        }}
        startTime={startTime}
      >
        <source src={src} />
        <BigPlayButton position="center" />
        <LoadingSpinner />
        <ControlBar autoHide={false} disableDefaultControls={true}>
          <PlayToggle />
        </ControlBar>
      </Player>
    </div>
  );
}

This is a simple video-react-based video player component, which only allows playing or stop the video file received with the src prop.

Cutting a Video in React

Your goal is to allow users to select a portion of the video and watch it before converting it into GIF. Implement this with the logic below:

import { Slider } from "antd"
import { sliderValueToVideoTime } from "../utils/utils"

function VideoEditor() {
  const [sliderValues, setSliderValues] = useState([0, 100])

  // logic to handle videoFile, videoPlayer, and videoPlayerState...

  useEffect(() => {
      const min = sliderValues[0]
      // when the slider values are updated, updating the
      // video time
      if (min !== undefined && videoPlayerState && videoPlayer) {
          videoPlayer.seek(sliderValueToVideoTime(videoPlayerState.duration, min))
      }
  }, [sliderValues])

  useEffect(() => {
      if (videoPlayer && videoPlayerState) {
          // allowing users to watch only the portion of
          // the video selected by the slider
          const [min, max] = sliderValues

          const minTime = sliderValueToVideoTime(videoPlayerState.duration, min)
          const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max)

          if (videoPlayerState.currentTime < minTime) {
              videoPlayer.seek(minTime)
          }
          if (videoPlayerState.currentTime > maxTime) {
              // looping logic
              videoPlayer.seek(minTime)
          }
      }
  }, [videoPlayerState])

  return (
          // other video editor components...
          <Slider
            disabled={!videoPlayerState}
            value={sliderValues}
            range={true}
            onChange={(values) => {
              setSliderValues(values)
            }}
            tooltip={{
              formatter: null,
            }}
          />
        // ...
}

This component equips users with the ability to select a portion of the uploaded video with the Antd Design Slider component and watch it in loop, before converting it into a GIF file.

Specifically, the logic here uses the sliderValueToVideoTime() function below to transform the slider values into time coordinates:

// src/utils/utils.js

export function sliderValueToVideoTime(duration, sliderValue) {
    return Math.round(duration * sliderValue / 100)
}

Note that the actual video processing slider will be performed by the next component through ffmpeg.

Converting a Video to GIF in React

To convert a video to GIF, create a VideoConversionButton component as follows:

// src/components/VideoConversionButton.js

import { Button } from 'antd';
import { fetchFile } from '@ffmpeg/ffmpeg';
import { sliderValueToVideoTime } from '../utils/utils';

function VideoConversionButton({
  videoPlayerState,
  sliderValues,
  videoFile,
  ffmpeg,
  onConversionStart = () => {},
  onConversionEnd = () => {},
  onGifCreated = () => {},
}) {
  const convertToGif = async () => {
    // starting the conversion process
    onConversionStart(true);

    const inputFileName = 'gif.mp4';
    const outputFileName = 'output.gif';

    // writing the video file to memory
    ffmpeg.FS('writeFile', inputFileName, await fetchFile(videoFile));

    const [min, max] = sliderValues;
    const minTime = sliderValueToVideoTime(videoPlayerState.duration, min);
    const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max);

    // cutting the video and converting it to GIF with an FFMpeg command
    await ffmpeg.run(
      '-i',
      inputFileName,
      '-ss',
      `${minTime}`,
      '-to',
      `${maxTime}`,
      '-f',
      'gif',
      outputFileName
    );

    // reading the resulting file
    const data = ffmpeg.FS('readFile', outputFileName);

    // converting the GIF file created by FFmpeg to a valid image URL
    const gifUrl = URL.createObjectURL(
      new Blob([data.buffer], { type: 'image/gif' })
    );
    onGifCreated(gifUrl);

    // ending the conversion process
    onConversionEnd(false);
  };

  return <Button onClick={() => convertToGif()}>Convert to GIF</Button>;
}

export default VideoConversionButton;

This component uses ffmpeg.wasm to launch an ffmpeg command. This command takes care of clipping the video to the time limits defined by sliderValues and converting it to GIF. Note that all these operations are performed client-side in the browser.

In detail, the ffmpeg -ss flag defines the time from which to start reading the video file, while -to defines the end time. Then, the gif option specifies that you want to produce a GIF file. Learn more about the flags and options offered by ffmpeg here.

The resulting GIF image is stored in the output.gif memory file, loaded as a valid "image/gif" Blob, and finally converted to a valid image URL with the [URL.createObjectURL()](https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL_static) native function.

Putting It All Together

Now, let’s see the full code of the VideoEditor component:

// src/components/VideoEditor.js

import { createFFmpeg } from '@ffmpeg/ffmpeg';
import { useEffect, useState } from 'react';
import { Slider, Spin } from 'antd';
import { VideoPlayer } from './VideoPlayer';
import { sliderValueToVideoTime } from '../utils/utils';
import VideoUpload from './VideoUpload';
import VideoConversionButton from './VideoConversionButton';
const ffmpeg = createFFmpeg({ log: true });

function VideoEditor() {
  const [ffmpegLoaded, setFFmpegLoaded] = useState(false);
  const [videoFile, setVideoFile] = useState();
  const [videoPlayerState, setVideoPlayerState] = useState();
  const [videoPlayer, setVideoPlayer] = useState();
  const [gifUrl, setGifUrl] = useState();
  const [sliderValues, setSliderValues] = useState([0, 100]);
  const [processing, setProcessing] = useState(false);

  useEffect(() => {
    // loading ffmpeg on startup
    ffmpeg.load().then(() => {
      setFFmpegLoaded(true);
    });
  }, []);

  useEffect(() => {
    const min = sliderValues[0];
    // when the slider values are updated, updating the
    // video time
    if (min !== undefined && videoPlayerState && videoPlayer) {
      videoPlayer.seek(sliderValueToVideoTime(videoPlayerState.duration, min));
    }
  }, [sliderValues]);

  useEffect(() => {
    if (videoPlayer && videoPlayerState) {
      // allowing users to watch only the portion of
      // the video selected by the slider
      const [min, max] = sliderValues;

      const minTime = sliderValueToVideoTime(videoPlayerState.duration, min);
      const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max);

      if (videoPlayerState.currentTime < minTime) {
        videoPlayer.seek(minTime);
      }
      if (videoPlayerState.currentTime > maxTime) {
        // looping logic
        videoPlayer.seek(minTime);
      }
    }
  }, [videoPlayerState]);

  useEffect(() => {
    // when the current videoFile is removed,
    // restoring the default state
    if (!videoFile) {
      setVideoPlayerState(undefined);
      setSliderValues([0, 100]);
      setVideoPlayerState(undefined);
      setGifUrl(undefined);
    }
  }, [videoFile]);

  return (
    <div>
      <Spin
        spinning={processing || !ffmpegLoaded}
        tip={!ffmpegLoaded ? 'Waiting for FFmpeg to load...' : 'Processing...'}
      >
        <div>
          {videoFile ? (
            <VideoPlayer
              src={URL.createObjectURL(videoFile)}
              onPlayerChange={(videoPlayer) => {
                setVideoPlayer(videoPlayer);
              }}
              onChange={(videoPlayerState) => {
                setVideoPlayerState(videoPlayerState);
              }}
            />
          ) : (
            <h1>Upload a video</h1>
          )}
        </div>
        <div className={'upload-div'}>
          <VideoUpload
            disabled={!!videoFile}
            onChange={(videoFile) => {
              setVideoFile(videoFile);
            }}
          />
        </div>
        <div className={'slider-div'}>
          <h3>Cut Video</h3>
          <Slider
            disabled={!videoPlayerState}
            value={sliderValues}
            range={true}
            onChange={(values) => {
              setSliderValues(values);
            }}
            tooltip={{
              formatter: null,
            }}
          />
        </div>
        <div className={'conversion-div'}>
          <VideoConversionButton
            onConversionStart={() => {
              setProcessing(true);
            }}
            onConversionEnd={() => {
              setProcessing(false);
            }}
            ffmpeg={ffmpeg}
            videoPlayerState={videoPlayerState}
            sliderValues={sliderValues}
            videoFile={videoFile}
            onGifCreated={(girUrl) => {
              setGifUrl(girUrl);
            }}
          />
        </div>
        {gifUrl && (
          <div className={'gif-div'}>
            <h3>Resulting GIF</h3>
            <img
              src={gifUrl}
              className={'gif'}
              alt={'GIF file generated in the client side'}
            />
            <a
              href={gifUrl}
              download={'test.gif'}
              className={'ant-btn ant-btn-default'}
            >
              Download
            </a>
          </div>
        )}
      </Spin>
    </div>
  );
}

export default VideoEditor;

Since ffmpeg takes time to load, you need to let users know that they need to wait a while before they can use the video editor component. Also, ffmpeg operations take time to execute. This wait logic is implemented by using the Antd Design [Spin](https://ant.design/components/spin/) component.

Also, note that by passing the log: true option to the createFFmpeg() function, ffmpeg will log useful debug info in the console. For example, after ffmpeg gets loaded, you should be able to see the following lines in the browser console:

[info] use ffmpeg.wasm v0.11.5
createFFmpeg.js:43 [info] load ffmpeg-core
createFFmpeg.js:43 [info] loading ffmpeg-core
createFFmpeg.js:43 [info] ffmpeg-core loaded

What the VideoComponent does is use all the components presented earlier to allow users to upload a video, select a portion through a slider, convert it to GIF, and download the resulting GIF file.

Et voilà! You just learned how to build a client-side video editor in React without using any backend functionality!

Commercial Alternative

VideoEditor SDK by IMG.LY provides powerful video editing features, including cropping and trimming videos in your project. You will receive staples of video editing, including straightening videos, filters, brightness, color adjustments, and more.

Conclusion

Building a client-side video processing and editing application that uses only the browser possible. With WebAssembly, you can run high-performance code written in C or C++ in JavaScript. This allows you to perform complex operations at lightning speed directly in the browser, without having to delegate the work to a server.

In this tutorial, you learned how to use ffmpeg.wasm to build a video editor in React. ffmpeg.wasm is the Wasm port of ffmpeg and enables video and audio processing and editing directly in the browser. Using it to build a web application to upload a video, cut it, and export it to GIF requires only a few lines of code, and here we have seen how to implement such a video editing application.

If your app goes beyond merely displaying video, and you want to allow your users to also edit video or create video based templates in the browser, explore our VideoEditor SDK for the Web – a performant video editing solution based on WASM and WebCodecs. Try the editor for yourself in action with our showcases.

Thanks for reading! Let us know what you think on Twitter! To stay in the loop, subscribe to our Newsletter.

FFmpeg - The Ultimate Guide

Csaba — Mon, 21 Nov 2022 12:16:19 GMT

In this guide, we’ll go through the hot topics of FFmpeg. But before that, we’ll cover some base ground to help you understand basic media concepts and FFmpeg. Feel free to skip the parts that are already trivial for you!

Introduction to FFmpeg

FFmpeg.org’s definition is the following: “FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge. No matter if they were designed by some standards committee, the community or a corporation.”

I think of FFmpeg as the go-to application for audio/video manipulation in an automated or scripted manner.

When you need to implement a service that manipulates video, or just have 300 media files that need to be converted into a different format, FFmpeg is your - nerdy - friend.

FFmpeg can do large chunks of the basic functionalities of a modern Non-linear (NLE) video editors, e.g., Davinci Resolve Studio or Premiere Pro. But, it does not have a graphical interface in that sense as those behemoths do, and unarguably it is way less friendly.

In a general NLE, you might do things like these:

Click to import a file
Drop it into the timeline
Trim and Cut
Add an overlay image
Crop that overlay
Add vignette
Add some color changing effects, e.g. change the hue
Add an extra audio track to the mix
Change the volume
Add some effects, e.g.: echo
Export into various formats
Export into a deployable video format
Export the master audio in wav

Learn how to crop and trim videos in Flutter. Or, to achieve the exact same thing, you could also execute this command:

ffmpeg -y  \
    -ss 20 -t 60 -i bbb_sunflower_1080p_60fps_normal.mp4 \
    -i train.jpg \
    -ss 4 -i voice_recording.wav \
    -filter_complex "[0:v]hue=h=80:s=1[main] ; [1:v]crop=w=382:h=304:x=289:y=227[train] ; [main][train]overlay=x=200:y=200,vignette=PI/4[video] ; [2:a]volume=1.5,aecho=0.8:0.9:100:0.3[speech] ; [0:a][speech]amix=duration=shortest,asplit[audio1][audio2]" \
    -map '[video]' -map '[audio1]' -metadata title="Editor's cut" bbb_edited.mp4 \
    -map '[audio2]' bbb_edited_audio_only.wav

Yes, it isn’t friendly at all, but it is very, very powerful once you become friends with FFmpeg.

Check out this comparison of the original and the edited one:

If you want to try this command out, get the example files and see it for yourself!

Installing FFmpeg

FFmpeg is available for most common and even uncommon platforms and architectures. You can be on Linux, Mac OS X or Microsoft Windows, and you’ll be able to run or link to FFmpeg.

Installing FFmpeg is easy on most platforms! There is no installer, usually just a compressed archive you need to get for your platform and architecture.

In the case of Linux, most distributions include a pre-built FFmpeg in their software repositories. Therefore, you can install FFmpeg from those even more quickly.

FFmpeg history

The project was started in 2000 by the awesome Fabrice Bellard. The name is a concatenation of “FF” meaning “fast-forward” and MPEG, the name of a video standards group. It has been very well, active and alive since then, releasing a new release about every three months.

FFmpeg supported codecs and formats

The default FFmpeg shipped with my Ubuntu Linux distribution supports about 460 codecs and 370 formats.

See it for yourself:

ffmpeg -codecs
ffmpeg -formats

Compilation of FFmpeg

Keep in mind that the supported codecs and formats (and filters, demuxers, muxers, input and output methods, etc.) are highly dependent on the so-called compilation flags.

This means that the above number only represents the fact that it supports at least this many codecs and formats. Still, there are even more that the package builders excluded for various reasons, e.g.: licensing, architecture, size considerations, etc.

Since FFmpeg is open source, you can compile FFmpeg for yourself at any time.

Suppose for example, that you care about your layer’s size (therefore the bootstrap speed) in AWS Lambda. In this case, you can compile an FFmpeg binary that only contains the mp3 encoder for example, and nothing else. For a full tutorial on running FFmpeg on AWS Spot Instances, see our cloud guide. Prefer Google Cloud? Our guide on running FFmpeg on Google Cloud Platform shows you how.

Also, you might not want to run into licensing issues and leave out stuff that would cause problems for your use case. Therefore you choose to leave out particular codecs/formats. I highly recommend checking out the “—enable-gpl”, “—enable-nonfree” and “—enable-version3” compilation flags in this case, as well as this.

Or you might want to have a standalone FFmpeg binary in your project (e.g.: embedded, or some cloud instance), that does not depend on any operating system libraries. Then you want to make a so-called static build, that compiles in all the libraries into a single binary file, and does not depend on your OS’ libraries and the runtime loading of other FFmpeg libraries. Search around for “—enable-static” in this case.

Finally, you can find pre-built static FFmpeg builds right here too. Alternatively, you can package FFmpeg in a Docker container for consistent environments - our Docker guide covers this approach.

FFmpeg’s strengths

FFmpeg reads and writes most video and audio formats that matter for most of us. It is a very capable and high-performance tool for converting and manipulating these formats.

But FFmpeg can do even more! For examples of these operations integrated into an automated pipeline, read our article on a batch video processing server.

Filtering

FFmpeg has vast amounts of filters for audio and video. Therefore, video manipulation is also a key feature of FFmpeg.

Hardware acceleration

It does support many kinds of hardware accelerations! Video encoding is a very resource-intensive operation, and you might come across quite a few hardware devices or features that might speed up your process!

Most notably, if you have an NVIDIA card, you can increase your H.264 or H.265 encoding and decoding throughput by multipliers compared to your CPU. But other things, such as VDPAU, VAAPI, or OpenCL, can be leveraged to boost your pipeline’s throughput.

Learn more about the supported hardware acceleration methods here.

Versatile input/output methods

FFmpeg is also very capable when it comes to accessing input and output data.

Just to name a few: it can use your webcam, record from your microphone, grab your screen, or capture from your Blackmagic DeckLink. But FFmpeg can download directly from a web address, open all kinds of streams, read from a pipe, a socket, and of course, from files.

The same holds true for outputting the data. It can write to your webcam, play audio on your microphone… Just kidding:) It can output to files, streams, pipes, sockets and so on.

Running example commands

This article is full of FFmpeg commands that are working examples. The reason for that is that you could test these out for yourself! But the command line interfaces of different operating systems are slightly different, so the commands in this article are meant to be executed in a Linux bash shell.

To adopt these command lines to Microsoft Windows, you might need to:

Change (cd) into the directory where you extracted the ffmpeg.exe. Alternatively, add that directory to the path to make it callable from anywhere.
You might need to replace “ffmpeg” to “ffmpeg.exe”
You will need to replace ”\“-s (backslashes) at the end of the lines with ”^“-s (hats)
You’ll need to replace the fontfile argument’s value to something like this: fontfile=/Windows/Fonts/arial.ttf to get commands with the drawtext filter working.

MacOS users will need steps #1 and #4.

Introduction to media concepts

Now let’s have a quick overview of media concepts. These concepts will be vital for us if we want to understand the latter sections of this article and FFmpeg’s workings. To keep this section brief, it is a higher-level, simplified explanation of these concepts.

Audio

We’ll briefly cover the following terms:

Sampling rate
Bitrate
Channels

Sampling Rate

The sampling rate is the factor that shows how many times we measure/scan/sample the input data stream.

The image below shows the measurement windows (quantization) as gray bars.

Why does this matter? Because it is a balancing act. If we measure the signal less often, we’ll lose more details (bad). Also, by having fewer samples, we’ll have less data in the end. Therefore the file size will be smaller (good).

Here are some ballpark values:

8 kHz (GSM - Low quality)
44.1 kHz (CD - High quality)
48 kHz (Very high quality)
88.2 kHz (Insane - usually for production only)
96 kHz (Insane - usually for production only)

There are no definite “right answers” here. The question is what is “good enough” for your use case? GSM focuses on speech, and not even quality but understandability and the least possible amount of data. Therefore, they found that 8 kHz is enough (there are quite a few more tricks), for their purposes.

The “CD quality” aimed for high quality. Therefore they chose 44.1 kHz, that number has some history in it, but the main reason for aiming above 40 kHz lies in physics and how the human ear works.

There were two very smart guys whose theorem basically says that if you want a quite good signal representation, you have to sample it at twice the speed as its original frequency. Human hearing generally works up until about 20 kHz, so if you want “good quality”, you should aim for at least 40 kHz. And 40 kHz + some headroom + some more physics + historical reasons = 44.1 kHz! :)

As for the higher rates, those are only used when very high-quality audio editing is needed.

Bitrate

Bitrate represents the amount of data per second that results from our transcoding/quantization process. If it is 1411 kbit/s, that means that for every second of audio data, about 1411 kbit of output data will be produced.

Therefore, you can say that 1 minute of audio with 1411 kbit/sec will require:

(1411 kbit / 8) kbyte * 60 second = 10582 kbyte = 10.33 mbyte

Now, it is only easy like that with raw audio data and with a few simple codecs, e.g. PCM in WAVs.

Codecs compressing hard might throw your numbers around a little, as input data might be compressible with different rates. Variable bitrate is usually happening to save space. The encoder might output a lower bitrate if the data is “simple” and does not require high precision.

Here are some ballpark values:

13 kbits/s (GSM quality)
320 kbit/s (High-quality MP3)
1411 kbit/s (16bit WAV, CD quality, PCM)

Channels

Inside of most audio formats, you can have more audio channels. This means multiple, separated audio streams can be in the same file.

Many times, multiple channels have their own name:

If you have a single microphone, you will most probably record it into a single channel called Mono.
General music from the FM radio or streaming services usually has two channels in a so-called “Stereo” configuration.

With stereo, there could be several methods how the audio “image” can be made richer by leveraging audio panning, time and phase-shifting and much more. There is a special recording technique too, called Binaural recording, which is super awesome. Wear headphones for this, and don’t be scared:)

For example, here are Big Buck Bunny’s audio waveforms in Audacity:

You can see that there are two lines of waveforms and also that they are pretty similar. That is normal, as you usually hear the same thing with your two ears, but the matter is in the subtle differences between the two. That’s where directionality, richness, and all kinds of other effects lie.

But why stop at two? The list continues:

2.1, as it is often called, means three channels: 2 for stereo and one for the LFE (“low-frequency effects” a.k.a.: “bass”).
5.1 is similar, with five directional channels (2 front, 1 center, 2 rear) and the LFE.

So channels are just separate “recordings” or “streams” of audio signals.

Image properties

For images, there are quite a few parameters, but we’ll check out only these:

Resolution
Bit-depth
Transparency

Resolution

An image consists of pixels, single points that have a single color. The resolution of an image determines how many columns and rows of pixels are in an image. In other words: an image has a width and a height.

This image shows the first 10 pixels in the first row.

Here are some ballpark values for resolution:

“HD” or “Full HD” or “1K” or “1080p” means 1920x1080 pixels.
“4K” could mean a few values, but it should be about 3840x2160 pixels.
A regular 16mp photo you make of your cat is about 4608x3456 pixels.
General social media image posts are about 1080x1080 pixels.

Bit-depth

Bit-depth represents the number of bits used for storing a single pixel’s color value. This is the same balancing game, and you need to decide between quality or file size.

General ballpark values for bit-depth:

Bits	Colors	Notes
1	2	Black & White
8	256	B/W or Limited color palette
24	16.7m	3x8 bit for R-G-B “True color”
30	1073m	3x10 bit for R-G-B “Deep color”

These last two sometimes are referred to as “8 bit” or “10 bit” respectively, especially when talking about videos. That means 8/10 bits per single color channel.

Transparency

Some image formats support an additional channel together with the red, green, and blue components: the alpha channel. The alpha channel determines how transparent a single pixel is, and it can have different bit-depths, it is usually either 1, 8 or 16 bits.

If the alpha channel is 1 bit, then the format can encode a pixel to be either transparent or non-transparent. If it is 8 or more bits, then the format can encode 256 or more steps of transparency.

Video properties

Video data is built by single images shown right after each other. This brings in most attributes of images and a few more!

So a video has a resolution that is its width and height.

Then the first obvious parameter of a video is the framerate, which defines how many images are shown in a second. Common values for this are 24, 25, 30, or 60.

A video file also has a codec assigned to it, which is the format describing how all those images were compressed into this video file. There are many more attributes of videos, but this is a good start.

Video codecs

Compression is a super important thing when it comes to video because you have thousands of images to keep together. If you aren’t doing it in a smart way, then the resulting video will be very, very large.

Just imagine a 2-minute video, with 30 fps. That means it will have 60 s * 2 * 30 fps = 3600 frames! I have just taken a screenshot of an HD video, which was 730 kbyte in JPEG format. Now 3600 frame * 730 kbyte equals 2.5 gigabytes!

Can you imagine that? I hope not, and that’s because compression brings that way, way down, to the level of tens of megabytes. These days a video of that size is quite high quality and about 2 hours long. Also, don’t forget, that JPEG is already compressed, a single frame would be 6 mbyte when uncompressed. Now that 2-minute video would be 21 gigabytes if we’d store it uncompressed.

Standard codecs such as H.264 and H.265 are doing very clever and complex operations to achieve high compression ratios with good quality.

Just think about that, most frames in a video are quite similar, only containing small differences. So if we could only store that little difference between frames, we’d won a huge bonus! And that’s just one of the many tricks codecs do.

Codec designers are also exploiting the weaknesses and features of the human eye. Such as the fact that we are more sensitive to light intensity changes than color changes (say hello to YUV). And they can get away with lower quality details for parts that are moving fast, and so on.

Because why lose precious bits for things that you can’t even notice?!

There are many codecs out there, with different goals in mind, although the majority focus on keeping the file size low.

H.264, H.265: These are the most common ones, with the widest support in browsers, phones, players, etc. It focuses on small file sizes with good quality. (At the cost of resource intensiveness.)
Apple ProRes, DNxHD: These are common formats for production. They focus on quality and ease of processing and not on file size.

Audio codecs

The goal of audio codecs is the same as what we saw with the video codecs. It is just harder to demonstrate it as audio does not consist of single image frames but audio frames/packets. So an analog audio signal is of an almost infinite, or at least very high quality if you think of it.

At the lowest level, the speed and amplitude resolution is very high. We could say “atomic”, as we need to measure and store the speed and direction of atoms. So if you want to store that exactly, that will require a super high-quality measurement, which will also result in a very high bitrate data stream.

Thankfully, the sound is at least not propagating with light speed so we can save quite a lot just by that fact. (There’s no need for an extreme sampling rate.) Then our hearing is very limited if we take the previous paragraph as a scale, so we win there again. We don’t need most of that high precision that is there.

But still, if we take our hearing capability and want to store raw audio data with about 44.1 kHz of sample rate with about 1 Mbit/sec bitrate, we’d still get quite a lot of data. Check the calculations in the audio bitrate section above.

So raw audio can be compressed further, which is what many popular codecs do. They also exploit the human senses, but this time the human ear. We started with the basics that the human ear has a limit on the frequencies it can detect. Therefore, we can save a lot by cutting out the range of frequencies outside our hearing range. Unless you are a bat, you are fine between 20-20khz! :)

But there are other tricks, for example, auditory masking. That means that the presence of one frequency can affect your capability to detect a different frequency. From the codec’s viewpoint, it can skip encoding a few frequencies if it is smart enough to know which ones you’ll not notice. I’m sure there are a lot more tricks, let me know if you know about a few more interesting ones!

Here is a list of common codecs:

MP3, AAC, OGG: These are common lossy audio formats.
PCM (e.g. in a WAV container), FLAC: These are lossless formats.
MIDI: It is a funny format. It is like a music sheet that might sound different on different players or settings. It is usually not made from real audio data, but from recording a digital keyboard or as an output from an audio composing software.

Containers

Now we got through the fundamental building blocks, the image, the video, the video codecs, and the audio codecs, and we reached the top of this iceberg: the containers.

A container is a format specification, that combines all these streams into a single file format. It defines how to put all these data together, how to attach metadata (e.g. author, description, etc), how to synchronize these streams, and sometimes a container even contains indexes to aid seeking.

So, for example, a MOV container can contain an H.264 video stream and an AAC audio stream together.

Common containers:

MOV
MP4
MKV
WebM
WAV (audio only)

Example Material

I will use these example materials as inputs in the following parts of this article. If you’d like to follow along, save these files for yourself!

Name	Resource
Big Buck Bunny	http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4
Train	train.jpg
Smiley	smiley.png
Voice recording	voice_recording.wav
Big Buck Bunny’s audio	ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 bbb_audio.wav

And we will make our own audio file by extracting the audio from the Big Buck Bunny movie! We’ll use this file as an example, so after downloading the video file, please execute this:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 bbb_audio.wav

By the middle of this article, you’ll understand this command, but for now, just make sure to have the WAV file next to your video file to test out the commands later in the article.

We’ll use these files in the following parts of this article. Therefore make sure to get them!

FFplay and FFprobe

FFmpeg is the name of the main binary and the project itself, but it is shipped together with two other binaries, ffplay and ffprobe.

Let’s check them out quickly, right in the command line!

FFplay

FFplay is a basic video player, that can be used for playing media. It’s not a friendly video player, but it is a good testing ground for various things.

To execute it, just simply supply a media file:

ffplay bbb_sunflower_1080p_60fps_normal.mp4

If you want to test this exact command, you’ll need to get the example files.

For example, it can be used to preview filters (we’ll discuss those later), but let’s see an example:

ffplay -vf "drawtext=text='HELLO THERE':y=h-text_h-10:x=(w/2-text_w/2):fontsize=200:f

FFprobe

FFprobe, as its name implies, is a tool for getting information about media files.

This command:

ffprobe bbb_sunflower_1080p_60fps_normal.mp4

Will return us some general information about the video file:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':
  Metadata:
[...]
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
[...]
  Stream #0:0[0x1](und): Video: h264 [...]
[...]
  Stream #0:1[0x2](und): Audio: mp3 [...]
[...]
  Stream #0:2[0x3](und): Audio: ac3 [...]

I have abbreviated it heavily, as we’ll check this out later.

But FFprobe is way more powerful than just this!

With the following command, we can get the same listing in JSON format, which is machine-readable!

ffprobe -v error -hide_banner -print_format json -show_streams bbb_sunflower_1080p_60fps_normal.mp4

The explanation of this command is the following:

“-v error -hide_banner”: This part hides extra output, such as headers and the default build information.
“-print_format json”: Obviously, this causes ffprobe to output a JSON.
“-show_streams” is the main switch that requests the stream information.

{
  "streams": [
    {
      "index": 0,
      "codec_name": "h264",
      "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
      "width": 1920,
      "height": 1080,
      "bit_rate": "4001453",
      "duration": "634.533333",
      "############################": "[~50 lines removed]"
    },
    {
      "index": 1,
      "codec_name": "mp3",
      "channels": 2,
      "bit_rate": "160000",
      "############################": "[~40 lines removed]"
    },
    {
      "index": 2,
      "codec_name": "ac3",
      "channels": 6,
      "############################": "[~20 lines removed]"
    }
  ]
}

In this output, you can see three streams of data in this video file. The first (index: 0) is a video stream, that is an HD video with an H.264 codec. Then we have two audio streams, the first (index: 1) is a simple mp3 stream with stereo audio, and the second (index: 2) is an ac3 stream with 6 channels, most likely in an 5.1 configuration.

I have removed quite a lot of output for brevity, but you can get way more information out of these streams, e.g. fps for the video stream and so on.

Other than -show_streams, there are 3 more: -show_format, -show_packets and -show_frames. Unless you are really deep in the rabbit hole, you’ll not need the last two, but -show_format could be useful:

ffprobe -v error -hide_banner -print_format json -show_format bbb_sunflower_1080p_60fps_normal.mp4

{
  "format": {
    "filename": "bbb_sunflower_1080p_60fps_normal.mp4",
    "nb_streams": 3,
    "nb_programs": 0,
    "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
    "format_long_name": "QuickTime / MOV",
    "start_time": "0.000000",
    "duration": "634.533333",
    "size": "355856562",
    "bit_rate": "4486529",
    "probe_score": 100,
    "tags": {
      "major_brand": "isom",
      "minor_version": "1",
      "compatible_brands": "isomavc1",
      "creation_time": "2013-12-16T17:59:32.000000Z",
      "title": "Big Buck Bunny, Sunflower version",
      "artist": "Blender Foundation 2008, Janus Bager Kristensen 2013",
      "comment": "Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net",
      "genre": "Animation",
      "composer": "Sacha Goedegebure"
    }
  }
}

This is an overview of “what is this file”. As we see, it is a MOV file (format_name), with three streams (nb_streams), and it is 634 seconds long. Also, there are some tags where we can see the title, the artist, and other information.

FFmpeg concepts

Here is a quick intro to how FFmpeg actually works!

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

FFmpeg opens the file, decodes it into memory, then encodes the in-memory packets back and puts them into some container: some output file. The term “codec” is a mix of the words “coder & encoder”. Those are the magic parts before and after the “decoded frames”.

The decoded frames are uncompressed images in-memory, e.g. the most basic pixel format for video frames is called “rgb24”. This just stores red, green, and blue values right after each other in 3x8 bits, or 3x1 byte, which could hold 16m colors.

The importance of this is that other than a few exceptions, you can only manipulate or encode the decoded frames. So when we get to different audio/video filters or transcoding, you’ll need the decoded frames for all that. But don’t worry, FFmpeg does this automatically for you.

Inputs

So you see and probably guessed, that FFmpeg must access the input data somehow. FFmpeg knows how to handle most media files, as the awesome people who develop FFmpeg and the related libraries made encoders and decoders for most formats available!

Don’t think that it is a trivial thing. Many formats are reverse engineered, a hard task requiring brilliant people.

So although we often refer to input files, the input could come from many sources, such as the network, a hardware device and so on. We’ll learn more about that later on in this article.

Many media files are containers for different streams, meaning that a single file might contain multiple streams of content.

For example, a .mov file might contain one or more streams:

video tracks
audio tracks (e.g. for the different languages or audio formats such as stereo or 5.1)
subtitle tracks
thumbnails
…

All these are streams of data from the viewpoint of FFmpeg. Input files and their streams are numerically differentiated with a 0-based index. So, for example, 1:0 means the first(0) stream of the second(1) input file. We’ll learn more about that later too!

Important to note that FFmpeg can open any number of input files simultaneously, and the filtering and mapping will decide what it will do with those. Again more on that later!

Streams

As we have seen in the previous section, streams are the fundamental building blocks of containers. So every input file must have at least one stream. And that’s what you can list by the simple ffmpeg -i command for example.

A stream might contain an audio format such as MP3, or a video format such as an H.264 stream.

Also, a stream, depending on the codec, might contain multiple “things”. For example, an mp3 or a WAV stream might include various audio channels.

So the building block hierarchy, in this case is: File → Stream → Channels.

Outputs

Of course, an output could be a local file, but it doesn’t need to be. It could be a socket, a stream and so on. In the same way as with inputs, you could have multiple outputs, and the mapping determines what goes into which output file.

The output also must have some format or container. Most of the time FFmpeg can and will guess that for us, mostly from the extension, but we can specify it too.

Mapping

Mapping refers to the act of connecting input file streams with output file streams. So if you give 3 input files and 4 output files to FFmpeg, you must also define what should go to where.

If you give a single input and a single output, then FFmpeg will guess it for you without specifying any mapping, but make sure you know how exactly that happens, to avoid surprises. More on all that later!

Filtering

Filtering stands for the feature of FFmpeg to modify the decoded frames (audio or video). Other applications might call them effects, but i’m sure there is a reason why FFmpeg calls them filters.

There are two kinds of filtering supported by FFmpeg, simple and complex. In this article we’ll only discuss the complex filters, as it is a superset of the simple filters, and this way, we avoid confusion and redundant content.

Simple filters are a single chain of filters between a single input and output. Complex filters can have more chains of filters, with any number of inputs and outputs.

The following figure extends the previous overview image with the filtering module:

A complex filter graph is built from filter chains, which are built from filters.

So a single filter does a single thing, for example, changes the volume. This filter is quite trivial, it has a single input, changes the volume, and it has a single output.

For video, we could check out the scale filter, which is also quite straightforward: it has a single input, scales the incoming frames, and it has a single output too.

You can chain these filters, meaning that you connect the output of one to the input of the next one! So you can have a volume filter after an echo filter, for example, and this way, you’ll add echo, and then you change the volume.

This way, your chain will have a single input, and it will do several things with it and will output something at the end.

Now, the “complex” comes in when you have multiple chains of these filters!

But before we go there, you should also know that some single filters might have multiple inputs or outputs!

For example:

The overlay filter puts 2 video streams above each other and will output a single video stream.
The split filter splits a single video stream into 2+ video streams (by copying).

So let’s discuss a complex example from a bird’s eye view! I have two video files, I want to put them above each other, and I want the output in two files/sizes, 720p and 1080p.

Now, that’s where complex filtering will be faithful to its name: to achieve this, you’ll need several filter chains!

Chain 1: [input1.mp4] [input2.mp4] → overlay → split → [overlaid1] [overlaid2]
Chain 2: [overlaid1] → scale → [720p_output]
Chain 3: [overlaid2] → scale → [1080p_output]

As you see, you can connect chains, and you can connect chains to output files. There is a rule that you can only consume a chain once, and that’s why we used split instead of the same input for chains 2 and 3.

The takeaway is this: with complex filter graphs (and mapping), you can:

build individual chains of filters
connect input files to filter chains
connect filter chains to filter chains
connect filter chains to output files

FFmpeg’s command line system

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

FFmpeg CLI

Finally, we arrived at FFmpeg, and trust me, we’ll execute it quite a lot of times! Let’s see how FFmpeg’s command line options are organized, as that is the first tricky part we need to understand!

FFmpeg mostly thinks about input and output files and their options together with global options. You specify input files with the “-i” flag followed by a file name. For the output file, specify it as-is without any preceding CLI (command line interface) flag.

Specifying an input file

Let’s specify just an input file:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4

The following image helps to understand the output:

First, you get the “banner”, where you see the build information and lib versions. If you watch closely, you’ll see the compilation flags, starting with —, e.g. —enable-shared.
Then you get the same output as we have seen with ffprobe earlier.
And then you get a complaint that there is no output file(s) specified. That’s fine for now.

You can remove the banner here with “-hide_banner”, but for brevity’s sake I’ll not include that anymore in the commands here, and I will leave it out from the outputs too.

Now, let’s get brave, and specify an output file!

Specifying an output

As I’ve said earlier, the output file is understood by FFmpeg as it is just a filename. But more specifically, it is after the input(s) specifications, and it is not a value of any other switches.

Don’t be confused for now, but yes, FFmpeg can have as many inputs and outputs as you’d like. We’ll cover that in more detail soon!

This command line specifies a single output file:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

Before taking a look at the output, let me congratulate you! You have just converted a video file into an audio file, by keeping just the audio content!

This is how you transcode! Of course, you’ll want to specify more parameters later on.

So, here is the output:

Let’s analyze it!

(1) First, we have our input metadata printing, which we saw many times already.

(2) Then we have something called “stream mapping”. We forced FFmpeg into a decision situation, as we specified an input file with 1 video and 2 audio streams. We said we wanted an audio output (guessed from the .wav extension). But we didn’t specify which audio stream we wanted, so let’s see what FFmpeg decided:

“Stream #0:2” means “The first input file’s third stream” or “input file index 0’s stream with index 2.” This is the input.
”-> #0:0” means the first output file’s first stream. This is the output.
Here you can learn more about how FFmpeg decide this.
Later on, we’ll manually override the mapping.
Summary: FFmpeg decided to convert the third stream in the input file (the ac3 5.1 audio) into the first stream of the output file.

(3) Then we have our output metadata information. This reveals what FFmpeg will output. It usually copies most of the metadata, and here you also see the container/format information too.

(4) And then we see the output summary. For example, the transcoding was 181x faster than the playback speed. Nice!

Understanding the command line order

Before going further, let’s understand FFmpeg’s command line arguments from a bird’s eye view!

In the manual, you’ll see this:

ffmpeg [global_options] {[input_file_options] -i input_url} ... {[output_file_options] output_url} ...

(Parts in […] are meant to be optional, and parts in {…} are meant to be specified 1 or more times.)

This is the general outline of how to specify inputs, outputs, input options, output options, and global options. The order matters, but it is easy to remember: global options, inputs and outputs. Also, i/o options come BEFORE the i/o specification.

Let’s put these into pseudo command line options, to understand it better:

# One inputs, one output, nothing fancy
ffmpeg -i input1.mp4 output1.wav

# Two inputs, one output
ffmpeg -i input1.mp4 -i input2.mp4 output1.wav

# Two inputs, two outputs
ffmpeg -i input1.mp4 -i input2.mp4 output1.wav output2.mp3

# One input, one output, with options
ffmpeg [input1 options] -i input1.mp4 [output2 options] output1.wav

# Two inputs, two outputs with options
ffmpeg [input1 options] -i input1.mp4 \
       [input2 options] -i input2.mp4 \
       [output1 options] output1.wav \
       [output2 options] output2.mp3

As for the global options, these are the ones you might care about:

-hide_banner: To skip printing the banner.
-y: To overwrite the output even if it exists.

For example, you can run this as many times as you want:

ffmpeg -y -hide_banner -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

And it will overwrite the output and be less verbose than earlier.

Without explaining the options themselves, let’s just see some real-world examples with options:

And here it is with two inputs and two outputs:

Mapping files

We saw above that this command:

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 audio_only.wav

… will result in an audio file that contains one of the audio streams from the input video chosen by FFmpeg. This automatic stream selection is usually handy when it is trivial. For example, when you have one stream as input and one output file, you don’t need to specify any mapping manually.

But in cases where it is not so trivial, you are usually better off manually specifying what you really want to do.

The following image summarises what our current situation is:

The video stream was not matched, as the output format was an audio file (.wav). But then FFmpeg chose Stream #2, because it has more channels.

So what if we’d like to get the stereo track instead? That is where mapping comes in! The mapping is a parameter of the OUTPUT file. Therefore the mapping arguments should come right before our output file definition!

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav

The argument -map 0:1 means, that in the output (since we specify it as an output option) we’d like to have Input #0’s (the first input file) Stream #1!

Let’s see the relevant parts from the output!

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':

[...]

Stream mapping:
  Stream #0:1 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))

[...]

Output #0, wav, to 'stereo_audio_only.wav':
  Metadata:
[...]
    Stream #0:0(und): [...] stereo [...]

The “Stream #0:1 -> #0:0” part means that we have successfully overridden the mapping, to get the mp3 stream (0:1) into our output! Also, the output metadata reveals that we’ll get a stereo result instead of the 5.1 earlier.

Multiple outputs

You can have multiple outputs from a single input, let’s see when that might be useful!

Let’s say, we want to extract BOTH audio streams into two separate WAV files! It is super easy:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav -map 0:2 ac3_audio_only.wav

See? I have just specified two output files with two mapping specifications! Also, I have sneaked in the “-y” to have it overwrite our previous file!

Let’s check out the relevant parts of the output!

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':

[...]

Stream mapping:
  Stream #0:1 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))
  Stream #0:2 -> #1:0 (ac3 (native) -> pcm_s16le (native))

[...]

Output #0, wav, to 'stereo_audio_only.wav':
    Stream #0:0(und): [...] stereo

[...]

Output #1, wav, to 'ac3_audio_only.wav':
    Stream #1:0(und): Audio: [...] 5.1(side)

Now the mapping reveals two lines, as we have two outputs! And indeed, you’ll get two .wav files as the output, one is stereo, and one is 5.1!

There might be several other reasons why you’d want to get multiple outputs. Let’s briefly check out a few!

Different formats:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 stereo_audio_only.wav  stereo_audio_only.mp3

Wow, did you catch that? We just created a WAV and an mp3 in a single command line! I’ve reverted to the automatic stream selection for brevity’s sake.

A bit closer to real-life needs, you might want different output qualities:

ffmpeg -y  -i bbb_sunflower_1080p_60fps_normal.mp4  \
-map 0:1 -b:a 320k stereo_audio_only_high_quality.mp3 \
-map 0:1 -b:a 64k  stereo_audio_only_low_quality.mp3

Here -b:a 320k means “bitrate of audio should be around 320 kbit/sec”. So I have requested FFmpeg to make two mp3s for me, from the stereo stream of the input.

Checking on the files, this is what we got:

 25Mb stereo_audio_only_high_quality.mp3
4,9Mb stereo_audio_only_low_quality.mp3

One more common reason for having multiple outputs or using mapping is when we introduce filters into our pipeline, but that will be discussed later!

Now you understand the foundations of how to communicate your basic requirements to FFmpeg via its command line! Great job! Now we can dive even deepert.

Hands-on with FFmpeg

In this section, we will discover and even try out some common features of FFmpeg!

For those who are just joining in: please get the example assets if you want to test out the commands shown in this chapter!

Inputs

Let’s see the common ways FFmpeg is fed with different data!

File

Of course, you have already seen that if you have a local file on your filesystem, FFmpeg is happy to read it!

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -map 0:1 stereo_audio_only.wav

This command which is exactly the same as one of our previous ones just reads a local file. Really, that’s it.

Network

Did you know, that FFmpeg can open a file directly on the network?!

ffmpeg -t 5 -i http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4 bbb_first_5_seconds.mp4

The command above opens the file directly from the network and saves the first 5 seconds into a local file!

I wanted to spare bandwidth for these awesome guys over renderfarming.net, so I added the duration flag: -t 5. FFmpeg doesn’t even download the full video for this operation. Isn’t that wonderful?!

Webcam

FFmpeg can also open your webcam!

This is an example command for Linux:

ffmpeg -f v4l2 -framerate 25 -video_size 640x480 -t 10 -i /dev/video0 10seconds_of_webcam.webm

This would record 10 seconds of your webcam!

Accessing the webcam happens differently on different platforms. Also specifying parameters is different for each platform, so for this reason, if you’d like to access your webcam with FFmpeg, please refer to the documentation:

Microphone

Let’s record some audio directly from your microphone!

List microphones:

arecord -l

Start 10 seconds of recording:

ffmpeg -f alsa -i hw:0,0 -t 10 out.wav

This command was meant to work on Linux, but you can check out how to do that on Microsoft Windows or macOS.

Pipe

Finally, FFmpeg can read from a pipe, and also output to a pipe.

On Linux, you could do something like this:

cat bbb_sunflower_1080p_60fps_normal.mp4 | ffmpeg -i - -f wav pipe:1 | pv > output.wav

# Alternative, without pv:
cat bbb_sunflower_1080p_60fps_normal.mp4 | ffmpeg -i - -f wav pipe:1 > output.wav

This command would use the cat program to simply read in the video file and output it to its standard output. Then this output is piped INTO FFmpeg, through its standard input. The combination “-i -” means “read from standard input”. By the way, standard input would be your keyboard otherwise, if we wouldn’t use any redirection here.

Then we specify the required output format for FFmpeg, with “-f wav”. This is needed because now we’ll have no output file name, and FFmpeg will not be able to guess the format. Then we specify “pipe:1” as an output, meaning we’d like FFmpeg to output to its standard output.

From then, we pipe the data into a program called “pv”, it is just a metering tool, that dumps information on the throughput (from its stdin to its stdout). Finally, we redirect pv’s output into a WAV file.

You might ask why we’d want to do that, why we talk about this. Piping can be useful if you build a complex pipeline from different programs or if you want to spare reading and writing to a local file.

For example, the node package fluent-ffmpeg can leverage this functionality by supplying input and output streams. For example, you can read from an S3 bucket and write to one directly.

But be warned, hell is awaiting you on that road. No kidding. You need to research the limitations of this technique. For example, many formats can not be streamed in this manner, as they need random access to the output data to write the indices at the beginning of the file after processing.

Outputs

FFmpeg can output into many protocols, from local file storage and ftp to message queue protocols all the way to streaming protocols.

For more information, check out the documentation here.

Transcoding audio with FFmpeg

In this chapter, we’ll be going to see how to transcode into audio with FFmpeg!

The general formula is:

ffmpeg -i {input audio or video file with audio} [output options] output_audio.ext

Choosing a format

FFmpeg is quite smart, and by the extension, it can determine which codec to use. If you specify “audio.wav” or “audio.mp3” for example, FFmpeg will use the appropriate codec to do the encoding.

It is perfectly guessing most of the time. But if you want to specify the format manually, then the “-f” flag is your friend.

For this, you might want to consult the list of formats:

ffmpeg -formats

So, these three commands will do exactly the same, but the last two requires the -f flag.

# Output codec is determined from the extension
ffmpeg -i bbb_audio.wav bbb_audio.mp3

# No extension in the filename
ffmpeg -i bbb_audio.wav -f mp3 bbb_audio

# Piped output therefore no filename, so no extension to use for guessing
ffmpeg -i bbb_audio.wav -f mp3 pipe:1 > bbb_audio

Setting the bitrate

In most cases. you want to specify the target bitrate you expect from your codec to output. If you are unsure what bitrate is, please read this article’s audio bitrate section.

To specify the audio bitrate, use the “-b:a” option with a corresponding value, e.g.:

-b:a 320k: For the mp3 codec this is considered high quality.
-b:a 128k: Lower quality.
-b:a 64k: Low quality.

For example:

ffmpeg -i bbb_audio.wav -b:a 320k bbb_audio_320k.mp3

Setting the sample rate

You may want to specify the sample rate to ensure quality or low output file size. Half the sample rate could mean half the output file size. If you are unsure what the sample rate is, please read the “audio sample rate” section of this article.

To specify the audio sample rate, use the “-ar” option with a corresponding value, e.g.:

-ar 48000: For high quality.
-ar 44100: For CD quality (still high).
-ar 22500: A bit of a compromise, not recommended for music, but for speech, it might be enough.
-ar 8000: Low quality, e.g. if you only want “understandable” speech.

For example:

ffmpeg -i bbb_audio.wav -ar 44100 bbb_audio_44100khz.mp3

Setting the channel count

Setting the channel count can be useful, for example, if you have a stereo recording of a single person’s speech. In that case, you might be content with just a mono output half the size of the original recording.

If you are unsure what an audio channel is, please read the “audio channels” section of this article.

To specify the channel count use the “-ac” option with a corresponding value, e.g.:

-ac 1: For mono
-ac 2: For stereo
-ac 6: For 5.1

For example:

ffmpeg -i bbb_audio.wav -ac 1 bbb_audio_mono.mp3

Complete command line for converting audio with FFmpeg

This is how you produce a high-quality output:

# Convert wav to mp3
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.mp3

# Convert wav to m4a (aac)
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.m4a

# Convert wav to ogg (vorbis)
ffmpeg -i bbb_audio.wav -ac 2 -ar 44100 -b:a 320k bbb_audio_hqfull.ogg

Check out this documentation about good quality audio transcoding too!.

Lossless formats

If you want to convert audio into a lossless format, here are a few choices for you:

# Convert to flac (Free Lossless Audio Codec)
ffmpeg -i bbb_audio.wav -compression_level 12 bbb_audio_lossless_12.flac # Best compression, slowest
ffmpeg -i bbb_audio.wav -compression_level 5 bbb_audio_lossless_5.flac   # Default
ffmpeg -i bbb_audio.wav -compression_level 0 bbb_audio_lossless_0.flac   # Least compression, fastest


# Convert to wav
cp bbb_audio.wav bbb_audio_lossless.wav # Just kidding:)

# Convert to wav
ffmpeg -i any_audio.ext bbb_audio_lossless.wav

It’s good if you know that flac results in a smaller file than WAV, as WAV doesn’t actually compress by default:

117M bbb_audio.wav
52M  bbb_audio_lossless_0.flac
45M  bbb_audio_lossless_5.flac
43M  bbb_audio_lossless_12.flac

WAV is generally thought of as a lossless format, but keep in mind that the WAV container can contain lossy content too, but by default FFmpeg uses the pcm_s16le format, which is the 16 bit PCM, that could be understood as lossless.

Learn more here and here.

Transcoding video with FFmpeg

In this chapter, we’ll be going to see how to transcode a video file into the two most common formats!

Converting to H.264

H264 is one of the most popular video codecs. Most devices, browsers and video players understand how to play it. It is efficient in storing video content, but as with most advanced video codecs, it is a resource intensive-process to encode and decode.

A complete command line for a high-quality H.264 transcoding with high-quality AAC audio is the following:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 \
-c:v libx264 -preset slow -crf 22 \
-profile:v main -g 250 -pix_fmt yuv420p \
-map 0:0 -map 0:1 \
-acodec aac -ar 44100 -b:a 320k bbb_transcoded_h264_HQ.mov

Make sure to understand this command and to customize it to match your needs.

To help you do that, let’s dissect this command!

Global options:

-y: Overwrite the output.

Input options:

-i bbb_sunflower_1080p_60fps_normal.mp4: The input file.

Output options:

-c:v libx264: Set the codec to libx264.

-preset slow: libx264 has a lot of variables that you can be tune, and most of them balance the coding speed and the resulting file size. To make your life easier, there are presets by which you can easily declare what you need: small size or speed.

-crf 22: This is the constant rate factor, the main option for setting image quality. It is a number between 0-51, where 0 is lossless, and 51 is the worst quality. Generally, you want something between 17 and 28. This is the option to tune the balance between image quality and file size. Check my comparison video here.

-profile:v main -g 250 -pix_fmt yuv420p: These are advanced options, guaranteeing you a quite backward compatible result. (See this, this, and this.)

-map 0:0 -map 0:1: You might not need this: these options are selecting the correct video and audio streams. In our case, we have two audio streams, and we need the stereo one to avoid some issues with our aac stream.

-acodec aac: Select the AAC (Advanced Audio Coding) codec for the audio in the output. We need to be more specific than just -f for the format. We need to specify the audio codec here manually.

-ar 44100: Set the audio sampling rate (learn more about that in previous chapters of this article).

-b:a 320k: Set the audio bitrate (learn more about that in previous chapters of this article).

30seconds_of_bb.mkv: The output file name. All the options since the last -i (or the last output file) considered to be a modifier for this output.

Let’s see the output:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_normal.mp4':

[...]

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (mp3 (mp3float) -> aac (native))

[...]

Output #0, mov, to 'bbb_transcoded_h264_HQ.mov':
    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 60 fps, 15360 tbn, 60 tbc (default)
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, 5.1(side), fltp, 320 kb/s (default)

[...]

frame=38074 fps= 35 q=-1.0 Lsize=  324855kB time=00:10:34.51 bitrate=4194.1kbits/s dup=2 drop=0 speed=0.58x

From this, we understand that FFmpeg chose the mp3 stream from the input file because we told it to do so. (Remember, it has two audio streams in it, a stereo mp3 and a 5.1 ac3.) We also see that my machine could transcode with 35fps (0.58 times the playback speed), and our settings resulted in an average video bitrate of 4200 kbit/s.

The video bitrate is an interesting question in this mode. With the CRF option, we specify the “constant visual quality” we want. To reach a constant visual quality, the encoder works hard to guess how much it can compress certain parts of every frame, and the result of that guess defines the final average video bitrate.

If you want even better results with H.264, and you can afford a bit more processing time and a bit more complicated process, check out the 2-pass encoding instead of the constant rate factor method introduced above.

To learn more about these two different rate control methods, read the awesome Understanding Rate Control Modes article. And to learn more about the intricacies of H.264 encoding, check out the H264 encoding guide.

Finally, later on, I will show you a comparison video that shows how different CRF values perform!

Converting to H.265

H.265 is the successor of H.264, according to the official FFmpeg manual, it offers 25-50% bitrate savings while retaining the same visual quality.

A complete command line for a high-quality H.265 transcoding with high-quality AAC audio is the following:

ffmpeg -y -i bbb_sunflower_1080p_60fps_normal.mp4 \
-c:v libx265 -preset slow -crf 27 \
-profile:v main -g 250 -pix_fmt yuv420p \
-map 0:0 -map 0:1 \
-acodec aac -ar 44100 -b:a 320k bbb_transcoded_h265_HQ.mov

And the result is:

...
encoded 38074 frames in 3384.84s (11.25 fps), 1720.32 kb/s, Avg QP:35.29

H.265 also has multiple rate control algorithms, I used the CRF method here. If you want to use a different rate control algorithm, then you may check out the H.265 encoding guide. Also, check out the next section, where I’ll reveal how different CRF values perform!

This command is almost the same as what we used in the H.264 example above, so please refer to that section to understand the arguments.

If we compare H.264 and H.265 with our commands above, taking into account this 10-minute long video on my system, these are the results:

H.264 is 3 times faster (35 fps vs 11 fps)
H.264 produces a 2 times larger file (318 mb vs 156 mb)

Comparing CRF values with H.264 and H.265

I have created a video for your convenience, that shows the different crf values in action. The selected frame had some movement on it with the leaves in the bunny’s hand. Movement is important with video codecs, as usually that’s where quality losses are first visible.

This video shows how the different CRF values perform, from 0-51 with the H.264 and H.265 formats!

H.264 & H.265 CRF comparison video

(Can you guess which program I was using to make this?:))

Basic editing with FFmpeg

In this section, we’ll achieve basic editing tasks by using FFmpeg only!

We’ll just get a basic mp4 with default settings in these examples to keep things simple. But to encode the result in a proper, high quality way, please check the earlier sections where we learned how to encode into H.264 and H.265!

Trimming from the beginning of the clip

It is possible to specify an in-point for a media file. By doing that, you essentially cut off the specified amount from the beginning of the input file. Therefore, FFmpeg will skip the first part of the file and only transcode the remainder!

For this, you need the “-ss” flag! The value can be specified in seconds (5 or 5.2) or as a timestamp (HOURS:MM:SS.MILLISECONDS).

To get the outro only, we could seek all the way to the end of the video! (It is 00:10:34.53 or 635 seconds long!)

# Get
# 635 - 4 = 631
ffmpeg -y -ss 631 -i bbb_sunflower_1080p_60fps_normal.mp4 last_4_seconds.mp4


# 00:10:34.53 - 4 = 00:10:30.53
ffmpeg -y -ss 00:10:30.53 -i bbb_sunflower_1080p_60fps_normal.mp4 last_4_seconds.mp4

Seeking can be a bit tricky, so you may want to learn more about seeking here.

Trimming from the end of the clip

You can also set an out-point for an input file, therefore shortening it. There are two options for this:

-t: This sets the duration.
-to: This sets the timestamp where the input video should stop.

These two are mutually exclusive, and also they do the same if no -ss is specified. The value can be specified in seconds (5 or 5.2) or as a timestamp (HOURS:MM:SS.MILLISECONDS).

Let’s experiment with them!

# "Get 30 seconds of the input."
ffmpeg -y -t 30 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -t 00:00:30.0 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4

# "Get everything until the content's 30th second."
ffmpeg -y -to 30 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4
ffmpeg -y -to 00:00:30.0 -i bbb_sunflower_1080p_60fps_normal.mp4 first_30_seconds.mp4

All four above commands result in exactly the same video. (For nerds: even the md5sum is the same.)

But let’s see how they perform when we introduce seeking!

# "Seek to the 10th second and get me 30 seconds of the input."
ffmpeg -y -ss 10 -t 30 -i bbb_sunflower_1080p_60fps_normal.mp4 part_between_10_and_40.mp4

# "Seek to the 10th second and get the content until the 30th second."
ffmpeg -y -ss 10 -to 30 -i bbb_sunflower_1080p_60fps_normal.mp4 part_between_10_and_30.mp4

The first command will result in a 30 second long video, while the second command will be 20 seconds long only!

The figure below shows the difference:

Editing without reencoding

FFmpeg can do something I’m not aware of in any other popular NLE: it can edit videos without reencoding them!

The usual workflow is to decode the data frames (a/v) into memory, modify them as much as we like and then encode them into a new video file. The problem with this is that unless you work with raw or lossless codecs, you’ll lose some quality in the process. Another issue with this approach is that it is computationally intensive.

For certain operations, you can configure FFmpeg, to keep the data frames intact, and this way, you can avoid decoding and encoding them! This is incredibly faster than regular transcoding, usually hundreds of times faster.

The “certain operations” are those that don’t need to modify the data frames themselves. For example, you can cut and trim this way. Also, you can manipulate streams while keeping others, like you can replace the audio track without touching the video frames.

All this is a bit of magic, and there are caveats you need to prepare for, but it is good if you know about this, as it is often handy!

The trick lies in two options:

-c:v copy: The “copy” video codec
-c:a copy: The “copy” audio codec

Let’s see a few examples!

Remove audio while keeping the video without reencoding

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -c:v copy -an copied_video_only.mp4

Here, we used the “-an” option, which removes all audio streams. I remembered it as “audio no”, but that is just my mnemonic:)

Let’s see how fast it was:

frame=38072 fps=20950 q=-1.0 Lsize=  310340kB time=00:10:34.51 bitrate=4006.7kbits/s speed= 349x

So It processed the whole 10 minutes of video in 2 seconds, 349x faster than playback, with 20950 fps!

Remove video while keeping the audio without reencoding

ffmpeg -i bbb_sunflower_1080p_60fps_normal.mp4 -c:a copy -vn copied_audio_only.wav

Here, we used the “-vn” option, which removes all video streams. I remembered it as “video no”.

Let’s see how fast it was:

size=   24772kB time=00:10:34.14 bitrate= 320.0kbits/s speed= 776x

776x faster than playback, finished in about a second, not bad!

Cut and trim without reencoding

ffmpeg -ss 10 -t 10  -i bbb_sunflower_1080p_60fps_normal.mp4 -c:a copy -c:v copy part_from_10_to_20_copied.mp4

There could be precision issues with seeking while you do this, so you may want to learn more about seeking and copying here.

Replace audio on video file without reencoding

We have removed audio and video already, but what if we want to swap them?

ffmpeg -y \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-i voice_recording.wav \
-map "0:v" -map "1:a" \
-c:v copy -c:a copy \
bbb_with_replaced_audio.mov

There is quite a lot going on in here, so let’s explain the parts!

First, we have two inputs (-i), meaning we are better off manually specifying the mapping. The command would work without the “-map” options, but it would ignore our second input.

-map "0:v" -map "1:a" means that please use the first file’s (first) video stream and the second file’s (first) audio stream.

With -c:v copy -c:a copy, we require FFmpeg to copy the already encoded data packets without touching them. Therefore FFmpeg’s work is mostly really just copying bytes, no decoding, no encoding.

Not surprisingly, that’s what we see in the stream mapping too:

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=38072 fps=9750 q=-1.0 Lsize=  320645kB time=00:10:34.51 bitrate=4139.7kbits/s speed= 162x

And since it is just copying, it was crazy fast, 162x of the playback speed, or almost 10k frames per second!

But!

Execute the exact same command, but with “bbb_with_replaced_audio.mp4” (.mp4 container instead of .mov) as an output file! You’ll get this:

Could not find tag for codec pcm_s16le in stream #1, codec not currently supported in container

The message is quite clear. You can not have a pcm_s16le (raw WAV, say that 10 times:)) stream in an MP4 container. I’m not sure if it is FFmpeg’s or the container’s lack of support, but we need to solve this. If you run into this situation, you might consider two solutions:

Change the container: I’ve just tried MOV, and it worked.
Encode the audio: We still copy the video data, and encoding audio isn’t that painful.

I just showed you option #1, so let’s see option #2:

ffmpeg -y \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-i voice_recording.wav \
-map "0:v" -map "1:a" \
-c:v copy \
-c:a aac -b:a 320k -ar 44100 \
bbb_with_replaced_audio_aac.mp4

This copies the video frames and encodes our WAV into a supported codec to be held in the mp4 container. You can refer back to the audio encoding section if you want to learn more about that.

Here is the output:

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
...
frame=38072 fps=2176 q=-1.0 Lsize=  313058kB time=00:10:34.51 bitrate=4041.8kbits/s speed=36.3x

“Only” 36x faster than playback, 2176 fps, still not that bad!

Filtering overview

FFmpeg supports many audio and video filters. Currently, there are 116 audio and 286 video filters, but there are a bit more if we count the hardware accelerated ones too.

So how do we leverage them?

There are two ways to define filters, but I’m going to explain the complex filter, as the difference is not much, but it is more versatile. So there is a global option for FFmpeg, called: -filter_complex. With quite a weird syntax, you can specify all your filters and their parameters right after this option.

You can imagine the process with the following image:

Basically, your filter graph can access all the inputs (-i a.mp4 -i b.mp4 -i c.mp4), and it can produce as many outputs as you like (-map might be needed).

Basic syntax

Let’s take a look at a simple, basic example:

ffmpeg -y  -t 5 \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex1.mp4

Although -filter_complex is a global option, I like to put it after the inputs and before the outputs as it is a bit easier to overlook the whole command that way. Thankfully the command line parser of FFmpeg is smart enough, and it works.

The command above produces a 5-second-long video, where the text “HELLO THERE” is overlaid on the intro of Big Buck Bunny.

Let’s understand the weird format for specifying filters!

We’ll go bottom-up, and we’ll build it from there. So the most basic format is this:

FILTER_NAME=ARGUMENT1=VALUE1:ARGUMENT2=VALUE2

For example:

drawtext=text='HELLO THERE':y=20:x=30

The first thing before the first equal (=) sign is the filter’s name, which is the drawtext filter in this case. Then we have our first argument, “text” and its value “‘HELLO THERE’”. Right after that, separated with a colon (:) comes the next argument, “y” with a value of “20”.

You can guess what each of the text, y, x, fontsize and fontfile arguments do, as it is quite self-explaining. But especially for the first time, you’ll heavily rely on the filtering documentation to understand every filter and every argument.

Also, several characters are reserved, such as: , : = and a few others depending on your environment, so sooner or later you need to learn about escaping too.

To recap, our pipeline looks like this now:

Multiple filters in a chain

This previous command is a single filter chain that consists of a single filter only, but you could have more filters put right after each other! It means that the output of one filter will be the input for the next! The way to do this is by separating them with a comma!

Let’s draw two boxes with the drawbox filter!

ffmpeg -y  -t 5 \
-i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "  drawbox=x=10:y=10:w=100:h=100:color=red  ,  drawbox=x=200:y=200:w=100:h=100:color=blue  " \
filter_complex2.mp4

See? The output of the first filter is passed to the output of the second filter!

Let’s visualize our pipeline again:

Input and output pads

Now, we have skipped something this far, because for simple uses FFmpeg is smart enough to do it for us. And this is the specification of a chain’s input and output pads!

Let’s draw just a single rectangle for now:

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex "drawbox=x=10:y=10:w=100:h=100:color=red" filter_complex3.mp4

FFmpeg sees that the input for our filter chain is a single video file, and the output is a single output video file. Therefore, it safely assumes that we want that single input as the input of our single filter chain. And that single output should be the single output of our single output chain.

That’s really nice, as, in simple situations like this, we don’t need to assign and map inputs and outputs manually! But when we get more inputs, filter chains, or outputs, it is no longer possible. Therefore, we need to understand how to assign inputs and outputs!

First of all, let’s compare the following two command lines. They result in exactly the same result, but the second one represents what FFmpeg does internally (roughly):

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex "drawbox=x=10:y=10:w=100:h=100:color=red" filter_complex3.mp4

ffmpeg -y  -t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 -filter_complex "[0:v]drawbox=x=10:y=10:w=100:h=100:color=red[out_link_0]" -map "[out_link_0]" filter_complex3.mp4

Do you see the difference? Before our filter chain, an “input pad” is defined: [0:v]. The expected format between the square brackets is documented in the stream selection section of the official documentation, and this article already covered it.

But, a quick summary:

0:v: This means the first video stream of the first input file.
0:v:0: Means exactly the same thing but in a long form.
0:0: Means the first stream of the first input file (not recommended, as it could be anything in theory. It could be a subtitle stream, a thumbnail, a video or an audio stream…)
0:a: This means the first audio stream of the first input file.
0:a:0: Means exactly the same thing but in a long form.
0:a:1: Means the second (index #1) audio stream of the first input file.

So we can specify which input file should be connected to which input of the filter graph!

Also, something similar is going on at the end! Do you see, the [out_link_0] output pad definition at the end of our filter chain?

The naming here is easier, as basically you can specify any arbitrary name in here. It roughly means, “please store the output data under this name”.

And when you specify your output file, you can or need to map it by selecting one of your filter graph outputs! Therefore, we must add the -map “[out_link_0]” option before our output file.

This map option means this: “Please save the data stream with this name into the following output file.”

This is how you can visualize this input/output mapping:

Multiple chains

Coming from the previous sections, you are now ready to see and understand an even more complicated configuration, which has multiple input files, output files, and filter chains!

ffmpeg -y  \
-i train.jpg \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "[0:v]drawbox=x=10:y=10:w=100:h=100:color=red[train_box] ; [1:v]drawbox=x=10:y=10:w=100:h=100:color=red[bbb_box]" \
-map "[train_box]" filter_complex4_train.jpg \
-map "[bbb_box]" filter_complex4_bbb.mp4

Let’s see the output (two files next to each other):

We had two inputs, and we got two output files, an image, and a video, with a red rectangle on them, with a single command!

Are you still here? I hope! Let’s understand what happened in that crazy command! We have two input files:

-i train.jpg: A simple image file
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4: Our video file, but to make it quick, just the first five seconds of it

Then the first thing to note is that we have two filter chains! They are separated with a ”;”.

Our first filter graph is this: [0:v]...[train_box]

This requests the first input file as an input
Draws a red box
Saves the output into the “train_box” output pad

Our second filter graph is this: [1:v]...[bbb_box]

This requests the second input file as an input
Draws a red box
Saves the output into the “bbb_box” output pad

And finally, we got two outputs, each mapping to one of the outputs of the filter graph:

-map “[train_box]” filter_complex4_train.jpg
-map “[bbb_box]” filter_complex4_bbb.mp4

Here is the same thing visually:

If you are thinking about making it even more complex and making filter graphs that combine multiple inputs into one for example, you are on the right track! It is possible, and we will get to that!

This was the introduction to the filtering system and its syntax.

Editing video

Now let’s get to know a few filters and make some interesting stuff!

Resizing or scaling

The scale filter is a simple one, yet it is quite powerful!

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "scale=width=600:height=-1:force_original_aspect_ratio=decrease" \
filter_complex5_scaled1.mp4

The arguments speak for themselves, but a few things:

Specifying -1 to either width or height means rescaling while keeping the aspect ratio.
“force_original_aspect_ratio” can be increase, decrease. Meaning it will increase or decrease the image to fit the specified bounding box while keeping the aspect ratio.

Adding text

We have already covered this a little, so let’s dive deeper!

This is what we used earlier:

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex1.mp4

Now let’s discover how to align the text!

Many filters, including drawtext, support variables in some of its argument’s values. If you scroll down in the documentation of drawtext, you’ll find this:

“The parameters for x and y are expressions containing the following constants and functions: ”

And after this part, you’ll see many variables which you can include in your x and y variables!

Let’s see:

# Align the text to the center
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=h/2-text_h/2:x=w/2-text_w/2:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex6_center.mp4
# y=h/2-text_h/2 means: y position = (image height / 2) - (text height / 2)

# Align the text to the right:
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=30:x=w-text_w-20:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex6_right.mp4
# x=w-text_w-20 means: x position = image width - text width - 20pixel padding


# Align the text to the bottom:
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='HELLO THERE':y=h-text_h-20:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex6_bottom.mp4
# y=h-text_h-20 means: y position = image height - text height - 20pixel padding

And this is what we’ll get in the end:

I need to mention one good trick that might not be obvious at first. So the text_h variable is a tricky one, because different text will be of different height! E.g.: ”____” and “WWW” will result in a different height.

For this reason, you do not always want to use text_h or even just a constant y=value expression but rather, you need to align text by its baseline. So just remember to use the “ascent” variable whenever you need to align text vertically!

Check out these two examples! Each has two drawtext filters printing ”_” and “_H”:

# This one uses y=200 for both, still the text isn't aligned properly!
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='_':y=200:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf,drawtext=text='_H':y=200:x=500:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex7_bad_text.mp4

# This one uses y=200-ascent for both and the text is aligned as expected!
ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-filter_complex "drawtext=text='_':y=200-ascent:x=30:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf,drawtext=text='_H':y=200-ascent:x=500:fontsize=200:fontfile=/usr/share/fonts/truetype/freefont/FreeSerif.ttf" \
filter_complex7_good_text.mp4

Now let’s compare the difference:

See? This is the difference between aligning the “top left” or the “baseline” of the text!

Adding an overlay

Overlaying is a very interesting thing to do with FFmpeg. Let’s jump right in!

Basic

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "overlay" \
filter_complex8_overlay1.mp4

Easy as that!

Of course, the overlay filter has a ton of options, but I wanted to demonstrate the easiest possible command line. We don’t even need to mess with input/output pads, as FFmpeg automatically understands the situation: two inputs for the overlay filter and its single output into a single output.

But just to exercise, we could have executed it like this:

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "[0:v][1:v]overlay[output]" \
-map "[output]" filter_complex8_overlay2.mp4

And this would result in the same output! Check it out, now I have specified the two inputs for the overlay: [0:v][1:v]!

Aligned

Let’s align the smiley into the center!

As we have seen with the drawtext, the overlay filter’s arguments also support a few dynamic variables. We’ll use those to achieve what we want!

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "overlay=x=main_w/2-overlay_w/2:y=main_h/2-overlay_h/2" \
filter_complex8_overlay3.mp4

Preprocessing the input for overlay

Let’s get a bit creative!

I want to make it smaller, and I also want to blur it!

Now pause for a minute, and think about it, how you’d do that?!

…

Ready?

ffmpeg -y  \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-i smiley.png \
-filter_complex "[1:v]scale=w=200:h=-1,gblur=sigma=3[smiley] ; [0:v][smiley]overlay=x=100:y=100" \
filter_complex8_overlay4.mp4

For this we needed to have two filter graphs!

The first one is this: [1:v]scale=w=200:h=-1,gblur=sigma=3[smiley]

Scales the input image (the smiley).
Then the scaled output is also blurred.
Then the output is saved into the output pad named “smiley”.

Then, we have our second filter graph: [0:v][smiley]overlay=x=100:y=100

This takes as input the first input file (the video).
This also takes as input the output pad named “smiley”. (We are connecting two chains this time!)
Then the overlay filter does its overlaying thing, and we trust FFmpeg to pair the unnamed output with the single output file we specified.

Reusing content

Let’s do one more, a really complicated one!

Let’s have the outro overlaid over the intro!

ffmpeg -y \
-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-t 5 -ss 00:09:40 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-filter_complex " [1:v]scale=w=1920/2:h=-1[outro]; [0:v][outro]overlay" \
filter_complex8_overlay5.mp4

We could have achieved it in several ways, e.g. we could use the trim filter, but to keep it easy, we just open the same file twice and seek/trim them.

-t 5 -i bbb_sunflower_1080p_60fps_normal.mp4: Open the video, and keep the first five seconds of it.
-t 5 -ss 00:09:40 -i bbb_sunflower_1080p_60fps_normal.mp4: Open the same video again, but seek to the end and keep five seconds from there.

Then we have two filter graphs again, one scales down the outro, and the second is just an overlay.

Are you excited?:) I hope these made-up examples opened up your eye for the possibilities, and I hope you’ll create very creative stuff with this knowledge!

Chroma keying, green screen, blue screen

In this section, we’ll use chroma keying to remove the background from Big Buck Bunny’s intro, and then we will put the transparent logo over the original video, as if it would be some kind of a logo overlay!

ffmpeg -y \
-ss 0.5 -t 2 -i bbb_sunflower_1080p_60fps_normal.mp4 \
-ss 10 -i bbb_sunflower_1080p_60fps_normal.mp4  \
-filter_complex " [0:v]chromakey=color=0xfdfdfd:similarity=0.1:blend=0.2 , scale=w=-1:h=300 , loop=loop=-1:start=0:size=120[intro] ; [1:v][intro]overlay=x=-40:y=-40" \
-t 10 filter_complex9.mp4

So just to recap, Big Buck Bunny’s first few seconds are like this:

And this is the result:

Also, the butterfly moves its wings repeatedly!

Let’s examine the command!

-ss 0.5 -t 2 -i bbb_sunflower_1080p_60fps_normal.mp4: We read in the intro from 0.5 to 2.5 seconds.
-ss 10 -i bbb_sunflower_1080p_60fps_normal.mp4: We read in the video, starting from the 10th second.

Then we have two filter graphs, the first being this:

[0:v]chromakey=color=0xfdfdfd:similarity=0.1:blend=0.2 , scale=w=-1:h=300 , loop=loop=-1:start=0:size=120[intro]

As we see, we have three filters in here!

chromakey: This one takes a color and a few parameters as input, and outputs transparent frames. The specified color + the blended areas will be the transparent sections. In our case we replaced the white-ish (#fdfdfd) background color with transparency.
scale: We resize the full 1080p image into something around 300px high.
loop: With the loop filter, we repeat all the 2 seconds worth of 120 frames (60*2) over and over again, to have the butterfly move its wings continuously.

And then, finally we have the second filter graph:

[1:v][intro]overlay=x=-40:y=-40

Nothing fancy, just an overlay of the original video and our chrome keyed intro.

What else?

You might want to check out a few more filters, that I didn’t cover here.

Here are just a few interesting ones:

Audio manipulation

In this chapter, we’ll be going to check out some audio manipulation techniques with FFmpeg!

First of all, let’s see our example file:

It is a voice recording, and it is intentionally… well, quite bad.

From the waveform, it is obvious that there are very different volume ranges in it. This is an example recording where each sentence was read in different strengths: “normal”, “whisper” or “powerful”, this is why you see repeating patterns of amplitude ranges on the image.

It isn’t visible, but it has some noise too, and of course, it is not normalized or enhanced in any way. Yet.

Please note that there are different scenarios, requirements, and ways to enhance audio. This is a simplified method to show the outline of the process in this article. I’m not an audio engineer, although I have some experience in the area. So if you know it better, feel free to fine-tune it for yourself even more, or contact me and recommend improvements!

I’m showing an example here with a very rough input, one that you’d just reject in real life as it would be useless due to its quality. But it is an excellent example to show the different steps of the enhancing process and to see what can be done to it!

The following steps are built upon each other, and we’ll reach the complete command at the end!

Don’t forget that these settings are specific to this voice recording. Sadly this can not be generalized too much.

Gate

Let’s start with the gate filter!

A gate is like a switch that opens only if the signal is stronger than the threshold. So if the signal level is lower than the threshold, it cuts to complete silence. Although you might soften or delay this cut with the knee, attack, and release arguments.

We’ll use this filter as a basic noise reduction method now! This helps us remove the noise between words and sentences by cutting it to silence. It doesn’t remove noise in any other way, e.g. it doesn’t touch the static on the voice itself.

Check this out!

ffmpeg -y \
-i voice_recording.wav \
-filter_complex "agate=threshold=0.01:attack=80:release=840:makeup=1:ratio=3:knee=8" \
gate.wav

Let’s hear it: gate.wav

And let’s see it:

As you can see, the “silent” parts were attenuated heavily, while the above-the-threshold parts remained similar. Those parts were still affected by the knee, attack, and release arguments determining how hard (knee) and quick (attack/release) the cut is.

I’ve left a quite high release timeout here to avoid sudden dips in the amplitude.

This is where we are right now:

The silent parts are more silent than before, but still, the amplitude range or the dynamic range is quite high. You must change your volume levels to hear everything and void blowing your speakers/brain out.

Equalization

Before fixing that, let’s do a bit more housekeeping. Let’s do some equalization and frequency filtering!

We’ll use these filters:

ffmpeg -y \
-i gate.wav  \
-filter_complex "highpass=f=100:width_type=q:width=0.5 , lowpass=f=10000 , anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1" \
gate_eq.wav

Let’s hear it: gate_eq.wav

This command gradually attenuates frequencies below 100hz, as there are not much valuable content in there, but it can really lower the clarity of the speech.

Then we do the same, but for frequencies above 10 kHz. This is mostly needed because we have a lot of high-frequency noise, so this is a workaround for those. Also, a male voice is generally deeper than a woman’s, so you might want to pay attention to how low you can put the bar.

Then comes anequalizer, which has a crazy an exceptional way of setting its arguments:

This: anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1 means:

at 250hz with a width of 100hz boost by 2 db, with Chebyshev type 1 filter on channel 0.
at 700hz with a width of 500hz attenuate by 5 db, with Chebyshev type 1 filter on channel 0.
at 2000hz with a width of 1000hz attenuate by 2 db, with Chebyshev type 1 filter on channel 0.

I agree. You might have used a friendlier equalizer in your life than this one:)

Those values are based on experimentation and common recommendations for voice. Feel free to tune it for your own needs!

Let’s compare the frequency plots before and after:

Tip: To see the frequency plot in Audacity, open a file, select all, and choose Analyze → Plot spectrum!

Compression

The compressor filter applies dynamic range compression on the incoming audio data. To simplify this, the compressor varies the attenuation based on the incoming signal level. Basically, when you watch a badly mastered movie, this is what you are doing. When it is way too loud in some action scene, you reach for the remote control or mouse to lower the volume, but in the next moment, you will not hear what your heroes are saying, so you increase it back again.

Dynamic range compression roughly does the same. You may set it up in a way so that it would attenuate louder parts, therefore keeping the overall volume range relatively small.

It often happens that performers on the stage use a high dynamic range. Many performers will shout at one moment and then whisper in the next to increase drama or keep the attention. If you want to avoid manually adjusting the volume in real-time (while blowing off your speakers and pulling your hair out), then a compressor will save you in these situations!

This is why our example audio consists of different speaking strengths, so that we could see the dramatic effect of this filter.

ffmpeg -y \
-i gate_eq.wav \
-filter_complex "acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6" \
gate_eq_comp.wav

Let’s hear it: gate_eq_comp.wav

And let’s compare the result of this with the original waveform!

Original:

Result:

Quite dramatic, isn’t it?:)

Let’s analyze this: acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6

First, level_in=6 sets the input gain. It is 1 by default, but since our example, audio is extremely silent at places, we boost up the whole thing before processing.

Then threshold=0.025 defines that everything above 0.025 should be attenuated.

Based on the image below, I’ve decided to cut at this point, as this is above most of the whispering, which cuts hard pops and “s”-es even in the “whisper zone”.

Then ratio=20 means 1:20 in attenuation ratio, which means that if the level rises 20 dB above the threshold, it will be only 1 dB above the line after the attenuation. Basically, this is a very strong compression ratio, it is almost a limiter.

This far, we boosted the signal, then turned down everything that was above our “whisper line” with a quite strong ratio, and now, everything is basically at the whisper level, even the parts that are shouting.

Finally, with the makeup=6 we just bring back everything to the level where the “normal” parts were before.

Let’s take a look back now, to understand why we used the gate and did the equalization before the compressor.

Generally, you want to remove unneeded parts and frequencies before compression, as the compressor will likely increase those too! So by removing most of the noise in the gaps, we avoided level_in=6 to increase them too! And the same goes for the high- and lowpass filtering.

Changing the volume

Now, if we want to make the result a bit louder, we could increase the previous step’s makeup argument, or leverage the volume filter.

While we are at it, let’s cut the first 4 seconds too with -ss 4.

ffmpeg -y \
-ss 4 -i gate_eq_comp.wav \
-filter_complex "volume=1.1" \
gate_eq_volume_comp.wav

Let’s hear it: gate_eq_volume_comp.wav

Let’s make audio gate again

Excuse me for that title:)

So as I’ve described earlier, compression can amplify the noises, so you might want to run the result through a gate again:

ffmpeg -y \
-i gate_eq_volume_comp.wav \
-filter_complex "agate=threshold=0.1:attack=50:release=50:ratio=1.5:knee=4" \
gate_eq_volume_comp_gate.wav

Let’s hear it: gate_eq_volume_comp_gate.wav

In this case, I’ve used a softer gate, with ratio=1.5. Because of this, I could use shorter attack and release delays too, as the attenuation is not that strong, it isn’t causing hard dips in the audio.

Putting it all together

Just a single command could have achieved all the steps above:

ffmpeg -y \
-i voice_recording.wav \
-filter_complex "agate=threshold=0.01:attack=80:release=840:makeup=1:ratio=3:knee=8 , highpass=f=100:width_type=q:width=0.5 , lowpass=f=10000 , anequalizer=c0 f=250 w=100 g=2 t=1|c0 f=700 w=500 g=-5 t=1|c0 f=2000 w=1000 g=2 t=1 , acompressor=level_in=6:threshold=0.025:ratio=20:makeup=6 , volume=1.1 , agate=threshold=0.1:attack=50:release=50:ratio=1.5:knee=4" \
gate_eq_volume_comp_gate_together.wav

I just copy-pasted all the filters right after each other with a comma between them.

Isn’t it beautiful? Yeah, it isn’t, but it is very practical:)

For the last time, check out the difference:

Original: voice_recording.wav
Final: gate_eq_volume_comp_gate.wav

It has less noise, more clear voice, and a small volume range. Therefore it is easy on your ears!

What else?

You might want to check out a few more filters that I didn’t cover here.

Here are just a few interesting ones:

Documentation

For your convenience, let me list the most important documentations that might be important for you! Most of these were already linked many times in this article.

If you got this far from top to bottom, then you are a true hero! I hope you enjoyed this, and I also hope that it inspired you to create something awesome with FFmpeg! Please consider donating to FFmpeg – they are fantastic.

If you’re looking to take your creative projects to the next level, check out our products - Creative Editor SDK, Video Editor SDK, and Photo Editor SDK. These versatile tools empower you to bring your vision to life, whether you’re editing images, crafting stunning videos, or unleashing your artistic talents.

Thanks for reading! Let us know what you think on Twitter! To stay in the loop, subscribe to our Newsletter.

How to Apply Filters and Effects to an Image in Flutter

Natalia — Tue, 15 Nov 2022 10:51:25 GMT

Introduction

Having previously discussed how Flutter deals with resizing and various overlays, in this tutorial, we will talk about a more popular type of image manipulation; applying filters and effects. At least since the ubiquity of social media platforms such as Instagram, Snapchat or more recently TikTok, applying various filters and effects has become an expected feature of any application dealing with image manipulation.

Applying Filters and Effects in Flutter demo application

In this tutorial, we go through the process of applying filters and effects to images in Flutter. Like the example given in the video, we will examine the options for developing an image editor and the effects available in the Flutter framework. The Git repository supporting this article can be cloned with the command provided below:

git clone git@github.com:nataliakzm/Applying_Filters_and_Effects_to_Images_Flutter.git

One of the leading solutions proposed by the Flutter team on this issue was the ImageFiltered class. Overall, the widget deals with major image manipulations (from blurring to matrix transformation, rotation etc.), transforming and rearranging image pixels. To create an ImageFilter, you need to follow a particular structure and apply the ImageFilter class to its child elements:

Moreover, in order to start working with ImageFiltered within your Flutter application, first, don’t forget to import this library on the top of your .dart file.

const ImageFiltered({
  super.key,
  required this.imageFilter,
  super.child,
  this.enabled = true,
}) : assert(imageFilter != null);

import 'dart:ui';

Then, ImageFiltered offers several filter choices for image processing, depending on the specification of your app or visual asset.

So, for example, with the help of ImageFilter.blur or the so-called Gaussian blur, the pixels of the image can be blurred, which can, for instance, be used to create a graphic element that can serve as a background image.
On the other hand, two other filters such as ImageFilter.dilate and ImageFilter.erode affect the pixels’ value by either enlarging them to the max significance within the specified range along the x and y axes or by shrinking pixels to their minimum values;
You can also easily rotate, scale and change your asset in other ways by transforming its matrix with ImageFilter.matrix;
Finally, ImageFilter.compose is handy for merging two filters and combining their effects.

At this point, the purpose of each effect seems much clearer, but we still miss some crucial steps to implement these filters in your Flutter application. Let’s see some real examples:

ImageFilter.blur constructor

To apply a blurring effect on your image, you have to call an ****ImageFiltered.blur(), and adjust the values of sigmaX and sigmaY arguments for blurring the image. Note that the asset must be called as a child element of the filter to achieve the desired effect.

Container(
		child: Center(
				child: Column(
						mainAxisAlignment: MainAxisAlignment.center,
						children: [
      // Blur an Image
      ImageFiltered(
					imageFilter: ImageFilter.blur(sigmaY: 5, sigmaX: 5),
					child: Image.asset("assets/logo.png"))

ImageFilter.matrix constructor

The situation is slightly different with ImageFilter.matrix filter since the effect can be initialized by one of two methods – Float64List.fromList or Matrix4.rotationZ . Thus, for the first method, the user needs to directly provide the matrix as a Float64List as it is done below:

// Matrix Transformation
ImageFiltered(
		imageFilter: ImageFilter.matrix(Float64List.fromList([
			1, 0.5, 0.0, 0.0,
			0.0, 1, 0.0, 0.0,
			0.0, 0.0, 1.0, 0.0,
			0.0, 0.0, 0.0, 1.0,
    ])),
		child: Image.asset("assets/logo.png", scale: 3.5)) // Note that scale is optional and depends on the image size

While the Matrix4 class simply provides rotation in Z directions:

// Matrix4 Rotation
ImageFiltered(
		imageFilter: ImageFilter.matrix(Matrix4.rotationZ(0.2).storage),
		child: Image.asset("assets/logo.png", scale: 6)) //Note that scale is optional and depends on the image size

ColorFiltered Effects

Alternatively, Flutter has several options to integrate some filters without calling any extra library. One such example is the ColorFilter function executed within the widget of the same name. ColorFiltered processes two colors and outputs only one of them, which is eventually displayed at the top of the layer.

 // Black & White Effect
ColorFiltered(
		colorFilter: const ColorFilter.mode(Colors.grey, BlendMode.saturation),
		child: Image.asset("assets/logo.png"))

Users can also engage with several construction methods: from directly calling the sRGB gamma curve with ColorFilter.linearToSrgbGamma() widget or its inverse prototype with ColorFilter.srgbToLinearGamma() to transforming the color by a 5x5 matrix within ColorFilter.matrix() function or simply applying the blend mode with a given color specified in ColorFilter.mode().

Final Thoughts

In conclusion, Flutter’s image processing capabilities are limited by a small variety of effects and often depend on the developer’s ability to handle complex concepts like matrix or sRGB curves. In this article, we reviewed some of the main image processing techniques proposed by the Flutter team and examined how to integrate them into photo editing apps. However, it is not always possible to allocate several days to calculate the correct matrix for transforming just one image when it comes to processing a large number of graphic assets. In such cases, you may want to think about using the PhotoEditor SDK for your next Flutter app. Just like in the interactive example below, PhotoEditor SDK has many different effects and filters that will suit any of your requests.

Applying Filters via PhotoEditor SDK

To learn more, check out our article on PhotoEditor SDK and how to set it up in your app and official PE.SDK guides. You can also follow our guide on how to integrate video editor for Flutter into your app. For any of your questions that might occur in the process, you can contact our Support, who will be glad to help you.

How to Draw on Images in React Native

Natalia — Thu, 20 Oct 2022 13:09:38 GMT

It has become a commonplace requirement for many applications to manipulate images in one way or another before users are ready to share them. Therefore, it is crucial to foresee end-user needs while working on an app involving any interaction with graphical content. Image annotation can be necessary in various cases, such as simple inspection apps, bug reporting software, or machine learning applications relying on manual user input.

In this article, we are shedding light on how to draw on images in React Native with the help of a simple HTML5

Enable the characteristics of the
Set the thickness and color of the brush instrument;
Define at what moment your brush needs to be activated and when the drawing process will finish;
Start drawing!

Let’s first clear up the purpose of

Canvas Integration for Images

<canvas> is an HTML component commonly used for creating and manipulating graphics, animation, and other visualization features. This element provides JavaScript APIs enabling image overlays and handling user input to make drawing possible. Moreover, while we have several methods for image manipulation with

<! -- HTML example of using canvas: -->
<canvas id="myCanvas" width="200" height="100"></canvas>

You might wonder why you cannot easily substitute <canvas> with some lines of HTML and CSS in your code. The answer lies on the surface: <canvas> creates a single flattened graphic rather than multiple components lying on top of each other (i.e., typically the output of HTML and CSS).

Now, let’s see how to implement this powerful element in your React Native application:

Drawing with Canvas

import React, { useEffect } from 'react';

// CanvasContext here is used for drawing onto the canvas
import { useCanvas } from './CanvasContext';

export function Canvas() {
  const { canvasRef, prepareCanvas, startDrawing, finishDrawing, draw } =
    useCanvas();

  useEffect(() => {
    prepareCanvas();
  }, []);

  return (
    <canvas
      onMouseDown={startDrawing}
      onMouseUp={finishDrawing}
      onMouseMove={draw}
      ref={canvasRef}
    />
  );
}

We start by enabling the characteristics of our <canvas> container, which are meant to render the CanvasContext component presented below.

The size of our future canvas is determined with prepareCanvas, which includes both the height (canvas.height) and width (canvas.width) of the HTML element. For the drawing process itself, it is necessary to assign not only the thickness and color of the brush instrument (context) but also take into account its activation (startDrawing with onMouseDown) and how the user can finish a stroke (finishDrawing with onMouseUp).

import React, { useContext, useRef, useState } from 'react';

// enabling drawing on the blank canvas
const CanvasContext = React.createContext();

export const CanvasProvider = ({ children }) => {
  const [isDrawing, setIsDrawing] = useState(false);
  const canvasRef = useRef(null);
  const contextRef = useRef(null);

  // defining width & height of the canvas
  const prepareCanvas = () => {
    const canvas = canvasRef.current;
    canvas.width = window.innerWidth * 2;
    canvas.height = window.innerHeight * 2;
    canvas.style.width = `${window.innerWidth}px`;
    canvas.style.height = `${window.innerHeight}px`;

    // defining the thickness and colour of our brush
    const context = canvas.getContext('2d');
    context.scale(2, 2);
    context.lineCap = 'round';
    context.strokeStyle = 'black';
    context.lineWidth = 5;
    contextRef.current = context;
  };

  const startDrawing = ({ nativeEvent }) => {
    const { offsetX, offsetY } = nativeEvent;
    contextRef.current.beginPath();
    contextRef.current.moveTo(offsetX, offsetY);
    setIsDrawing(true);
  };

  const finishDrawing = () => {
    contextRef.current.closePath();
    setIsDrawing(false);
  };

  const draw = ({ nativeEvent }) => {
    if (!isDrawing) {
      return;
    }
    const { offsetX, offsetY } = nativeEvent;
    contextRef.current.lineTo(offsetX, offsetY);
    contextRef.current.stroke();
  };

  // Once the canvas is cleared it return to the default colour
  const clearCanvas = () => {
    const canvas = canvasRef.current;
    const context = canvas.getContext('2d');
    context.fillStyle = 'white';
    context.fillRect(0, 0, canvas.width, canvas.height);
  };

  return (
    <CanvasContext.Provider
      value={{
        canvasRef,
        contextRef,
        prepareCanvas,
        startDrawing,
        finishDrawing,
        clearCanvas,
        draw,
      }}
    >
      {children}
    </CanvasContext.Provider>
  );
};

export const useCanvas = () => useContext(CanvasContext);

Drawbacks of the Canvas Method

Despite the canvas component offering powerful features, it is also quite demanding to use in its original form. For example, it may be problematic to adjust the properties of the brush to ensure that you can draw with smooth movements. Thus, implementing any solution for this issue would further increase the coding time and reduce the compatibility between different frameworks. Also, separately defining canvas and brush properties overcomplicates the code and requires advanced sources for successful integration into your React Native app. Although some temporary solutions aim to facilitate this process (such as react-native-image-draw or react-native-sketch-draw), none provides a permanent solution with clear documentation and guidance.

If you wonder about a magic door to avoid this coding nightmare, you can always choose a commercial, all-in-one solution like PhotoEditor SDK. Implementing this SDK only takes a few lines of code, saves time and expenses in building an app, and you can expect great support from the developers. This way, you can skirt the hassle of coding an editor and instead focus on creating a great product.

PhotoEditor SDK Integration for Drawing on Your Images

To get started integrating the PhotoEditor SDK in your React Native app, refer to this guide from the official documentation. You can then ensure comfortable image interaction for your users with the optimized brush tool, as in the example below.

Conclusion

This article aimed to examine image editing instruments suitable for React Native framework. The HTML <canvas> element could have big potential as it provides fundamental tools for creating graphics or animation. However, its application is time-consuming and requires comprehensive knowledge in the field. Therefore, if you want to guarantee a smooth and easy integration of image drawing tools for your app, consider using advanced software like PhotoEditor SDK.

Comparing the Top 5 Open Source Image Manipulation Libraries for React Native

Natalia — Thu, 08 Sep 2022 06:56:00 GMT

While integrating visual content into your application, it is crucial to take into account image manipulations that can significantly enhance the quality of your product.

Thus, for example, reducing the background noise in combination with cropping the image or resizing can direct the user’s attention by eliminating distracting objects, while efficient caching of large images can speed up the page load. In fact, the variety of manipulation techniques is limited only by your needs and skills. In this guide, we are discussing five of the best image processing libraries that could be easily employed in your Reactive Native application. The image editing libraries are ranked according to their complexity, so even if you are new to the framework, you can always find something useful.

Native Photo Editor

Let’s start with one of the most comprehensive RN packages, which combines multiple features at once and provides access to a variety of fundamental techniques. By employing the Native Photo Editor API, one can immediately modify the image within the native UI through Text or Stickers Integration, Animation, Cropping or Color, and Object Manipulation. Regarding prerequisites, this library needs to be installed on React Native version 61 or higher, whereas for iOS-based applications, make sure you have CocoaPods dependencies enabled.

Overall, the Native Photo Editor can meet your needs and be handy in cases where you are required to edit photos in the application’s sandbox.

Example

First, run the following command in order to install the library.

$ npm i react-native-photo-editor

Afterwards, the integration of its API slightly differs for iOS and Android platforms: for iOS, you will need to update your Podfile by adding ios/Podfile to the script and run:

use_native_modules!

  pod 'RNPhotoEditor', :path => '../node_modules/react-native-photo-editor/ios'
  use_frameworks! :linkage => :static
  pod 'iOSPhotoEditor', :git => '<https://github.com/prscX/photo-editor>', :branch => 'master'

  post_install do |installer|
    installer.pods_project.targets.each do |target|
      if target.name.include?('iOSPhotoEditor')
        target.build_configurations.each do |config|
          config.build_settings['SWIFT_VERSION'] = '5'
        end
      end
    end
  end

  # Follow [Flipper iOS Setup Guidelines](<https://fbflipper.com/docs/getting-started/ios-native>)
  # This is required because iOSPhotoEditor is implemented using Swift and we have to use use_frameworks! in Podfile
  $static_framework = ['FlipperKit', 'Flipper', 'Flipper-Folly',
    'CocoaAsyncSocket', 'ComponentKit', 'Flipper-DoubleConversion',
    'Flipper-Glog', 'Flipper-PeerTalk', 'Flipper-RSocket', 'Yoga', 'YogaKit',
    'CocoaLibEvent', 'OpenSSL-Universal', 'boost-for-react-native']

  pre_install do |installer|
    Pod::Installer::Xcode::TargetValidator.send(:define_method, :verify_no_static_framework_transitive_dependencies) {}
    installer.pod_targets.each do |pod|
        if $static_framework.include?(pod.name)
          def pod.build_type;
            Pod::BuildType.static_library
          end
        end
      end
  end

Whereas for Android, make sure to add the following to your build.gradle:

buildscript {
    repositories {
        maven { url "<https://jitpack.io>" }
        ...
    }
}
allprojects {
    repositories {
        maven { url "<https://jitpack.io>" }
        ...
    }
}

Then, you can integrate this piece into your code in order to start playing with the library:

import PhotoEditor from 'react-native-photo-editor';

PhotoEditor.Edit({
  path: RNFS.DocumentDirectoryPath + '/photo.jpg',
});

Photo Manipulator

We are now moving to a more sophisticated image processing package, which opens up the possibility of programmatically modifying your visual asset. The Photo Manipulator package can be employed for projects built on React Native starting from version 60. Among the features it provides are Cropping and Resizing, Quality Optimization, Image Overlay, Text Integration and Batching of all features simultaneously.

This library can save you time when integrating image manipulation into your application due to its easy usage and powerful batch application of all features. The typical use case could be a photo editor like Instagram.

Example

Start your exciting adventure with Photo Manipulator by running in your terminal:

$ npm i react-native-photo-manipulator

Then, you can directly import its API to your code:

import RNPhotoManipulator from 'react-native-photo-manipulator';

const image = '<https://unsplash.com/photos/qw6qQQyYQpo/download?force=true>';

// Crop and resize the image to 200 x 150:
const cropRegion = { x: 5, y: 30, height: 400, width: 250 };
const targetSize = { height: 200, width: 150 };

RNPhotoManipulator.crop(image, cropRegion, targetSize).then((path) => {
  console.log(`Result image path: ${path}`);
});

// Optimize your image by enhancing its quality to 90, and save the result in jpeg format:
const quality = 90;

RNPhotoManipulator.optimize(image, 90).then((path) => {
  console.log(`Result image path: ${path}`);
});

// Overlay image on top of background image:
const overlay = '<https://www.iconfinder.com/icons/1174949/download/png/128>';
const position = { x: 5, y: 20 };

RNPhotoManipulator.overlayImage(image, overlay, position).then((path) => {
  console.log(`Result image path: ${path}`);
});

// Add text to your image:
const texts = [
  {
    position: { x: 50, y: 30 },
    text: 'Text 1',
    textSize: 30,
    color: '#000000',
  },
  {
    position: { x: 50, y: 30 },
    text: 'Text 1',
    textSize: 30,
    color: '#FFFFFF',
    thickness: 3,
  },
];

RNPhotoManipulator.printText(image, texts).then((path) => {
  console.log(`Result image path: ${path}`);
});

// Crop, resize and do operations (overlay and printText) on image:
const cropRegion = { x: 5, y: 30, height: 400, width: 250 };
const targetSize = { height: 200, width: 150 };
const operations = [
  {
    operation: 'text',
    options: {
      position: { x: 50, y: 30 },
      text: 'Text 1',
      textSize: 30,
      color: '#000000',
    },
  },
  {
    operation: 'overlay',
    overlay: '<https://www.iconfinder.com/icons/1174949/download/png/128>',
    position: { x: 5, y: 20 },
  },
];
const quality = 90;

RNPhotoManipulator.batch(
  image,
  cropRegion,
  targetSize,
  operations,
  quality
).then((path) => {
  console.log(`Result image path: ${path}`);
});

Image Crop Picker

Image Crop Picker is another example of a powerful open-source library for iOS and Android applications. It’s designed specifically for modifying the camera-based static assets and dynamic videos through cropping, compressing, and multiplication. Thus, the advantage that distinguishes this package from similar ones is its ability to crop and compress images, which is extremely important when creating an image editing app, since large files can cause issues with performance.

Moreover, it’s recommended to use version 0.25 and higher for the React Native based on v.60 (and above); for older versions, you need to apply older versions of the Image Crop Picker library.

Example

Let’s begin with the installation of the library:

$ npm i react-native-image-crop-picker

Once the system is set up, you can import the Image Crop Picker API with the following:

import ImagePicker from 'react-native-image-crop-picker';

// Crop a single image to a size 300 x 400:
ImagePicker.openPicker({
  width: 300,
  height: 400,
  cropping: true,
}).then((image) => {
  console.log(image);
});

// Crop several images with one command:
ImagePicker.openPicker({
  multiple: true,
}).then((images) => {
  console.log(images);
});

// Select image from camera and crop it to a size 300 x 400:
ImagePicker.openCamera({
  width: 300,
  height: 400,
  cropping: true,
}).then((image) => {
  console.log(image);
});

//This library also supports manipulations with video; check the documentation for more details.

Image Filter Kit

The following library could be an ideal manipulation tool for a photo editing application. Image Filter Kit library differs significantly from previous packages by the number of filters it supports. Thus, an asset can be modified in terms of its color/convolve matrix or composition, whereas the distracting objects could be blended, blurred, or extracted. Moreover, Android-only or iOS-only filters could also be implemented depending on the system you are working with.

The Image Filter Kit can be employed for the Reactive Native systems of version 64 and higher (note that in this case, it also limits Android to min v.21 and iOS min v.9). For the projects based on the RN from 57.1 (and up to v.64), the min Android version should be set to 17 and for iOS to v.9.

Example

Start by once again installing the API of this library to your device:

$ npm i react-native-image-filter-kit

The set of features supported by the library is vast, so here we give just a brief example from which you could start working with Image Filter Kit:

// Import the library
import {
  SoftLightBlend, // Blend filter
  Emboss, // Convolve matrix filter
  Earlybird, // CSSGram filter
  Invert,
  RadialGradient,
} from 'react-native-image-filter-kit';

const result = (
  <Earlybird
    image={
      <SoftLightBlend
        resizeCanvasTo={'dstImage'}
        dstTransform={{ scale: 'CONTAIN' }}
        dstImage={
          <Emboss // Emboss an image to create a realistic outline effect
            image={
              <Image
                style={{ width: 320, height: 320 }}
                source={require('./parrot.png')}
                resizeMode={'contain'}
              />
            }
          />
        }
        srcTransform={{
          anchor: { x: 0.5, y: 1 },
          translate: { x: 0.5, y: 1 },
        }}
        srcImage={
          <Invert // Invert and Adjust Radial Gradient of the image to red
            image={
              <RadialGradient
                colors={['rgba(0, 0, 255, 1)', '#00ff00', 'red']}
                stops={[0.25, 0.75, 1]}
                center={{ x: '50w', y: '100h' }}
              />
            }
          />
        }
      />
    }
  />
);

FastImage

Once you have finished modifying images, you could think about their final presentation on your users’ devices and the memory problems that can occur when large resources are incorrectly cached. The FastImage library is designed to handle such challenges and correctly process image caching information in order to avoid flickering, cache misses and low performance. Among other things the package is capable of are the support of GIF assets, Border radius, Preload images, and Headers Authorization. The only disadvantage of this package is the low support for the older versions of React Native (such as below v.60).

Example

To install the library:

$ npm i react-native-fast-image

Then, we are going to import the library and set up some main settings:

import FastImage from 'react-native-fast-image';

const YourImage = () => (
  <FastImage
    style={{ width: 200, height: 200 }}
    source={{
      uri: '<https://unsplash.it/400/400?image=1>',
      headers: { Authorization: 'someAuthToken' }, // Add authorization headers;
      priority: FastImage.priority.normal, // Adjust the priority of images;
    }}
    resizeMode={FastImage.resizeMode.contain}
  />
);

React Native Skia

In the end, we have a bonus library with extremely powerful 2D graphic capabilities. React Native Skia originates from the open-source Skia Graphics Library, which powers the graphics engine of Google Chrome, Chrome OS, Firefox OS Android, Flutter and many other platforms. Thus, by deploying this package into your RN product, you can benefit from fantastic drawings or even UI effects like Neumorphism and Glassmorphism.

Example

Don’t forget to install the library dependencies:

npm install @shopify/react-native-skia

//Alternatively, you can opt for running the library in your web browser with the expo (See [here](<https://shopify.github.io/react-native-skia/docs/getting-started/web/>) for more details):
//expo install @shopify/react-native-skia // Don't forget to uncomment this line

Once everything is set up, you can turn on the Skia engine with the following declarative API:

import { Canvas, Circle, Group } from '@shopify/react-native-skia';
export const HelloWorld = () => {
  const size = 256;
  const r = size * 0.33;
  return (
    <Canvas style={{ flex: 1 }}>
      <Group blendMode="multiply">
        <Circle cx={r} cy={r} r={r} color="cyan" />
        <Circle cx={size - r} cy={r} r={r} color="magenta" />
        <Circle cx={size / 2} cy={size - r} r={r} color="yellow" />
      </Group>
    </Canvas>
  );
};

CONCLUSION

Thus, the best React Native image processing libraries vary in terms of their functionality, from simple ones providing limited image manipulation capabilities to more advanced libraries capable of modifying the image and how it works in your user interface system.

If you are looking for a production-grade photo editor with all the above features and a customizable professional UI have a look at our PhotoEditor SDK. If you want to go beyond simple image editing and your project requires support for templates, creative automation, or more complex designs, you should check out the CreativeEditor SDK.

How To Crop and Trim Videos In Kotlin for Android

Paul — Fri, 02 Sep 2022 10:18:21 GMT

Cropping and trimming videos is a notoriously difficult task to achieve on Android. One way to implement this functionality is by using FFmpeg a free open-source suite of tools that can perform a wide range of tasks, from video converting to editing. Normally FFmpeg is used from the command line, to use it correctly in Android you have to understand its underlying APIs and how to use them.

In this tutorial, you will learn how to crop and trim videos in Android by using FFmpeg. Even if you are a beginner you should be able to follow the steps to achieve the desired results.

I try to summarise the most important basics that you need to know to manipulate videos with FFmpeg. After reading this article you should be able to use it in your own applications. Furthermore, I have developed a sample application and library that can be used to trim and crop videos with an Android device.

What is FFmpeg

FFmpeg is a great multimedia framework that is able to mux, demux, decode, encode, transcode, filter, stream, and play most media content that exists. FFmpeg supports different tools that can be used to develop an application to manipulate any kind of media to the desired output. You do not have any limitations on what to do with multimedia when using FFmpeg.

Unfortunately, to use FFmpeg in an Android App using Kotlin you have to compile and build the libraries to use them as a dependency in your project. This process is not straightforward because you have to manually compile a C/C++ library with help of the Android NDK.

Luckily, many people already have done this and we are able to use their compiled library in our project. However, if you want to compile FFmpeg from scratch you can do this. All necessary information should be available within the android developer documentation.

To show you how we can use FFmpeg in our app I will use a compiled FFmpeg library that can be found here. To find other FFmpeg libraries you can have a look at the official FFmpeg wiki where several pre-packaged sources are listed.

Setting up the project

In order to use the FFmpeg library of WritingMinds (or any other library), we have to follow some simple steps.

1. Add dependency for the FFmpeg library to your app-level build.gradle

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'com.writingminds:FFmpegAndroid:0.3.2'
}

2. Check if your device supports the current implementation of FFmpeg or not

fun initialize() {
        val ffmpeg = FFmpeg.getInstance(ctx.applicationContext)
        try {
            ffmpeg.loadBinary(object : LoadBinaryResponseHandler() {
                override fun onFinish() {
                    super.onFinish()
                }

                override fun onSuccess() {
                    super.onSuccess()
                }

                override fun onFailure() {
                    super.onFailure()
                }

                override fun onStart() {
                    super.onStart()
                }
            })
        } catch (e: FFmpegNotSupportedException) {
            Log.e("FFmpeg", "Your device does not support FFmpeg")
        }
    }

3. Initialize the FFmpeg module (leave command blank)

val ffmpeg = FFmpeg.getInstance(ctx)
       ffmpeg.loadBinary(object : FFmpegLoadBinaryResponseHandler {
            override fun onFinish() {
                Log.d("FFmpeg", "onFinish")
            }

            override fun onSuccess() {
                Log.d("FFmpeg", "onSuccess")
                val command = //TODO: the command will added here later
                try {
                    ffmpeg.execute(command, object : ExecuteBinaryResponseHandler() {
                        override fun onSuccess(message: String?) {
                            super.onSuccess(message)
                            Log.d(TAG, "onSuccess: " + message!!)
                        }

                        override fun onProgress(message: String?) {
                            super.onProgress(message)
                            Log.d(TAG, "onProgress: " + message!!)
                        }

                        override fun onFailure(message: String?) {
                            super.onFailure(message)
                            Log.e(TAG, "onFailure: " + message!!)
                        }

                        override fun onStart() {
                            super.onStart()
                            Log.d(TAG, "onStart")
                        }

                        override fun onFinish() {
                            super.onFinish()
                            Log.d(TAG, "onFinish")
                        }
                    })
                } catch (e: FFmpegCommandAlreadyRunningException) {
                            Log.e("FFmpeg", "FFmpeg runs already")
                }
            }

            override fun onFailure() {
                Log.e("FFmpeg", "onFailure")
            }

            override fun onStart() {
            }
        })

4. Implement the commands you want to use in your app

All commands that are supported by FFmpeg can be included in your app with help of an array. This is done by passing every command line argument as a single element within the array. The array will then be translated into an FFmpeg command using the execute method from FFmpeg:

ffmpeg.execute(command, object : ExecuteBinaryResponseHandler() { ... }

As we want to implement Trim and Crop I will show how this can be done using the arrayOf function.

Trim:

val command = arrayOf("-y", "-i", input, "-ss", startPos, "-to", endPos, "-c", "copy", output)

“-y”: overwrites output files without asking
“-i”: specifies an input file
input: the path of the source video to trim
“-ss”: specifies that the next value will be the starting point of the resulting video
startPos: the starting position in “%d:%02d:%02d” format
“-to”: specifies that the next value will be the end position of the resulting video
endPos: the end position in “%d:%02d:%02d” format
“-c” AND “copy”: defines that the stream will not be encoded. The resulting video will only be saved.
output: the path of the resulting video

To better understand every argument you can have a look at the official FFmpeg documentation.

Additionally, to provide arguments for startPos and endPos you normally would have to use a utility function that converts the timestamp of the start and end position into the desired “%d:%02d:%02d” format (a string):

fun convertTimestampToString(timeInMs: Float): String {
        val totalSeconds = (timeInMs / 1000).toInt()
        val seconds = totalSeconds % 60
        val minutes = totalSeconds / 60 % 60
        val hours = totalSeconds / 3600
        val formatter = Formatter()
        return if (hours > 0) {
            formatter.format("%d:%02d:%02d", hours, minutes, seconds).toString()
        } else {
            formatter.format("%02d:%02d", minutes, seconds).toString()
        }
    }

Crop:

val command = arrayOf("-i", input, "-filter:v", "crop=$w:$h:$x:$y", "-threads", "5", "-preset", "ultrafast", "-strict", "-2", "-c:a", "copy", output)

“-i”: specifies an input file
input: the path of the source video to trim
“-filter:v”: defines that a filtergraph is used
“crop=$w:$h:$x:$y”: use crop functionality to crop a part from the video start at point x:y and having a width(w) and height(h).
“-threads”: specifies that the next value will set the thread count
“5”: the number of threads to use
“-preset”: specifies that the next value will set the encoding preset
“ultrafast”: the encoding preset to use
“-strict”: specifies how strictly the standards should be followed
“-2”: the strict value
“-c:a” AND “copy” defines that the stream will not be encoded and ALL audio streams will also be used.
output: the path of the resulting video

Also, check the official FFmpeg documentation to better understand every argument.

5. Implement the UI

To create a fancy UI you should create a custom view that shows some of the frames available in the video (VideoPreviewView). Also, it should contain a SeekBar for cropping and a RangeSeekBar for trimming. The SeekBar will be used to switch to a certain timestamp within the video to see what is actually trimmed or cropped. The RangeSeekBar will only be used within the trimming UI to define the start and end position of the resulting video.

While SeekBar is an Android Widget the RangeSeekBar is more complicated but there exist several implementations that can be used: By Tiszideepan.

You can implement the VideoPreviewView with an easy approach. Divide the video into a specific number of frames that are based on the view’s width and display every frame sequentially within a single view to have a preview. To do this you need the width of the view and the duration of the video that can be found using the MediaMetadataRetriever. With the MediaMetadataRetriever, you can use getFrameAtTime() to fetch a single frame at a specific timestamp. If you now want to display a complete video preview you need to display viewWidth / frameWidth frames. Unfortunately, depending on the video length and video width, it could happen that only a few frames will be present within the VideoPreviewView. To fix this problem you have to maintain a threshold to ensure that a specific number of frames are displayed. This means that you have to crop the frames to a certain width until calculated frames equal the threshold.

The following code snippet will show how you can achieve this:

private fun createPreview(viewWidth: Int) {
        BackgroundExecutor.execute(object : BackgroundExecutor.Task("", 0L, "") {
            override fun execute() {
                try {
                    val threshold = 11
                    val thumbnails = LongSparseArray<Bitmap>()
                    val mediaMetadataRetriever = MediaMetadataRetriever()
                    mediaMetadataRetriever.setDataSource(context, videoUri)
                    val videoLengthInMs = (Integer.parseInt(
                        mediaMetadataRetriever.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION)
                    ) * 1000).toLong()
                    val frameHeight = viewHeight
                    val initialBitmap = mediaMetadataRetriever.getFrameAtTime(
                        0,
                        MediaMetadataRetriever.OPTION_CLOSEST_SYNC
                    )
                    val frameWidth =
                        ((initialBitmap.width.toFloat() / initialBitmap.height.toFloat()) * frameHeight.toFloat()).toInt()
                    var numThumbs = ceil((viewWidth.toFloat() / frameWidth)).toInt()
                    if (numThumbs < threshold) {
                        numThumbs = threshold
                    }
                    val cropWidth = viewWidth / threshold
                    val interval = videoLengthInMs / numThumbs
                    for (i in 0 until numThumbs) {
                        var bitmap = mediaMetadataRetriever.getFrameAtTime(
                            i * interval,
                            MediaMetadataRetriever.OPTION_CLOSEST_SYNC
                        )
                        bitmap?.let {
                            try {
                                bitmap = Bitmap.createScaledBitmap(
                                    bitmap,
                                    frameWidth,
                                    frameHeight,
                                    false
                                )
                                bitmap = Bitmap.createBitmap(bitmap, 0, 0, cropWidth, bitmap.height)
                            } catch (e: Exception) {
                                Log.e(TAG, "error while create bitmap: $e")
                            }
                            thumbnails.put(i.toLong(), bitmap)
                        }
                    }
                    mediaMetadataRetriever.release()
                    returnBitmaps(thumbnails)
                } catch (e: Throwable) {
                    Thread.getDefaultUncaughtExceptionHandler()
                        .uncaughtException(Thread.currentThread(), e)
                }
            }
        })
    }

Additionally, for cropping a video you need to display a crop rectangle within the UI to let users decide which part of the video they want to crop. Also, you have to give users the ability to reposition the crop area. To do this, you can use the ImageCropper library from ArthurHub which is initially designed to crop an image but can also be used for videos as you only need the values from the rectangle. Add the ImageCropper to your project:

dependencies {
    implementation 'com.theartofdev.edmodo:android-image-cropper:2.8.0'
}

Now, you can extend a CropperActivity which loads the current frame of the video (that you can extract with the VideoPreviewView) by inserting the CropImageView layout to get the cropping bounds. Once, you select an area in the UI, the rectangle values can be extracted and used to calculate the needed values for video cropping with this function:

val rect = cropFrame.cropRect
val w = abs(rect.left - rect.right)
val h = abs(rect.top - rect.bottom)
val x = rect.left
val y = rect.top

These values will then be used as input variables for the FFmpeg crop command.

To help you use all this information I created a project that uses all explained snippets to create a sample app that can crop and trim videos. The source code can be found on my personal GitHub.

Fixing error=13, Permission denied

Unfortunately, if you implement the described functionality and use the compiled FFmpeg library you will run into this problem if you compile your app for SDK 29 and onwards.

Permission denied error due to wrong target SDK The problem is caused because from Android Q onwards it is not allowed to execute binaries in your app’s private data directory. The error is described in the official issue tracker from google.

One solution will be that you compile your app for SDK 28 but this could lead to problems if you try to put your app into the Android Play Store.

Another solution will be that you compile your app for SDK 29 and stop putting binaries in any not supported directory. Unfortunately, the code that moves the binaries is within the 3rd party lib from WritingMinds and has to be removed by you (with a Pull Request) or by the maintainer of the library. A future-proof solution will be that you stop using external binaries and start compiling dependencies as an NDK project. This will be a lot of work but you can find help within a famous repository that compiles CPP to Java and also has FFmpeg

Add more commands

After you implemented crop and trim you maybe want to add more cool features to your Android app. To do this with FFmpeg and the previously described code you can do this by simply creating a new function within the VideoCommands() and translate the FFmpeg command into a VideoCommands() function.

The next sections will describe some fancy functions that would improve the usability of your app. Just add a new function with this command and fill the callbacks with a toast or anything else. Then call the function from anywhere within your app (because you do not need UI for this).

Removing audio from videos

You might run into a scenario where you’d only like to keep the visuals of a video and remove the audio track, for instance for voicing over certain footage or removing background noise:

val command = arrayOf("-i", input, "-an", output)

Compress a video

Make big videos smaller to save your valuable disk resources.

val command = arrayOf("-i", input, "-vf", "scale=$w:$h", "-c:v", "libx264", "-preset", "veryslow", "-crf", "24", output)

$w and $h are the new sizes (should be smaller). If one is -1 it will keep the ratio and is automatically adjusted in relation to the other one.

Change playback speed

Increase the playback speed to make long videos faster or decrease it for showing something really awesome very slowly.

 val command = arrayOf("-i", input, "-vf", "\"setpts=$scale*PTS\"", output)

$scale should be > 1 for slowing down a video or < 1 for fastening a video.

To find more cool commands search the official argument documentation and implement them by putting every argument into a value of the array function as described earlier.

For production-grade application you may be better served by using our VideoEditorSDK which in addition to professional trim and crop features allows your users to add filters, text and audio overlays. Recently, we released time-based sprites as well allowing users to time the appearance of text and stickers in videos.

Closing Notes

In this article, you learned that you can use FFmpeg to crop and trim videos on Android. Unfortunately, it does not work out of the box and you have to compile the library using complex techniques. Luckily, there exist some useful libraries that do the heavy lifting for you.

Using my app as a baseline you are able to crop and trim videos. Additionally, you can implement more commands easily if they do not need a GUI because creating a fancy-looking GUI was a complicated part of developing a cropping and trimming app.

Unfortunately, there are problems with the library that was used because Android Q introduces a security fix that forbids apps to execute binaries within their data folder. But, this is only a problem if you want to create a “Google Play Store” ready app because you can avoid this problem if compile your app for SDK 28 (and lower). Keep in mind that if you want to create a private app that you distribute on your website instead of using the Play Store you can do this. But, if you want to upload your app to the Google Play Store you have to search for another FFmpeg library within the documentation or build one from scratch so you can compile your app for the current needed SDK.

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

I hope you enjoyed reading this article and are able to create an Android app that can crop and trim videos. If you have any questions, need help, or want to give feedback @ us on Twitter. We are happy to help.

Build a Simple Real-Time Video Editor with Metal for iOS

Walter — Tue, 30 Aug 2022 20:06:49 GMT

In this tutorial you’ll see how to extract frames from live camera streams, regular movie files and streamed movie files and display them on a MetalKit View. Using Metal allows you to have a great deal of control over how the pixels are rendered and ensures that the GPU is used for rendering, which keeps from slowing down the CPU.

The tutorial code will always convert the video frames into CIImage objects before display. This is to help you adapt the code to your own applications. Applying filters, resizing and other effects are fast and easy when working with CIImage. Additionally, as long as you set your CIContext to the GPU, you can be sure that your code uses the GPU and the CPU efficiently.

When working with Video, Apple provides a number of frameworks that operate at different levels. For the higher level frameworks such as AVKit or CoreImage, the system determines when to use the GPU and when to use the CPU for rendering and computation. The higher-level frameworks also offer a tradeoff between ease of use and granularity of control. If you want to have direct access to tell the GPU how to render every pixel, then you will want to use Metal. Remember, though, that you are now responsible for keeping video and audio in sync, encoding and decoding data and determining a reasonable UI for your user. For many use cases, using a higher level framework is probably a better choice. Apple’s engineers have worked to ensure the graphics frameworks use the GPU and CPU in reasonable ways. However, when you need to use Metal, you you need it. So, let’s get started.

Setting Up a MTKView

An MTKView is a subclass of a UIView so it has a frame and bounds and other properties. Its drawing and rendering is backed directly by drawables and textures rendered on the GPU, so drawing to it can be quite fast. In addition to displaying video, you can render 3D model objects and graphics shaders, so for a graphics rich application it can be a powerful tool.

Before you can send data to the view it needs to be configured. MetalKit Views are still heavily influenced by UIKit, so if you are working in a SwiftUI project, you will need to wrap them in ViewRepresentable code.

//MetalKit Variables
@IBOutlet var displayView: MTKView!
var metalDevice : MTLDevice!
var metalCommandQueue : MTLCommandQueue!

//CoreImage Variables
var ciContext : CIContext!
var filteredImage: CIImage?
var cleanImage: CIImage?

//get a reference to the GPU
metalDevice = MTLCreateSystemDefaultDevice()

//link the GPU to our MTKView
displayView.device = metalDevice

//link our command queue variable to the GPU
metalCommandQueue = metalDevice.makeCommandQueue()

//associate our CIContext with the metal stack
ciContext = CIContext(mtlCommandQueue: metalCommandQueue)

You will need to get a reference to the GPU of the iOS device. At least for now, iOS devices only have one GPU. Then you will need to link the displayView and the metalCommandQueue to the metalDevice. Finally we will link the ciContext to the GPU so that all of our image manipulation code will run in the same place.

//tell our MTKView that we want to call .draw() to make updates
displayView.isPaused = true
displayView.enableSetNeedsDisplay = false
//let it's drawable texture be writen to dynamically
displayView.framebufferOnly = false

//set our code to be our MTKView's delegate
displayView.delegate = self

In the code above, we set the MTKView to be in a isPaused state and do not enable setNeedsDisplay. This will ensure that it will only redraw when we explicitly tell it to redraw using a .draw() method in our delegate. By setting framebufferOnly to false you are telling the view that you will be writing to it multiple times and may also read from it.

Drawing in an MTKView

Now the MTKView is configured. The next step is to update the delegate methods. The MTKViewDelegate has two methods. If your code needs to respond to the view dimensions changing (to support device rotation, or if you want the user to be able to resize the window) make your adjustments in func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize). For this example we are only interested in the delegate method: func draw(in view: MTKView). The code below draws a CIImage to the MTKView. When showing video, we can extract each frame of the video as a CIImage

//create a buffer to hold this round of draw commands
guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
  return
  }

//grab the filtered or a clean image to display
guard let ciImage = filteredImage ?? cleanImage else {
  return
  }

//get a drawable if the GPU is not busy
guard let currentDrawable = view.currentDrawable else {
  return
  }

//make sure frame is centered on screen
let heightOfImage = ciImage.extent.height
let heightOfDrawable = view.drawableSize.height
let yOffset = (heightOfDrawable - heightOfImage)/2

//render into the metal texture
self.ciContext.render(ciImage,
             to: currentDrawable.texture,
  commandBuffer: commandBuffer,
         bounds: CGRect(origin: CGPoint(x: 0, y: -yOffset),
                          size: view.drawableSize),
     colorSpace: CGColorSpaceCreateDeviceRGB())

//present the drawable and buffer
commandBuffer.present(currentDrawable)

//send the commands to the GPU
commandBuffer.commit()

For each pass of the draw method, we will create a new MTLCommandBuffer then we will get the MTKView’s .currentDrawable which contains a texture we can send pixel data to. The texture will have a size and a color space. Though we are using Metal to render images to the screen, Metal can be used to send any valid commands to the GPU for things like computations. Once the buffer has been filled with commands it can be assigned to the drawable and committed. When the buffer is committed, the GPU will execute all of the commands. If another .draw() gets called while the buffer is being executed, the GPU will not interrupt the current buffer in order to start the new one.

The bounds of the render allow you to move where the CIImage gets displayed in the MTKView and resize the CIImage within the view. This can be valuable when compositing multiple images or video streams onto the same MTKView. Remember that unlike a UIView the origin point is the bottom left of the texture and CIImage.

Since iOS 11 Apple provides a new lighter weight API for sending renders to the buffer. Use whichever form makes sense to you.

//render into the metal texture
let destination = CIRenderDestination(mtlTexture: currentDrawable.texture,
                                   commandBuffer: commandBuffer)
do {
 try self.ciContext.startTask(toRender: ciImage, to: destination)
} catch {
 print(error)
}

The CIRenderDestination has some other optional parameters such as height and width but will always display with an origin of 0,0. So, unlike the earlier call, if you need to rotate or change the dimensions of the image, you will need to do it to the CIImage earlier in the code. However, if your app always displays the video at full size in the MTKView this may be easier. Also the startTask call will return immediately, instead of waiting for other tasks on the CPU to complete.

It is important to remember that you can have any number of renders in the commandBuffer before the calls to .present and .commit. This means that the same MTKView can display video from multiple streams as well as static images or animations.

Working with Local Files

In order to extract individual frames from a local file, we can use an AVAssetReader to get pixel data to render in the MTKView

When using higher-level frameworks, a quick way to display a video file for playback is to load it into an AVPlayer. However, an alternative is to use an AVAssetReader to read the tracks and the individual frames from the video tracks.

let asset = AVAsset(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)
let reader = try! AVAssetReader(asset: asset)

guard let track = asset.tracks(withMediaType: .video).last else {
  return
  }

let outputSettings: [String: Any] = [
        kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32ARGB
    ]
let trackOutput = AVAssetReaderTrackOutput(track: track, outputSettings: outputSettings)
    reader.add(trackOutput)
    reader.startReading()

It is important that the outputSettings of the reader match the image format of the video track. If there is a mismatch the color may be off or the reader may be unable to extract a usable buffer. If your code is generating empty or black buffers, the outputSettings are the first place to troubleshoot. Once the reader starts reading then it can extract pixel buffers. You can use a while loop to get all of the buffers from the track and send them to the MTKView for display.

var sampleBuffer = trackOutput.copyNextSampleBuffer()
while sampleBuffer != nil {
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer!) else {
   return
  }
  print(track.preferredTransform)
  print(sampleBuffer?.outputPresentationTimeStamp)

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()

  sampleBuffer = trackOutput.copyNextSampleBuffer()
  }
}

In the code above, use CMSampleBufferGetImageBuffer to ensure that we have a valid image. Then assign the image to a CoreImage and execute the .draw method of the MTKView. Afterwards, get the next sample buffer. This will continue until the end of the track, when .copyNextSampleBuffer() will return nil.

In the code above, there are two print statements to note some valuable information that your app may want to store for later use. Remember, when working with buffers and the GPU directly, much of the higher-level metadata about the video is lost, so you’ll need to keep track of it in your code if you want to use it later. An image may be rotated because of the way it was originally generated in the video. The preferredTransform will provide data that your app can use to transform the image to the “proper” orientation before display. Additionally the outputPresentationTimeStamp tells when the image represented by the buffer appeared in the original video. This can be helpful when you are trying to sync individual frames back to audio tracks or if your app only wants to modify specific frames and then reinsert them into the original clip.

Working with Streams

In addition to local files, an app may want to use a video stream as the input. The process is largely the same, as there is a method to extract a CVPixelBuffer from an AVPlayer that is streaming. You can find the complete example code for pushing pixel buffers from a stream to an MTKView in this blog post about streaming. However, the important part of the code perhaps looks familiar by now:

let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) {
  if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
     let frameImage = CIImage(cvImageBuffer: buffer)
     self.currentFrame = frameImage //a CIImage var
     self.videoView.draw() //our MTKView
   }
}

The code above uses a CADisplayLink to query the stream on a regular basis for new video frames. It then extracts a buffer and sends it over the MTKView for display.

Working with the Cameras

Much like a video stream with an AVPlayerItemVideoOutput to output pixel buffer data, a standard object to use with the camera is an AVCaptureVideoDataOutput object. First, you need to set up a standard capture session for either the front or back camera. Then attach a video data output object to the session.

videoOutput = AVCaptureVideoDataOutput()
let videoQueue = DispatchQueue(label: "captureQueue", qos: .userInteractive)
videoOutput.setSampleBufferDelegate(self, queue: videoQueue)

if captureSession.canAddOutput(videoOutput) {
  captureSession.addOutput(videoOutput)
} else {
  fatalError("configuration failed")
}

Whenever the camera has collected enough data to generate a pixel buffer it will send that to its delegate. The delegate can then convert that data to a CIImage and send it to the MTKView to get rendered. The method in a AVCaptureVideoDataOutputSampleBufferDelegate would look like this.

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
  //get a CVImageBuffer from the camera
  guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    return
  }

  //get a CIImage out of the CVImageBuffer
  cleanImage = CIImage(cvImageBuffer: cvBuffer)
  displayView.draw()
}

Going Further

In this tutorial, you saw how to extract pixel buffers from the camera, local file and remote streams and convert them to CIImage Then you saw how to render a CIImage to fill all or part of an MTKView. If your application needs to resize or apply filters to the images, you can do that before calling the .draw method of the MTKView. If the only reason you are considering using MKTViews is because of a Metal or OpenGL kernel you want to use, consider our tutorial on how to wrap kernels into Core Image filters.

If your application needs to gather streams of video, filter and then render them to the screen with precision, then drawing to an MTKView may be sufficient. However, if you want to let your users dictate how and where to filter the streams, then you may want to consider an SDK like VideoEditor SDK or CreativeEditor SDK.

How to Stream Videos Using Javascript and HTML5

Mark — Mon, 29 Aug 2022 20:36:00 GMT

If you would tell someone from the early 2000s what you’re about to do, they would be amazed and would give everything to understand how you did it. Because back then HTML was much simpler and didn’t even support video playback. So to stream your video inside of your browser you were forced to use some third-party services like Flash or Silverlight. Now we have HTML5 and new versions of JavaScript, and by combining these two new technologies you’ll end up with results that rival native applications. Streaming video inside of your web application nowadays is a breeze.

The `<video>` tag

You can easily start streaming your own video by using only one tag in HTML5. I’m talking about the <video> tag. It embeds a fully-fledged media player in the document and similar to the img tag accepts a path to the media we want to play inside the src attribute as well as accepting parameters for width and height.

In addition, there are video-specific properties such as whether the video should loop or autoplay and whether to display the default browser controls.

To handle the case of browsers not supporting the video tag you can provide fallback content inside the <video></video> tags.

Combining with JavaScript

You can also combine this simple video streaming in HTML with JavaScript and make it more professional and be able to control your streaming manually. When you get an element from the DOM of a video tag, the object that you get exposes various methods and attributes which can be used for manipulating media content. Firstly, we need to access the DOM and declare an object which will be used for our manipulations. Now we can use myVideo to programmatically pause, change the playback rate and current time of the video.

const myVideo = document.getElementById('#video1');

//pause the video
myVideo.pause();

//Change the speed of playback
myVideo.playbackRate = 3.0;

//Change current time
myVideo.currentTime = 4;

However, most videos that we see on the web today such as YouTube, Twitch, or any other social media display much more complex behaviors than just changing playback speed or pausing the video. On YouTube, you can change the quality of your video, add subtitles, or even add features like autoplay if you have several other videos queued up and don’t want to switch it manually.

All those services actually still use a video tag. But instead of streaming a video file through the “src” attribute inside of your tag, they use powerful tools such as Media Source Extensions or other adaptive streaming API that will help stream your video more efficiently.

What are Media Source Extensions (MSE)

Media Source Extensions is a W3C specification that allows JavaScript to send streams to media codecs within Web browsers that support HTML5 video and audio that most browsers implement today. In other words, it allows JavaScript to generate streams and facilitate a variety of use cases like adaptive streaming and time-shifting live streams. It’s quite an advanced solution if you want to make your video playback more customizable and professional. MSE gives us finer grained control over how much and how often content is fetched, and some control over memory usage details, such as when buffers are evicted.

Wrap your resource via the Media Source Extensions API instead of dealing with the URL directly (We’ll talk more closely about it a little bit later).

This specification was designed with the following goals:

Let JavaScript create a media stream regardless of how the media is fetched.
Leverage the browser cache as much as possible.
Provide requirements for byte stream format specifications.
Minimize the need for media parsing in JavaScript.

As I mentioned, we are still using HTML5 video tags, and even more surprising is that we’ll still use the “src” attribute. Only this time, instead of adding a link to the video, add a link to the MediaSource object.

If you’re a little bit confused, then don’t worry, I’ll explain everything. To enable this type of use case, the W3C has defined a static method URL.createObjectURL. You can use this API to create a URL that directly points to a JavaScript object created on the client, rather than actually pointing to an online resource.

You can see how it works in this example:

const videoTag = document.getElementById('my-video');

// creating the MediaSource, just with the "new" keyword, and the URL for it
const myMediaSource = new MediaSource();
const url = URL.createObjectURL(myMediaSource);

// attaching the MediaSource to the video tag
videoTag.src = url;

Now that you have a MediaSource, what should you do? That’s not the only MSE specification. It also defines another concept, Source Buffers.

Source Buffers

The SourceBuffer interface represents a chunk of media to be passed into an HTMLMediaElement and played via a MediaSource object. To put the video directly into the MediaSource for playback, you’ll need Source Buffers. MediaSource contains one or more instances of it. Each is associated with a content type, such as Audio, Video, or both of them at the same time. Source buffers are all associated with a single MediaSource and are used in JavaScript to add video data directly to HTML5 video tags. Separating video and audio allows managing them separately on the server. Doing so has several advantages such as control over your traffic and increasing performance. This is how it works:

// -- Create a MediaSource and attach it to the video (We already learned about that) --

const videoTag = document.getElementById('my-video');
const myMediaSource = new MediaSource();
const url = URL.createObjectURL(myMediaSource);
videoTag.src = url;

// 1. add source buffers

const audioSourceBuffer = myMediaSource.addSourceBuffer(
  'audio/mp4; codecs="mp4a.40.2"'
);
const videoSourceBuffer = myMediaSource.addSourceBuffer(
  'video/mp4; codecs="avc1.64001e"'
);

// 2. download and add our audio/video to the SourceBuffers

// for the audio SourceBuffer
fetch('<http://server.com/audio.mp4>')
  .then(function (response) {
    // The data has to be a JavaScript ArrayBuffer
    return response.arrayBuffer();
  })
  .then(function (audioData) {
    audioSourceBuffer.appendBuffer(audioData);
  });

// the same for the video SourceBuffer
fetch('<http://server.com/video.mp4>')
  .then(function (response) {
    // The data has to be a JavaScript ArrayBuffer
    return response.arrayBuffer();
  })
  .then(function (videoData) {
    videoSourceBuffer.appendBuffer(videoData);
  });

Adaptive Streaming

Many video players have an “auto quality” feature that automatically selects quality based on network speed. If you have a slow internet connection then you’ll end up watching a video in the lowest resolution (probably 360p or even lower), on the other hand, if the connection is good then the video will be in the highest resolution (probably 1080p or even higher). It depends on how many options the specific video has. In addition, your hardware capability is also taken into account.

On the server side, we create different segments for every quality we want to support. For example, we would put the following files on our server:

./video/
  ├── ./360p/
  |     ├── video_0.mp4
  |     ├── video_1.mp4
  |     └── video_2.mp4
	├── ./720p/
  |     ├── video_0.mp4
  |     ├── video_1.mp4
  |     └── video_2.mp4
  └── ./1080p/
        ├── video_0.mp4
        ├── video_1.mp4
        └── video_2.mp4

The web player then automatically selects the correct video file to load when the network or CPU state changes. This is entirely done in JavaScript. For video segments, it could, for example, look as follows:


/**
 * We first derive the desired quality from the bandwidth
 * Then pass it to the pushVideoSegment()
 * which fetches each segment with the desired quality in turn and returns a promise
**/

function pushVideoSegment(nb, wantedQuality) {
 const url = "<http://my-server/video/>" +
 wantedQuality + "/segment" + nb + ".mp4");
return fetch(url)
 .then((response) => response.arrayBuffer());
 .then(function(arrayBuffer) {
 videoSourceBuffer.appendBuffer(arrayBuffer);
 });
}

/**
 * Translate an estimated bandwidth to the right audio
 * quality as defined on server-side.
 * @param {number} bandwidth
 * @returns {string}
 */
function fromBandwidthToQuality(bandwidth) {
 return bandwidth > 320e3 ? "360p" : "720p";
}

// first estimate the bandwidth. Most often, this is based on
// the time it took to download the last segments
const bandwidth = estimateBandwidth();

const quality = fromBandwidthToQuality(bandwidth);

pushVideoSegment(0, quality)
 .then(() => pushVideoSegment(1, quality))
 .then(() => pushVideoSegment(2, quality));

Conclusion

Congratulations, you’ve learned multiple ways how to stream a video file into your browser. We have explored several tools that are helping you not just change some video parameters by using vanilla JS and HTML, and also Media Source Extensions (MSE) that are helping all developers around the world to build professional web players which you’re using yourself, such as YouTube, Twitch, TikTok and many others.

If you want to give your users the ability to also edit videos within your mobile application check out our Video Editor SDK offering features such as trimming, transforms, video composition and more. It’s fully customizable to match the look and feel of your app and the most performant solution on the market.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

If you have questions, feedback or comments reach out to us on Twitter.

How To Apply Image Filters in WebGL

Antonello — Fri, 08 Apr 2022 13:06:16 GMT

In this article, you will learn how to apply custom filters to an image in WebGL. You can achieve this goal without using any external library. That is because the HTML5 canvas element natively supports WebGL and does not require the use of plug-ins.

As we have already covered in previous articles, <canvas> is a powerful tool that equips you with everything required to manipulate images. This is true regardless of the use of WebGL. So, you can filter images also without using WebGL.

Now, let’s see how to filter images in Vanilla JavaScript with WebGL in your browser. Follow this step-by-step tutorial and learn how to build the following demo:

Getting Started with WebGL

If you are not familiar with WebGL, you need to learn a few things before approaching it. To build an application in WebGL, start with the following three concepts.

1. Textures

Keep in mind that to draw an image in WebGL you have to use a texture. To use a texture, WebGL requires you to define the texture coordinates. These coordinates go from 0.0 to 1.0, regardless of the texture size.

2. Vertex shader

The vertex shader is a function you have to write in GLSL that is in charge of computing the vertex positions. Thanks to it, WebGL can rasterize the draw primitives, which include points, lines, and triangles. When rasterizing these primitives, WebGL calls another user-defined function called fragment shader. In other words, WebGL interpolates the values provided in the vertex shader function while it draws each pixel using the fragment shader function execution.

3. Fragment shader

The fragment shader is a function you have to write in GLSL whose goal is to generate a color for each pixel of the draw primitive currently being drawn. This function has little info per pixel, but you can provide it with everything required by using the varyings variables. These allow you to pass values from the vertex shader function to the fragment shader function.

Filtering Images with Kernels in WebGL with `<canvas>`

There are many ways to filter images, but the most common one involves the convolution operation. This is because when used on images, convolution applies a filter by taking the weighted sum of a square of pixels and assigning the resulting value to the current pixel. This logic is applied to every pixel the image consists of. Therefore, you can now imagine why convolution is one of the most relevant concepts when it comes to image processing.

The coefficients used to perform the weighted sum come from a matrix called kernel. The kernel represents the filter you want to apply through the convolution operation. So, by changing the kernel, the resulting image will change accordingly. Some kernels are more useful than others and can be used for blurring, sharpening, performing edge detection, and other operations. You can find a list of the most popular kernels on Wikipedia.

Now, let’s see how to implement kernel image filtering in WebGL.

Clone the GitHub repository that supports this article with the following command:

git clone https://github.com/Tonel/how-to-filter-an-image-in-webgl-imgly

Additionally, try the demo application by launching how-to-filter-an-image-in-webgl-imgly/index.html in your browser.

Otherwise, you can find the JavaScript function taking care of implementing the filter logic in WebGL below:

function filterImage(canvas, originalImage, kernel) {
  // assuming the kernel is a square matrix
  const kernelSize = Math.sqrt(kernel.length);
  const gl = canvas.getContext('webgl');

  // clearing the canvas
  gl.clearColor(1, 1, 1, 1);
  gl.clear(gl.COLOR_BUFFER_BIT);

  const vertexShaderSource = `
  attribute vec2 position;
  varying vec2 v_coordinate;
  void main() {
    gl_Position = vec4(position, 0, 1);
    v_coordinate = gl_Position.xy * 0.5 + 0.5;
  }
`;

  const fragmentShaderSource = `
  precision mediump float;
  // the varible defined in the vertex shader above
  varying vec2 v_coordinate;
  uniform vec2 imageSize;
  uniform sampler2D u_texture;
  
  void main() {
    vec2 position = vec2(v_coordinate.x, 1.0 - v_coordinate.y);
    vec2 onePixel = vec2(1, 1) / imageSize;
    vec4 color = vec4(0);
    mat3 kernel = mat3(
      ${kernel.join(',')}
    );
    // implementing the convolution operation
    for(int i = 0; i < ${kernelSize}; i++) {
      for(int j = 0; j < ${kernelSize}; j++) {
        // retrieving the sample position pixel
        vec2 samplePosition = position + vec2(i - 1 , j - 1) * onePixel;
        // retrieving the sample color
        vec4 sampleColor = texture2D(u_texture, samplePosition);
        sampleColor *= kernel\[i\][j];
        color += sampleColor;
      }
    }
    color.a = 1.0;
    gl_FragColor = color;
 }
`;

  const vertexShader = compileShader(gl, gl.VERTEX_SHADER, vertexShaderSource);
  const fragmentShader = compileShader(
    gl,
    gl.FRAGMENT_SHADER,
    fragmentShaderSource
  );

  // iniziailing the program
  const program = createProgram(gl, vertexShader, fragmentShader);

  const positionAttributeLocation = gl.getAttribLocation(program, 'position');

  const imageSizeLocation = gl.getUniformLocation(program, 'imageSize');

  // binding the position buffer to positionBuffer
  const positionBuffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, positionBuffer);

  // using the program defined above
  gl.useProgram(program);
  // enabling the texcoord attribute
  gl.enableVertexAttribArray(positionAttributeLocation);
  // setting up the size of the image
  gl.uniform2f(imageSizeLocation, canvas.width, canvas.height);
  // telling positionAttributeLocation how to retrieve data out of positionBuffer
  gl.vertexAttribPointer(positionAttributeLocation, 2, gl.FLOAT, false, 0, 0);
  // provide the texture coordinates
  gl.bufferData(
    gl.ARRAY_BUFFER,
    new Float32Array([-1, -1, -1, 1, 1, -1, 1, 1, 1, -1, -1, 1]),
    gl.STATIC_DRAW
  );

  // loading the original image as a texture
  const texture = gl.createTexture();
  texture.image = new Image();

  // setting the anonymous mode
  // Learn more about it here:
  // https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/crossOrigin
  texture.image.crossOrigin = '';
  texture.image.src = originalImage.src;
  texture.image.onload = function () {
    gl.bindTexture(gl.TEXTURE_2D, texture);
    // setting the parameters to be able to render any image,
    // regardless of its size
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
    // loading the original image as a texture
    gl.texImage2D(
      gl.TEXTURE_2D,
      0,
      gl.RGBA,
      gl.RGBA,
      gl.UNSIGNED_BYTE,
      texture.image
    );
    gl.drawArrays(gl.TRIANGLES, 0, 6);
  };
}

function compileShader(gl, type, shaderSource) {
  const shader = gl.createShader(type);
  gl.shaderSource(shader, shaderSource);
  gl.compileShader(shader);

  const outcome = gl.getShaderParameter(shader, gl.COMPILE_STATUS);

  if (outcome === false) {
    // logging the error message on failure
    console.error(gl.getShaderInfoLog(shader));
    gl.deleteShader(shader);
  }

  return shader;
}

function createProgram(gl, vertexShader, fragmentShader) {
  const program = gl.createProgram();
  gl.attachShader(program, vertexShader);
  gl.attachShader(program, fragmentShader);
  gl.linkProgram(program);

  const outcome = gl.getProgramParameter(program, gl.LINK_STATUS);

  if (outcome === false) {
    // logging the error message on failure
    console.error(gl.getProgramInfoLog(program));
    gl.deleteProgram(program);
  }

  return program;
}

The filterImage() function is where the magic happens. It takes the following three parameters:

canvas: an Element object representing an HTML canvas element
originalImage: an Element object representing an HTML img element
kernel: an array containing the values of the kernel to use in the convolution operation

The first part of the function takes care of extracting the WebGLRenderingContext object representing a three-dimensional rendering context from the canvas element. Then, the vertex shader and fragment shader functions are defined as strings written in GLSL. Next, they are compiled and finally used to create a WebGL program.

The last part of the function creates a texture starting from the original image passed as a parameter and passes it to the WebGLRenderingContext to produce the final result. This represents the filtered image and is finally displayed by the browser in the canvas element.

The results produced by the filterImage() function depend on the kernel chosen, as you can verify by playing with the live demo you can find at the beginning of the article.

Et voilà! You just learned how to filter images in WebGL!

Final Considerations

As shown above, you can filter images with WebGL in Vanilla JavaScript with a hundred lines. At the same time, this cannot be considered an easy task to achieve. The reason is that getting into WebGL takes time and effort. Plus, you have to learn how to use GLSL to write the vertex shader and the fragment shader functions. So, things can get more complicated than expected.

Plus, you learned how to perform non-complex filters, but several filtering techniques are complex and do not involve the convolution operation. That also means that implementing them can be challenging and result in inefficient algorithms.

To avoid a headache, you should consider a complete and all-in-one solution like PhotoEditor SDK. Commercial solutions make things easier and shield you from all difficulties, offering features that would be complex, time-consuming, and challenging to implement. That is particularly true when it comes to using WebGL, which is PhotoEditor SDK’s main renderer.

You will be able to harness WebGL’s power without writing a single line of GLSL or knowing about its existence. WebGL will be used behind the scene for you! Keep also in mind that you would not be alone, since developers at IMG.LY are happy to provide support.

Filtering Images With PhotoEditor SDK

Read the article from the official documentation to learn how to get started with PhotoEditor SDK in HTML and Vanilla JavaScript. In detail, the Filters feature gives you more than 60 high-quality filters to play with. Achieve the following result with a few clicks:

Check out this feature on the PhotoEditor SDK demo page.

Conclusion

In this article, we looked at how to filter an image in WebGL with the HTML canvas element. Implementing a feature allowing users to filter images through a kernel-based approach in WebGL involves only a dozen of lines of code. However, understanding how WebGL works and coding in GLSL cannot be considered easy tasks.

As a result, you might want to avoid dealing with WebGL entirely. In this case, consider a commercial and easy-to-adopt solution using WebGL behind the scene – such as PhotoEditor SDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.‌

How To Add Overlays to a Video in React Native

Antonello — Mon, 28 Mar 2022 16:23:32 GMT

In this article, you will learn to add a text and image overlay to a video with JavaScript in React Native. All you need is React Native Video, the correct styling rules, and just a few lines of code.

By following this guide, you will easily achieve a logo and text overlay:

First, let’s learn how to implement a React Native component that allows you to add an overlay image and overlay text to a video.

What is React Native Video?

Mobile apps come with several native interfaces and features. This is what makes React Native different from React, which is based on browser rendering. So, <div> becomes <View>, <p> becomes <Text>, and so on. In particular, React Native supports only a limited amount of components. You can find the entire list here.

To this day, React Native does not have a native <Video> component. That is why react-native-video was born - a widely used library without considering alternatives. This 500kB package equips you with everything you need to get started with videos in React Native.

Prerequisites

Here is the list of all the prerequisites for the demo application you are going to build:

You can add react-native-video to your project’s dependencies by launching the following npm command:

npm install react-native-video

Or if you are a yarn user:

yarn add react-native-video

Then, follow this guide to learn how to link react-native-video to your project. This step is not required for Android users with React Native 0.60 and above.

Adding Overlay Image and Text to a Video in React Native

You can clone the GitHub repository supporting this article and launch the following commands to try the demo application:

git clone https://github.com/Tonel/how-to-add-overlays-to-videos-in-react-native-imgly
cd how-to-add-overlays-to-videos-in-react-native-imgly
npm i
npm start

The project is a React Native Create App project, and you can learn more about how it works and how to run it here.

Otherwise, keep following this step-by-step tutorial and learn how to build a React Native component for adding overlays to a video.

1. Initializing a React Native Project

Create React Native App is the officially supported and easiest way to create an empty and working React Native application. Install it globally with the following command:

npm install -g create-react-native-app

Now, you can initialize a new project called react-native-video-overlays-demo with the following command:

create-react-native-app react-native-video-overlays-demo

The react-native-video-overlays-demo folder should now contain a demo project with the following file structure:

react-native-video-overlays-demo
├── .buckconfig
├── .gitattributes
├── .gitignore
├── App.js
├── app.json
├── babel.config.js
├── index.js
├── metro.config.js
├── package.json
├── package-lock.json
├── yarn.lock
├── android
│   └── ...
└── ios
    └── ...

Enter the react-native-video-overlays-demo folder:

cd react-native-video-overlays-demo

Then launch the development server with one of the following commands:

for launching an Expo app

npm start

for Android development on your phone or an emulator

npm run android

for iOS development on your phone or an emulator

npm run ios

for react-native-web development

npm run web

2. Defining a Component to Add Overlays on a Video

Let’s see how to build a VideoWithOverlays component that allows you to add overlays such as text and an image to a video. You can implement it as follows:

import React, { useEffect, useState } from 'react';
import { Image, StyleSheet, Text, View } from 'react-native';
import Video from 'react-native-video';

export default function VideoWithOverlays({
  videoComponentProps,
  text,
  imageSrc,
}) {
  const {
    video,
    overlayText,
    videoWithOverlays,
    overlayTextView,
    overlayImageView,
    overlayImage,
  } = styles;

  const [height, setHeight] = useState(0);
  const [overlayHeight, setOverlayHeight] = useState(0);

  useEffect(() => {
    // setting the overlay height to 10px starting
    // from the bottom of the video
    setOverlayHeight(height / 2 + height / 3 - 10);
  }, [height]);

  return (
    <View style={videoWithOverlays}>
      <Video
        {...videoComponentProps}
        style={video}
        onLayout={(event) => {
          const { height } = event.nativeEvent.layout;
          setHeight(height);
        }}
      />
      <View style={{ ...overlayTextView, top: overlayHeight }}>
        {text && <Text style={overlayText}>{text}</Text>}
      </View>
      <View style={{ ...overlayImageView, top: overlayHeight }}>
        {imageSrc && (
          <Image style={overlayImage} source={imageSrc} resizeMode="contain" />
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  overlayTextView: {
    position: 'relative',
    alignItems: 'flex-end',
    justifyContent: 'flex-end',
    right: 5,
  },
  overlayImageView: {
    position: 'relative',
    alignItems: 'flex-start',
    justifyContent: 'flex-end',
    left: 5,
  },
  overlayImage: {
    position: 'absolute',
    height: 40,
  },
  overlayText: {
    fontSize: 25,
    fontWeight: 'bold',
    color: 'white',
  },
  video: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
    marginTop: '50%',
  },
  videoWithOverlays: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
  },
});

The react-native-video Video component is used to play the video and its props are exposed thanks to the videoComponentProps prop. Then, after being placed into the layout, its height depending on the size of the device screen is stored in the height variable. This is used to calculate the height the overlays should be placed at in the useEffect hook. In detail, the overlays are placed at 10px from the bottom edge of the video.

The first internal View represents the optional text overlay. The overlay logic depends entirely on the custom overlayTextView and overlayText StyleSheet fields, representing the CSS classes responsible for displaying the text passed as a prop with text as an overlay. By default, the text overlay is placed in the bottom-right corner of the video.

The second internal View represents the optional image overlay. The overlay logic depends entirely on the custom overlayImageView and overlayImage StyleSheet fields, representing the CSS classes taking care of showing the image passed as a prop with imageSrc as an overlay. By default, the image overlay is placed in the bottom-left corner of the video. As you can see, both overlays are optional.

3. Putting It All Together

Let’s see the VideoWithOverlays component defined above in action. All you need to do is replace the App.js file with this:

import React from 'react';
import { StyleSheet, View } from 'react-native';
import VideoWithOverlays from './src/components/VideoWithOverlays/VideoWithOverlays';

export default function App() {
  // a static image stored into the assets folder
  const imgLy = require('./src/assets/images/IMG_LY.png');

  return (
    <View style={styles.container}>
      <VideoWithOverlays
        videoComponentProps={{
          // replace this free video source with your video
          source: {
            uri: 'https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4',
          },
        }}
        // overlay text
        text={'IMG.LY'}
        // overlay image
        imageSrc={imgLy}
      />
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#fff',
    alignItems: 'center',
    justifyContent: 'center',
  },
});

Here, <VideoWitOverlays> is initialized with a free video source retrieved from a public URL and two overlays. The text overlay states “IMG.LY” and it is placed in the bottom-right corner of the video. While the image overlay represents the IMG.LY logo and is placed in the bottom-left corner of the video. This is a screenshot of the VideoWitOverlays in action:

Final Considerations

Adding overlays elements to a video in React Native is not complex and takes just a few lines of code. On the other hand, the approach presented above is just a frontend trick and does not modify the video. You are just showing it behind text or an image, giving the illusion to the users that it is a single source of content.

If you wanted to let users modify their videos by adding overlays or other elements and then export them, this would involve complex logic and require an advanced UI. Building such a React Native component is a time-consuming task that represents a waste of energy. If that is your goal, you should take into consideration a complete and ready-to-use SDK solution, such as VideoEditorSDK. This would provide your users with the ability to add overlays and use many other cool features.

Add Overlays to a Video With React Native module for VideoEditor SDK

Visit this page from the official documentation to learn how to get started with VideoEditorSDK in React Native. Then, load your video and start editing it by adding overlays, text, and stickers. Follow the previous links to learn more about each feature.

Download our free mobile demo app from the App Store or Google Play, and try an extensive set of tools to edit videos.

Conclusion

In this tutorial, we learned how to define a React Native component that allows you to reproduce a video and add overlays. Specifically, we used react-native-video and defined everything required to place text or an image on your video. However, if you intend to export your edit, you will have to build a complex video editing application. In this scenario, you will want to consider a fully-featured, cutting-edge, and easy-to-adopt solution – such as VideoEditorSDK.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How To Resize and Compress an Image in JavaScript for Upload

Antonello — Fri, 04 Mar 2022 16:13:56 GMT

In this article, you will learn to compress an image in JavaScript and then upload it to Imgur. Similar to our previous guide on resizing images, and the one on drawing on images, you can accomplish everything with the HTML5 <canvas> element and we will involve any external libraries for this.

Smartphones cameras have become increasingly accurate and enhanced their photo quality for years. Consequently, their file size has grown as well. Since the speed of the average network has not improved at the same pace, it is essential to compress the images before uploading them.

Compressing is about downscaling an image. In other words, you want to reduce either its size or quality or both. By doing so, you can avoid uploading large images, saving the end-user time and money.

Compressing an Image With `<canvas>`

You can clone the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/how-to-compress-an-image-in-javascript-imgly

Then, you can try the demo application by opening the index.html file in your browser.

Otherwise, keep following this step-by-step tutorial and learn how to build the demo application.

1. Implementing the Compression Logic

You can compress an image by solely using the HTML <canvas> element. This is a powerful image manipulation tool that allows you to achieve many results, as we have already explored in our blog.
Now, let’s delve into how to compress an image with canvas:

function compressImage(imgToCompress, resizingFactor, quality) {
  // resizing the image
  const canvas = document.createElement('canvas');
  const context = canvas.getContext('2d');

  const originalWidth = imgToCompress.width;
  const originalHeight = imgToCompress.height;

  const canvasWidth = originalWidth * resizingFactor;
  const canvasHeight = originalHeight * resizingFactor;

  canvas.width = canvasWidth;
  canvas.height = canvasHeight;

  context.drawImage(
    imgToCompress,
    0,
    0,
    originalWidth * resizingFactor,
    originalHeight * resizingFactor
  );

  // reducing the quality of the image
  canvas.toBlob(
    (blob) => {
      if (blob) {
        // showing the compressed image
        resizedImage.src = URL.createObjectURL(resizedImageBlob);
      }
    },
    'image/jpeg',
    quality
  );
}

This function is an extension of the resizeImage() function defined in this article. So, follow the link to that tutorial to learn more about it.

What is new here are the last few lines, which take care of reducing the quality of the uploaded image based on the quality parameter. As you can see, the last part of the compressImage()is based on the toBlob() function. This transforms the image stored in the canvas into a Blob object and compresses it based on the last parameter passed to the function. This last parameter represents the quality of the target image file.

Just like resizingFactor, quality must contain a Number between 0 and 1.

Also, since the toBlob() function returns a Blob object, you can store it in a global variable. Then, you can use the blob object representing the compressed image file to upload it to your server. Let’s see how.

2. Uploading an Image to Imgur

First, you need an Imgur account. If you already have one, log in here. Otherwise, create a new account for free here.

Now, register an application here to have access to the Imgur API program. Fill out the form as follows:

Then, click on “Submit” and you should get access to this page.

Store your Client-ID in a safe place. You will need it later.
Now, you have everything required to start uploading images to your Imgur application.

Uploading an image to Imgur in JavaScript is easy and can be achieved with just a bunch of lines of code, as below:

// compressedImageBlob represents the compressed image Blob to upload
const formdata = new FormData();
formdata.append('image', compressedImageBlob);

fetch('https://api.imgur.com/3/image/', {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    Authorization: 'Client-ID YOUR_CLIENT_ID',
  },
  body: formdata,
}).then((response) => {
  if (response?.status === 403) {
    console.error('Unvalid Client-ID!');
  } else if (response?.status === 200) {
    // retrieving the URL of the image
    // just uploaded to Imgur
    response.json().then((jsonResponse) => {
      console.log(`URL: ${jsonResponse.data?.link}`);
    });
  } else {
    console.error(response);
  }
});

Replace YOUR_CLIENT_ID with the Client-ID retrieved before, and you should now be able to use this snippet to upload your images to Imgur.

3. Putting It All Together

Now it is time to see the compressImage() function in action through a simple example.

<!DOCTYPE html>
<html>
  <body>
    <h1>Compress and Resize an Image</h1>
    <p>Upload an image and compress it or use the following demo image</p>
    <input id="upload" type="file" accept="image/*" />
    <div>
      <h2>Original Image</h2>
      <img
        style="margin-top: 5px;"
        id="originalImage"
        src="demo.jpg"
        crossorigin="anonymous"
      />
    </div>
    <div style="margin-top: 5px;">
      <span>Resizing: </span>
      <input type="range" min="1" max="100" value="80" id="resizingRange" />
    </div>
    <div style="margin-top: 5px; margin-left: 8px;">
      <span>Quality: </span>
      <input type="range" min="1" max="100" value="80" id="qualityRange" />
    </div>
    <h2>Compressed Image</h2>
    <div><b>Size:</b> <span id="size"></span></div>
    <img id="compressedImage" />
    <div>
      <button id="uploadButton">Upload to Imgur</button>
    </div>
    <script src="src/index.js"></script>
  </body>
</html>

const fileInput = document.querySelector('#upload');
const originalImage = document.querySelector('#originalImage');

const compressedImage = document.querySelector('#compressedImage');
const resizingElement = document.querySelector('#resizingRange');

const qualityElement = document.querySelector('#qualityRange');
const uploadButton = document.querySelector('#uploadButton');

let compressedImageBlob;

let resizingFactor = 0.8;
let quality = 0.8;

// initializing the compressed image
compressImage(originalImage, resizingFactor, quality);

fileInput.addEventListener('change', async (e) => {
  const [file] = fileInput.files;

  // storing the original image
  originalImage.src = await fileToDataUri(file);

  // compressing the uplodaded image
  originalImage.addEventListener('load', () => {
    compressImage(originalImage, resizingFactor, quality);
  });

  return false;
});

resizingElement.oninput = (e) => {
  resizingFactor = parseInt(e.target.value) / 100;
  compressImage(originalImage, resizingFactor, quality);
};

qualityElement.oninput = (e) => {
  quality = parseInt(e.target.value) / 100;
  compressImage(originalImage, resizingFactor, quality);
};

uploadButton.onclick = () => {
  // uploading the compressed image to
  // Imgur (if present)
  if (compressedImageBlob) {
    const formdata = new FormData();
    formdata.append('image', compressedImageBlob);

    fetch('https://api.imgur.com/3/image/', {
      method: 'POST',
      headers: {
        Accept: 'application/json',
        Authorization: 'Client-ID YOUR_CLIENT_ID',
      },
      body: formdata,
    }).then((response) => {
      if (response?.status === 403) {
        alert('Unvalid Client-ID!');
      } else if (response?.status === 200) {
        // retrieving the URL of the image
        // just uploaded to Imgur
        response.json().then((jsonResponse) => {
          alert(`URL: ${jsonResponse.data?.link}`);
        });
        alert('Upload completed succesfully!');
      } else {
        console.error(response);
      }
    });
  } else {
    alert('Rezind and compressed image missing!');
  }
};

function compressImage(imgToCompress, resizingFactor, quality) {
  // showing the compressed image
  const canvas = document.createElement('canvas');
  const context = canvas.getContext('2d');

  const originalWidth = imgToCompress.width;
  const originalHeight = imgToCompress.height;

  const canvasWidth = originalWidth * resizingFactor;
  const canvasHeight = originalHeight * resizingFactor;

  canvas.width = canvasWidth;
  canvas.height = canvasHeight;

  context.drawImage(
    imgToCompress,
    0,
    0,
    originalWidth * resizingFactor,
    originalHeight * resizingFactor
  );

  // reducing the quality of the image
  canvas.toBlob(
    (blob) => {
      if (blob) {
        compressedImageBlob = blob;
        compressedImage.src = URL.createObjectURL(compressedImageBlob);
        document.querySelector('#size').innerHTML = bytesToSize(blob.size);
      }
    },
    'image/jpeg',
    quality
  );
}

function fileToDataUri(field) {
  return new Promise((resolve) => {
    const reader = new FileReader();
    reader.addEventListener('load', () => {
      resolve(reader.result);
    });
    reader.readAsDataURL(field);
  });
}

// source: https://stackoverflow.com/a/18650828
function bytesToSize(bytes) {
  var sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB'];

  if (bytes === 0) {
    return '0 Byte';
  }

  const i = parseInt(Math.floor(Math.log(bytes) / Math.log(1024)));

  return Math.round(bytes / Math.pow(1024, i), 2) + ' ' + sizes[i];
}

The input element allows users to upload an image. This is then passed to the compressImage() function along with the resizingFactor and quality values retrieved from the respective range input HTML elements. This function takes care of compressing the image, displaying it, and storing its Blob representation to the global compressedImageBlob variable. Finally, compressedImageBlob is uploaded to Imgur when the “Upload to Imgur” button is clicked.
Notice that these two snippets are what allow you to implement the live example you can find at the beginning of the article.

Final Considerations

Compressing an image in Vanilla JavaScript is easy. You can achieve this with no extra libraries and in a dozen of lines of code. At the same time, the resizing part of the process relies on an image interpolation algorithm that changes according to the browser in use. This can lead to different results based on the end-user’s browser. Also, compressing while preserving quality is always tricky and can easily become a grueling goal to achieve.
If you want to avoid this stress, consider adopting a commercial and browser-consistent solution like PhotoEditorSDK. This library provides access to a wide set of tools that allow you to manipulate your images in many ways and with an advanced and easy-to-use UI.

Resizing an Image with PhotoEditor SDK

First, read this article from the official documentation to get started with PhotoEditorSDK in HTML and JavaScript. Then, you can use the transform tool to resize your image, as shown below:

Check out this feature on the PhotoEditorSDK demo page.

Conclusion

In this article, we learned how to downscale an image in JavaScript before uploading it to your server. In detail, we resized and reduced the quality of an uploaded image before uploading it to Imgur. Everything was achieved by using only the HTML5 <canvas> element. This is a powerful tool supported by most browsers that allows you to resize and change the quality of an image with just a few lines of code. On the other hand, this process may not be browser-consistent. That is why we also introduced a commercial and more reliable solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions. To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How To Add a Sticker to a Texture With Three.js

Antonello — Fri, 25 Feb 2022 15:10:58 GMT

Disclaimer: This article has been updated in June 2023 to reflect the changes introduced by Three.js r150

In this article, you will learn to add a sticker to a texture with WebGL. In detail, you will see how to use the Three.js JavaScript library to add an overlay image to a texture. Check out our guide to explore other image editing operations, such as resizing an image in React.

Dealing with 3D graphics used to be a challenging, complex, and time-consuming task. Thanks to libraries like three.js, it has become more accessible than ever. Follow this easy step-by-step how-to and achieve the following result:

Prerequisites

Here are all the prerequisites you need to meet to follow this tutorial and build the demo application shown above:

Explore the official documentation page to learn about how to install three.js.

What is Three.js?

As stated in the GitHub page of the project, the goal of the Three.js project is to create a lightweight 3D JavaScript library with a very low level of complexity. In other words, Three.js aims to make 3D graphics more accessible, especially considering that Three.js only requires a browser to run.

To be more specific, it allows you to create a 3D graphics project where you can modify elements in your browser and see visual feedback. The library is currently built on top of a WebGL renderer, but it also supports WebGPU, SVG, and CSS3D renderers.

If you are not familiar with these concepts, WebGL stands for “Web Graphics Library” and is a JavaScript API for rendering high-performance interactive 3D and 2D graphics. You can use it in many web browsers without external plugins or libraries.

As you can verify from the MDN compatibility page, browsers currently support the most popular WebGL features. However, keep in mind that the user’s device must meet hardware requirements for some features to work.

WebGL is undoubtedly a powerful tool but requires a lot of skill and involves a lot of boilerplate code. Here is where Three.js comes into play, making 3D development much easier.

Now, let’s see it in action!

Adding an Overlay Image to a Texture in WebGL

Clone the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/how-to-add-a-sticker-with-threejs-imgly

Additionally, try the demo application by launchingthe command below:

npm install
npm run start

Then, visit http://localhost:5000 in the browser.

Now that you know what the demo app looks like, keep following this tutorial and learn how to build it.

Specifically, this is the entire JavaScript code required to achieve the goal:

import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

// set up the scene, camera, and renderer
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(
  40,
  window.innerWidth / window.innerHeight,
  1,
  1000
);
camera.position.set(150, 100, 150);
new OrbitControls(camera, document.body);

// initialize the WebGL renderer
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// add light to the scene to make the texture visible
const light = new THREE.AmbientLight(0xffffff);
scene.add(light);

// add the axes helper indicator to the scene
scene.add(new THREE.AxesHelper(20));

// load the base texture
const texture = new THREE.TextureLoader().load(
  'https://threejs.org/examples/textures/hardwood2_diffuse.jpg'
);

// create a new buffer geometry and place it
// to the horizontal plane
const textureGeometry = new THREE.PlaneGeometry(50, 50);
textureGeometry.rotateX(-90 * THREE.MathUtils.DEG2RAD);
// translate the plane to the center of the scene
textureGeometry.translate(25, 0, 25);

// create a MeshBasicMaterial with the base texture
const textureMaterial = new THREE.MeshBasicMaterial({ map: texture });

// create a mesh using the geometry and material
// and add it to the scene
const textureMesh = new THREE.Mesh(textureGeometry, textureMaterial);
scene.add(textureMesh);

// repeat the same logic to place the sticker image
// over the texture
const sticker = new THREE.TextureLoader().load(
  'https://i.imgur.com/IYh17Rv.png'
);

const stickerGeometry = new THREE.PlaneGeometry(20, 20);
stickerGeometry.rotateX(-90 * THREE.MathUtils.DEG2RAD);
stickerGeometry.translate(25, 0, 25);
const stickerMaterial = new THREE.MeshBasicMaterial({
  map: sticker,
  transparent: true,
});
// create a mesh using the image geometry and material
const imageMesh = new THREE.Mesh(stickerGeometry, stickerMaterial);
scene.add(imageMesh);

// adjust the position and rotation of the sticker mesh
// by setting the y-coordinate to 1
// in order for the sticker to rest on the texture
imageMesh.position.set(0, 1, 0);

// render the scene
function launchThreeJs() {
  requestAnimationFrame(launchThreeJs);
  renderer.render(scene, camera);
}

launchThreeJs();

First, the code imports the necessary components from the Three.js library. Next, it initialize a Scene and set up a PerspectiveCamera with OrbitsControl, allowing the user to move around. Then, it creates a WebGL renderer and sets its size to match the window’s dimensions. The scene is lit by an AmbientLight instance, which illuminates all objects in the scene equally.

The snippet loads a base texture using TextureLoader from the given URL, creates a PlaneGeometry, defines a mesh with some geometries, and then add the resulting element to the scene. This logic will take care of rendering the texture image in the scene.

To add the sticker to the texture, the same procude gets repeated. This time, Three.js is instructed to load a sticker image and position it over the original texture. Note that the position of the sticker mesh is adjusted by setting its y-coordinate to 1 to ensure the new image rests on top of the old one.

Et voilà! You just learned how to add a sticker to a WebGL texture with three.js!

Adding a Sticker With `PHOTOEDITORSDK`

PhotoEditorSDK uses WebGL as the main renderer. You can effortlessly add a sticker to your textures without writing any line of code. WebGL will be used behind the scene. Consequently, you do not need to know WebGL or code in GLSL to harness its power.

Adding a sticker to an image only takes a few seconds here. Just jump to this article from the official documentation and learn how to get started with PhotoEditorSDK in Vanilla JavaScript. Then, upload your texture image and a sticker with the Sticker feature. As shown below, you can position a sticker on top of the original image with a point-and-click UI:

Check out this feature on the PhotoEditorSDK demo page.

Conclusion

Today we learned how to add an overlay image to a WebGL texture in JavaScript with three.js. This tool makes it easier to deal with WebGL and 3D graphics. In particular, by using one of the many plugins supporting the Three.js project, we were able to add a sticker to a texture with just a bunch of lines of code.

3D graphics remains a complex topic, especially when it comes to developing in the OpenGL Shading Language. Three.js tries to hide this encumbrance as much as possible, but sometimes you still require it. If you want to avoid using WebGL directly to achieve your goal, consider an all-in-one and easy-to-use solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Make Videos from Still Images with AVFoundation and Swift

Walter — Thu, 10 Feb 2022 15:35:45 GMT

Whether making a slide show or breaking up video tracks with title scenes, inserting still images into a video file can be tricky. This tutorial will show you how to convert still images into standard Quicktime video files so you can work with them further in your favorite video editor. This tutorial was created and tested with Xcode 13 and Swift 5. This tutorial will discuss two strategies:

convert each image to a separate movie and stitch them together later (this is best suited for times you want to interleave the images and other video files using some other editing workflow)
create a “blank” video and use the applyingCIFiltersWithHandler variation to overlay the images (this is best suited for times you want to make a quick slide show with no further editing)

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with this code (and some extra goodies) can be found on GitHub.

The Problem with an Image

The basic models that are used in AVFoundation for making video files are AVAsset and AVAssetTrack. You can compose multiple tracks and assets together to make a single video file. All tracks must have some media data and a duration. Static image files don’t have a duration. This is the primary problem you have to overcome when you want to add static images to video.
Fortunately, you can use many of the same AVFoundation tools that are used for video capture and frame manipulation. However instead of using a CADisplayLink and AVPlayerItemVideoOutput or an AVCaptureDevice to provide frame buffers to save, you will create the frame buffers manually. Then you can use AVAssetWriter to convert the frame buffers into a video file.

Some items to consider before you actually generate the video file are:

What framerate to use? Because the image doesn’t have any motion, you can use a very low frame rate to create smaller file sizes. However, this might cause issues if you are interleaving it with regular video, which usually has a frame rate of 30 or 60 fps (or higher, for slow motion video)
What dimensions will the final video have? It is easy to resize an image and add padding using CoreImage or some image editing software before it becomes a video. As the image becomes a video and moves through different steps in an AVFoundation pipeline, the different classes in AVFoundation will resize or crop the images in different ways when the input dimensions and output dimension don’t match.

Writing Buffers using `AVAssetWriter`

The AVAssetWriter exists to encode media to a file on disk. Though this tutorial will use a single video input AVAssetWriter supports multiple inputs of different kinds (audio, video, metadata). It is quite similar to AVAssetExportSession. The difference is that AVAssetWriter uses multiple inputs and each input is a single track where the AVAssetExportSession takes a single AVAsset as an input. The result of both is a single file.

As indicated above, our basic strategy will be:

Create a pixel buffer that is the correct color space and size to hold the image
Render the image into the pixel buffer
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop, and each time through the loop, append the pixel buffer
Clean up

Create the Pixel Buffer

The code below creates a CIImage from an image file. Then it creates a pixel buffer the same size as the image and uses a standard color space. Finally, a CIContext renders the image into the pixel buffer.

//create a CIImage
guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
}
//create a variable to hold the pixelBuffer
var pixelBuffer: CVPixelBuffer?
//set some standard attributes
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
     kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
//create the width and height of the buffer to match the image
let width:Int = Int(staticImage.extent.size.width)
let height:Int = Int(staticImage.extent.size.height)
//create a buffer (notice it uses an in/out parameter for the pixelBuffer variable)
CVPixelBufferCreate(kCFAllocatorDefault,
                    width,
                    height,
                    kCVPixelFormatType_32BGRA,
                    attrs,
                    &pixelBuffer)
//create a CIContext
let context = CIContext()
//use the context to render the image into the pixelBuffer
context.render(staticImage, to: pixelBuffer!)

Though we generally think of CIImage as an image, Apple is clear that it is not an image by itself. CIImage needs a context for rendering. This is where a lot of the power of CoreImage and filters and GPU rendering come from, the fact that CIImage is just the instructions for creating an image. Note: You can also use CGImage and the CoreGraphics framework to create pixel buffers. I find it easier to use the CoreImage framework.

Configure AVAssetWriter

Now with a buffer created, you can configure the AVAssetWriter. It will take several parameters as a dictionary to determine the dimensions and format of the outputs. You can either set them individually or else Apple provides presets.
One of the most important settings is the output dimension. If the output dimension matches the original image, the video file will show no distortion or letterboxing. However, if the output is smaller, it will compress the image until it fits. If the output is larger, the image will expand until its width or height matches the output size and then will letterbox. Note: if an image expands too much, it will appear grainy. The image below shows different output sizes for a 640 x 480 input image.

To create your settings for the AVAssetWriter first create a dictionary containing the different values. The example here sets an output dimension of 400 x 400 and provides for .h264 encoding.

let assetWriterSettings = [AVVideoCodecKey: AVVideoCodecType.h264, AVVideoWidthKey : 400, AVVideoHeightKey: 400] as [String : Any]

To use one of the presets, use the AVOutputSettingsAssistant. You can read about the different settings in the Apple documentation. An example to create 1080p video output would look like this:

let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings

Write the pixel buffer to the video file

With the settings configured, the asset writer can loop through and append the contents of the pixel buffer to create each frame of the video. Something important to notice in the code below is that AVFoundation has a philosophy to preserve data. So, it will make copies, and it will generally not overwrite files. That is why you have to delete any old files before you can create the new AVAssetWriter.

//generate a file url to store the video. some_image.jpg becomes some_image.mov
guard let imageNameRoot = imageName.split(separator: ".").first, let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("\(imageNameRoot).mov") else {
  throw ConstructionError.invalidURL //an error i made up
}
//delete any old file
do {
  try FileManager.default.removeItem(at: outputMovieURL)
} catch {
  print("Could not remove file \(error.localizedDescription)")
}
//create an assetwriter instance
guard let assetwriter = try? AVAssetWriter(outputURL: outputMovieURL, fileType: .mov) else {
  abort()
}
//generate 1080p settings
let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings
//create a single video input
let assetWriterInput = AVAssetWriterInput(mediaType: .video, outputSettings: settingsAssistant)
//create an adaptor for the pixel buffer
let assetWriterAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: assetWriterInput, sourcePixelBufferAttributes: nil)
//add the input to the asset writer
assetwriter.add(assetWriterInput)
//begin the session
assetwriter.startWriting()
assetwriter.startSession(atSourceTime: CMTime.zero)
//determine how many frames we need to generate
let framesPerSecond = 30
//duration is the number of seconds for the final video
let totalFrames = duration * framesPerSecond
var frameCount = 0
while frameCount < totalFrames {
  if assetWriterInput.isReadyForMoreMediaData {
    let frameTime = CMTimeMake(value: Int64(frameCount), timescale: Int32(framesPerSecond))
    //append the contents of the pixelBuffer at the correct time
    assetWriterAdaptor.append(pixelBuffer!, withPresentationTime: frameTime)
    frameCount+=1
  }
}
//close everything
assetWriterInput.markAsFinished()
assetwriter.finishWriting {
  pixelBuffer = nil
  //outputMovieURL now has the video
  Logger().info("Finished video location: \(outputMovieURL)")
}
}

Now, your image has become a Quicktime video of whatever duration you specified and is ready for use as any other video file! You can now continue with the like of VideoEditor SDK to trim videos, and add filters or overlays. You might also want to combine the video with other video files to make a new creation.

Say hi to my dog! A plain video from still images

Creating a Blank Video

The previous example created a single .mov file for each of your images. The most robust way to get these images into a video with sound, transitions, etc., is to use AVMutableComposition with multiple tracks and layer instructions. However, for a quick slideshow with just a few elementary transitions, a strategy is to create a blank movie and then use AVVideoComposition with (asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void). That will let you use CIImage and CIFilter to paint each frame of the blank movie with images.

Our basic strategy for this method will be:

Create a pixel buffer that is the right color space and size and fill it with a solid color
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop and each time through the loop append the pixel buffer
Create an AVVideoComposition to add the images to the individual frames of the video
Let AVPlayer combine the video and the composition

In the first strategy, you started by making a pixel buffer and using an AVAssetWriter to create the video file using this code:

guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
    }

However, this time, instead of using a source image, you create a CIImage that is a single color.

let staticImage = CIImage(color: bgColor).cropped(to: CGRect(x: 0, y: 0, width: 960, height: 540))

The CIImage(color:) initializer creates an image that is of infinite size and is just a single color. Using the .cropped(to:) modifier lets you make it the same size as the AVAssetWriter output so that there won’t be any letterboxing. The rest of the code is the same as before and the end result will be a video file that is just the single color.

Adding a Composition

Once the video file has been created and saved to disk, load it as an asset, then create an AVVideoComposition. This allows you to generate individual frames. The function below uses the request property of the composition. This has a CIImage representation of the current frame as well as the timestamp of when this frame will appear. The .fetchSlide(forTime:) is a helper function in the demo app that returns the appropriate CIImage for the slide to display. Then the .sourceOverCompositing filter combines the original frame image with the slide. Using a CGAffineTransform moves the slide across the screen as the video plays.

func createComposition(_ asset: AVAsset) {
  let slideshowComposition = AVVideoComposition(asset: asset) {[weak self] request in
    guard let self = self else { return }
    let slide = self.fetchSlide(forTime: request.compositionTime)
    let compose = CIFilter.sourceOverCompositing() //filter to join two images
    compose.backgroundImage = request.sourceImage
    compose.inputImage = slide?.transformed(by: CGAffineTransform(translationX: request.compositionTime.seconds * 50, y: 0))
    //always finish with the last output of the pipeline
    request.finish(with: compose.outputImage!, context: nil)
  }
  self.outputSlideshow = slideshowComposition
}

Because the creation of the video is done asynchronously, you can composite images over the original frame and add filters without the danger of impacting the final frame rate. Recall that CIFilter works as a pipeline, each CIFilter has an .outputImage and most have .inputImage and some configuration settings. Feeding the output from one filter to the input of the next will let you build complex compositions. Once a composition has been created, AVPlayer can join the original video with the composition for playback or export.

let item = AVPlayerItem(asset: self.outputMovie!)
item.videoComposition = outputSlideshow
self.player = AVPlayer(playerItem: item)

The code above might generate a video like this one.

As with the first video, the final video is a plain .mov file.

Going Further

In this tutorial, you saw two ways to convert static images into .mov files that you can then edit with any video editor. The sample project also has some code for exporting your creation as a .mov file and code for adding sound to the creation.

These are not the only ways to go about solving this problem. For example, instead of creating a blank video and overlaying the images, using a stock video of waves at the beach and overlaying the images might work better for your project. Conversely, because the initial pixel buffer is created from a CIImage, there is no reason that the CIImage cannot be the end of a long pipeline of CIFilters. This way the composition is done at the beginning, and the AVAssetWriter is the only tool needed. The best strategy is the one that makes sense to you and that works with the kinds of media you have.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How To Scale, Crop, Flip, and Filter an Image in CSS

Antonello — Wed, 09 Feb 2022 19:41:38 GMT

In this article, you will learn how to perform popular image processing operations in CSS. That is possible thanks to the several features introduced by CSS3, which allow you to change how the images look to end-users. To learn instead about image manipulation operations for other frameworks, you can head on here: React or vanilla JS.

First, let’s delve into how image manipulation works in CSS. Then, let’s finally see the most common CSS image processing operations in action.

How CSS Image Manipulation Work

CSS does not allow you to deal with files, data formats, or data transformation. In other words, CSS cannot change an image on a deep level, yet with CSS, you can determine how a browser displays elements.

As a result, as opposed to what would happen when using an HTML canvas element, no pixel manipulation will be executed behind-the-scene. Therefore, your CSS rules won’t affect the file that represents your image. What changes is how your image is displayed frontend-side.

You can flip, blur, and crop an image in CSS, but when downloading the image with the right-click option, the image saved will always be a copy of the original image file. Read this page from MDN Web Docs, if you want to learn more about this phenomenon.

Prerequisites

The CSS properties you will see are widely based on CSS3 and are not supported by Internet Explorer. To verify if your target browsers support a CSS property, visit MDN Web Docs and explore the Browser compatibility section.

Image Processing in CSS

Let’s look at how to perform scaling, cropping, flipping, and filtering an image in CSS.

Scaling an image with `object-fit: scale-down`

The object-fit CSS property allows you to define how the content of an HTML <img> element should be resized to fit its container. In particular, scale-down acts as if none or contain were set, depending on which would result in a smaller image.

When it acts like none, the image will keep its original intrinsic size and not be scaled down. On the contrary, when it acts like contain the image will be scaled down to fit the container while maintaining its aspect ratio.

Test this CSS rule in the live demo below:

In particular, this is the CSS rule where the magic happens:

.manipulated-image {
  object-fit: scale-down;
  height: 100px;
}

Cropping an image with `object-fit` and `object-position`

By employing object-fit in conjunction with object-position, you can show a part of your image, just if it has been cropped. In detail, the object-position CSS property allows you to specify the positioning of your <img> element within its container box. This happens by passing this CSS property a position parameter containing from one to four values. This parameter is what defines the position of the <img> element.

By using object-fit to set how your <img> element should fit its container and by positioning it with object-position, you can show only a part of your image as shown in the live example below:

This is the CSS rule to look attention to:

.manipulated-image {
  object-fit: cover;
  object-position: -150px -25px;
}

Flipping an image with `transform`

The transform CSS property is a powerful tool that gives you the ability to rotate, scale, skew, or translate an<img> element. Specifically, you can flip an image by using the scaleX() and scaleY() transformation functions, or by adopting the rotateX() or rotateY() functions.

scaleX() and scaleY() define a transformation that resizes an HTML element along the x-axis and y-axis, respectively. Both accept a number representing the scaling factor as a parameter. Learn how to flip an image vertically with scaleX() in the example below:

In detail, this is the CSS rule to flip an image vertically:

.manipulated-image {
  transform: scaleX(-1);
}

Similarly, rotateX() and rotateY() define a transformation that rotates an HTML element without deforming it around the horizontal axis and vertical axis, respectively. Both require an angle representing the angle of rotation as a parameter. Now, let’s see how to flip an image vertically with rotateY() in the example below:

Specifically, this is what the CSS rule to flip an image vertically looks like:

.manipulated-image {
  transform: rotateY(180deg);
}

Filtering an image with `filter`

The filter CSS property allows you to apply visual effects on an <img> element. This property provides access to several built-in effects, such as blur(), grayscale(), contrast(), and many others. If required, you can also define your custom image filtering functions.

CSS filters are a powerful tool that is typically used to adjust how an image is displayed by the browser and shown to the end-user. Using them is really easy, as shown in the live demo below:

These are the CSS rules where the filtering operations are defined:

.blurred-image {
  filter: blur(2px);
}
.grayscale-image {
  filter: grayscale(80%);
}
.high-contrast-image {
  filter: contrast(150%);
}

CSS-Only Image Editor vs PhotoEditor SDK

Image manipulation in CSS is an easy and straightforward way to manipulate how an image is displayed with no effort. On the other hand, you cannot save or export the manipulated images with CSS. This means that it is not possible to implement a file image editor based solely on CSS.

Therefore, you can rely on CSS only if you are interested in changing how your images are rendered by the browser. Conversely, if you are looking for a way to manipulate your images and then export them as files, then you need a much more advanced tool. In this case, a commercial solution like IMG.LY’s Photo Editor SDK will provide you with a complete, fast, and easy-to-use image editor.

Conclusion

Today we learned how to use CSS to control how browsers display images. In detail, we delved into how to scale, crop, flip, and filter an image in CSS. These represent the most common image manipulation operations and allow you to transform your image CSS-side. As we have learned, these operations only determine how your browser renders an image. They do not change an image as a data structure.
This means that it is not possible to create an image editor that allows users to export manipulated images while using only CSS. So, if this is what you were looking for, you should consider adopting a fast, advanced, and ready-to-use solution – such as PhotoEditorSDK. And you can get right into the PhotoEditorSDK or VideoEditorSDK and its myriad integrations by starting with a free trial over here.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How To Manipulate an Image With Jimp in React

Antonello — Wed, 26 Jan 2022 13:36:27 GMT

In this article, you will learn how to use jimp in React to perform the most common image processing operations. As you are about to learn, this popular JavaScript library allows you to effortlessly manipulate an image with a few lines of code.

What is Jimp?

Jimp stands for JavaScript Image Manipulation Program, and it is one of the most popular open-source image processing libraries, with more than one million weekly downloads. As stated on the official GitHub page, Jimp is a Promise-based library that relies entirely on Vanilla JavaScript. This means that it has no native or external dependencies, as you can verify here. This makes it more reliable and lightweight.

Jimp comes with several functions that allows you to transform your images in endless ways. Particularly, it includes the resize(), blur(), crop(), rotate() functions and many others. At the time of writing, these are image formats supported by Jimp:

Let’s now see how to use the Jimp transformation functions.

Prerequisites

This is the list of prerequisites you need to make the React applications using Jimp we are about to build works:

You can add jimp to your project’s dependencies by launching the following command:

npm install jimp

Now, you have everything required to start getting your hands dirty with Jimp in React.

Basic Image Processing in Jimp

All the Jimp usage examples presented below share the same logic. So, let’s address it immediately.

Specifically, this is what a basic JimpDemo component looks like:

import Jimp from 'jimp';
import { useEffect, useState } from 'react';

export function JimpDemo({ imageUrl }) {
  const [jimpImage, setJimpImage] = useState(undefined);
  const [image, setImage] = useState(undefined);
  const [transformedImage, setTransformedImage] = useState(undefined);

  // loading an image every time imageUrl changes
  useEffect(() => {
    const loadImage = async () => {
      // generating the Jimp data structure
      // loading an image from an URL
      const jimpImage = await Jimp.read(imageUrl);
      setJimpImage(jimpImage);

      // transforming jimpImage into its Base64 representation
      // and storing it
      const image = await jimpImage.getBase64Async(Jimp.MIME_JPEG);
      setImage(image);
    };

    loadImage();
  }, [imageUrl]);

  // generating the transformed image
  // as soon as the Jimp data structure is ready
  useEffect(() => {
    if (jimpImage) {
      const transformImage = async () => {
        // performing the Jimp image processing operation
        // on jimpImage...

        // e.g. jimpImage.crop(100, 100)

        // storing the transformed image
        // in Base64 format
        const transformedImage = await jimpImage.getBase64Async(Jimp.MIME_JPEG);
        setTransformedImage(transformedImage);
      };

      transformImage();
    }
  }, [jimpImage, height, width]);

  return image && jimpImage ? (
    <>
      <h1>Original Image</h1>
      <img className="originalImage" src={image} alt="Original" />
      <h1>Transformed Image</h1>
      <img
        className="transformedImage"
        src={transformedImage}
        alt="Transformed"
      />
    </>
  ) : (
    <>Loading...</>
  );
}

First, the imageUrl prop value storing the URL to the image you want to transform is used to generate a Jimp data structure. This happens in the first useEffect() function when launching Jimp.read(). This function takes the path to a file or a URL and returns a Promise with the Jimp data structure you call the manipulation functions on. Also, the Base64 representation of the original image is stored to be able to show it in the HTML <img> tag.

Keep in mind that the function passed to useEffect() should not be async. This is why an internal async function was defined and then called instead.

Then, the second useEffect() function is called as soon as jimpImage is initialized and takes care of performing the transformation function on the original image. Again, the Base64 representation of the transformed image is saved.

Finally, both the original and transformed images are rendered by the component. As you can imagine, what will change in the following examples will be the Jimp image transform function called, which might also require extra props.

Cropping an image

The Jimp crop() function crops an image at a given point to a given size.

jimpImage.crop(x, y, width, height)

x: the x coordinate to crop from
y: the y coordinate to crop from
width: the width of the crop region
height: the height of the crop region

See it in action in the live example below:

Resizing an image

The Jimp resize() function resizes an image to set width and height using a 2-pass bilinear algorithm.

jimpImage.resize(width, height);

width: the width to resize the image
height: the height to resize the image to

See it in action in the live example below:

Rotating an image

The Jimp rotate() function rotates an image clockwise by a number of degrees. Note that the width and height of the image will be resized accordingly by default.

jimpImage.rotate(degrees);

degrees: the number of degrees to rotate the image by

See it in action in the live example below:

Filtering an image

Although Jimp does not come with a single filtering function, it offers several functions to achieve the desired results. For example, you can implement an image filtering feature by harnessing the blur(), color(), dither(), normalize(), or threshold() functions.

Let’s see some of them in action in the live demo below:

Jimp vs PhotoEditor SDK

Undoubtedly, Jimp is a powerful tool. It’s an amazingly lightweight and simple library that offers so many image manipulations. On the other hand, achieving complex results requiring multiple manipulations to be performed on the end user’s needs could become challenging. In detail, this would require an advanced, robust, reliable, and easy-to-use UI that only a commercial solution such as IMG.LY’s PhotoEditor SDK can offer.

You could build such a UI yourself on top of Jimp, but this would take a lot of time and effort. This is especially true if you want to implement a general-purpose application, such as an image processing application. For limited and specific tasks without any user interaction, Jimp is the ideal solution.

Also, now you can take a look at the CreativeEditor SDK which is a more advanced API solution and obvious alternative to Jimp!

Conclusion

Today we learned about the jimp JavaScript library, its dependencies, how it works, and how to use it in React real-world scenarios. Specifically, this npm library comes with several image manipulation functions that can be used with a few lines of code. However, building a complex image editing application on top of it would require an entire UI, and you might want to avoid spending time on it. In this case, you should look into a more advanced, fully-featured, and ready-to-use solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Draw on an Image With JavaScript

Antonello — Mon, 23 Aug 2021 15:20:14 GMT

In this article, you will learn to implement a brush to draw on an image in JavaScript, without using any dedicated external libraries, like those of React for image manipulations. As you are going to see, this can be achieved effortlessly using the HTML5 <canvas> element.

In the last few years, touchscreens have become more and more common. This technology has made drawing on screens much easier and accessible to everyone. Thus, allowing users to draw on photos or pictures is a more common feature than ever before. For example, we all expect to be able to modify pictures we took before sending them in messaging applications. This is just one of the several real-world case scenarios where you may need to implement this increasingly valuable feature.

So, let’s see how to draw on an image with Vanilla JavaScript. Follow this step-by-step guide to achieve the following result:

Drawing on Images With `<canvas>`

Clone the GitHub repository that supports this article by launching the following command:

git clone https://github.com/Tonel/how-to-draw-on-an-image-with-javascript-img-ly

Then, launch the index.html page to try the demo application.

Otherwise, continue following this tutorial to see how to implement the demo application step by step.

1. Implementing the Brush Feature

The HTML <canvas> element natively offers everything required to implement a brush feature. So, it is actually the only prerequisites. Despite it is typically used for game graphics, animations, real-time video processing, and data visualization, you can also employ it as a tool to draw on images. This powerful tool allows you to implement many cool features, such as resizing an image. Follow this link to find out how.
In this case, it takes only a handful of lines of code to implement a brush. Let’s see how together:

function drawOnImage(image = null) {
  const canvasElement = document.getElementById('canvas');

  const context = canvasElement.getContext('2d');

  // if an image is present,
  // the image passed as a parameter is drawn in the canvas
  if (image) {
    const imageWidth = image.width;
    const imageHeight = image.height;
    // rescaling the canvas element
    canvasElement.width = imageWidth;
    canvasElement.height = imageHeight;
    context.drawImage(image, 0, 0, imageWidth, imageHeight);
  }

  let isDrawing;
  canvasElement.onmousedown = (e) => {
    isDrawing = true;
    context.beginPath();
    context.lineWidth = 10;
    context.strokeStyle = 'black';
    context.lineJoin = 'round';
    context.lineCap = 'round';
    context.moveTo(e.clientX, e.clientY);
  };

  canvasElement.onmousemove = (e) => {
    if (isDrawing) {
      context.lineTo(e.clientX, e.clientY);
      context.stroke();
    }
  };

  canvasElement.onmouseup = function () {
    isDrawing = false;
    context.closePath();
  };
}

First, a canvas already presents on the HTML page is retrieved. Then, a 2D rendering context is initialized. This will be used to draw on the surface of the blank canvas or on the optional image passed as a parameter. In this last case, the canvas element is resized to fit the image dimensions. Also, the image is drawn by employing the drawImage() function. Next, the logic necessary to draw is implemented.

On mousedown, brush color, thickness, and opacity are defined. Also, the lineJoin and lineCap properties are set to round to make edges less sharp and more rounded. Then, the pointer is moved to the coordinates corresponding to the mouse click position employing the moveTo function.

On mousemove, a line to the new coordinates of the mouse position is drawn by harnessing lineTo and then stroke. Since the mouse position was moved on the mousedown event, a straight line connecting the new (x, y) coordinates is drawn.

Finally, on mouseup, the drawing operation is stopped by assigning the isDrawing variable to false. This flag was introduced to avoid drawing when moving the mouse on canvas without clicking on it first.

Note that the two just mentioned event-handler functions used to draw are enclosed by beginPath and closePath. This ensures that every line drawn represents a new sub-path for the canvas. This allows each line to be completely independent of the others and to be plotted in the canvas as an unmodifiable drawing element.

Et voilà! As you can see, implementing a brush to draw on an image in Vanilla JavaScript takes only a few lines of code.

2. Drawing in Action

Now, let’s take a look at the drawOnImage() function in action through a complete real-world example:

<!DOCTYPE html>
<html>
  <body>
    <div>
      <canvas
        id="canvas"
        width="500"
        height="200"
        style="border: 1px solid black;"
      ></canvas>
    </div>
    <div style="margin-top:5px">
      <span>Size: </span>
      <input
        type="range"
        min="1"
        max="50"
        value="10"
        class="size"
        id="sizeRange"
      />
    </div>
    <div style="margin-top:5px">
      <span>Color: </span>
      <input type="radio" name="colorRadio" value="black" checked />
      <label for="black">Black</label>
      <input type="radio" name="colorRadio" value="white" />
      <label for="black">White</label>
      <input type="radio" name="colorRadio" value="red" />
      <label for="black">Red</label>
      <input type="radio" name="colorRadio" value="green" />
      <label for="black">Green</label>
      <input type="radio" name="colorRadio" value="blue" />
      <label for="black">Blue</label>
    </div>
    <div style="margin-top:5px">
      <button id="clear">Clear</button>
    </div>
    <br />
    <input id="upload" type="file" accept="image/*" />
    <p>
      Start drawing on the blank canvas or upload an image and use the brush to
      modify on it
    </p>
    <script src="src/index.js"></script>
  </body>
</html>

const fileInput = document.querySelector('#upload');

// enabling drawing on the blank canvas
drawOnImage();

fileInput.addEventListener('change', async (e) => {
  const [file] = fileInput.files;

  // displaying the uploaded image
  const image = document.createElement('img');
  image.src = await fileToDataUri(file);

  // enabling the brush after the image
  // has been uploaded
  image.addEventListener('load', () => {
    drawOnImage(image);
  });

  return false;
});

function fileToDataUri(field) {
  return new Promise((resolve) => {
    const reader = new FileReader();
    reader.addEventListener('load', () => {
      resolve(reader.result);
    });
    reader.readAsDataURL(field);
  });
}

const sizeElement = document.querySelector('#sizeRange');
let size = sizeElement.value;
sizeElement.oninput = (e) => {
  size = e.target.value;
};

const colorElement = document.getElementsByName('colorRadio');
let color;
colorElement.forEach((c) => {
  if (c.checked) color = c.value;
});
colorElement.forEach((c) => {
  c.onclick = () => {
    color = c.value;
  };
});

function drawOnImage(image = null) {
  const canvasElement = document.getElementById('canvas');

  const context = canvasElement.getContext('2d');

  // if an image is present,
  // the image passed as parameter is drawn in the canvas
  if (image) {
    const imageWidth = image.width;
    const imageHeight = image.height;

    // rescaling the canvas element
    canvasElement.width = imageWidth;
    canvasElement.height = imageHeight;

    context.drawImage(image, 0, 0, imageWidth, imageHeight);
  }

  const clearElement = document.getElementById('clear');
  clearElement.onclick = () => {
    context.clearRect(0, 0, canvasElement.width, canvasElement.height);
  };

  let isDrawing;
  canvasElement.onmousedown = (e) => {
    isDrawing = true;
    context.beginPath();
    context.lineWidth = size;
    context.strokeStyle = color;
    context.lineJoin = 'round';
    context.lineCap = 'round';
    context.moveTo(e.clientX, e.clientY);
  };

  canvasElement.onmousemove = (e) => {
    if (isDrawing) {
      context.lineTo(e.clientX, e.clientY);
      context.stroke();
    }
  };

  canvasElement.onmouseup = function () {
    isDrawing = false;
    context.closePath();
  };
}

As you can see, the HTML page contains everything needed to change brush size, and color. By interacting with them, users can achieve several possibilities and unleash their creativity. Also, in case of errors, they can press the Clear button and return to the initial conditions.

From a technical point of view is that the input element is used to upload an optional image. Then, it is passed to the aforementioned drawOnImage() function. This enables the brush feature on the image or the blank canvas and lets users draw on it. Please, note that the example that was just implemented corresponds to the fiddle you can find at the beginning of the article.

Final Considerations

Although implementing a brush feature to draw on an image in Vanilla JavaScript is definitely an easy task, there are a few downsides that must be addressed. First, making the drawing process smooth while moving the pointer quickly is tricky. This is an optimization problem, which may require you to adopt caching or debouncing techniques. Thus, implementing this feature effectively and efficiently can easily turn into a nightmare. Second, whenever you want to add a new style to the brush, you must spend time implementing all the front-end components required to enhance this feature. Despite not being difficult, this definitely involves boilerplate code.

When you do not want to spend hours writing boring yet unavoidable code to add new colors or brushes, a commercial and all-in-one solution like PhotoEditorSDK should be the preferred approach. In fact, it allows you to no longer stressing due to having to tackle these tedious and time-consuming tasks. Plus, whenever you need help, you can ask for support from the IMG.LY developers who built the SDK.

Drawing on an Image with `photoeditorsdk`

First, you should read integration tutorial article from the official documentation on how to get started with PhotoEditorSDK in HTML and JavaScript. Then, by using the Brush Engine you can start enjoying a brush feature optimized for touch screen interaction. Moreover, it natively supports different brush strokes. They can also be tweaked in terms of color, hardness, size, and opacity. This way, you should be able to achieve the desired result:

Conclusion

In this article, we looked at how to draw on an image with JavaScript. Implementing a brush to draw on images by using the <canvas> HTML element is not complex. However, it inevitably involves boilerplate code. Specifically, when implementing all the minor options such a feature should offer to the end-users. Also, making the brush smooth and optimized for touch screen and fast movements is quite complex. In fact, this requires optimizing or devising an intelligent algorithm aimed at not wasting resources. So, if you want to avoid these issues and use an easy an all-in-one solution, you should consider using more advanced and complete software – such as CreativeEditor SDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.
Like to try out our product? Get in touch or start a free trail today!

How To Resize an Image in React

Antonello — Fri, 30 Jul 2021 10:57:19 GMT

In this article, you will see how to resize an image in JavaScript. In particular, here you will learn to achieve this goal with the react-image-file-resizer React library. If you are looking for a pure Javascript solution, here’s a quick rundown of the HTML API usage.

Providing users with features to resize images has become almost unavoidable. This is because images are larger than ever. It is not a secret that the quality of the images and therefore their file sizes have been increasing for years.

The problem is that dealing with large files is time-consuming. Plus, it may cost money in bandwidth when uploading or downloading them. This is why it is so important to shrink the size of images by resizing them. Also, these issues fall on end-users, and this should be avoided.

So, let’s see how to resize an image in React with react-image-file-resizer. By following this step-by-step tutorial, you will achieve the following result:

Prerequisites

This is the list of all the prerequisites for the demo application you are going to build:

Node.js and npm 5.2+ and higher
react-image-file-resizer >= 0.4.7

Resizing an Image with `react-image-file-resizer`

You can clone the GitHub repository that supports this article and try the demo application by launching the following commands:

git clone https://github.com/Tonel/resize-image-react-demo-imgly
cd resize-image-react-demo-imgly
npm i
npm start

Otherwise, you can continue following this tutorial and build the demo application step by step.

1. Creating a React Project

Generate an empty working project in React with Create React App, the officially supported way to create single-page React applications. You can initialize a new project called react-image-resizer-demo with the following command:

npx create-react-app react-image-resizer-demo

You will now have a demo project located in the react-image-resizer-demo folder with the following file structure:

react-image-resizer-demo
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Move into the react-image-resizer-demo folder and start a local server by launching these commands:

cd react-image-resizer-demo
npm start

Reach http://localhost:3000/ in your browser. You should now be able to see the default Create React App screen, as follows:

2. Installing `react-image-file-resizer`

First, you have to add the react-image-file-resizer library to your project’s dependencies. You can do it by running the following command:

npm install --save react-image-file-resizer

Thus, your package.json file will be updated accordingly. You should now be able to spot react-image-file-resizer as a dependency.

Now, all prerequisites have been met. So, you can start building your image resizer component. Let’s see together how.

3. Building the Image Resizer Component

First, make a components folder inside src. Then, create an ImageResizer folder containing index.js and index.css. These two files will contain the resizer component definition and style respectively.

Initialize index.js with the following snippet:

import React from 'react';

function ImageResizer() {
  return <div>{/*TODO*/}</div>;
}

export default ImageResizer;

This way, you have just defined an empty ImageResizer component.
Now, import the Resizer utility from the react-image-file-resizer library. Add it to the ImageResizer imports:

import Resizer from 'react-image-file-resizer'

This is what the final ImageResizer will look like:

import React, {useEffect, useState} from "react";
import Resizer from "react-image-file-resizer";

function ImageResize(props) {
    const {imageToResize, onImageResized, resizeAspect, resizeQuality} = props;

    const [imageToResizeUri, setImageToResizeUri] = useState();
    const [imageToResizeWidth, setImageToResizeWidth] = useState();
    const [imageToResizeHeight, setImageToResizeHeight] = useState();

    useEffect(() => {
        if (imageToResize) {
            const reader = new FileReader();

            reader.addEventListener('load', () => {
                setImageToResizeUri(reader.result);
            });

            reader.readAsDataURL(imageToResize);
        }
    }, [imageToResize])

    useEffect(() => {
        if (imageToResize && imageToResizeWidth && imageToResizeHeight) {
            Resizer.imageFileResizer(
                imageToResize,
                imageToResizeWidth * resizeAspect,
                imageToResizeWidth * resizeAspect,
                "JPEG",
                resizeQuality,
                0,
                (uri) => {
                    onImageResized(uri)
                },
                "base64"
            );
        }
    }, [
        imageToResize, imageToResizeWidth, imageToResizeHeight,
        onImageResized, resizeAspect, resizeQuality
    ]);

    return (
        <img
            src={imageToResizeUri}
            onLoad= {(e) => {
                const img = e.target;
                setImageToResizeWidth(img.width);
                setImageToResizeHeight(img.height);
            }}
            crossorigin="anonymous" // to avoid CORS-related problems
        />
    );
}

ImageResize.defaultProps = {
    onImageResized: () => {},
    resizeAspect: 0.5,
    resizeQuality: 100
}

export default ImageResize;

imageToResize is the source Blob representing the image received from the props. First, imageToResize is transformed into a Base64 URI by the first useEffect() function, and then showed. After being loaded by the <img> HTML tag, the image’s width and height are saved to be used during the resizing operation. This is done by the second useEffect() function, which takes care of resizing the Blob image received by employing the Resizer utility.

Please, note that resizeAspect and resizeQuality are two props in charge of defining the resize aspect and quality percentage, respectively. By default, they are assigned to 0.5 and 100. This means that the resized image will be 50% smaller and no compression will be applied during the resizing process.

4. Putting It All Together

Now, let’s see the ImageResizer component defined above in action. All you need to do, is change the App.js file as follows:

import React, { useState } from 'react';
import './App.css';
import ImageResize from './components/ImageResizer';

function App() {
  const [imageToResize, setImageToResize] = useState(undefined);
  const [resizedImage, setResizedImage] = useState(undefined);

  const onUploadFile = (event) => {
    if (event.target.files && event.target.files.length > 0) {
      setImageToResize(event.target.files[0]);
    }
  };

  return (
    <div className="app">
      <h1>Image Resizer</h1>
      <p>
        Please, upload an image and it will be showed both original and resized
        by 50%
      </p>
      <input type="file" accept="image/*" onChange={onUploadFile} />
      <div>
        <ImageResize
          imageToResize={imageToResize}
          onImageResized={(resizedImage) => setResizedImage(resizedImage)}
        />
      </div>
      {resizedImage && (
        <div>
          <h2>Resized Image</h2>
          <img alt="Resize Image" src={resizedImage} />
        </div>
      )}
    </div>
  );
}

export default App;

The input element allows users to upload an image. This is stored in the imageToResize state variable, and then passed to ImageResizer as a props. This will take care of resizing it accordingly. The resulting resized image is saved thanks to the setResizedImage function, and finally displayed in the Resize Image section.

If you are a Next.js user, you can use the <Image /> component to make it resize images for you. React will then render resized images automatically.

Final Considerations on `react-image-file-resizer`

Resizing an image cannot be considered a complex task. However, using a library like react-image-file-resizer makes everything easier. In fact, you can achieve your goal with just a handful of lines of code. On the other hand, you should take into account what using such specific libraries implies. In fact, when you need to perform many image-related operations, you may be ending up with as many libraries as operations required.

Not only might they be complex to make them coexist, but they may have very different UIs as well. This is why, harnessing a commercial and more complete solution such as PhotoEditorSDK could be a better approach. With only one library, you would get several tools to deal with images as you need. This, while preserving the consistency of your application’s UI. Plus, whenever you need help, you can ask for support from the img.ly developers who built the SDK.

Resizing an Image with PhotoEditorSDK

First, you should read this article from the official documentation on how to get started with PhotoEditorSDK in React. Then, by using the transform tool you can perform cropping, resizing, flipping, and rotation operations with just one feature. This way, you should be able to achieve the desired result

Conclusion

In this article, we looked at how to resize an image in React. Although this cannot be considered a complex feature to implement, using a library such as react-image-file-resizer is recommended. As we have seen, you can resize an image effortlessly and with only a few lines of code. On the other hand, it is a library with a very specific and limited purpose. So, if you needed to perform more than one operation on your images, you might want to take advantage of a more advanced and complete solution – such as PhotoEditorSDK.

Tech – IMG.LY Blog

How to Build a Cross-Platform Editing Experience for Web-to-Print

The Foundation: Platform Agnostic Graphical Engine

Ensuring Consistent Cross-Platform Output

Designing UIs - Balancing Requirements and Constraints

Template Creation on the Web

Mobile UIs

Conclusion

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Setting Up Your Project

Asking for Permissions

Making Video Recordings

Configuring the Camera

Presenting the Controller

Building Your Creation

Configuring the Editors Assets

Exporting the Finished Video

Setting Up a Playback Controller

Where to Go From Here

How to Add Stickers and Overlays to a Video in Flutter

How does Flutter handle video assets?

Prerequisites:

Adding Overlay Stickers and Text to a Video in Flutter

Summing up

Adding Overlays to a Video with Flutter package for VideoEditor SDK

How To Build a Video Editor With Wasm in React

What is WebAssembly?

What is ffmpeg.wasm?

Building a Video Editor in React with WASM

Prerequisites

Initializing a React Project

Installing the Project’s Dependencies

Uploading a Video to the Video Editor

Playing a Video in React

Cutting a Video in React

Converting a Video to GIF in React

Putting It All Together

Commercial Alternative

Conclusion

FFmpeg - The Ultimate Guide

Introduction to FFmpeg

Installing FFmpeg

FFmpeg history

FFmpeg supported codecs and formats

Compilation of FFmpeg

FFmpeg’s strengths

Filtering

Hardware acceleration

Versatile input/output methods

Running example commands

Introduction to media concepts

Audio

Sampling Rate

Bitrate

Channels

Image properties

Resolution

Bit-depth

Transparency

Video properties

Video codecs

Audio codecs

Containers

Example Material

FFplay and FFprobe

FFplay

FFprobe

FFmpeg concepts

Inputs

Streams

Outputs

Mapping

Filtering

FFmpeg’s command line system

FFmpeg CLI

Specifying an input file

Specifying an output

Understanding the command line order

Mapping files

Multiple outputs

What is `ffmpeg.wasm`?