Mobile App Development – IMG.LY Blog

CE.SDK v1.70 Release Notes

Neslihan — Tue, 10 Mar 2026 16:08:17 GMT

Fresh off v1.69 (our biggest developer experience release yet), v1.70 now strengthens what’s already there. The Android video timeline gets a professional-grade overhaul, iOS and Android gain video duration controls. Under the hood, significant stability improvements across every platform.

Let’s dive in!

Edit Videos with Clarity on Android

These updates to the Android video timeline improve how editing looks and feels for your mobile users.

Use Real Audio Waveforms for Accurate Video Editing

Audio clips now display actual waveform visualization, replacing the previous placeholder. Users can see exactly where audio peaks and drops — making precise edits faster and more intuitive.

→ Explore Audio Waveform Docs

Keep Your Timeline Clutter-Free

Images, stickers, and shapes now display one thumbnail per clip, instead of repeating across the track. The timeline is cleaner and easier to read, keeping complex video compositions tidy on mobile screens.

Keep Text Visible Throughout Your Timeline

Text clips now show their actual text content alongside the clip label, so users can identify and navigate text layers without opening each one.

Improvements

Set Video Duration Limits on iOS and Android
You can now define minimum and maximum video duration constraints directly in CE.SDK on both iOS and Android.

Timeline markers make the boundaries visible to users while they edit, and export validation ensures clips meet the defined limits before rendering.

Edit Colors like a Pro – in One Panel

The color panel for web is now unified: preview, saved colors, and custom editing controls all live in one place. Users can pick, adjust, and reuse colors in a single continuous workflow, without a separate modal or extra clicks.

CMYK Color Mode for Web
Additionally, the web editor supports restricting the color picker to a specific color mode: RGB, CMYK. When CMYK is enforced, elements using a different color mode show a clear convert option rather than broken inputs.
New pages, asset library blocks, and fill type switches automatically convert colors to match the configured mode.

Full Changelog

All technical details, breaking changes, and migration notes are in the v1.70.0 Changelog.

Thank you for building with IMG.LY.

Best Video SDKs for Mobile Applications: A Comprehensive Comparison for Developers

Klaudia — Tue, 02 Dec 2025 14:54:37 GMT

If you’re building a mobile app that needs video editing capabilities, you’re facing a tough decision: should you integrate a commercial SDK, work with open-source tools, or try to build something yourself?

Video editing on mobile isn’t just hard, it’s complex in ways that surprise even experienced developers. You’re dealing with device fragmentation, battery constraints, memory management, and user expectations shaped by apps like TikTok and CapCut. And unlike web apps, mobile video editing requires native performance optimization to feel responsive.

In this article, we’ll compare the most popular video SDK options for mobile applications: IMG.LY, Banuba, Meishe, BytePlus, FFmpeg, and GStreamer and show you how they stack up across features, platforms, integration complexity, and pricing. We’ll be honest about where each solution shines and where you’ll hit roadblocks.

We’ll also dig into what it actually takes to build with each option: difficulty of integration, performance considerations, and what happens when you need to scale beyond a prototype.

The Challenge: Building Video Editing Into Mobile Apps

Before diving into specific SDKs, let’s talk about why video editing on mobile is uniquely challenging.

Processing Power vs Battery Life

Video processing is resource-intensive. Every frame you manipulate — trimming, filtering, compositing, exporting — requires significant CPU and GPU work. On mobile devices, this creates a constant tension between delivering fast performance and not draining your user’s battery in minutes.
Approximately 60% of battery usage in media apps comes from data transmission and processing. If you’re not careful with how you handle encoding, memory management, and background processes, your app will get uninstalled fast.

Device Fragmentation

Unlike web apps where you can control the environment, mobile apps run on thousands of device configurations. An iPhone 15 Pro handles 4K video editing effortlessly; a three-year-old Android phone might struggle with 1080p. You need to account for different chipsets, GPU capabilities, screen sizes, and OS versions — all while maintaining consistent UX.

User Expectations Are Higher Than Ever

Users expect mobile video editors to feel as responsive as native camera apps. They want real-time preview, smooth timeline scrubbing, instant filter application, and fast exports. Anything that feels sluggish or unpolished gets compared unfavorably to TikTok, Instagram, or CapCut — apps with massive engineering teams and years of optimization work.

Building this level of polish from scratch is a multi-month (or multi-year) undertaking. That’s why most teams reach for an SDK.

Understanding Video SDK Options

The video SDK landscape breaks down into three broad categories:

Commercial SDKs (IMG.LY Banuba, Meishe, BytePlus) provide pre-built editors with UI components, effects libraries, and dedicated support. They’re designed to integrate quickly but come with licensing costs.

Open-source frameworks (FFmpeg, GStreamer) give you powerful processing engines but no UI, no templates, and no hand-holding. You’ll need to build everything user-facing yourself.

Hybrid approaches combine open-source processing with custom UI development, giving you flexibility at the cost of significant engineering time.

When evaluating options, consider:

Do you need a user-facing editor or just backend processing?
How important is cross-platform consistency?
What’s your timeline and team size?
What happens when you need to scale or add new features?

Now let’s break down each solution.

1. IMG.LY CE.SDK (The Complete Solution)

IMG.LY CreativeEditor SDK is an enterprise-grade video editing solution built for teams that need production-ready editing features, cross-platform consistency, and automation — without spending months building and maintaining a custom solution.

Unlike processing libraries (FFmpeg, GStreamer) or social-first SDKs (Banuba, BytePlus), IMG.LY provides both interactive editing and server-side automation in one platform. You can embed the SDK for user-facing editing and use the same engine for headless creative automation on your backend.

Core Capabilities

IMG.LY delivers a complete video editing platform with timeline-based editing, design tools, and AI features:

Timeline-Based Editing:

Arrange videos on multi-track timelines
Trim, crop, and adjust duration
Merge videos and create collages
Audio overlay and mixing
Layer management and compositing

Creative Tools:

60+ filters and effects
Adjustments (brightness, contrast, saturation, exposure)
Text design with custom fonts and styles
Dynamic stickers and overlays
Frames with multiple blend modes

Template System:

Design templates with text variables and placeholders
Lockable elements for brand consistency
Automated variation generation
Server-side rendering for mass personalization

AI & Automation:

Native AI features: video generation, captions, voiceovers, thumbnails
Model-agnostic plugin system (integrate any AI model or API)
Background removal and style transfer
Smart filters and enhancements

Customization Framework:

Headless API for completely custom interfaces
Theming to match your brand
Toolbar configuration and element positioning
Multi-language localization
External asset library integration (Getty Images, Soundstripe)

Platform Support

Web: JavaScript, React, Angular, Vue.js, Svelte, Next.js
Mobile: Android (Kotlin/Java), iOS (Swift), React Native, Flutter
Desktop: macOS, Mac Catalyst, Electron
Server: Node.js for backend automation

Technical Architecture

Client-side encoding means fast editing operations run directly on devices without requiring backend infrastructure. This cuts bandwidth, reduces upload times for users, and eliminates server costs for standard editing.

Server-side rendering enables creative automation at scale — design templates once, generate thousands of variations programmatically from any data source with 100% consistent rendering.

Ideal Use Cases

Social media creation: In-app video recording with dual-camera support, advanced audio overlays, and real-time effects
Template-based workflows: Automate video creation for marketing materials, product videos, social media ads
Enterprise automation: Server-side rendering for high-volume ad generation, creative pipelines, mass personalization
E-commerce: Product video creation, user-generated content, personalized demos
SaaS platforms: In-app video editing for user-generated content, brand-controlled templates

Real Customer Examples

Social Media & Content Creation:

Mobile apps use IMG.LY SDK for in-app video recording with dual-camera support, allowing creators to capture picture-in-picture content directly
Advanced audio overlay features enable users to mix music tracks, voiceovers, and original audio in real-time
Real-time effects and filters provide instant creative feedback during recording and editing

Template-Based Video Production:

Marketing teams design video templates once and generate thousands of personalized variations programmatically using server-side rendering
E-commerce platforms automate product video creation by mapping product data to video templates, producing consistent branded content at scale
Brand teams create locked templates with variable elements, ensuring on-brand output while allowing distributed teams to customize messaging

Enterprise Creative Automation:

Ad platforms use CE.SDK’s Node.js capabilities for high-volume programmatic video generation, creating thousands of ad variations from single templates
DAM systems integrate IMG.LY for in-workflow video editing, allowing teams to make quick adjustments without leaving their asset management platform
Publishing platforms automate social media content by rendering video templates with dynamic data feeds, producing daily content without manual editing

500+ million creations per month powered by IMG.LY’s video editing and automation infrastructure across 600+ customers

Limitations

Paid solution. Unlike open-source frameworks, CE.SDK requires an enterprise license. This makes it less accessible for hobby projects or very early-stage prototypes, but the investment pays off in reduced development time, ongoing support, and scalability.

Enterprise-focused. The SDK is built for teams that need reliability, support, and scale. If you’re building a quick prototype or low-traffic app, open-source tools might be more cost-effective initially (though you’ll pay in engineering time).

Pricing / Licensing

Custom enterprise licensing based on platform, features, and usage
Dedicated support included: onboarding, SLAs, regular updates
Integration time: Most teams have a working editor in hours, not weeks

Who It’s For

Businesses that need a complete, production-ready video editing solution with cross-platform support, automation capabilities, enterprise scalability, and ongoing support. Teams that want to ship faster and focus on their unique product features rather than rebuilding video editing infrastructure.

Why IMG.LY SDK Stands Out

While other SDKs focus on either interactive editing (Banuba, BytePlus) or backend processing (FFmpeg, GStreamer), IMG.LY is the only solution that combines:

Cross-platform video editing (Web, iOS, Android, React Native, Flutter, Node.js)
Polished editor UI with complete customization options
Template system with server-side automation
AI-powered creative features
Dual-purpose architecture (client-side editing + server-side rendering)
Enterprise-grade scalability and dedicated support

If you’re evaluating other SDKs, ask yourself: how much engineering time will you spend building, maintaining, and scaling a custom solution? Our SDK eliminates that overhead so your team can focus on what makes your product unique.

2. Meishe SDK

Meishe is a Chinese video editing SDK provider with a comprehensive suite covering video editing, beauty filters, short video creation, and audio processing. They’re particularly strong in the Asian market and have been adopted by major platforms like Bilibili, OPPO, Xiaomi, and Himalaya.

Core Capabilities

Meishe offers film and television-level processing capabilities with a focus on high-quality output:

Video Editing Features:

Real-time preview through LiveWindow (preview effects during editing without pre-processing)
Support for multiple file formats with input up to 4K
Video output up to 1080P with configurable quality levels (1080P, 720P, 480P)
Unlimited audio track editing with multi-segment support
Transition effects between clips with customization options

Creative Tools:

Real-time skin beautification with adjustable parameters
Animation stickers with customizable properties
Special effects filters
Full-link HDR support
Professional video/audio editing with free function combination

Technical Details:

4K shooting capability
Real-time special effects processing
Full-link HDR processing
Professional-grade video and audio editing

Platform Support

Native: iOS, Android
Cross-Platform: Flutter (with unified display across platforms)

Ideal Use Cases

Enterprise applications requiring 4K video support
Professional video editing apps
Social media platforms (especially in Asian markets)
Streaming and content creation apps
Apps requiring HDR video processing

Limitations

Documentation primarily in Chinese. While Meishe has English documentation, many resources and examples are primarily available in Chinese, which can create barriers for Western development teams.

Less Western market presence. Most case studies and customer examples are from Chinese companies, which can make it harder to assess fit for Western markets.

Customization complexity. Getting deep customization may require more direct engagement with Meishe’s team compared to more self-service SDKs.

Pricing / Licensing

Pricing information is not publicly available — you’ll need to contact Meishe directly for quotes. This is common for enterprise-focused SDKs but can slow down evaluation.

Who It’s For

Development teams (especially in Asian markets) building professional-grade video editing apps that need 4K support, HDR processing, and advanced real-time capabilities.

How it compares to IMG.LY

Meishe offers strong professional-grade video processing with 4K and HDR support, particularly well-suited for Asian markets. IMG.LY provides similar 4K capabilities but adds cross-platform consistency (including web and server), template-driven automation, and a more developer-friendly integration experience for global teams. Meishe’s documentation is primarily in Chinese, while IMG.LY offers comprehensive English documentation and Western market support. If you’re building for Asian markets with HDR requirements, Meishe is a strong choice. For global deployment with automation needs, IMG.LY’s unified platform across client and server offers broader flexibility.

3. BytePlus Video Editor SDK

BytePlus Video Editor SDK comes from ByteDance (the company behind TikTok and CapCut). It’s designed to replicate the short-form, social-first editing experience that made those platforms successful, giving app developers similar workflows and effects libraries.

Core Capabilities

BytePlus provides three main feature sets built around social media video creation:

Camera Feature:

Real-time video capture and recording
4K video recording on any mobile device
Live filters and live beauty filters during capture

Preview Editor Feature:

Video editing and composition tools
More than 80,000 effects and filters
AR stickers and visual enhancements

Track Editor Feature:

Advanced multi-track timeline editing
Speed change effects
Transitions and audio features
Multi-layer composition

Export Options:

Support for 540p, 720p, 1080p, 4K resolutions
Frame rates: 25fps, 30fps, 50fps, 60fps

Platform Support

Native: iOS (Cocoapods, SPM, Objective-C, SwiftUI), Android (Gradle, Java, Jetpack Compose)
Cross-Platform: React Native, Flutter
Magic Template SDK: Available for both platforms

Ideal Use Cases

Social media apps focused on user-generated content
Short-form video platforms
Apps that want TikTok-like editing workflows
Content creation tools with extensive effects libraries
Apps requiring machine learning-powered visual enhancements

Limitations

Effects and beauty features are separate. The standard and lite versions of the SDK don’t include Camera props, filters, and beauty effects — you need to purchase those separately, which can complicate pricing and integration.

Customization beyond theming is limited. While you can match the SDK to your brand through theming, deeper UI customization requires more effort compared to headless SDKs.

Annual licensing only. The SDK operates on an annual subscription basis, which may not work for all business models or early-stage products.

Pricing / Licensing

Annual subscription: License valid for 1 year covering both iOS and Android
Contact sales for pricing details
Integration time: Estimated 2 weeks per platform (iOS & Android) for one developer
Customization may extend integration timeline

Who It’s For

Teams building social media-style video apps that want to replicate TikTok’s editing experience, especially if you need access to a massive effects library and ByteDance’s proven video editing architecture.

How it compares to IMG.LY

BytePlus brings TikTok’s proven social video editing architecture with 80,000+ effects and filters, making it ideal for replicating short-form social experiences. IMG.LY takes a different approach: instead of a massive pre-built effects library, it focuses on customization, templates, and automation. BytePlus excels at client-side social video creation with extensive effects; IMG.LY excels at template-driven workflows, server-side automation, and cross-platform consistency (including web and Node.js). If you want TikTok-like effects out of the box, BytePlus delivers. If you need creative automation, programmatic content generation, or want to build custom workflows with full control over the editing experience, IMG.LY offers more flexibility and a dual-purpose architecture.

4. Banuba Video Editor SDK

Banuba is a commercial video editing SDK focused on social media apps, with a strong emphasis on AR effects and beauty filters. It’s designed for teams building TikTok-like experiences or apps where user-generated video content is central to the product.

Core Capabilities

Banuba provides a full-featured video editor with AI-powered tools and creative effects:

Video Editing Features:

Picture-in-picture for multi-video layouts
AI-powered clipping that detects important segments
Customizable templates with replaceable elements
Trimming, merging, and timeline-based editing
AI-generated captions (English, Mandarin, Spanish, Portuguese)

Creative Tools:

Face AR masks with morphing effects and animated backgrounds
Video and transition effects (including “Rave” and “Cathode Flash”)
Audio editing with recording and mixing
Text and GIF overlays
Beauty effects (skin smoothing, teeth whitening)
Voice-change effects (Elf, Robot, Squirrel, Giant, Echo, Baritone)

Technical Details:

AI algorithms work offline — all processing happens on the device
No user content transmitted to servers
Requires OpenGL ES 3.0 minimum (3.1 for GPU-accelerated neural networks)

Platform Support

Native: iOS 15.0+, Android 6.0+
Cross-Platform: React Native, Flutter, NativeScript

Ideal Use Cases

Social media applications with AR filters
Photo/video editing apps for content creators
E-commerce platforms with video demos
Lifestyle and educational apps
Short-form video content creation

Limitations

Pricing complexity. Banuba’s pricing is custom and depends on platform, features, custom development, and payment terms. You’ll need to contact sales for accurate quotes, which can slow down evaluation.

AR licensing. Face AR features are optional and may require separate licensing, adding complexity to your commercial agreement.

Heavy focus on AR. If you don’t need beauty filters or AR masks, you’re paying for features you won’t use.

Pricing / Licensing

Free trial: 14 days with all features
Commercial: Custom pricing based on platform, features, and usage
Priced per platform per month

Who It’s For

Teams building social media apps or content creation tools that need AR effects, beauty filters, and TikTok-like creative features out of the box.

How it compares to IMG.LY

Banuba excels at AR effects and beauty filters for social media apps, while IMG.LY focuses on template-based workflows and cross-platform automation. If you need face masks and beauty effects, Banuba is purpose-built for that. But if you need server-side rendering, creative automation, or want the same editing engine across web, mobile, and backend, IMG.LY provides broader platform coverage and a dual-purpose architecture. Banuba is optimized for client-side social video creation; IMG.LY handles both interactive editing and programmatic content generation at scale.

5. FFmpeg (via FFmpegKit)

FFmpeg is the industry-standard open-source tool for video processing. It’s not a video SDK in the traditional sense — it’s a command-line tool and set of libraries that handle encoding, decoding, transcoding, filtering, and nearly every video operation imaginable.

FFmpegKit is a wrapper that makes FFmpeg accessible on mobile platforms (Android, iOS, React Native, Flutter) through platform-specific APIs. It’s the primary way developers integrate FFmpeg into mobile apps.

Core Capabilities

FFmpeg handles the entire video processing pipeline:

Video Operations:

Encoding and decoding (every codec imaginable)
Format conversion (MP4, MOV, AVI, MKV, WebM, etc.)
Trimming, cutting, merging, splitting
Filtering and effects
Subtitle burning
Video stabilization
Audio extraction and mixing

Technical Features:

Command-line interface (via wrapper APIs)
Support for hardware acceleration
Concurrent command execution
Extensive codec library
Professional-grade output quality

Platform Support

Native: Android (Java API), iOS (Objective-C API), Linux (C++ API), macOS, tvOS
Cross-Platform: Flutter (Dart API), React Native (JavaScript API with TypeScript definitions)
Eight prebuilt packages: Distributed via Maven Central, CocoaPods, pub, npm

Ideal Use Cases

Backend video processing and transcoding
Format conversion at scale
Apps that need specific codec support
Video compression and optimization
Building custom video processing pipelines
Server-side automation

Limitations

No UI whatsoever. FFmpeg is a processing engine. If you want users to interact with video — timelines, filters, previews — you’ll need to build the entire interface yourself from scratch.

Steep learning curve. FFmpeg’s command-line syntax is powerful but complex. Even simple tasks require understanding flags, options, and codec parameters.

GPL licensing complexity. While FFmpegKit itself is LGPL v3.0, including GPL-licensed libraries makes the entire bundle GPL v3.0, which has implications for commercial apps.

FFmpegKit has been officially retired. As of June 2025, FFmpegKit is no longer maintained. Community-maintained forks exist, but official support is ending.

Pricing / Licensing

Free and open-source: LGPL v3.0 for FFmpegKit library
GPL v3.0 when GPL-licensed libraries are included
Licensing depends on which codecs and libraries you bundle

Who It’s For

Developers who need powerful backend video processing, format conversion, or transcoding — and are willing to build all user-facing features themselves. Not suitable for teams that need a ready-to-use editor.

How it compares to IMG.LY

FFmpeg is a processing engine, not an editing SDK. It handles transcoding, format conversion, and video manipulation via command-line operations — but provides no UI, no templates, and no interactive editing components. IMG.LY includes a production-ready editor, template system, and user-facing tools out of the box, while also supporting server-side automation through Node.js. If you only need backend processing and are building all UI yourself, FFmpeg (via community forks) can work. If you need both interactive editing and automation, IMG.LY eliminates months of development by providing both in one platform. IMG.LY also avoids GPL licensing complications and the uncertainty of using a retired library with community forks.

6. GStreamer (with GStreamer Editing Services)

GStreamer is a mature open-source multimedia framework used across Linux, Android, iOS, Windows, and macOS. It’s designed for building media applications — streaming, playback, recording, and non-linear editing — with a plugin-based architecture that makes it extremely flexible.

GStreamer Editing Services (GES) is a higher-level library built on top of GStreamer to simplify video editing application development. It was originally funded by Nokia for mobile video editing (“video editing in your pocket”).

Core Capabilities

GStreamer provides the foundation for building multimedia applications:

Core Features:

Audio and video playback, recording, streaming
Non-linear editing capabilities via GES
Extensive plugin library (encoders, decoders, filters)
Hardware acceleration support
Low-latency streaming (RTSP, WebRTC)
Support for diverse codecs and formats

GStreamer Editing Services:

High-level API for creating editing applications
Timeline-based editing
Transition effects
Audio mixing
Asset management

Performance Optimizations:

Offloading work to specialized hardware (DSPs, GPUs)
Low power consumption for embedded processors
Support for ARM, MIPS, SPARC architectures

Platform Support

Cross-Platform: Linux, Android, iOS, Windows, macOS
Language Bindings: C++ (primary), Python, Java, JavaScript
Mobile: Official SDK binaries for Android and iOS

Ideal Use Cases

Custom video editing apps requiring deep control
Streaming applications (RTSP, WebRTC)
Media players and recording apps
Research and academic projects
Embedded systems and IoT devices
Apps requiring specialized codec support

Limitations

Steep learning curve. GStreamer’s plugin-based architecture is powerful but complex. Understanding pipelines, bins, and element linking takes time even for experienced developers.

No pre-built UI. GES provides editing APIs, but you’ll need to build the entire interface yourself. This means designing timelines, playback controls, effect panels, export dialogs — everything.

Limited mobile-specific examples. While GStreamer officially supports iOS and Android, production-ready mobile editing examples are scarce. You’ll be pioneering integration patterns yourself.

Integration complexity. Getting GStreamer running on mobile requires compiling binaries, managing dependencies, and debugging platform-specific issues. This isn’t a “drop in and go” integration.

Pricing / Licensing

Free and open-source: LGPL license
GStreamer Editing Services included in main distribution
No commercial support by default, but third-party companies (like Fluendo) offer commercial support and custom development

Who It’s For

Teams with strong multimedia engineering expertise who need deep control over video processing and are willing to build everything from the ground up. Not suitable for teams that need fast time-to-market or lack video processing experience.

How it compares to IMG.LY

GStreamer provides powerful low-level control over multimedia processing through its plugin-based architecture, making it ideal for specialized use cases like streaming, embedded systems, or custom pipelines. IMG.LY provides a higher-level, production-ready solution with UI components, templates, and automation built in. GStreamer requires significant expertise and months of development to build user-facing features; IMG.LY offers a working editor in hours with customization options that don’t require multimedia engineering expertise. If you need deep control over codecs, streaming protocols, or embedded deployment, GStreamer’s flexibility is unmatched. If you need to ship a production-ready video editor quickly with cross-platform support and automation capabilities, IMG.LY eliminates the complexity and provides enterprise support.

Comparison Table: Video SDKs for Mobile Applications

Feature / SDK	Banuba	Meishe	BytePlus	FFmpeg	GStreamer	IMG.LY SDK
Platform Support	iOS 15+, Android 6+, React Native, Flutter, NativeScript	iOS, Android, Flutter	iOS, Android, React Native, Flutter	Android, iOS, React Native, Flutter, Linux, macOS	Linux, Android, iOS, Windows, macOS	Web, iOS, Android, React Native, Flutter, Desktop, Node.js
UI Included	✅ Pre-built editor	✅ Pre-built editor	✅ Pre-built editor	❌ Processing only	❌ GES provides APIs only	✅ Production-ready editor + Headless API
Timeline Editing	✅ Yes	✅ Yes	✅ Multi-track	❌ Command-line only	✅ Via GES	✅ Advanced multi-track
AR / Beauty Filters	✅ Strong focus	✅ Real-time beauty	✅ 80,000+ effects	❌ None	❌ None	✅ Filters + extensible AI plugins
Templates	✅ Customizable	⚠️ Limited info	✅ Magic Templates	❌ None	❌ None	✅ Advanced template system with variables
Server-Side Automation	❌ No	❌ No	❌ No	✅ CLI-based	✅ Possible but complex	✅ Native support (same engine)
AI Features	✅ AI clipping, captions	⚠️ Limited info	✅ ML-powered effects	❌ None	❌ None	✅ Native AI + model-agnostic plugins
4K Support	✅ Yes	✅ Up to 4K input	✅ 4K recording	✅ Yes	✅ Yes	✅ Yes
Customization Level	⚠️ UI/UX + optional API	⚠️ Requires team engagement	⚠️ Theming primarily	✅ Complete control	✅ Complete control	✅ Headless API + theming + UI config
Integration Complexity	⚠️ Moderate (GitHub samples)	⚠️ Moderate	⚠️ Moderate (~2 weeks)	⚠️ Complex (retired library)	⚠️ Very complex	✅ Simple (hours to integrate)
Documentation	✅ English, developer portal	⚠️ Primarily Chinese	✅ English, comprehensive	✅ Extensive community docs	✅ Extensive but complex	✅ Comprehensive with samples
Licensing / Pricing	Custom (free 14-day trial)	Contact for quote	Annual subscription	Free, LGPL/GPL	Free, LGPL	Enterprise license
Enterprise Support	⚠️ Custom agreements	⚠️ Direct engagement	⚠️ ByteDance support	❌ Community only	⚠️ Third-party (Fluendo)	✅ Dedicated support + SLAs
Best For	Social media apps with AR focus	Professional apps in Asian markets	TikTok-like social video apps	Backend processing / transcoding	Custom multimedia apps	Cross-platform editing + automation at scale

Making Your Decision: Which Video SDK Is Right for You?

Here’s the honest breakdown based on your use case:

Choose Banuba if:

You’re building a social media app and AR effects are central to your product
You need beauty filters, face masks, and creative effects out of the box
Your primary audience expects TikTok-style creative tools
You’re comfortable with custom pricing and optional feature licensing

Choose Meishe if:

You’re targeting Asian markets where Meishe has strong adoption
You need professional-grade 4K and HDR video processing
Real-time preview and film-quality output are critical
You have access to Chinese documentation resources or engineering support

Choose BytePlus if:

You want to replicate TikTok/CapCut’s editing experience
You need access to 80,000+ effects and filters
Machine learning-powered visual enhancements are important
You’re comfortable with annual licensing and want ByteDance’s proven architecture

Choose FFmpeg if:

You need backend video processing, transcoding, or format conversion
You’re building automation pipelines, not user-facing editors
You have strong video engineering expertise
You’re okay with command-line complexity and building all UI yourself
Be aware: FFmpegKit is officially retired; plan for community forks

Choose GStreamer if:

You need deep control over multimedia processing
You’re building custom streaming, playback, or editing applications
You have strong multimedia engineering expertise
You’re willing to invest significant time in integration and UI development
You need specialized codec support or embedded system deployment

Choose IMG.LY CE.SDK if:

You need both interactive editing and server-side automation
Cross-platform consistency is critical (Web, iOS, Android, React Native, Flutter)
You want a production-ready editor that integrates in hours, not months
Template-based workflows and creative automation are important
You need enterprise support, SLAs, and ongoing updates
Your team wants to focus on product features, not rebuilding video editing infrastructure

Build vs Buy: What’s the Real Cost?

Let’s be honest about the engineering investment required:

Building on FFmpeg or GStreamer:

Months to build a basic timeline editor with preview
Ongoing maintenance for device compatibility, codec updates, performance optimization
No UI components — you’re building everything from scratch
Scaling challenges when you need templates, automation, or new features

Commercial SDKs (Banuba, Meishe, BytePlus):

Days to weeks for basic integration
Pre-built UI and effects libraries
Dedicated support for troubleshooting
Licensing costs vary; evaluate based on your revenue model

IMG.LY SDK:

Hours to integrate a working editor
Production-ready UI with full customization
Dual-purpose: client-side editing + server-side automation
Template system and AI features included
Enterprise support and regular updates

Ask yourself:

What’s your timeline? If you need to ship in weeks, open-source frameworks won’t get you there.
What’s your team’s expertise? Video processing is complex; building from scratch requires specialized knowledge.
What happens when you scale? Adding templates, automation, or AI features to a custom-built solution takes months of additional work.

Commercial SDKs cost money upfront but save significant engineering time. Open-source tools are “free” but expensive in development hours and opportunity cost.

Conclusion: The Right SDK Depends on Your Goals

There’s no one-size-fits-all answer, but here’s the summary:

FFmpeg and GStreamer are powerful processing engines for developers who need backend automation or custom pipelines. They give you complete control but require significant expertise and offer no user-facing components. Great for transcoding, format conversion, and specialized workflows — not for building interactive editors.

Banuba, Meishe, and BytePlus are excellent for social media apps that need AR effects, beauty filters, and TikTok-style editing. They provide pre-built editors with extensive effects libraries, but customization beyond theming can be limited, and they’re focused primarily on client-side editing (no server automation).

IMG.LY CE.SDK is the only solution that combines production-ready interactive editing with server-side creative automation. We provide cross-platform consistency, a complete template system, AI features, and enterprise support — all in one platform. Where other SDKs require you to choose between editing and automation, we give you both.

If you’re building an app where video editing is a core feature (not a side project), and you need to ship fast without compromising on quality or scalability, our SDK eliminates months of development work and ongoing maintenance headaches.

The real question isn’t “which SDK is cheapest?” It’s “how much engineering time will we spend building, maintaining, and scaling a custom solution — and what could we build instead if we had that time back?”

Ready to see what IMG.LY SDK can do? Explore our demos or get in touch to discuss your project.

CE.SDK v1.19 Release Notes

Neslihan — Fri, 22 Dec 2023 14:26:48 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we’re thrilled to introduce CE.SDK v1.19! With this release, you can:

Unify Design Power with Block Unification

Introducing Block Unification for CE.SDK on Mobile and Web—a feature that simplifies design versatility. Videos, images, shapes, and stickers now seamlessly come together as one Graphic Block, easily modified within a unified inspector. All supported settings for your selected block, including strokes, filters, and adjustments, are conveniently displayed in a single, user-friendly space.

Unified Graphic Block
Experience a singular Graphic Block supporting diverse fills (images, videos, colors, gradients) and shapes. Switch fill types (video, image, color) effortlessly, enabling dynamic designs in one place.

Easy Fine-Tuning
Block Unification integrates effects, filters, adjustments, and blur uniformly across all design elements. Elevate your designs with precision and consistency.

A Treat for Users and Developers
For developers, Block Unification means streamlined code maintenance. The consolidation of design blocks into one Graphic Block simplifies development, allowing more focus on crafting exceptional design experiences with IMG.LY.

Enhance Interaction with Mouse Wheel Support

In our latest update, our engine now directly registers mouse wheel events on the canvas, eliminating the need for manual camera manipulation. This seamless integration brings a host of new features, including smooth zooming, centered around the cursor, and support for pinch gestures on multitouch trackpads. Enjoy a more intuitive and unified interaction experience across various inputs, simplifying and enriching your design process.

Migrate to CE.SDK v1.19

The structural changes of CE.SDK v1.19 bring a more composable and powerful editing experience. To benefit from our new features, and to attain to licensing changes, head to our documentation on migrating to v.1.19 for an easy transition.

Wait. There’s more. As we explore the extensive capabilities of our SDK, we’re unveiling new showcases to demonstrate your possibilities. Plus, stick around until the end for an exciting announcement that will elevate your video creation experience!

New Showcase: Automated Resizing

In our new live showcase for web, effortlessly generate size variations of your designs and seamlessly scale your marketing materials across platforms. The automated resizing feature empowers you to easily leverage marketing materials without requiring the involvement of a design team, ensuring efficiency and adaptability.

New Showcase: Version History

Discover the convenience of Version History, enabling you to monitor changes made to each design and seamlessly restore past versions when needed. Explore our straightforward Version History showcase for a practical design experience.

Outlook: Offer TikTok-Inspired Video Editing

Video remains the reigning champion for attracting and retaining your audience. Yet, building a video editor from the ground up is a pain.

To simplify your journey and save valuable time and resources, we are excited to unveil a groundbreaking SDK for creating captivating short-form videos in January 2024!

Empower your users to arrange video, audio, text, and graphics seamlessly on a sleek video timeline. Our new camera introduces popular features like Voiceover, Zoom, Tap to Record, and more. Plus, bring the beloved Split Screen Modes for Reactions and Duets, akin to TikTok and Instagram!

Be the first to access our new IMG.LY SDK—join our waitlist now!

Thanks for reading! Subscribe to our newsletter to stay in the loop with the latest news and updates.

CE.SDK v1.12 Release

Neslihan — Thu, 15 Jun 2023 12:19:24 GMT

In our last release, we introduced exciting features such as cutout stickers and shapes, letter case options, effortless layouting, and more.
Now, get ready for CE.SDK v1.12, packed with more features and essential fixes to enhance your creative editing experience. Let’s explore what’s included:

Auto-Resize Designs

Interfaces: Advanced UI, API
Platforms: All
Easily auto-resize or adapt your designs to different page sizes without the need for separate templates. Our intelligent solution adjusts the elements of your design while preserving full-size elements and optimizing image fill modes. This time-saving feature empowers you to reuse templates with ease, ensuring consistent and professional results. Whether it’s creating graphics for social media, presentations, or printed materials, our auto-resize feature simplifies the process.
Learn more in our documentation.

Enhance Readability with Paragraphs

Interfaces: Default UI, Advanced UI, API
Platforms: All
Achieve better text readability and design aesthetics by modifying the spacing between paragraphs. This feature seamlessly integrates with our advanced text styling options, providing a comprehensive toolkit for creating visually stunning text-centric products, such as postcards and more. Learn more about paragraph spacing in our documentation.

Avoid Cut Off Text

Interfaces: Default UI, Advanced UI, API
Platforms: All
We’ve made a significant improvement related to text clipping that ensures the top of the text is no longer cut off. This enhancement greatly improves the visual presentation and communication of your content, providing a seamless user experience.

Achieve Perfect Design Alignment

Interfaces: Default UI, Advanced UI, Touch UI
Platforms: All
We’ve addressed the problem of snapping to the wrong elements, ensuring a more accurate and reliable snapping experience. Precise alignment is crucial in design, and our continuous improvement efforts will further enhance snapping capabilities in future updates. Enjoy a seamless and frustration-free design process with our enhanced snapping precision.

Take Control of Text Editing with Precision

Interfaces: API
Platforms: All
getTextCursorRange empowers you to access the current cursor range within the text editor. This feature opens up a world of possibilities, whether you’re building a custom user interface or implementing text-related automation. You can:

Query the text cursor offset and the current selected text range.
Automate text generation and insertion with precision.
Seamlessly integrate this functionality into your workflow and unlock new possibilities for your text-centric projects.

Say goodbye to guesswork and welcome a more accurate and efficient text editing experience. Retrieve the current cursor range effortlessly, and precisely insert content exactly where you need it. Learn more in our documentation.

Manage Templates from Multiple Sources

Interfaces: API
Platforms: All
Introducing an improved template configuration feature that allows you to define asset sources for templates. Now, you can effortlessly browse and query a vast database of templates, opening up a world of possibilities for your projects.

By aligning the template configuration process with the existing asset management workflow, we’ve streamlined the experience, saving you time and effort. Moreover, you have the flexibility to define remote asset sources directly from databases, expanding your template options even further.

Enjoy a more streamlined workflow with CE.SDK v1.12, enhanced visual presentation, and the freedom to customize your designs with ease.
Get started for free and bring creativity to your users!

Subscribe to our Newsletter to stay in the loop with the latest news and updates. Thank you for choosing IMG.LY.

IMG.LY Partners with Soundstripe to Infuse Video Editing with Epic Royalty-Free Music & SFX

Jan — Wed, 24 May 2023 07:31:24 GMT

We are excited to announce our exciting new partnership with Soundstripe, the leading provider of royalty-free music for video creators. Our VideoEditor SDK now features a seamless integration with Soundstripe, offering users access to a vast collection of over 9,000 hand-curated songs from 150+ musicians, along with an extensive library of 70,000 sound effects.

Starting from version 11, our VideoEditor SDK now comes pre-equipped with a seamless Soundstripe integration, as detailed in our documentation.

Streamlining Music Licensing for Digital Platforms

Finding the right music for digital platforms hasn’t been easy. Music licensing complexities and the challenge of finding high-quality tracks that align with your brand standards can be overwhelming.

But now, with our partnership with Soundstripe, you can effortlessly enhance your app with royalty-free music and video editing.

Our mission has always been to provide the ultimate toolkit for building captivating creative experiences, and video editing plays a central role in achieving this. We recognize the significant role that music plays in video storytelling, as it has the power to evoke emotions, set the mood, tone, and atmosphere, and greatly influence the pacing and energy of a video. With Soundstripe’s diverse range of over 50 genres, your users will have no trouble finding the perfect track to complement their visual narrative.

Effortless Music Selection

We have developed an intuitive search interface that allows users to explore music by title, genre, and description effortlessly. Once users have found their desired song, they can easily position any section of the track over their video. Furthermore, we are delighted to provide every one of our customers with seven complimentary sample songs, allowing them to get a taste of the fantastic library available. To unlock the full audio library, simply head over to Soundstripe and request an API key.

We are eagerly looking forward to working closely with Soundstripe to deliver the most exceptional video editing experience on the market. We invite you to try out VideoEditor SDK in your Android or iOS app and witness the transformative impact it will have on your users’ creativity. Together, we can take your video editing capabilities to new heights.

Thanks for reading! Stay ahead of the curve with our newsletter and sign up now!

Why Mobile-First is the Future of Print

Klaudia — Thu, 23 Feb 2023 13:11:09 GMT

Over the last 30 years, mobile phones have heavily influenced lifestyles across the globe. They have changed the way we communicate, spend leisure time, look for information, and shop.

Thanks to its accessibility and powerful capabilities, mobile has eclipsed the desktop. Statista reported last year, 59% of all global web traffic came from mobile, with almost five billion people accessing the internet through mobile devices. Moreover, since 2018, app usage has exceeded browser usage on mobile devices, with shopping apps noting 49% year-over-year usage increase between 2020 & 2021.

This means companies have to not only provide a seamless mobile experience if they wish to attract and convert users, but also invest in app development to stay ahead of the curve.

Market Trends

The printing industry is showing no signs of slowing down. In 2021, the global print-on-demand market was valued at $4.90 billion, and it’s projected to generate $39.40 billion in revenue by 2030, according to industry reports from Grand View Research. The report also suggests that print-on-demand is expected to grow at a compound annual growth rate of 26.1% from 2022 to 2030, indicating significant potential for further expansion.

However, print-on-demand is not the only promising sector to watch. The Print Advertising segment is projected to have ad spending worth $46.87 billion in 2023. This sector has remained resilient to the global decline in the print industry during the Covid-19 pandemic, registering an 18% revenue increase in the US between 2019 and 2020.

With the printing industry’s rapid growth, competition is likely to increase, making it essential for companies to prioritize excellence in order to attract and retain customers.

User Preference

It’s undeniable that mobile is becoming increasingly dominant in e-commerce. According to Shopify, almost 50% of all e-commerce transactions are predicted to occur via mobile devices by 2024, with an expected revenue of $620.97 billion. Being able to quickly and effectively serve customers on mobile is now essential for businesses looking to thrive. In 2022, Shopify launched shop.app, a mobile-only e-commerce platform, raising the bar for the industry. To stay competitive, other e-commerce businesses will need to develop similarly sophisticated solutions for mobile users.

Having a seamless mobile experience is also an important factor for B2B companies. Although most interactions still occur on desktop, modern working arrangements allow people to work on the go, without being limited to one device. Providing customers with flexible options is a significant advantage. For example, a web-to-print editor for print advertising should be cross-platform, providing the same experience across all devices during design creation and print validation processes. Offering agile solutions ahead of industry standards can position a company as a go-to service provider. However, an excellent editor alone is not enough; covering the basics of a good mobile experience is crucial.

Game-Changing Mobile Factors

Customers are becoming more unforgiving when it comes to poor user experience. According to Google, 61% of mobile users won’t return to a site they had trouble accessing, with 40% turning to a competitor’s site instead. For print companies relying almost exclusively on online purchases, this poses a substantial threat, hence, the need to fulfill a long list of expectations to satisfy users.

While the ultimate recommendation is to build a mobile app, customers will most likely want to check out the web page first, hence, the importance of providing a spotless web experience.

Here are some of the most prominent factors of mobile UX to optimize.

Design Editing

While for most e-commerce stores developing a smooth purchase experience is rather straightforward, many printing companies have an added layer of complication - catering to custom-made products. Once a rarity, personalization of products has proven to be a constantly growing trend many companies choose to embrace, and rightly so. The global Custom T-shirt Printing market alone was valued at $3.7 billion in 2021 and is projected to reach $6.98 billion by 2028.

With a growing demand for custom prints, follows a necessity for a web-to-print editor that will provide a seamless design process - on both desktop and mobile. Web-to-print editor simplifies artwork supply, jumpstarts the creative process, and takes care of print validation. The ideal tool should be intuitive to navigate and have a quick load time while providing a quality design experience. The same approach should be adopted by print service providers, breaching a gap between multiple software and communication channels clients have to endure.

A powerful web-to-print editor doesn’t just enhance the user experience on your website - it can also open new doors for your business, such as scaling to native mobile apps, print kiosks, and terminals. Developing separate tools for each platform can be time-consuming and costly. With ready-made solutions (as we will discover below), you can speed up the development process and achieve a seamless editing experience across all devices without starting from scratch each time.

Site Speed

In a study conducted by Deloitte, they observed that a difference of 0.1 seconds has improved conversion rate by 8% and increased purchase value by 10% in the retail sector. The highest conversion rate occurs on pages with a load time of 0-2 seconds, and as reported by Google, 53% of users abandon the page if the load time takes more than 3 seconds. Moreover, it has been estimated that slow-loading websites cost retailers $2.6 billion in lost sales every year.

It’s also worth mentioning that Core Web Vitals monitoring page speed is a Google ranking factor. Poor page speed performance will in turn take a hit on rankings - and with a loss of each position on the Search Engine Result Page, the click-through rate falls significantly.

Investing in optimizing page speed and providing a smooth user experience is a win-win on every front.

Mobile-First Design

Responsive web design is not enough anymore. Websites have to be built with a mobile-first approach in mind. What’s the difference? Responsive design ensures that a page can be viewed on all devices, changing its dimensions to fit different screen sizes. The content and the layout of the page, however, remain unchanged.

While responsive web design is just a part of the design workflow making sure a website is accessible for mobile, mobile-first defines the design approach and focuses on creating a website that is intended for mobile users. It means that every aspect of mobile interaction is taken into account when designing a page.

While the general layout of a page is important for both desktop and mobile, clickable elements of design such as buttons and navigation menus can have larger importance for mobile users. Navigating with a thumb or a finger poses more challenges than a neat computer cursor. The mobile-first design makes sure that every part of the page - content, media, navigation, and calls-to-action, feels intuitive to navigate and use. However, even the best mobile-first web design can’t compare to native mobile apps, which usability and performance are far superior.
Hence, mobile-first should be a necessity embraced by all, those who want to go a step further and provide a fully optimized user experience should turn their eyes to app development.

Payments

The simplicity of mobile payments is yet another reason why mobile purchases are on the rise. Mobile wallet solutions like Google Pay, Apple Pay, and Samsung Pay allow users to complete a transaction with just one click, removing the need for credit card input.

Nowadays, providing an easy and quick checkout experience is more important than ever. Over the last few years, the cart abandonment rate significantly increased, from 59.80% back in 2006 to 88.05% in 2020. Focusing on optimizing the checkout and payment processes could reduce the risk of not completing a purchase. A great example of a seamless payment experience is Shopify’s Shop.app with a one-tap checkout, making it as easy as possible for users to convert.

The checkout funnel is also a great space to utilize personalization hacks and recommend similar products. Instapage reported that shopping cart recommendations influence 92% of users, making it a great tool to increase basket size.

Beyond Browser - Investing in Mobile Apps

Mobile app downloads increased by over 80% since 2016, reaching a mighty 255 billion in 2022. The growing popularity of mobile apps is quite understandable, apps provide a better native experience, are easier to navigate, and usually provide personalized recommendations and remember users’ last activity (58% of users consider this important).

In-app shopping, when correctly optimized, is quick and easy, with enhanced product presentation, intuitive navigation, and no unnecessary hassle of filling in forms and adding credit card details.

Print is once again one of the industries that could leverage all advantages mobile apps have to offer. An example of a company that did it right is IMG.LY’s customer Shutterfly, which combined the emerging trends of personalization and going mobile. The company developed an app that automatically creates personalized items users can purchase. They simply ask for permission to access user’s photo gallery, identify pictures with faces and apply them to product templates. This way, customers can be presented with unique products with zero effort of creating an actual design.

Even without automatically generated custom products, an app-to-print editor provides a much smoother editing experience and interface, with native hardware acceleration, local storage, and easier access to photos or even a camera. Moreover, apps present the possibility to provide more accurate recommendations and personalized content, like in-app notifications, reminders, and activity-based discounts. Hence, developing a quality mobile app could not only boost sales, but also increase brand loyalty and customer satisfaction.

Streamlining Mobile Web-to-Print Editing with CE.SDK

With the undeniable advantages of web-to-print design tools come significant challenges in their development. However, CreativeEditor SDK offers a powerful design editor that delivers consistent experiences across all platforms. Our cross-platform engine can power any editor UI, ensuring 100% consistency in the rendered designs across different devices.

Discover how CE.SDK can streamline your mobile web-to-print process with the following capabilities:

Efficient Client-Side Encoding

Our client-side encoding ensures a smooth and fast design process, as the editor runs directly on the user’s mobile device or web browser. This not only reduces upload time and bandwidth costs, but also allows for easy scalability without incurring additional server costs.

Seamless Cross-Platform Integration

CE.SDK simplifies the process of building an application for the web, iOS, Android, and server by providing a creative engine that’s portable across all platforms. This means that you can easily integrate our solution into your existing systems, saving valuable time and resources. And with consistent design and functionality across all devices, your users can switch between platforms with ease.

Powerful Editing Features

With CE.SDK, your users can unleash their creativity and design high-quality prints effortlessly. Take advantage of our powerful editing features, including the ability to capture and edit photos. CE.SDK also allows you to predefine the design process with adjustable placeholders and constraints, making sure that your final product meets all of the necessary print criteria.

Preview & Design Validation

Imagine being able to preview your design in real-time, ensuring your product expectations are exceeded every time. Plus, with automatic design validation, you can rest easy knowing that no element will extend beyond the bleed margins, minimizing the risk of misprints. Don’t just take our word for it, check out our showcases and see for yourself how CE.SDK can transform your design process.

Conclusion

As we navigate the ever-changing landscape of technology, it’s important to stay ahead of the curve and keep up with emerging trends. With mobile devices becoming increasingly essential in our lives, it’s an crucial time to invest in creating a delightful mobile experience for your customers. By designing an intuitive website, prioritizing fast page load times, and providing a smooth checkout process, you can create a seamless experience for your users. And for print companies, taking it a step further with an exceptional in-app custom-print design tool is a must to help users bring their dream creations to life. These investments can not only enhance customer satisfaction but also give your business a competitive edge in the market.

However, building a cross-platform editor from scratch can be a time-consuming and costly venture with numerous technical challenges. Fortunately, IMG.LY has developed a solution for those who want a quicker and more efficient option. Check out our CreativeEditor SDK, which enables seamless integration of a robust editor for web-to-print across desktop, mobile, and server.

If you’re interested in integrating CE.SDK, our documentation provides detailed information on how to get started, or you can try it out for free!

Thanks for reading. To stay in the loop, subscribe to our Newsletter

How to Apply Filter Effects to Video using VidEffects library on Android

Neslihan — Tue, 24 May 2022 11:41:23 GMT

Video manipulation is a complex topic on Android because there is no out-of-the-box support from the Android SDK, and using FFmpeg requires not just a very elaborate setup but having to climb a steep learning curve to learn its CLI commands.

As an aside, we have also put together how-to blogs to elaborate on other video manipulation operations like merging still images to form videos, compressing and resizing videos, or background removal.

The VidEffects library is much faster to set up than FFmpeg, and it works by applying effects to GLSurfaceView. This blog post will demonstrate how to apply Filter effects to a video in an Android app using the VidEffects library.

VidEffects does support storing videos on the disk, but it has limitations in this regard which we will discuss in a moment.

Before we begin, keep in mind that the VidEffects library differentiates between Effects and Filters. As an example, we will apply both and discuss the crucial difference between the two.

Without much further ado, let’s start coding.

Create a VidEffects Example App

Create a New Project with an Empty Activity in Android Studio. Select Kotlin as the language (or Java if you wish), and set **Minimum SDK to API 21.

You can also download the final code from GitHub.

Add Dependency

In your module-level build.gradle (app/build.gradle) file add the latest VidEffects dependency.

dependencies {
    ...

    implementation "com.github.krazykira:videffects:1.1.1"

    ...
}

Next, enable viewBinding by adding the following lines within the android{} closure.

android {
    ...
    buildFeatures {
        viewBinding true
    }
}

By using View Binding, we can avoid the findViewById() method.

Click Sync Now to sync the project with gradle files.

Add Video Asset File

Place the sample video file in app/src/main/assets/ directory. Ideally, your app will load the video file from disk – but to avoid complexity, we are placing the video in the assets directory.

Create a Layout

Create the above layout with a VideoSurfaceView, three Filter buttons, and one Reset Button.

Replace your activity_main.xml`s code with the following.

<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">
    <com.sherazkhilji.videffects.view.VideoSurfaceView
        android:id="@+id/mVideoSurfaceView"
        android:layout_width="match_parent"
        android:layout_height="300dp"
        app:layout_constraintTop_toTopOf="parent" />
    <androidx.constraintlayout.widget.ConstraintLayout
        android:id="@+id/wrapper"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@id/mVideoSurfaceView">
        <androidx.appcompat.widget.AppCompatButton
            android:id="@+id/bwButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Black and White"
            app:layout_constraintStart_toStartOf="parent"
            app:layout_constraintTop_toTopOf="parent" />
        <androidx.appcompat.widget.AppCompatButton
            android:id="@+id/grainButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Grain"
            app:layout_constraintStart_toEndOf="@id/bwButton"
            app:layout_constraintTop_toTopOf="parent" />
        <androidx.appcompat.widget.AppCompatButton
            android:id="@+id/duotoneButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="DuoTone"
            app:layout_constraintStart_toEndOf="@id/grainButton"
            app:layout_constraintTop_toTopOf="parent" />
    </androidx.constraintlayout.widget.ConstraintLayout>
    <androidx.appcompat.widget.AppCompatButton
        android:id="@+id/resetButton"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Reset"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/wrapper" />
</androidx.constraintlayout.widget.ConstraintLayout>

Initialize the video

Before initializing the video, set up an instance of the Binding class to use it within the Activity.

Setup View Binding

Add the following code in MainActivity.kt before calling setContentView().

If you do not wish to use View Binding and instead prefer using findViewById(), skip to the next step Initializing the Sample Video.

private lateinit var binding: ActivityMainBinding
class MainActivity : AppCompatActivity() {
		private lateinit var binding: ActivityMainBinding
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        ...
    }
    companion object {
        private const val TAG = "MainActivity"
    }
}

The ActivityMainBinding class is auto-generated. If it did not get auto-generated, then sync the project with the gradle file from File > Sync Project with Gradle Files.

Next, pass the root view to the setContentView().

class MainActivity : AppCompatActivity() {
    ...
    override fun onCreate(savedInstanceState: Bundle?) {
        ...
        setContentView(binding.root)
    }
}

Initializing the Sample Video

Next, declare and initialize the MediaPlayer variable.

class MainActivity : AppCompatActivity() {
    ...
    private lateinit var mMediaPlayer: MediaPlayer
        override fun onCreate(savedInstanceState: Bundle?) {
            ...
            mMediaPlayer = MediaPlayer()
    }
}

Then open the video file from our assets directory and pass it as the data source to the Media Player.

override fun onCreate(savedInstanceState: Bundle?) {
    ...
    try {
        val afd = assets.openFd("sample.mp4")
        mMediaPlayer.setDataSource(
            afd.fileDescriptor,
            afd.startOffset, afd.length
        )
    } catch (e: Exception) {
        Log.e(TAG, e.message, e)
    }
}

Now, all that is remaining is to initialize the VideoSurfaceView.

override fun onCreate(savedInstanceState: Bundle?) {
        ...
        binding.mVideoSurfaceView.init(mMediaPlayer, NoEffectFilter())
    }

Your app should run. While the buttons still do not work, we will setOnClickListeners() next.

Apply VidEffects Effects

The VidEffects library offers multiple effects, but we will be applying three: Black and White, Grain, and Duotone.

In MainActivity’s onCreate(), copy the following code to setup all the setOnClickListeners.

override fun onCreate(savedInstanceState: Bundle?) {
    ...
    //Black and White Effect
    binding.bwButton.setOnClickListener {
        binding.mVideoSurfaceView.shader = BlackAndWhiteEffect()
    }
    //Grain Filter
    binding.grainButton.setOnClickListener {
        val grainFilter = GrainFilter(10, 10)
        grainFilter.setIntensity(0.5f)
        binding.mVideoSurfaceView.filter = grainFilter
    }
    //Duotone Effect
    binding.duotoneButton.setOnClickListener {
        binding.mVideoSurfaceView.shader =
            DuotoneEffect(Color.BLUE, Color.YELLOW)
    }
    //Reset with AutoFixFilter
    binding.resetButton.setOnClickListener {
        binding.mVideoSurfaceView.filter = NoEffectFilter()
    }
}

Run the app.

Difference Between Effects and Filters

As mentioned before, Effects and Filters are two separate functions in the VidEffects library.

Effects are temporary, view-only overlays that cannot be stored on the disk. Filters can be viewed and stored on the disk. Unfortunately, a majority of the overlays offered by VidEffects are Effects and therefore only useful for playback only. At the time of writing this article, only three Filters are available.

Limitations

One of the biggest limitations of this library, as mentioned above, is that the majority of the effects provided by the library are just view-only.

Moreover, to save videos with filters, the app must target minSdk 23 at least or use FFmpeg.

Many of these Effects do not have an intensity parameter and therefore we cannot fine-tune these effects.

Alternative

If you are looking for an easy-to-integrate solution that overcomes these limitations, take a look at IMG.LY’s VideoEditor SDK. It provides more than 60 high-quality adjustable filters out-of-the-box. Moreover, you can easily add custom filters using LUT files. VideoEditor SDK works for Android SDK version 21 and above and thus targets more devices.

If the purpose is to apply effects during the playback, the VidEffects library is a solid and free open-source solution. While it handles all the complex video editing, developers will have to spend some time developing the layout.

How To Add Overlays to a Video in React Native

Antonello — Mon, 28 Mar 2022 16:23:32 GMT

In this article, you will learn to add a text and image overlay to a video with JavaScript in React Native. All you need is React Native Video, the correct styling rules, and just a few lines of code.

By following this guide, you will easily achieve a logo and text overlay:

First, let’s learn how to implement a React Native component that allows you to add an overlay image and overlay text to a video.

What is React Native Video?

Mobile apps come with several native interfaces and features. This is what makes React Native different from React, which is based on browser rendering. So, <div> becomes <View>, <p> becomes <Text>, and so on. In particular, React Native supports only a limited amount of components. You can find the entire list here.

To this day, React Native does not have a native <Video> component. That is why react-native-video was born - a widely used library without considering alternatives. This 500kB package equips you with everything you need to get started with videos in React Native.

Prerequisites

Here is the list of all the prerequisites for the demo application you are going to build:

You can add react-native-video to your project’s dependencies by launching the following npm command:

npm install react-native-video

Or if you are a yarn user:

yarn add react-native-video

Then, follow this guide to learn how to link react-native-video to your project. This step is not required for Android users with React Native 0.60 and above.

Adding Overlay Image and Text to a Video in React Native

You can clone the GitHub repository supporting this article and launch the following commands to try the demo application:

git clone https://github.com/Tonel/how-to-add-overlays-to-videos-in-react-native-imgly
cd how-to-add-overlays-to-videos-in-react-native-imgly
npm i
npm start

The project is a React Native Create App project, and you can learn more about how it works and how to run it here.

Otherwise, keep following this step-by-step tutorial and learn how to build a React Native component for adding overlays to a video.

1. Initializing a React Native Project

Create React Native App is the officially supported and easiest way to create an empty and working React Native application. Install it globally with the following command:

npm install -g create-react-native-app

Now, you can initialize a new project called react-native-video-overlays-demo with the following command:

create-react-native-app react-native-video-overlays-demo

The react-native-video-overlays-demo folder should now contain a demo project with the following file structure:

react-native-video-overlays-demo
├── .buckconfig
├── .gitattributes
├── .gitignore
├── App.js
├── app.json
├── babel.config.js
├── index.js
├── metro.config.js
├── package.json
├── package-lock.json
├── yarn.lock
├── android
│   └── ...
└── ios
    └── ...

Enter the react-native-video-overlays-demo folder:

cd react-native-video-overlays-demo

Then launch the development server with one of the following commands:

for launching an Expo app

npm start

for Android development on your phone or an emulator

npm run android

for iOS development on your phone or an emulator

npm run ios

for react-native-web development

npm run web

2. Defining a Component to Add Overlays on a Video

Let’s see how to build a VideoWithOverlays component that allows you to add overlays such as text and an image to a video. You can implement it as follows:

import React, { useEffect, useState } from 'react';
import { Image, StyleSheet, Text, View } from 'react-native';
import Video from 'react-native-video';

export default function VideoWithOverlays({
  videoComponentProps,
  text,
  imageSrc,
}) {
  const {
    video,
    overlayText,
    videoWithOverlays,
    overlayTextView,
    overlayImageView,
    overlayImage,
  } = styles;

  const [height, setHeight] = useState(0);
  const [overlayHeight, setOverlayHeight] = useState(0);

  useEffect(() => {
    // setting the overlay height to 10px starting
    // from the bottom of the video
    setOverlayHeight(height / 2 + height / 3 - 10);
  }, [height]);

  return (
    <View style={videoWithOverlays}>
      <Video
        {...videoComponentProps}
        style={video}
        onLayout={(event) => {
          const { height } = event.nativeEvent.layout;
          setHeight(height);
        }}
      />
      <View style={{ ...overlayTextView, top: overlayHeight }}>
        {text && <Text style={overlayText}>{text}</Text>}
      </View>
      <View style={{ ...overlayImageView, top: overlayHeight }}>
        {imageSrc && (
          <Image style={overlayImage} source={imageSrc} resizeMode="contain" />
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  overlayTextView: {
    position: 'relative',
    alignItems: 'flex-end',
    justifyContent: 'flex-end',
    right: 5,
  },
  overlayImageView: {
    position: 'relative',
    alignItems: 'flex-start',
    justifyContent: 'flex-end',
    left: 5,
  },
  overlayImage: {
    position: 'absolute',
    height: 40,
  },
  overlayText: {
    fontSize: 25,
    fontWeight: 'bold',
    color: 'white',
  },
  video: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
    marginTop: '50%',
  },
  videoWithOverlays: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
  },
});

The react-native-video Video component is used to play the video and its props are exposed thanks to the videoComponentProps prop. Then, after being placed into the layout, its height depending on the size of the device screen is stored in the height variable. This is used to calculate the height the overlays should be placed at in the useEffect hook. In detail, the overlays are placed at 10px from the bottom edge of the video.

The first internal View represents the optional text overlay. The overlay logic depends entirely on the custom overlayTextView and overlayText StyleSheet fields, representing the CSS classes responsible for displaying the text passed as a prop with text as an overlay. By default, the text overlay is placed in the bottom-right corner of the video.

The second internal View represents the optional image overlay. The overlay logic depends entirely on the custom overlayImageView and overlayImage StyleSheet fields, representing the CSS classes taking care of showing the image passed as a prop with imageSrc as an overlay. By default, the image overlay is placed in the bottom-left corner of the video. As you can see, both overlays are optional.

3. Putting It All Together

Let’s see the VideoWithOverlays component defined above in action. All you need to do is replace the App.js file with this:

import React from 'react';
import { StyleSheet, View } from 'react-native';
import VideoWithOverlays from './src/components/VideoWithOverlays/VideoWithOverlays';

export default function App() {
  // a static image stored into the assets folder
  const imgLy = require('./src/assets/images/IMG_LY.png');

  return (
    <View style={styles.container}>
      <VideoWithOverlays
        videoComponentProps={{
          // replace this free video source with your video
          source: {
            uri: 'https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4',
          },
        }}
        // overlay text
        text={'IMG.LY'}
        // overlay image
        imageSrc={imgLy}
      />
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#fff',
    alignItems: 'center',
    justifyContent: 'center',
  },
});

Here, <VideoWitOverlays> is initialized with a free video source retrieved from a public URL and two overlays. The text overlay states “IMG.LY” and it is placed in the bottom-right corner of the video. While the image overlay represents the IMG.LY logo and is placed in the bottom-left corner of the video. This is a screenshot of the VideoWitOverlays in action:

Final Considerations

Adding overlays elements to a video in React Native is not complex and takes just a few lines of code. On the other hand, the approach presented above is just a frontend trick and does not modify the video. You are just showing it behind text or an image, giving the illusion to the users that it is a single source of content.

If you wanted to let users modify their videos by adding overlays or other elements and then export them, this would involve complex logic and require an advanced UI. Building such a React Native component is a time-consuming task that represents a waste of energy. If that is your goal, you should take into consideration a complete and ready-to-use SDK solution, such as VideoEditorSDK. This would provide your users with the ability to add overlays and use many other cool features.

Add Overlays to a Video With React Native module for VideoEditor SDK

Visit this page from the official documentation to learn how to get started with VideoEditorSDK in React Native. Then, load your video and start editing it by adding overlays, text, and stickers. Follow the previous links to learn more about each feature.

Download our free mobile demo app from the App Store or Google Play, and try an extensive set of tools to edit videos.

Conclusion

In this tutorial, we learned how to define a React Native component that allows you to reproduce a video and add overlays. Specifically, we used react-native-video and defined everything required to place text or an image on your video. However, if you intend to export your edit, you will have to build a complex video editing application. In this scenario, you will want to consider a fully-featured, cutting-edge, and easy-to-adopt solution – such as VideoEditorSDK.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How To Resize and Compress an Image in JavaScript for Upload

Antonello — Fri, 04 Mar 2022 16:13:56 GMT

In this article, you will learn to compress an image in JavaScript and then upload it to Imgur. Similar to our previous guide on resizing images, and the one on drawing on images, you can accomplish everything with the HTML5 <canvas> element and we will involve any external libraries for this.

Smartphones cameras have become increasingly accurate and enhanced their photo quality for years. Consequently, their file size has grown as well. Since the speed of the average network has not improved at the same pace, it is essential to compress the images before uploading them.

Compressing is about downscaling an image. In other words, you want to reduce either its size or quality or both. By doing so, you can avoid uploading large images, saving the end-user time and money.

Compressing an Image With `<canvas>`

You can clone the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/how-to-compress-an-image-in-javascript-imgly

Then, you can try the demo application by opening the index.html file in your browser.

Otherwise, keep following this step-by-step tutorial and learn how to build the demo application.

1. Implementing the Compression Logic

You can compress an image by solely using the HTML <canvas> element. This is a powerful image manipulation tool that allows you to achieve many results, as we have already explored in our blog.
Now, let’s delve into how to compress an image with canvas:

function compressImage(imgToCompress, resizingFactor, quality) {
  // resizing the image
  const canvas = document.createElement('canvas');
  const context = canvas.getContext('2d');

  const originalWidth = imgToCompress.width;
  const originalHeight = imgToCompress.height;

  const canvasWidth = originalWidth * resizingFactor;
  const canvasHeight = originalHeight * resizingFactor;

  canvas.width = canvasWidth;
  canvas.height = canvasHeight;

  context.drawImage(
    imgToCompress,
    0,
    0,
    originalWidth * resizingFactor,
    originalHeight * resizingFactor
  );

  // reducing the quality of the image
  canvas.toBlob(
    (blob) => {
      if (blob) {
        // showing the compressed image
        resizedImage.src = URL.createObjectURL(resizedImageBlob);
      }
    },
    'image/jpeg',
    quality
  );
}

This function is an extension of the resizeImage() function defined in this article. So, follow the link to that tutorial to learn more about it.

What is new here are the last few lines, which take care of reducing the quality of the uploaded image based on the quality parameter. As you can see, the last part of the compressImage()is based on the toBlob() function. This transforms the image stored in the canvas into a Blob object and compresses it based on the last parameter passed to the function. This last parameter represents the quality of the target image file.

Just like resizingFactor, quality must contain a Number between 0 and 1.

Also, since the toBlob() function returns a Blob object, you can store it in a global variable. Then, you can use the blob object representing the compressed image file to upload it to your server. Let’s see how.

2. Uploading an Image to Imgur

First, you need an Imgur account. If you already have one, log in here. Otherwise, create a new account for free here.

Now, register an application here to have access to the Imgur API program. Fill out the form as follows:

Then, click on “Submit” and you should get access to this page.

Store your Client-ID in a safe place. You will need it later.
Now, you have everything required to start uploading images to your Imgur application.

Uploading an image to Imgur in JavaScript is easy and can be achieved with just a bunch of lines of code, as below:

// compressedImageBlob represents the compressed image Blob to upload
const formdata = new FormData();
formdata.append('image', compressedImageBlob);

fetch('https://api.imgur.com/3/image/', {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    Authorization: 'Client-ID YOUR_CLIENT_ID',
  },
  body: formdata,
}).then((response) => {
  if (response?.status === 403) {
    console.error('Unvalid Client-ID!');
  } else if (response?.status === 200) {
    // retrieving the URL of the image
    // just uploaded to Imgur
    response.json().then((jsonResponse) => {
      console.log(`URL: ${jsonResponse.data?.link}`);
    });
  } else {
    console.error(response);
  }
});

Replace YOUR_CLIENT_ID with the Client-ID retrieved before, and you should now be able to use this snippet to upload your images to Imgur.

3. Putting It All Together

Now it is time to see the compressImage() function in action through a simple example.

<!DOCTYPE html>
<html>
  <body>
    <h1>Compress and Resize an Image</h1>
    <p>Upload an image and compress it or use the following demo image</p>
    <input id="upload" type="file" accept="image/*" />
    <div>
      <h2>Original Image</h2>
      <img
        style="margin-top: 5px;"
        id="originalImage"
        src="demo.jpg"
        crossorigin="anonymous"
      />
    </div>
    <div style="margin-top: 5px;">
      <span>Resizing: </span>
      <input type="range" min="1" max="100" value="80" id="resizingRange" />
    </div>
    <div style="margin-top: 5px; margin-left: 8px;">
      <span>Quality: </span>
      <input type="range" min="1" max="100" value="80" id="qualityRange" />
    </div>
    <h2>Compressed Image</h2>
    <div><b>Size:</b> <span id="size"></span></div>
    <img id="compressedImage" />
    <div>
      <button id="uploadButton">Upload to Imgur</button>
    </div>
    <script src="src/index.js"></script>
  </body>
</html>

const fileInput = document.querySelector('#upload');
const originalImage = document.querySelector('#originalImage');

const compressedImage = document.querySelector('#compressedImage');
const resizingElement = document.querySelector('#resizingRange');

const qualityElement = document.querySelector('#qualityRange');
const uploadButton = document.querySelector('#uploadButton');

let compressedImageBlob;

let resizingFactor = 0.8;
let quality = 0.8;

// initializing the compressed image
compressImage(originalImage, resizingFactor, quality);

fileInput.addEventListener('change', async (e) => {
  const [file] = fileInput.files;

  // storing the original image
  originalImage.src = await fileToDataUri(file);

  // compressing the uplodaded image
  originalImage.addEventListener('load', () => {
    compressImage(originalImage, resizingFactor, quality);
  });

  return false;
});

resizingElement.oninput = (e) => {
  resizingFactor = parseInt(e.target.value) / 100;
  compressImage(originalImage, resizingFactor, quality);
};

qualityElement.oninput = (e) => {
  quality = parseInt(e.target.value) / 100;
  compressImage(originalImage, resizingFactor, quality);
};

uploadButton.onclick = () => {
  // uploading the compressed image to
  // Imgur (if present)
  if (compressedImageBlob) {
    const formdata = new FormData();
    formdata.append('image', compressedImageBlob);

    fetch('https://api.imgur.com/3/image/', {
      method: 'POST',
      headers: {
        Accept: 'application/json',
        Authorization: 'Client-ID YOUR_CLIENT_ID',
      },
      body: formdata,
    }).then((response) => {
      if (response?.status === 403) {
        alert('Unvalid Client-ID!');
      } else if (response?.status === 200) {
        // retrieving the URL of the image
        // just uploaded to Imgur
        response.json().then((jsonResponse) => {
          alert(`URL: ${jsonResponse.data?.link}`);
        });
        alert('Upload completed succesfully!');
      } else {
        console.error(response);
      }
    });
  } else {
    alert('Rezind and compressed image missing!');
  }
};

function compressImage(imgToCompress, resizingFactor, quality) {
  // showing the compressed image
  const canvas = document.createElement('canvas');
  const context = canvas.getContext('2d');

  const originalWidth = imgToCompress.width;
  const originalHeight = imgToCompress.height;

  const canvasWidth = originalWidth * resizingFactor;
  const canvasHeight = originalHeight * resizingFactor;

  canvas.width = canvasWidth;
  canvas.height = canvasHeight;

  context.drawImage(
    imgToCompress,
    0,
    0,
    originalWidth * resizingFactor,
    originalHeight * resizingFactor
  );

  // reducing the quality of the image
  canvas.toBlob(
    (blob) => {
      if (blob) {
        compressedImageBlob = blob;
        compressedImage.src = URL.createObjectURL(compressedImageBlob);
        document.querySelector('#size').innerHTML = bytesToSize(blob.size);
      }
    },
    'image/jpeg',
    quality
  );
}

function fileToDataUri(field) {
  return new Promise((resolve) => {
    const reader = new FileReader();
    reader.addEventListener('load', () => {
      resolve(reader.result);
    });
    reader.readAsDataURL(field);
  });
}

// source: https://stackoverflow.com/a/18650828
function bytesToSize(bytes) {
  var sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB'];

  if (bytes === 0) {
    return '0 Byte';
  }

  const i = parseInt(Math.floor(Math.log(bytes) / Math.log(1024)));

  return Math.round(bytes / Math.pow(1024, i), 2) + ' ' + sizes[i];
}

The input element allows users to upload an image. This is then passed to the compressImage() function along with the resizingFactor and quality values retrieved from the respective range input HTML elements. This function takes care of compressing the image, displaying it, and storing its Blob representation to the global compressedImageBlob variable. Finally, compressedImageBlob is uploaded to Imgur when the “Upload to Imgur” button is clicked.
Notice that these two snippets are what allow you to implement the live example you can find at the beginning of the article.

Final Considerations

Compressing an image in Vanilla JavaScript is easy. You can achieve this with no extra libraries and in a dozen of lines of code. At the same time, the resizing part of the process relies on an image interpolation algorithm that changes according to the browser in use. This can lead to different results based on the end-user’s browser. Also, compressing while preserving quality is always tricky and can easily become a grueling goal to achieve.
If you want to avoid this stress, consider adopting a commercial and browser-consistent solution like PhotoEditorSDK. This library provides access to a wide set of tools that allow you to manipulate your images in many ways and with an advanced and easy-to-use UI.

Resizing an Image with PhotoEditor SDK

First, read this article from the official documentation to get started with PhotoEditorSDK in HTML and JavaScript. Then, you can use the transform tool to resize your image, as shown below:

Check out this feature on the PhotoEditorSDK demo page.

Conclusion

In this article, we learned how to downscale an image in JavaScript before uploading it to your server. In detail, we resized and reduced the quality of an uploaded image before uploading it to Imgur. Everything was achieved by using only the HTML5 <canvas> element. This is a powerful tool supported by most browsers that allows you to resize and change the quality of an image with just a few lines of code. On the other hand, this process may not be browser-consistent. That is why we also introduced a commercial and more reliable solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions. To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Make Videos from Still Images with AVFoundation and Swift

Walter — Thu, 10 Feb 2022 15:35:45 GMT

Whether making a slide show or breaking up video tracks with title scenes, inserting still images into a video file can be tricky. This tutorial will show you how to convert still images into standard Quicktime video files so you can work with them further in your favorite video editor. This tutorial was created and tested with Xcode 13 and Swift 5. This tutorial will discuss two strategies:

convert each image to a separate movie and stitch them together later (this is best suited for times you want to interleave the images and other video files using some other editing workflow)
create a “blank” video and use the applyingCIFiltersWithHandler variation to overlay the images (this is best suited for times you want to make a quick slide show with no further editing)

As with most AVFoundation code, the Xcode simulator is not always the best platform for running the code. Test on an actual device if you can. A demo project with this code (and some extra goodies) can be found on GitHub.

The Problem with an Image

The basic models that are used in AVFoundation for making video files are AVAsset and AVAssetTrack. You can compose multiple tracks and assets together to make a single video file. All tracks must have some media data and a duration. Static image files don’t have a duration. This is the primary problem you have to overcome when you want to add static images to video.
Fortunately, you can use many of the same AVFoundation tools that are used for video capture and frame manipulation. However instead of using a CADisplayLink and AVPlayerItemVideoOutput or an AVCaptureDevice to provide frame buffers to save, you will create the frame buffers manually. Then you can use AVAssetWriter to convert the frame buffers into a video file.

Some items to consider before you actually generate the video file are:

What framerate to use? Because the image doesn’t have any motion, you can use a very low frame rate to create smaller file sizes. However, this might cause issues if you are interleaving it with regular video, which usually has a frame rate of 30 or 60 fps (or higher, for slow motion video)
What dimensions will the final video have? It is easy to resize an image and add padding using CoreImage or some image editing software before it becomes a video. As the image becomes a video and moves through different steps in an AVFoundation pipeline, the different classes in AVFoundation will resize or crop the images in different ways when the input dimensions and output dimension don’t match.

Writing Buffers using `AVAssetWriter`

The AVAssetWriter exists to encode media to a file on disk. Though this tutorial will use a single video input AVAssetWriter supports multiple inputs of different kinds (audio, video, metadata). It is quite similar to AVAssetExportSession. The difference is that AVAssetWriter uses multiple inputs and each input is a single track where the AVAssetExportSession takes a single AVAsset as an input. The result of both is a single file.

As indicated above, our basic strategy will be:

Create a pixel buffer that is the correct color space and size to hold the image
Render the image into the pixel buffer
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop, and each time through the loop, append the pixel buffer
Clean up

Create the Pixel Buffer

The code below creates a CIImage from an image file. Then it creates a pixel buffer the same size as the image and uses a standard color space. Finally, a CIContext renders the image into the pixel buffer.

//create a CIImage
guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
}
//create a variable to hold the pixelBuffer
var pixelBuffer: CVPixelBuffer?
//set some standard attributes
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
     kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
//create the width and height of the buffer to match the image
let width:Int = Int(staticImage.extent.size.width)
let height:Int = Int(staticImage.extent.size.height)
//create a buffer (notice it uses an in/out parameter for the pixelBuffer variable)
CVPixelBufferCreate(kCFAllocatorDefault,
                    width,
                    height,
                    kCVPixelFormatType_32BGRA,
                    attrs,
                    &pixelBuffer)
//create a CIContext
let context = CIContext()
//use the context to render the image into the pixelBuffer
context.render(staticImage, to: pixelBuffer!)

Though we generally think of CIImage as an image, Apple is clear that it is not an image by itself. CIImage needs a context for rendering. This is where a lot of the power of CoreImage and filters and GPU rendering come from, the fact that CIImage is just the instructions for creating an image. Note: You can also use CGImage and the CoreGraphics framework to create pixel buffers. I find it easier to use the CoreImage framework.

Configure AVAssetWriter

Now with a buffer created, you can configure the AVAssetWriter. It will take several parameters as a dictionary to determine the dimensions and format of the outputs. You can either set them individually or else Apple provides presets.
One of the most important settings is the output dimension. If the output dimension matches the original image, the video file will show no distortion or letterboxing. However, if the output is smaller, it will compress the image until it fits. If the output is larger, the image will expand until its width or height matches the output size and then will letterbox. Note: if an image expands too much, it will appear grainy. The image below shows different output sizes for a 640 x 480 input image.

To create your settings for the AVAssetWriter first create a dictionary containing the different values. The example here sets an output dimension of 400 x 400 and provides for .h264 encoding.

let assetWriterSettings = [AVVideoCodecKey: AVVideoCodecType.h264, AVVideoWidthKey : 400, AVVideoHeightKey: 400] as [String : Any]

To use one of the presets, use the AVOutputSettingsAssistant. You can read about the different settings in the Apple documentation. An example to create 1080p video output would look like this:

let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings

Write the pixel buffer to the video file

With the settings configured, the asset writer can loop through and append the contents of the pixel buffer to create each frame of the video. Something important to notice in the code below is that AVFoundation has a philosophy to preserve data. So, it will make copies, and it will generally not overwrite files. That is why you have to delete any old files before you can create the new AVAssetWriter.

//generate a file url to store the video. some_image.jpg becomes some_image.mov
guard let imageNameRoot = imageName.split(separator: ".").first, let outputMovieURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent("\(imageNameRoot).mov") else {
  throw ConstructionError.invalidURL //an error i made up
}
//delete any old file
do {
  try FileManager.default.removeItem(at: outputMovieURL)
} catch {
  print("Could not remove file \(error.localizedDescription)")
}
//create an assetwriter instance
guard let assetwriter = try? AVAssetWriter(outputURL: outputMovieURL, fileType: .mov) else {
  abort()
}
//generate 1080p settings
let settingsAssistant = AVOutputSettingsAssistant(preset: .preset1920x1080)?.videoSettings
//create a single video input
let assetWriterInput = AVAssetWriterInput(mediaType: .video, outputSettings: settingsAssistant)
//create an adaptor for the pixel buffer
let assetWriterAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: assetWriterInput, sourcePixelBufferAttributes: nil)
//add the input to the asset writer
assetwriter.add(assetWriterInput)
//begin the session
assetwriter.startWriting()
assetwriter.startSession(atSourceTime: CMTime.zero)
//determine how many frames we need to generate
let framesPerSecond = 30
//duration is the number of seconds for the final video
let totalFrames = duration * framesPerSecond
var frameCount = 0
while frameCount < totalFrames {
  if assetWriterInput.isReadyForMoreMediaData {
    let frameTime = CMTimeMake(value: Int64(frameCount), timescale: Int32(framesPerSecond))
    //append the contents of the pixelBuffer at the correct time
    assetWriterAdaptor.append(pixelBuffer!, withPresentationTime: frameTime)
    frameCount+=1
  }
}
//close everything
assetWriterInput.markAsFinished()
assetwriter.finishWriting {
  pixelBuffer = nil
  //outputMovieURL now has the video
  Logger().info("Finished video location: \(outputMovieURL)")
}
}

Now, your image has become a Quicktime video of whatever duration you specified and is ready for use as any other video file! You can now continue with the like of VideoEditor SDK to trim videos, and add filters or overlays. You might also want to combine the video with other video files to make a new creation.

Say hi to my dog! A plain video from still images

Creating a Blank Video

The previous example created a single .mov file for each of your images. The most robust way to get these images into a video with sound, transitions, etc., is to use AVMutableComposition with multiple tracks and layer instructions. However, for a quick slideshow with just a few elementary transitions, a strategy is to create a blank movie and then use AVVideoComposition with (asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void). That will let you use CIImage and CIFilter to paint each frame of the blank movie with images.

Our basic strategy for this method will be:

Create a pixel buffer that is the right color space and size and fill it with a solid color
Create an AVAssetWriter with a single video input
Decide how many frames will be required to create a video of the desired duration
Make a loop and each time through the loop append the pixel buffer
Create an AVVideoComposition to add the images to the individual frames of the video
Let AVPlayer combine the video and the composition

In the first strategy, you started by making a pixel buffer and using an AVAssetWriter to create the video file using this code:

guard let uikitImage = UIImage(named: imageName), var staticImage = CIImage(image: uikitImage) else {
  throw ConstructionError.invalidImage //this is an error type I made up
    }

However, this time, instead of using a source image, you create a CIImage that is a single color.

let staticImage = CIImage(color: bgColor).cropped(to: CGRect(x: 0, y: 0, width: 960, height: 540))

The CIImage(color:) initializer creates an image that is of infinite size and is just a single color. Using the .cropped(to:) modifier lets you make it the same size as the AVAssetWriter output so that there won’t be any letterboxing. The rest of the code is the same as before and the end result will be a video file that is just the single color.

Adding a Composition

Once the video file has been created and saved to disk, load it as an asset, then create an AVVideoComposition. This allows you to generate individual frames. The function below uses the request property of the composition. This has a CIImage representation of the current frame as well as the timestamp of when this frame will appear. The .fetchSlide(forTime:) is a helper function in the demo app that returns the appropriate CIImage for the slide to display. Then the .sourceOverCompositing filter combines the original frame image with the slide. Using a CGAffineTransform moves the slide across the screen as the video plays.

func createComposition(_ asset: AVAsset) {
  let slideshowComposition = AVVideoComposition(asset: asset) {[weak self] request in
    guard let self = self else { return }
    let slide = self.fetchSlide(forTime: request.compositionTime)
    let compose = CIFilter.sourceOverCompositing() //filter to join two images
    compose.backgroundImage = request.sourceImage
    compose.inputImage = slide?.transformed(by: CGAffineTransform(translationX: request.compositionTime.seconds * 50, y: 0))
    //always finish with the last output of the pipeline
    request.finish(with: compose.outputImage!, context: nil)
  }
  self.outputSlideshow = slideshowComposition
}

Because the creation of the video is done asynchronously, you can composite images over the original frame and add filters without the danger of impacting the final frame rate. Recall that CIFilter works as a pipeline, each CIFilter has an .outputImage and most have .inputImage and some configuration settings. Feeding the output from one filter to the input of the next will let you build complex compositions. Once a composition has been created, AVPlayer can join the original video with the composition for playback or export.

let item = AVPlayerItem(asset: self.outputMovie!)
item.videoComposition = outputSlideshow
self.player = AVPlayer(playerItem: item)

The code above might generate a video like this one.

As with the first video, the final video is a plain .mov file.

Going Further

In this tutorial, you saw two ways to convert static images into .mov files that you can then edit with any video editor. The sample project also has some code for exporting your creation as a .mov file and code for adding sound to the creation.

These are not the only ways to go about solving this problem. For example, instead of creating a blank video and overlaying the images, using a stock video of waves at the beach and overlaying the images might work better for your project. Conversely, because the initial pixel buffer is created from a CIImage, there is no reason that the CIImage cannot be the end of a long pipeline of CIFilters. This way the composition is done at the beginning, and the AVAssetWriter is the only tool needed. The best strategy is the one that makes sense to you and that works with the kinds of media you have.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

Looking to integrate video capabilities into your app? Check out our Video Editor SDK, Short Video Creation, and Camera SDK!

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Add a Filter to a Video Stream in iOS

Walter — Mon, 08 Nov 2021 15:33:55 GMT

Using filters and effects with video streams requires different strategies than applying them to files. In this tutorial, you will see how to apply effects and filters to video streams in real-time. The code in this tutorial compiles with Xcode 13.

As with most AVFoundation code, the Xcode simulator is not the best platform for running the code. Test on an actual device. A project with code that supports this tutorial can be found on GitHub.

Streams or Files

For video files stored on the device, you can use an AVMutableVideoComposition to apply filters. When streaming video from a remote location, this strategy will not work. The AVMutuableVideoComposition classes are not designed to work in real-time with streams. Apple provides a way to extract the pixels from the stream at regular intervals. You can manipulate the pixels and then render them to the screen. An AVPlayerItemVideoOutput object gives access to the pixels that make up a video frame. Apple also supplies a CADisplayLink object to ensure you are extracting the right pixels at the right time. The display link is a specialized timer that will fire in sync with the redraw rate of the current display.

So, to filter a video stream, the strategy becomes:

Set up an AVPlayer as normal and assign it the URL of the stream
Attach an AVPlayerItemVideoOutput object to the stream
Attach a CADisplayLink object to the run loop
Extract the current pixel buffer from the AVPlayerItem every time the screen redraws
Convert the pixel buffer to a CoreImage object
Apply filters
Display the image

Setting up AVPlayer

You won’t use the AVPlayerLayer or AVPlayerViewController to display the video, but the AVPlayer plays a central role. The AVPlayer keeps the video in sync with the audio, handles decoding whatever format the stream uses, controls buffering, and more. The basic setup is the same as with any other project:

//create a player
let videoItem = AVPlayerItem(url: streamURL)
self.player = AVPlayer(playerItem: videoItem)

The URL can either point to a local resource or a stream.

Using AVPlayerItemVideoOutput

The AVPlayerItemVideoOutput allows you to query the AVPlayerItem for the pixel buffer (one screen worth of pixels) at any given time. You can specify different pixel formats and other options for the output. For this tutorial, you will you can create the object with a simple instantiation using the default formats.

let playerItemVideoOutput = AVPlayerItemVideoOutput()

Later you will add this video output to your player item. Then you can ask it to generate the pixel buffers as needed. In this tutorial, you will ask for the pixel buffer of the current frame. You could ask for the frame for any timestamp though.

Creating the Display Link

The CADisplayLink class is a specialized NSTimer that synchronizes itself to the screen refresh rate of the display. This ensures that the pixel buffer you extract will be from the correct time of your video stream. It will get rendered at the correct time in the screen refresh. Syncing a standard NSTimer to the screen refresh has always been problematic. With the new variable refresh rate screens that Apple makes, it becomes impossible. However, CADisplayLink adjusts its rate as the screen refresh rate changes. Apple explains how this works in the 2021 WWDC Session Optimize for Variable Refresh Rate Displays. When you create the display link, you give it the name of a function to call each time it executes. A sample for creating a link could look like this:

lazy var displayLink: CADisplayLink = CADisplayLink(target: self,
                                                  selector: #selector(displayLinkFired(link:)))

Elsewhere in the code, a function called displayLinkFired(link: CADisplayLink) extracts the pixel buffer from the AVPlayerItem.

Starting the Player

Apple notes that loading a video file takes a measurable amount of time. It can take even longer when the video file is streaming across a network. AVFoundation allows your program to continue to execute while the setup steps and initial buffering are occurring in the background. You can run into issues if you load a video into AVFoundation and then immediately attempt to work with it. AVFoundation may not show an error or status message, but it will not perform as expected either.

The most important step in this entire process is using an observer to wait until the AVPlayerItem has a .readyToPlay status before attaching the output and displaying link objects. In the example below statusOberserver, player, playerItemVideoOutput and displayLink are all declared at the class level. They need to persist the entire time the video is being used.

//create a player
let videoItem = AVPlayerItem(url: streamURL!)
self.player = AVPlayer(playerItem: videoItem) //1
//*important* add the display link and the output only after it is ready to play
self.statusObserver = videoItem.observe(\.status, //2
                       options: [.new, .old],
                 changeHandler: { playerItem, change in
    if playerItem.status == .readyToPlay { //3
      playerItem.add(self.playerItemVideoOutput)
      self.displayLink.add(to: .main, forMode: .common)
      self.player?.play() //4
    }
})

Here is what to notice about the code above

Creating the player will take a large amount of time, but the program execution will not stop and wait
NSKeyValueObservation is how swift defines key value observers the code will execute every time the videoItem.status property changes values
Checks to see if the .status property is .readyToPlay
After adding the display link and the video output objects start playing the video

Extracting the Pixel Buffer

Once the video begins to play, the display link function gets called for each screen refresh. Here is an example of how to extract the pixel buffer and render it to a UIImageView.

@objc func displayLinkFired(link: CADisplayLink) { //1
  let currentTime = playerItemVideoOutput.itemTime(forHostTime: CACurrentMediaTime())
  if playerItemVideoOutput.hasNewPixelBuffer(forItemTime: currentTime) { //2
    if let buffer = playerItemVideoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
      let frameImage = CIImage(cvImageBuffer: buffer) //3
      //4
      let pixelate = CIFilter(name: "CIPixellate")!
      pixelate.setValue(frameImage, forKey: kCIInputImageKey)
      pixelate.setValue(self.filterSlider.value, forKey: kCIInputScaleKey)
      pixelate.setValue(CIVector(x: frameImage.extent.midX, y: frameImage.extent.midY), forKey: kCIInputCenterKey)
     let newFrame = pixelate.outputImage!.cropped(to: frameImage.extent)
     self.videoView.image = UIImage(ciImage: newFrame) //5
    }
  }
}

Here is what the code is doing:

Marks the func with @objc so that it works with the display link selector
Asks if there is a new pixel buffer to display. Depending on the refresh rate of the screen and the frame rate of the video, there will not always be a new buffer.
After extracting the pixel buffer, convert it to a CoreImage object
Apply any filters. This example applies the CIPixellate filter. It uses a slider to let the user change the intensity as the video plays
Convert the filtered image to a UIImage and assign it to the UIImageView

Now as the pixellate scale changes the image will be filtered but the video will continue to play smoothly.

Going Further

Using the strategy above you should be able to apply different filters to the video stream in real-time. Remember that you only have a few milliseconds though, then the video will begin to stutter. CoreImage filters are optimized to make use of the GPU and should be fast enough in most cases. For better performance, you can render the filtered image to an MTKView and use Metal. Apple has also begun to create Metal Performance Shaders which are even more efficient than CoreImage filters when used with a MetalKit view. As of now, there are many more CoreImage filters, so that may give you the most flexibility.
The strategy in this tutorial is adequate for filtering video streams. If you want to give your users more flexibility with filters and other video tweaks, an editor such as VideoEditorSDK can let you focus on your application’s core functions and let a team of professionals worry about the video. Using VideoEditorSDK your users can apply filters to video as well as text annotations and more.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions or suggestions.

You like what we do? Check our careers page. Even if you can’t find your role listed, we might be interested since you are already here!

How To Record Screen Video Using ReplayKit for iOS

Walter — Tue, 21 Sep 2021 15:05:08 GMT

In this article, you will see how to use Swift and ReplayKit to add screen recording to your application. You will also see how to use the new rolling clips feature, introduced in iOS 15. The code in this article uses Swift 5 and Xcode 13. Clone this repository for a sample project and example code.

ReplayKit Basics

Apple introduced the ReplayKit framework in 2015 with iOS 9. They have refined and expanded it since then, but it still has a small API. ReplayKit uses the same shared components as AirPlay and the device screen recorder (accessible from the device Control Center). For this reason, ReplayKit functions will not work when using AirPlay or playing video. Also, the screen recording functions do not work on the Xcode simulators. You should test using a physical device.

Your app makes all its calls to the RPScreenRecorder.shared() object. This is a singleton object for your app that ReplayKit provides. There is no need to store it in a variable or pass it around in your code.

Rolling Clip Recording

In iOS 15, Apple introduced rolling clip recording. ReplayKit will maintain an updated video buffer of the most recent fifteen seconds. Your app can request the video at any time. ReplayKit comes from GameCenter. Apple’s idea for this feature is for creating a short, sharable video of the moments before an exciting event in a game. Another use can be for a user to record the steps leading up to a defect during testing. When the user is unlikely to know in advance that they will want a screen recording, rolling clips may be a good solution.

To start rolling clip buffering, include a method such as this one

RPScreenRecorder.shared().startClipBuffering { err in
  if let err = err {
    //either the user denied permission or something like
    //AVPlayer or AirPlay is using the shared recorder resources.
    print("Error attempting to start buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering started.")
  }
}

Screen recording has privacy implications. ReplayKit presents the user with a modal, requesting to record before it starts. If the user denies the request or if the buffering is unable to start, ReplayKit passes an Error object to the closure. Unlike other privacy modals, you do not have an opportunity to add your own language to the message. The modal appears as soon as you execute the .startClipBuffering command, so be sure that the user is not surprised by it. Execute the command at a logical transition in your application. After the user grants permissions, the modal will not appear again unless eight minutes pass before the next call to .startClipBuffering.

While the clip buffering runs, at any time, you can save the buffer to a URL on the device. The clip you save can be any duration up to fifteen seconds. If the recorder is unable to write the clip to disk, it will send an error.

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().exportClip(to: clipURL, duration: TimeInterval(15)) { err in
  if let err = err {
    print("An error? \(err.localizedDescription )")
  } else {
    print("Clip saved to url \(clipURL)")
}

The code above stores a fifteen-second clip at a URL in the user’s temporary directory. The temporary directory gets cleared by the system, but not while the app is running. To ensure that the files are not deleted except by the user, store them in the user’s documents directory instead.
When your app is done collecting video, use .stopClipBuffering to allow the system to clean up resources and stop the recording.

RPScreenRecorder.shared().stopClipBuffering { err in
  if let err = err {
    print("Error attempting to stop buffering \(err.localizedDescription)")
  } else {
    print("Clip buffering stopped.")
  }
}

Recording Screen and App Audio

Rolling clip buffering runs in the background. You can also allow the user to control when to start and stop screen recording. The API looks similar and is available since iOS 10.

RPScreenRecorder.shared().startRecording { err in
  guard err == nil else { print(err.debugDescription); return }
  //code to display recording indicator
}

As with the rolling clip buffering, the system will display a permissions dialog. If the user denies permission you will get an error and the recording will not start.

The same eight-minute rule applies as for rolling clip recording. But, since the user has just taken some action to start recording, they should expect to see this modal.
Apple provides a basic view controller for editing and sharing the video.

ReplayKit passes the RPPreviewViewController to the .stopRecording handler.

RPScreenRecorder.shared().stopRecording { preview, err in
  guard let preview = preview else { print("no preview window"); return }
  //update recording controls
  preview.modalPresentationStyle = .overFullScreen
  preview.previewControllerDelegate = self
  self.present(preview, animated: true)
}

The preview object above is the RPPreviewViewController editor. You can display it as a modal or as a popover depending on your needs. The Apple provided editor will allow the user to save to their camera roll or share the clip after editing.

Notice in the code above that there is a previewControllerDelegate. You will need to add some code to the delegate to cleanup and dismiss the editor when the user is done sharing or saving. An example is below.

func previewControllerDidFinish(_ previewController: RPPreviewViewController) {
  previewController.dismiss(animated: true) { [weak self] in
  //reset the UI and show the recording controls
  }
}

Using Many `UIWindow` Objects

The ReplayKit recording will only capture the main window of your application. When designing for screen recording, you can place any UI elements in a separate UIWindow object. Although applications typically only have one UIWindow, iOS allows many windows. A UIWindow requires a .rootViewController to provide content. Then use it like any other view in your application.
The UI for this application consists of three UIWindow objects.

Apple places the status bar in a separate window from your application. The recording controls are in a separate window from the main application and will not appear in the screen recording.

To add some recording controls or a recording indicator to your app. Create a new UIWindow and add it to the same windowScene as your view.

controlsWindow = UIWindow(frame: CGRect(x: 0, y: 0, width: view.frame.width, height: 45 + view.safeAreaInsets.top)) //1
controlsWindow?.windowScene = view.window?.windowScene //2
controlsWindow?.makeKeyAndVisible() //3
let recordingIndicator = UIButton.systemButton(with: UIImage(systemName: "record.circle")!, target: self, action: #selector(recordingToggled(_:))) //4
let vc = UIViewController()
controlsWindow?.rootViewController = vc //5
vc.view.addSubview(recordingIndicator) //6
recordingIndicator.center = CGPoint(x: vc.view.center.x, y: vc.view.center.y + 20)

The code above creates the red/blue recording button and banner at the top of the view. It relies on a controlsWindow variable created at the class level. Here is what it is doing:

Creates a new UIWindow that is the width of the view and tall enough for the button and the safe area (the area around the notch and status bar)
Adds the window to the same windowScene as the current view
Makes the window visible (UIWindow objects are hidden when created) and brings it to the front of the stack of windows
Creates a button and links it to a method in your class recordingToggled(_:)
Adds a view controller to the window so that there will be some content
Adds the button to the view controller

Because the recording indicator and button are in a separate window, they will not appear in any screen recording. Be mindful when showing modal views or alert views, as they may appear behind the extra window. A good strategy is to hide the extra window when showing these views. You can use a similar strategy to hide any other UI elements that should not appear in screen recordings.

Going Further

Though Apple provides a basic editor, you can provide your own editing tools. The examples above showed the rolling clip editor writes the video to the disk. The regular recorder can also write the video to disk. You can stop recording using this code:

let clipURL: URL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("\(UUID().description).mov")
RPScreenRecorder.shared().stopRecording(withOutput: clipURL) { err in
  //check for an error and process the url
}

This method would replace the .stopRecording { preview, err in method shown above.
With the video saved, you can use an editor such as the VideoEditor SDK to allow your users to add annotations, text and filters to their clips. They can even add audio or combine clips to make a great creation.

Wrapping Up

In this article, you saw how to capture a rolling screen recording as well as how to capture the screen manually. Further, you saw how using an SDK such as VideoEditor SDK allows you to annotate and enhance your clips before sharing.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions!

How to Trim and Crop Video in Swift

Walter — Tue, 31 Aug 2021 16:03:25 GMT

In this article, you will see how to use Swift and AVKit to crop a video clip and how to trim a video’s timeline. Then you will learn how to use AVAssetExportSession to write your edited video to disk. The code in this article uses Swift 5 and Xcode 12.5. Clone this repository for a sample project and example code.

Anatomy of a Video File

The AVFoundation and AVKit frameworks are what Swift and iOS use to manage audio and video. When starting out, you may use an AVPlayerViewController for playback with the same controls and features as the native players. You can also use an AVPlayer object and provide your own playback and editing controls. Whichever you choose, you begin by loading a media file (.mp3, .mov, etc.) into an AVPlayerItem. Inside the AVPlayerItem, the media becomes an AVAsset which may have many tracks of video, audio, text, closed captions etc.

To manipulate an AVAsset, Apple provides the AVComposition class. The AVComposition can act on a single track or many tracks to filter and mix underlying media and produces a single output. An AVPlayer shows the output of an AVComposition on a device. When it is time to export, you can use an AVAssetExportSession with the same AVComposition to write to disk or upload to a server.

Cropping Video to a Rectangle

Apple provides some different AVComposition classes optimized for common tasks. Apple recommends using one of these classes instead of writing custom AVComposition classes whenever possible. The reason for this is that when Apple introduces new technologies (like HDR Video), they will ensure it works with their classes. If you’ve written your own, you will have to update it to work with the new technologies. In this article you will crop the video to a rectangle and let the audio pass through as recorded. A good class to use for this task will be the AVMutableVideoComposition with the init(asset: AVAsset, applyingCIFiltersWithHandler applier: @escaping (AVAsynchronousCIImageFilteringRequest) -> Void) initializer. This will allow you to apply a CIFilter to each frame of the video. Apple provides an efficient cropping filter called CICrop.

func transformVideo(item: AVPlayerItem, cropRect: CGRect) {

  let cropScaleComposition = AVMutableVideoComposition(asset: item.asset, applyingCIFiltersWithHandler: {request in

    let cropFilter = CIFilter(name: "CICrop")! //1
    cropFilter.setValue(request.sourceImage, forKey: kCIInputImageKey) //2
    cropFilter.setValue(CIVector(cgRect: cropRect), forKey: "inputRectangle")


    let imageAtOrigin = cropFilter.outputImage!.transformed(by: CGAffineTransform(translationX: -cropRect.origin.x, y: -cropRect.origin.y)) //3

    request.finish(with: imageAtOrigin, context: nil) //4
    })

  cropScaleComposition.renderSize = cropRect.size //5
  item.videoComposition = cropScaleComposition  //6
}

The applyingCIFiltersWithHandler will execute for every frame in the asset of the AVPlayerItem. Here is what the code above will do:

Create a CICrop filter (note that this demo uses ! to force unwrap, production code should handle failure)
Add the .sourceImage (a CIImage) from the request to the filter.
Move the cropped image to the origin of the video frame. When you resize the frame (step 4) it will resize from the origin.
Output the transformed frame image
Set the size of the video frame to the cropped size
Attach the composition to the videoComposition property of the asset

Notice that the transformation does not alter the underlying asset. It creates a videoComposition filter to attach to the item during playback. After the transformation, AVPlayer will display the new creation when executing its .play() function.

Sharp eyed readers will have noticed that the init method mentions “applying CIFilters” plural. Instead of one filter, you can create an entire pipeline of CIFilter objects to manipulate the visual properties of the frames. The request object also contains the compositionTime so filters can change at different parts of the video.

Trimming the Time of a Video

When an AVPlayer loads an AVItem the start time of the video will be at CMTime.zero. The end will be the duration property of the AVItem. To adjust the playback times, you call the .seek function to move to the new start time and set the forwardPlaybackEndTime to the new end time. Now the player will only play the part of the clip between those times.

//The player object is already created and configured to play video in the ViewController

//load a video .mov file
self.player = AVPlayer(url: Bundle.main.url(forResource: "grocery-train", withExtension: "mov")!)

//Set the start time to 5 seconds.
let startTime = CMTimeMakeWithSeconds(5, preferredTimescale: 600)

//Convert the duration of the video to seconds
let videoDurationInSeconds = self.player!.currentItem!.duration.seconds

//Subtract 5 seconds from the end time
let endTime = CMTimeMakeWithSeconds(videoDurationInSeconds - 5, preferredTimescale: 600)

//Assign the new values to the start and end time
self.player?.seek(to: startTime)
self.player?.currentItem?.forwardPlaybackEndTime = endTime

//Play the video
self.player?.play()

The CMTime (Core Media Time) object uses timescales and Int values to map to the individual frames of tracks. Video tracks commonly come to your app with 24, 30, 60 or 120 frames per second. Converting between these different formats using Floats or Doubles would be imprecise. When converting seconds to CMTime with video, 600 is the commonly used timescale because it is a common multiple of all of the standard frames per second.

Exporting the Video

As with the videoComposition property, setting the start and end times of the player only modifies the playback of the video clip. The underlying asset remains unchanged. When you use the AVAssetExportSession, the new video file will have the change. The export session applies time range and video composition objects as it writes the file. A function to export might look like:

func export(_ asset: AVAsset, to outputMovieURL: URL, startTime: CMTime, endTime: CMTime, composition: AVVideoComposition) {

    //Create trim range
  let timeRange = CMTimeRangeFromTimeToTime(start: startTime, end: endTime)

    //delete any old file
  do {
    try FileManager.default.removeItem(at: outputMovieURL)
  } catch {
    print("Could not remove file \(error.localizedDescription)")
  }

    //create exporter
  let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality)

    //configure exporter
  exporter?.videoComposition = composition
  exporter?.outputURL = outputMovieURL
  exporter?.outputFileType = .mov
  exporter?.timeRange = timeRange

    //export!
  exporter?.exportAsynchronously(completionHandler: { [weak exporter] in
    DispatchQueue.main.async {
      if let error = exporter?.error {
        print("failed \(error.localizedDescription)")
      } else {
        print("Video saved to \(outputMovieURL)")
      }
    }
  })
}

Going Further

AVKit and AVFoundation provide simple objects for manipulating video files. The difficulty when working with video and audio tracks usually comes while providing editing controls for a user. The image below shows how the original video dimensions might change from creation to display.

The original 2160x3840 video appears on an iPhone in a 357x635 frame. Additionally the origin point (green dot) of the video file and the origin point of the UIViews are not equal. Passing the frame of the red, cropping rectangle to an AVComposition would not work as expected. Inside of the AVComposition the video resumes it’s 720x1280 dimensions while the 250x250 cropping rectangle would remain 250x250.

Before using a UIView rectangle with an AVComposition it needs to resize and the origin point needs to align to the video’s origin. Your app must apply a CGAffineTransform to reorient the origin points.

var cropRect = self.croppingView.frame //1
let originFlipTransform = CGAffineTransform(scaleX: 1, y: -1)
let frameTranslateTransform = CGAffineTransform(translationX: 0, y: renderingSize.height)
cropRect = cropRect.applying(originFlipTransform) //2
cropRect = cropRect.applying(frameTranslateTransform) //3

Now you apply regular ratio math to the dimensions of the cropping rectangle to match the video dimensions.

let renderingSize = playerItem.presentationSize

let xFactor = renderingSize.width / playerView.bounds.size.width
let yFactor = renderingSize.height / playerView.bounds.size.height

let newX = croppingView.frame.origin.x * xFactor
let newW = croppingView.frame.width * xFactor
let newY = croppingView.frame.origin.y * yFactor
let newH = croppingView.frame.height * yFactor
var cropRect = CGRect(x: newX, y: newY, width: newW, height: newH)

The transformations above are standard when working with AVKit and UIKit or SwiftUI in an app. The math is not complex, but it is tedious. Unless your app is a custom video editing application, you may find that using a commercial solution such as VideoEditorSDK is a better approach. By adding the VideoEditorSDK, your app can have professional appearing trim and crop controls as well as filtering, text and more.

Wrapping Up

In this article, you saw how to crop a video image and how to trim time from a track, all in Swift. Further, you saw how using an SDK such as VideoEditorSDK allows you to provide full-featured video editing for any application.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How to Create Image Filters for iOS

Walter — Tue, 24 Aug 2021 08:09:37 GMT

In this tutorial, you will learn how to write a custom CIFilter in Metal and Swift. You can use this filter in any CoreImage pipeline.

Apple offers over 200 image filters in the CoreImage framework, but sometimes you need a little extra to make your images perfect. Apple provides two ways to create customize image filters. You can chain CIFilters together in a CIFilter subclass or wrap a CIKernel in a CIFilter subclass. Either method creates a filter that CoreImage can execute on the GPU. That makes the filter fast. CoreImage filters can filter live video without impacting the frame rate. Before iOS 11 the only way to write a custom CIKernel was to pass a string containing the code to the GPU at runtime using the OpenGL Shading Language. This had two drawbacks: it was difficult to find errors in the string of commands and the kernel was not compiled until runtime. This tutorial will show you how to add a custom filter with a Metal-based CIKernel to any Swift project in Xcode. The code samples in this tutorial use Xcode 12.5 and Swift 5.

Clone this repository for a sample project and example code that supports this tutorial. The repo also contains some other custom filters, demonstrating other aspects of Metal filters.

Adding Metal Support to an Xcode Project

The first step is to add a Metal source code file to your project.

Find the “Metal File” template, select it and click Next. Name the file and save it. At compile time, Xcode combines all of the .metal files in your project into a single .metallib file. So, organize your Metal code into lots of files or just one file as you prefer. The last step before you start writing code is to add compiler flags for CIKernel objects.

In the Build Settings, find the other Metal Compiler Flags and add an entry of -fcikernel.

Then find the Other Metal Linker Flags and add an entry of -cikernel.

Important note: any project without .metal files hides the Metal Compiler and Linker sections from build Settings.

Setting Up a Metal File for CoreImage Kernels

Metal is a general-purpose technology for writing code that will execute on the GPU. The compiler flags tell Metal that you will be writing code to work with CoreImage. With a Metal source file in your project and the compiler flags set, you’re finally ready to write some code! Open the Metal file you created above. It should be empty except for

#include <metal_stdlib>
using namespace metal;

Below these lines, import the CoreImage headers by adding:

#include <CoreImage/CoreImage.h>

and add a stub for your filter code with:

extern "C" {
  namespace coreimage {
    // KERNEL GOES HERE
  }
}

Prefix your kernel code with extern "C" and the CoreImage namespace. You write kernel code in the Metal Shader Language, which is a variation of C. If you only know Swift, you should be able to rely on Xcode’s code-completion and symbol lookup to help you as you’re learning to write Metal code. In addition to the general language reference, Apple provides a Metal Shading Language for Core Image Kernels document.

The example below creates a CIColorKernel. This kernel is optimized for changing the color value of each pixel in an image. It will change each pixel to a grayscale version of itself. A filter applying his kernel will send the color value of each pixel of the image to the kernel.

float4 grayscaleFilterKernel(sample_t s) { //1
  float gray = (s.r + s.g + s.b) / 3;      //2
  return float4(gray, gray, gray, s.a);    //3
}

Here is how this kernel works:

The filter provides the color of the current pixel to the kernel as a sample_t type. The kernel will return the new color for that pixel as a float4 type. The sample_t type is equivalent to a float4 type and is only named differently because of historical convention. A float4 type contains four float values. In the case of color, they are the red, green, blue and alpha values for the pixel. Each float has a value between zero and 1.
Calculate a grayscale version of the pixel by averaging the three color channels.
Create a new float4 with the averaged value for the three color channels and the original alpha value for the fourth.

CoreImage also provides a CIWarpKernel class that is optimized for changing the position of each pixel and a CIBlendKernel for blending two images. For more complex tasks, CoreImage also provides a general CIKernel. Apple suggests that chaining together optimized kernels in a pipeline is usually more efficient than trying to write a larger, general kernel. A color-optimized kernel only has access to the color of the current pixel. A transform-optimized kernel can only affect the position of the current pixel. A general kernel can impact color and position as well as read other values from the source image.

Wrapping a `CIKernel` with a `CIFilter`

Now you’ll make a CIFilter subclass as the interface between the Metal code and your Swift code. Every CIFilter has an outputImage variable that returns a CIImage. Generally, input variables for a filter begin with input. The grayscale filter you are making has a single input and a single output. Create a new Swift file in your project and title it “GrayscaleFilter.swift”. Then be sure to import CoreImage by adding

import CoreImage

to the top of the file. Then create the filter as a subclass of CIFilter.

class GrayscaleFilter: CIFilter {

Next, create a kernel variable as either static or lazy, so that it only gets created one time, regardless of how often it’s called.

static var kernel: CIColorKernel = { () -> CIColorKernel in
    let url = Bundle.main.url(forResource: "default",
                            withExtension: "metallib")! //1
    let data = try! Data(contentsOf: url)
    return try! CIColorKernel(functionName: "grayscaleFilterKernel",
                      fromMetalLibraryData: data) //2
}()

Look in the bundle for the compiled metal code. The metal code will have a filename of “default”.
The .metalib may contain many kernels so use the one called “grayscaleFilterKernel”

Now add a variable to hold the input image.

var inputImage: CIImage?

Finally override the outputImage variable.

override var outputImage: CIImage? {
    guard let inputImage = inputImage else { return .none }
    return GrayscaleFilter.kernel.apply(extent: inputImage.extent,
                                   roiCallback: { (index, rect) -> CGRect in
                                                  return rect
                                  }, arguments: [inputImage])
}

The apply function of the kernel takes an extent argument. For a CIImage the extent property is the width and height of the image. The function has an roiCallback that allows you to specify what part of the image the kernel uses for each calculation. For a color filter, you generally just pass back the rect. Finally, pass in any arguments as an array. Notice that there is not any type checking here. It is up to you to pass in the correct types in the correct order.

Using the Filter

With the Metal code wrapped in a CIFilter you can now apply the filter images the same as using one of the built-in filters. Create an instance of the filter, then assign the inputImage variable a CIImage and display the outputImage. Your code might look something like this:

let filter = GrayscaleFilter()
filter.inputImage = image

displayView.image = UIImage(ciImage: (filter.outputImage ?? image) ?? CIImage())

and it can render a grayscale version of an image, like in this example.

Going Further

You can write custom filters and chain them to the built-in filters to invent new effects. Many of the math formulas for effects are available on the Internet. Most are easy to translate into the Metal Shader Language to make them perfect for your application. Writing an entire image or video editing application to showcase your filters is a much larger task. Using an SDK like the IMG.LY PhotoEditorSDK or VideoEditorSDK can help you provide an app that has the features that users will expect in addition to your cool filters.

To add your filter to the PhotoEditorSDK you first wrap your CIFilter with an Effect. For a basic filter, you only need to override the newEffectFilter variable.

import PhotoEditorSDK

class GrayscaleEffect: Effect {
  override var newEffectFilter: CIFilter? {
      GrayscaleFilter()
  }
}

Then add the filter to the Effect.all array when configuring the SDK, using something like

Effect.all = [NoEffect(), GrayscaleEffect(identifier: "grayscaleFilter", displayName: "Best Gray")]

Now your filter appears the same as the other 60+ filters that ship with the SDK.

It applies to live camera previews as well as still images.

Wrapping Up

In this tutorial, you learned how to configure Xcode for custom Metal code. You also learned how to create a color filter kernel in Metal and wrap it in a CIFilter. By combining your filters with Apple’s filters in pipelines, you can create unique effects for your next app. Further, using an SDK such as PhotoEditorSDK or VideoEditorSDK allows you to showcase your filters in a full-featured image or video editing application. The PhotoEditor SDK for iOS documentation will give you deeper look into all other image adjustments including, ofcourse, the image filtering we discussed above.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How To Resize an Image in React

Antonello — Fri, 30 Jul 2021 10:57:19 GMT

In this article, you will see how to resize an image in JavaScript. In particular, here you will learn to achieve this goal with the react-image-file-resizer React library. If you are looking for a pure Javascript solution, here’s a quick rundown of the HTML API usage.

Providing users with features to resize images has become almost unavoidable. This is because images are larger than ever. It is not a secret that the quality of the images and therefore their file sizes have been increasing for years.

The problem is that dealing with large files is time-consuming. Plus, it may cost money in bandwidth when uploading or downloading them. This is why it is so important to shrink the size of images by resizing them. Also, these issues fall on end-users, and this should be avoided.

So, let’s see how to resize an image in React with react-image-file-resizer. By following this step-by-step tutorial, you will achieve the following result:

Prerequisites

This is the list of all the prerequisites for the demo application you are going to build:

Node.js and npm 5.2+ and higher
react-image-file-resizer >= 0.4.7

Resizing an Image with `react-image-file-resizer`

You can clone the GitHub repository that supports this article and try the demo application by launching the following commands:

git clone https://github.com/Tonel/resize-image-react-demo-imgly
cd resize-image-react-demo-imgly
npm i
npm start

Otherwise, you can continue following this tutorial and build the demo application step by step.

1. Creating a React Project

Generate an empty working project in React with Create React App, the officially supported way to create single-page React applications. You can initialize a new project called react-image-resizer-demo with the following command:

npx create-react-app react-image-resizer-demo

You will now have a demo project located in the react-image-resizer-demo folder with the following file structure:

react-image-resizer-demo
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Move into the react-image-resizer-demo folder and start a local server by launching these commands:

cd react-image-resizer-demo
npm start

Reach http://localhost:3000/ in your browser. You should now be able to see the default Create React App screen, as follows:

2. Installing `react-image-file-resizer`

First, you have to add the react-image-file-resizer library to your project’s dependencies. You can do it by running the following command:

npm install --save react-image-file-resizer

Thus, your package.json file will be updated accordingly. You should now be able to spot react-image-file-resizer as a dependency.

Now, all prerequisites have been met. So, you can start building your image resizer component. Let’s see together how.

3. Building the Image Resizer Component

First, make a components folder inside src. Then, create an ImageResizer folder containing index.js and index.css. These two files will contain the resizer component definition and style respectively.

Initialize index.js with the following snippet:

import React from 'react';

function ImageResizer() {
  return <div>{/*TODO*/}</div>;
}

export default ImageResizer;

This way, you have just defined an empty ImageResizer component.
Now, import the Resizer utility from the react-image-file-resizer library. Add it to the ImageResizer imports:

import Resizer from 'react-image-file-resizer'

This is what the final ImageResizer will look like:

import React, {useEffect, useState} from "react";
import Resizer from "react-image-file-resizer";

function ImageResize(props) {
    const {imageToResize, onImageResized, resizeAspect, resizeQuality} = props;

    const [imageToResizeUri, setImageToResizeUri] = useState();
    const [imageToResizeWidth, setImageToResizeWidth] = useState();
    const [imageToResizeHeight, setImageToResizeHeight] = useState();

    useEffect(() => {
        if (imageToResize) {
            const reader = new FileReader();

            reader.addEventListener('load', () => {
                setImageToResizeUri(reader.result);
            });

            reader.readAsDataURL(imageToResize);
        }
    }, [imageToResize])

    useEffect(() => {
        if (imageToResize && imageToResizeWidth && imageToResizeHeight) {
            Resizer.imageFileResizer(
                imageToResize,
                imageToResizeWidth * resizeAspect,
                imageToResizeWidth * resizeAspect,
                "JPEG",
                resizeQuality,
                0,
                (uri) => {
                    onImageResized(uri)
                },
                "base64"
            );
        }
    }, [
        imageToResize, imageToResizeWidth, imageToResizeHeight,
        onImageResized, resizeAspect, resizeQuality
    ]);

    return (
        <img
            src={imageToResizeUri}
            onLoad= {(e) => {
                const img = e.target;
                setImageToResizeWidth(img.width);
                setImageToResizeHeight(img.height);
            }}
            crossorigin="anonymous" // to avoid CORS-related problems
        />
    );
}

ImageResize.defaultProps = {
    onImageResized: () => {},
    resizeAspect: 0.5,
    resizeQuality: 100
}

export default ImageResize;

imageToResize is the source Blob representing the image received from the props. First, imageToResize is transformed into a Base64 URI by the first useEffect() function, and then showed. After being loaded by the <img> HTML tag, the image’s width and height are saved to be used during the resizing operation. This is done by the second useEffect() function, which takes care of resizing the Blob image received by employing the Resizer utility.

Please, note that resizeAspect and resizeQuality are two props in charge of defining the resize aspect and quality percentage, respectively. By default, they are assigned to 0.5 and 100. This means that the resized image will be 50% smaller and no compression will be applied during the resizing process.

4. Putting It All Together

Now, let’s see the ImageResizer component defined above in action. All you need to do, is change the App.js file as follows:

import React, { useState } from 'react';
import './App.css';
import ImageResize from './components/ImageResizer';

function App() {
  const [imageToResize, setImageToResize] = useState(undefined);
  const [resizedImage, setResizedImage] = useState(undefined);

  const onUploadFile = (event) => {
    if (event.target.files && event.target.files.length > 0) {
      setImageToResize(event.target.files[0]);
    }
  };

  return (
    <div className="app">
      <h1>Image Resizer</h1>
      <p>
        Please, upload an image and it will be showed both original and resized
        by 50%
      </p>
      <input type="file" accept="image/*" onChange={onUploadFile} />
      <div>
        <ImageResize
          imageToResize={imageToResize}
          onImageResized={(resizedImage) => setResizedImage(resizedImage)}
        />
      </div>
      {resizedImage && (
        <div>
          <h2>Resized Image</h2>
          <img alt="Resize Image" src={resizedImage} />
        </div>
      )}
    </div>
  );
}

export default App;

The input element allows users to upload an image. This is stored in the imageToResize state variable, and then passed to ImageResizer as a props. This will take care of resizing it accordingly. The resulting resized image is saved thanks to the setResizedImage function, and finally displayed in the Resize Image section.

If you are a Next.js user, you can use the <Image /> component to make it resize images for you. React will then render resized images automatically.

Final Considerations on `react-image-file-resizer`

Resizing an image cannot be considered a complex task. However, using a library like react-image-file-resizer makes everything easier. In fact, you can achieve your goal with just a handful of lines of code. On the other hand, you should take into account what using such specific libraries implies. In fact, when you need to perform many image-related operations, you may be ending up with as many libraries as operations required.

Not only might they be complex to make them coexist, but they may have very different UIs as well. This is why, harnessing a commercial and more complete solution such as PhotoEditorSDK could be a better approach. With only one library, you would get several tools to deal with images as you need. This, while preserving the consistency of your application’s UI. Plus, whenever you need help, you can ask for support from the img.ly developers who built the SDK.

Resizing an Image with PhotoEditorSDK

First, you should read this article from the official documentation on how to get started with PhotoEditorSDK in React. Then, by using the transform tool you can perform cropping, resizing, flipping, and rotation operations with just one feature. This way, you should be able to achieve the desired result

Conclusion

In this article, we looked at how to resize an image in React. Although this cannot be considered a complex feature to implement, using a library such as react-image-file-resizer is recommended. As we have seen, you can resize an image effortlessly and with only a few lines of code. On the other hand, it is a library with a very specific and limited purpose. So, if you needed to perform more than one operation on your images, you might want to take advantage of a more advanced and complete solution – such as PhotoEditorSDK.

A Photo and Video Editor for React Native Apps

Julia — Fri, 29 May 2020 10:30:01 GMT

How to integrate a Photo and Video Editor into your React Native App

This tutorial walks you through the integration process of PhotoEditor SDK and VideoEditor SDK into your React Native app for iOS and Android. You’ll learn how to use our React Native modules to facilitate the integration and to customize our editors. For this tutorial we presume that all the necessary development tools for building an iOS and Android app are met, so make sure to complete the official React Native CLI Quickstart guides for iOS and Android beforehand.

Please make sure to acquire the licenses for PhotoEditor SDK and VideoEditor SDK before integrating them.

https://www.youtube.com/embed/e9JiCMQKrJY?feature=oembed

In this tutorial, we’re going to show how to integrate PhotoEditor SDK and VideoEditor SDK for iOS and Android into your React Native app. Therefore, we created React Native modules for our products to simplify this process for you as much as possible. We’re going to use VideoEditor SDK’s README, which is in most parts identical to the PhotoEditor SDK’s README, and Visual Studio Code. So, let’s get started.

First, we create a React Native project with the name “Demo” based on the default template by using the command npx react-native init Demo. The project will now be initialized and automatically install all dependencies of the current React Native version. Afterwards, we can find the new React Native project ready to use in the folder “Demo”. So, we’ll speed this up a little.

We already prepared another folder with resources and assets that we want to integrate in our app. Here we chose an image, the required licenses for our PhotoEditor SDK and VideoEditor SDK for both target platforms, a video and two logos that we will later use to show how we can customize our editors. We copy these resources into the root of our project to make the resources accessible for our app.

Now, we switch to the folder of the “Demo” project*.* We can now copy and execute the command yarn add react-native-videoeditorsdk from the README to install the dependencies to the React Native module for our VideoEditor SDK … and to the React Native module for PhotoEditor SDK by issuing the command yarn add react-native-photoeditorsdk.

Now, we’re going to set up the dependencies for our native iOS libraries. We can simply copy the command cd ios && pod install && cd .. from the README and execute it to install all iOS dependencies. They include the native PhotoEditor SDK and VideoEditor SDK libraries that are required by our React Native modules.

And now, we set up the dependencies for our native Android libraries. The required steps that we will now take are described in detail in the README.

android {
  defaultConfig {
    multiDexEnabled true
  }
}

dependencies {
  implementation ‘androidx.multidex:multidex:2.0.1’
}

We copy the lines and add them at the end of our android/app/build.gradle file. Now we need to change the superclass of our MainApplication class to enable Multidex. Next, we add the img.ly repository and the plugin by copying the following lines and add them at the top of our android/build.gradle file located in our Android folder.

buildscript {
  repositories {
    jcenter()
    maven { url "https://plugins.gradle.org/m2/" }
    maven { url "https://artifactory.img.ly/artifactory/imgly" }
  }

  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.61"
    classpath 'ly.img.android.sdk:plugin:7.1.8'
  }
}

Now, we can configure our PhotoEditor SDK and VideoEditor SDK by opening the android/app/build.gradle file and add these lines under apply plugin: “com.android.application".

apply plugin: 'ly.img.android.sdk'
apply plugin: 'kotlin-android'

// Comment out the modules you don't need, to save size.
imglyConfig {
  modules {
    include 'ui:text'
    include 'ui:focus'
    include 'ui:frame'
    include 'ui:brush'
    include 'ui:filter'
    include 'ui:sticker'
    include 'ui:overlay'
    include 'ui:transform'
    include 'ui:adjustment'
    include 'ui:text-design'
    include 'ui:video-trim'

    // This module is big, remove the serializer if you don't need that feature.
    include 'backend:serializer'

    // Remove the asset packs you don't need, these are also big in size.
    include 'assets:font-basic'
    include 'assets:frame-basic'
    include 'assets:filter-basic'
    include 'assets:overlay-basic'
    include 'assets:sticker-shapes'
    include 'assets:sticker-emoticons'
  }
}

Getting back to our iOS version, we can now launch our demo project on iOS, which will currently look like a plain React Native project that we initialized with the first command.

The main difference to an off-the-shelf React Native project is that our React Native modules are installed and ready-to-use in the App.js file once the native projects are compiled. Then, it won’t be necessary to recompile the native projects for the remainder of this tutorial. We sped up the compilation a little and here we go — our React Native app is running on the iOS simulator. Now, we do the same for the Android version and wait until the project is compiled.

Now, the demo project launched on both platforms as we can see on the right on the iOS simulator at the top, and on the Android emulator at the bottom of the screen.

We’ve decided that we want to start our photo editor by pressing a button. So next, we’re going to actually customize our React Native app by adding this button. Therefore, we open the App.js file and import the Button component in order to create a button with the title “Edit a sample image”. For now, we leave the onPress function empty.

We save the App.js file to trigger a refresh of the running apps and immediately see the result on the right. The new button appears in both, the iOS and the Android app.

Now we create a second button with the title “Edit a sample video”. This will respectively start our video editor. And again, we save the App.js file and see the second button appear on the right side.

Next, we are going to add the code that actually opens our editors when we press the buttons. Visual Studio Code automatically imported the respective React Native PhotoEditor SDK module for us at the very top of the file, while writing the code that makes use of our SDK. We do the same for the VideoEditor SDK. We use the require function to make static assets available to our app. Here, we “require” our sample image and our sample video that we copied to the app’s folder in the beginning and pass them as the first argument to our openEditor functions. The first argument can also be a regular URI.

We save the App.js file again and now we can click the buttons to start our photo editor or video editor. There we go! We still see a watermark here. The reason for this watermark is that we haven’t unlocked our SDKs so far which we will do next.

We unlock both products with our licenses to get rid of the watermark. If not unlocked, the watermark will be on both the image and video previews as well as on the exported images and videos. To unlock the products, we use the unlockWithLicense function of each SDK. In total, we need four license files, one license file for each product and platform combination. The license files should be named pesdk_license and vesdk_license with platform-specific extensions .ios.json and .android.json. React Native will then automatically pick the right file for the corresponding platform. After this, the watermarks will be removed for PhotoEditor SDK and VideoEditor SDK on both platforms. And now you can also see it in the simulator — no watermarks anymore.

In the next step, we’re going to change the configuration of the editors. If no changes are made to the configuration, our default stickers are available with the editor. To customize them, we need to import the Configuration from either the PhotoEditor SDK or VideoEditor SDK. The configurations are compatible between both products. So, for this tutorial, we decided to use the VideoEditor SDK configuration which we are now using by adding Configuration to the react-native-videoeditorsdk imports. We decided that we want to add custom stickers to our editors. Therefore we define a non-default configuration to the sticker tool.

const configuration: Configuration = {
  sticker: {}
}

To customize the sticker assets, we need to define the sticker “categories” array. Here, we define a new category and name the identifier demo_sticker_category. These asset identifiers must always be unique. Next, we set a name for the category and we name it “Logos”.

Each category also requires a thumbnail image to be displayed in the editor. For the thumbnail, we use the React logo that we added to the folder of our app at the very beginning. Next, we define the items for this new sticker category. These items are the actual stickers that we can apply to the edited image or video. We now create a new sticker for the React logo. Therefore, we call the identifier demo_sticker_react and name it “React”. These sticker names won’t appear in the UI, but they are used for accessibility. Now, we need to define the actual image that should be used for that sticker. Here we use the React image again.

To create a second sticker, we can now copy and paste the code of the first sticker. We create a sticker with our img.ly logo and rename the identifier of the pasted code to demo_sticker_imgly. Accordingly, we set the name to “img.ly” and change the file to imgly.png.

In addition, we want to specify a non-default tint mode for our second sticker by using tintMode: TintMode.SOLID which enables us to change the color of the sticker. The TintMode type is automatically added to the VideoEditor SDK imports for us by Visual Studio Code. Now, that we completed our configuration, we need to pass it as the second argument to the openEditor functions in order to take effect.

We save the App.js file again to refresh the running apps and we can see the result live after starting a new editing session. Please note that you cannot alter the configuration of a running editor instance. You always need to start a new editing session to see configuration changes.

We want to use another feature of our SDKs which is called serialization. With the serialization feature, we can capture all image and video editing operations that are applied in the editor and export them. This allows us to import the editing operations in later sessions and continue editing. The serializations are compatible between both products as well. The input serialization is the third parameter of the openEditor functions of our SDKs and the output serialization is optionally part of their result type.

First, we check if the result is “null”. This is the case when a user clicks the “Discard” button in the editor and thus does not export an image or video. If the result is not ”null”, we know that the user exported an image or video. Then we can assign the exported serialization to the previously defined global serialization variable which will then be input to the next editing session. We copy the code and add it to the video editor as well to enable the serialization function here too.

Now, one thing is left to enable the actual serialization export in the configuration. The serialization export is disabled per default because not every user needs the serialization feature. Here, we enable it now and also change the export type to object. By doing so, the result type of the editor will contain the serialization as an object. Per default, the serialization is exported to a file and that file name is returned as part of the export result. Writing the serialization to a file is a reasonable default as serializations can be quite large, especially if large amounts of binary data for personal stickers are embedded.

Now, we can run the app on the simulator and use all the parameters that we configured in this tutorial. First, we can add our custom stickers, both the React logo and the img.ly logo. Here we can also change the colors which we enabled with the tint mode.

We can also use the text design tool to add a phrase to our image. Here we can pick different designs, so we’re trying a couple and place the text design fitting to the image and logo.

Next, we export our image with the serialization. With the serialization function enabled, it is now possible to import the editing operations into our video editor. This allows us to keep on editing because the serialization is compatible between both products. So here we can add further words to our text design. We can also put filters on our video. For example, we can choose the peach duo tone and increase the contrast a little. And here we go! We successfully integrated PhotoEditor SDK and VideoEditor SDK into our React Native app.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Case Study: W | Bear & PhotoEditor SDK

Felix — Mon, 17 Sep 2018 00:00:00 GMT

W | Bear is the first global photo and video blogging social community for gay men. The free app allows its users to post pictures and videos to a personal feed, express their creativity and connect with like-minded bears. The blogging community provides a safe place where people can build and shape a social environment, chat, flirt and date. W | Bear was created and developed by the private and public cloud solution provider gNetLabs. We sat together with Xavier Nicolle, CEO at gNetLabs to talk about the vision behind W | Bear, time to market and providing users with tailored assets to amplify engagement.

“W | Bear is aimed at the bear subculture of the LGBT community. People express themselves, their story and their life through photos and videos. Our users are very happy with the app. 18 months after the release we reached around 150.000 users worldwide, which is beyond our expectation and it’s getting faster and faster. We get a lot of positive reactions and amazing feedback from users that tell us, that our application helped them to gain more self-esteem and to better integrate into the community,” Xavier Nicolle says.

As pictures were going to be a vital part of the Instagram like photo blogging application (to this day, more than 2 million pictures and videos have been posted on the app), the folks at gNetLabs started exploring options as their CEO explains: “Time to market was critical for us, and we were kind of in a rush to develop the app. From our background as a hosting company, we’d usually develop and host everything ourselves, however, due to time problems, that wasn’t a viable option regarding the photo editing functionalities. So, we jumped into PhotoEditor SDK to save time, really liked it and eventually kept it. The features and pricing perfectly fit our needs, the integration was straightforward, and the SDK is of great quality and open enough for fine-tuning. On top of that, unlike Adobe Creative SDK for example, it is not linked to any other service. And that is exactly what we were looking for, a piece of software that we could simply plug in that isn’t linked to a service that we don’t need.”

To offer the best experience possible with W | Bear, the folks at gNetLabs pay close attention to how people engage with the app and its features. “Our users mostly work with stickers, frames, and overlays when editing their pictures,” Xavier Nicolle says, “so, bearing that in mind, we want to provide more content and assets that are tailored for the community so our users will hopefully make more use of the editor. Currently, we’re working together with artists that develop stickers that we’re going to introduce using the SDK.”

But it doesn’t stop with W | Bear, as Xavier Nicolle describes the vision behind the community: “W | Bear is the first piece of a larger social network we want to develop that is dedicated and aimed at the whole LGBT community. We’re going to release more applications using the same technology as W | Bear to address more subcultures. On top of that, there are going to be corresponding websites for each app, so the user will be able to find the same functionalities and tools across all platforms.”

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

How to build Instagram’s Story Editor in a Day

Malte — Wed, 13 Dec 2017 00:00:00 GMT

A common question we get asked from our customers, is whether they’ll be able to create an entirely different user interface using our PhotoEditor SDK. The SDK comes with it’s own customizable UI, but customization are of course limited to a certain extent. As our SDK is used in many different use cases and contexts, we like to explore what’s possible with the PhotoEditor SDK and its included components. Here, we decided to build a UI similar to Instagram’s Stories or Snapchat using our SDK. By now, this ‘Story UI’ has become a popular way of quickly designing with different elements (stickers, brush and text) rather than enhancing and styling the image only.

So we grabbed our own docs and headed out to create a demo in which we recreate the Instagram Stories using components from the PhotoEditor SDK. This article presents our approach by starting with a general overview and then diving into the different view controllers to look at specific implementation details. You can follow along by downloading the accompanying code.

Architecture and Overview

Any image editing interface is naturally centered around some sort of canvas or preview. This is where the image with all operations applied is rendered and any changes are shown to the user. All tools add their own interface elements to allow modifications like adding stickers, text or brush strokes to the image. To create such a hierarchy of tools, the PhotoEditor SDK makes heavy use of an iOS pattern called view controller containment. The root view controller, an EditViewController in this case, manages a series of child view controllers:

A PhotoEditPreviewController that handles the internal model and all rendering, as well as the rendering canvas itself
A TextSpriteEditController above the PhotoEditPreviewController that allows selection and manipulation of text sprites
A StickerSpriteEditController above the PhotoEditPreviewControllermanages selection and manipulation of stickers

The child view controllers may manage additional view controllers themselves, but we only need to manage the topmost objects and wire their interfaces together. This is done within the EditViewController who is also responsible for presenting the view controllers that implement the different tools, StickerViewController, BrushViewController and TextViewController.

These are mainly responsible for managing the interface and adjusting the model accordingly, while the ‘real’ work is being done in the background by the PhotoEditPreviewController whenever the model gets updated. Thanks to the SDK, creating our BrushViewController boils down to creating and wiring a BrushEditController and adding its view, color and size controls to the BrushViewControllers view. This creates a fully fledged brush tool with OpenGL rendering, color and brush size adjustments, and great performance in just 89 lines of code.

EditViewController

As described in the previous section, the EditViewController is the root view controller of our demo application and is responsible for showing a preview, presenting tools and rendering the final output. To do so, we add a PreviewEditViewController and SpriteEditControllers to the root view and register the EditViewController as their delegate. Registering as a delegate, allows the EditViewController to wire all child view controllers and pass data between them. In order to do so, we just need to return objects of other view controllers or react to model changes. All controllers then use the EditViewController to ask for objects or data they need, for example the current size of the preview view, in order to rearrange interface elements or calculate model updates. As an example, the spriteEditControllerPreviewView(_ spriteEditController:) method of the SpriteEditControllerDelegate protocol, asks for the current preview view. All we need to do in our EditViewController is to get this view from our PreviewEditViewController and return it:

Sprite Handling

Stickers and text sprites are both sprites, but there are different SpriteEditController subclasses for each sprite type, because of differing gestures and UI elements. The Instagram UI allows the editing of both sticker and text sprites within a single view, so we need to add both specific controllers, StickerSprite- and TextSpriteEditController to our EditViewController and dynamically enable the appropriate one depending on the currently selected sprite.

StickerViewController

For adding stickers, the Instagram Stories UI offers a single button that presents a grid of all available stickers. Replicating this using the PhotoEditor SDK is easy to do, as the SDK uses a collection view for presentation as well. This collection view is embedded in a StickerSelectionController which we add to our StickerViewController. Once again, after registering as the StickerSelectionControllers delegate, we get notified upon selection of any sticker and just need to change the PhotoEditModel accordingly. To do so, we create a link between the root and the currently presented view controller using the StickerViewControllerDelegate protocol. Once a sticker has been selected, we create or update a StickerSpriteModel, pass it to the EditViewController, which updates the PreviewEditViewController model:

This triggers a new rendering pass and the sticker appears on screen. All that’s left to do now is closing the StickerViewController. Again, we use the delegate pattern to ask the EditViewController about the referenceSize that’s needed to calculate the sticker’s normalized size.

BrushViewController

This view controller, just as described above, is just a wrapper around the BrushEditController that adds a color and size selection. Once again, all interaction is handled through delegates and upon closing of the brush tool, the internal model is updated. To support undo, we need to pass the PreviewEditViewController’s undoController and begin a new group, whenever we fire up the tool.

TextViewController

The text editor Instagram created within their Stories UI is essentially a fullscreen textfield with additional controls for color and size selection. Ignoring the customized design, this can easily be recreated using stock iOS components and essentially only requires listening for keyboard notifications to handle the layout and adjusting the textfield’s properties according to user interactions. Once finished, we create a corresponding TextSpriteModel that is positioned below the UITextField and notify the EditViewController using the delegate. After dismissing the TextViewController the text rendering is handled by the PhotoEditPreviewController. When reselecting an existing label, we just prefill the UITextField with it’s contents and update the model when the label textfield changes. To ensure the UITextField is the only place the text is currently visible, we hide the spriteView we previously selected and show it again, once editing has finished.

The Result

Recreating the Instagram Stories UI seemed rather challenging at first, but using our PhotoEditor SDK, it quickly turned into a very rewarding project. Being able to create something like a brush tool without the need to actually implement any drawing or gesture handling is amazing and helped to create a working prototype in no time. Tricky details like sprite transformations, remote loading of stickers or undo management were entirely handled by the SDK and we could completely focus on the interface itself.

For more details and some actual code, take a look at the repository. We documented all tricky parts and you should be able to start your own implementation using the code. Please keep in mind, that the code may target a slightly older PhotoEditor SDK version, so check for a new version before you start your journey.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.