Video Editing – IMG.LY Blog

Build in a Day: AI Video Clipping with CE.SDK

Eray — Thu, 05 Feb 2026 12:17:28 GMT

Introduction

We built a video shortener in a single day using Claude Code and CE.SDK. It extracts 3-4 short clips from long-form video, handles transcription, identifies the best moments via AI, detects speakers, and outputs vertical/horizontal/square formats—all running in the browser.

Features:

Extracts 3-4 clips per video (highlights, summaries, or cleaned-up edits)
Outputs 9:16 (vertical), 16:9 (landscape), or 1:1 (square)
Detects speakers and maps them to faces with user confirmation
Auto-crops to follow the active speaker
Adds captions and text hooks
Non-destructive: change aspect ratio or template without re-processing

Best suited for: Videos with speech/dialogue (podcasts, interviews, presentations, vlogs)

Why Client-Side?

CE.SDK’s CreativeEngine runs in the browser via WebAssembly. Video decoding, timeline manipulation, effects, and preview all happen on the user’s device.

Benefits:

No upload/download wait — edits preview instantly
Non-destructive — switch aspect ratio or template without rendering
Lower infrastructure costs — your costs don’t scale with video length or user count

Tech Stack

Frontend: Next.js + React
Video Engine: CE.SDK (CreativeEngine)
Transcription: ElevenLabs Scribe v2
AI Analysis: Google Gemini

Architecture Overview

High-Level Flow

Required API Keys

Service	Purpose	Environment Variable
CE.SDK	Video editing engine	`NEXT_PUBLIC_CESDK_LICENSE`
ElevenLabs	Speech-to-text transcription	`ELEVENLABS_API_KEY`
Gemini (via OpenRouter or direct)	AI highlight detection	`OPENROUTER_API_KEY` or `GEMINI_API_KEY`

Setting Up CE.SDK

What is CE.SDK?

CE.SDK (CreativeEngine SDK) is a browser-based engine for video, image, and design editing—a programmable video editor you can embed in your app.

Key Concepts:

Engine: The runtime that manages the editing session
Scene: The document/project containing all elements
Blocks: Individual elements (video clips, text, shapes, audio)
Timeline: Time-based arrangement of blocks for video editing

Installation

npm install @cesdk/cesdk-js

Initializing the CreativeEngine

import CreativeEngine from '@cesdk/cesdk-js';

const engine = await CreativeEngine.init({
  license: process.env.NEXT_PUBLIC_CESDK_LICENSE,
});

// Create a video scene
const scene = engine.scene.createVideo();

// Get the page (timeline container)
const pages = engine.scene.getPages();
const page = pages[0];

// Configure page dimensions for your target aspect ratio
engine.block.setWidth(page, 1080); // 9:16 vertical
engine.block.setHeight(page, 1920);

Uploading Video to CE.SDK

CE.SDK works with video through a fill-based system. The graphic block is the container, while the video fill holds the actual media source and playback properties.

// Create a video block
const videoBlock = engine.block.create('graphic');
const videoFill = engine.block.createFill('video');

// Set the video source
engine.block.setString(
  videoFill,
  'fill/video/fileURI',
  videoUrl // Can be a blob URL or remote URL
);

// Apply fill to block
engine.block.setFill(videoBlock, videoFill);

// Add to timeline
engine.block.appendChild(page, videoBlock);

Extracting Audio for Transcription

// Configure audio-only export
const mimeType = 'audio/mp4';

// Export just the audio track
const audioBlob = await engine.block.export(page, mimeType, {
  targetWidth: 0,
  targetHeight: 0,
});

// audioBlob can now be sent to transcription API

Setting both dimensions to 0 tells CE.SDK to skip video encoding entirely, making this export much faster than exporting the full video.

Getting Video Metadata

// Get video duration
const duration = engine.block.getDuration(videoBlock);

// Get dimensions from the fill
const videoFill = engine.block.getFill(videoBlock);
const sourceWidth = engine.block.getSourceWidth(videoFill);
const sourceHeight = engine.block.getSourceHeight(videoFill);

console.log(`Video: ${sourceWidth}x${sourceHeight}, ${duration}s`);

AI-Powered Transcription & Highlight Detection

The Pipeline

Audio → Transcription: Send extracted audio to ElevenLabs Scribe
Transcription → Analysis: Feed word-level transcript to Gemini
Analysis → Timestamps: Map AI suggestions back to precise video times

Transcription with Speaker Diarization

ElevenLabs Scribe v2 provides:

Word-level timestamps (start/end time for each word)
Speaker diarization (which speaker said what)

The output is a structured transcript where each word has a precise timestamp, enabling frame-accurate editing.

AI Highlight Detection with Gemini

The prompt structure matters. Here’s what works:

You are analyzing a video transcript to identify segments for short-form content.

TRANSCRIPT:
[Word-by-word transcript with timestamps]

TASK:
Identify 3-4 segments that work as standalone short videos. For each segment:
1. Find the exact starting and ending words
2. Ensure clean sentence boundaries (no mid-sentence cuts)
3. Aim for 30-60 second segments

OUTPUT FORMAT (JSON):
{
  "concepts": [
    {
      "id": "concept_1",
      "title": "Hook title",
      "description": "Why this segment works as a standalone clip",
      "trimmed_text": "The exact transcript text to keep...",
      "estimated_duration_seconds": 45
    }
  ]
}

CRITERIA FOR SELECTION:
- Strong hooks (surprising statements, questions, bold claims)
- Complete thoughts (don't cut mid-explanation)
- Emotional peaks (humor, insight, controversy)
- Standalone value (makes sense without context)


Before finalizing each segment, ask: "If someone started watching here,
would they understand what's being discussed?"

Mapping Back to Timestamps

Once Gemini returns the trimmed_text, we match it against our word-level transcript to find exact timestamps:

AI returns:     "The secret to success is actually quite simple..."
Transcript has: [{ word: "The", start: 45.2 }, { word: "secret", start: 45.4 }, ...]

Result:         Trim video from 45.2s to 52.8s

This text-matching approach is more reliable than asking the AI to output timestamps directly. LLMs can hallucinate timestamps or miscalculate offsets, but they’re excellent at identifying the right words—so we let the transcript data provide the ground truth for timing.

Working with the CE.SDK Timeline

Understanding Blocks

// Video/Image content
const graphic = engine.block.create('graphic');

// Audio track
const audio = engine.block.create('audio');

// Text overlay
const text = engine.block.create('text');

// Each block can be positioned on the timeline
engine.block.setTimeOffset(block, startTimeInSeconds);
engine.block.setDuration(block, durationInSeconds);

Manipulating Trim Points

Trimming controls which portion of the source media is shown:

const videoFill = engine.block.getFill(videoBlock);

// Set where in the source video to start (in seconds)
engine.block.setTrimOffset(videoFill, 45.2);

// Set how long to play from that point
engine.block.setTrimLength(videoFill, 30.0);

// Also update the block's duration to match
engine.block.setDuration(videoBlock, 30.0);

Working with Fills and Their Timing

// Get the fill (contains the actual media)
const fill = engine.block.getFill(block);

// Fills have their own timing properties
const trimStart = engine.block.getTrimOffset(fill);
const trimDuration = engine.block.getTrimLength(fill);

// The block's duration should typically match the fill's trim length
engine.block.setDuration(block, trimDuration);

Think of the fill as the media source (which part of the original video to use) and the block as the timeline placement (when and how long it appears). Both need to be updated together for clean edits.

Creating Time-Based Edits from Transcript Words

interface TranscriptWord {
  word: string;
  start: number;
  end: number;
  speaker_id?: string;
}

function applyTranscriptTrim(
  engine: CreativeEngine,
  videoBlock: number,
  words: TranscriptWord[]
) {
  if (words.length === 0) return;

  const startTime = words[0].start;
  const endTime = words[words.length - 1].end;
  const duration = endTime - startTime;

  const fill = engine.block.getFill(videoBlock);

  engine.block.setTrimOffset(fill, startTime);
  engine.block.setTrimLength(fill, duration);
  engine.block.setDuration(videoBlock, duration);
}

Generating Speaker Thumbnails

async function generateSpeakerThumbnail(
  engine: CreativeEngine,
  videoBlock: number,
  timestampSeconds: number,
  size: number = 128
): Promise<string> {
  const fill = engine.block.getFill(videoBlock);

  // Seek to the specific timestamp
  engine.block.setTrimOffset(fill, timestampSeconds);
  engine.block.setTrimLength(fill, 0.1); // Just a single frame

  // Export as image
  const blob = await engine.block.export(videoBlock, 'image/jpeg', {
    targetWidth: size,
    targetHeight: size,
  });

  return URL.createObjectURL(blob);
}

We sample multiple timestamps throughout each speaker’s talk time to show different facial angles and expressions—this helps users identify the right person even if they’re looking away or mid-gesture in one frame.

Speaker Detection & Face Tracking

Why Semi-Automatic?

Fully automatic speaker detection fails often enough that we added a confirmation step. Users verify detected faces against speaker names from the transcript—takes a few seconds and prevents bad crops on the entire video.

How It Works

Sample frames throughout the video
Detect & cluster faces using face-api.js (runs in browser, no server needed)
User confirms speaker identities via thumbnails
Correlate with transcript diarization to map speakers → face locations

This gives us verified speaker-to-face mapping for dynamic cropping and picture-in-picture layouts.

Multi-Speaker Templates & Dynamic Switching

The Concept

When a video has multiple speakers, we can create layouts that show:

The active speaker prominently
Other speakers in smaller picture-in-picture views
Dynamic switching as the conversation flows

Creating Picture-in-Picture with CE.SDK

// Duplicate the video block for each speaker slot
const pipBlock = engine.block.duplicate(originalVideoBlock);

// Position and size the PiP
engine.block.setWidth(pipBlock, 200);
engine.block.setHeight(pipBlock, 200);
engine.block.setPositionX(pipBlock, 20); // 20px from left
engine.block.setPositionY(pipBlock, 20); // 20px from top

// Enable cropping
engine.block.setClipped(pipBlock, true);
engine.block.setContentFillMode(pipBlock, 'Cover');

Key Technique: Muting Duplicate Audio

When duplicating video blocks for multi-speaker layouts, each copy has its own audio track. We must mute all but one. The setMuted API operates on the video fill, not the block itself:

// For each speaker slot after the first, mute the video fill
if (slotIndex > 0) {
  const videoFill = engine.block.getFill(duplicatedBlock);
  if (videoFill) {
    engine.block.setMuted(videoFill, true);
  }
}

Dynamic Speaker Switching

As the active speaker changes throughout the video, we:

Detect which speaker is talking (from transcript diarization)
Swap speaker positions in the template
Keep the active speaker in the prominent position

The layout updates automatically as the conversation switches between speakers. We apply different trim offsets to each duplicated block based on the transcript timing—so the main speaker slot shows the person currently talking while PiP slots show the listeners.

Preview, Playback & Export

Setting Up the Canvas

const container = document.getElementById('cesdk-canvas');
engine.element.attachTo(container);

Playback Controls

engine.player.play();
engine.player.pause();
engine.player.setPlaybackTime(30.5); // seek to 30.5 seconds

const currentTime = engine.player.getPlaybackTime();
const isPlaying = engine.player.isPlaying();

Syncing UI State

engine.player.onPlaybackTimeChanged(() => {
  const time = engine.player.getPlaybackTime();
  updateTimeDisplay(time);
  updateProgressBar(time / totalDuration);
});

engine.player.onPlaybackStateChanged(() => {
  updatePlayButton(engine.player.isPlaying());
});

Export Options

const exportOptions = {
  targetWidth: 1080,
  targetHeight: 1920,
  framerate: 30,
  videoBitrate: 8_000_000, // 8 Mbps
};

const blob = await engine.block.export(
  page,
  'video/mp4',
  exportOptions,
  (progress) => updateProgressBar(progress * 100)
);

// Trigger download
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = 'shortened-video.mp4';
a.click();

For longer videos, consider showing estimated time remaining or allowing background export. Browser export is single-threaded and blocks the tab—a 5-minute export of a 60-second clip isn’t unusual on average hardware, so user feedback is critical.

The Finished App

The user flow:

Upload → Drop a long-form video into the browser
Configure → Pick output mode (highlights/summary/cleanup) and aspect ratio (9:16, 16:9, 1:1)
Verify speakers → Match detected faces to transcript speaker names
Review clips → Browse the 3-4 AI-suggested segments, adjust if needed
Choose template → Solo speaker, sidecar, stacked, etc.
Preview → Scrub through the timeline, see exactly what you’ll get
Export → Download the final video directly from the browser

What’s Next

Ideas for Extension

Caption style controls: Custom fonts, animations, and positioning for subtitles
B-roll insertion: Automatically add relevant stock footage
Music & sound effects: AI-selected background audio
Brand templates: Custom overlays, intros, outros
Batch processing: Process multiple videos in sequence

Taking It Server-Side

Client-side processing for large files strains browser memory, and users must keep the tab open during export. A hybrid approach works better for production—upload in the background while users edit, then render on a server. You can also offload just the export step—let users build their edits in the browser, then send the CE.SDK scene JSON to your backend for faster, background rendering.

CE.SDK runs server-side with the same API. For batch processing, background jobs, or offloading rendering from user devices, see the CE.SDK Renderer for creative automation.

Resources

Made by IMG.LY with CE.SDK

Automating Video Creation at Scale: Why Templates, Timelines, and Rendering Matter

Klaudia — Mon, 19 Jan 2026 10:46:07 GMT

You’ve automated image generation—your marketing team can produce thousands of product banners, social posts, and email headers without touching Photoshop. But when someone asks, “Can we do this for video?” the conversation gets complicated fast.

Video automation isn’t just “image automation with a timeline.” It’s fundamentally different. A static banner renders in 100 milliseconds. A 30-second video takes 10-30 seconds to render. Your 10MB image template becomes a 50MB video file. And that’s before you factor in audio tracks, transitions, platform-specific encoding requirements, and the reality that “vertical for TikTok” means different specs than “vertical for Instagram Stories.”

Most teams discover these challenges the hard way—after they’ve already committed to video automation and hit unexpected walls. The template that looked perfect in the editor breaks when the headline is 30% longer in German. The rendering pipeline that handled images fine collapses under video workloads. The licensing you thought you had doesn’t cover H.264 distribution.

Video automation is solvable. But it requires understanding what makes video different, designing templates that can handle variability, and building infrastructure that respects the complexity.

In this article we will cover:

Why video automation is different
Designing templates that scale
Architecture patterns for generation
Platform constraints & encoding
Production pitfalls & monitoring
When automation does (and doesn’t) make sense

Why Video Automation Is Fundamentally Different

If you’ve built image automation pipelines, video introduces layers of complexity that change how you think about templates, rendering, and infrastructure. The core differences:

Timeline complexity: Video adds time as a dimension—every element has duration, entry/exit timing, and potential animation
Multi-asset coordination: Dozens of assets (video clips, audio, overlays) must stay synchronized across time
Rendering performance: 100x slower than images (30-second video = 900 frames at 30fps)
File size: 10MB PNG becomes 50MB MP4, impacting storage and distribution
Platform requirements: Stricter, more varied specs (aspect ratios, duration limits, codec preferences)

Timeline Complexity vs. Static Composition

Images are spatial—elements positioned on a 2D canvas with fixed relationships. Video adds time. Every element now has duration, timing, and animation across the timeline.

When you generate 10,000 image variants with different text lengths, the layout adjusts spatially. With video, text length affects duration—do you extend the scene to accommodate longer copy, or speed up the animation? Neither is automatic.

A timeline-based engine like CE.SDK’s video editor handles this through track systems similar to TikTok or Instagram: foreground tracks for video clips and audio, background tracks that define overall video length. Video clips can be trimmed, repositioned, and arranged. Audio strips sync with video timing but remain independent—you can adjust one without breaking the other.

The automation challenge: when you inject variable data (product names, customer messages, CTAs), how does that affect the timeline? Template design needs to account for temporal variability, not just spatial.

Multi-Asset Coordination and Audio Sync

A single image template might reference 5-10 assets. A video template coordinates dozens: multiple video clips, background music, sound effects, overlay images, animated text, transitions. Each has its own timing, and they must stay synchronized when you generate variations.

Audio is particularly tricky:

Background music needs to match video duration (fade cleanly, not cut mid-phrase)
Voiceovers sync with specific visual moments
Sound effects trigger at precise timestamps
When automation extends duration (German translation longer), does audio stretch or stay fixed?

Timeline-based engines support audio playback with independent positioning controls. Audio strips appear in the timeline (not on canvas), and you can trim, reposition, and adjust timing independently of video tracks.

Key takeaway: Video templates aren’t just asset collections—they’re choreographed sequences where timing relationships matter as much as visual composition.

Rendering Performance and Infrastructure Implications

Static image rendering is near-instantaneous. Video rendering is fundamentally slower—every frame is a separate render. A 30-second video at 30fps is 900 frames.

Infrastructure implications:

GPU acceleration isn’t optional (CPU-only rendering is prohibitively slow)
Batch processing requires queuing systems (can’t render 1,000 videos synchronously)
Progress monitoring becomes essential (users need status, not guessing)
Export failures are costlier (failed image = 100ms wasted; failed video = 30 seconds)

Most video export APIs provide progress callbacks for real-time status:

const videoBlob = await engine.block.exportVideo(page, {
  mimeType: 'video/mp4',
  onProgress: (rendered, encoded, total) => {
    console.log(`Progress: ${Math.round((encoded / total) * 100)}%`);
  },
});

Designing Templates That Scale

Once you understand why video is different, the next challenge is designing templates that handle variability without manual intervention. Template design determines whether automation breaks or scales.

Scene-Based Modular Structure

Instead of a single 30-second video, think in scenes: 5-second intro, 15-second product showcase, 5-second CTA, 5-second outro. Each scene is self-contained.

Benefits:

Variable duration: Extend the product scene by 3 seconds for longer German translations without affecting intro/outro
Scene reordering: A/B test different sequences (CTA-first vs. product-first)
Conditional scenes: Include/exclude based on data (show “Limited Time Offer” scene only for promotions)
Reusable components: Same intro across multiple video types, swap middle scenes

The template creator defines scene boundaries, timing, and which elements automation can modify. Automation code loads the template and injects data without worrying about timeline complexity.

Text Duration and Audio Sync Strategies

Text management approaches:

Extend scene duration: Add seconds based on character count (Math.ceil(text.length / 40) seconds)
Reduce text: Pass pre-truncated strings that fit fixed timeline
Multi-line display: Let text wrap across frames (works for some designs, breaks others)

Audio sync strategies:

Fixed-duration music: Choose tracks matching video length exactly (less flexible)
Loopable music: Tracks designed to loop seamlessly—calculate loops needed, fade out at end (most reliable)
Adaptive music: Music that adapts to duration changes (complex but professional)

For automation, loopable music with fade-out is safest. Calculate final video duration after data injection, loop the audio track to cover that duration, apply 1-2 second fade at end.

Video Clip Placeholders and Safe Zones

Video placeholder logic:
Your template might designate a 5-second product demo slot, but actual product videos are 3, 7, or 12 seconds. Options:

Trim to fit: Longer clips trimmed to match; shorter clips looped (keeps consistent duration, may cut content)
Extend the scene: Adjust placeholder to match clip length (preserves content, variable duration)
Speed adjustment: Speed up/slow down clip to fit (works 0.75x-1.5x, extreme adjustments look awkward)

Safe zones for multi-format export:

Text safe zone: Keep all text 10% from edges (device overscan, UI overlays)
Critical content zone: Logos, faces, key products within central 70% (survives aspect ratio cropping)
Action-safe zone: Don’t place CTAs in corners where platform UI might cover them

When designing templates that export to multiple aspect ratios (16:9, 9:16, 1:1), test how cropping affects composition.

Architecture Patterns for Video Generation Pipelines

Once your templates can handle variability, the next challenge is scale: how do you actually generate thousands of videos reliably? Three proven patterns:

Pattern 1: Template-Based Batch Rendering

Use case: Marketing team uploads CSV with 500 products. Overnight, system generates 500 personalized video ads.

High-level workflow:

Frontend upload interface (CSV, database query, API)
Job queue (Redis, RabbitMQ, AWS SQS) holds generation requests
Worker nodes pull jobs, generate videos, upload to storage
Notification system alerts when batch completes

Why it works: Batch jobs don’t need real-time feedback. Optimize for throughput over latency—process 100 videos in parallel across 10 worker nodes.

Implementation pattern (simplified):

// Worker node processing one batch item
const engine = await CreativeEditorSDK.create('#cesdk_container', {
  license: process.env.CESDK_LICENSE,
});

const scene = await engine.scene.loadFromURL(
  'templates/product-ad-video.scene'
);

// Inject product data via Variables API
engine.variable.setString('productName', product.name);
engine.variable.setString('productPrice', product.price);

const videoBlob = await engine.block.exportVideo(scene, {
  mimeType: 'video/mp4',
  h264Profile: 77,
  framerate: 30,
  onProgress: (rendered, encoded, total) => {
    updateJobProgress(jobId, encoded / total);
  },
});

await uploadToS3(videoBlob, `output/${product.id}.mp4`);

Scaling: Add more worker nodes horizontally. With GPU-enabled instances, one node can process 10-20 videos per minute depending on complexity.

For large batches (10,000+ videos), use server-side rendering via Docker for maximum throughput. The CE.SDK Renderer handles GPU acceleration automatically and exports to your specified output directory.

Licensing note: For production deployments, use licensed Renderer variants (not open-source) to ensure proper H.264/H.265 codec coverage.

Pattern 2: Real-Time Personalization

Use case: User signs up for service. Within 30 seconds, they receive personalized welcome video with their name and company logo.

High-level workflow:

User action (signup, purchase, milestone) fires event
Serverless function receives event data
Engine generates video in real-time (typically 10-30 seconds)
Video URL returned or sent via email/notification
Cache generated videos if multiple users might trigger identical content

Why it works: Real-time generation provides immediate personalization. Latency (10-30 seconds for a 30-second video) is acceptable when delivered asynchronously.

When to use:

Video duration is short (under 60 seconds)
User data is simple (text substitution, not complex asset swapping)
Delivery is asynchronous (email, notification—not immediate playback)
Volume is moderate (hundreds per hour, not thousands per minute)

For higher volumes, shift to Pattern 1 (batching) or Pattern 3 (hybrid).

Pattern 3: Hybrid UI + Automation

Use case: Marketing manager designs campaign video template in visual editor. System then automatically generates 50 variants for different audience segments, products, or A/B testing.

High-level workflow:

Designer uses visual editor to create master template with variables
Saved template becomes reusable asset
Marketing team specifies which data to inject (audience segments, products)
Backend loads template, injects data, generates variants
Generated videos go through approval workflow before deployment

Why it works: Separates creative work (design) from scale (automation). Designers don’t write code. Automation doesn’t require developer involvement for template updates.

Timeline-based engines like CE.SDK are explicitly designed for this pattern—unified workflow between editor and automation. The same template works in both contexts. What the designer sees in the visual editor is exactly what automation generates. No conversion, no drift, perfect consistency.

This pattern is covered in detail in our creative automation infrastructure guide, which explains why teams need both editor and API capabilities within unified workflows.

Platform Constraints and Encoding Optimization

Generating videos is only half the problem—distribution platforms impose constraints that directly affect how you export.

H.264 Profiles, Levels, and Trade-offs

Video export engines support H.264 encoding with configurable parameters affecting quality, compatibility, and file size:

H.264 Profile:

Baseline (66): Broadest device support, lower compression efficiency
Main (77): Good balance (default), works on most modern devices
High (100): Best quality and compression, requires recent devices

H.264 Level (multiply by 10):

Level 3.1 (31) = Up to 720p at 30fps
Level 4.1 (41) = Up to 1080p at 30fps
Level 5.2 (52) = Up to 4K at 60fps

Recommended settings by distribution channel:

Social media ads (Instagram, TikTok, Facebook): Main profile (77), Level 4.1 (41), 5-8 Mbps bitrate
YouTube videos: High profile (100), Level 5.2 (52), 8-12 Mbps bitrate
Maximum compatibility (email, older devices): Baseline profile (66), Level 3.1 (31), 2-4 Mbps bitrate

Critical constraint: H.264 doesn’t support transparency. Transparent areas render with black backgrounds. Handle this at the template level (solid backgrounds or pre-composited elements), not during export.

Social platforms prefer specific aspect ratios:

9:16 (Vertical): TikTok, Instagram Reels, YouTube Shorts, Stories
1:1 (Square): Instagram Feed, Facebook Feed, LinkedIn Feed
16:9 (Landscape): YouTube, Facebook Watch, LinkedIn Video
4:5 (Portrait): Instagram Feed alternative, Pinterest

Your automation pipeline needs to export multiple formats from a single template:

const formats = [
  { name: 'tiktok', width: 1080, height: 1920 }, // 9:16
  { name: 'instagram', width: 1080, height: 1080 }, // 1:1
  { name: 'youtube', width: 1920, height: 1080 }, // 16:9
];

for (const format of formats) {
  const videoBlob = await engine.block.exportVideo(page, {
    mimeType: 'video/mp4',
    targetWidth: format.width,
    targetHeight: format.height,
    framerate: 30,
  });

  await saveToStorage(videoBlob, `${product.id}-${format.name}.mp4`);
}

Template design consideration: Design for the narrowest format (9:16 vertical) as your base. Ensure critical content stays within the central “safe zone” that remains visible when cropped to 1:1 or extended to 16:9.

Browser vs. Server-Side Rendering Trade-offs

Timeline-based video engines support two rendering environments:

Browser rendering (client-side):

Pros: No server infrastructure, renders on user’s device, immediate preview
Cons: Limited by device performance, requires modern web codecs, mobile web not supported
Best for: Interactive editing, single video exports, preview generation

Server-side rendering (Docker):

Pros: GPU acceleration, consistent performance, handles high volumes, licensed codecs
Cons: Requires infrastructure setup, Docker deployment, GPU-enabled instances
Best for: Batch processing, automation pipelines, production deployments

When to use which:

Client-side: User-initiated exports, preview generation, low volumes (< 10 videos/hour)
Server-side: Automated generation, batch jobs, high volumes (100+ videos/hour), production reliability

For hybrid workflows, use client-side rendering for template creation and preview, then switch to server-side for automation and scale.

Production Pitfalls and Monitoring

You’ve built a prototype that generates a few test videos. Moving to production requires thinking through edge cases, failure modes, and operational concerns that don’t surface during development.

Error Handling and Recovery

Video rendering can fail: corrupted assets, invalid template structure, insufficient memory, encoding errors, network timeouts. Your automation needs to handle these gracefully.

Common failure modes:

Asset loading failure: Verify asset URLs before generation, use fallback assets
Memory exhaustion: Reduce batch size, add memory monitoring, restart workers periodically
Template validation: Check template structure before generation, catch invalid variable names early
Encoding errors: Log exact error messages, validate H.264 profile compatibility, check for transparency issues

Failure recovery pattern (simplified):

async function generateVideoWithRetry(job, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const videoBlob = await engine.block.exportVideo(scene, {
        mimeType: 'video/mp4',
        framerate: 30,
      });
      return { success: true, blob: videoBlob };
    } catch (error) {
      if (attempt === maxRetries) {
        await logFailedJob(job, error);
        return { success: false, error: error.message };
      }
      await sleep(Math.pow(2, attempt) * 1000); // Exponential backoff
    }
  }
}

Monitoring and Observability

Production video automation needs visibility:

Key metrics to track:

Generation rate (videos per minute/hour)
Success rate (percentage completing successfully)
Average render time (detect performance degradation)
Queue depth (identify backlogs early)
Storage usage (monitor file growth and costs)
Error types (group by category: asset loading, encoding, timeout)

Alert thresholds:

Success rate drops below 95%
Average render time increases by 50%
Queue depth exceeds 1,000 pending jobs
Error rate for specific template exceeds threshold

Scaling and Cost Optimization

Horizontal scaling (more worker nodes):

Queue depth consistently high
Generation rate needs to double or more
Batch job completion times exceed acceptable limits

Vertical scaling (bigger instances):

Individual video rendering is slow
Memory constraints causing failures
GPU utilization low (CPU bottleneck)

Cost optimization:

Use spot instances for batch processing (significant savings, acceptable interruption risk)
Pre-warm worker nodes during off-peak hours for predictable demand spikes
Archive old generated videos to cold storage after 90 days (S3 Glacier)
Cache frequently used templates and assets in memory

Why Most Teams Don’t Build Video Automation In-House

Before committing to custom development, understand what you’re actually building:

Timeline engines are hard to maintain:

Frame-accurate editing across web, mobile, desktop platforms
Synchronization between video tracks, audio, overlays, effects
Undo/redo systems that work across temporal changes
Template systems with variable timing and duration

Codec licensing is non-trivial:

H.264/H.265 require patent licenses for distribution
Open-source codecs lack broad device compatibility
Licensing costs and compliance complexity

GPU rendering infrastructure requires expertise:

Docker orchestration with GPU passthrough
NVIDIA Container Toolkit configuration
Performance optimization for parallel rendering
Cross-platform consistency (web, server, mobile)

Editor + automation parity is rare:

What you design in the editor must match what automation generates
Same rendering engine, same output quality, no conversion drift
Template sharing between interactive and headless contexts

Timeline-based engines like CE.SDK already solved these problems and expose them via both UI and API. Most teams building video automation use existing engines rather than building from scratch.

Real-World Use Cases

Personalized Video Campaigns at Scale

Scenario: Financial services company sends 10,000 customers personalized year-end investment summary videos.

Template design:

40-second video: 5s intro → 20s portfolio visualization → 10s personalized message → 5s CTA
Variables: customerName, portfolioReturn, topHolding1/2/3, advisorName, advisorMessage

Results:

10,000 videos generated overnight across 50 worker nodes
Average generation time: 15 seconds per video
Total processing time: ~3 hours with parallelization
Delivery: Videos embedded in personalized emails

This mirrors the use case in our CE.SDK Renderer article, where a customer generated “up to 100,000 unique versions” of personalized videos from demographic targeting data.

Scenario: E-commerce brand launches 200 new products. Marketing needs video ads for each product across TikTok (9:16), Instagram Feed (1:1), and YouTube Shorts (9:16) with different CTA variations for A/B testing.

Math: 200 products × 3 formats × 2 CTA variants = 1,200 videos

Distribution: Videos automatically upload to Facebook Marketing API and Google Ads API with campaign tags, ready for activation.

When Video Automation Makes Sense (and When It Doesn’t)

Video automation isn’t right for every situation.

Good Fit: High-Volume, Template-Driven Use Cases

Video automation works well when:

High volume: Dozens, hundreds, or thousands of videos regularly
Repetitive structure: Videos follow consistent patterns with variable content
Data-driven: Content from databases, APIs, or structured data sources
Time-sensitive: Manual production can’t meet speed requirements
Cost-prohibitive: Hiring video editors or agencies for this volume isn’t viable

Examples: Product videos for e-commerce catalogs, personalized marketing campaigns, social media ads with A/B testing, localized versions (same video in 10+ languages)

Poor Fit: Creative-Intensive, One-Off Productions

Video automation struggles when:

Low volume: 1-5 videos per month (manual production is faster)
High creative variation: Each video needs unique storytelling or cinematography
Complex motion graphics: Advanced animation that templates can’t capture
Live-action footage: Real people, locations, scenarios that vary significantly
Artistic direction: Videos requiring subjective creative decisions

Examples: Brand commercials with custom concepts, documentary-style content, event recap videos, testimonial videos

The Middle Ground: Hybrid Approaches

Automated foundation + manual refinement:

Generate base videos automatically
Creative team polishes specific versions for key campaigns
Use automation for scale, manual work for flagship content

Manual template creation + automated generation:

Designer creates polished templates once
Automation generates thousands of variants
No manual work after template is finalized

What to Do Next

Before committing to video automation infrastructure:

Evaluate if your use case is template-driven (repetitive structure with variable data)
Identify which parts must be automated vs. manual (scale vs. creative direction)
Test one template end-to-end with real data (uncover edge cases early)
Measure render time, cost, and failure rates (validate assumptions with real numbers)

Video Automation: Solvable, But Not Simple

Video automation is harder than image automation. The timeline adds complexity. Rendering takes 100x longer. File sizes are larger. Platform requirements are stricter. Audio coordination matters. There’s no magic solution that makes these challenges disappear.

But video automation is solvable when you respect the complexity and build accordingly:

Design templates that handle variability (text length changes, scene duration flexibility, multi-asset coordination)
Choose the right architecture (batch processing for volume, real-time for immediacy, hybrid for collaboration)
Optimize infrastructure (GPU acceleration, server-side rendering, proper codec licensing)
Plan for production (error handling, monitoring, template versioning, cost management)

The teams that succeed with video automation don’t try to make it simple. They acknowledge it’s complex, design systems that handle that complexity, and automate what can be automated while keeping humans involved where judgment matters.

Automation doesn’t eliminate work; it shifts work from repetitive execution to thoughtful system design.

For more context on building creative automation systems that combine human creativity with programmatic scale, see our article on creative automation infrastructure. And if you’re ready to explore how timeline-based engines handle video automation specifically, check out the automated video generation documentation and video timeline editor guide.

Video automation at scale is possible. It just requires understanding what makes video different and building systems that respect those differences.

Ready to test it? Start a free trial or speak with our sales team.

Best Open Source Video Editor SDKs: 2026 Roundup

Vatsala — Mon, 10 Nov 2025 11:13:59 GMT

If you’ve ever tried building a video editing feature from scratch, you know how quickly things can get complicated. Codecs, pipelines, formats, integrations, and performance tuning all come into play.

That’s why developers often turn to open-source SDKs that make video processing more accessible and flexible.

In this guide, we’ll break down some of the most widely used open-source video SDKs, focusing on how easy these tools are to build with, their use cases, and who they’re best for. We’ll also look at where open-source flexibility meets its limits, and how commercial SDKs like IMG.LY can save you time and complexity when building production-ready video products.

1. FFmpeg

Let’s start the comparison with one of the most popular open source video editor SDKs. FFmpeg is a command-line multimedia framework that powers much of the modern video ecosystem. It supports everything from transcoding to advanced automation, making it one of the most widely adopted open-source solutions for video processing.

Features & capabilities: FFmpeg includes transcoding, filtering, cropping, resizing, overlaying, color grading, audio processing, and format conversion. Its flexibility comes from its powerful command-line interface, allowing developers to script nearly any video transformation task.
Platforms: The tool offers broad cross-platform compatibility, running natively on Windows, macOS, and Linux. Its portability allows it to integrate into a wide range of development environments.
Building difficulty & potential roadblocks: FFmpeg is quite difficult to learn and implement for beginners. It requires strong knowledge of scripting, codecs, and CLI-based workflows. Due to its steep learning curve, debugging complex filter chains can be time-consuming, and integrating it into GUI-based environments is also challenging. Developers must also navigate potential H.264 commercial licensing restrictions while using this tool.
Use Cases: FFmpeg is best suited for server-side batch processing, video pipelines, and automated transcoding. While it excels in backend automation, it lacks an interactive editor UI or cross-platform embedding.
Future-proofing, support & pricing: FFmpeg is a mature and stable project used across many industries for years, although its innovation is slow. Most developers using the tool rely on community forums and documentation for troubleshooting and setup help. It’s completely free and open-source, but encoding with certain codecs like H.264 may still require a separate commercial license.
Who it’s for: This video editor SDK is ideal for developers handling large-scale backend video processing or transcoding tasks. It’s powerful and reliable, but not beginner-friendly or suited for products needing a visual editing interface.

Comparison to IMG.LY

FFmpeg is purely backend and lacks an interactive editor or cross-platform embedding capabilities. However, it performs exceptionally well in large-scale automation pipelines, where speed, reliability, and format support are critical. IMG.LY CE.SDK, by contrast, gives you a fully embeddable video editor, UI components, and automation workflows in a unified SDK. All without the complexity of manual scripting.

Check out why IMG.LY is a great alternative to FFmpeg here.

2. MoviePy

The next open source video editor on the list is MoviePy. It’s a Python library for programmatic video editing. It allows developers to edit video through code, making it particularly useful for automated workflows or content prototyping.

Features & capabilities: It supports cutting, compositing, adding text overlays, transitions, and visual effects. Developers can use Python scripts to build video sequences and automate rendering.
Platforms: MoviePy works across all major platforms that support Python. It runs in standard Python environments without requiring extra setup, which makes it accessible for most developers.
Building difficulty & potential roadblocks: MoviePy is easier to use than FFmpeg, but still requires scripting. It’s suitable for Python developers for prototyping, but it can struggle with performance when handling large video files. Developers often face challenges with dependency management and the absence of a real-time UI.
Use cases: This tool is best for automated content generation, research, and rapid prototyping of video workflows. It’s ideal for proof-of-concept projects where scripting flexibility matters more than performance.
Future-proofing, support & pricing: MoviePy is actively maintained by its niche community of developers. While it serves a smaller user base compared to larger frameworks, its reliability and accessibility make it a consistent choice for Python-based video projects. You can find support through community documentation that explains most functions clearly. It’s free and open-source, which makes it especially appealing for experimentation, research, and smaller-scale automation setups.
Who its for: MoviePy is a great choice if you’re a Python developer wanting automation or experimental workflows. However, it’s not ideal for production-scale or interactive use cases due to performance limits and the absence of a UI.

Comparison to IMG.LY

MoviePy is designed for scripting and backend automation. It works best for creating short automated clips or experimental workflows where developers want full control through Python code.

IMG.LY CE.SDK goes beyond backend scripting by offering ready-to-embed editors, automation APIs, and full multi-platform support for web and mobile. This makes it a more complete choice if your team needs both automation and interactive editing within products.

3. OpenShot Library / MLT Framework

The OpenShot Library, built on the MLT Framework, is a C++/Python library for video compositing and timeline editing. It powers the OpenShot video editor and can be integrated into custom desktop applications.

Features & capabilities: It provides timeline editing, transitions, effects, and audio mixing. Developers can implement advanced video composition features similar to traditional desktop editors.
Platforms: The library is cross-platform, working reliably across major desktop operating systems, including Windows, macOS, and Linux. It can be integrated into various environments, which makes it useful if you’re building desktop editors or applications that require timeline-based video editing.
Building difficulty & potential roadblocks: The integration complexity is medium-high, as the API can be unstable, and managing dependencies can be difficult. Developers may also encounter documentation gaps, making it challenging to set up and maintain. Integrating the library into a GUI is not straightforward, and limited server-side automation means it is less suited for scalable or cloud-based workflows.
Use cases: OpenShot Library is suitable for you if you’re looking to build desktop-based editors, experimental timeline tools, or custom video composition applications.
Future-proofing, support & pricing: The framework is actively maintained and works well for desktop video editing applications. However, it is not designed for server-side automation or large-scale deployment. The support is community-driven, so the updates and fixes depend on volunteer contributions, which can affect long-term reliability for enterprise use. The framework is free and open-source, making it accessible to anyone looking to experiment or build lightweight video editing tools.
Who its for: OpenShot/MLT fits developers building desktop-focused editors who need timeline control but can manage API and dependency challenges. It’s less practical for cloud or mobile workflows.

Comparison to IMG.LY

While OpenShot/MLT excels in desktop-based timeline editing, it is better suited for applications that prioritize local processing and desktop workflows.

IMG.LY CE.SDK, on the other hand, offers embedded, cross-platform editors with built-in automation, making it a more versatile choice for SaaS and enterprise environments that need scalability, faster integration, and an intuitive editing experience for end users.

4. GStreamer

GStreamer is a multimedia framework for building video and audio pipelines. It’s a foundational tool in many streaming and broadcast systems and allows developers to create complex pipelines for capturing, processing, and outputting media in real time.

Features & capabilities: This open source video editor supports real-time processing, filters, and complex pipelines for media processing and streaming. Developers can construct customized processing chains for advanced workflows.
Platforms: GStreamer is cross-platform, supporting multiple programming languages with bindings for C, Python, Java, and Rust. You can run it on Windows, macOS, and Linux, and integrate it into both desktop and server environments. This makes it flexible for building diverse media workflows.
Building difficulty & potential roadblocks: Building with GStreamer is complex. It requires deep knowledge of pipelines and plugins, and debugging can be time-intensive. UI integration adds another layer of difficulty.
Use cases: Despite its building difficulty, developers use GStreamer for real-time video streaming, server-side processing, and for broadcast applications requiring precise control over data flow.
Future-proofing, support & pricing: GStreamer is mature and widely used in the media industry. It continues to be a reliable choice for building streaming and broadcast systems, but it requires significant development expertise to configure and maintain. Developers often rely on community resources for support, though some commercial assistance is available. It’s free and open-source, which makes it accessible for experimentation and custom media workflows.
Who it’s for: The tool is ideal for developers who need real-time video pipelines or streaming capabilities. But it may not be a fit for applications requiring easy embedding or interactive UIs.

Comparison to IMG.LY

GStreamer operates at the infrastructure level, ideal for backend media processing where you need precise control over data flow and performance. It gives you the flexibility to build custom video pipelines, but doing so requires significant technical effort and time.

IMG.LY provides a ready-to-embed, interactive editor with automation and multi-platform support, so you can focus on building user experiences rather than managing low-level media infrastructure.

5. Remotion

Remotion enables developers to create videos programmatically using React components. Developers can define video scenes, transitions, and animations directly in React. While it’s a flexible approach, building long or high-resolution videos can demand careful optimization to ensure smooth performance.

Features & capabilities: It supports timeline-based video composition, text overlays, animations, audio integration, and programmatic transitions, all rendered through React. Developers can create videos entirely with code, defining scenes and effects inside React components.
Platforms: Remotion is built for Node.js and browser-based environments via headless rendering. You can render videos directly in the browser or on a Node.js server, which gives developers flexibility in how they deploy and automate video creation.
Building difficulty & potential roadblocks: Building difficulty using Remotion is at a medium level because it requires React and JavaScript proficiency. The potential issues you might face while using this include debugging challenges with high-resolution or long-duration renders, along with limited server-side optimization that can affect scalability and performance.
Use cases: Remotion is well-suited for automated video generation, marketing content, and dynamic SaaS video templates. You can use it to create personalized videos or promotional content at scale, using data or user inputs to generate customized output.
Future-proofing, support & pricing: The tool is actively maintained and is built with a modern web stack using React and TypeScript, which ensures long-term stability and adaptability. You can find help through detailed documentation, GitHub issues, and an engaged developer community. While it doesn’t offer formal support, its open-source nature encourages collaboration and contributions that keep the framework improving. Like other tools we’ve discussed, this is also free and open-source, making it an accessible choice for developers exploring programmatic video creation.
Who it’s for: Based on its features and capabilities, it’s best for React developers who want to generate videos programmatically. However, it lacks an interactive editor and may not scale easily for enterprise automation.

Comparison to IMG.LY

Remotion excels in programmatic video creation, allowing you to build videos entirely through code with precision and flexibility. However, it lacks interactive editing tools or native SDKs that can be directly embedded into products.

IMG.LY CE.SDK, by comparison, gives you both automation capabilities and user-facing editors across platforms, helping teams build complete video experiences without maintaining separate systems. You can find more information on how IMG.LY compares to Remotion here.

6. IMG.LY CE.SDK

A commercial solution that offers enterprise-grade functionality and developer-friendly implementation.

IMG.LY CreativeEditor SDK is built for interactive video editing and automation. It gives developers the tools to build powerful, embeddable video editors without the complexity of starting from scratch. It balances the flexibility developers expect from open-source frameworks with the stability and support of a commercial-grade platform.

Features & capabilities: It includes timeline editing, multi-layer video composition, templates, AI-assisted editing, transitions, video export, and server-side automation. The SDK simplifies complex workflows by offering prebuilt editors and APIs for integration.
Platforms: Available on Web, iOS, Android, React Native, Flutter, Node.js, and Electron. This broad platform support allows developers to build consistent video editing experiences across desktop, mobile, and web environments. Whether you’re integrating into an existing app or launching a new product, the IMG.LY CE SDK ensures a unified editing interface and workflow across all platforms.
Building difficulty & potential roadblocks: Integration is straightforward, with minimal setup required, making it the easiest SDK to build with among all the tools we compared in this list. Most implementation effort lies in customizing workflows rather than solving technical barriers. This simplicity comes from well-documented APIs, prebuilt UI components, and consistent cross-platform behavior, which together reduce the development time. Potential roadblocks are minimal, allowing you to focus on building product features instead of managing infrastructure or compatibility issues.
Use cases: IMG.LY is best suited for SaaS apps, marketing automation, social content tools, and video personalization platforms that need seamless interactive editing.
Future-proofing, support & pricing: Frequent releases, an expanding AI roadmap, and a scalable engine ensure long-term reliability for developers and enterprises. You get access to dedicated onboarding engineers, solution architects, and enterprise SLAs that make implementation and scaling easier. Pricing follows a custom enterprise licensing model, offering flexibility for different business needs. But you can take a free trial to get started.
Who it’s for: IMG.LY is designed for companies that need scalable, interactive video editing and automation in one package. Its ideal for teams seeking low development overhead and enterprise-grade reliability.

Why IMG.LY stands out as the best open source video editor SDK

IMG.LY combines interactive editing, server-side automation, and multi-platform SDKs in one stack, giving you everything needed to build scalable, modern video products. You can create, edit, and automate workflows within the same ecosystem without relying on separate tools. None of the open-source options on this list deliver this level of integration and consistency across web, mobile, and desktop environments.

Check out how IMG.LY has helped over 600 innovative startups, government entities, and Fortune 500 companies to streamline their design, video, and photo editing workflows. Read our case studies here.

Overview table

Here’s a brief overview of all the video editor SDKs we’ve discussed to help you choose the right one.

Feature / Use Case	FFmpeg	MoviePy	OpenShot / MLT	GStreamer	Remotion	IMG.LY CE.SDK
Timeline editing	❌ No GUI timeline; timeline-like operations only possible via scripting in CLI.	⚠️ Timeline functionality simulated through Python scripts; suitable for small-scale projects.	✅ Native timeline-based video editing with GUI; library supports composition though GUI integration can be limited.	⚠️ Pipeline-centric; timeline must be assembled programmatically, no native GUI timeline.	✅ Timeline created programmatically using React components; no interactive GUI timeline.	✅ Full-featured interactive timeline, multi-layer support, drag-and-drop UI
Filters & Effects	✅ Supports CLI-based filters, overlays, color correction; requires manual configuration	✅ Python effects library; scripting needed to combine filters	✅ Built-in video effects and transitions; limited prebuilt UI	✅ Plugin-based filters; flexible but requires pipeline setup	✅ Animations, text overlays, audio effects via scripting; programmatic transitions	✅ AI-enhanced effects, transitions, multi-layer composition, background removal
Server-side automation	✅ Excellent for backend batch processing and transcoding; CLI automation is robust	✅ Python scripting enables automated content generation	⚠️ Limited server-side automation; mainly library-level	✅ Suitable for large-scale pipelines and streaming	✅ Automated video rendering via Node.js; works with server-side scripts	✅ Full automation APIs; combines server-side rendering + interactive editing in one SDK
Ease of embedding	❌ CLI only; difficult to embed in apps	⚠️ Python scripts can be integrated, but no GUI	⚠️ Library-only; integration requires custom UI	⚠️ Complex pipelines; requires deep development knowledge	⚠️ Node.js / headless browser integration possible; UI must be custom-built	✅ Ready-to-embed SDKs for Web, iOS, Android, React Native, Flutter; includes prebuilt editors
Difficulty building / integration	High — complex CLI usage, codec handling, and scripting required	Medium — Python knowledge required, scripting-intensive	Medium-High — API can be unstable, dependencies must be managed	High — pipeline setup and plugin management are challenging	Medium — React knowledge needed; performance optimization may be required	Low — SDK provides prebuilt editor components and APIs for easy integration
Potential roadblocks	CLI complexity, codec licensing (H.264), debugging pipelines, no GUI	Large videos may be slow, performance issues, dependency management	API instability, documentation gaps, integration with UI requires effort	Steep learning curve, pipeline debugging, plugin version conflicts	Rendering performance for long/high-resolution videos, debugging programmatic scripts	Minimal; most challenges relate to workflow decisions during integration
Cross-platform support	✅ Desktop/server; Windows, macOS, Linux	✅ Desktop/server; cross-platform Python	✅ Desktop/server; cross-platform	✅ Desktop/server; cross-platform	✅ Node.js + headless browser; works across server environments	✅ Web + mobile + desktop; consistent UI across platforms
AI / Automation	❌ None	❌ None	❌ None	❌ None	⚠️ Programmatic automation via scripting	✅ AI-assisted editing, automation workflows, plugin ecosystem
Scalability	✅ Excellent for batch and server pipelines	⚠️ Limited; performance depends on scripting and hardware	⚠️ Medium; suitable for desktop-scale apps	✅ Large-scale pipelines, real-time streaming	✅ Programmatic generation can scale, but may need server resources	✅ Enterprise-grade; cloud-ready, multi-platform, high-volume video workflows
Pricing	Free, OSS; H.264 encoding may require commercial license	Free, OSS	Free, OSS	Free, OSS	Free, OSS	Custom enterprise licensing
Best For	Developers building automation pipelines, server-side video processing, or streaming workflows	Python developers building scripted video content and automation	Developers building desktop-focused video apps with timeline editing	Developers needing real-time pipelines or advanced streaming	Developers using React for programmatic video generation and templating	Companies needing interactive, embedded video editing + AI automation + multi-platform SDK

Choosing the right video SDK: How to make the right choice

When choosing the right video SDK, the best option depends on what you’re building and who it’s for.

Choose open-source SDKs like FFmpeg, MoviePy, OpenShot/MLT, GStreamer, and Remotion if you prefer flexibility and complete control. These tools are free, customizable, and well-suited for automation, research, or proof-of-concept projects. However, they often require significant development time, scripting knowledge, and ongoing maintenance, which can slow down production or scaling.

For teams building commercial or enterprise-ready video products, commercial SDKs offer a more practical route. They save you time on integration, provide dedicated support, and come with consistent updates to ensure reliability across platforms.

That’s why developers choose IMG.LY SDK.

It combines interactive editing, automation, and cross-platform SDKs in a single stack, something open-source solutions can’t fully replicate. It’s designed for SaaS, social, and enterprise use cases where scalability, ease of use, and performance matter most. You can embed editors, automate workflows, and scale confidently without managing multiple systems.

So if you want a complete, production-ready platform with AI features, automation, and an intuitive editing experience, IMG.LY is the right choice.

Get in touch with our team to see how it’s deployed in over 500 applications and used by millions of users across 29 countries.

Start a free trial or book your call today!

Animate Between Images - AI-Native Video Workflows with CE.SDK and Veo 3

Jan — Wed, 22 Oct 2025 14:10:31 GMT

With the release of Veo 3.1, we wanted to show just how effortless it is to embed generative AI capabilities directly into creative workflows. In this quick demo, we integrated Veo 3 into CreativeEditor SDK (CE.SDK) enabling users to animate between two still images in just a few clicks.

From Still Images to Motion

In our demo, we start with two images of the same person, one wearing a hat, the other without. Inside the editor, users simply select both images, click the AI context button, and choose “Animate between images.”

The images are loaded into a side panel where users can optionally add a text prompt to guide the transition. Once generated, the resulting short video is placed directly on the canvas ready for editing, compositing, or export.

What’s particularly impressive is the generation speed. In this example, Veo 3.1 produced a smooth 8-second transition in just 9 seconds a major improvement compared to earlier versions. This speed makes iterative creative workflows feel fluid and responsive, bridging the gap between prompt-driven generation and real-time editing.

Try It Out

You can check out the implementation on GitHub and give Veo 3.1 a spin inside your CE.SDK instance (sign up for a free trial if you haven’t already).

A Glimpse of AI-Native Editing

This simple feature highlights how easy it is to make your editor AI-native, combining traditional editing tools with generative intelligence.

Practical use cases include:

E-commerce
Show products “in action” or animate between styles and configurations.
Marketing
Create quick product reveal animations from static assets.
Content creation
Generate short motion clips or “tween” between creative scenes.

And if users want longer clips, they can simply add another 8-second track and transition seamlessly into the next.

The Future of Creative Workflows

With Veo 3 integrated, CE.SDK becomes a powerful playground for AI-driven creativity from image-to-video to scene interpolation and contextual animation.

We already empowered over 600 innovative startups, government entities, and Fortune 500 companies to add powerful design, video, and photo editing workflows to their products. Get in touch, to see how we can do the same for you.

How to Run FFmpeg on AWS Spot Instances for Scalable, Low-Cost Video Processing

Robin — Mon, 06 Oct 2025 08:25:43 GMT

In this guide, we’ll explore how to harness Amazon Web Services (AWS) for video processing with FFmpeg. You’ll learn why the cloud is so helpful for handling resource-heavy video tasks, how to set up your AWS Spot Instance and S3 bucket, and how to run FFmpeg jobs at scale. Along the way, we’ll get acquainted with AWS Spot Instances, a cost-saving compute option that can make large-scale video processing far more affordable. If you’re new to the tool itself, start with our Ultimate Guide to FFmpeg.

By the end of this article, you’ll know how to:

Set up AWS with S3 storage and a Spot instance.
Install and test FFmpeg in the cloud.
Run video processing jobs on a Spot Instance while keeping costs low.

Whether you’re new to cloud computing or just looking for a practical way to scale your FFmpeg workflows, this guide will walk you through the essentials step by step.

Lets dive into by setting up the AWS environment.

Set Up Your AWS Environment

Before you can run FFmpeg in the cloud, you need to prepare your AWS environment. This involves creating an account, setting up storage, and launching an instance where you’ll install and run FFmpeg. Let’s break it down step by step.

Create an AWS Account and S3 Bucket

If you don’t already have one, sign up for an AWS account. Once logged into the AWS Management Console, the first thing to set up is an Amazon S3 bucket. S3 (Simple Storage Service) is AWS’s object storage system, and it’s ideal for hosting both your input and output video files. For example, you can upload raw videos to S3, process them with FFmpeg, and then store the converted files back in the same bucket. When creating your bucket, be sure to choose a region close to where you expect to run your compute resources, as this reduces latency and transfer costs.

Launch a Spot Instance

Spot Instances

AWS Spot Instances are virtual machines that let you access unused cloud capacity at significantly reduced prices, sometimes up to 90% cheaper than regular On-Demand Instances. The catch is that they can be interrupted by AWS with little notice if the capacity is needed elsewhere. For workloads like video processing, which can be restarted or distributed across multiple nodes, Spot Instances are a perfect fit. They allow developers to take advantage of lower costs without sacrificing performance, provided they design workflows with fault tolerance in mind. This balance of affordability and scalability makes Spot Instances a popular choice for running FFmpeg jobs in the cloud.

Setup a Spot Instance

From the AWS console, go to the EC2 dashboard and launch a new instance. Choose Ubuntu Server or Amazon Linux as the operating system, since both have excellent community support and are well-documented. When selecting an instance type, start with something lightweight like t2.micro (eligible for the free tier), but keep in mind you may need more powerful instances (with higher CPU or GPU) for larger video workloads. Open the Advanced Details, under Purchasing option, check the box labeled Spot Instances to tell AWS you want to use discounted spare capacity instead of On-Demand pricing. You can also set a maximum price per instance hour if you want to control costs more tightly, but in most cases leaving it at the default ensures you’ll simply pay the current Spot market price, which automatically adjusts with supply and demand and is never higher than the standard On-Demand rate.

You can also choose between a persistent or one-time request type in the Customize Spot Instance options.

Once the instance is created, you can confirm that its lifecycle is spot under the instance details section. This indicates that a Spot Instance has been successfully created.

Connect to Your Instance via SSH

Once your Spot instance is running, you can connect to it remotely using SSH (Secure Shell). During instance setup, AWS will prompt you to create or download a key pair (.pem file). Keep this file safe, it’s your private key for secure access. On your local machine, open a terminal at the location where you saved the key pair file and run:

ssh -i "your-key.pem" <username>@<your-instance-public-dns>

Replace your-key.pem with the path to your private key file and <your-instance-public-dns> with the public DNS address of your Spot instance (visible in the console). You can get the exact SSH command for your instance by clicking the Connect button in the EC2 dashboard and selecting the SSH client option. Once connected, you’ll be inside the environment where you can install FFmpeg and start running commands. For a smooth operation, make yourself familiar with basic shell commands for the operating system you opted for in the instance. Examples for various shell commands on a linux system can be found here.

Install FFmpeg on the Spot Instance

With your spot instance up and running, the next step is to install FFmpeg so you can start processing videos. Installation is straightforward, and once complete, you’ll test it with a simple command to confirm everything works.

Update Your Package Lists

Before installing any software, it’s a good practice to make sure your system’s package list is up to date. Run:

sudo apt update && sudo apt upgrade -y

This command refreshes the list of available software and ensures your instance has the latest security patches. (If you’re using Amazon Linux instead of Ubuntu, replace this with sudo yum update -y.)

Install FFmpeg

On Ubuntu, FFmpeg is available directly through the official package repository:

sudo apt install ffmpeg -y

For Amazon Linux, the process may require enabling extra repositories first:

sudo amazon-linux-extras enable epel
sudo yum install ffmpeg -y

This will download and install FFmpeg along with its dependencies. To avoid dependency issues, consider running FFmpeg inside a Docker container.

Verify the Installation

After installation, check that FFmpeg is available by running:

ffmpeg -version

You should see output showing the installed FFmpeg version and configuration details.

Run FFmpeg on a Spot Instance

Now that you’ve installed FFmpeg on a regular Spot instance, it’s time to make evrything work together. In this section, upload a video to Amazon S3, and then run FFmpeg on the spot instance to process the video and save the output.

While it’s possible to upload videos directly to the instance and process them there, this approach isn’t reliable since the instance can be terminated at any time when AWS reclaims capacity. The better and safer practice is to use the S3 bucket, which ensures your files are stored safely and don’t need to be re-uploaded each time. This is especially important when working with large video files or multiple inputs, as S3 provides a more reliable and scalable storage solution. Prefer Google Cloud? See our tutorial on running FFmpeg on GCP for similar steps.

Upload a Sample Video to S3

Before running FFmpeg, you’ll need a video file available in your S3 bucket. From your local machine terminal (not the instance terminal), use the AWS CLI to upload a sample file:

aws s3 cp sample.mp4 s3://your-bucket-name/input/sample.mp4

This stores sample.mp4 inside the input folder of your bucket. Organizing files into input and output directories is a good practice for video workflows. You can do the same using the S3 dashboard in AWS. If your workflow involves cropping or trimming videos before upload, check out our Flutter guide on crop and trim videos in Flutter.

If you are using AWS CLI for the first time in you machine, then setup your credentials using aws configure command. See here for more help.

Run FFmpeg on the Spot Instance

Once your Spot Instance is running, connect to it via SSH (the same way you did earlier). Before we get the video from the S3 bucket, you should set an IAM Role for the instance to access the S3 bucket. For that, go to IAM>Roles>Create Role, select EC2 from use case dropdown menu. On the next page, search for AmazonS3FullAccess and select it, give a role name and finaly create it. Now the instances can access the files from the bucket.

Since the instance is a new system, you will need to install awscli from here. You will also have to setup credentials using aws configure command as mentioned above. Then, copy the input file from S3 down to your instance:

aws s3 cp s3://your-bucket-name/input/sample.mp4 sample.mp4

Now, run FFmpeg to process the video. For example, converting MP4 to WebM:

ffmpeg -i sample.mp4 output.webm

After processing, upload the result back to S3:

aws s3 cp output.webm s3://your-bucket-name/output/output.webm

You can now access your processed file directly in S3 and even configure it for delivery via Amazon CloudFront if needed.

By running FFmpeg jobs on Spot Instances, you can save up to 90% of costs compared to On-Demand pricing, making this setup ideal for batch video processing.

You can refer to this github repository for examples.

Spot Instances vs AWS Lambda for FFmpeg

When running FFmpeg on AWS, you have more than one option for compute. Two of the most common choices are AWS Lambda and EC2 Spot Instances. Each has its strengths and trade-offs, and the best fit depends on the type of video workload you’re running.

AWS Lambda for Lightweight Tasks

AWS Lambda is Amazon’s serverless compute service—you don’t need to manage servers, and your code runs only when triggered. It’s easy to set up and integrates seamlessly with other AWS services such as S3 and API Gateway. For example, you can automatically run an FFmpeg job as soon as a new video is uploaded to your S3 bucket.

However, Lambda comes with limitations: functions can only run for a maximum of 15 minutes, and they have limited temporary storage (512MB by default, extendable up to 10GB). This makes Lambda a great choice for lightweight tasks like generating thumbnails, clipping short videos, or converting smaller files, but not for large-scale processing.

EC2 Spot Instances for Heavy Workloads

EC2 Spot Instances, by contrast, are designed for long and heavy workloads. Spot Instances give you access to unused EC2 capacity at discounts of up to 90% compared to On-Demand pricing. Since they are full-fledged EC2 machines, Spot Instances don’t impose runtime limits or storage caps.

This makes them ideal for large video files, long-running transcodes, and GPU-accelerated workloads such as 4K/8K encoding or complex filter pipelines. The trade-off is that Spot Instances can be interrupted by AWS with two minutes of notification when EC2 needs the capacity back, so jobs need to be either restartable or split into smaller tasks.

Rule of Thumb

A simple guideline to choose between the two is:

Small, quick jobs → use AWS Lambda
Large, long jobs but restartable → use EC2 Spot Instances

By matching the right AWS service to your workload, you get the best balance of cost, performance, and reliability for running FFmpeg in the cloud.

Handling Spot Interruptions

Two-Minute Warning

One important aspect of using Spot Instances is that they can be interrupted at any time if AWS needs the capacity back. When this happens, AWS provides a two-minute warning before the instance is terminated. For video processing workflows, this means you need to plan for interruptions so that work isn’t lost midway.

Break Videos into Smaller Chunks

A common strategy is to break long videos into smaller chunks before processing. For example, instead of transcoding a two-hour video in one run, you can split it into 10-minute segments, process each segment separately, and then stitch them back together later. This way, if an interruption occurs, only the current segment is affected rather than the entire job. FFmpeg supports segmenting input files using options like -ss (start time) and -t (duration), which makes this approach straightforward.

Store Partial Results in S3

Another best practice is to store partial results in Amazon S3 as soon as they are completed. For instance, after processing each video chunk, upload the output back to your S3 bucket immediately. This ensures that even if the Spot Instance is terminated, your progress is safe, and you can resume processing from the last completed chunk instead of starting over. Using S3 as a central storage layer makes your workflow fault-tolerant and resilient against interruptions.

Reliable and Cost-Effective Workflows

By designing your FFmpeg workflows with these strategies, you can take full advantage of the cost savings of Spot Instances without risking data loss. This makes Spot instances a reliable option even for long-running or large-scale video processing jobs, provided you build in restart and recovery mechanisms.

Tips & Next Steps

Once you’ve set up FFmpeg on AWS and experimented with Spot Instances, there are a few ways to refine your workflow and explore more advanced options. These next steps will help you get the most out of your cloud-based video processing setup.

1. Try Different Instance Types

AWS offers many EC2 instance families, each optimized for different workloads. For general transcoding and compression tasks, compute-optimized instances like the c5 family are often a good fit, providing strong CPU performance at a reasonable cost. If you’re working with high-resolution videos or need hardware acceleration, GPU-based instances such as the g4dn family can dramatically speed up encoding with FFmpeg’s GPU-enabled libraries. Experimenting with instance types lets you balance performance and cost for your specific use case.

2. Orchestrate Jobs with SQS or Step Functions

As your workflows grow, running jobs manually can become inefficient. AWS provides orchestration tools to help automate FFmpeg jobs. Amazon SQS (Simple Queue Service) can be used to queue video processing tasks, allowing multiple Spot Instances to work on jobs in parallel. For a full Docker‑based pipeline with REST APIs and worker queues, see our production‑ready batch video processing server guide. For more complex workflows, AWS Step Functions provide a way to chain together multiple tasks such as fetching input files, running FFmpeg, handling errors, and saving outputs into a reliable, serverless state machine. These services ensure your video pipelines are scalable and fault-tolerant without requiring you to build custom orchestration logic.

3. Consider CE.SDK as an Alternative

If you’re building custom editing workflows on AWS, consider integrating a frontend editor like our CreativeEditor SDK. While FFmpeg and AWS handle the heavy lifting of transcoding and storage, CE.SDK provides a powerful in-browser editing interface for trimming, adding overlays, applying filters, or creating design templates. This combination allows you to deliver both scalable backend video processing and interactive, user-facing editing tools, similar to what platforms like Canva provide.

Conclusion

Running FFmpeg on AWS Spot Instances gives you a powerful and cost-effective way to handle large-scale video transcoding. By leveraging Spot pricing, you can cut costs by up to 90% compared to On-Demand instances, making it possible to process even long, high-resolution videos without breaking the budget. With strategies like chunking videos and storing partial outputs in S3, interruptions become manageable, allowing you to balance affordability with reliability.

For smaller, lightweight, or event-driven workflows, AWS Lambda is a perfect complement. Its serverless nature makes it easy to trigger FFmpeg jobs automatically, for example, generating thumbnails or clipping short clips as soon as a file is uploaded to S3. Although Lambda has strict runtime and storage limits, it shines when simplicity and automation are priorities.

As a next step, you can begin experimenting with automation and orchestration using services like SQS, Step Functions, or even AWS Batch to build resilient pipelines that combine the strengths of both Spot Instances and Lambda. For teams looking for a fully managed alternative, CE.SDK is also worth exploring.

By now, you’ve seen how to install FFmpeg, run it on spot instance, and handle serverless workflows with Lambda. With these tools and strategies, you’re ready to design flexible, scalable, and affordable video processing pipelines in the AWS cloud.

Join 3,000+ creative professionals who get early access to new features and updates—subscribe.

How To: Video Generation With Javascript

Antonello — Wed, 22 Jan 2025 09:53:00 GMT

Follow this tutorial to learn how to programmatically create videos in JavaScript directly in the browser.

In 2025 video is firmly established as powerhouse for engagement, gaining even more traction with the rise of short-form videos. After all, as humans, we naturally resonate with video content more than any other type of media.

The problem is that video generation is challenging—and doing it within the browser through JavaScript makes things even more complex. Fortunately, production-grade solutions like CreativeEditor SDK (CE.SDK) make the process not only possible but also easy to implement.

In this guide, you will learn how to programmatically generate videos in the browser via JavaScript using CreativeEditor SDK. We will cover various applications, including merging videos, adding audio tracks, and even integrating AI features.

Let’s dive in!

Why Programmatic Video Generation Matters

Videos are dominating social media, marketing, and online platforms such as e-commerce platforms (e.g., to showcase products). Numerous studies have proven that video is the most effective form of media for marketing, as it resonates deeply with us.

As the demand for personalized and dynamic content increases, traditional video production methods are becoming inefficient. The proliferation of channels with different demands on size and video quality as well as the need to create personalized videos at scale means that automated video creation is becoming indispensable. Most existing solutions run batch processing of videos on the server after gathering input data from various source. However, while server-side processing quickly becomes costly and introduces significant communication overhead client devices have become more powerful than ever.

CreativeEditor SDK bridges that gap by simplifying programmatic video generation in JavaScript within modern browsers. With CE.SDK, you can automate repetitive tasks and define scalable, personalized, and engaging video creation workflows, ensuring each generated video resonates with its target audience.

Get Started with CE.SDK for JavaScript

Now that we established why programmatic video generation is important, you are ready to get started with the tool designed for it: CreativeEditor SDK.

Learn how to integrate CE.SDK for video editing into a vanilla JavaScript application!

You can access the code for this tutorial on Github.

Requirements

The only prerequisites to follow this tutorial are:

A modern browser: CreativeEditor SDK runs directly in your browser’s JavaScript engine, so no additional setup is required. Any browser supporting WASM is enough.
A CreativeEditor SDK license: If you do not have one yet, sign up for a free trial.

If your web application uses Node.js, ensure you have the latest stable versions of both Node.js and NPM installed.

Import the library

In a vanilla JavaScript application, create a JavaScript module file (e.g., video-editor.js) and import the CreativeEditor SDK engine library inside it:

import CreativeEngine from '<https://cdn.img.ly/packages/imgly/cesdk-engine/1.42.0/index.js>';

Note: In this example, the SDK is served from our CDN for convenience. In a production environment, it is recommended to host all assets and libraries on your own servers for improved control and performance.

To use the video-editor.js module in an HTML page, import it with this line:

<script type="module" src="video-editor.js"></script>

Note the type=“module” attribute tells the browser to treat the script as an ES6 module, so that you can use import and export statements.

Alternatively, if you are using a module bundler like Webpack, Rollup, Parcel, or Vite, add the @cesdk/engine npm package to your project’s dependencies:

npm install @cesdk/cesdk-js

In this case, you can then import the headless SDK into your JavaScript code as follows:

import CreativeEngine from '@cesdk/engine';

Well done!

Setting Up Your Environment

In your JavaScript module, right below the library import, initialize the CreativeEditor SDK headless engine using the following code:

// src/video-editor.js

import CreativeEngine from '<https://cdn.img.ly/packages/imgly/cesdk-engine/1.42.0/index.js>';

// your CE.SDK license and user configs
const config = {
  license: '<YOUR_LICENSE>',
};

CreativeEngine.init(config).then(async (engine) => {
  // attach the engine canvas to the DOM
  document.getElementById('cesdk_container').append(engine.element);

  // do something with your instance of CreativeEditor SDK...

  // detach the engine and clean up its resource
  engine.element.remove();
  engine.dispose();
});

Replace <YOUR_LICENSE> with the license key provided in your CreativeEditor SDK subscription.

Ensure that the HTML page importing video-editor.js contains the following <div> element with the cesdk_container ID:

<div id="cesdk_container" style="width: 100%; height: 100vh;"></div>

The CreativeEngine.init() method will mount the editor engine component within this div.

Congratulations! You have successfully integrated the video editing engine.

Load a Video

Use the logic below inside the body of the CreativeEndine.init() callback function to load a video in CE.SDK:

// initialize a new video scene
const scene = engine.scene.createVideo();

// create a page block and attach it to the scene
const page = engine.block.create('page');
engine.block.appendChild(scene, page);

// set the dimensions of the video page
engine.block.setWidth(page, 720);
engine.block.setHeight(page, 1280);

// create a graphic block to hold the video content
// and set its shape to a rectangle
const video = engine.block.create('graphic');
engine.block.setShape(video, engine.block.createShape('rect'));

// create a fill type for the video and set the video file URI
const videoFill = engine.block.createFill('video');
engine.block.setString(
  videoFill,
  'fill/video/fileURI',
  '<https://cdn.img.ly/assets/demo/v2/ly.img.video/videos/pexels-drone-footage-of-a-surfer-barrelling-a-wave-12715991.mp4>'
);

// apply the video fill to the video graphic
engine.block.setFill(video, videoFill);

// create a track block to manage the video timeline
// and attache the track to the page and the video graphic
// to the track
const track = engine.block.create('track');
engine.block.appendChild(page, track);
engine.block.appendChild(track, video);

// make the track cover the parent block
engine.block.fillParent(track);

This initializes a video scene using createVideo() and creates a "graphic" block to display the video in a rectangle, adding it to the page. Next, it loads a video from a remote resource and appends it as a track to the parent block.

For a complete explanation of how the above code works, refer to the documentation.

If you comment out the cleanup part, this is what you should see in your browser application:

Notice how the video from the remote URI has been loaded into the

element where CreativeEditor SDK is attached to the page’s DOM.Time to explore more programmatic video generation features!

Programmatic Video Generation Features

Follow the use cases below to see how CE.SDK makes automated video generation in JavaScript easier.Generate Video in Different Formats CE.SDK supports video export in MP4 format (with MP3 audio) with different resolutions and aspect ratios.To export a video with a specific aspect ratio, first adjust the width and height of the parent block to which the video block is attached:

// specify an aspect ratio of 9:16 (ideal for vertical videos)
engine.block.setWidth(page, 720);
engine.block.setHeight(page, 1280);

In this case, the video will be automatically adjusted to fit the specified dimensions. Horizontal videos will be cropped to match the defined vertical outline of the block, while vertical videos will be rendered as they are within the specified block.

Next, you can export the video with the configured 9:16 aspect ratio and a given resolution with:

// define the MIME type for the video export (MP4 format)
const mimeType = 'video/mp4';

// define a callback function to track the progress of video rendering and encoding
const progressCallback = (renderedFrames, encodedFrames, totalFrames) => {
  console.log(
    'Rendered', // log the number of rendered frames
    renderedFrames,
    'frames and encoded', // log the number of encoded frames
    encodedFrames,
    'frames out of', // log the total frames to be processed
    totalFrames
  );
};

// define the video export options
const videoOptions = {
  duration: 5, // video duration in seconds
  framerate: 30, // video framerate (frames per second)
  targetWidth: 480, // target video width in pixels (for the resolution)
  targetHeight: 852, // target video height in pixels (for the resolution)
};

// export the page as an MP4 video
const blob = await engine.block.exportVideo(
  page, // the page containing the design video block
  mimeType, // the MIME type for the video (MP4)
  progressCallback, // the callback to track video rendering progress
  videoOptions // the options for the video export
);

// create an anchor element to trigger the video download
const anchor = document.createElement('a');
anchor.href = URL.createObjectURL(blob);
anchor.download = 'video.mp4'; // output video name
anchor.click();

The exportVideo() function produces a blob video file of the specified MIME type. Note that the export occurs over multiple iterations of the update loop, with a frame being encoded in each iteration.

The targetWidth and targetHeight options are optional. If used, the video will be resized to fit the target dimensions while maintaining its aspect ratio. This is useful for setting a specific resolution for the output video while preserving the parent block’s width and height on the output video. If omitted, the produced video will match the resolution of the width and height specified with setWidth() and setHeight(), respectively.

The progressCallback() function will track and display the video generation progress in the browser’s console:

The final lines of the above code snippet trigger the download operation, mimicking the action of clicking a link to download a resource. This is a common JavaScript technique to programmatically trigger a file download.

Once the export process completes, the output video file (video.mp4) will be automatically downloaded to the browser’s download directory. Open its properties and this is what you will see:

Note the video file’s resolution and the fact that its duration is set to 5 seconds, as defined by the duration attribute.

These options enable you to easily create clips with different lengths, aspect ratios, and resolutions, optimized for various audiences and social media platforms like LinkedIn, Instagram, Facebook, X, TikTok, YouTube, and more.

Add Audio Tracks

Adding an audio track for narration or background music to your clip is straightforward. First, create an “audio” block and add your audio resource to it:

const audio = engine.block.create('audio');
engine.block.appendChild(page, audio);
engine.block.setString(
  audio,
  'audio/fileURI',
  'https://cdn.img.ly/assets/demo/v1/ly.img.audio/audios/far_from_home.m4a'
);

Next, you can easily adjust the volume, and apply fade-in and fade-out effects as follows:

// set the volume level to 80%
engine.block.setVolume(audio, 0.8);

// start the audio after 1 second of playback
engine.block.setTimeOffset(audio, 1);

// set the audio block's duration to 5 seconds
engine.block.setDuration(audio, 5);

Here is breakdown of the functions used above:

setVolume(): Adjusts the audio volume of the block, with a range from 0 (0%) to 1 (100%).
setTimeOffset(): Sets the time offset of the block relative to its parent. This determines when the block starts playing in the timeline.
setDuration(): Sets the playback duration of the audio block in seconds, defining how long the block will play during the scene.

In this example, the audio starts at the 1-second mark and plays for 5 seconds (from 1s to 6s), with the volume set to 80% of the original.

Merge Videos

To stitch multiple video clips together in CE.SDK, you first need to import and create separate video blocks:

// create the first video block
const video1 = engine.block.create('graphic');
engine.block.setShape(video1, engine.block.createShape('rect'));
const videoFill1 = engine.block.createFill('video');
engine.block.setString(
  videoFill1,
  'fill/video/fileURI',
  'https://cdn.img.ly/assets/demo/v2/ly.img.video/videos/pexels-drone-footage-of-a-surfer-barrelling-a-wave-12715991.mp4'
);
engine.block.setFill(video1, videoFill1);

// create the second video block
const video2 = engine.block.create('graphic');
engine.block.setShape(video2, engine.block.createShape('rect'));
const videoFill2 = engine.block.createFill('video');
engine.block.setString(
  videoFill2,
  'fill/video/fileURI',
  'https://cdn.img.ly/assets/demo/v2/ly.img.video/videos/pexels-kampus-production-8154913.mp4'
);
engine.block.setFill(video2, videoFill2);

At this point, you have two different video clips loaded from separate video files.

Now, assume you want to produce a 10-second video that includes both clips. Start by setting the duration of the page block to 10 seconds with the setDuration() method:

engine.block.setDuration(page, 10);

Then, suppose you want the first clip (video1) to play for 7 seconds, and the second clip (video2) to fade in after that and play for 3 seconds (from second 1 to second 4 of the original second clip). This is how you can set that up:

// set the duration of the first and second video clips
engine.block.setDuration(video1, 7); // video1 plays for 7 seconds
engine.block.setDuration(video2, 3); // video2 plays for 3 seconds

// wait for the second video to be fully loaded before applying the trim options
await engine.block.forceLoadAVResource(videoFill2);
// trim the second video to start at 1 second and play up to second 4
engine.block.setTrimOffset(videoFill2, 1);
engine.block.setTrimLength(videoFill2, 4);

// add a fade-in animation to the second video
const fadeInAnimation = engine.block.createAnimation("fade");
engine.block.setInAnimation(video2, fadeInAnimation);

These are the methods used above:

setDuration(): Defines the duration (in seconds) that a video block will be active during playback.
**forceLoadAVResource()**: Ensures that the audio or video resource (in this case, video2) is fully loaded before applying trim operations.
setTrimOffset(): Sets the start point (in seconds) within the video where playback begins.
setTrimLength(): Defines the length of the video to be played from the trim offset.
createAnimation(): Creates a fade-in animation for the second video (video2) to smoothly transition into the clip.

Finally, add both video blocks to the track block for playback:

const track = engine.block.create("track");
engine.block.appendChild(page, track);
engine.block.appendChild(track, video1); // add the first video to the track
engine.block.appendChild(track, video2); // add the second video to the track
engine.block.fillParent(track);

The result will be:

Note how the second clip fades in after 7 seconds of the first clip, lasting for 3 seconds as specified.

Incorporate Text Variables

Now, suppose you want to display cool words or messages in your videos. You can easily achieve this by adding a "text" block to your page, as shown below:

const text = engine.block.create("text");
// set the vluae of the text variable
engine.block.replaceText(text, "Surfing\nis\nCOOL!");
// set the text color to white and semi-transparent
engine.block.setTextColor(text, { r: 255.0, g: 255.0, b: 255.0, a: 0.8 });
// set the font size for the text
engine.block.setTextFontSize(text, 30);

// positioning the text on an absolute position on the video
engine.block.setWidthMode(text, "Auto");
engine.block.setHeightMode(text, "Auto");
engine.block.setPositionX(text, 130);
engine.block.setPositionY(text, 800);

// append the text block to the page
engine.block.appendChild(page, text);

The video canvas will now display a white “Surfing is COOL!” message as follows:

In this example, the content of the text block is static. However, it can easily be retrieved from a database or any other source for programmatic video generation with different messages, including personalized ones for tailored outreach (e.g., for custom greetings or offers).

Advanced Use Case: Prompt-Based Video Generation Using AI

As highlighted in our recent LinkedIn post, IMG.LY has just experimented with an MVP that turns keywords into fully edited short videos, complete with a script, voiceover, and visuals—all automated.

That is made possible by integrating CE.SDK with an LLM for coding and scriptwriting, ElevenLabs for voice synthesis, and Flux (via fal.ai) for visuals. Specifically, the LLM uses CE.SDK for video composition, animation, and rendering.

This is just the beginning, showing how AI can be combined with CE.SDK to programmatically generate video scripts—or even templates. A possible scenario would be to use the Creative Editor SDK plugin API to extend CE.SDK with custom plugins that adds AI directly to the design editor engine.

For example, you could develop a plugin with an LLM integration that allows users to input a prompt like: “Create a 15-second video with the text ‘Welcome to our App!’ and upbeat background music.” The LLM will process the request, using Creative Editor SDK to automate video creation by leveraging its powerful features.

Alternatively, the AI could produce a video that you can then automatically edit and enhance using CE.SDK capabilities. One thing is certain: the possibilities are endless with CE.SDK + AI!

Conclusion: The Future of Programmatic Video Generation

CE.SDK for JavaScript offers exceptional versatility in automating video workflows, allowing you to easily generate, edit, and render videos with just a few lines of code. You can stitch multiple video files, add background audio tracks, and incorporate dynamic text variables to finally produce videos in the browser—all thanks to the CE.SDK headless engine.

Looking ahead, AI integration with CE.SDK is unlocking even more powerful possibilities, such as prompt-based video content creation and automatic editing directly in the browser.

With the headless engine provided by CreativeEditor SDK, programmatic video generation in JavaScript can be implemented in just a few minutes. Explore the video capabilities of CE.SDK and dive into the docs.

Stay tuned for more updates, and please reach out if you have any questions. Thank you for reading.

Over 3,000 creative professionals gain early access to our new features, demos, and updates—don’t miss out, and subscribe to our newsletter.

Javascript Video Editing: Ultimate Guide for Developers and PMs

Jan — Fri, 20 Dec 2024 14:06:55 GMT

When developing or integrating a JavaScript-based video editor for the web, you must consider a number of factors to ensure the solution is both efficient and robust. This post is the definitive guide for anyone embarking on such a project. It explores the key technologies involved, their strengths and weaknesses, and how different use cases influence the choice of tech stack. We’ll examine diverse use cases, from lightweight, browser-based editors for quick edits to more advanced tools requiring complex processing and rendering. We’ll then discuss how these scenarios drive the selection of features and technology.

Additionally, we will show you the best open-source solutions that can accelerate development. This technical analysis will help you make an informed “build vs. buy” decision, ensuring you select the right approach for your project. Throughout this post, we will use the example of the IMG.LY’s JavaScript video editor (see here for a demo), showing you how these considerations shaped its architecture and feature set. You will learn practical insights into the decision-making process behind a successful web-based video editor.

Modern web technologies like WebGL, WebCodecs and WebAssembly have enabled browsers to efficiently perform resource-intensive tasks such as video editing that had hitherto been the sole domain of desktop applications. As a result Javascript based video editing tools such as veed.io have increased in popularity and users expects ever more sophisticated video editing capabilities inside modern video based web apps.

Video Editing Use Cases

Let’s first outline use cases and requirements of video editing applications on the web. These will inform our discussion of technical features of respective solutions below and grant a conceptual framework for evaluating potential benefits and drawbacks of each solution.

Simple Video Editing

Let’s start with the base case, simple video edits including trimming, cropping and resizing and simple effects such as adjusting brightness or saturation. This is usually sufficient feature set to support video editing for the following use cases:

Sales Outreach Videos: Sales teams often need to quickly edit customer-specific videos to personalize their outreach. This may involve trimming irrelevant portions, adding company logos, or adjusting brightness to ensure visual clarity.

Messaging Applications: These often require basic editing tools to allow users to crop, trim, or apply simple filters to shared videos, ensuring they’re concise and visually appealing.

Screencasting: Screencasting tools benefit from trimming and resizing capabilities to focus on key parts of recorded screens. Adding effects like brightness adjustment can make tutorials clearer and more professional.

CMS Systems: Content management systems may offer built-in video editing to help users optimize media assets for specific platform requirements, such as resizing for web embeds or adding subtle branding.

Screen Recording Applications: Screen recording applications often include simple editing options for cleaning up recorded content by trimming extraneous sections or cropping to highlight the most relevant parts.

Video Annotation

Next, users might want to add an additional layer of information to videos and overlay other assets, such as stickers, shapes, overlays, or text. Crucially these assets need to be time-based—that is, users need control over when the asset is shown so that particular video sequences can be referenced. This introduces the need for a timeline to arrange different video components relative to each other in time, as well as features like voice-over and audio track support.

E-commerce Reviews: Sellers and reviewers can annotate product demo videos with callouts, price tags, and feature highlights to make the content more engaging and informative.

Claims Management / Insurance: Insurance companies can use annotation to highlight key details in submitted video claims, such as timestamps of damage or explanatory text overlaying critical sections of footage.

Real Estate: Realtors might annotate property walkthroughs by adding labels, dimensions, or descriptive text overlays to highlight key features of the home or property.

Educational Applications: Instructors can use annotation tools to emphasize key moments in lectures or tutorials, such as overlaying text with formulas or concepts, or adding visual shapes to guide attention.

Productivity Tools: Users can annotate meeting recordings with timestamps, text notes, or overlay diagrams to summarize key decisions or action points.

Healthcare and Telemedicine: Medical professionals might annotate diagnostic videos or procedure recordings to explain findings or highlight areas of interest for training purposes.

Customer Support & Onboarding Tools: Companies can add annotations to video tutorials or troubleshooting guides to direct users through specific steps or highlight important information.

Video Composition

As we’re moving away from single media editing to creating video compositions of several types of media such as audio tracks, text, images, animations, and effects to create appealing visual designs, a well-designed timeline becomes even more important. Users need to manage many different video components in time, requiring user-friendly ways to browse and integrate external assets. This editor variant is most relevant for the following use cases:

Marketing Tech (Promotional Videos): Marketers can create visually stunning promotional videos by combining custom animations, background music, and text overlays for branding.

Social Media (Stories and Reels): Social media creators can quickly craft short, engaging videos that combine multiple assets like stickers, animations, and dynamic transitions tailored to platform-specific formats.

Event Highlight Reels: Event organizers can compile videos, photos, and music into cohesive highlight reels that encapsulate the essence of the occasion.

Template-based Video Creation

For many applications, users need starting points and examples for their designs. Starting from a blank canvas is rarely necessary for most use cases. Take a simple product video that includes promotional text, a brand logo, and animations. There is no need to reinvent the wheel for these types of videos. Instead, design applications should offer template libraries to accelerate user workflows.

Once we introduce workflows involving several stakeholders—such as designers setting up a certain brand framework and marketers working with and adapting these preconfigured designs—templates need to be able to constrain what edit operations adaptors can perform. These requirements are particularly important for these use cases:

Digital Asset Management: Organizations can manage and deploy branded video templates across teams, ensuring consistent design and messaging.

Social Media Publishing: Social media managers can utilize templates for quick turnaround on platform-specific video formats, such as Instagram Stories or LinkedIn posts.

Marketing Tech: Marketing teams can rely on pre-designed templates to churn out campaign videos at scale while maintaining brand consistency.

Training Videos: HR or L&D departments can adapt existing templates to quickly produce training materials customized for different teams or scenarios.

Creative Automation for Video

Finally, you can leverage data and automation to generate variations of your templates at scale, significantly boosting the productivity of use cases where users need to either test a large number of designs, publish to different channels with different requirements, or adapt designs to many slightly different instances, for example, menu designs for a franchise.

Marketing Tech: Marketers can generate hundreds of ad variations for A/B testing or localization by dynamically adjusting text, visuals, or calls to action.

Social Media Publishing: Social media teams can automate the creation of videos tailored to platform specifications, such as aspect ratios or resolution, while retaining core branding.

E-commerce: Retailers can produce personalized product showcase videos for different customer segments, featuring tailored offers, pricing, or recommendations.

Hospitality Industry: Hotels and restaurants can dynamically generate location-specific promotional videos showcasing seasonal offers, menus, or events.

Essential Video Editing Features

The above use cases are enabled by a set of features from simple transforms to complex compositions. Some video editors are focused more on manipulating individual video clips while others are more oriented towards video creation providing fully-fledged multi-media composition tool. This concise overview serves as a reference for evaluating video editing solutions:
Transforms

Cut, Trim, and Split: Basic editing tools for segmenting video clips.
Resize and Scale: Adjusts clip dimensions, ensuring proper fit or aspect ratio.
Crop and Rotate: Controls framing and orientation.
Zooming Capabilities: A UX feature enabling detailed editing and close-up views for precision.

Adjustments

Basic Adjustments: Controls brightness, contrast, and other visual enhancements.
Filters and Effects: Sets the visual appearance and atmosphere of videos.
Audio Tracks and Mixing: Supports multi-track audio adjustments and volume balancing.
Text and Overlay Options: Allows visual enhancements with text, captions, and emojis, providing opacity and layering options.

Composition

Multi-media Composition: Overlaying images, stickers, text as well as audio tracks are essential for creating videos through composition.
Multi-Track Editing: Elements and layers need to be position relative to each other in time allowing the creation of complex composition.
Canvas-Based Editing: Provides a flexible workspace for positioning and layering media elements.
Timeline Management: Split, join and arrange clips on a timeline. Manages the positioning of clips and timing for seamless transitions.
Animations: Make static elements dynamic and control their behavior in time. Can be use to create videos from scratch and enhance the storytelling workflow of existing ones.

The Technology Landscape

Some time in the early 2000s the web started to be viewed as a platform to run applications not just a distributed database to deliver static html documents. While Javascript development was not yet at a point to serve as foundation for the kind of application development we see today, Adobe Flash filled the role of development platform. Video editing in the early days of the web was mainly provided by Flash-based tools involving simple transform operations such as trimming and basic transitions. Java applets were another way to deliver processing intensive programs via the browser while executing them outside of the browser inside the JVM. These early attempts were either too clunky, since they could not run natively in the browser or lacked the performance to be useful for high quality editing.

When HTML5 came onto the scene in the early 2000s it became possible to handle multimedia content natively in the browser. Coupled with significant performance advances of Javascript and the introduction of WebRTC this opened the door for video editing on the web.

Some early cloud-based editors, such as WeVideo and Magisto, while more convenient and user-friendly than its predecessors still suffered from performance issues due to latency and lacked the depth of features found in desktop software.

In the past few years advances such as WebAssembly and Javascript libraries built upon it e.g. ffmpeg.js have revolutionised browser-based video editing. User can now perform complex editing tasks like multi-track timelines, video effects and real-time in-browser rendering. Modern web-based solution have even come to rival desktop apps, because they can take advantage of GPU acceleration, modern APIs like WebGL and UI frameworks such as React.

Let’s have a look at the state of the art web technologies that allow us to create performant video editing experiences inside the browsers. We are going to explore their relative strength, their complexities and conclude with when to best use each of these technologies:

Canvas API

The HTML5 Canvas API should be most familiar to most modern web developers, since it’s a familiar and popular way to render 2D graphics in the browser. It’s use for video editing is limited and restricted to simple overlays, trimming, composition and animation:

Complexity: Canvas is the least complex of the technologies introduced below and overlaps with the WebGL feature set to some extend, it’s API is well-known to most web developers.
Advantages: Canvas is lightweight, compatible across all browsers, and easy to implement, making it a great choice for simple use cases that require basic effects and no timeline.
Downsides: While well suited for simpler tasks Canvas lacks the GPU-acceleration of WebGL, hence it is not recommended for resource intensive operation or performance sensitive use cases.
When to Use: Choose Canvas for simple editing tasks, like trimming or basic overlays, or if you need a fallback technology that works well on legacy browsers and devices.

WebGL

WebGL opens up the power of GPU-accelerated graphics to the browser, making it the default choice for any use case requiring high performance rendering. As such modern Javascript video editing solution are well served by webGL when looking to render visual effects in real time, add transitions or facilitate multi-layer compositions by leveraging the GPU.

Complexity: WebGL comes with a steep learning curve especially if you are unfamiliar with 3D graphics processing, shaders and GPU programming.
Advantages: WebGL delivers high performance for processing complex video effects and can handle significant volume of data without latency. That makes it ideal for advanced, highly responsive video editing features.
Downsides: It can be challenging to engineer modular and maintainable WebGL code, debugging shaders and ensuring consistent performance across devices is an art in itself. WebGL might also be overpowered for simple editing use cases that do not justify the added complexity.
When to Use: Use WebGL if you’re creating a video editor with high-performance demands, particularly if real-time effects, transitions, or multi-layer compositing are needed.

WebCodecs API

Next up in our arsenal of formidable web technologies is a recent addition to modern browsers, WebCodecs, provide direct access to hardware accelerated video encoding and decoding.

The WebCodecs API is a recent addition to the browser, allowing direct access to hardware-accelerated video decoding and encoding. This API makes it easier to handle high-resolution video without delays, thanks to efficient decoding directly in the browser.

Complexity: WebCodecs is relatively easy to use for those familiar with media formats. However, you may need other technologies for rendering and complex editing.
Advantages: This API offers quick access to hardware acceleration, ideal for managing large video files. With it, you can efficiently handle high-definition video playback and streaming.
Downsides: Browser support for WebCodecs is still expanding, so compatibility might be a concern. Additionally, WebCodecs alone doesn’t provide a full editing suite; it needs to be combined with rendering and compositing technologies.
When to Use: WebCodecs is perfect if you need efficient decoding and encoding, especially for handling high-definition playback or live streaming.

WebAssembly (Wasm)

The next performance frontier browser pushed towards was enabling near-native performance by compiling languages like C++ or Rust into a format that efficiently runs in Javascript. This format is WebAssembly, which makes it possible to handle computationally expensive tasks like encoding, decoding or frame manipulation inside the browser.

Complexity: Working with WebAssembly is significantly more complex that the technologies discussed above. Developers need knoweldge of C++ or Rust as well as experience compiling to Wasm.
Advantages: WebAssembly enables browsers to handle complex video processing tasks, such as encoding and decoding, with near-native performance. Libraries such as FFmpeg build upon WebAssembly to make familiar video tools available to web developers.
Downsides: Development with Wasm is complex, and debugging can be a challenge. Additionally, Wasm modules may increase load times.
When to Use: Use WebAssembly when building native libraries for complex operation and expensive computations, such as encoding and decoding.

Our article on building a video editor with React and Wasm gives you a real-world starting point for your own apps.

MediaStream API

The MediaStream API is essential for gaining easy access to live video sources from within the browser, such as webcams. If your video editing app includes a feature for real-time recording or streaming the MediaStream API is an essential tool.

Complexity: MediaStream exposes a fairly simple API and is relatively easy to learn for experienced web developers.
Advantages: This API is well-suited for video feeds with low latency such as recording or streaming and does require addition encoding or decoding.
Downsides: Any use case requiring post-production video has to rely on additional technologies or frameworks, as the MediaStream API is limited to live feeds. The quality of the stream depends on the input source and can vary.
When to Use: Use MediaStream for applications with live recording or streaming functionality.

Web Audio API

The final web technology essential for video editing is the Web Audio API. Whether you want to add effects or mix multiple audio tracks, this API offers sophisticated tools for audio manipulation.

Complexity: For developers unfamiliar with audio processing the learning curve can be stepp, however the API is well designed and there is a host of educational resources.
Advantages: Web Audio API allows precise control over audio enabling your app to offer audio adjustments such as effects or reverb and multi-track editing.
Downsides: It can be challenging to accurately synchronize audio with video, especially if your app includes a real-time component.
When to Use: Opt for Web Audio API when audio editing is a core feature of your video editor.

Using These Technologies Together

Together, these technologies can be use to build a comprehensive browser based editor that is virtually indistinguishable from its desktop based counterparts. Different parts of the video editing workflow can be handled by each technology, while some choices have to made depending on the complexity and requirements of your projects, e.g. whether you’ll need resource intensive processing or an integration will real-time sources.

Video Decoding - WebCodecs API (for efficient video loading and playback).
Rendering - WebGL (for high-performance effects) and Canvas API (for simpler editing features).
Complex Processing - WebAssembly (for encoding/decoding and using native libraries like FFmpeg).
Audio Processing - Web Audio API (for mixing and adding effects to soundtracks).
Live Recording - MediaStream API (to capture live video from devices).

Open Source Javascript Video Editing Libraries

Fortunately in many cases someone has already done the heavy lifting and abstracted those browser APIs into simple to use libraries and frameworks. We’ll explore the most prominent ones distinguishing between projects that are for processing raw video and those that already provide an API for you to use. It is important to note that the libraries introduced below do not provide a video editing UI, unfortunately there are no open source video editors offering an end user interface that meet the standards for inclusion in a production grade application. None of the projects we evaluated met the demands we place on third-party libraries with respect to robustness, stability, maintenance and future development. However many can provide good starting points and references on how to build your own UI such as Reactive Video for React UIs.

FFmpeg.js

FFmpeg.js is the JavaScript port of the popular FFmpeg library compiled into WebAssembly. It enables developers to manipulate and process raw video files entirely in the browser. You can trim, convert formats, extract audio, and apply filters to videos without needing external software. However, FFmpeg just gives you the raw toolchain for building video editing features, you are still tasked with building the entire UX and editing workflows. We have an extensive guide on FFmpeg that you can consult to explore its capabilities and get started learning FFmpeg syntax.

Advantages:
- Most comprehensive library for web based video and audio processing.
- Fully client-side, no need for server-side processing.
- Highly flexible, allowing fine-grained control over video workflows.
Downsides:
- Fairly complex for beginners with a steep learning curve, requires familiarity with FFmpeg’s command-line syntax.
- FFmpeg is licensed under LGPL which makes it ill-suited for most commercial projects.
**When to Use:**FFmpeg.js is best suited for projects with custom requirements and UI making it necessary to exert fine-grained control over video processing tasks, especially for use cases like format conversion, advanced editing pipelines, or serverless video workflows.

The following two libraries allow you to programatically create and edit videos, we focus on the most stable, popular libraries here, while there might be promising up and comers we want to ensure a degree of dependability of the third-party code to be used in production systems.

Remotion

Remotion is a React-based framework that enables developers to edit or create videos programmatically in a declarative fashion using React components. React’s declarative syntax makes it comparatively simple to manage video content and Remotion provides a convenient UI for previewing and editing videos. While its studio UI cannot be embedded directly, Remotion can serve as the graphics processing engine for custom UIs. Although it is a commercial project, it offers a generous free tier for individuals and small businesses.

Advantages:
- Leverages React’s declarative model for seamless video creation.
- Provides robust video preview capabilities for iterative development.
- Flexible integration with custom-built UIs.
- Generous free tier for non-commercial and small-scale projects.
Downsides:
- The built-in studio UI cannot be embedded, requiring developers to create their own UI for embedding.
- React-specific, limiting use in non-React environments.
**When to Use:**Use Remotion when building React-based video applications that require programmatic video generation or dynamic video content. It’s ideal for teams already familiar with React and for applications where embedding is not a core requirement.

Etro.js

Similarly to Remotion, Etro.js is a JavaScript framework for programmatic video editing, it is framework agnostic and provides an API for simple composition of layers, effects and exporting videos.

Advantages:
- Includes support for advanced effects using GLSL shaders.
- Designed for in-browser video workflows.
Downsides:
- Minimal community support compared to larger projects.
- Requires additional effort to create a full UI.
When to Use: Suitable for applications where programmatic control and customization are key.

Extensibility

An important consideration when deciding to buy a prebuilt video editing solution is the degree of customization and extensibility required by your use case versus how critical speed to market is. While most SDKs provide enough flexibility by exposing cor API to build a solution that best suits your needs, this level of technical control comes at the expense of higher development and maintenance efforts. White-label solutions on the other end of the spectrum offer limited customization often relying on iFrames for embedding while providing maximum speed to market. If you are still in the phase where you need to make the business case for video editing features these might be key to allowing to quickly iterate. To learn more about weighing the pros and cons of SDK vs. White-label explore our extensive guide on the topic.

IMG.LY’s CE.SDK achieves extensibility by providing an interface for plugins that allow developers to add custom features and UI elements.

Cross Platform Video Editing

While this post is dedicated specifically to Javascript video editing an important consideration for choosing a video editing tech stack is whether there exists an intermediary output format that preserves editing operations so that videos can be edited on other devices and platforms. This is important for collaborative video editing use cases as well as use cases where videos are captured and basic edits performed on mobile and the finishing touches performed on the desktop.
These consideration are particular pertinent for the Real Estate, Telemedicine and Productivity use cases described above, since they involve either a multi-device or collaborative component.

At most common libraries or SDKs can achieve that by offering support for some form of serialization, that is exporting the edit operations along with the unedited video to then reapply edits upon import.

IMG.LY’s CE.SDK, on the other hand, is the only video editing SDK that is truly cross-platform. That means it is built atop a single creative engine that is portable to any platform. Whether iOS, Android, Desktop, or the Web, every platform uses the same underlying tech and uses the same custom format for scenes which contain all assets and edits making error-prone serializations obsolete.

Furthermore:

iOS and Android make use of the same underlying API.
Cross platform feature parity is baked into the cake. Since core functionality is implemented at the engine level, features are guaranteed to be available on both platform, although the timeline might differ somewhat.
Designs are 100% interoperable and consistent between platforms. Exporting and importing design files across platforms works seamlessly and final renderings are guaranteed to be consistent.

Javascript Video Editing & Generative AI

Generative AI is impacting video editing and generation across the board. Video generation obvious application of generative AI, but AI is also changing the way we we compose, create an edit videos in more subtle ways.

Features such as automated transitions, color grading, voice enhancement, voice over generation, background removal or inpainting are

Currently, these capabilities are offered by multiple specialized providers, such as ElevenLabs and OpenAI’s Sora. Firstly, that means modern Javascript video editors need to be open enough to serve as integrators for various, programmatically combining voice tracks, video clips, text, audio and other media before presenting users with an editing UI. Secondly, traditional video editing needs to enable the human in the loop. AI is still imperfect and manually refining unchangeable artifacts through prompts is inefficient and tedious. Instead the output by AI must remain editable and any editor solution smoothly interface with those artifacts to allow users to add the necessary polish, swap out media and regenernate individual parts of the scene.

To explore what this might look like in practice have a look at our CEO Eray Basar’s latest project building an AI video generator on top of CE.SDK.

Conclusion

When building or integrating a JavaScript video editor, developers and PMs have to navigate a complex technology landscape and make myriad decisions along the way. This guide has covered everything from essential use cases and feature sets to the cutting-edge technologies that underly browser-based video editing. Whether your project requires basic trimming tools or advanced multimedia composition, understanding the benefits and downsides of the various solutions like Canvas, WebGL, WebCodecs, WebAssembly, and more is crucial for choosing the right stack.

We’ve also explored how open-source tools like FFmpeg.js, Remotion, and Etro.js can accelerate development, while SDKs like IMG.LY’s CE.SDK offer unmatched extensibility and cross-platform compatibility. For those navigating the impact of generative AI on video editing, the need for seamless integration and user-centric design remains important.

If you’re looking for a robust, modern video editing solution for your web application, explore our showcase and start a free trial.

CE.SDK Video Editor Now Available on Android

Jan — Tue, 22 Oct 2024 11:44:02 GMT

We are excited to announce the release of CE.SDK’s video editing and video content creation tools for Android!

After the successful iOS launch, CE.SDK’s professional video editing capabilities are now available for Android. Whether your app focuses on social media, e-commerce, or marketing technology, CE.SDK enables users to create high-quality videos directly within your app.

Getting Started

Getting up and running with an instance of the CE.SDK video editor for Android is easy.

After downloading your license (get your trial license here) from your dashboard, simply invoke VideoEditor as composable function, Activity or Fragment.

Our Video Editing Backstory

Since we launched VideoEditor SDK almost ten years ago, we have seen an astounding range of creative and unique use cases of our SDK in customer projects. From Cameo connecting celebrities and their fans in new ways, over Multibrain supercharging their users marketing campaigns with editable video content, to Zigazoo powering video content within their social network for kids while putting safety and parental control first.

Since then, millions of users have experienced IMG.LY’s VideoEditor SDK through our customers’ products, shaping how they expect to interact with and modify videos. Continuous feedback from both our customers and their users has also influenced the way we approach video interface design, helping us refine and innovate along the way.

Our goal was to combine the reliability and robustness of the VideoEditor SDK with true cross-platform compatibility and the flexibility to meet diverse needs. By applying the lessons we’ve learned, we aimed to deliver an exceptional video editing user experience straight out of the box. We described how those learnings shaped the design of the mobile video editing interface.

Key Capabilities

With CE.SDK’s sleek timeline interface, your Android users can now take their video editing from simple transforms and adjustments to complex composition and multi-track editing:

Timeline-Based Editing
Split, trim, arrange video and audio tracks. Position elements such as images and stickers on the timeline and control their duration.

Multi-Track Editing
Work with multiple video and audio tracks, allowing for complex composition.

Transform & Enhance
Crop, flip, rotate, and use presets for common video formats. Add filters, effects, blur and essential adjustments.

Customizable Templates
Create and import templates using our web video editor, to jumpstart your users’ video composition and designs.

Asset Integration
Integrate your own asset libraries for audio tracks, sound effects, images, and more.

Cross-platform Interoperability
CE.SDK’s video capabilities extends across iOS, Android and the Web, enabling cross-platform applications to offer seamless video editing capabilities. Users can edit and save work in progress and pick up on any device.

Check out the showcase app for an interactive demo of these features.

Customization

The ability to tailor the video editor to your app’s particular use case has been a front and center concern in its development.

UI Customization: While theming is currently limited to dark and light model, future releases will introduce more flexible theming, allowing you to modify the look and feel of the editor to match your app’s branding. Furthermore, we expose callbacks that make it possible to hook into editor events and overlays for custom dialogues.
Configure Video Presets: You can configure the editor to create video presets based on your users’ needs. For instance, you may want to fix the video’s aspect ratio or optimize for certain resolutions, depending on the target platform.
Video Templates: With the CE.SDK video editor web UI, you can create an unlimited number of video templates, such as collages, text designs, and animations, providing your users with professional-quality starting points for their video designs.
Assets: Provide custom filters, fonts, and stickers to enhance the user experience. For niche apps, this can be an excellent way to engage users by offering content that feels unique to your community.

CE.SDK’s new Android video editing capabilities are perfect for a variety of use cases, including:

Social Media Applications: The CE.SDK video editor for Android excels at short-form video, ideal for social media applications, whether your users are publishing to other networks or you run a network yourself.
Ecommerce: Allow your users to create engaging product showcases, ads, and promotional videos, testimonials or reviews.
Marketing Tech Platforms: Users can leverage our versatile templating to easily create polished social media and marketing graphics from within your app.

What’s Next?

This Android release is just the beginning! We are about to launch a Camera UI for Android and are working on more advanced template creation features. So stay tuned.
To see some of these features in action, check out our iOS demo, where the next phase of innovation is unfolding.

Get Started

Explore our Video UI Showcase and Documentation for Android to discover how CE.SDK can streamline your content creation projects and make Android video editing easier than ever.

Over 3,000 creative professionals gain early access to new features and updates—don’t miss out, and subscribe to our newsletter.

A Modern Video Editor SDK for your Flutter App

Jan — Fri, 13 Sep 2024 11:26:15 GMT

Video has been the only type of content with steadily growing demand, and it will remain a staple and an expected medium for users to create and consume.

With platforms such as TikTok, Instagram Reels, and YouTube Shorts dominating and shaping user habits, the ability to create and edit videos directly within your app can significantly increase engagement.

In this post, we’ll show how to integrate a video editor into a Flutter app using CreativeEditor SDK and customize it for specific use cases, like creating TikTok-style short videos.

You can check out the code for the default and custom Flutter video editor on GitHub

Why add a Video Editor to Your Flutter App?

The last decade has made one thing clear: video content is king. Particularly, short-form dominates digital content. At the forefront of this trend are apps like TikTok, which made people more comfortable creating, editing, and sharing bite-sized clips.

These apps have capitalized on their ability to let users apply filters, and add audio overlays, text, and more, empowering users at all skill levels to make professional-looking videos with minimal effort.

A host of different use cases stand to profit from this trend by enhancing their user experience with video editing features, particularly marketing tech tooling and apps with a messaging or social media component.

In general, video creation lowers the threshold for user-generated content and can positively impact distribution and product engagement.

Flutter, Google’s UI toolkit for building apps across platforms from a single code base, is a perfect fit for IMG.LY’s Video Editor. What often motivates the choice to use Flutter is that It ensures performance, flexibility, and a seamless user experience on iOS and Android devices. Similarly, our SDK is designed to offer a native cross-platform experience, while relying on the same underlying graphics processing engine, ensuring consistency and interoperability.

Getting Started: Integrating the Video Editor in Flutter

Let’s dive into how to set up a video editor in your Flutter app using CreativeEditor SDK.

Requirements

Before we get into the details, here’s what you’ll need:

Flutter 3.16.0+
Dart 2.12.0+
iOS 16 or later (as the video editor is currently only available for iOS, Android coming soon)
Swift 5.10 and Xcode 15.4

First, ensure your pubspec.yml file includes the necessary dependencies:

dependencies:
  flutter:
    sdk: flutter
  imgly_editor: 1.34.0

Once your environment is set up, you’re ready to start coding.

Setting Up the Editor

After adding the imgly_editor dependency, import the package in your code:

import "package:imgly_editor/imgly_editor.dart";

The next step is to initialize the editor by providing an instance of EditorSettings. This setup requires a license key, which you can obtain from CreativeEditor SDK, and an optional userId that helps track monthly active users (MAUs) across different devices.

Here’s a basic example of how to open the video editor:

class VideoEditorSolution {
  /// Opens the editor.
  void openEditor() async {
    final settings = EditorSettings(
        license: "YOUR_LICENSE",
        userId: "YOUR_USER_ID"
    );

    const preset = EditorPreset.video;

		/// Open the editor and retrieve the editing result.
    final result = await IMGLYEditor.openEditor(
        preset: preset,
        settings: settings
    );
  }
}

This launches the editor with the video preset, enabling users to trim, cut, and add filters, text overlays, music, and more to their videos.

Use Cases: Building a TikTok-Like Experience

Now that we have integrated the basic editor configuration into your Flutter app, let’s revisit some prominent use cases and how to configure the editor to support them.

Short-Form Video Creation

For a short-form video, TikTok-like user experience, the most important factor is the ease of use of any video editor and content such as stickers or audio overlays.

The following SDK capabilities can help you achieve that goal:

Control Video Duration and Position in Time: Users should be able to trim video clips on a timeline and position them relative to other tracks, such as audio clips.
Filters & Effects: Filters are essential for setting the tone of the video. Whether users want a retro, high contrast, or soft pastel look, filters provide creative control over the visual style.
Text & Stickers: Allow users to add text captions—especially important for content often watched on mute. Stickers and emojis can add a playful element to videos.
Music & Audio: Like TikTok, you can allow users to add background music or sound effects. Providing a library of popular tracks or sound snippets can enhance the content creation experience.
Video Templates: Give your users a head start with customizable video templates for different themes, topics, and occasions. IMG.LY’s CE.SDK comes with a web video editor to help build and deliver these templates to your Flutter app. You can explore a range of different templates in our video editor demo.

By integrating these features, you’ll empower users to create engaging, shareable content, elevating your app’s video creation experience to the level of consumer-grade social media apps.

Influencers and Marketing

If your app is targeting influencers or businesses, the ability to quickly create branded video content is key. You can offer:

Branded Templates: Pre-made templates that align with a brand’s style, allowing users to easily drop in their video footage and text while maintaining brand consistency. Again, check out the web editor demo to see how to build video templates.
Watermarks: Adding a watermark or logo as a brand signifier to videos can help with brand visibility and make sure content always points back to the creator.

This setup allows influencers and brands to create professional-looking videos that they can share across social media platforms with ease.

eCommerce and User-Generated Content

Video is a powerful medium to showcase products, especially on marketplaces that can leverage user-generated content and enable vendors as well as customers to create product or review videos.

For example, users could create unboxing videos, product reviews, or tutorials using the same trimming, filter, and music tools offered in a TikTok-style editor. These videos can then be shared on social media or used within the app to enhance the shopping experience.

Likewise, showcasing products using authentic demo videos tap into familiar patterns of video consumption and can boost sales for vendors.

Customization Options with CreativeEditor SDK

We have designed the Flutter plugin to be adaptable to match your brand and use case-specific requirements. Soon the plugin will also support more advanced customization options such as the ability to activate or deactivate features by default.

UI Customization: Modify the look and feel of the editor to match your app’s branding. We provide options for theming, styling color palettes, and callbacks to hook into editor events.
Video Presets: You can create custom video presets based on your users’ needs. For instance, you may want to limit video length or optimize for certain resolutions, depending on the target platform.
Video Templates: You can use the CE.SDK web UI to generate any number of video templates including collages, text designs, and animations to give your users professional-looking starting points for their video designs.
Assets: Provide custom filters, fonts, and stickers to enhance the user experience. For niche apps, this can be an excellent way to engage users by offering content that feels unique to your community.

Conclusion

Integrating a video editor into your Flutter app will improve your UX, and help boost engagement, retention, and potential distribution of your product, whether you’re building a social media platform, an influencer tool, or an e-commerce app. With CreativeEditor SDK, you can create a TikTok-like video editing experience or offer specialized tools for businesses and creators.

By following the steps outlined in this post, you can bring professional-level video editing features to your users. Explore CE.SDK’s video capabilities and dive into the docs.

Stay tuned for more updates, and please reach out if you have any questions. Thank you for reading.

Over 3,000 creative professionals gain early access to our new features, demos, and updates—don’t miss out, and subscribe to our newsletter.

Designing A Timeline For Mobile Video Editing

Frank — Tue, 20 Aug 2024 13:08:58 GMT

This article documents the most surprising insights from our journey building timeline user interface. We’ll explore everything from overcoming usability challenges, visualizing data, prioritizing information, and refining touch interactions.

IMG.LY’s CE.SDK for iOS now comes with a native video editor, built from scratch in SwiftUI. I was responsible for coding and polishing the timeline user interface, working with both our design team and our cross-platform engine team. When I started implementing the timeline, we had the basic design concept sketched out in Figma. However, for an advanced UI like this, nothing beats rapid prototyping and iteration between design and development to make it feel right.

I. Visualizing Time

Timelines are a well-known convention, at least to modern humans. Timelines show durations in time as segments in space, distributed along the horizontal axis, with multiple tracks for things that happen in parallel. This is a simple yet effective way of representing moments in time visually. As obvious as it may seem today, this flavor of data visualization first appeared only 250 years ago. One of the earliest known examples is Joseph Priestley’s A Chart of Biography, published in 1765. Here is Priestley’s A New Chart of History from 1769, which is even more impressive:

Video editing timelines are built on this exact principle of mapping time to space: A digital video composition consists of tracks for simultaneous audio and video layers. These tracks contain individual clips that can be rearranged, trimmed, and transformed. Navigating the video’s time dimension is as easy as scrolling left and right. The tracks and clips serve as landmarks for orientation, forming the mental model of what the video composition is.

Timelines map time to space

https://www.youtube.com/embed/cKcksLW95-Y?feature=oembed

Horizontal Direction

Modern iOS user interfaces appear flipped in right-to-left languages: the buttons and controls are in different horizontal positions. The layout follows the reading direction of the language. Even the familiar drill-down navigation animates to the left when drilling deeper into the hierarchy, and the Back button sits in the upper right corner, its chevron pointing to the right. So, naturally, one of the first things we checked was whether timelines should run right-to-left in writing systems like Arabic or Hebrew. According to a trustworthy source, this is not the case: we found an old version of Apple’s developer guides stating that editor timelines in software are universally left-to-right, even if the surrounding interface layout is flipped by the operating system for right-to-left system languages.

Cognitive Load vs. Discoverability

When designing our video timeline, we experimented with both the size of the clips and with the dimensions of the timeline itself. How much can you fit on a screen before it looks cluttered? How large can the controls be without being clumsy and inefficient? The only way to find out is to try it out right on the device.

Hiding features in dropdown menus is easy. Banishing them to separate screens is even easier. But hiding functionality hurts discoverability. Out of sight, out of mind. We think user interfaces in productivity apps should be the opposite. They should prioritize high information density and avoid extra navigation steps as much as possible.

We increased the density of the toolbar that sits below the timeline: Our goal was to make all the important features easily accessible, despite the small screen. We achieved this by changing the single floating Add button from the initial designs to a more granular set of buttons to add specific asset types, like stickers and text. This not only increases efficiency for frequent users, it also makes the capabilities of the video editor more obvious and approachable.

Adaptive Height

As space is at a premium on a small screen, we wanted to make the video preview as large as possible. We decided to allow the user to collapse the timeline at any time to focus on the video canvas. And even without manually minimizing the timeline, it doesn’t waste space: the timeline adjusts its height to fit snugly around its contents. It only grows as you add more clips.

Units and Scale

The numeric scale is an essential part of any timeline. It helps you see the in and out time of a clip at a glance, and the permanently visible ruler makes changes in the zoom level easy to understand. We added dots as subdivisions between the ruler’s numeric labels, which gives it a nice tool-like character. At full zoom—when the ruler shows 5-second intervals—we decided to use four dots in between, so there is one dot for each second. We think this detail adds a welcome sense of precision.

The clips themselves have a duration label when they are selected. If the duration is below 10, you get a fractional digit (e.g., 4.2s). For the ruler, we went with the same short format (5s, 10s, 30s, etc.) for all values below one minute, and the technical format (1:00, 1:30, etc.) for the rest. Of course, we use iOS’s built-in number formatters to get properly formatted and localized strings for our labels.

An important detail for the label that shows the current playback timecode: The seconds must always be rounded down. If the playhead sits between two seconds, the lower number must be displayed until the playhead crosses the indicator for the higher second. Simple rounding to the nearest integer would be confusing, because it doesn’t match our expectations when dealing with timelines. Five seconds do not count as five seconds until they have elapsed in their entirety.

II. Manipulating Time

What looks good is not guaranteed to feel good when you interact with it. A video editor timeline is more than just a visualization; it’s also a user interface. As such, it should be self-explanatory and provide constant feedback as the user manipulates it.

Fingertip-sized

The clips must be large enough to be easy to touch and at the same time detailed enough to give a decent overview. The size that can be easily tapped by most people is around one square centimeter, which matches the 44 × 44 pt default size of iPhone system controls. Our testing showed that we could go a little smaller. Editing video is a focused task, so the tap targets don’t need to be as robust as for tasks that happen while you, say, run for the bus. And all user interfaces should be scalable for accessibility anyhow, so we didn’t hard-code any sizes and we also made the dimensions adjustable. And, speaking of adapting to individual user needs: of course, all the text labels in our timeline are automatically scaled depending on your device settings.

Handles

Selected clips in the timeline get a bold blue frame with large handles on either side. These handles are used to trim the clip’s length. For comfortable tapping and dragging, the handles must be larger than their visual size. In fact, we made the hit area more than twice as wide as the visual appearance suggests.

Early on in our design process, we realized that the trim handles had to sit on the outside of the clip, extending beyond the left and right edge. This made it hard to see the clip’s actual size, so we added little grooves: little triangular indents that point to the start and end of the selected clip.

Some clips in the timeline can be extended to any duration, like text or stickers. Other clips have a finite footage duration, like video and audio clips. To make this difference visible, clips with a finite footage duration reveal a ghost outline in the background while they are resized, featuring marching ants. The handle icon changes from an outward-pointing chevron to a vertical line when the clip can’t be made longer.

Scrolling and Zooming

A rather invisible detail was one of the trickiest things to get right: the coordination of all the different finger gestures on the timeline.

Horizontal Scrolling: Timeline scrolls horizontally to move in time.
Vertical Scrolling: Timeline scrolls vertically providing room for tracks.
Clip Movement: Dragging horizontally on a selected clip moves it.
Duration Adjustment: Dragging the trim handles adjusts the clip’s duration.
Pinch-to-Zoom: Pinching with two fingers changes the zoom level of the visualization.

All these gestures must behave as expected and not interfere with each other. We found it unexpectedly difficult to fine-tune and harmonize these interactions with pure SwiftUI, so we used some proven legacy iOS techniques to get it right. If you’re interested in the technical details of our solution with SwiftUIIntrospect and UIGestureRecognizers, check the source code.

Snapping

Video editing must feel precise, even on a small screen. When you trim or move clips, they should snap to other clips, to the playhead, and to the start and the end of the timeline. To show which position the selected clip snaps to, we designed animated dotted lines. We use the iPhone’s haptic engine to provide subtle feedback for snapping, which really helps to make the interaction feel natural—like a notch snapping into a detent. Something that we didn’t get right at first: Snapping should only respect those snapping points that are currently visible in the viewport. Of course! Otherwise, a partially visible clip may snap to something offscreen, which is very confusing.

Scrubbing Preview

For editing a video clip to feel precise, you need to be able to see exactly where you’re trimming. But the playback position in our mobile timeline can’t change while you’re trimming, because that would mean scrolling the timeline—which would move the very clip you’re trimming. The trim preview needs to be independent of the composition’s playback position. We solved this by adding a temporary overlay with its own playback time to the canvas. When you start dragging one of the handles on either side of the clip, the overlay shows the exact trim position of the video clip. When you release, the preview returns to the playback position.

Try it out!

“The only way to experience an experience is to experience it,” said interaction designer Bill Moggridge, and we couldn’t agree more! If you’re curious how our new video timeline feels, download our iOS app and try editing your own videos in our brand-new timeline experience.

If you’re a developer, check out the video editor documentation. With IMG.LY’s SDK, you can add all of these video editing capabilities to your own apps in a breeze.

CE.SDK v1.25 Release Notes

Neslihan — Tue, 07 May 2024 11:43:59 GMT

Welcome to the latest update from our team! Since our last release, we have been working to bring more creative capabilities to your app and improve user experience. With this release, you can:

Recolor Images like T-shirt Mockups

Replace colors in any image with our new recoloring feature. Select a color you want to change, choose its replacement, and apply the changes. Recoloring is ideal for tasks like altering the color of specific elements, such as t-shirts in product photos.

Simply choose your image, navigate to Effects, and select “Recolor”. Choose the color you want to change (Source Color), and the color you want to replace it with (Target Color). For a more refined result, adjust “Smoothness”, “Color Match”, and “Brightness Match”.

This feature is available on Web, Android, and iOS platforms.

Check our documentation on filters and effects to learn more.

Create Transparency with a Green Screen

Easily make green screen backgrounds transparent in your images and videos on the Web, and for images on Android and iOS. Our Green Screen Effect is tailored to identify and eliminate green backdrops—or any other color you specify. Perfect for content creators and videographers who need to isolate subjects without manually editing backgrounds.

Select an image or video clip with a color background you wish to remove. Then choose “Green Screen” in the Effects, and our effect handles the rest. Fine-tune the effect to your liking.

User Interface Improvements

In our latest update, we’ve focused on enhancing the usability and functionality of our interface. Here’s what’s new:

Audio Preview

Choosing the right audio clip is now easier with our new Audio Preview feature. Preview audio clips directly within the interface and make the perfect selection for your projects.

Selection Frame Improvements

We’ve upgraded our selection frames with new icons, hover effects, enhanced drag functionality, and shadow rendering. These improvements make manipulating elements within our platform more intuitive and visually appealing. Whether you’re rearranging components or resizing elements, our refined handling ensures a smoother interaction.

Please note that these release notes also include improvements from v1.24.

Thanks for reading. Join over 3000 specialists with powerful apps, and subscribe to our newsletter. We keep you in the loop with brand-new features, early access, and updates.

CE.SDK v1.23 Release Notes

Neslihan — Thu, 28 Mar 2024 09:47:34 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. In the past month, we have received valuable insight from you in our exclusive Beta testing. Now we are excited to announce CE.SDK v1.23!

Record and Edit Short-Form Video Content on iOS

Video Content Creation is here! Your users can record videos, blend them with audio, text, and graphics on a timeline. This includes features like countdown timer, and the popular split screen modes for reactions and duets. You may recognize these from Instagram Reels or TikTok. Making adjustments is straightforward using filters, effects, and adjustment settings.

Record with a New Camera

Record a video and utilize the Dual Camera feature. This allows you to record simultaneously with your front and back camera, producing a split-screen video. Determine when the recording begins using a countdown timer.

Edit on a Timeline

Users can reorder, arrange and split clips on a timeline, and use filters, visual effects, along with other adjustments to modify the appearance of clips and images. Add audio tracks, image or video overlays, format text, integrate stickers, and use tools for cropping, filtering, blurring, and adjusting images and videos. Try recording and editing Video Content with our Demo App for iOS.

Get started with our documentation.

Quickly Add Editors to Mobile Apps with UI Packages

In the past, integrating our editors into your app required developers to manually collect code from our showcases and piece together the UI. This not only demanded effort but also necessitated frequent updates to ensure compatibility with each of our new releases.

To free your time and resources, we are now introducing ready-to-use UI packages for iOS and Android, that bring advanced editors directly to your app. This eliminates the previous labor-intensive process. These packages include a wide range of options, from a fully equipped Video Editor (iOS) and Camera UI (iOS) to specialized examples like an Apparel Editor or Postcard Editor.

We maintain UI packages for you, ensuring compatibility across devices and development environments. Now, you can immediately start with fully featured editors, staying up-to-date with just a click, and without manual intervention. Get started with our documentation.

Integrate a User-Friendly Editor

We’ve made enhancements for a more user-friendly editor setup of CE.SDK.

To enhance legibility and perception of our UI elements, we have reworked our UI colors, making sure they meet the minimal contrast requirements for an improved experience. Additionally, the active state of settings is now more apparent than it was previously.

Next, libraries and all inspectors open on the same side of our default editor now, instead of on both sides. This reduces mouse travel, saves space, and streamlines workflows.

Additionally, you can now resize images more easily by dragging from any side, not just using the handles. This same feature is also applicable when cropping your image.

Lastly, elements stay selected after grouping or ungrouping. In the past, changing group settings would deselect your elements.

Integrate A Smooth Video Editor into your App

We recently launched our Video Editor for the Web. Since the release, we significantly improved the playback performance, leading to smoother editing and playback.

The timeline lets you arrange and edit videos, photos, text, audio, shapes, and stickers. Enhance projects with filters or adjustments, and add a personal touch with customizable audio. Precisely split, trim, and replace elements. Add audio files to make your project stand out. Try editing yourself with our Showcase for Sales Outreach Videos.

Please note that these release notes also include improvements from v1.22.

Thanks for reading. Join over 3000 specialists with powerful apps, and subscribe to our newsletter. We keep you in the loop with brand-new features, early access, and updates.

CE.SDK v1.21 Release Notes

Neslihan — Thu, 22 Feb 2024 16:22:28 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we are happy to introduce CE.SDK v1.21!

Create Videos With A New Timeline for Web

Create eye-catching videos within your web app for any platform, including Instagram, TikTok, or YouTube! Here’s what’s new: the integrated timeline supports the arrangement of elements like videos, photos, text, audio, shapes and stickers. Users can fine-tune their projects with adjustments or filters, and personalize their creations with audio from a customizable library.

To simplify your content creation, use our formats that automatically set your canvas size for social media and other popular video resolutions.

Effectively arrange and edit multiple elements within your videos, offering precise control over the editing process. Easily split, trim, and replace. Lastly, make it pop by adding audio files to the composition.

Creating templates with placeholders for your users is just as easy. Simply select an element, click ‘Placeholder’, and decide which elements can be edited.

To integrate, browse our Guide. Try the video editor live demo yourself. This showcase contains templates, audio and video examples for you to try.

Keep Designs in Focus on iOS

Our new Camera Zoom Limits make editing a design easier. Your page stays in focus to avoid accidental scrolling outside the design. The zoom level is set within a minimum and maximum range, ensuring a practical, usable view at all times. When you select an element in your design, it stays in focus when opening adjustments or other inspectors. It allows you to immediately see changes made to the selected element. This improvement is available for iOS and Android, and can be tested in our Design Editor Demo on the App Store and Google Play.

Ensure Color Purity in Print

Our newly introduced Underlayer API guarantees the preservation of color purity whenever you export designs using our CE.SDK for print. This solution specifically addresses the challenge of maintaining the richness and vibrancy of colors, irrespective of the color of the canvas, which could range from apparel to paper.

When you’re ready to export your design, this feature adds a special ‘underbase’ layer, underneath your design. This layer, typically a white or specific primer coat, makes sure the final printed colors match your original vision perfectly.

Very soon: Integrate Short-Form Video Creation

Video remains key for attracting, engaging and retaining audiences, especially on mobile. Creating a video editor can be tough, thus we’re thrilled to soon release our turnkey solution for mobile video content creation.

Combined with our new Camera SDK, users can seamlessly blend video, audio, text, and graphics on a sleek timeline. It includes features like voiceover, zoom, tap to record, and the popular split screen modes for reactions and duets. You may recognize these from Instagram or TikTok!

Last chance: Secure your waitlist spot to our Video Content Creation Release. You will also receive access to our TestFlight app.
Registration is only open until March 6, 2024.

? Gain Access to our Video Content Creation Release
? Download Feature Exposé instantly (PDF)

We can’t wait to hear your feedback!

Thanks for reading. Join over 3000 specialists with powerful apps, and subscribe to our newsletter. We keep you in the loop with brand-new features and updates.

CE.SDK v1.19 Release Notes

Neslihan — Fri, 22 Dec 2023 14:26:48 GMT

Since our last release, we’ve been crafting new features to empower your users’ creative journey. Today, we’re thrilled to introduce CE.SDK v1.19! With this release, you can:

Unify Design Power with Block Unification

Introducing Block Unification for CE.SDK on Mobile and Web—a feature that simplifies design versatility. Videos, images, shapes, and stickers now seamlessly come together as one Graphic Block, easily modified within a unified inspector. All supported settings for your selected block, including strokes, filters, and adjustments, are conveniently displayed in a single, user-friendly space.

Unified Graphic Block
Experience a singular Graphic Block supporting diverse fills (images, videos, colors, gradients) and shapes. Switch fill types (video, image, color) effortlessly, enabling dynamic designs in one place.

Easy Fine-Tuning
Block Unification integrates effects, filters, adjustments, and blur uniformly across all design elements. Elevate your designs with precision and consistency.

A Treat for Users and Developers
For developers, Block Unification means streamlined code maintenance. The consolidation of design blocks into one Graphic Block simplifies development, allowing more focus on crafting exceptional design experiences with IMG.LY.

Enhance Interaction with Mouse Wheel Support

In our latest update, our engine now directly registers mouse wheel events on the canvas, eliminating the need for manual camera manipulation. This seamless integration brings a host of new features, including smooth zooming, centered around the cursor, and support for pinch gestures on multitouch trackpads. Enjoy a more intuitive and unified interaction experience across various inputs, simplifying and enriching your design process.

Migrate to CE.SDK v1.19

The structural changes of CE.SDK v1.19 bring a more composable and powerful editing experience. To benefit from our new features, and to attain to licensing changes, head to our documentation on migrating to v.1.19 for an easy transition.

Wait. There’s more. As we explore the extensive capabilities of our SDK, we’re unveiling new showcases to demonstrate your possibilities. Plus, stick around until the end for an exciting announcement that will elevate your video creation experience!

New Showcase: Automated Resizing

In our new live showcase for web, effortlessly generate size variations of your designs and seamlessly scale your marketing materials across platforms. The automated resizing feature empowers you to easily leverage marketing materials without requiring the involvement of a design team, ensuring efficiency and adaptability.

New Showcase: Version History

Discover the convenience of Version History, enabling you to monitor changes made to each design and seamlessly restore past versions when needed. Explore our straightforward Version History showcase for a practical design experience.

Outlook: Offer TikTok-Inspired Video Editing

Video remains the reigning champion for attracting and retaining your audience. Yet, building a video editor from the ground up is a pain.

To simplify your journey and save valuable time and resources, we are excited to unveil a groundbreaking SDK for creating captivating short-form videos in January 2024!

Empower your users to arrange video, audio, text, and graphics seamlessly on a sleek video timeline. Our new camera introduces popular features like Voiceover, Zoom, Tap to Record, and more. Plus, bring the beloved Split Screen Modes for Reactions and Duets, akin to TikTok and Instagram!

Be the first to access our new IMG.LY SDK—join our waitlist now!

Thanks for reading! Subscribe to our newsletter to stay in the loop with the latest news and updates.

CE.SDK v1.14 Release Notes

Neslihan — Fri, 25 Aug 2023 15:37:00 GMT

Since our last release, we’ve been hard at work developing new features and improvements to enhance your creative editing experience. We are excited to announce that CE.SDK v1.14 is now available!

With this release, you can:

Generate Thumbnails Fast for Smooth Performance

Interfaces: Engine
Platforms: All

We have made significant enhancements to the way video thumbnails are generated, resulting in a smoother and more efficient workflow. Our intelligent engine now renders thumbnails seamlessly across multiple frames, ensuring there are no delays or interruptions while you work. Instead of slow JPEG compression, we have implemented a faster method using raw pixel buffers that maintain high-quality results.

If you’re using web applications, the thumbnail width adapts to fit your screen, providing a seamless and enjoyable user experience. Additionally, our dynamic thumbnail generation adjusts based on the size of your clips and zoom level, optimizing resources and improving overall performance.

Enhance Templating with Improved Placeholders

Interfaces: Engine, Editor
Platforms: All

Introducing an enhanced placeholder behavior for text and image content, allowing for seamless editing and replacement. With the new PlaceholderBehavior concept, you can define whether a block should initially look and behave as a placeholder or allow editing/replacement. This feature provides a smooth editing experience for users and offers flexibility for template designers to differentiate between placeholder and desired content.

Bring Creative Editing to Your macOS App

We’re excited to announce the addition of macOS platform support to our lineup. Extend your application’s reach to macOS users and provide a seamless editing experience across different platforms. Get started, and initialize our engine in your macOS app with our documentation.

Expand Editing Capabilities: macOS & Catalyst Binding

With this new feature, we provide a comprehensive solution for developers looking to enhance their iOS applications with our CE.SDK’s editing capabilities. Learn how to initialize our creative engine in your iOS app in our documentation.

Thank you for reading! Never miss out on updates and subscribe to our newsletter.

CE.SDK v1.13 Release Notes

Neslihan — Wed, 19 Jul 2023 13:08:50 GMT

Since our last release, we’ve been busy crafting new features and enhancements to empower your creative editing journey. Moreover, we have changed the configuration of CreativeEngine and CreativeEditor SDK to make it more versatile. We are excited to announce that CE.SDK v1.13 is now available!

Enhance Video Composition: Duration and Time Offset Control

Interfaces: API
Platforms: All

Whether you want to add a captivating video overlay or include background music, the possibilities are endless. What’s included:

Time Offset and Duration control: Give each block a specific time offset and duration, allowing them to appear and disappear at desired intervals. For example, you can overlay a sticker on your video for a brief moment, creating eye-catching effects.
API integration: Currently, this feature can be controlled via API, providing flexibility and control for advanced customization.

We are thrilled to announce that this engaging feature will soon be available across all our user interfaces. Learn how to configure video and audio in documentation.

Elevate iOS Video Experience with More Codecs

Interfaces: Touch UI
Platforms: iOS

With CE.SDK’s H.265 support for iOS, you can deliver high-quality video playback on iOS devices while enjoying the benefits of improved video compression and reduced file sizes. Enhance your users’ experience with faster video streaming, enhanced storage efficiency, and exceptional visual quality.

Easily Select Blocks in Complex Designs

Interfaces: Default UI, Studio UI, Touch UI, API
Platforms: All

Selecting blocks in complex designs can be tedious and time-consuming. But with our refined selection behavior, we’ve made it effortless. With this enhancement, you can now effortlessly select and interact with elements in your designs, even in the most intricate compositions. Our advanced algorithm considers the shape of the block and the transparency of the fill, ensuring precise and intuitive selections.

Simplify Group Interaction and Enhance Mobile Usability

Interfaces: Default UI, Studio UI, Touch UI
Platforms: All

Simplify the management and editing of groups in your designs with our latest enhancements. Experience a seamless workflow on both desktop and mobile devices, as you navigate and manipulate group elements. With clear hover-states and selection visualizations, group relationships become more apparent, allowing for precise editing and arrangement of elements. Your users can achieve the perfect composition for their designs with ease.

Configure via API

Interfaces: All
Platforms: Web

We’re thrilled to introduce a more versatile approach to configuring the engine with our latest update. Now, you have the power to fine-tune every aspect of the engine’s settings using comprehensive API calls. This enhanced configurability empowers you to customize your editor precisely according to your requirements. Embrace the flexibility and control to create an editing experience that aligns perfectly with your vision. Learn more about the changes in detail and how to migrate to CE.SDK v1.13 in our documentation.

Outlook: Easily Remove Backgrounds in Your Browser

@imgly/background-removal is a powerful open-source JavaScript library that allows seamlessly background removal of images directly in the browser. With its unique features and capabilities, this package offers enhanced productivity, cost-efficiency, and data privacy while achieving stunning results. Try it out in our background removal showcase. This highly requested feature will soon be available within CE.SDK

Thank you for reading! Never miss out on updates and subscribe to our newsletter.

IMG.LY Partners with Soundstripe to Infuse Video Editing with Epic Royalty-Free Music & SFX

Jan — Wed, 24 May 2023 07:31:24 GMT

We are excited to announce our exciting new partnership with Soundstripe, the leading provider of royalty-free music for video creators. Our VideoEditor SDK now features a seamless integration with Soundstripe, offering users access to a vast collection of over 9,000 hand-curated songs from 150+ musicians, along with an extensive library of 70,000 sound effects.

Starting from version 11, our VideoEditor SDK now comes pre-equipped with a seamless Soundstripe integration, as detailed in our documentation.

Streamlining Music Licensing for Digital Platforms

Finding the right music for digital platforms hasn’t been easy. Music licensing complexities and the challenge of finding high-quality tracks that align with your brand standards can be overwhelming.

But now, with our partnership with Soundstripe, you can effortlessly enhance your app with royalty-free music and video editing.

Our mission has always been to provide the ultimate toolkit for building captivating creative experiences, and video editing plays a central role in achieving this. We recognize the significant role that music plays in video storytelling, as it has the power to evoke emotions, set the mood, tone, and atmosphere, and greatly influence the pacing and energy of a video. With Soundstripe’s diverse range of over 50 genres, your users will have no trouble finding the perfect track to complement their visual narrative.

Effortless Music Selection

We have developed an intuitive search interface that allows users to explore music by title, genre, and description effortlessly. Once users have found their desired song, they can easily position any section of the track over their video. Furthermore, we are delighted to provide every one of our customers with seven complimentary sample songs, allowing them to get a taste of the fantastic library available. To unlock the full audio library, simply head over to Soundstripe and request an API key.

We are eagerly looking forward to working closely with Soundstripe to deliver the most exceptional video editing experience on the market. We invite you to try out VideoEditor SDK in your Android or iOS app and witness the transformative impact it will have on your users’ creativity. Together, we can take your video editing capabilities to new heights.

Thanks for reading! Stay ahead of the curve with our newsletter and sign up now!

How to Build a TikTok Clone for iOS with Swift & CreativeEditor SDK

Walter — Thu, 30 Mar 2023 11:20:26 GMT

Now that TikTok is facing a ban in the US any day now, we better wait in the wings with an alternative ready to go and scoop up those millions of hobby dancers, micro bloggers, and would-be influencers.

In this article, you’ll learn how to use Swift, SwiftUI, and the IMG.LY CreativeEditor SDK to build a simple iOS app for recording, editing, and viewing short videos. This app features views providing core functionalities similar to making a post in a video-sharing platform like TikTok.

The editing controller is the central hub. It allows users to capture video clips and enhance their videos by adding filters, stickers, music, and other effects. CreativeEditor SDK calls the overall project a “scene” and the parts (clips, stickers, text, etc.) “blocks”. Once the scene is ready, the user can compose it into a standard video file and export it. Finally, the playback controller showcases the finished projects using Apple’s standard VideoPlayer struct. For recording and editing, the CreativeEditor SDK’s Camera and Video Editor handle the heavy lifting, enabling swift development of a robust app. You can easily extend its features, filters, and creative options as your users demand.

Follow along to see how to capture video, enhance it, and export the finished product as a polished movie file.

Setting Up Your Project

You can try out the TikTok clone by cloning the GitHub repository supporting the article. Otherwise, follow this step-by-step tutorial and learn how to build the TikTok clone with CreativeEditor SDK by yourself.

To add the CreativeEditor SDK to an Xcode project, include it with your other package dependencies. Use the URL below to add the package, either to your Package.swift file or using File > Add Packages...

https://github.com/imgly/IMGLYUI-swift

The package includes a number of different frameworks:

IMGLYUI, an umbrella framework of all of the different editors and a camera.
IMGLYCamera, a standalone camera View. This is the same camera as the editors use.
IMGLYApparelEditor, IMGLYDesignEditor, IMGLYPostcardEditor, and IMGLYVideoEditor which are four different configurations to support different use cases.

Each of these frameworks uses the IMG.LY Engine for image processing. To include them in your project, navigate to the ‘Frameworks, Libraries, and Embedded Content’ section in your app’s General settings. Click the ’+’ button and select the desired frameworks. Initially, consider adding the IMGLYUI package, as it encompasses all other packages. Before releasing your app, review and include only the necessary components to minimize app size and avoid unused code.

Once Xcode downloads and resolves the packages, you just need to include IMGLYVideoEditor in any of the files that will reference the editor in your app. When you’re working with the underlying engine directly you’ll also want to include IMGLYEngine. You’ll only work with the engine as you start to customize the workflow.

Asking for Permissions

Before any app can access the phone’s microphone, camera, documents or photo library, the user must specifically give permission to record audio or video as well as access the user’s Photo library. If your app doesn’t secure these permissions properly before trying to use the camera the app will crash and also probably won’t get through Apple’s app review. In earlier versions of iOS you could develop an app on the Simulator without asking permissions, but always on the device the app will crash. In current versions of iOS the app crashes regardless of platform. The dialogue requesting permissions is a system dialog and looks like this:

You do not have the ability to change this dialog. You can supply the reason you are asking for the permission using a key in the info.plist file in the Xcode project. In the example above “Lets you record videos” is in the info.plist. For the video camera you will have to ask for video and audio permission. The first step is to add a short message to the user in the info.plist.

For video the key to add is NSCameraUsageDescription and for the microphone you need to add NSMicrophoneUsageDescription. Whenever your app attempts to use the camera, iOS will first check to see if the user has already granted access. If not, it will then display the dialogs using your entries in the info.plist or crash if you have not provided entries. The user might be surprised by these dialog boxes and may accidentally tap Don't Allow if they are trying to quickly launch the camera. It is better practice to set up some kind of onboarding view and secure the permissions before you need them. You might have a view that explains what you are going to ask for and then displays the permission dialog.

switch AVCaptureDevice.authorizationStatus(for: .video) {
  case .authorized:
    //the user has authroized permission!
    break
  case .denied:
    //the user has previously denied permission!
    //perhaps we should ask them again
    break
  case .restricted:
    //the user cannot grant permission!
    break
  case .notDetermined:
    //the user has never been asked for permission
    //so let's ask them now
    AVCaptureDevice.requestAccess(for: .video) { accessGranted in
      if accessGranted {
        //the user has authorized permission!
      } else {
        //the user has denied permission!
      }
    }
  @unknown default:
    break
}

The snippet above lets your app read the permission status for the video camera and ask for permission if it has not been granted. To request access to the microphone pass .audio to the AVCaptureDevice.authorizationStatus(for:) and AVCaptureDevice.requestAccess(for:) functions. The AVCaptureDevice.requestAccess(for:) command is what displays the system dialog actually requesting access. The accessGranted variable reports back to your app what the user chose. You can take care of getting the permissions any time but when the CreativeEditor SDK first launches it will attempt to access the camera, so the dialog will appear if the user has not already granted permission.

The .restricted case is fairly new. In this case, there are policies on the phone that prohibit the user from granting permission to your app. In addition to asking permissions during onboarding, it is good practice to check for permission every time before you attempt to run any code that uses the camera. The user can change permissions using the Settings app on their phone at any time. If the user has denied permissions and you present the video camera anyway, your app will record black video frames and silence on audio tracks.

In addition to asking for camera and microphone permissions, your app will probably want to access the photos on the user’s phone. You will need to add an entry to the info.plist for NSPhotoLibraryUsageDescription. As with the camera and the microphone, the dialog will appear when the Video Editor first attempts access, but you can give your user some ability to grant permission during onboarding. For the user’s photos, you can check the authorization status using

let status = PHPhotoLibrary.authroizationStatus(for: .readWrite)

As with the camera, the photo library has PHPhotoLibrary.requestAuthroization(for: .readWrite) but instead of just returning a “granted” or “denied” status, this command returns the actual new status. In addition to the status values for the camera, the photo library may return a .limited status meaning that the user has granted permission to only some of the photos. If the user has chosen to share only some of their photos, Apple provides some views you can present so that the user can manage the photos. Any videos that your app saves to the user’s photo library will always be available when the user has chosen .limited. You can read more about how to work with permissions and the user’s photo library by reading this Apple article Delivering an Enhanced Privacy Experience in Your Photos App.

Making Video Recordings

The start of a great TikTok is the camera. When our user wants to make a new video clip with our app, they tap on the button at the bottom of the initial screen to start the creation workflow.

Here there is a decision to make. TikTok and the CreativeEditor SDK can each start with the creation controls and a video preview. When using the CreativeEditor SDK though, you can also start with some video that came from somewhere else. So if you already have some camera code you like, or if you want to start with some video clip from somewhere else, you can insert that into the editor when it launches. You can do that by leveraging the onCreate modifier of the editor.

let engine = EngineSettings(license: "<your license code>",
                              userID: "<your unique user id>")

VideoEditor(engine)
  .imgly.onCreate({ engine in
    try await engine.scene.create(fromVideo: Bundle.main.url(forResource: "dog_water", withExtension: "mov")!)

    //set other defaults
  }

In the code above, the app initializes an instance of the Video Editor view and then reads the dog_water.mov file from the app bundle. launches the editor with that.

Whether starting with a video or starting with a blank canvas a user can add video clips to their creation using the embedded camera.

The image above compares two different camera controllers: the left one from the TikTok app and the right showing the default settings of the MobileCamera, either standalone or within a Video Editor scene. Key controls in each are marked with blue numbers:

1 - Flip camera button
2 - Flash button
3 - Recording time indicator
4 - Add video and finish buttons
5 - Cancel button

While the TikTok camera integrates several editing functions during video capture, the Video Editor separates these tasks, focusing the camera purely on recording. Both systems allow users to start and stop video recording to compile a series of clips before moving to the editing phase.

Configuring the Camera

The camera used inside of the Video Editor UI isn’t customizable. You have limited options to customize when you use the standalone camera though. You can set the colors of the buttons, helpful if you have a preferred color scheme, and you can set a maximum duration for the video recording allowed.

let config = CameraConfiguration(
  recordingColor: .orange,
  highlightColor: .blue,
  maxTotalDuration: 30,
  allowExceedingMaxDuration: false)

Camera(engine, config: config) { result in
 //handle the recorded video collection
}

The code above sets the record button to orange while recording and the clip view to orange. There is a maximum duration of 30 seconds for all videos in the collection and the button to save the clips highlighted in blue.

If your app requires further customization, you’ll probably want to make your own camera view and work with the IMG.LY Creative Engine directly.

The documentation website is a great resource for understanding how much you can add to and customize the IMGLYUI resources and when you’ll need to drop down to the level of the engine.

Presenting the Controller

As demonstrated above, both the Video Editor and the Camera are View structures that can be presented directly or through overlays such as overlay or fullScreenCover. However, they are typically enclosed in a NavigationView to utilize the navigation bar for displaying export and other buttons. Therefore, when triggering the Video Editor from a button, it’s crucial to embed it within a NavigationView.

Button("Edit the Video") {
  isPresented = true
}
 .fullScreenCover(isPresented: $isPresented) {
   NavigationView {
     VideoEditor(engine)
      //any setup modifiers
   }

Building Your Creation

After capturing a video, the app displays an editor to complete the creation. The editing tools are in the red circles. TikTok provides about ten different tools on the editing screen. Additionally, TikTok provided some editing tools on the capture screen. The CreativeEditor SDK apps provide a scrolling set of asset libraries along the bottom for items to add to the scene. But when any of the elements of the scene, like your video or an audio clip, are highlighted, the bottom row becomes tools to manipulate that element. The basic Asset Library types are uploads, videos, audio, images, text, shapes, and stickers.

How to configure and customize each of these tools is beyond the scope of this tutorial. But, a lot of the customization of the assets, fonts, blurs and even filters can be done without code. By design, the CreativeEditor SDK downloads assets from a central server. This is a great way for you to be able to push updated filters, stickers, and other assets to your users without going through the app update process. In the next section, we’ll walk through that and provide some links for further research.

Configuring the Editors Assets

The IMG.LY Engine looks for assets for stickers, vector shapes, LUT filters, color filters, color palettes, blurs, and fonts using a URL by default. The design of the system is that you would have your own server supporting your app. However, assets in the app bundle are also accessible by URL, so as long as the assets are in the folder format that the engine expects, it doesn’t matter whether they are local or remote.

In the documentation for this process you can find a URL to the default asset bundle. Once you download that bundle you can add it to your app. The bundle is structured much like a regular iOS Assets.catalog with some JSON containing metadata about the asset and a file that provides the actual asset. For some of the assets like filters and vectors, no file is needed as everything can be defined in the JSON.

In the image above, you can see the JSON to describe the ape sticker and the emoji_ape image file after the IMGLYAssets bundle has been added to our app. You can edit these assets and add your own.

Once you’re done editing the asset bundle, you can use it instead of the default bundle during the onCreate modifier like this:

VideoEditor(engine)
  .imgly.onCreate({ engine in
     try await engine.scene.load(from: VideoEditor.defaultScene)
              // Add asset sources
     let bundleURL = Bundle.main.url(forResource: "IMGLYAssets", withExtension: "bundle")!
     try await engine.addDefaultAssetSources(baseURL: bundleURL)
     try await engine.addDemoAssetSources(sceneMode: engine.scene.getMode(),
       withUploadAssetSources: true)
     try await engine.asset.addSource(TextAssetSource(engine: engine))
  })

In the code above, we create a blank scene using VideoEditor.defaultScene static method. Then set a URL to point to the bundle in the app instead of a remote bundle and make that the source of addDefaultAssetSources. Now our app will use the assets from the bundle.

Exporting the Finished Video

Once the user has finished with their creation, they tap the share button and the Video Editor composes all of the video, audio, filters, and static items into a single .mp4 file. Then it displays a share sheet. If you’d rather do something different, like save the creation locally so you can play it back, you can override this default behavior.

In your app, after the .onCreate modifier, you can define a .onExport modifier. When the user taps on the share button you can export the video and then save it to the documents directory to playback later.

In the documentation for modifiers you can find the source code for the Video Editor’s default behavior. Instead of duplicating it in detail here, we can just summarize and provide a little snippet for our change.

First, the .onCreate handler takes an engine and a eventHandler parameter. The engine is the underlying IMG.LY Engine object we’ve been using. The eventHandler lets us send information back to the Video Editor UI about the progress or any errors.

In the documentation, there is a helper function to export the video. After it runs it returns the data of the exported video and a mimeType, which will be .mp4.

let data: Date, mimeType: MIMEType
let mode = try mainEngine.scene.getMode()
guard mode == .video else { throw CallbackError.unknownSceneMode; return }
(data, mimeType) = try await exportVideo()

The code in the documentation handles static files from the DesignEditor as well as videos from the VideoEditor. The above code makes it so that only video can be exported, since our app only works with video.

In the exportVideo function, the code first checks to make sure it has a properly formed scene object to work with. Then it starts the export using mainEngine.block.exportVideo(_, mimeType:). This begins an async throwing stream during the export. The VideoExport objects in the stream are either .progress or .finished. When a .progress object arrives, the exportVideo function sends an update to the eventHandler to display in the UI. When the .finished object arrives, it has an associated object that is the data for the video.

Now instead of opening the share sheet, we can write the video to the documents directory for the app.

do {
      guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
        else {
          logger.error("Documents directory not found"); return
        }

  let filePath = documentDirectory.appending(component: UUID().uuidString)
  try data.write(to: filePath)
} catch (let error as NSError) {
  logger.error("could not write finished file: \(error.localizedDescription)")
}

eventHandler.send(.exportCompleted {})

The code above takes the data from the exportVideo function and writes it to the app’s documents directory, giving it a UUID string value as a filename. Then it updates the UI to let it know the export is done but sends an empty action block.

Setting Up a Playback Controller

For this example app, the playback controller will play any clips in the app’s Documents directory. Each clip plays on a loop. The user can swipe up to get the next clip and tap to pause or restart the clip. If there are no clips, the user will be encouraged to make a new one.

When the video player view appears, the first task is to see if there are any videos to load. Any video files in the Documents directory are assumed to be ready to play.

func loadVideos() {
  //get a handle to the documents directory for this app
  guard let documentDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true) else { logger.error("Documents directory not found"); return}

  if let files = try? FileManager.default.contentsOfDirectory(at: documentDirectory, includingPropertiesForKeys: nil) {
    videos = files
  }
}

The code above reads all of the filenames from the documents directory into a files array and then assigns that array to a videos variable.

Each time the app is launched or the VideoEditor view is dismissed, the video player view will get the onAppear message. This is a good place to check for videos using the loadVideos function. After the videos are loaded, the app can start to play the first one.

func setupVideoPlayer() {
  guard let currentVideo = videos.first else {logger.error("No videos to play"); return}

  self.shouldHideEmptyDirectoryText = true

       let endMonitor = NotificationCenter.default.publisher(for: NSNotification.Name.AVPlayerItemDidPlayToEndTime)

  self.player = AVPlayerItem(url: currentVideo)
  self.currentVideo = currentVideo

}

In the code above, we first check to see if there are any video files. If there are, then hide the message about videos and create a new AVPlayerItem using the URL of the first video. The endMonitor, shouldHideEmptyDirectoryText, and player are both being watched by the View. The AVPlayer gets rendered in a regular SwiftUI View.

VideoPlayer(player: player)
  .frame(width: 640, height: 360)
  .onAppear {
    player.play()
  }
  .onReceive(endMonitor) { _ in
    player.seek(to: .zero)
    player.play()
}

When the video reaches the end, it will seek(to: .zero) which rewinds the video and loops it.

In the TikTok app, you can advance to the next video by swiping up. Additionally, you can start and stop any video by tapping on the screen. We can add both of those features using gesture recognizers.

The code for play and pause is attached to a Tap Gesture modifier we can append to the player.

.onTapGesture {
  if player.rate > 0.0 {
    player.rate = 0.0
  } else {
    player.rate = 1.0
  }
}

We can just adjust the playback rate. A value of 1.0 is a normal speed and 0.0 is paused. The player also has a .pause and .play methods, but sometimes these seem buggy.

Swipe is a little more difficult since SwiftUI only has swiping on List items. So we can use a drag gesture and then evaluate the translations when it’s complete.

.gesture(
    DragGesture(minimumDistance: 3.0, coordinateSpace: .local)
        .onEnded { value in
            // Check for a vertical, upward swipe with minimal horizontal deviation
          if abs(value.translation.width) < 100 &&
             value.translation.height < 0 {
                // It was an up swipe!
		viewModel.advanceVideo()
            }
        }
)

Then advancing the video is a simple matter of refreshing the player

func advanceVideo() {
  guard let currentVideo = self.currentVideo else { logger.error("No current video."); return}

  let currentIndex = videos.firstIndex(of: currentVideo)
  var nextVideo = videos.index(after: currentIndex!)
  if nextVideo == videos.count {
    nextVideo = 0
  }

  self.currentvideo = videos[nextVideo]
  self.player = AVPlayer(url: self.currentVideo!)

}

This code will get the next file URL from the array of videos. When it reaches the end, it will loop back and get the file at index 0. Then it creates a new player. Because the view is observing that player variable, it will update with the new video.

With this view, the user can create new videos by tapping the creation button and swipe to view all of their created videos.

Where to Go From Here

This tutorial has focused on how the CreativeEditor SDK can help you quickly make a video creation app like TikTok. Good next steps would be to further customize the editing tools and build out the network for sharing and tagging videos. Something that is important to consider is the data structure for each video. The iOS system is optimized to read only as much of a video file off of disk as is needed at any moment. Your app should use those optimizations to run faster. So, don’t load the whole video into memory as a Data object. Your data structures should keep the large video files somehow separate from the much smaller metadata (likes, comments, etc.). Consider storing the URL or filename of the video in the same object as the likes and comments. This will also allow you to cache video files after they have been downloaded so that you don’t need to redownload them when other data such as comments or number of likes changes.

Thanks for reading! We hope that you’ve gotten a better understanding for how a tool like the CreativeEditor SDK can bring your ideas to market faster. Feel free to reach out to us with any questions, comments, or suggestions.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

To stay in the loop, subscribe to our Newsletter or follow us on LinkedIn and Twitter.

How to Add Stickers and Overlays to a Video in Flutter

Natalia — Fri, 24 Mar 2023 09:32:33 GMT

When working with interactive content, it may happen that you will need to add remote or local assets on top of your video file. The assets varying from images and stickers to videos and fonts are usually grouped and deployed within your application and are available at runtime. This tutorial will teach you how to apply stickers and overlays to a video in Flutter and make your video content more personalizable and user-oriented. We start by considering the prerequisites for successful video integration, then talk about how Flutter manages overlays in order to personalize our video file.

How does Flutter handle video assets?

Unlike images, Flutter displays a video file by employing a special video_player plugin that provides playback, stores a file, and manages speed and sound control. For iOS-based applications, the video is integrated via AVPlayer, whereas the Android system employs ExoPlayer. Crucially, to create a simple video player, one should follow the steps below:

First, to ensure that pubspec.yaml file contains the video_player dependency;
Then permit access to videos for the application;
Make a VideoPlayerController and set it up;
Finally, don’t forget to display and play the video player;

So, let’s closely look at the setup process.

Prerequisites:

We start by adding the following dependencies of video_player plugin to pubspec.yaml file.

dependencies:
  flutter:
    sdk: flutter
  video_player: ^2.4.7

Afterwards, change configurations of both android and ios systems to guarantee that an application has the proper authorization for video streaming: e.g. for Android-oriented programs, proceed to the <project root>/android/app/src/main/AndroidManifest.xml directory – where it’s necessary to add the next line to the AndroidManifest.xml.

<manifest xmlns:android="<http://schemas.android.com/apk/res/android>">
    <application ...>

    </application>

    <uses-permission android:name="android.permission.INTERNET"/>
</manifest>

For iOS-based apps the Info.plist file is stored at <project root>/ios/Runner/Info.plist. Once again, don’t forget to add the following specification to the original code:

<key>NSAppTransportSecurity</key>
<dict>
  <key>NSAllowsArbitraryLoads</key>
  <true/>
</dict>

Successful video integration in Flutter is managed through the VideoPlayer widget and a VideoPlayerController, which sets the connection to the asset and initialize the controller for playback.

/// Stateful widget to fetch and then display video content.
class VideoApp extends StatefulWidget {
  const VideoApp({Key? key}) : super(key: key);

  @override
  _VideoAppState createState() => _VideoAppState();
}

class _VideoAppState extends State<VideoApp> {
  late VideoPlayerController _controller;

  @override
  void initState() {
    super.initState();
    _controller = VideoPlayerController.asset('assets/bee.mp4')

      /// Specify the path to your video asset here
      ..initialize().then((_) {
        /// Ensure the first frame is shown after the video is initialized,
        /// even before the play button has been pressed.
        setState(() {});
      });
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }
}

Note that video tends to take up as much space on the screen as possible by default, which can significantly deteriorate the video’s quality. Therefore, the Flutter team suggests employing the AspectRatiowidget to adjust the video proportions.

? AspectRatio(
    aspectRatio: _controller.value.aspectRatio)

Adding Overlay Stickers and Text to a Video in Flutter

To see the complete code of our demo application, you can clone the GitHub repository or put the following command to your Terminal:

git clone git@github.com:nataliakzm/Adding_Stickers_and_Overlays_to_video_in_Flutter.git

Similarly to how we handled resizing images in Flutter, the same method is used when overlaying an object on a video. The above example consists basically of three containers, one with a sticker, another with the textual watermark and the last with a local video asset. However, the critical difference here is that a Container filled with a sticker and a Container filled with a textual watermark cannot be simply listed one by one in your code but rather must be both placed within a Stack widget.

The Stack container is designed as a “mother”-widget which retains multiple layers of widgets on the screen. Stack structures these branches of children into a hierarchical system from bottom to top. Thus, the uppermost widget goes to the background, and the bottommost item is displayed in the foreground.

Stack(
  children: <Widget>[
    BottomWidget(),
    MiddleWidget(),
    TopWidget(),
  ]),

Since the size of the Stack widget is aimed to be the largest size among layers, we will place our video asset on this layer. Then, to implement the overlays, it’s essential to position and/or align each child’s container.


child: Stack(children: [
  VideoPlayer(_controller),

  /// Don't forget to align the position of the Сontainer
  Positioned( bottom: 5, left: 10,
      child: Container( width: 80, height: 40,

        /// In case you want to check the position/size of the Container uncomment the next line
        //color: Color(0xff0360da),
        child: Align(
            alignment: Alignment.center,

            /// Integrate Sticker overlay into a video
            child: Image.asset('assets/sticker.png')))),

  Positioned( top: 5, right: 10,
      child: Container( width: 80, height: 40,
        child: Align(
            alignment: Alignment.center,

            /// Integrate Text overlay into a video
            child: Text("IMG.LY",
                style: TextStyle(
                    color: Colors.white,
                    fontSize: 20,
                    fontWeight: FontWeight.bold))))),
                        ]))

  /// If there is no video, a blank page will be return   : Container()),

Summing up

Thus, you can see from the code below that integrating multiple layers over interactive assets can be a tricky task which requires bearing in mind the position and order of each layer, as well as managing how the children are aligned to a video file and the screen size of the user. Conversely, the discussed method allows personalizing an application without significant modifications to the original file.

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';

/// Import video_player package to integrate a video asset
import 'package:video_player/video_player.dart';

void main() => runApp(const VideoApp());

/// Stateful widget to fetch and then display video content.
class VideoApp extends StatefulWidget {
  const VideoApp({Key? key}) : super(key: key);

  @override
  _VideoAppState createState() => _VideoAppState();
}

class _VideoAppState extends State<VideoApp> {
  late VideoPlayerController _controller;

  @override
  void initState() {
    super.initState();
    _controller = VideoPlayerController.asset('assets/bee.mp4')

      /// Specify the path to your video asset here
      ..initialize().then((_) {
        /// Ensure the first frame is shown after the video is initialized,
        /// even before the play button has been pressed.
        setState(() {});
      });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
        debugShowCheckedModeBanner: false,
        home: Scaffold(
            appBar: AppBar(
                title: Text('How to add Stickers and Overlays'),
                backgroundColor: Color(0xff0360da)),
            body: Center(
                child: _controller.value.isInitialized

                    /// First, we specify the AspectRatio of a video frame
                    ? AspectRatio(
                        aspectRatio: _controller.value.aspectRatio,

                        /// Then, we initialize a Stack widget
                        child: Stack(children: [
                          VideoPlayer(_controller),

                          /// Don't forget to align the position of the Сontainer
                          Positioned( bottom: 5, left: 10,
                              child: Container( width: 80, height: 40,

                                  /// Uncomment the following line to check the position/size of the Container
                                  //color: Color(0xff0360da),
                                  child: Align(
                                      alignment: Alignment.center,

                                      /// Integrate Image overlay into a video
                                      child: Image.asset('assets/sticker.png')))),

                          Positioned( top: 5, right: 10,
                              child: Container( width: 80, height: 40,
                                  child: Align(
                                      alignment: Alignment.center,

																			/// Integrate some Text overlay into a video
                                      child: Text("IMG.LY",
                                          style: TextStyle(
                                              color: Colors.white,
                                              fontSize: 20,
                                              fontWeight: FontWeight.bold))))),
                        ]))

                    /// If there is no video, a blank page will be return
                    : Container()),
            floatingActionButton: FloatingActionButton.extended(
                backgroundColor: const Color(0xff0360da),
                foregroundColor: Colors.white,
                onPressed: () {
                  setState(() {
                    _controller.value.isPlaying
                        ? _controller.pause()
                        : _controller.play();
                  });
                },
                icon: Icon(_controller.value.isPlaying
                    ? Icons.pause
                    : Icons.play_arrow),
                label: Text('Play for IMG.LY'))));
  }

  @override
  void dispose() {
    super.dispose();
    _controller.dispose();
  }
}

Adding Overlays to a Video with Flutter package for VideoEditor SDK

However, if you would like to offer more advanced sticker and overlay functionality to your user and you might need a more complex UI structure. Since this can be an extremely time consuming undertaking you might want to opt for a ready-to-use solution such as VideoEditor SDK which among a host of other essential video editing features allows users to easily add stickers and different overlays to their videos.

To replicate the example above, go to the official documentation to discover how to get started with VideoEditor SDK or follow our guide on how to integrate video editor for Flutter into your app. Try to upload your video file and modify it with various stickers, text and other overlays. Alternatively, you can download our free mobile demo app from the App Store or Google Play, and test a comprehensive set video editing tools.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

How To Build a Video Editor With Wasm in React

Antonello — Mon, 16 Jan 2023 16:31:57 GMT

You may think that video processing can only be performed on a server, but this is not true. Thanks to WebAssembly (Wasm), you can run high-performance code written in C or C++ in your browser. This opens up a lot of possibilities and gives you the ability to build a client-side video editor. Let’s now learn how to create a Wasm-based video editor in React.

Follow this tutorial and learn how to build the application below:

This React video editor allows users to upload a video, select a portion of it, convert it to GIF, and download the resulting image file – all of this in your browser.

What is WebAssembly?

WebAssembly (also called Wasm) is a new type of code that modern web browsers can run and understand. Specifically, Wasm provides new functionality to web development and brings significant performance benefits. This applies to both frontend and backend, since Wasm can be used by both clients and servers.

Wasm code should not be written by hand. This is because Wasm is a binary instruction format. So, Wasm is designed to be a compilation target for source languages such as C, C++, Rust, and others. Learn more about Wasm on the WebAssembly MDN Web Docs page.

Wasm provides a way to run code written in several languages on the Web at near-native speed. This means that WebAssembly forms can be easily imported and executed by a Web client or server application. In other words, you can use WebAssembly functions via JavaScript and achieve results that were not previously possible on a browser, especially in terms of performance.

What is `ffmpeg.wasm`?

As stated on the GitHub page of the project, ffmpeg.wasm is a WebAssembly and JavaScript port of FFmpeg. If you are not familiar with it, FFmpeg is an open-source software suite that includes several libraries and programs for managing audio, video, streams, and other media files. All of these tools can be invoked and executed via the ffmpeg command-line instruction.

Therefore, ffmpeg.wasm enables video and audio processing in JavaScript-based web applications. In detail, it enables video and audio recording, format transcoding, video and audio editing, and video scaling. Even though it is a WebAssembly-based library, you can use it in your JavaScript code just like any other npm library. This is the power of Wasm.

Since it transpiles to Wasm, you can take advantage of ffmpeg.wasm directly in your browser without performance concerns. This means that ffmpeg.wasm opens the door to client-side audio and video processing. Let’s now learn how to use ffmpeg.wasm to build a client-side video editor in React!

Building a Video Editor in React with WASM

In this step-by-step tutorial, you will learn how to build a Wasm-based video editor in React with ffmpeg.wasm. This React video editor application will allow you to upload a video, trim it, and convert it to GIF in the browser, without using any backend functionality or external API.

Clone the GitHub repository that supports the tutorial and try the video editor application by launching the following commands:

git clone https://github.com/Tonel/video-editor-wasm-react
cd video-editor-wasm-react
npm install
npm run start

Let’s waste no more time and see how to build a video editor application in React.

Prerequisites

This is the list of libraries the video editor application you are about to build depends on:

Node.js and npm 8+ and higher
React 18.2+
[ffmpeg.wasm](https://github.com/ffmpegwasm/ffmpeg.wasm)
[antd](https://www.npmjs.com/package/antd) >= 4.3
[video-react](https://www.npmjs.com/package/video-react) >= 0.15

antd is one of the most popular UI libraries for React. In this tutorial, its [Slider](https://ant.design/components/slider/) component will be used to implement the video cutting feature. So, any other UI library including a slider component with a range option will do.

video-react is one of the most advanced and reliable HTML5 video players for React. You can replace it with any other React video player library.

Initializing a React Project

Let’s initialize a new Create React App React project called video-editor-wasm-react with the command below:

npx create-react-app video-editor-wasm-react

Your video-editor-wasm-react directory should now contain the following files:

video-editor-wasm-react
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    ├── reportWebVitals.js
    └── setupTests.js

Enter the video-editor-wasm-react folder in your terminal and launch a React local server with the commands below:

cd video-editor-wasm-react
npm start

Now, visit the [http://localhost:3000/](http://localhost:3000/) page in your browser, and you should be seeing the default Create React App screen.

Before installing ffmpeg.wasm, you need to get your server ready. As stated in the official documentation, fmmpeg.wasm depends on [SharedArrayBuffer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer). This means that only browsers that support SharedArrayBuffer can run ffmpeg.wasm. You can find a complete list of all browsers here.

In the most popular browsers, SharedArrayBuffer is only available to cross-origin isolated webpages. To enable this, you need to set the following headers in your server hosting the React application:

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

In a Create-React-App application, you achieve this by creating a src/setupProxy.js file and making sure it contains the following code:

module.exports = function (app) {
    app.use(function (req, res, next) {
        res.setHeader("Cross-Origin-Opener-Policy", "same-origin")
        res.setHeader("Cross-Origin-Embedder-Policy", "require-corp")
        next()
    })
}

Your local development server is now ready to use ffmpeg.wasm!

Installing the Project’s Dependencies

It is now time to add the aforementioned required dependencies to your project.

First, install ffmpeg.wasm with the following command:

npm install @ffmpeg/ffmpeg @ffmpeg/core

Now, let’s install Antd Design:

npm install antd

Finally, it is time to install Video-React:

npm install video-react redux

Note that video-react also requires [redux](https://www.npmjs.com/package/redux) to work.

You have now everything you need to start developing your React video editor!

Uploading a Video to the Video Editor

Create a VideoUpload component as below:

// src/components/VideoUpload.js

import { Button, Upload } from 'antd';

function VideoUpload({ disabled, onChange = () => {}, onRemove = () => {} }) {
  return (
    <>
      <Upload
        disabled={disabled}
        beforeUpload={() => {
          return false;
        }}
        accept="video/*"
        onChange={(info) => {
          if (info.fileList && info.fileList.length > 0) {
            onChange(info.fileList[0].originFileObj);
          }
        }}
        showUploadList={false}
      >
        <Button>Upload Video</Button>
      </Upload>
      <Button
        danger={true}
        disabled={!disabled}
        onClick={() => {
          onRemove(undefined);
        }}
      >
        Remove
      </Button>
    </>
  );
}

export default VideoUpload;

As you can see, this component uses the Antd Design Upload component to allow users to upload a video file. Then, a “Remove” button gives users the ability to remove the uploaded video. Note that when the Upload component is enabled, the “Remove” button is disabled and vice versa. This way, users can deal with only one video at a time.

Playing a Video in React

You now need a VideoPlayer component to play the uploaded video. You can build it as below:

// src/components/VideoPlayer.js

import {
  BigPlayButton,
  ControlBar,
  LoadingSpinner,
  Player,
  PlayToggle,
} from 'video-react';
import 'video-react/dist/video-react.css';
import { useEffect, useState } from 'react';

export function VideoPlayer({
  src,
  onPlayerChange = () => {},
  onChange = () => {},
  startTime = undefined,
}) {
  const [player, setPlayer] = useState(undefined);
  const [playerState, setPlayerState] = useState(undefined);

  useEffect(() => {
    if (playerState) {
      onChange(playerState);
    }
  }, [playerState]);

  useEffect(() => {
    onPlayerChange(player);

    if (player) {
      player.subscribeToStateChange(setPlayerState);
    }
  }, [player]);

  return (
    <div className={'video-player'}>
      <Player
        ref={(player) => {
          setPlayer(player);
        }}
        startTime={startTime}
      >
        <source src={src} />
        <BigPlayButton position="center" />
        <LoadingSpinner />
        <ControlBar autoHide={false} disableDefaultControls={true}>
          <PlayToggle />
        </ControlBar>
      </Player>
    </div>
  );
}

This is a simple video-react-based video player component, which only allows playing or stop the video file received with the src prop.

Cutting a Video in React

Your goal is to allow users to select a portion of the video and watch it before converting it into GIF. Implement this with the logic below:

import { Slider } from "antd"
import { sliderValueToVideoTime } from "../utils/utils"

function VideoEditor() {
  const [sliderValues, setSliderValues] = useState([0, 100])

  // logic to handle videoFile, videoPlayer, and videoPlayerState...

  useEffect(() => {
      const min = sliderValues[0]
      // when the slider values are updated, updating the
      // video time
      if (min !== undefined && videoPlayerState && videoPlayer) {
          videoPlayer.seek(sliderValueToVideoTime(videoPlayerState.duration, min))
      }
  }, [sliderValues])

  useEffect(() => {
      if (videoPlayer && videoPlayerState) {
          // allowing users to watch only the portion of
          // the video selected by the slider
          const [min, max] = sliderValues

          const minTime = sliderValueToVideoTime(videoPlayerState.duration, min)
          const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max)

          if (videoPlayerState.currentTime < minTime) {
              videoPlayer.seek(minTime)
          }
          if (videoPlayerState.currentTime > maxTime) {
              // looping logic
              videoPlayer.seek(minTime)
          }
      }
  }, [videoPlayerState])

  return (
          // other video editor components...
          <Slider
            disabled={!videoPlayerState}
            value={sliderValues}
            range={true}
            onChange={(values) => {
              setSliderValues(values)
            }}
            tooltip={{
              formatter: null,
            }}
          />
        // ...
}

This component equips users with the ability to select a portion of the uploaded video with the Antd Design Slider component and watch it in loop, before converting it into a GIF file.

Specifically, the logic here uses the sliderValueToVideoTime() function below to transform the slider values into time coordinates:

// src/utils/utils.js

export function sliderValueToVideoTime(duration, sliderValue) {
    return Math.round(duration * sliderValue / 100)
}

Note that the actual video processing slider will be performed by the next component through ffmpeg.

Converting a Video to GIF in React

To convert a video to GIF, create a VideoConversionButton component as follows:

// src/components/VideoConversionButton.js

import { Button } from 'antd';
import { fetchFile } from '@ffmpeg/ffmpeg';
import { sliderValueToVideoTime } from '../utils/utils';

function VideoConversionButton({
  videoPlayerState,
  sliderValues,
  videoFile,
  ffmpeg,
  onConversionStart = () => {},
  onConversionEnd = () => {},
  onGifCreated = () => {},
}) {
  const convertToGif = async () => {
    // starting the conversion process
    onConversionStart(true);

    const inputFileName = 'gif.mp4';
    const outputFileName = 'output.gif';

    // writing the video file to memory
    ffmpeg.FS('writeFile', inputFileName, await fetchFile(videoFile));

    const [min, max] = sliderValues;
    const minTime = sliderValueToVideoTime(videoPlayerState.duration, min);
    const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max);

    // cutting the video and converting it to GIF with an FFMpeg command
    await ffmpeg.run(
      '-i',
      inputFileName,
      '-ss',
      `${minTime}`,
      '-to',
      `${maxTime}`,
      '-f',
      'gif',
      outputFileName
    );

    // reading the resulting file
    const data = ffmpeg.FS('readFile', outputFileName);

    // converting the GIF file created by FFmpeg to a valid image URL
    const gifUrl = URL.createObjectURL(
      new Blob([data.buffer], { type: 'image/gif' })
    );
    onGifCreated(gifUrl);

    // ending the conversion process
    onConversionEnd(false);
  };

  return <Button onClick={() => convertToGif()}>Convert to GIF</Button>;
}

export default VideoConversionButton;

This component uses ffmpeg.wasm to launch an ffmpeg command. This command takes care of clipping the video to the time limits defined by sliderValues and converting it to GIF. Note that all these operations are performed client-side in the browser.

In detail, the ffmpeg -ss flag defines the time from which to start reading the video file, while -to defines the end time. Then, the gif option specifies that you want to produce a GIF file. Learn more about the flags and options offered by ffmpeg here.

The resulting GIF image is stored in the output.gif memory file, loaded as a valid "image/gif" Blob, and finally converted to a valid image URL with the [URL.createObjectURL()](https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL_static) native function.

Putting It All Together

Now, let’s see the full code of the VideoEditor component:

// src/components/VideoEditor.js

import { createFFmpeg } from '@ffmpeg/ffmpeg';
import { useEffect, useState } from 'react';
import { Slider, Spin } from 'antd';
import { VideoPlayer } from './VideoPlayer';
import { sliderValueToVideoTime } from '../utils/utils';
import VideoUpload from './VideoUpload';
import VideoConversionButton from './VideoConversionButton';
const ffmpeg = createFFmpeg({ log: true });

function VideoEditor() {
  const [ffmpegLoaded, setFFmpegLoaded] = useState(false);
  const [videoFile, setVideoFile] = useState();
  const [videoPlayerState, setVideoPlayerState] = useState();
  const [videoPlayer, setVideoPlayer] = useState();
  const [gifUrl, setGifUrl] = useState();
  const [sliderValues, setSliderValues] = useState([0, 100]);
  const [processing, setProcessing] = useState(false);

  useEffect(() => {
    // loading ffmpeg on startup
    ffmpeg.load().then(() => {
      setFFmpegLoaded(true);
    });
  }, []);

  useEffect(() => {
    const min = sliderValues[0];
    // when the slider values are updated, updating the
    // video time
    if (min !== undefined && videoPlayerState && videoPlayer) {
      videoPlayer.seek(sliderValueToVideoTime(videoPlayerState.duration, min));
    }
  }, [sliderValues]);

  useEffect(() => {
    if (videoPlayer && videoPlayerState) {
      // allowing users to watch only the portion of
      // the video selected by the slider
      const [min, max] = sliderValues;

      const minTime = sliderValueToVideoTime(videoPlayerState.duration, min);
      const maxTime = sliderValueToVideoTime(videoPlayerState.duration, max);

      if (videoPlayerState.currentTime < minTime) {
        videoPlayer.seek(minTime);
      }
      if (videoPlayerState.currentTime > maxTime) {
        // looping logic
        videoPlayer.seek(minTime);
      }
    }
  }, [videoPlayerState]);

  useEffect(() => {
    // when the current videoFile is removed,
    // restoring the default state
    if (!videoFile) {
      setVideoPlayerState(undefined);
      setSliderValues([0, 100]);
      setVideoPlayerState(undefined);
      setGifUrl(undefined);
    }
  }, [videoFile]);

  return (
    <div>
      <Spin
        spinning={processing || !ffmpegLoaded}
        tip={!ffmpegLoaded ? 'Waiting for FFmpeg to load...' : 'Processing...'}
      >
        <div>
          {videoFile ? (
            <VideoPlayer
              src={URL.createObjectURL(videoFile)}
              onPlayerChange={(videoPlayer) => {
                setVideoPlayer(videoPlayer);
              }}
              onChange={(videoPlayerState) => {
                setVideoPlayerState(videoPlayerState);
              }}
            />
          ) : (
            <h1>Upload a video</h1>
          )}
        </div>
        <div className={'upload-div'}>
          <VideoUpload
            disabled={!!videoFile}
            onChange={(videoFile) => {
              setVideoFile(videoFile);
            }}
          />
        </div>
        <div className={'slider-div'}>
          <h3>Cut Video</h3>
          <Slider
            disabled={!videoPlayerState}
            value={sliderValues}
            range={true}
            onChange={(values) => {
              setSliderValues(values);
            }}
            tooltip={{
              formatter: null,
            }}
          />
        </div>
        <div className={'conversion-div'}>
          <VideoConversionButton
            onConversionStart={() => {
              setProcessing(true);
            }}
            onConversionEnd={() => {
              setProcessing(false);
            }}
            ffmpeg={ffmpeg}
            videoPlayerState={videoPlayerState}
            sliderValues={sliderValues}
            videoFile={videoFile}
            onGifCreated={(girUrl) => {
              setGifUrl(girUrl);
            }}
          />
        </div>
        {gifUrl && (
          <div className={'gif-div'}>
            <h3>Resulting GIF</h3>
            <img
              src={gifUrl}
              className={'gif'}
              alt={'GIF file generated in the client side'}
            />
            <a
              href={gifUrl}
              download={'test.gif'}
              className={'ant-btn ant-btn-default'}
            >
              Download
            </a>
          </div>
        )}
      </Spin>
    </div>
  );
}

export default VideoEditor;

Since ffmpeg takes time to load, you need to let users know that they need to wait a while before they can use the video editor component. Also, ffmpeg operations take time to execute. This wait logic is implemented by using the Antd Design [Spin](https://ant.design/components/spin/) component.

Also, note that by passing the log: true option to the createFFmpeg() function, ffmpeg will log useful debug info in the console. For example, after ffmpeg gets loaded, you should be able to see the following lines in the browser console:

[info] use ffmpeg.wasm v0.11.5
createFFmpeg.js:43 [info] load ffmpeg-core
createFFmpeg.js:43 [info] loading ffmpeg-core
createFFmpeg.js:43 [info] ffmpeg-core loaded

What the VideoComponent does is use all the components presented earlier to allow users to upload a video, select a portion through a slider, convert it to GIF, and download the resulting GIF file.

Et voilà! You just learned how to build a client-side video editor in React without using any backend functionality!

Commercial Alternative

VideoEditor SDK by IMG.LY provides powerful video editing features, including cropping and trimming videos in your project. You will receive staples of video editing, including straightening videos, filters, brightness, color adjustments, and more.

Conclusion

Building a client-side video processing and editing application that uses only the browser possible. With WebAssembly, you can run high-performance code written in C or C++ in JavaScript. This allows you to perform complex operations at lightning speed directly in the browser, without having to delegate the work to a server.

In this tutorial, you learned how to use ffmpeg.wasm to build a video editor in React. ffmpeg.wasm is the Wasm port of ffmpeg and enables video and audio processing and editing directly in the browser. Using it to build a web application to upload a video, cut it, and export it to GIF requires only a few lines of code, and here we have seen how to implement such a video editing application.

If your app goes beyond merely displaying video, and you want to allow your users to also edit video or create video based templates in the browser, explore our VideoEditor SDK for the Web – a performant video editing solution based on WASM and WebCodecs. Try the editor for yourself in action with our showcases.

Thanks for reading! Let us know what you think on Twitter! To stay in the loop, subscribe to our Newsletter.

Video Editing – IMG.LY Blog

Build in a Day: AI Video Clipping with CE.SDK

Introduction

Why Client-Side?

Tech Stack

Architecture Overview

Setting Up CE.SDK

What is CE.SDK?

Installation

Initializing the CreativeEngine

Uploading Video to CE.SDK

Extracting Audio for Transcription

Getting Video Metadata

AI-Powered Transcription & Highlight Detection

The Pipeline

Transcription with Speaker Diarization

AI Highlight Detection with Gemini

Mapping Back to Timestamps

Working with the CE.SDK Timeline

Understanding Blocks

Manipulating Trim Points

Working with Fills and Their Timing

Creating Time-Based Edits from Transcript Words

Generating Speaker Thumbnails

Speaker Detection & Face Tracking

Why Semi-Automatic?

How It Works

Multi-Speaker Templates & Dynamic Switching

The Concept

Creating Picture-in-Picture with CE.SDK

Key Technique: Muting Duplicate Audio

Dynamic Speaker Switching

Preview, Playback & Export

Setting Up the Canvas

Playback Controls

Syncing UI State

Export Options

The Finished App

What’s Next

Ideas for Extension

Taking It Server-Side

Resources

Automating Video Creation at Scale: Why Templates, Timelines, and Rendering Matter

Why Video Automation Is Fundamentally Different

Timeline Complexity vs. Static Composition

Multi-Asset Coordination and Audio Sync

Rendering Performance and Infrastructure Implications

Designing Templates That Scale

Scene-Based Modular Structure

Text Duration and Audio Sync Strategies

Video Clip Placeholders and Safe Zones

Architecture Patterns for Video Generation Pipelines

Pattern 1: Template-Based Batch Rendering

Pattern 2: Real-Time Personalization

Pattern 3: Hybrid UI + Automation

Platform Constraints and Encoding Optimization

H.264 Profiles, Levels, and Trade-offs

Multi-Format Export for Social Platforms

Browser vs. Server-Side Rendering Trade-offs

Production Pitfalls and Monitoring

Error Handling and Recovery

Monitoring and Observability

Scaling and Cost Optimization

Why Most Teams Don’t Build Video Automation In-House

Real-World Use Cases

Personalized Video Campaigns at Scale

Social Media Content Automation

When Video Automation Makes Sense (and When It Doesn’t)

Good Fit: High-Volume, Template-Driven Use Cases

Poor Fit: Creative-Intensive, One-Off Productions

The Middle Ground: Hybrid Approaches

What to Do Next

Video Automation: Solvable, But Not Simple

Best Open Source Video Editor SDKs: 2026 Roundup

1. FFmpeg

Comparison to IMG.LY

2. MoviePy

Comparison to IMG.LY

3. OpenShot Library / MLT Framework

Comparison to IMG.LY