How to Edit Videos

In addition to static designs, CE.SDK also allows you to create and edit videos. Working with videos introduces the concept of time into the scene, which requires you to switch the scene into the "Video" mode. The "playback/time" property of the scene then controls the progress of time through the scene.

Such a video scene only displays one page at a time. The pages are shown one after the other in the order in which they are added as children of the scene's stack block. By default, each page is visible for five seconds, but this value can be adjusted for each page via the API.

In order to add videos to your pages, you can add a block with a "//ly.img.ubq/fill/video" fill. As the playback time of the scene progresses, the corresponding point in time of the video fill is rendered by the block.

You can also customize the video fill's trim in order to control the portion of the video that should be looped while the page is visible.

//ly.img.ubq/audio blocks can be added to the scene in order to play an audio file during playback. The playback/timeOffset property controls after how many seconds the audio should begin to play, while the duration property defines how long the audio should play.

Finally, the whole scene can be exported as a video file using the block.exportVideo function.

Explore a full code sample of the integration on CodeSandbox or view the code on GitHub

Setup#

This example uses the headless Creative Engine. See the Setup article for a detailed guide. To get started right away, you can also access the block API within a running CE.SDK instance via cesdk.engine.block. Check out the APIs Overview to see that illustrated in more detail.

Creating a Video Scene#

First, we create a scene that is set up for video editing by calling the scene.createVideo() API. Then we add two pages to the scene's stack in the order in which they should appear in the video.

Setting Page Durations#

Next, we define for how long each page should be shown in the video using the setDuration(block: number, duration: number): void API. Here, we want the first page to be visible for four seconds before then switching to the second page and displaying that for another 20 seconds.

Our final exported video will therefore be 24 seconds long, which can also be queried using getTotalSceneDuration(scene: number): number.

Adding Videos#

In order to add existing videos to our scene, we create a 'video' fill and assign it to a rect shape. Adding the rect shape as a child of the second page will now show the video for the same 20 seconds that the second page is visible.

If the video is longer than the duration of the page, only its first 20 seconds will be played. If it is too short, the video will automatically loop for as long as its parent page is visible.

We can also manually define the portion of our video that should loop within the page using the setTrimOffset(block: number, offset: number): void and setTrimLength(block: number, length: number): void APIs. We use the trim offset to cut away the first second of the video and the trim length to only play 10 seconds of the video. Since our page is twice as long, the video will be played a total of two times.

We can control if a video will loop back to its beginning by calling setLooping(block: number, looping: boolean): void. Otherwise, the video will simply hold its last frame instead and audio will stop playing. Looping behavior is activated for all blocks by default.

Audio#

If the video of a video fill contains an audio track, that audio will play automatically by default when the video is playing. We can mute it by calling setMuted(block: number, muted: boolean): void.

We can also add audio-only files to play together with the contents of the scene by adding an 'audio' block to the scene and assigning it the URL of the audio file.

Since the audio block is added to the scene and not to a page, it can play its audio simultaneously with multiple pages as they become appear and disappear throughout the duration of the scene playback.

We can adjust the volume level of any audio block or video fill by calling setVolume(block: number, volume: number): void. The volume is given as a fraction in the range of 0 to 1.

By default, our audio block will start playing at the very beginning of the scene. We can change this by specifying how many seconds into the scene it should begin to play using the setTimeOffset(block: number, offset: number): void API.

By default, our audio block will have a duration of 5 seconds. We can change this by specifying its duration in seconds by using the setDuration(block: number, duration: number): void API.

Exporting Video#

You can start exporting the entire scene as a video file by calling exportVideo(). The encoding process will run in the background. You can get notified about the progress of the encoding process by the progressCallback. It will be called whenever another frame has been encoded.

Since the encoding process runs in the background the engine will stay interactive. So, you can continue to use the engine to manipulate the scene. Please note that these changes won't be visible in the exported video file because the scene's state has been frozen at the start of the export.

File:

import CreativeEngine from 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.16.1/index.js';

const config = {
  baseURL: 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.16.1/assets'
};

CreativeEngine.init(config).then(async (engine) => {
  // Attach engine canvas to DOM
  document.getElementById('cesdk_container').append(engine.element);
  const scene = engine.scene.createVideo();
  const stack = engine.block.findByType('stack')[0];

  const page1 = engine.block.create('page');
  const page2 = engine.block.create('page');
  engine.block.appendChild(stack, page1);
  engine.block.appendChild(stack, page2);

  engine.block.setWidth(page1, 1280);
  engine.block.setHeight(page1, 720);
  engine.block.setWidth(page2, 1280);
  engine.block.setHeight(page2, 720);

  /* Show the first page for 4 seconds and the second page for 20 seconds. */
  engine.block.setDuration(page1, 4);
  engine.block.setDuration(page2, 20);

  const rectShape = engine.block.create('shapes/rect');
  engine.block.destroy(engine.block.getFill(rectShape));
  const videoFill = engine.block.createFill('video');
  engine.block.setFill(rectShape, videoFill);

  engine.block.setString(
    videoFill,
    'fill/video/fileURI',
    'https://cdn.img.ly/assets/demo/v1/ly.img.video/videos/pexels-drone-footage-of-a-surfer-barrelling-a-wave-12715991.mp4'
  );

  engine.block.appendChild(page2, rectShape);
  engine.block.setPositionX(rectShape, 0);
  engine.block.setPositionY(rectShape, 0);
  engine.block.setWidth(rectShape, engine.block.getWidth(page2));
  engine.block.setHeight(rectShape, engine.block.getHeight(page2));

  /* Make sure that the video is loaded before calling the trim APIs. */
  await engine.block.forceLoadAVResource(videoFill);
  engine.block.setTrimOffset(videoFill, 1);
  engine.block.setTrimLength(videoFill, 10);

  engine.block.setLooping(videoFill, true);

  engine.block.setMuted(videoFill, true);

  const audio = engine.block.create('audio');
  engine.block.appendChild(scene, audio);
  engine.block.setString(
    audio,
    'audio/fileURI',
    'https://cdn.img.ly/assets/demo/v1/ly.img.audio/audios/far_from_home.m4a'
  );

  /* Set the volume level to 70%. */
  engine.block.setVolume(audio, 0.7);

  /* Start the audio after two seconds of playback. */
  engine.block.setTimeOffset(audio, 2);

  /* Give the Audio block a duration of 7 seconds. */
  engine.block.setDuration(audio, 7);

  /* Export scene as mp4 video. */
  const mimeType = 'video/mp4';
  const progressCallback = (renderedFrames, encodedFrames, totalFrames) => {
    console.log('Rendered', renderedFrames, 'frames and encoded', encodedFrames, 'frames out of', totalFrames);
  };
  const blob = await engine.block.exportVideo(scene, mimeType, progressCallback, {});

  /* Download blob. */
  const anchor = document.createElement('a');
  anchor.href = URL.createObjectURL(blob);
  anchor.download = 'exported-video.mp4';
  anchor.click();
});