Add Captions

For video scenes, open captions can be added in CE.SDK. These allow to follow the content without the audio.

Two blocks are available for this. The Caption blocks hold the text of individual captions and the CaptionTrack is an optional structuring block to hold the Caption blocks, e.g., all captions for one video.

The "playback/timeOffset" property of each caption block controls when the caption should be shown and the "playback/duration" property how long the caption should be shown. Usually, the captions do not overlap. As the playback time of the page progresses, the corresponding caption is shown.

With the "caption/text" property, the text of the caption can be set. In addition, all text properties are also available for captions, e.g., to change the font, the font size, or the alignment.

Position, size, and style changes on caption blocks are automatically synced across all caption blocks.

Finally, the whole page can be exported as a video file using the block.exportVideo function.

A Note on Browser Support#

Video mode heavily relies on modern features like web codecs. A detailed list of supported browser versions can be found in our Supported Browsers. Please also take note of possible restrictions based on the host platform browsers are running on.

Creating a Video Scene#

First, we create a scene that is set up for captions editing by calling the scene.createVideo() API. Then we create a page, add it to the scene and define its dimensions. This page will hold our composition.

  const scene = engine.scene.createVideo();

  const page = engine.block.create('page');
  engine.block.appendChild(scene, page);

  engine.block.setWidth(page, 1280);
  engine.block.setHeight(page, 720);

Setting Page Duration#

Next, we define the duration of the page using the setDuration(block: number, duration: number): void API to be 20 seconds long. This will be the total duration of our exported captions in the end.

engine.block.setDuration(page, 20);

Adding Captions#

In this example, we want to show two captions, one after the other. For this, we create two caption blocks.

const caption1 = engine.block.create('caption');
engine.block.setString(caption1, 'caption/text', 'Caption text 1');
const caption2 = engine.block.create('caption');
engine.block.setString(caption2, 'caption/text', 'Caption text 2');

As an alternative to manually creating the captions, changing the text, and adjusting the timings, the captions can also be loaded from a caption file, i.e., an SRT or VTT file with the createCaptionsFromURI API. This return a list of caption blocks, with the parsed texts and timings. These can be added to a caption track as well.

// Captions can also be loaded from a caption file, i.e., from SRT and VTT files.
// The text and timing of the captions are read from the file.
const captions = await api.block.createCaptionsFromURI('https://img.ly/static/examples/captions.srt');
for (let i = 0; i < captions.length; i++) {
  api.block.appendChild(captionTrack, captions[i]);
}

Creating a Captions Track#

While we could add the two blocks directly to the page, we can alternatively also use the captionTrack block to group them.

Caption tracks themselves cannot be selected directly by clicking on the canvas, nor do they have any visual representation.

We create a captionTrack block, add it to the page and add both captions in the order in which they should play as the track’s children.

The dimensions of a captionTrack are always derived from the dimensions of its children, so you should not call the setWidth or setHeight APIs on a track, but on its children instead.

const captionTrack = engine.block.create('captionTrack');
engine.block.appendChild(page, captionTrack);
engine.block.appendChild(captionTrack, caption1);
engine.block.appendChild(captionTrack, caption2);

Modifying Captions#

By default, each caption block has a duration of 3 seconds after it is created. If we want to show it on the page for a different amount of time, we can use the setDuration API.

engine.block.setDuration(caption1, 3);
engine.block.setDuration(caption2, 5);

The position and size of the captions is automatically synced across all captions that are attached to the scene. Therefore, changes only need to be made on one of the caption blocks.

// Once the captions are added to the scene, the position and size are synced with all caption blocks in the scene so only needs to be set once.
engine.block.setPositionX(caption1, 0.05);
engine.block.setPositionXMode(caption1, "Percent");
engine.block.setPositionY(caption1, 0.8);
engine.block.setPositionYMode(caption1, "Percent");
engine.block.setHeight(caption1, 0.15);
engine.block.setHeightMode(caption1, "Percent");
engine.block.setWidth(caption1, 0.9);
engine.block.setWidthMode(caption1, "Percent");

The styling of the captions is also automatically synced across all captions that are attached to the scene. For example, changing the text color to red on the first block, changes it on all caption blocks.

// The style is synced with all caption blocks in the scene so only needs to be set once.
engine.block.setColor(caption1, "fill/solid/color", { r: 0.9, g: 0.9, b: 0.0, a: 1.0 });
engine.block.setBool(caption1, "dropShadow/enabled", true);
engine.block.setColor(caption1, "dropShadow/color", { r: 0.0, g: 0.0, b: 0.0, a: 0.8 });

Exporting Video#

You can start exporting the entire page as a captions file by calling exportVideo(). The encoding process will run in the background. You can get notified about the progress of the encoding process by the progressCallback. It will be called whenever another frame has been encoded.

Since the encoding process runs in the background the engine will stay interactive. So, you can continue to use the engine to manipulate the scene. Please note that these changes won’t be visible in the exported captions file because the scene’s state has been frozen at the start of the export.

  /* Export page as mp4 video. */
  const blob = await engine.block.exportVideo(
    page,
    {
      mimeType: 'video/mp4',
      onProgress: (renderedFrames, encodedFrames, totalFrames) => {
        console.log(
          'Rendered',
          renderedFrames,
          'frames and encoded',
          encodedFrames,
          'frames out of',
          totalFrames
        );
      }
    }
  );

  /* Download blob. */
  const anchor = document.createElement('a');
  anchor.href = URL.createObjectURL(blob);
  anchor.download = 'exported-video.mp4';
  anchor.click();

Full Code#

Here’s the full code:

import CreativeEngine from 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.55.1/index.js';

const config = {
  license: 'mtLT-_GJwMhE7LDnO8KKEma7qSuzWuDxiKuQcxHKmz3fjaXWY2lT3o3Z2VdL5twm',
  userId: 'guides-user',
  baseURL:
    'https://cdn.img.ly/packages/imgly/cesdk-engine/1.55.1/assets',
};


CreativeEngine.init(config).then(async (engine) => {
  // Attach engine canvas to DOM
  document.getElementById('cesdk_container').append(engine.element);
  const scene = engine.scene.createVideo();

  const page = engine.block.create('page');
  engine.block.appendChild(scene, page);

  engine.block.setWidth(page, 1280);
  engine.block.setHeight(page, 720);

  engine.block.setDuration(page, 20);

  const caption1 = engine.block.create('caption');
  engine.block.setString(caption1, 'caption/text', 'Caption text 1');
  const caption2 = engine.block.create('caption');
  engine.block.setString(caption2, 'caption/text', 'Caption text 2');

  const captionTrack = engine.block.create('captionTrack');
  engine.block.appendChild(page, captionTrack);
  engine.block.appendChild(captionTrack, caption1);
  engine.block.appendChild(captionTrack, caption2);

  engine.block.setDuration(caption1, 3);
  engine.block.setDuration(caption2, 5);

  engine.block.setTimeOffset(caption1, 0);
  engine.block.setTimeOffset(caption2, 3);

  // Captions can also be loaded from a caption file, i.e., from SRT and VTT files.
  // The text and timing of the captions are read from the file.
  const captions = await api.block.createCaptionsFromURI('https://img.ly/static/examples/captions.srt');
  for (let i = 0; i < captions.length; i++) {
    api.block.appendChild(captionTrack, captions[i]);
  }

  // Once the captions are added to the scene, the position and size are synced with all caption blocks in the scene so only needs to be set once.
  engine.block.setPositionX(caption1, 0.05);
  engine.block.setPositionXMode(caption1, "Percent");
  engine.block.setPositionY(caption1, 0.8);
  engine.block.setPositionYMode(caption1, "Percent");
  engine.block.setHeight(caption1, 0.15);
  engine.block.setHeightMode(caption1, "Percent");
  engine.block.setWidth(caption1, 0.9);
  engine.block.setWidthMode(caption1, "Percent");

  // The style is synced with all caption blocks in the scene so only needs to be set once.
  engine.block.setColor(caption1, "fill/solid/color", { r: 0.9, g: 0.9, b: 0.0, a: 1.0 });
  engine.block.setBool(caption1, "dropShadow/enabled", true);
  engine.block.setColor(caption1, "dropShadow/color", { r: 0.0, g: 0.0, b: 0.0, a: 0.8 });

  /* Export page as mp4 video. */
  const blob = await engine.block.exportVideo(
    page,
    {
      mimeType: 'video/mp4',
      onProgress: (renderedFrames, encodedFrames, totalFrames) => {
        console.log(
          'Rendered',
          renderedFrames,
          'frames and encoded',
          encodedFrames,
          'frames out of',
          totalFrames
        );
      }
    }
  );

  /* Download blob. */
  const anchor = document.createElement('a');
  anchor.href = URL.createObjectURL(blob);
  anchor.download = 'exported-video.mp4';
  anchor.click();
});