Add Captions

For video scenes, open captions can be added in CE.SDK. These allow to follow the content without the audio.

Two blocks are available for this. The DesignBlockType.caption blocks hold the text of individual captions and the DesignBlockType.captionTrack is an optional structuring block to hold the Caption blocks, e.g., all captions for one video.

The "playback/timeOffset" property of each caption block controls when the caption should be shown and the "playback/duration" property how long the caption should be shown. Usually, the captions do not overlap. As the playback time of the page progresses, the corresponding caption is shown.

With the "caption/text" property, the text of the caption can be set. In addition, all text properties are also available for captions, e.g., to change the font, the font size, or the alignment.

Position, size, and style changes on caption blocks are automatically synced across all caption blocks.

Finally, the whole page can be exported as a video file using the block.exportVideo function.

Creating a Video Scene#

First, we create a scene that is set up for captions editing by calling the scene.createCaptions() API. Then we create a page, add it to the scene and define its dimensions. This page will hold our composition.

let scene = try engine.scene.createVideo()
let page = try engine.block.create(.page)
try engine.block.appendChild(to: scene, child: page)
try engine.block.setWidth(page, value: 1280)
try engine.block.setHeight(page, value: 720)
try engine.editor.setSettingBool("features/videoCaptionsEnabled", value: true)

Setting Page Durations#

Next, we define the duration of the page using the func setDuration(_ id: DesignBlockID, duration: Double) throws API to be 20 seconds long. This will be the total duration of our exported captions in the end.

try engine.block.setDuration(page, duration: 20)

Adding Captions#

In this example, we want to show two captions, one after the other. For this, we create two caption blocks.

let caption1 = try engine.block.create(.caption)
try engine.block.setString(caption1, property: "caption/text", value: "Caption text 1")
let caption2 = try engine.block.create(.caption)
try engine.block.setString(caption2, property: "caption/text", value: "Caption text 2")

As an alternative to manually creating the captions, changing the text, and adjusting the timings, the captions can also be loaded from a caption file, i.e., an SRT or VTT file with the createCaptionsFromURI API. This return a list of caption blocks, with the parsed texts and timings. These can be added to a caption track as well.

// Captions can also be loaded from a caption file, i.e., from SRT and VTT files.
// The text and timing of the captions are read from the file.
let captions = try await engine.block
  .createCaptionsFromURI(URL(string: "https://img.ly/static/examples/captions.srt")!)
for caption in captions {
  try engine.block.appendChild(to: captionTrack, child: caption)
}

Creating a Captions Track#

While we could add the two blocks directly to the page, we can alternatively also use the captionTrack block to group them.

Caption tracks themselves cannot be selected directly by clicking on the canvas, nor do they have any visual representation.

The dimensions of a captionTrack are always derived from the dimensions of its children, so you should not call the setWidth or setHeight APIs on a track, but on its children instead.

let captionTrack = try engine.block.create(.captionTrack)
try engine.block.appendChild(to: page, child: captionTrack)
try engine.block.appendChild(to: captionTrack, child: caption1)
try engine.block.appendChild(to: captionTrack, child: caption2)

Modifying Captions#

By default, each caption block has a duration of 3 seconds after it is created. If we want to show it on the page for a different amount of time, we can use the setDuration API.

try engine.block.setDuration(caption1, duration: 3)
try engine.block.setDuration(caption2, duration: 5)

The position and size of the captions is automatically synced across all captions that are attached to the scene. Therefore, changes only need to be made on one of the caption blocks.

// Once the captions are added to the scene, the position and size are synced with all caption blocks in the scene so
// only needs to be set once.
try engine.block.setPositionX(caption1, value: 0.05)
try engine.block.setPositionXMode(caption1, mode: .percent)
try engine.block.setPositionY(caption1, value: 0.8)
try engine.block.setPositionYMode(caption1, mode: .percent)
try engine.block.setHeight(caption1, value: 0.15)
try engine.block.setHeightMode(caption1, mode: .percent)
try engine.block.setWidth(caption1, value: 0.9)
try engine.block.setWidthMode(caption1, mode: .percent)

The styling of the captions is also automatically synced across all captions that are attached to the scene. For example, changing the text color to red on the first block, changes it on all caption blocks.

// The style is synced with all caption blocks in the scene so only needs to be set once.
try engine.block.setColor(caption1, property: "fill/solid/color", color: Color.rgba(r: 0.9, g: 0.9, b: 0.0, a: 1.0))
try engine.block.setBool(caption1, property: "dropShadow/enabled", value: true)
try engine.block.setColor(caption1, property: "dropShadow/color", color: Color.rgba(r: 0.0, g: 0.0, b: 0.0, a: 0.8))

Exporting Video#

You can start exporting the entire page as a video file by calling func exportVideo(_ id: DesignBlockID, mimeType: MIMEType). The encoding process will run in the background. You can get notified about the progress of the encoding process by the async stream that’s returned.

Since the encoding process runs in the background the engine will stay interactive. So, you can continue to use the engine to manipulate the scene. Please note that these changes won’t be visible in the exported video file because the scene’s state has been frozen at the start of the export.

// Export page as mp4 video.
let mimeType: MIMEType = .mp4
let exportTask = Task {
  for try await export in try await engine.block.exportVideo(page, mimeType: mimeType) {
    switch export {
    case let .progress(renderedFrames, encodedFrames, totalFrames):
      print("Rendered", renderedFrames, "frames and encoded", encodedFrames, "frames out of", totalFrames)
    case let .finished(video: videoData):
      return videoData
    }
  }
  return Blob()
}
let blob = try await exportTask.value

Full Code#

Here’s the full code:

import Foundation
import IMGLYEngine

@MainActor
func editVideoCaptions(engine: Engine) async throws {
  let scene = try engine.scene.createVideo()
  let page = try engine.block.create(.page)
  try engine.block.appendChild(to: scene, child: page)
  try engine.block.setWidth(page, value: 1280)
  try engine.block.setHeight(page, value: 720)
  try engine.editor.setSettingBool("features/videoCaptionsEnabled", value: true)

  try engine.block.setDuration(page, duration: 20)

  let caption1 = try engine.block.create(.caption)
  try engine.block.setString(caption1, property: "caption/text", value: "Caption text 1")
  let caption2 = try engine.block.create(.caption)
  try engine.block.setString(caption2, property: "caption/text", value: "Caption text 2")

  let captionTrack = try engine.block.create(.captionTrack)
  try engine.block.appendChild(to: page, child: captionTrack)
  try engine.block.appendChild(to: captionTrack, child: caption1)
  try engine.block.appendChild(to: captionTrack, child: caption2)

  try engine.block.setDuration(caption1, duration: 3)
  try engine.block.setDuration(caption2, duration: 5)

  try engine.block.setTimeOffset(caption1, offset: 0)
  try engine.block.setTimeOffset(caption2, offset: 3)

  // Captions can also be loaded from a caption file, i.e., from SRT and VTT files.
  // The text and timing of the captions are read from the file.
  let captions = try await engine.block
    .createCaptionsFromURI(URL(string: "https://img.ly/static/examples/captions.srt")!)
  for caption in captions {
    try engine.block.appendChild(to: captionTrack, child: caption)
  }

  // Once the captions are added to the scene, the position and size are synced with all caption blocks in the scene so
  // only needs to be set once.
  try engine.block.setPositionX(caption1, value: 0.05)
  try engine.block.setPositionXMode(caption1, mode: .percent)
  try engine.block.setPositionY(caption1, value: 0.8)
  try engine.block.setPositionYMode(caption1, mode: .percent)
  try engine.block.setHeight(caption1, value: 0.15)
  try engine.block.setHeightMode(caption1, mode: .percent)
  try engine.block.setWidth(caption1, value: 0.9)
  try engine.block.setWidthMode(caption1, mode: .percent)

  // The style is synced with all caption blocks in the scene so only needs to be set once.
  try engine.block.setColor(caption1, property: "fill/solid/color", color: Color.rgba(r: 0.9, g: 0.9, b: 0.0, a: 1.0))
  try engine.block.setBool(caption1, property: "dropShadow/enabled", value: true)
  try engine.block.setColor(caption1, property: "dropShadow/color", color: Color.rgba(r: 0.0, g: 0.0, b: 0.0, a: 0.8))

  // Export page as mp4 video.
  let mimeType: MIMEType = .mp4
  let exportTask = Task {
    for try await export in try await engine.block.exportVideo(page, mimeType: mimeType) {
      switch export {
      case let .progress(renderedFrames, encodedFrames, totalFrames):
        print("Rendered", renderedFrames, "frames and encoded", encodedFrames, "frames out of", totalFrames)
      case let .finished(video: videoData):
        return videoData
      }
    }
    return Blob()
  }
  let blob = try await exportTask.value
}