Control Audio & Video

In this example, we will show you how to use the CreativeEditor SDK's CreativeEngine to configure and control audio and video through the block API.

Setup#

This example uses the headless CreativeEngine. See the Setup article for a detailed guide. To get started right away, you can also access the block API within a running CE.SDK instance via cesdk.engine.block. Check out the APIs Overview to see that illustrated in more detail.

Time Offset and Duration#

The time offset determines when a block becomes active during playback on the page's timeline, and the duration decides how long this block is active. Blocks within tracks are a special case in that they have an implicitly calculated time offset that is determined by their order and the total duration of their preceding blocks in the same track. As with any audio/video-related property, not every block supports these properties. Use supportsTimeOffset and supportsDuration to check.

JavaScript


supportsTimeOffset(id: DesignBlockId): boolean

Returns whether the block has a time offset property.

id: The block to query.
Returns true, if the block has a time offset property.

JavaScript


setTimeOffset(id: DesignBlockId, offset: number): void

Set the time offset of the given block relative to its parent. The time offset controls when the block is first active in the timeline. The time offset is not supported by the page block.

id: The block whose time offset should be changed.
offset: The new time offset in seconds.

JavaScript


getTimeOffset(id: DesignBlockId): number

Get the time offset of the given block relative to its parent.

id: The block whose time offset should be queried.
Returns the time offset of the block.

JavaScript


supportsDuration(id: DesignBlockId): boolean

Returns whether the block has a duration property.

id: The block to query.
Returns true if the block has a duration property.

JavaScript


setDuration(id: DesignBlockId, duration: number): void

Set the playback duration of the given block in seconds. The duration defines for how long the block is active in the scene during playback. The duration is ignored when the scene is not in "Video" mode.

id: The block whose duration should be changed.
duration: The new duration in seconds.

JavaScript


getDuration(id: DesignBlockId): number

Get the playback duration of the given block in seconds.

id: The block whose duration should be returned.
Returns The block's duration.

JavaScript


setNativePixelBuffer(id: number, buffer: HTMLCanvasElement | HTMLVideoElement): void

Update the pixels of the given pixel stream fill block.

id: The pixel stream fill block.
buffer: Use pixel data from a canvas or a video element.

Trim#

You can select a specific range of footage from your audio/video resource by providing a trim offset and a trim length. The footage will loop if the trim's length is shorter than the block's duration. This behavior can also be disabled using the setLooping function.

JavaScript


supportsTrim(id: DesignBlockId): boolean

Returns whether the block has trim properties.

id: The block to query.
Returns true, if the block has trim properties.

JavaScript


setTrimOffset(id: DesignBlockId, offset: number): void

Set the trim offset of the given block or fill. Sets the time in seconds within the fill at which playback of the audio or video clip should begin. This requires the video or audio clip to be loaded.

id: The block whose trim should be updated.
offset: The new trim offset.

JavaScript


getTrimOffset(id: DesignBlockId): number

Get the trim offset of this block. * This requires the video or audio clip to be loaded.

id: The block whose trim offset should be queried.
Returns the trim offset in seconds.

JavaScript


setTrimLength(id: DesignBlockId, length: number): void

Set the trim length of the given block or fill. The trim length is the duration of the audio or video clip that should be used for playback. After reaching this value during playback, the trim region will loop. This requires the video or audio clip to be loaded.

id: The object whose trim length should be updated.
length: The new trim length in seconds.

JavaScript


getTrimLength(id: DesignBlockId): number

Get the trim length of the given block or fill.

id: The object whose trim length should be queried.
Returns the trim length of the object.

Playback Control#

You can start and pause playback and seek to a certain point on the scene's timeline. There's also a solo playback mode to preview audio and video blocks individually while the rest of the scene stays frozen. Finally, you can enable or disable the looping behavior of blocks and control their audio volume.

JavaScript


setPlaying(id: DesignBlockId, enabled: boolean): void

Set whether the block should be playing during active playback.

id: The block that should be updated.
enabled: Whether the block should be playing its contents.

JavaScript


isPlaying(id: DesignBlockId): boolean

Returns whether the block is playing during active playback.

id: The block to query.
Returns whether the block is playing during playback.

JavaScript


setSoloPlaybackEnabled(id: DesignBlockId, enabled: boolean): void

Set whether the given block or fill should play its contents while the rest of the scene remains paused. Setting this to true for one block will automatically set it to false on all other blocks.

id: The block or fill to update.
enabled: Whether the block's playback should progress as time moves on.

JavaScript


isSoloPlaybackEnabled(id: DesignBlockId): boolean

Return whether the given block or fill is currently set to play its contents while the rest of the scene remains paused.

id: The block or fill to query.
Returns Whether solo playback is enabled for this block.

JavaScript


supportsPlaybackTime(id: DesignBlockId): boolean

Returns whether the block has a playback time property.

id: The block to query.
Returns whether the block has a playback time property.

JavaScript


setPlaybackTime(id: DesignBlockId, time: number): void

Set the playback time of the given block.

id: The block whose playback time should be updated.
time: The new playback time of the block in seconds.

JavaScript


getPlaybackTime(id: DesignBlockId): number

Get the playback time of the given block.

id: The block to query.
Returns the playback time of the block in seconds.

JavaScript


isVisibleAtCurrentPlaybackTime(id: DesignBlockId): boolean

Returns whether the block should be visible on the canvas at the current playback time.

id: The block to query.
Returns Whether the block should be visible on the canvas at the current playback time.

JavaScript


supportsPlaybackControl(id: DesignBlockId): boolean

Returns whether the block supports a playback control.

block: The block to query.
Returns Whether the block has playback control.

JavaScript


setLooping(id: DesignBlockId, looping: boolean): void

Set whether the block should restart from the beginning again or stop.

block: The block or video fill to update.
looping: Whether the block should loop to the beginning or stop.

JavaScript


isLooping(id: DesignBlockId): boolean

Query whether the block is looping.

block: The block to query.
Returns Whether the block is looping.

JavaScript


setMuted(id: DesignBlockId, muted: boolean): void

Set whether the audio of the block is muted.

block: The block or video fill to update.
muted: Whether the audio should be muted.

JavaScript


isMuted(id: DesignBlockId): boolean

Query whether the block is muted.

block: The block to query.
Returns Whether the block is muted.

JavaScript


setVolume(id: DesignBlockId, volume: number): void

Set the audio volume of the given block.

block: The block or video fill to update.
volume: The desired volume with a range of 0, 1.

JavaScript


getVolume(id: DesignBlockId): number

Get the audio volume of the given block.

block: The block to query.
Returns The volume with a range of 0, 1.

JavaScript


getVideoWidth(id: DesignBlockId): number

Get the video width in pixels of the video resource that is attached to the given block.

block: The video fill.
Returns The video width in pixels or an error.

JavaScript


getVideoHeight(id: DesignBlockId): number

Get the video height in pixels of the video resource that is attached to the given block.

block: The video fill.
Returns The video height in pixels or an error.

Resource Control#

Until an audio/video resource referenced by a block is loaded, properties like the duration of the resource aren't available, and accessing those will lead to an error. You can avoid this by forcing the resource you want to access to load using forceLoadAVResource.

JavaScript


forceLoadAVResource(id: DesignBlockId): Promise<void>

Begins loading the required audio and video resource for the given video fill or audio block.

id: The video fill or audio block whose resource should be loaded.
Returns A Promise that resolves once the resource has finished loading.

JavaScript


unstable_isAVResourceLoaded(id: DesignBlockId): boolean

Returns whether the audio and video resource for the given video fill or audio block is loaded.

id: The video fill or audio block.
Returns The loading state of the resource.

JavaScript


getAVResourceTotalDuration(id: DesignBlockId): number

Get the duration in seconds of the video or audio resource that is attached to the given block.

id: The video fill or audio block.
Returns The video or audio file duration.

Thumbnail Previews#

For a user interface, it can be helpful to have image previews in the form of thumbnails for any given video resource. For videos, the engine can provide one or more frames using generateVideoThumbnailSequence. Pass the video fill that references the video resource. In addition to video thumbnails, the engine can also render compositions of design blocks over time. To do this pass in the respective design block. The video editor uses these to visually represent blocks in the timeline.

In order to visualize audio signals generateAudioThumbnailSequence can be used. This generates a sequence of values in the range of 0 to 1 that represent the loudness of the signal. These values can be used to render a waveform pattern in any custom style.

Note: there can be at most one thumbnail generation request per block at any given time. If you don't want to wait for the request to finish before issuing a new request, you can cancel it by calling the returned function.

JavaScript


generateVideoThumbnailSequence(id: DesignBlockId, thumbnailHeight: number, timeBegin: number, timeEnd: number, numberOfFrames: number, onFrame: (frameIndex: number, result: ImageData | Error) => void): () => void

Generate a sequence of thumbnails for the given video fill or design block. Note: there can only be one thumbnail generation request in progress for a given block.

id: The video fill or design block.
thumbnailHeight: The height of a thumbnail. The width will be calculated from the video aspect ratio.
timeBegin: The time in seconds relative to the time offset of the design block at which the thumbnail sequence should start.
timeEnd: The time in seconds relative to the time offset of the design block at which the thumbnail sequence should end.
numberOfFrames: The number of frames to generate.
onFrame: Gets passed the frame index and RGBA image data.
Returns A method that will cancel any thumbnail generation request in progress for this block.

JavaScript


generateAudioThumbnailSequence(id: DesignBlockId, samplesPerChunk: number, timeBegin: number, timeEnd: number, numberOfSamples: number, numberOfChannels: number, onChunk: (chunkIndex: number, result: Float32Array | Error) => void): () => void

Generate a thumbnail sequence for the given audio block or video fill. A thumbnail in this case is a chunk of samples in the range of 0 to 1. In case stereo data is requested, the samples are interleaved, starting with the left channel.

id: The audio block or video fill.
samplesPerChunk: The number of samples per chunk. onChunk is called when this many samples are ready.
timeBegin: The time in seconds at which the thumbnail sequence should start.
timeEnd: The time in seconds at which the thumbnail sequence should end.
numberOfSamples: The total number of samples to generate.
numberOfChannels: The number of channels in the output. 1 for mono, 2 for stereo.
onChunk: This gets passed an index and a chunk of samples whenever it's ready, or an error.


// Time Offset and Duration
engine.block.supportsTimeOffset(audio);
engine.block.setTimeOffset(audio, 2);
engine.block.getTimeOffset(audio); /* Returns 2 */

engine.block.supportsDuration(page);
engine.block.setDuration(page, 10);
engine.block.getDuration(page); /* Returns 10 */

// Trim
engine.block.supportsTrim(videoFill);
engine.block.setTrimOffset(videoFill, 1);
engine.block.getTrimOffset(videoFill); /* Returns 1 */
engine.block.setTrimLength(videoFill, 5);
engine.block.getTrimLength(videoFill); /* Returns 5 */

// Playback Control
engine.block.setPlaying(page, true);
engine.block.isPlaying(page);

engine.block.setSoloPlaybackEnabled(videoFill, true);
engine.block.isSoloPlaybackEnabled(videoFill);

engine.block.supportsPlaybackTime(page);
engine.block.setPlaybackTime(page, 1);
engine.block.getPlaybackTime(page);
engine.block.isVisibleAtCurrentPlaybackTime(block);

engine.block.supportsPlaybackControl(videoFill);
engine.block.setLooping(videoFill, true);
engine.block.isLooping(videoFill);
engine.block.setMuted(videoFill, true);
engine.block.isMuted(videoFill);
engine.block.setVolume(videoFill, 0.5); /* 50% volume */
engine.block.getVolume(videoFill);

// Resource Control
await engine.block.forceLoadAVResource(videoFill);
const isLoaded = engine.block.unstable_isAVResourceLoaded(videoFill);
engine.block.getAVResourceTotalDuration(videoFill);
const videoWidth = engine.block.getVideoWidth(videoFill);
const videoHeight = engine.block.getVideoHeight(videoFill);

// Thumbnail Previews
const cancelVideoThumbnailGeneration = engine.block.generateVideoThumbnailSequence(
  videoFill /* video fill or any design block */,
  128 /* thumbnail height, width will be calculated from aspect ratio */,
  0.5 /* begin time */,
  9.5 /* end time */,
  10 /* number of thumbnails to generate */,
  async (frame, width, height, imageData) => {
    const image = await createImageBitmap(imageData);
    // Draw the image...
  }
);
const cancelAudioThumbnailGeneration = engine.block.generateAudioThumbnailSequence(
  audio /* audio or video fill */,
  20 /* number of samples per chunk */,
  0.5 /* begin time */,
  9.5 /* end time */,
  10 * 20 /* total number of samples, will produce 10 calls with 20 samples per chunk */,
  2, /* stereo, interleaved samples for the left and right channels */,
  (chunkIndex, chunkSamples) => {
    drawWavePattern(chunkSamples);
  }
);

// Piping a native camera stream into the engine
const pixelStreamFill = engine.block.createFill('pixelStream');
const video = document.createElement('video');
engine.block.setNativePixelBuffer(pixelStreamFill, video);