Control Audio & Video
In this example, we will show you how to use the CreativeEditor SDK's CreativeEngine to configure and control audio and video through the block
API.
Setup#
This example uses the headless CreativeEngine. See the Setup article for a detailed guide.
To get started right away, you can also access the block
API within a running CE.SDK instance via cesdk.engine.block
.
Check out the APIs Overview to see that illustrated in more detail.
Offset and Duration#
The offset determines when a block becomes active during playback on the scene's timeline, and the duration decides how long this block is active.
Page blocks are a special case in that they have an implicitly calculated offset that is determined by their order and the total duration of their preceding pages.
As with any audio/video-related property, not every block supports these properties. Use hasTimeOffset
and hasDuration
to check.
hasTimeOffset(id: number): bool
returns true if the block has a time offset property.
setTimeOffset(id: number, TimeOffset: number): void
sets the time offset of the given block in seconds.
getTimeOffset(id: number): number
returns the time offset of the given block in seconds.
hasDuration(id: number): bool
returns true if the block has a duration property.
setDuration(id: number, duration: number): void
sets the playback duration of the given block in seconds.
getDuration(id: number): number
returns the playback duration of the given block in seconds.
getTotalSceneDuration(id: number): number
returns the total scene duration in seconds.
Trim#
You can select a specific range of footage from your audio/video resource by providing a trim offset and a trim length. The footage will loop if the trim's length is shorter than the block's duration. This behavior can also be disabled using the setLooping
function.
hasTrim(id: number): bool
returns true if trim is applicable to this block.
setTrimOffset(id: number, offset: number): void
sets the trim offset of the given block in seconds.
getTrimOffset(id: number): number
returns the trim offset of the given block in seconds.
setTrimLength(id: number, length: number): void
sets the trim length of the given block in seconds.
getTrimLength(id: number): number
returns the trim length of the given block in seconds.
Playback Control#
You can start and pause playback and seek to a certain point on the scene's timeline. There's also a solo playback mode to preview audio and video blocks individually while the rest of the scene stays frozen. Finally, you can enable or disable the looping behavior of blocks and control their audio volume.
setPlaying(id: number, enabled: bool): void
sets whether the block should be playing during playback.
isPlaying(id: number): bool
returns true if the block is playing during playback.
setSoloPlaybackEnabled(id: number, enabled: bool): void
sets whether the block should play while the rest of the scene remains paused.
isSoloPlaybackEnabled(id: number): bool
returns true if the solo playback mode is active for the given block.
hasPlaybackTime(id: number): bool
returns true if the block has a playback time property.
setPlaybackTime(id: number, time: number): void
sets the playback time of the given block in seconds.
getPlaybackTime(id: number): number
returns the playback time of the given block in seconds.
hasPlaybackControl(id: number): bool
returns true if the block supports playback control.
setLooping(id: number, looping: bool): void
sets whether the block should restart from the beginning again or just stop.
isLooping(id: number): bool
returns true if the block is looping.
setMuted(id: number, muted: bool): void
sets whether the block's audio is muted.
isMuted(id: number): bool
returns true if the block is muted.
setVolume(id: number, volume: bool): void
sets the volume of the block in the range of [0, 1].
getVolume(id: number): bool
returns the volume of the block in the range of [0, 1].
Resource Control#
Until an audio/video resource referenced by a block is loaded, properties like the duration of the resource aren't available, and accessing those will lead to an error. You can avoid this by forcing the resource you want to access to load using forceLoadAVResource
.
async forceLoadAVResource(id: number): Promise<void>
returns a promise that resolves once the resource has finished loading.
getAVResourceTotalDuration(id: number): number
returns the duration in seconds of the video or audio resource referenced by the given block.
Thumbnail Previews#
For a user interface, it can be helpful to have image previews in the form of thumbnails for any given video resource. For videos, the engine can provide a single frame using getVideoFillThumbnail
. Or you can obtain an atlas. That is a two-dimensional grid of frames taken at an evenly spaced interval across the entire video from left to right, top to bottom. Finally, the engine can generate a complete page composition atlas over time. The video editor uses these to visually represent and preview pages in the timeline. They can be regenerated whenever an edit is made to a page.
async getVideoFillThumbnail(videoFill: number, thumbnailHeight: number): Promise<Blob>
returns a thumbnail encoded as JPEG. The thumbnail width will be calculated from the video's aspect ratio.
async getVideoFillThumbnailAtlas(videoFill: number, numberOfColumns: number, numberOfRows: number, thumbnailHeight: number): Promise<Blob>
returns a thumbnail atlas encoded as JPEG. The width will be calculated from the video's aspect ratio.
async getPageThumbnailAtlas(videoFill: number, numberOfColumns: number, numberOfRows: number, thumbnailHeight: number): Promise<Blob>
returns a thumbnail atlas encoded as JPEG. The width will be calculated from the page's aspect ratio.
// Offset and Durationengine.block.hasTimeOffset(audio);engine.block.setTimeOffset(audio, 2);engine.block.getTimeOffset(audio); /* Returns 2 */engine.block.hasDuration(page);engine.block.setDuration(page, 10);engine.block.getDuration(page); /* Returns 10 */engine.block.getTotalSceneDuration(scene);// Trimengine.block.hasTrim(videoFill);engine.block.setTrimOffset(videoFill, 1);engine.block.getTrimOffset(videoFill); /* Returns 1 */engine.block.setTrimLength(videoFill, 5);engine.block.getTrimLength(videoFill); /* Returns 5 */// Playback Controlengine.block.setPlaying(scene, true);engine.block.isPlaying(scene);engine.block.setSoloPlaybackEnabled(videoFill, true);engine.block.isSoloPlaybackEnabled(videoFill);engine.block.hasPlaybackTime(scene);engine.block.setPlaybackTime(scene, 1);engine.block.getPlaybackTime(scene);engine.block.hasPlaybackControl(videoFill);engine.block.setLooping(videoFill, true);engine.block.isLooping(videoFill);engine.block.setMuted(videoFill, true);engine.block.isMuted(videoFill);engine.block.setVolume(videoFill, 0.5); /* 50% volume */engine.block.getVolume(videoFill);// Resource Controlawait engine.block.forceLoadAVResource(videoFill);engine.block.getAVResourceTotalDuration(videoFill);// Thumbnail Previewsconst videoThumb = await engine.block.getVideoFillThumbnail(videoFill, 128);const videoThumbAtlas = await engine.block.getVideoFillThumbnailAtlas(videoFill, 4, 4, 128);const pageThumbAtlas = await engine.block.getPageThumbnailAtlas(page, 4, 4, 128);