Skip to content

Record Video

Other than having pre-recorded video in your scene you can also have a live preview from a camera in the engine. This allows you to make full use of the engine’s capabilities such as effects, strokes and drop shadows, while the preview integrates with the composition of your scene. Simply swap out the VideoFill of a block with a PixelStreamFill. This guide shows you how the PixelStreamFill can be used in combination with a camera.

We create a video scene with a single page. Then we create a PixelStreamFill and assign it to the page. To demonstrate the live preview capabilities of the engine we also apply an effect to the page.

import CreativeEngine from 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.51.0/index.js';


const config = {
  license: 'mtLT-_GJwMhE7LDnO8KKEma7qSuzWuDxiKuQcxHKmz3fjaXWY2lT3o3Z2VdL5twm',
  userId: 'guides-user',
  baseURL: 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.51.0/assets'
};


CreativeEngine.init(config).then(async (engine) => {
  document.getElementById('cesdk_container').append(engine.element);


  engine.scene.createVideo();
  const stack = engine.block.findByType('stack')[0];
  const page = engine.block.create('page');
  engine.block.appendChild(stack, page);


  const pixelStreamFill = engine.block.createFill('pixelStream');
  engine.block.setFill(page, pixelStreamFill);


  engine.block.appendEffect(page, engine.block.createEffect('half_tone'));

Orientation

To not waste expensive compute time by transforming the pixel data of the buffer itself, it’s often beneficial to apply a transformation during rendering and let the GPU handle this work much more efficiently. For this purpose the PixelStreamFill has an orientation property. You can use it to mirror the image or rotate it in 90° steps. This property lets you easily mirror an image from a front facing camera or rotate the image by 90° when the user holds a device sideways.

// Horizontal mirroring
engine.block.setEnum(
  pixelStreamFill,
  'fill/pixelStream/orientation',
  'UpMirrored'
);

Camera

We use the MediaDevices.getUserMedia() API to prompt the user for permission to use a media input video device. This can be a camera, video recording device, or a screen sharing service. To access frames of the MediaStream we create an HTML video element. Once the video meta data is available we adjust the page size to the match the video resolution and zoom the camera to include the entire page.

navigator.mediaDevices.getUserMedia({ video: true }).then(
  (stream) => {
    const video = document.createElement('video');
    video.autoplay = true;
    video.srcObject = stream;
    video.addEventListener('loadedmetadata', () => {
      engine.block.setWidth(page, video.videoWidth);
      engine.block.setHeight(page, video.videoHeight);
      engine.scene.zoomToBlock(page, 40, 40, 40, 40);

Updating the Fill

We use the video’s requestVideoFrameCallback to update the contents of the PixelStreamFill. The callback is fired whenever a new video frame is available. We use the engine’s setNativePixelBuffer method to update the fill with the new video frame. This method is optimized to avoid unnecessary copies of the pixel data. The method accepts either an HTMLVideoElement or an HTMLCanvasElement as input.

const onVideoFrame = () => {
  engine.block.setNativePixelBuffer(pixelStreamFill, video);
  video.requestVideoFrameCallback(onVideoFrame);
};
video.requestVideoFrameCallback(onVideoFrame);

Full Code

Here’s the full code:

import CreativeEngine from 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.51.0/index.js';


const config = {
  license: 'mtLT-_GJwMhE7LDnO8KKEma7qSuzWuDxiKuQcxHKmz3fjaXWY2lT3o3Z2VdL5twm',
  userId: 'guides-user',
  baseURL: 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.51.0/assets',
};


CreativeEngine.init(config).then(async engine => {
  document.getElementById('cesdk_container').append(engine.element);


  engine.scene.createVideo();
  const stack = engine.block.findByType('stack')[0];
  const page = engine.block.create('page');
  engine.block.appendChild(stack, page);


  const pixelStreamFill = engine.block.createFill('pixelStream');
  engine.block.setFill(page, pixelStreamFill);


  engine.block.appendEffect(page, engine.block.createEffect('half_tone'));


  // Horizontal mirroring
  engine.block.setEnum(
    pixelStreamFill,
    'fill/pixelStream/orientation',
    'UpMirrored',
  );


  navigator.mediaDevices.getUserMedia({ video: true }).then(
    stream => {
      const video = document.createElement('video');
      video.autoplay = true;
      video.srcObject = stream;
      video.addEventListener('loadedmetadata', () => {
        engine.block.setWidth(page, video.videoWidth);
        engine.block.setHeight(page, video.videoHeight);
        engine.scene.zoomToBlock(page, 40, 40, 40, 40);
        const onVideoFrame = () => {
          engine.block.setNativePixelBuffer(pixelStreamFill, video);
          video.requestVideoFrameCallback(onVideoFrame);
        };
        video.requestVideoFrameCallback(onVideoFrame);
      });
    },
    err => {
      console.error(err);
    },
  );
});