How to Use the Camera

Other than having pre-recorded video in your scene you can also have a live preview from a camera in the engine. This allows you to make full use of the engine's capabilities such as effects, strokes and drop shadows, while the preview integrates with the composition of your scene. Simply swap out the VideoFill of a block with a PixelStreamFill. This guide shows you how the PixelStreamFill can be used in combination with a camera.

Explore a full code sample on Stackblitz or view the code on Github.

Setup#

This example uses the headless CreativeEngine. See the Setup article for a detailed guide. To get started right away, you can also access the block API within a running CE.SDK instance via cesdk.engine.block. Check out the APIs Overview to see that illustrated in more detail.

We create a video scene with a single page. Then we create a PixelStreamFill and assign it to the page. To demonstrate the live preview capabilities of the engine we also apply an effect to the page.

Orientation#

To not waste expensive compute time by transforming the pixel data of the buffer itself, it's often beneficial to apply a transformation during rendering and let the GPU handle this work much more efficiently. For this purpose the PixelStreamFill has an orientation property. You can use it to mirror the image or rotate it in 90° steps. This property lets you easily mirror an image from a front facing camera or rotate the image by 90° when the user holds a device sideways.

Camera#

We use the MediaDevices.getUserMedia() API to prompt the user for permission to use a media input video device. This can be a camera, video recording device, or a screen sharing service. To access frames of the MediaStream we create an HTML video element. Once the video meta data is available we adjust the page size to the match the video resolution and zoom the camera to include the entire page.

Updating the Fill#

We use the video's requestVideoFrameCallback to update the contents of the PixelStreamFill. The callback is fired whenever a new video frame is available. We use the engine's setNativePixelBuffer method to update the fill with the new video frame. This method is optimized to avoid unnecessary copies of the pixel data. The method accepts either an HTMLVideoElement or an HTMLCanvasElement as input.


import CreativeEngine from 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.39.0/index.js';

const config = {
  license: 'mtLT-_GJwMhE7LDnO8KKEma7qSuzWuDxiKuQcxHKmz3fjaXWY2lT3o3Z2VdL5twm',
  userId: 'guides-user',
  baseURL: 'https://cdn.img.ly/packages/imgly/cesdk-engine/1.39.0/assets'
};

CreativeEngine.init(config).then(async (engine) => {
  document.getElementById('cesdk_container').append(engine.element);

  engine.scene.createVideo();
  const stack = engine.block.findByType('stack')[0];
  const page = engine.block.create('page');
  engine.block.appendChild(stack, page);

  const pixelStreamFill = engine.block.createFill('pixelStream');
  engine.block.setFill(page, pixelStreamFill);

  engine.block.appendEffect(page, engine.block.createEffect('half_tone'));

  // Horizontal mirroring
  engine.block.setEnum(
    pixelStreamFill,
    'fill/pixelStream/orientation',
    'UpMirrored'
  );

  navigator.mediaDevices.getUserMedia({ video: true }).then(
    (stream) => {
      const video = document.createElement('video');
      video.autoplay = true;
      video.srcObject = stream;
      video.addEventListener('loadedmetadata', () => {
        engine.block.setWidth(page, video.videoWidth);
        engine.block.setHeight(page, video.videoHeight);
        engine.scene.zoomToBlock(page, 40, 40, 40, 40);
        const onVideoFrame = () => {
          engine.block.setNativePixelBuffer(pixelStreamFill, video);
          video.requestVideoFrameCallback(onVideoFrame);
        };
        video.requestVideoFrameCallback(onVideoFrame);
      });
    },
    (err) => {
      console.error(err);
    }
  );
});