Batch Processing

In the browser, batch processing means automating the same CreativeEngine workflow while the tab stays open. Instead of the user editing/exporting items one by one, your front-end:

1. Loops through a dataset.
2. Produces a series of outputs.

This guides helps you understand how the CE.SDK can work in a batch process workflow.

What You’ll Learn#

• Two different batch processing approaches:

  • On the main thread
  • In an export worker

• How to batch:

  • Templates population with data.
  • Exports to different formats (PNG, JPEG, PDF, MP4).
  • Thumbnails generation.

• How to optimize memory usage.

Batch Processing Strategies#

The Engine is single-threaded, so a second instance in the same tab shares the main thread and buys no parallelism. What you choose is where each export renders:

• On the main thread: a single Engine loop. Each export freezes the page while it renders.
• In an export worker: the same loop with the exportWorker feature flag on, so each export renders in a Web Worker and the page stays interactive.

The following examples show both approaches when running a batch export in the browser:

import CreativeEngine from '@cesdk/engine';

async function downloadBlob(blob, filename) {
  const url = URL.createObjectURL(blob);
  const link = Object.assign(document.createElement('a'), { href: url, download: filename });
  link.click();
  URL.revokeObjectURL(url);
}

const engine = await CreativeEngine.init({
  license: 'YOUR_CESDK_LICENSE_KEY',
});

for (const record of records) {
  await engine.scene.load(record.scene);
  const [page] = engine.scene.getPages();
  const blob = await engine.block.export(page, { mimeType: 'image/png' });
  await downloadBlob(blob, `${record.id}.png`);
}

engine.dispose();

import CreativeEngine from '@cesdk/engine';

// downloadBlob as in the previous tab
const engine = await CreativeEngine.init({
  license: 'YOUR_CESDK_LICENSE_KEY',
  featureFlags: { exportWorker: true },
});

for (const record of records) {
  await engine.scene.load(record.scene);
  const [page] = engine.scene.getPages();
  const blob = await engine.block.export(page, { mimeType: 'image/png' });
  await downloadBlob(blob, `${record.id}.png`);
}

engine.dispose();

The following table summarizes the pros and cons of each approach:

How To Batch Template Population#

For this operation, you generate personalized outputs at scale by combining:

• Templates
• Structured data

Set the Data Sources#

Batch workflows can use a variety of data sources to populate a template, such as:

• CSV files with parsing libraries
• JSON from REST APIs
• Databases (SQL, NoSQL)
• Stream data

The following examples show how to set three different data sources:

await fetch('path/to/dataset.json').then((r) => r.json());

await fetch('https://api.example.com/dataset').then((r) => r.json());

// Define key variables
let textVariables = {
  first_name: '',
  last_name: '',
  address: '',
  city: '',
};

Update the Template#

You can automate template population, update media, and show conditional content based on data. Find some examples in existing guides:

Batch Export the Design#

The CE.SDK provides a set of format options when exporting the edited designs:

Check all the export options in the Export section.

Batch Thumbnail Generation from Static Scenes#

The export feature allows you to automate thumbnails generation by tweaking the format and the size of the design, for example:

import CreativeEngine from '@cesdk/engine';

// CreativeEngine.init is already headless: it renders to its own canvas, which
// stays out of the DOM until you append engine.element. It still runs on the
// main thread, so enable the export worker to keep thumbnails off it.
const thumbnailEngine = await CreativeEngine.init({
  license: 'YOUR_CESDK_LICENSE_KEY',
  featureFlags: { exportWorker: true },
});

async function generateThumbnail(sceneData) {
  await thumbnailEngine.scene.load(sceneData);
  const [page] = thumbnailEngine.scene.getPages();

  // Generate small preview
  const thumbnail = await thumbnailEngine.block.export(page, {
    mimeType: 'image/jpeg',
    targetWidth: 200,
    targetHeight: 200,
    jpegQuality: 0.7,
  });

  return thumbnail;
}

Set targetWidth and targetHeight together, or neither applies. They define a box the render fills while keeping the page’s aspect ratio, so a non-square page produces a thumbnail larger than 200×200 on one axis.

Read more about thumbnails generation in the Engine guide.

Batch Thumbnail Generation from Video Scenes#

Extract representative frames from videos efficiently, and automate this action using the dedicated CE.SDK features:

The following code shows how to generate thumbnails from a video:

import CreativeEngine from '@cesdk/engine';

const engine = await CreativeEngine.init({
  license: 'YOUR_CESDK_LICENSE_KEY',
});
await engine.scene.load('/assets/video-scene.imgly');

const [page] = engine.scene.getPages();
const videoBlock = engine.block
  .getChildren(page)
  .find(
    (child) =>
      engine.block.supportsFill(child) &&
      engine.block.getType(engine.block.getFill(child)) ===
        '//ly.img.ubq/fill/video'
  );

if (videoBlock) {
  // Decode the video before seeking, so the frame exists to render.
  await engine.block.forceLoadAVResource(engine.block.getFill(videoBlock));

  // Seek on the page: a fill's own playback time only drives rendering
  // under solo playback.
  engine.block.setPlaybackTime(page, 4.2);

  const thumbnail = await engine.block.export(page, {
    mimeType: 'image/png',
    targetWidth: 640,
    targetHeight: 360
  });

  await downloadBlob(thumbnail, 'scene-thumb.png');
}

engine.dispose();

The preceding code:

1. Loads a scene containing a video.
2. Finds the block whose fill is a video. getType returns the full type id, and video is a fill type, so there is no video block type to match on.
3. Waits for the video resource to decode.
4. Seeks the page to 4.2 s, which cascades the time down to the blocks it contains.
5. Exports the page as a PNG and saves the thumbnail.

Optimize Memory Usage#

Every export produces and accumulates:

• Blobs
• URLs
• Engine state

Proper cleanup ensures batch processes complete without resource exhaustion. Without proper cleanup, the browser might:

• Hits memory ceiling.
• Crash.
• Slow down.

Consider the following actions to avoid exhausting the client:

Treat cleanup as part of each loop iteration, by either:

• Freeing resources after each item.
• Chunking resources, by loading smaller parts of your datasets at a time.

Apply Error Handling#

Batch runs often work with large records of data. Some factors can make the job crash, such as:

• A malformed asset
• Timeouts

When your job encounters one of these errors, you can proactively avoid the job’s failure using the following patterns:

• Catch errors inside each loop iteration.
• Log failing records so you can retry them later.
• Decide whether to keep going or stop when an error happens.
• Collect a summary of all failures for post-run review.

For example, the preceding code to generate thumbnails now handles errors gracefully to avoid crashes:

import CreativeEngine from '@cesdk/engine';

let engine;
try {
  engine = await CreativeEngine.init({
    license: 'YOUR_CESDK_LICENSE_KEY',
  });
  await engine.scene.load('/assets/video-scene.imgly');

  const [page] = engine.scene.getPages();
  if (!page) throw new Error('Scene has no pages.');

  const videoBlock = engine.block
    .getChildren(page)
    .find(
      (child) =>
        engine.block.supportsFill(child) &&
        engine.block.getType(engine.block.getFill(child)) ===
          '//ly.img.ubq/fill/video'
    );
  if (!videoBlock) throw new Error('No video block found.');

  await engine.block.forceLoadAVResource(engine.block.getFill(videoBlock));
  engine.block.setPlaybackTime(page, 4.2);

  const thumbnail = await engine.block.export(page, {
    mimeType: 'image/png',
    targetWidth: 640,
    targetHeight: 360
  });

  await downloadBlob(thumbnail, 'scene-thumb.png');
} catch (error) {
  console.error('Failed to generate thumbnail', error);
} finally {
  engine?.dispose();
}

Use Retry Logic#

Some errors are temporary due to factors such as:

• Network hiccup
• Rate limits
• Busy CDN

To avoid saturating the related service, you can use smart retries after a short delay. If the error persist:

1. Double the delay.
2. Retry
3. Double again the delay exponentially after each retry.

This strategy allows you to identify temporary failures that could be resolved later.

For API failures, consider using circuit breaking patterns that:

• Pause the calls on repeated errors.
• Test again after a delay.

Check the Input Data Before Processing#

Lightweight checks can help you with:

• Catching bad inputs early.
• Preventing waste of time and compute on batches that’ll fail.

Add checks before:

• Launching the CE.SDK.
• Loading scenes.
• Exporting large scenes.

The following table contains some checks examples:

For example, the following data validation function checks:

• The record type
• The id
• The HTTPS template URL
• The presence of variants

It throws descriptive errors if any of these elements are missing.

function validateRecord(record) {
  if (typeof record !== 'object' || record === null) {
    throw new Error('Record must be an object');
  }
  if (typeof record.id !== 'string') {
    throw new Error('Missing record id');
  }
  if (!record.templateUrl?.startsWith('https://')) {
    throw new Error('Invalid template URL');
  }
  if (!Array.isArray(record.variants) || record.variants.length === 0) {
    throw new Error('Record requires at least one variant');
  }
  return true;
}

Batch Process on Production#

When running on production, enhance browser-based batch processes with architecture and UX decisions that help the user run the workflow, such as:

• User-initiated batches: keep work tied to explicit user actions; show confirmation dialogs for large jobs.
• Chunked processing: split datasets into small slices (for example, 20 records) and yield to the browser between slices. On the main thread this only spreads the freeze out, because each export still blocks for its own duration. Enable the exportWorker feature flag to remove the freeze itself.
• Resource caps: document safe limits (for example, 50–100 exports per session) and enforce them in the UI.
• Persistence: use localStorage or IndexedDB to cache progress so reloads can resume work.

Monitor the Process#

Give users visibility inside the tab and send lightweight telemetry upstream.

• Render UI elements that show the state, such as:

  • Progress bars
  • Per-item status chips

• Send fetch calls to your backend for:

  • Error logs
  • Aggregated stats

• When a chunk fails:

  1. Show in-app notifications/snackbars.
  2. Offer retries.

For example, the following code:

• Structures logging.
• Renders it with timestamps.

function reportBatchMetrics(batchMetrics) {
  const entry = {
    timestamp: new Date().toISOString(),
    ...batchMetrics,
  };
  console.table([entry]);
  return fetch('/api/logs', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(entry),
  });
}

Troubleshooting#

Debugging Strategies#

Effective troubleshooting techniques for batch processing in web apps include:

• Retry with small batches.
• Console log detailed error information.
• Isolate problematic items.

Next Steps#

• Headless Mode - Learn headless engine operation basics
• Design Generation - Automate single design generation workflows
• Export Designs - Deep dive into export options and formats
• Text Variables - Work with dynamic text content in templates
• Source Sets - Specify assets sources for each block.