Generate Designs From Your Backend - CoDesign

Turn a design and a job, such as “make this poster an Instagram story and a LinkedIn banner”, into one finished design per format, from an HTTP endpoint or a worker. For a complete app, see the Agentic Resize starter kit.

The code on this page is the reference, and every function that is yours to write is named in it. The example job is a resize: one design in, one re-composed design per target format out, with no person in the loop.

Before You Start#

Run the Vercel AI SDK quickstart first. You reuse three things from that starter kit:

its executor script and .env, which start the executor, IMG.LY’s published Docker image imgly/codesign-executor:1.83.0; the executor holds CE.SDK, IMG.LY’s design engine, and runs the code the model writes
the model client in its design.mjs, which reaches the model through IMG.LY’s AI gateway, a model API paid with your CoDesign license key
the way its design.mjs hands the SDK’s tools to the AI SDK, including imagesAfterToolResults, which passes the images the tools return on to the model (Add Guardrails says why)

Two secrets, both on your servers only:

Your CoDesign license key (IMGLY_CESDK_LICENSE). Contact sales to get a CoDesign license key. The executor container runs with it and charges exports to your account ( Billing), and your job sends it to IMG.LY’s AI gateway as its API key for the model.
EXECUTOR_TOKEN, a shared secret. The executor accepts sessions only from a runner that sends it, so it goes into the container’s environment and into your job’s remoteExecutors options.

The kit’s .env holds both. Its token works as it is on your machine; in production, use a long random string of your own, such as from openssl rand -hex 32:

IMGLY_CESDK_LICENSE=<your CoDesign license key>
EXECUTOR_TOKEN=codesign-quickstart-token

Install the SDK at the image’s version, because your service and the executor must run the same SDK build ( Versions and Upgrades). The code uses names from version 7 of the AI SDK (instructions, isStepCount), so install ai at the version below. Then start the executor from the kit’s folder:

npm install --save-exact @imgly/codesign-sdk@1.83.0 ai@7.0.118
npm run executor

curl http://localhost:8080/health answers with "bound": false: the executor is up and serves no request yet.

A few more terms appear below. For how they fit together, see Architecture; for every option and type, see the API Reference.

The executor is one running container of that image: an HTTP server that holds one CE.SDK engine and runs the JavaScript the model writes. Your service, the agent runner, runs the agent loop: it sends the model the conversation, runs each tool (a function the SDK hands your agent framework) the model asks for, and sends back the result. It calls the executor.
An engine session is one design loaded into the executor’s engine, so a run of calls on that design loads it once.
A design is stored as a scene, a text document that describes every page, text, and image, or as an .imgly file, a zip of the scene with the images and fonts it uses.
conversation is your id for one piece of work. Every design of a conversation opens in one executor that serves no other conversation. Here, one caller’s request is one conversation.
remoteExecutors tells the SDK how to reach an executor: its acquire answers the executor of a conversation.
plugins are the CE.SDK plugin modules the executor’s engine applies: the fonts and images the agent can use.

How the Job Works#

For each requested format, the job does this:

Prepare the tools. Call prepareCoDesign with the request’s conversation, your remoteExecutors, and your plugins.
Load the caller’s original design. codesign.load(bytes) replaces the design with the file and returns its version, a value that changes on every save. Every tool result names the version, and the model’s next edit (a change it makes by running its own code) must name it, so the model never builds on a design it has not looked at. Put the version in the model’s first message, or its first edit is refused for lack of one (for why, see Concurrent Edits). load reads .imgly and scene files, images, and SVG, and, through the executor’s importers, Photoshop (PSD), InDesign (IDML), PowerPoint (PPTX), and PDF files. Those four importers report what they could not carry over, such as a substituted font, as warnings; any load also warns when a picture in the file stays inside the design because no source took it.
Let the model re-compose it. Hand codesign.tools to your agent loop, as in the quickstart, and append codesign.instructions to your system prompt. Your prompt says what the job is, such as the new canvas size, and names none of the tools.
Export the result. codesign.export('imgly') returns the .imgly file, the editable design with its images, and keepArchive, yours, puts it in your bucket and answers the URL the caller downloads. It is free, as a scene string is; every other file format is charged ( Billing). For video and audio, see Status.
Release the design. codesign.release() ends the design’s engine session on the executor at once. Without it, the session stays open until its 3-minute idle timeout, beside the next format’s.

resizeDesign takes the request and the functions that are yours to write (ResizeDeps: the model, fetchSource for the caller’s file, and keepArchive). It makes one conversation, runs the formats one after another, and returns one entry per format: the URL of its .imgly file, or the error that format failed with:

export async function resizeDesign(
  deps: ResizeDeps,
  request: ResizeRequest
): Promise<ResizedDesign[]> {
  const source = await deps.fetchSource(new URL(request.sourceUrl));
  // This request's conversation: every format's design opens in one executor
  // that no other request ever uses, because the model's code runs there.
  const conversation = crypto.randomUUID();
  const designs: ResizedDesign[] = [];
  try {
    for (const format of request.formats) {
      designs.push(await resizeOne(deps, source, format, conversation));
    }
  } finally {
    // The request is over: the pool forgets its executor, which exits 30
    // seconds after its last engine session ends ("Run the Executors").
    pool.release(conversation);
  }
  return designs;
}

Each format is one short job:

async function resizeOne(
  deps: ResizeDeps,
  source: Uint8Array,
  format: Format,
  conversation: string
): Promise<ResizedDesign> {
  // One CoDesign per format, with no `designs` store: it keeps one new design
  // in its own memory, which lives as long as this object.
  const codesign = prepareCoDesign({
    conversation,
    executors: executorsFor(),
    // Google Fonts for the model's font lookup and the PSD importer. The
    // images a design carries stay inside its .imgly file.
    plugins: [googleFonts()],
    gateway: false // a resize re-composes what the design has; { bearer } adds image generation
  });
  const guard = repeatedErrorGuard(); // one per format ("Add Guardrails")
  try {
    const { version } = await codesign.load(source); // the ORIGINAL, every time
    const run = await generateText({
      model: deps.model,
      // Your prompt, then the SDK's: how to work with its tools.
      instructions: `${PRODUCT_PROMPT}\n\n${codesign.instructions}`,
      tools: toolsFor(codesign.tools, guard),
      abortSignal: guard.signal,
      stopWhen: isStepCount(60), // without it, the run ends after one tool call
      prompt: resizeBrief(format, version)
    });
    // Anything but 'stop', such as the step cap reached or a reply cut off, is a failed format.
    if (run.finishReason !== 'stop') {
      throw new Error(`the model stopped early: ${run.finishReason}`);
    }
    const archive = await codesign.export('imgly'); // the editable design with its images; free
    return { ...format, archiveUrl: await deps.keepArchive(archive) };
  } catch (err) {
    return {
      ...format,
      error: err instanceof Error ? err.message : String(err)
    };
  } finally {
    // Ends this format's engine session now rather than after its idle timeout.
    await codesign.release().catch(() => undefined);
  }
}

The whole job, resize-design.ts

import { googleFonts, prepareCoDesign } from '@imgly/codesign-sdk';
import { generateText, isStepCount, type LanguageModel } from 'ai';

import { executorsFor, pool } from './executors.js';
import { repeatedErrorGuard, toolsFor } from './tools.js';

export interface ResizeRequest {
  sourceUrl: string; // the design to re-compose: an .imgly, a PSD, an image
  formats: { label: string; width: number; height: number }[];
}

type Format = ResizeRequest['formats'][number];
export type ResizedDesign = Format &
  ({ archiveUrl: string } | { error: string });

// Yours to write. Each is a few lines over something your service has.
export interface ResizeDeps {
  /** The model: IMG.LY's AI gateway through createOpenAICompatible, as in the quickstart, or your provider's AI SDK package. */
  model: LanguageModel;
  /** The caller's file, fetched with your own checks ("Secure the Endpoint"). */
  fetchSource(url: URL): Promise<Uint8Array>;
  /** Keeps a finished .imgly (your bucket); returns the URL the caller downloads. */
  keepArchive(archive: Uint8Array): Promise<string>;
}

// Your product's standing policy. It names none of the SDK's tools.
const PRODUCT_PROMPT = [
  'You re-compose designs for new canvas sizes for our customers.',
  'When you are done, say in one sentence what you changed.'
].join('\n');

export async function resizeDesign(
  deps: ResizeDeps,
  request: ResizeRequest
): Promise<ResizedDesign[]> {
  const source = await deps.fetchSource(new URL(request.sourceUrl));
  // This request's conversation: every format's design opens in one executor
  // that no other request ever uses, because the model's code runs there.
  const conversation = crypto.randomUUID();
  const designs: ResizedDesign[] = [];
  try {
    for (const format of request.formats) {
      designs.push(await resizeOne(deps, source, format, conversation));
    }
  } finally {
    // The request is over: the pool forgets its executor, which exits 30
    // seconds after its last engine session ends ("Run the Executors").
    pool.release(conversation);
  }
  return designs;
}

async function resizeOne(
  deps: ResizeDeps,
  source: Uint8Array,
  format: Format,
  conversation: string
): Promise<ResizedDesign> {
  // One CoDesign per format, with no `designs` store: it keeps one new design
  // in its own memory, which lives as long as this object.
  const codesign = prepareCoDesign({
    conversation,
    executors: executorsFor(),
    // Google Fonts for the model's font lookup and the PSD importer. The
    // images a design carries stay inside its .imgly file.
    plugins: [googleFonts()],
    gateway: false // a resize re-composes what the design has; { bearer } adds image generation
  });
  const guard = repeatedErrorGuard(); // one per format ("Add Guardrails")
  try {
    const { version } = await codesign.load(source); // the ORIGINAL, every time
    const run = await generateText({
      model: deps.model,
      // Your prompt, then the SDK's: how to work with its tools.
      instructions: `${PRODUCT_PROMPT}\n\n${codesign.instructions}`,
      tools: toolsFor(codesign.tools, guard),
      abortSignal: guard.signal,
      stopWhen: isStepCount(60), // without it, the run ends after one tool call
      prompt: resizeBrief(format, version)
    });
    // Anything but 'stop', such as the step cap reached or a reply cut off, is a failed format.
    if (run.finishReason !== 'stop') {
      throw new Error(`the model stopped early: ${run.finishReason}`);
    }
    const archive = await codesign.export('imgly'); // the editable design with its images; free
    return { ...format, archiveUrl: await deps.keepArchive(archive) };
  } catch (err) {
    return {
      ...format,
      error: err instanceof Error ? err.message : String(err)
    };
  } finally {
    // Ends this format's engine session now rather than after its idle timeout.
    await codesign.release().catch(() => undefined);
  }
}

function resizeBrief(
  { label, width, height }: Format,
  version: string
): string {
  return [
    `Resize the design already open to ${width}×${height} px (the "${label}" format).`,
    `It is at version ${version}.`,
    'Re-derive margins, type sizes and spacing from the new canvas; never stretch.'
  ].join('\n');
}

Five decisions shape the loop:

Load the original for every format. Every output is a re-composition of the source, never a resize of a resize. The bytes are fetched once and loaded into each format’s own design.
Your code loads and exports. The model only designs. The caller asked for one design per format and must get them back, whatever the model does. uploadExport, the prepareCoDesign option that lets the model hand files to people, is left out here; without it, the model cannot forget to export. Its tools let it read the handbook (IMG.LY’s design guidance for the model, with a guide for resizing), edit the design, look at a preview of it, and search what your plugins register. Starting and listing designs refuse in words here, because the job passes no design store.
The brief is about the job, not the tools. The rules for re-composing a design live in the handbook’s resize guide. Naming the SDK’s tools or handbook pages in your prompt makes a second copy that drifts when the SDK changes them.
One format’s failure is not the request’s. Catch the error per format and return a list with a result or an error for each one. Your endpoint maps it to a status: 200 when all landed, 207 when some did, 502 when none did.
Run formats one after another. All of a request’s formats open on its executor’s one engine, which runs one call at a time. Formats run in parallel would only take turns on that engine, and each switch reloads the other format’s design.

The job passes no designs, the prepareCoDesign option that reads and writes designs in your database. Without one, the design lives in the memory of that codesign object until codesign.export() hands your code the result. That fits a short job like this one. When your service edits designs it keeps, pass a design store as designs and the design’s id as the design option ( Connect Your Storage).

The images a loaded .imgly file or PSD carries, and any the model generates, stay inside the design, and each format’s .imgly file holds them, so plugins: [googleFonts()] is all this job configures. To keep them in your bucket instead, for a CDN that serves the designs, add a source that stores new files, as that guide shows.

Your code can also work on a design without the model, with the same engine and sources: codesign.create() starts a design from a template, a file, or a copy, codesign.edit() runs a function of yours on it to read or set data such as a text or a color, and codesign.export() renders it. For their arguments and results, see CoDesign in the API Reference.

Run the Executors#

Each request gets an executor of its own. An executor serves one conversation, the first that opens a design on it, and answers every other with the error kind executor_bound. It exits shortly after that conversation’s last engine session has closed, and only a new container is free again. So your acquire hands each request an executor no request used before, and the same one for every call of that request.

This example keeps a pool of started executors: EXECUTOR_URLS lists their addresses, comma-separated, and EXECUTOR_TOKEN is the token they were started with. acquire takes one whose GET /health answers "bound": false, and something of yours replaces each used container with a new one at the same address:

import { remoteExecutors, type AcquiredExecutor } from '@imgly/codesign-sdk';

// Every executor's address, e.g. 'http://executor-1:8080,http://executor-2:8080'.
const urls = (process.env.EXECUTOR_URLS ?? '').split(',').filter(Boolean);

// Hands out executors, one per request, and takes them back when it ends.
export const pool = (() => {
  const held = new Map<string, string>(); // conversation → its executor's url
  // A fresh executor: bound to no conversation yet, and accepting sessions.
  const unbound = async (url: string): Promise<boolean> => {
    try {
      const health = (await (await fetch(`${url}/health`)).json()) as {
        result?: { bound?: boolean; accepting?: boolean };
      };
      return health.result?.bound === false && health.result.accepting === true;
    } catch {
      return false; // down or restarting: not a candidate now
    }
  };
  const pick = async (conversation: string) => {
    const mine = held.get(conversation);
    if (mine !== undefined) return mine; // the request's own, every time
    const taken = new Set(held.values());
    for (const url of urls) {
      if (taken.has(url) || !(await unbound(url))) continue;
      held.set(conversation, url);
      return url;
    }
    return undefined; // every executor is busy: the call fails as executor_unreachable
  };
  // One pick at a time, so two requests never claim the same executor between
  // its health check and the claim.
  let queue: Promise<unknown> = Promise.resolve();
  return {
    acquire: async ({
      conversation
    }: {
      conversation: string;
    }): Promise<AcquiredExecutor | undefined> => {
      const picked = queue.then(() => pick(conversation));
      queue = picked.catch(() => undefined);
      const url = await picked;
      return url === undefined ? undefined : { url };
    },
    // The request ended. Its executor reads as bound until its container is
    // replaced, so the health check keeps it out of the pool until then.
    release: (conversation: string) => {
      held.delete(conversation);
    }
  };
})();

export const executorsFor = () =>
  remoteExecutors({
    acquire: pool.acquire,
    // The value the executors were started with as EXECUTOR_TOKEN.
    token: process.env.EXECUTOR_TOKEN
  });

You can also start one container per request instead of keeping a pool, such as an Amazon Elastic Container Service (ECS) task or a Kubernetes pod per conversation id. A pool answers faster, because a container started for the request must start its engine first.

When every executor is busy, acquire answers undefined and the format fails with kind executor_unreachable. This service does not wait; to wait, retry the format by kind. isCoDesignError(err) tells the SDK’s errors apart. If you branch on err.kind, keep a branch for kinds you do not know, because new kinds are added in minor versions. For how long an executor lives, how to replace it, and every way a call to it fails, see Host the Executor.

Add Guardrails#

The agent runs with no person watching, and every model call is charged. The SDK has no cost cap and no limit on failed tool calls, because the agent loop is yours. Add both before the first unattended run:

A repeated-error guard stops a run when tool calls keep failing the same way. Some errors are worded as retryable, so a starved machine otherwise produces an agent that retries until the budget is gone.
A cost cap per format. Price each step from the token counts your model client reports, and fail the run when a step reports none, since a cap that adds up zeros never stops. The example does not show one: the prices are your model’s.

The guard keys on the error text, not the tool, because a dead backend fails every tool. toolsFor records every tool result in it, and resizeDesign makes one per format and passes its signal to generateText:

/** Aborts a run after `limit` identical tool failures in a row, across all tools. */
export function repeatedErrorGuard(limit = 5) {
  const stop = new AbortController();
  let last: string | undefined;
  let count = 0;
  return {
    signal: stop.signal, // pass as generateText's abortSignal
    record(isError: boolean, text: string): void {
      const key = isError ? text.slice(0, 200) : undefined; // a varying tail is the same error
      count = key === undefined ? 0 : key === last ? count + 1 : 1;
      last = key;
      if (count >= limit) {
        stop.abort(
          new Error(`${count} identical tool failures in a row: ${key}`)
        );
      }
    }
  };
}

The tool mapping with the guard, tools.ts

import type { CoDesignTool, ToolResultPart } from '@imgly/codesign-sdk';
import { jsonSchema, tool } from 'ai';

// The SDK's tools in the Vercel AI SDK's shape (the `ai` package, version 7).
export function toolsFor(
  tools: readonly CoDesignTool[], // codesign.tools
  guard?: { record(isError: boolean, text: string): void } // repeatedErrorGuard, below
) {
  return Object.fromEntries(
    tools.map((t) => [
      t.name,
      tool({
        description: t.description,
        inputSchema: jsonSchema(t.inputSchema),
        execute: async (args, { abortSignal }) => {
          let result;
          try {
            result = await t.call(args, { signal: abortSignal });
          } catch (err) {
            // A call can throw too (an executor that went away): the same failure.
            guard?.record(
              true,
              err instanceof Error ? err.message : String(err)
            );
            throw err;
          }
          guard?.record(result.isError === true, textOf(result.content));
          // Throw on failure: the AI SDK then tells the model the call failed.
          if (result.isError) throw new Error(textOf(result.content));
          return result.content;
        },
        // Hand images to the model as images, not as JSON text.
        toModelOutput: ({ output }) => ({
          type: 'content' as const,
          value: output.map(toAiSdkPart)
        })
      })
    ])
  );
}

export function textOf(parts: readonly ToolResultPart[]): string {
  return parts
    .map((part) => (part.type === 'text' ? part.text : ''))
    .join('\n')
    .trim();
}

function toAiSdkPart(part: ToolResultPart) {
  return part.type === 'text'
    ? { type: 'text' as const, text: part.text }
    : {
        type: 'file' as const,
        mediaType: part.mimeType,
        data: { type: 'data' as const, data: part.data }
      };
}

/** Aborts a run after `limit` identical tool failures in a row, across all tools. */
export function repeatedErrorGuard(limit = 5) {
  const stop = new AbortController();
  let last: string | undefined;
  let count = 0;
  return {
    signal: stop.signal, // pass as generateText's abortSignal
    record(isError: boolean, text: string): void {
      const key = isError ? text.slice(0, 200) : undefined; // a varying tail is the same error
      count = key === undefined ? 0 : key === last ? count + 1 : 1;
      last = key;
      if (count >= limit) {
        stop.abort(
          new Error(`${count} identical tool failures in a row: ${key}`)
        );
      }
    }
  };
}

Each of these fails silently when left out of the mapping:

turning isError into a throw, or the model reads a failed edit as a success
passing image parts through as images, or the agent designs blind
setting stopWhen, or the AI SDK’s default ends the run after the first tool call
checking finishReason, or a reply cut off at the output limit looks like a finished run

IMG.LY’s gateway speaks the Chat Completions API, whose tool results hold only text, so the AI SDK’s provider for it sends a tool result’s images as text. The images must therefore also follow the tool result in a message of their own. The quickstart’s imagesAfterToolResults does that in prepareStep (see the Vercel AI SDK quickstart).

Pin the model with its full provider id in source. A short alias may not exist in your agent framework’s catalog, and fails on your server with “unknown model”.

Secure the Endpoint#

Fetch the caller’s file yourself, with an allow-list of hosts, a size limit, and a timeout, in your fetchSource, and pass the bytes to load. load with a URL fetches whatever the caller names, metadata endpoints and localhost included.
Put authentication in front of the endpoint. An open endpoint spends your IMG.LY credits with no limit. Make the check fail closed: when its configuration is missing, answer 503. An unauthenticated request must get 401, never a design.
Run the service as a job. A typical design job takes around 3 minutes, depending on the model and how hard the brief is, which exceeds many HTTP timeouts. Answer POST with 202 { jobId }, keep an outcome per format so a retry reruns only the failed ones, and let the caller poll or follow server-sent events.

Next Steps#

Host the Executor — run a new, hardened container per request, and see how long an executor lives.
Concurrent Edits — see what changes when your service edits designs it keeps.
Add a Design Chat — add your designs and sources per client, brands, your own tools, and files handed to the person.
Agentic Resize — run the same job as a complete app.
Troubleshooting — find what to check when a job fails.