Tools

A tool is a function the model may ask your agent loop to call; the agent loop is your code that calls the model and runs its tool calls. Each one is an object with a name, a description the model reads, an inputSchema (its arguments as a JSON Schema object), and call, which your loop runs. prepareCoDesign returns them as codesign.tools. Hand the model the whole list, and append codesign.instructions to your system prompt:

const system = `${yourProductPrompt}\n\n${codesign.instructions}`;

codesign.instructions tells the model to read the handbook (IMG.LY’s design rules for the model) before its first edit, to look at its work, to score a finished design against the handbook’s checklist, and to carry each design’s version. If you leave it out, nothing fails, but the model learns about versions only from its first refused edit.

Every design has a version that changes with each save, such as b7b86c2a8e11.7. An edit or export must name the version the last result gave, so the model never builds on a design it has not seen. If it names an older one, the edit is refused, and the refusal tells the model the current version ( Concurrent Edits).

The code on this page is type-checked against the SDK, but no example runs it: your executor lookup (acquire), your design store (designs), and your brand lookup (brandFor) are yours to write.

What the Agent Can Do#

Tool names and descriptions can change in every release. They are what the model reads, and IMG.LY tunes them as models change. So neither your code nor your prompts depend on them: match no tool by its name, name none in your prompt, and hand the model codesign.tools and codesign.instructions as they are.

Your options fix the list. Which tools exist, and their text, follow from your options alone (uploadExport, gateway, tools, designs, and design), never from the executor (IMG.LY’s container that runs the engine) or from what your plugins register. So the text is the same on every start, and a model provider’s prompt cache holds. A capability the current setup cannot serve is still offered, and refuses at the call with a sentence that names what is missing. For example, an export the model asks for in a video format is refused, because video export is not supported ( Status).

Every call on a design takes the design it works on. Left out, it is the design you passed as prepareCoDesign’s design option, which the model’s text calls “the design this conversation is about”; with a designs store and no design option, the model must name one. The tools also take your store’s own version, the version that codesign.load() and codesign.edit() return to your code, so a prompt of yours can pass that one on.

The read-only tools carry annotations: { readOnlyHint: true }, the field the Model Context Protocol (MCP, a standard for tool servers) uses for this. Keep it when you pass the tools to your framework: a framework that honors it runs the viewing calls at once and the changing calls in order. Parallel edits are safe either way: of two edits on one version, one is kept and the other is told to look again.

Tool Results: Text and Images#

A tool result is a list of parts, each a text or an image:

type ToolResult = {
  content: (
    | { type: 'text'; text: string }
    | { type: 'image'; data: string; mimeType: string } // data: base64
  )[];
  isError?: boolean;
};

Every successful result of an edit, a preview, a started design, and an export starts with a JSON text part that names the design and its version, such as { "design": "d42", "version": "b7b86c2a8e11.7" }. The model passes both back on its next call.

A call ends in one of three ways:

• Success. Hand content to the model.

• isError: true. The model made a mistake, or the design refused it: its code threw, an argument was wrong, or the design changed. The text says what to do next. Hand it to the model as a failed call.

• A rejected promise. Nothing the model can fix: no executor answers, an export or a generation was not charged, your store or one of your asset sources failed, or a design was saved by a newer CE.SDK. The error is a CoDesignError whose kind says why, and your loop decides whether to tell the person, retry, or end the run. For every kind, who gets it, and what to do, see Errors.

  The call was also canceled when your signal aborted it (you pass one as tool.call(args, { signal })). A retry is safe: an edit that was already saved is answered “already applied”, and the same export of the same version is charged once.

Images. Every image the tools render, the pages and blocks of a preview and the render at the end of an edit, is a PNG sized for a model’s context: about 512 × 512 pixels at the block’s own aspect ratio, no side longer than 1,568 pixels, and at most 10 images in one preview. Hand them to the model as images, not as text.

This fails silently in two places:

• The Vercel AI SDK sends a result as JSON text unless your tool maps it with toModelOutput. The model then reads a render as a long base64 string instead of seeing it.
• A chat-completions API (OpenAI’s Chat Completions, OpenRouter’s, and IMG.LY’s AI gateway) defines a tool result as text. Move each result’s images into a user message right after it. pi, the agent library, does this itself.

To check, ask the model to describe a render: if it quotes base64 or says it cannot see an image, the mapping is missing. Each quickstart kit shows the mapping for its framework, for example the Vercel AI SDK quickstart. For all six frameworks, start from the Quickstart.

Adding Your Own Tools#

Your own tools take the same shape as CoDesign’s. Pass them to prepareCoDesign as tools, and they come last in codesign.tools, unchanged. This one gives the model the tenant’s brand:

const brandTool: CoDesignTool = {
  name: 'brand_guidelines',
  // The model decides from this text when to call the tool.
  description:
    "The tenant's brand: typefaces, colors, and tone. Call it before the first edit when the brief mentions the brand.",
  inputSchema: {
    type: 'object',
    properties: {
      topic: {
        type: 'string',
        description: 'What the design is about, such as "spring sale".'
      }
    },
    required: ['topic'],
    additionalProperties: false
  },
  call: async (args) => {
    const { topic } = args as { topic?: unknown };
    if (typeof topic !== 'string') {
      // A mistake the model can fix: tell it in words, as an error.
      return {
        content: [{ type: 'text', text: 'Pass `topic` as a string.' }],
        isError: true
      };
    }
    const brand = await brandFor(tenantId, topic);
    return { content: [{ type: 'text', text: JSON.stringify(brand) }] };
  }
};

Pass it with the other options:

const codesign = prepareCoDesign({
  // Your id for this chat; all its designs open in one executor that serves no other.
  conversation: `${tenantId}.${chatId}`,
  // EXECUTOR_TOKEN: a long random string you make, set on the executors too.
  executors: remoteExecutors({ acquire, token: process.env.EXECUTOR_TOKEN }),
  // IMG.LY's Google Fonts catalog; list your own plugin modules here too.
  plugins: [googleFonts()],
  designs,
  // Leaves out generated images, video, and audio only; your agent loop calls the model itself.
  gateway: false,
  // Yours come last in codesign.tools, unchanged. A reserved name throws tool_name_clash.
  tools: [brandTool]
});

import {
  googleFonts,
  prepareCoDesign,
  remoteExecutors,
  type AcquiredExecutor,
  type CoDesignTool,
  type DesignStore
} from '@imgly/codesign-sdk';

// Yours: this tenant's brand, from wherever you keep brand kits.
async function brandFor(
  _tenantId: string,
  _topic: string
): Promise<{ typefaces: string[]; colors: string[]; tone: string }> {
  throw new Error('yours to write');
}

// Yours: the executor of a conversation, as { url }; undefined when none can be had (Host the Executor).
async function acquire(_req: {
  conversation: string;
  signal?: AbortSignal;
}): Promise<AcquiredExecutor | undefined> {
  throw new Error('yours to write');
}

// Yours: read and write over your design table (Connect Your Storage).
declare const designs: DesignStore;
declare const tenantId: string;
declare const chatId: string;

const brandTool: CoDesignTool = {
  name: 'brand_guidelines',
  // The model decides from this text when to call the tool.
  description:
    "The tenant's brand: typefaces, colors, and tone. Call it before the first edit when the brief mentions the brand.",
  inputSchema: {
    type: 'object',
    properties: {
      topic: {
        type: 'string',
        description: 'What the design is about, such as "spring sale".'
      }
    },
    required: ['topic'],
    additionalProperties: false
  },
  call: async (args) => {
    const { topic } = args as { topic?: unknown };
    if (typeof topic !== 'string') {
      // A mistake the model can fix: tell it in words, as an error.
      return {
        content: [{ type: 'text', text: 'Pass `topic` as a string.' }],
        isError: true
      };
    }
    const brand = await brandFor(tenantId, topic);
    return { content: [{ type: 'text', text: JSON.stringify(brand) }] };
  }
};

const codesign = prepareCoDesign({
  // Your id for this chat; all its designs open in one executor that serves no other.
  conversation: `${tenantId}.${chatId}`,
  // EXECUTOR_TOKEN: a long random string you make, set on the executors too.
  executors: remoteExecutors({ acquire, token: process.env.EXECUTOR_TOKEN }),
  // IMG.LY's Google Fonts catalog; list your own plugin modules here too.
  plugins: [googleFonts()],
  designs,
  // Leaves out generated images, video, and audio only; your agent loop calls the model itself.
  gateway: false,
  // Yours come last in codesign.tools, unchanged. A reserved name throws tool_name_clash.
  tools: [brandTool]
});

export { codesign };

• The description decides when the model calls it. Say what it returns and when to call it.
• Keep the schema simple if you use several model providers. CoDesign’s own schemas use only type, description, properties, required, items, enum, minimum, maximum, and a boolean additionalProperties, a subset that Anthropic, Gemini, and OpenAI (without strict mode) all accept.
• Answer a mistake with isError: true and a sentence the model can act on. Throw only for what the model cannot fix.
• Some names are reserved. A tool named like one of CoDesign’s, or like a word the handbook depends on, makes prepareCoDesign throw tool_name_clash, naming the tool. The list is in the API Reference, and it can grow.

Next Steps#

• Quickstart — run the tools in an agent loop, with the images mapped for the model.
• Errors — handle each kind a rejected call carries.
• API Reference — look up prepareCoDesign’s options and the reserved tool names.
• Serve the Design Tools as an MCP Server — hand the same tools to other AI assistants.