Automation – IMG.LY Blog

Javascript Video Editing: Ultimate Guide for Developers and PMs

Jan — Fri, 20 Dec 2024 14:06:55 GMT

When developing or integrating a JavaScript-based video editor for the web, you must consider a number of factors to ensure the solution is both efficient and robust. This post is the definitive guide for anyone embarking on such a project. It explores the key technologies involved, their strengths and weaknesses, and how different use cases influence the choice of tech stack. We’ll examine diverse use cases, from lightweight, browser-based editors for quick edits to more advanced tools requiring complex processing and rendering. We’ll then discuss how these scenarios drive the selection of features and technology.

Additionally, we will show you the best open-source solutions that can accelerate development. This technical analysis will help you make an informed “build vs. buy” decision, ensuring you select the right approach for your project. Throughout this post, we will use the example of the IMG.LY’s JavaScript video editor (see here for a demo), showing you how these considerations shaped its architecture and feature set. You will learn practical insights into the decision-making process behind a successful web-based video editor.

Modern web technologies like WebGL, WebCodecs and WebAssembly have enabled browsers to efficiently perform resource-intensive tasks such as video editing that had hitherto been the sole domain of desktop applications. As a result Javascript based video editing tools such as veed.io have increased in popularity and users expects ever more sophisticated video editing capabilities inside modern video based web apps.

Video Editing Use Cases

Let’s first outline use cases and requirements of video editing applications on the web. These will inform our discussion of technical features of respective solutions below and grant a conceptual framework for evaluating potential benefits and drawbacks of each solution.

Simple Video Editing

Let’s start with the base case, simple video edits including trimming, cropping and resizing and simple effects such as adjusting brightness or saturation. This is usually sufficient feature set to support video editing for the following use cases:

Sales Outreach Videos: Sales teams often need to quickly edit customer-specific videos to personalize their outreach. This may involve trimming irrelevant portions, adding company logos, or adjusting brightness to ensure visual clarity.

Messaging Applications: These often require basic editing tools to allow users to crop, trim, or apply simple filters to shared videos, ensuring they’re concise and visually appealing.

Screencasting: Screencasting tools benefit from trimming and resizing capabilities to focus on key parts of recorded screens. Adding effects like brightness adjustment can make tutorials clearer and more professional.

CMS Systems: Content management systems may offer built-in video editing to help users optimize media assets for specific platform requirements, such as resizing for web embeds or adding subtle branding.

Screen Recording Applications: Screen recording applications often include simple editing options for cleaning up recorded content by trimming extraneous sections or cropping to highlight the most relevant parts.

Video Annotation

Next, users might want to add an additional layer of information to videos and overlay other assets, such as stickers, shapes, overlays, or text. Crucially these assets need to be time-based—that is, users need control over when the asset is shown so that particular video sequences can be referenced. This introduces the need for a timeline to arrange different video components relative to each other in time, as well as features like voice-over and audio track support.

E-commerce Reviews: Sellers and reviewers can annotate product demo videos with callouts, price tags, and feature highlights to make the content more engaging and informative.

Claims Management / Insurance: Insurance companies can use annotation to highlight key details in submitted video claims, such as timestamps of damage or explanatory text overlaying critical sections of footage.

Real Estate: Realtors might annotate property walkthroughs by adding labels, dimensions, or descriptive text overlays to highlight key features of the home or property.

Educational Applications: Instructors can use annotation tools to emphasize key moments in lectures or tutorials, such as overlaying text with formulas or concepts, or adding visual shapes to guide attention.

Productivity Tools: Users can annotate meeting recordings with timestamps, text notes, or overlay diagrams to summarize key decisions or action points.

Healthcare and Telemedicine: Medical professionals might annotate diagnostic videos or procedure recordings to explain findings or highlight areas of interest for training purposes.

Customer Support & Onboarding Tools: Companies can add annotations to video tutorials or troubleshooting guides to direct users through specific steps or highlight important information.

Video Composition

As we’re moving away from single media editing to creating video compositions of several types of media such as audio tracks, text, images, animations, and effects to create appealing visual designs, a well-designed timeline becomes even more important. Users need to manage many different video components in time, requiring user-friendly ways to browse and integrate external assets. This editor variant is most relevant for the following use cases:

Marketing Tech (Promotional Videos): Marketers can create visually stunning promotional videos by combining custom animations, background music, and text overlays for branding.

Social Media (Stories and Reels): Social media creators can quickly craft short, engaging videos that combine multiple assets like stickers, animations, and dynamic transitions tailored to platform-specific formats.

Event Highlight Reels: Event organizers can compile videos, photos, and music into cohesive highlight reels that encapsulate the essence of the occasion.

Template-based Video Creation

For many applications, users need starting points and examples for their designs. Starting from a blank canvas is rarely necessary for most use cases. Take a simple product video that includes promotional text, a brand logo, and animations. There is no need to reinvent the wheel for these types of videos. Instead, design applications should offer template libraries to accelerate user workflows.

Once we introduce workflows involving several stakeholders—such as designers setting up a certain brand framework and marketers working with and adapting these preconfigured designs—templates need to be able to constrain what edit operations adaptors can perform. These requirements are particularly important for these use cases:

Digital Asset Management: Organizations can manage and deploy branded video templates across teams, ensuring consistent design and messaging.

Social Media Publishing: Social media managers can utilize templates for quick turnaround on platform-specific video formats, such as Instagram Stories or LinkedIn posts.

Marketing Tech: Marketing teams can rely on pre-designed templates to churn out campaign videos at scale while maintaining brand consistency.

Training Videos: HR or L&D departments can adapt existing templates to quickly produce training materials customized for different teams or scenarios.

Creative Automation for Video

Finally, you can leverage data and automation to generate variations of your templates at scale, significantly boosting the productivity of use cases where users need to either test a large number of designs, publish to different channels with different requirements, or adapt designs to many slightly different instances, for example, menu designs for a franchise.

Marketing Tech: Marketers can generate hundreds of ad variations for A/B testing or localization by dynamically adjusting text, visuals, or calls to action.

Social Media Publishing: Social media teams can automate the creation of videos tailored to platform specifications, such as aspect ratios or resolution, while retaining core branding.

E-commerce: Retailers can produce personalized product showcase videos for different customer segments, featuring tailored offers, pricing, or recommendations.

Hospitality Industry: Hotels and restaurants can dynamically generate location-specific promotional videos showcasing seasonal offers, menus, or events.

Essential Video Editing Features

The above use cases are enabled by a set of features from simple transforms to complex compositions. Some video editors are focused more on manipulating individual video clips while others are more oriented towards video creation providing fully-fledged multi-media composition tool. This concise overview serves as a reference for evaluating video editing solutions:
Transforms

Cut, Trim, and Split: Basic editing tools for segmenting video clips.
Resize and Scale: Adjusts clip dimensions, ensuring proper fit or aspect ratio.
Crop and Rotate: Controls framing and orientation.
Zooming Capabilities: A UX feature enabling detailed editing and close-up views for precision.

Adjustments

Basic Adjustments: Controls brightness, contrast, and other visual enhancements.
Filters and Effects: Sets the visual appearance and atmosphere of videos.
Audio Tracks and Mixing: Supports multi-track audio adjustments and volume balancing.
Text and Overlay Options: Allows visual enhancements with text, captions, and emojis, providing opacity and layering options.

Composition

Multi-media Composition: Overlaying images, stickers, text as well as audio tracks are essential for creating videos through composition.
Multi-Track Editing: Elements and layers need to be position relative to each other in time allowing the creation of complex composition.
Canvas-Based Editing: Provides a flexible workspace for positioning and layering media elements.
Timeline Management: Split, join and arrange clips on a timeline. Manages the positioning of clips and timing for seamless transitions.
Animations: Make static elements dynamic and control their behavior in time. Can be use to create videos from scratch and enhance the storytelling workflow of existing ones.

The Technology Landscape

Some time in the early 2000s the web started to be viewed as a platform to run applications not just a distributed database to deliver static html documents. While Javascript development was not yet at a point to serve as foundation for the kind of application development we see today, Adobe Flash filled the role of development platform. Video editing in the early days of the web was mainly provided by Flash-based tools involving simple transform operations such as trimming and basic transitions. Java applets were another way to deliver processing intensive programs via the browser while executing them outside of the browser inside the JVM. These early attempts were either too clunky, since they could not run natively in the browser or lacked the performance to be useful for high quality editing.

When HTML5 came onto the scene in the early 2000s it became possible to handle multimedia content natively in the browser. Coupled with significant performance advances of Javascript and the introduction of WebRTC this opened the door for video editing on the web.

Some early cloud-based editors, such as WeVideo and Magisto, while more convenient and user-friendly than its predecessors still suffered from performance issues due to latency and lacked the depth of features found in desktop software.

In the past few years advances such as WebAssembly and Javascript libraries built upon it e.g. ffmpeg.js have revolutionised browser-based video editing. User can now perform complex editing tasks like multi-track timelines, video effects and real-time in-browser rendering. Modern web-based solution have even come to rival desktop apps, because they can take advantage of GPU acceleration, modern APIs like WebGL and UI frameworks such as React.

Let’s have a look at the state of the art web technologies that allow us to create performant video editing experiences inside the browsers. We are going to explore their relative strength, their complexities and conclude with when to best use each of these technologies:

Canvas API

The HTML5 Canvas API should be most familiar to most modern web developers, since it’s a familiar and popular way to render 2D graphics in the browser. It’s use for video editing is limited and restricted to simple overlays, trimming, composition and animation:

Complexity: Canvas is the least complex of the technologies introduced below and overlaps with the WebGL feature set to some extend, it’s API is well-known to most web developers.
Advantages: Canvas is lightweight, compatible across all browsers, and easy to implement, making it a great choice for simple use cases that require basic effects and no timeline.
Downsides: While well suited for simpler tasks Canvas lacks the GPU-acceleration of WebGL, hence it is not recommended for resource intensive operation or performance sensitive use cases.
When to Use: Choose Canvas for simple editing tasks, like trimming or basic overlays, or if you need a fallback technology that works well on legacy browsers and devices.

WebGL

WebGL opens up the power of GPU-accelerated graphics to the browser, making it the default choice for any use case requiring high performance rendering. As such modern Javascript video editing solution are well served by webGL when looking to render visual effects in real time, add transitions or facilitate multi-layer compositions by leveraging the GPU.

Complexity: WebGL comes with a steep learning curve especially if you are unfamiliar with 3D graphics processing, shaders and GPU programming.
Advantages: WebGL delivers high performance for processing complex video effects and can handle significant volume of data without latency. That makes it ideal for advanced, highly responsive video editing features.
Downsides: It can be challenging to engineer modular and maintainable WebGL code, debugging shaders and ensuring consistent performance across devices is an art in itself. WebGL might also be overpowered for simple editing use cases that do not justify the added complexity.
When to Use: Use WebGL if you’re creating a video editor with high-performance demands, particularly if real-time effects, transitions, or multi-layer compositing are needed.

WebCodecs API

Next up in our arsenal of formidable web technologies is a recent addition to modern browsers, WebCodecs, provide direct access to hardware accelerated video encoding and decoding.

The WebCodecs API is a recent addition to the browser, allowing direct access to hardware-accelerated video decoding and encoding. This API makes it easier to handle high-resolution video without delays, thanks to efficient decoding directly in the browser.

Complexity: WebCodecs is relatively easy to use for those familiar with media formats. However, you may need other technologies for rendering and complex editing.
Advantages: This API offers quick access to hardware acceleration, ideal for managing large video files. With it, you can efficiently handle high-definition video playback and streaming.
Downsides: Browser support for WebCodecs is still expanding, so compatibility might be a concern. Additionally, WebCodecs alone doesn’t provide a full editing suite; it needs to be combined with rendering and compositing technologies.
When to Use: WebCodecs is perfect if you need efficient decoding and encoding, especially for handling high-definition playback or live streaming.

WebAssembly (Wasm)

The next performance frontier browser pushed towards was enabling near-native performance by compiling languages like C++ or Rust into a format that efficiently runs in Javascript. This format is WebAssembly, which makes it possible to handle computationally expensive tasks like encoding, decoding or frame manipulation inside the browser.

Complexity: Working with WebAssembly is significantly more complex that the technologies discussed above. Developers need knoweldge of C++ or Rust as well as experience compiling to Wasm.
Advantages: WebAssembly enables browsers to handle complex video processing tasks, such as encoding and decoding, with near-native performance. Libraries such as FFmpeg build upon WebAssembly to make familiar video tools available to web developers.
Downsides: Development with Wasm is complex, and debugging can be a challenge. Additionally, Wasm modules may increase load times.
When to Use: Use WebAssembly when building native libraries for complex operation and expensive computations, such as encoding and decoding.

Our article on building a video editor with React and Wasm gives you a real-world starting point for your own apps.

MediaStream API

The MediaStream API is essential for gaining easy access to live video sources from within the browser, such as webcams. If your video editing app includes a feature for real-time recording or streaming the MediaStream API is an essential tool.

Complexity: MediaStream exposes a fairly simple API and is relatively easy to learn for experienced web developers.
Advantages: This API is well-suited for video feeds with low latency such as recording or streaming and does require addition encoding or decoding.
Downsides: Any use case requiring post-production video has to rely on additional technologies or frameworks, as the MediaStream API is limited to live feeds. The quality of the stream depends on the input source and can vary.
When to Use: Use MediaStream for applications with live recording or streaming functionality.

Web Audio API

The final web technology essential for video editing is the Web Audio API. Whether you want to add effects or mix multiple audio tracks, this API offers sophisticated tools for audio manipulation.

Complexity: For developers unfamiliar with audio processing the learning curve can be stepp, however the API is well designed and there is a host of educational resources.
Advantages: Web Audio API allows precise control over audio enabling your app to offer audio adjustments such as effects or reverb and multi-track editing.
Downsides: It can be challenging to accurately synchronize audio with video, especially if your app includes a real-time component.
When to Use: Opt for Web Audio API when audio editing is a core feature of your video editor.

Using These Technologies Together

Together, these technologies can be use to build a comprehensive browser based editor that is virtually indistinguishable from its desktop based counterparts. Different parts of the video editing workflow can be handled by each technology, while some choices have to made depending on the complexity and requirements of your projects, e.g. whether you’ll need resource intensive processing or an integration will real-time sources.

Video Decoding - WebCodecs API (for efficient video loading and playback).
Rendering - WebGL (for high-performance effects) and Canvas API (for simpler editing features).
Complex Processing - WebAssembly (for encoding/decoding and using native libraries like FFmpeg).
Audio Processing - Web Audio API (for mixing and adding effects to soundtracks).
Live Recording - MediaStream API (to capture live video from devices).

Open Source Javascript Video Editing Libraries

Fortunately in many cases someone has already done the heavy lifting and abstracted those browser APIs into simple to use libraries and frameworks. We’ll explore the most prominent ones distinguishing between projects that are for processing raw video and those that already provide an API for you to use. It is important to note that the libraries introduced below do not provide a video editing UI, unfortunately there are no open source video editors offering an end user interface that meet the standards for inclusion in a production grade application. None of the projects we evaluated met the demands we place on third-party libraries with respect to robustness, stability, maintenance and future development. However many can provide good starting points and references on how to build your own UI such as Reactive Video for React UIs.

FFmpeg.js

FFmpeg.js is the JavaScript port of the popular FFmpeg library compiled into WebAssembly. It enables developers to manipulate and process raw video files entirely in the browser. You can trim, convert formats, extract audio, and apply filters to videos without needing external software. However, FFmpeg just gives you the raw toolchain for building video editing features, you are still tasked with building the entire UX and editing workflows. We have an extensive guide on FFmpeg that you can consult to explore its capabilities and get started learning FFmpeg syntax.

Advantages:
- Most comprehensive library for web based video and audio processing.
- Fully client-side, no need for server-side processing.
- Highly flexible, allowing fine-grained control over video workflows.
Downsides:
- Fairly complex for beginners with a steep learning curve, requires familiarity with FFmpeg’s command-line syntax.
- FFmpeg is licensed under LGPL which makes it ill-suited for most commercial projects.
**When to Use:**FFmpeg.js is best suited for projects with custom requirements and UI making it necessary to exert fine-grained control over video processing tasks, especially for use cases like format conversion, advanced editing pipelines, or serverless video workflows.

The following two libraries allow you to programatically create and edit videos, we focus on the most stable, popular libraries here, while there might be promising up and comers we want to ensure a degree of dependability of the third-party code to be used in production systems.

Remotion

Remotion is a React-based framework that enables developers to edit or create videos programmatically in a declarative fashion using React components. React’s declarative syntax makes it comparatively simple to manage video content and Remotion provides a convenient UI for previewing and editing videos. While its studio UI cannot be embedded directly, Remotion can serve as the graphics processing engine for custom UIs. Although it is a commercial project, it offers a generous free tier for individuals and small businesses.

Advantages:
- Leverages React’s declarative model for seamless video creation.
- Provides robust video preview capabilities for iterative development.
- Flexible integration with custom-built UIs.
- Generous free tier for non-commercial and small-scale projects.
Downsides:
- The built-in studio UI cannot be embedded, requiring developers to create their own UI for embedding.
- React-specific, limiting use in non-React environments.
**When to Use:**Use Remotion when building React-based video applications that require programmatic video generation or dynamic video content. It’s ideal for teams already familiar with React and for applications where embedding is not a core requirement.

Etro.js

Similarly to Remotion, Etro.js is a JavaScript framework for programmatic video editing, it is framework agnostic and provides an API for simple composition of layers, effects and exporting videos.

Advantages:
- Includes support for advanced effects using GLSL shaders.
- Designed for in-browser video workflows.
Downsides:
- Minimal community support compared to larger projects.
- Requires additional effort to create a full UI.
When to Use: Suitable for applications where programmatic control and customization are key.

Extensibility

An important consideration when deciding to buy a prebuilt video editing solution is the degree of customization and extensibility required by your use case versus how critical speed to market is. While most SDKs provide enough flexibility by exposing cor API to build a solution that best suits your needs, this level of technical control comes at the expense of higher development and maintenance efforts. White-label solutions on the other end of the spectrum offer limited customization often relying on iFrames for embedding while providing maximum speed to market. If you are still in the phase where you need to make the business case for video editing features these might be key to allowing to quickly iterate. To learn more about weighing the pros and cons of SDK vs. White-label explore our extensive guide on the topic.

IMG.LY’s CE.SDK achieves extensibility by providing an interface for plugins that allow developers to add custom features and UI elements.

Cross Platform Video Editing

While this post is dedicated specifically to Javascript video editing an important consideration for choosing a video editing tech stack is whether there exists an intermediary output format that preserves editing operations so that videos can be edited on other devices and platforms. This is important for collaborative video editing use cases as well as use cases where videos are captured and basic edits performed on mobile and the finishing touches performed on the desktop.
These consideration are particular pertinent for the Real Estate, Telemedicine and Productivity use cases described above, since they involve either a multi-device or collaborative component.

At most common libraries or SDKs can achieve that by offering support for some form of serialization, that is exporting the edit operations along with the unedited video to then reapply edits upon import.

IMG.LY’s CE.SDK, on the other hand, is the only video editing SDK that is truly cross-platform. That means it is built atop a single creative engine that is portable to any platform. Whether iOS, Android, Desktop, or the Web, every platform uses the same underlying tech and uses the same custom format for scenes which contain all assets and edits making error-prone serializations obsolete.

Furthermore:

iOS and Android make use of the same underlying API.
Cross platform feature parity is baked into the cake. Since core functionality is implemented at the engine level, features are guaranteed to be available on both platform, although the timeline might differ somewhat.
Designs are 100% interoperable and consistent between platforms. Exporting and importing design files across platforms works seamlessly and final renderings are guaranteed to be consistent.

Javascript Video Editing & Generative AI

Generative AI is impacting video editing and generation across the board. Video generation obvious application of generative AI, but AI is also changing the way we we compose, create an edit videos in more subtle ways.

Features such as automated transitions, color grading, voice enhancement, voice over generation, background removal or inpainting are

Currently, these capabilities are offered by multiple specialized providers, such as ElevenLabs and OpenAI’s Sora. Firstly, that means modern Javascript video editors need to be open enough to serve as integrators for various, programmatically combining voice tracks, video clips, text, audio and other media before presenting users with an editing UI. Secondly, traditional video editing needs to enable the human in the loop. AI is still imperfect and manually refining unchangeable artifacts through prompts is inefficient and tedious. Instead the output by AI must remain editable and any editor solution smoothly interface with those artifacts to allow users to add the necessary polish, swap out media and regenernate individual parts of the scene.

To explore what this might look like in practice have a look at our CEO Eray Basar’s latest project building an AI video generator on top of CE.SDK.

Conclusion

When building or integrating a JavaScript video editor, developers and PMs have to navigate a complex technology landscape and make myriad decisions along the way. This guide has covered everything from essential use cases and feature sets to the cutting-edge technologies that underly browser-based video editing. Whether your project requires basic trimming tools or advanced multimedia composition, understanding the benefits and downsides of the various solutions like Canvas, WebGL, WebCodecs, WebAssembly, and more is crucial for choosing the right stack.

We’ve also explored how open-source tools like FFmpeg.js, Remotion, and Etro.js can accelerate development, while SDKs like IMG.LY’s CE.SDK offer unmatched extensibility and cross-platform compatibility. For those navigating the impact of generative AI on video editing, the need for seamless integration and user-centric design remains important.

If you’re looking for a robust, modern video editing solution for your web application, explore our showcase and start a free trial.

IMG.LY Research: AI-based Generative Design Editing

Mirko — Tue, 16 Jul 2024 10:44:05 GMT

Generative AI is transforming the tech landscape, finding applications in virtually every field. At IMG.LY, we’re exploring how these advancements can revolutionize creative workflows. This article presents a research project, where we integrate Large Language Models (LLMs) with our flagship product, CreativeEditor SDK (CE.SDK), to enable natural language-driven design edits.

Our flagship product, CreativeEditor SDK (CE.SDK) allows for advanced creative workflows for countless use cases in industries ranging from print to marketing tech. Most use cases can be realized with the out-of-the-box feature set, but it also exposes a best-in-class API, called Engine API, to build complex custom workflows with designs and videos.

In this article, we will showcase how to combine our CE.SDK Engine API with LLMs to edit designs with natural language.

Introduction to LLMs

Generative AI is often associated with chatbots, but its capabilities stretch further. It is a versatile text processor that can transform any textual input into various structured outputs. This adaptability is due to its training on diverse textual patterns, allowing it to support a wide range of text-to-text applications beyond just generating conversational prose.

Carefully crafting an input text (prompt) in a way that instructs the LLM to output a specific structured output format allows us to use LLMs to solve almost any arbitrary text-based task.

Crafting prompts is an art in itself. When ensuring that we receive the required output we need to adhere to the following steps:

Consider what type of data our model was trained on to ensure the correct formatting of our input text. The most basic example is using English as our “main prompt language” since most LLMs are mainly trained on English text samples.
All necessary problem-specific information to solve the task needs to be included in the prompt. While LLMs often possess an inherent understanding of the world inferred from the vast amount of text they are trained on, they may not know much about our specific problem. Furthermore, LLMs have the notorious tendency to hallucinate, that is fill in missing context with incoherent or incorrect information. To ensure the best performance, the input to the LLM must provide as much context as possible.
Finally, we need to instruct the LLM well enough to output text-based data in a format we can then parse and process.

Human vs. AI Workflows for Executing Design Tasks

We started this project with the vision to use generative AI to magically handle requests like these (in increasing order of complexity):

Make the logo bigger
Translate this design into German
Adopt this template to our brand colors and brand assets
Transform this Instagram story portrait design into a landscape YouTube thumbnail

When trying to delegate a task to an AI, it’s best to start by thinking about how these tasks are currently solved by humans.

Let’s walk through how a human would complete a task such as
Make the logo bigger:

Humans would visually scan the design and automatically segment it by its elements such as objects, backgrounds, or text.
Humans would then read and comprehend the task “Make the logo bigger”
Finally, users would use their existing knowledge of how to move and interact with design software to fulfill the task by manipulating the individual elements in the design.

Based on these considerations we can extract the implicit knowledge necessary to fulfill a task and make it explicit for the benefit of our LLM.

Design Representation:

To enable the LLM to understand and manipulate a design, it is essential to provide a representation of the design. This can be either textual or a mix of textual and visual (if the LLM has vision capabilities) data. While supplying the current design as a raster image to the LLM is trivial, serializing a CE.SDK design into a textual format requires a custom serialization process. The textual representation is important since it allows the LLM to identify, address, and comprehend the different components of the design effectively.

Refer to Appendix: Use-Case dependent serialization of CE.SDK Designs for a more in-depth explanation of how to accomplish this.

Editing Protocol:

LLMs do not interact with design software using traditional human interfaces like a mouse, keyboard, or visual feedback. Therefore, we need a specific protocol for the LLM to propose changes to the design. We have developed a method where we pass a textual representation of the design to the LLM as part of the prompt so that the LLM can indicate changes to the design by returning a modification of this representation.

Practically, this means that if we pass in an element such as <Image id="1337" x=”100” y="100" .../>, the LLM can change those x and y attributes by simply returning <Image id="1337" x="0" y="0" .. /> inside its output text. Since we can identify the design element that was changed using the ID attribute, we can then calculate the programmatic changes that need to be applied to the design, like in this case engine.block.setPositionX(1337, 0) and engine.block.setPositionY(1337, 0).

Refer to Appendix: Parsing and transforming LLM response for a deeper look into this topic.

Based on what we have learned, we assemble a workflow, with the LLM as the center point that allows the LLM to execute design-related tasks on any of our CE.SDK designs. This workflow can be divided into two sub-tasks: Composing an input text (prompt) with all necessary context and parsing and applying the output of the LLM to the CE.SDK design.

Composing the Input Text

As seen in the graphic above we first compose the input text based on different components to provide the model with all necessary context to fulfill the user’s editing request: This includes a general text to “instruct” the model, the actual request that the user entered, an exemplary design representation to explain our output format to the model and a representation of the currently edited design.

The general, static text to “instruct” the model is composed of the following three parts.

What we are trying to achieve in general:
"You are an AI with expertise in design, specifically focused on XML representations of designs".
Output format instructions:
Your responses should only contain one XML document. Ensure that you do not introduce new attributes to any XML elements. You can change image elements by setting the alt attribute, which then will be used to search Unsplash for a fitting image. A sample alt text is “A mechanic changing tires with a pair of beautiful work gloves on”.
Additionally, pay close attention to the layout: verify that no elements in the XML document extend beyond the page boundaries. This constraint is critical for maintaining consistency and accuracy in XML formatting. Always double-check your XML output for these requirements.
The actual user request:
e.g., "Make the logo bigger"

By including a textual representation of a comprehensive example design we can show the model which layer types are available as well as which properties of those layers can be manipulated.

The model now has all the necessary context and building blocks to respond to user requests in a format that we can process downstream.

Applying the Output Text

In the first step, we scan the output text for an XML-like document and if we find one, attempt to parse it.

This will yield a structured data object we can compare to the one we passed in and calculate which elements have been modified, added, or removed.

The resulting change set can then be translated into specific calls to the CE.SDK Engine API to change the current design.

Issues faced

Latency: One issue with state-of-the-art models is their big latency: Each LLM response contains approximately 1000 tokens. That means each request takes 7-45 seconds (depending on the model) to complete. This long delay may be unacceptable for some user experiences. However, we see this issue as transitory and expect upcoming models to have much smaller latency while maintaining their capabilities.

Pricing: Each request/response with GPT-4 turbo as a backing model costs around 5 cents and restricts some use cases. We also expect the pricing to drop significantly. The new GPT-4o model for example reduces the price by half.

Hallucinations: LLMs do not always follow the instructions properly and, e.g., produce output that is not parsable. Hallucinations directly correlate with the capabilities of the model and this issue is not apparent at current state-of-the-art LLMs like e.g GPT-4/GPT-4o.

Conclusion

We present a novel and adaptable approach to use Generative AI and LLMs specifically to interact with IMG.LY’s CreativeEditor SDK. We showcase how this technology can be used to execute common design requests on arbitrary CE.SDK Designs. The proposition that LLMs can understand textual representations of visual elements was by no means obvious. This research project has revealed that it is very well within the scope of LLMs to translate instructions from a visual semantic context to its textual representation and back. This invites more inquiries into LLMs as assistants for tasks with a heavy visual component such as design.

While further research is needed to make this technology available in production environments, we are confident that Generative AI-based editing will play a big role in the future of Graphics and Video editing.

Appendix: Use-case Dependent Serialization of CE.SDK Designs

LLMs work based on “tokens” which are equal to words. However, a design, like e.g a poster design or a social media graphic, is highly visual. That means that we need a way to convert a design into text, a way to serialize it. Our CE.SDK engine can serialize an existing scene using our engine.block.saveToString method. However, this serialization contains a huge pile of information that is not necessary to do edits inside the file. LLMs are priced by token and their speed is also relative to the number of tokens the input and output have. Thus, the number of tokens should be reduced.

We looked at several ways to convert the current state of the design into a textual representation. Since GenAI is trained on a lot of (X)HTML which has an XML-like format, we decided to serialize any designs into a tag-based XML-like format.

The IMG.LY editor internally refers to design elements like images or texts as “blocks”. These blocks are uniquely identifiable and addressable using a numeric ID. We use this ID to be able to identify a serialized design block in the input and output of the LLM. Example: <Image id="12582927" x="0" y="0" width="800" height="399" />

For each of the CE.SDK block types like e.g “Text” or “Graphics” (representing images or vector shapes), we will the CE.SDK Engine API to query very specific data from the block. That means that for example we only have a text attribute for Text blocks.

This rather specific mapping of only certain properties from the CE.SDK design into the text serialization allows us to optimize the design serialization for different use cases. A use-case where we e.g. want to automatically name each layer does maybe not require fine-grained information about e.g the font size.

Appendix: Parsing and transforming LLM response

The LLM answers with arbitrary tokens. It’s not possible to restrict the response to a certain syntax. By settling on a well-defined and widely used format we instruct the model to also reply with an XML-like document, similar to the one we passed in as “current state”.

After receiving the LLM’s Response we first make sure that only a single XML document is present inside the response. We then compare the retrieved XML document with the state of the Design that we passed into the LLM and generate a change set. This change set contains entries like “Color of block with ID=123 has changed”. These change set entries are then converted into programmatic commands, like e.g engine.block.setColor(123) and executed on the current design.

One challenges when working with an LLM is the inability to restrict the output space. Thus, we are never guaranteed that the LLM did not add e.g new XML node names or that it even replies with a proper, valid XML-like document. The only lever to influence the probability of a proper XML-like document is to use strong prompting and LLM that are good at following those instructions.

In our tests, state-of-the-art models like GPT-4 can follow those instructions without any further tooling.

Further Research Topics

It’s also worth exploring fine-tuning an LLM specifically for this task which could improve the performance of the LLM for the specific tasks.

It would also be possible to use more advanced libraries like Guidance, which allows to define a grammar for the LLM response thus making sure that the output of the LLM is always parseable.

Another way to improve the performance would be to methodically test different prompt templates and find a way to measure and compare the output quality.

Thank you for reading!

3,000+ creative professionals gain exclusive access and hear of our releases first—subscribe to our newsletter and never miss out.

How Printers Can Leverage Creative Automation

Natalia — Tue, 04 Apr 2023 09:58:54 GMT

The emergence of print as a service and print-on-demand has changed the traditional way we think about printing, creating many possibilities for customized products, such as personalized photo books and bespoke t-shirts. However, to meet the ever-evolving needs of customers, printers must constantly innovate and streamline their operations. This is where creative automation wins.

According to Keypoint Intelligence, only a mere 7% of Print Service Providers in North America had adopted a fully or mostly automated approach across their departments in 2020. This underscores the crucial role that creative automation can play in transforming the print industry. But what exactly does creative automation mean, and how can it revolutionize print production? Let’s find out!

What is Creative Automation?

To understand the power of creative automation, it’s critical to draw a line between the production of creative assets and ad units.

Although many creative assets may eventually turn into ads, they first need to be integrated into a broader ad unit that incorporates other non-graphical elements like headlines, text, and landing pages. This ad unit is then assigned to a specific campaign.

By leveraging creative automation, each stage of the creative asset production process can be automated, enabling the entire ad production process across all placements to be streamlined.

To clarify, dynamic creative automates media buying, while creative automation utilizes advanced software to produce stunning designs quickly. This approach offers several benefits, including improved efficiency, reduced costs, and higher product quality.

Creative Automation for Print Service Providers and Print-on-Demand

The potential advantages of creative automation for Print Service Providers are significant, particularly in terms of streamlining the design process.

With tools like CE.SDK, printers can rapidly create bespoke designs for t-shirts, business cards, and other products, while also taking advantage of a wide range of templates, styles, and layouts that can be easily customized to suit any project. This not only saves time, but also ensures that each design is unique and of outstanding quality.

Print-on-Demand companies, on the other hand, can also leverage the power of creative automation to offer personalized products to customers in a rapid and efficient manner.

By utilizing creative automation tools, print-on-demand companies can develop high-quality designs for photo books, t-shirts, and mugs, without the need for manual intervention in the design process. This evolves into significant time savings, cost reductions, and increased profitability, allowing print-on-demand companies to stay ahead of the competition.

How to Apply Creative Automation in Print

So, how can printers apply creative automation in print? There are several key elements to consider, including:

Variable Data Printing is a printing technique that allows the personalization of each print piece, based on specific customer data. By automating this process, Print Service Providers can increase efficiency and accuracy, while reducing costs.
Constraints might seem like a limitation, but they’re actually a key factor in ensuring that designs are consistent during the creation process and meet industry standards. By automating the constraint process, printers can ensure that designs are consistent and meet expectations.
Templates are pre-designed layouts that can be used for specific products, such as business cards or brochures. Automation of the template process allows the quickly creating of high-quality designs, without having to start from scratch each time.
Styles are a set of design elements, such as colors, fonts, and images, that can be used to create a consistent look and feel across all designs. The style automation process brings the possibility to create a cohesive brand image, increasing brand recognition and customer loyalty.
Layouts refer to the arrangement of design elements on a page, such as images, text, and graphics. Their automation provides fast creation of stunning designs, without having to manually arrange each element.
Design Validation is like having an eagle eye that spots mistakes (such as incorrect dimensions) before they become costly printing errors. Automating the design validation process allows printers to quickly identify and correct errors, reducing the risk of printing errors and increasing customer satisfaction.

Creative Automation in Print with CE.SDK

CE.SDK is a powerful tool for creative automation in print, taking care of the entire design process, from layout to validation.

With CE.SDK, you can easily create brilliant designs for various products, without having to spend time on manual design processes. The platform also offers a range of templates, styles, and layouts that can be easily customized to meet the needs of each project.

For instance, let’s say you want to create a gorgeous photo book for a client. With CE.SDK, all you have to do is upload your images, select a template, and customize the design elements.

CE.SDK will take care of the rest, automatically arranging the images into a beautifully designed photo book that includes text, graphics, and other design elements. This not only saves time, but also ensures that each photo book is unique and of high quality.

Here are some easy steps to follow when creating a photo book:

Step 1: Organize your Photos

Choose the photos you want to include in your photo book and organize them in a way that tells a story. For instance, you can arrange them chronologically or by theme.

Step 2: Customize your Photo Book

Then, add a personal touch to your photo book by customizing the text, adding captions, or changing the font style and color.

Step 3

Once you’ve finished customizing your template, you can preview the final result, and send it out for printing.

Conclusion

In today’s ever-changing printing industry, printers need to be efficient, innovative, and creative to remain competitive. Creative automation provides an excellent solution, enabling printers to streamline their design process, reduce costs, and improve profitability while delivering exceptional, personalized products to their customers.

CE.SDK is a powerful tool for creative automation in print, offering a wide range of customizable templates, styles, and layouts that can be tailored to meet the unique requirements of each project. With CE.SDK, printers can create stunning designs for a range of products, from personalized t-shirts to exquisite photo books, all while saving time and money. Try CE.SDK for free and see the benefits of creative automation for yourself.

Variable Data Printing: What Is It & Why Do You Need It

Klaudia — Fri, 03 Mar 2023 09:45:33 GMT

Variable data printing (VDP) has been around since the 1980s, helping businesses automate manual processes, scale personalization efforts, and utilize the power of direct mail marketing. While the term “variable data printing” may not be familiar to everyone, we encounter its use regularly in our day-to-day lives through bills, event tickets, or membership cards.

VDP’s ability to leverage data and create personalized tangible items, such as postcards or brochures, provides a more intimate experience compared to digital communication. Although process automation is a significant advantage of VDP, there are many other ways to use it.

In this article, we’ll explore the most common applications of VDP and the best practices for working with it.

How It Works

To fully understand the advantages of variable data printing (VDP), let’s start with a clear definition. VDP, also referred to as variable data imaging, is a digital printing technique that enables the creation of numerous customized materials in a single print run. This process is powered by software that can generate data-driven designs or templates by pulling variables from databases or files.

The variables used in VDP can automatically generate unique and personalized materials based on data sources. Anything that can be customized, such as names, addresses, messages, or images, can be utilized as a variable. These elements can be changed on each printed piece without slowing down the printing process.

To illustrate how VDP works in practice, let’s consider a scenario where a company offers a loyalty program for customers to collect points when they make purchases. To encourage more spending, customers can use their accumulated points to receive discounts or freebies. Although points and bonuses can be accessed through an app or website, the company can create immediate gamification by sending a personalized card with every new purchase.

To personalize that card, we are looking at three separate variables: name, number of points, and type of discount that can be applied. To manually generate all variations of the same template would be counterproductive, but with variables connected to a database it’s an automated process. A name is connected to a number of points, which in turn are connected to discounts. The software does all the heavy lifting, while marketers and designers can focus on other tasks.

Common Applications of VDP

Although VDP was originally used to automate manual mail processes, its usefulness extends beyond that. Its ability to personalize marketing and advertising materials quickly and efficiently has made it widely appreciated and used in various industries.

While there are many ways to use VDP, we’ll focus on its most common applications in marketing, advertising, and customer relationship management. These areas benefit greatly from the ability to tailor messages to specific audiences and create unique, personalized experiences.

Marketing

From quickly generating business cards to creating custom QR codes, variable data printing is a big part of marketing efforts for most businesses. Apart from handling internal processes, VPD also has a hand in scaling personalized marketing campaigns. The growing preference for personalized messaging and products goes alongside increased usage of VDP.

Production of personalized labels, packaging, or brochures is one of the best ways to leverage VDP in marketing. It can be used as a step in a broader personalization campaign, like generating custom QR codes to bring users to personalized landing pages. Or be a venture on its own, like “Share a Coke” by Coca-Cola and its widely successful personalization of labels.

Advertising

Variable data printing brought new life to direct mail, moving away from what once was perceived as junk and irrelevant advertising, and bringing about a new age of valuable and relatable mail. A recent study reported that 70% of consumers see direct mail as more personal than online interactions, with 54% claiming they want direct mail from brands that interest them. Moreover, direct mail has the highest response rate among all advertising channels being five to nine times higher than other forms, with a simple act of adding a name to a direct mail piece that can increase response rates by 135%.

Taking advantage of VPD in advertising usually includes customized leaflets or coupons with images and content based on age or demographics. One of the personalization examples is including custom maps with steps on how to reach an advertised location, making it much more relatable and easy to act on.

Customer Relationship Management

While VDP helps marketing and advertising drive acquisition, it’s also a great way to help with customer retention. Small and yet important gestures of appreciation like custom-made thank you and birthday cards can go a long way. As mentioned before, VDP can also be used to generate membership cards or personalized bonuses and discounts based on shopping preferences. Tapping into the power of variables, every printed material sent to a customer can be created in a way that will feel personal and appreciative, in turn helping sustain loyal customers.

Why Use VDP

Variable data printing is not reserved for a single industry or business type. It can be leveraged across multiple customer touchpoints, from the awareness stage via direct mail advertising, to a personalized label and thank you card shipped with a purchased product. VDP can also be used internally to show gratitude and appreciation to employees with custom-made cards and packaging. In fact, variable data printing can be used on any piece of printed content or design that can be personalized.

Benefits

The advantages of variable data printing go hand-in-hand with the advantages of personalization. Studies show that customers respond much better to marketing materials they can connect to, with 60% of consumers saying they will likely become repeat buyers after a personalized shopping experience with a retailer.

VDP can help create a valuable experience from start to finish and facilitate a meaningful connection between a brand and a customer. Some of the most prominent benefits are:

Providing personalized shopping experience in the form of custom labels, packaging & cards
Building strong customer relationships with personalized thank you & birthday cards, custom messaging, and discounts
Delivering meaningful direct mail advertising with higher conversion rates
Generating personalized marketing campaigns & materials that customers can connect with

Bringing variable data printing to your workflows, whether it concerts marketing, advertising, HR, or sales, is a great way to tap into powerful benefits of personalization.

Challenges

The main challenge of VDP is having software that can efficiently work with variables and generate print-ready designs, while also providing advanced design editing features.

Since the popularization of personalization, variables can be leveraged in many different types of marketing materials. However, the design of those materials is equally important, and so it’s an automated print validation process.

This challenge applies to companies that use variable data to generate designs, as well as Print Service Providers (PSP) working with VDP. For the PSP, another challenge is file format submission, especially if they don’t have a web-to-print editor their customers can use to submit their orders. Flawed print validation, incompatible format, low resolution, or insufficient quality could be costly mistakes to make. Hence, the best option for print service providers is to develop their own web-to-print editor with variable data printing that will take care of those problems and limit manual work to a minimum.

Example of Leveraging Variable Data Printing with CE.SDK

IMG.LY has developed CreativeEditor SDK to help integrate VDP into all designs in an easy and intuitive way. CE.SDK is fully customizable and can be integrated into a customer-facing web-to-print editor, providing powerful editing features, or an in-house design editor for all creative needs. Here’s an example of how you can leverage variables with our editor.

Creating Custom Thank-You Card

As mentioned before, thank you cards are a great way to show appreciation and build stronger connections with both customers and employees. Creation of a custom thank-you card is pretty straightforward, here are the necessary steps:

Creating a Smart Template

To create a personalized card, we begin by designing a template with variable fields. The front design remains consistent, while the back features unique personalized messages.

2. Assigning Placeholders

The next step is to choose the elements that will be personalized. For example, we may select a personalized greeting, such as “Hello {{name}}”, and address information. To insert a variable, we use double curly brackets {{}} and a placeholder name, such as {{firstname}} or {{address}}.

3. Connecting Template to a Data Source

The last step is connecting your template to a data source with list of variables. This can be done by using the Creative Engine variable API to programmatically set values for the variables and render the final personalized card.

4. Design Validation

In the example above, there shouldn’t be any issue with text exceeding beyond bleed line or other common print errors, however, design validation is an important feature of CE.SDK. It will automatically detect any issues that might cause a print error, cause trademark infringement or generate harmful content.

And that’s it! We have created a template for custom thank-you card in three easy steps. Here is what a card would look like.

Try creating a custom card with CE.SDK yourself on our showcase page!

Conclusion

Variable data printing can be a valuable tool to boost performance and take initiatives to the next level. It can help scale personalization efforts, acquire new customers, add value to the final product, and make employees feel appreciated. By using VDP, businesses can efficiently grow and reap numerous benefits, with only a few obstacles that can be easily overcome with the right solution.

To take advantage of the benefits of VDP and overcome its challenges, consider using CreativeEditor SDK and see how it can help you scale personalization and bring variable data printing to your company and customers.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Personalize or Perish: Why Companies Need to Go Personal

Klaudia — Tue, 21 Feb 2023 11:34:58 GMT

In the past, tailor-made marketing campaigns and personalized products were seen as a luxury that only a few companies could afford. However, as we enter 2023, personalization has become essential for all industries. Consumers now expect to see relevant and relatable content on a regular basis, which means that leveraging personalization on at least one channel has become a must-have.

Many marketers already use personalization techniques to retain customers and boost conversions. However, with new trends constantly emerging, there’s a risk of falling behind. If you’re still unconvinced that personalization is the future, we’ve put together an overview of why it should be a top priority for your business strategy.

What is Personalization?

First, let’s clarify what personalization actually means. The most straightforward way to explain personalization would be tailoring of marketing materials, campaigns, and products to individuals’ specific characteristics, preferences, and needs in order to provide a more relevant and engaging experience.

With technological development, companies are finding new ways to apply personalization across their businesses, ways beyond mere “Hello [name]” emails. For industries like e-commerce or printers, the personalization approach has been extended all the way to the production stage - allowing customers to incorporate their own designs and purchase a unique product made exclusively for them. Others, rely on algorithms to make sure that customers are always presented with the most relevant and engaging content. Regardless of the industry, everyone is looking to take advantage of personalization approaches.

Benefits of Personalization

Personalization is by default a user-centric approach, prioritizing relevance, usability, and impact of provided content and features. Thoroughly planned personalized marketing campaigns can make or break new product launches, boost subscription numbers, or convert lingering customers.
For industries like e-commerce and print, enabling customers to design or customize their own product means higher sales and lower return rates - which translates to not only a higher profit margin but also a more sustainable shopping experience.

Companies efficiently utilizing personalization efforts can expect:

Deeper connection between brand and customer
Increased brand loyalty and customer retention
Higher conversion rates

Treating customers with individual attention and care creates a sense of connection going beyond just purchasing needs. When customers are faced with meaningful content they are able to find what they need more easily, creating a pleasant user experience. Instapage reported that 74% of customers feel frustrated when website content is not personalized. Making them more likely to stick with brands that go the extra mile.

Moreover, presenting customers with relevant recommendations or tailor-made discounts significantly increases chances of conversion. According to Business Insider, 47% of customers turn to Amazon if the brand they are shopping with doesn’t provide personalized product suggestions. The ability to quickly find what is needed might just be the difference between making a sale and losing a customer.

Leveraging Personalization

Applications of personalization are limited only by imagination and creativity. The right personalization approach can sometimes equal to the success of an entire business. Hard to imagine Spotify being so popular without its flawless personalization algorithms alongside giants like Netflix or Amazon which heavily rely on personalization to retain subscriptions and push sales.

Fortunately, personalization is not reserved for large enterprises only, it can be used just as efficiently by any other company, regardless of its size. Thanks to its versatility, it can be utilized across different channels, platforms, and consumer touchpoints. Some of the most popular uses of personalization include:

Marketing

For brands and marketing agencies, personalization usually translates to displaying personalized content - be it product, service, or piece of information, alongside ensuring relevant outreach - with tailor-made emails and ads. The success of marketing personalization efforts usually depends on quality of gathered data and right algorithms. Unfortunately, finding a winning approach means little without having a way to scale it. Companies that wish to succeed with their marketing personalization strategy usually have to turn to automation of workflows and creative materials.

E-Commerce

Apart from applying personalization in their marketing efforts, e-commerce companies have the advantage of extending it all the way to their products. Allowing customers to add a custom engraving or artwork creates a more intimate shopping experience with irreplaceable product added value. Companies that cannot offer those services have also found a way to leverage personalization, usually by providing options to add a custom greeting card, thank you note, or even record a custom message.

Print Products

Print products might have the most exciting and yet still underutilized personalization growth possibility. While photo printers have been profiting from personalization for quite some time, there is still a large portion of the industry lagging behind. Personalization of any printed materials like photos, posters, cards, marketing materials, or apparel items is the future. And certainly, the future customers are impatiently looking forward to.

However, personalization leaves nobody behind. For those unable to provide such services, there is a whole world of personalized packaging and labels to take advantage of and create a meaningful customer experience.

Personalization Challenges

With all the benefits and possibilities of personalization, it seems like every company should have mastered it by now. So why it isn’t so? Scaling personalization can proven to be a challenging task. It requires a strategy on how to gather data sensibly, translate it into initiatives, and bring those initiatives to life. Some of the most common challenges of scaling personalization are as follows.

Data Collection and Preparation

In order to create relevant personalization of marketing content and materials first there must be data that can support this. With topics of privacy policies and data protection on the rise, targeted ads and campaigns might seem obsolete. The key to leveraging personalization is understanding the fine line between providing experience that feels relevant and welcomed, not invasive and overbearing. Another factor contributing to positive results is the transparency of collected data and how it’s being used. Harvard Business Review conducted a study that reported that when customers were openly informed their ads were based on their online activity revenue grew by 38%. With data becoming a currency of its own, honest communication seems to be the key to successful personalization campaigns.

Creative Materials Production

Personalization is loved by marketers because they know how much it helps with engagement and conversion rates. However, it can be dreaded by designers and creative teams. Production of personalized marketing materials can create a painful bottleneck and freeze resources for extended periods of time. The only way to scale production and reduce efforts is by implementing software that can automate a large portion of the work with templates and variable data imports. Designers can create adjustable templates and the marketing team can adapt them to their needs, making the process fast and accessible to all.

Custom Prints

The most significant challenges of personalized print products usually revolve around artwork supply and print quality. Different resolutions, file formats, and design quality might mean the back and forth of proofreading, rejecting, and proofreading again. Custom prints might also pose a risk of misprints and trademark infringements, as well as mismatched expectations - especially if there is no preview function of the ready product. A solution for those headaches would be a web-to-print editor, which can not only improve the design creation process, but also automatically validate designs and generate previews.

Scale Personalization with CE.SDK

CreativeEditor SDK was developed with the thought of bringing automation and scalability to creative materials. Because of its versatility and extensive features, it can be applied across all industries and utilized on all channels and platforms, and addresses some of the most burning questions limiting the current use of personalization.

Personalized Cross-Channel Content

How do you create personalized content that can be implemented for a large audience across multiple channels, all with limited resources? The short answer is automation. Creating countless variations of marketing materials has to be automated, otherwise, it cannot be scaled.

The most efficient way is to work with adjustable templates and data variation imports. Designers can create as many templates as needed, making sure they stay on brand, while marketers and social media creators can adapt them to their needs.

CreativeEditor SDK offers the possibility of creating smart templates with adjustable placeholders and constraints. This way, editing of any design is done in a controlled and predefined way. The editor can be used by internal teams, as well as external clients and customers, making the production of all marketing materials easily scalable.

How to Develop a Web2print Editor in a Timely Manner?

The most time-efficient way is to use Software Development Kit. SDKs contain powerful features with quick and easy integration and give the opportunity to stay on brand with customizable UI.

Choosing CE.SDK, whether for photo prints, apparel, or marketing materials, can streamline the web2print process and solve many of the custom print challenges. With smart templates, companies can jumpstart their customers’ creative process, ensuring right design dimensions and quality. While automatic design validation safeguards the printing process with notifications of artwork exceeding bleed margin, as well as checking for harmful content or trademark infringements.

Conclusion

Regardless if you use it as a key component or part of a bigger strategy, personalization is here to stay, and it will most likely determine the success of many brands and industries. Staying up to date with current personalization trends is becoming a must for marketers, with a growing need to think outside the box. The ability to scale personalized marketing materials, and provide bespoke user experience and relevant recommendations, are the future of reaching conversion goals and building lasting relationships with customers.

Companies will have to find ways to easily and efficiently scale personalization and bring it to their users. IMG.LY developed tools to help with those efforts. Learn more about CE.SDK capabilities or see how to use it for Creative Automation process or Web-to-Print.

If you want to learn more about integrating CE.SDK, check out our documentation or try it for free!

Thanks for reading. To stay in the loop, subscribe to our Newsletter.

Content Moderation for User-Generated Designs with CE.SDK and Sightengine

Antonello — Mon, 13 Jun 2022 14:07:54 GMT

Creating content and sharing it with others has become one of the core features offered by online services. As users publish their content to your website, you want to consider implementing a system to prevent inappropriate posts. Ideally, content will be programmatically moderated before it is uploaded on your platform. That is what you will learn in this article!

Follow this step-by-step tutorial and learn how to integrate content moderation with Sightengine in your design process built with CreativeEditor SDK in React. At the end of this tutorial, you will achieve the following result:

Why You Should Moderate User-Generated Content

User-generated content represents a large portion of online content, whether text or multimedia. Empowering users with tools and features to create high-quality content is excellent. However, it is a common concern that potentially inappropriate content on your website will be associated with your brand. Naturally, you will want to avoid this.

A recent example from the web-to-print industry serves as a cautionary tale: the German e-commerce company Spreadshirt neglected to moderate content on its t-shirt printing platform, leading to a wave of outrage on social media. Even though the company reacted swiftly and stated an apology, Spreadshirt could have avoided most of the damage if it had implemented a system that automatically flags potentially harmful content. Of course, a religious or cultural symbol is not inherently inappropriate, but the misuse is.

It is highly advisable to take precautions to prevent your company from facing similar adversity. A common problem, however, is the costly and time-consuming manual moderation, epecially when controlling multimedia files like images.

Luckily, you can use CE.SDK and Sightengine to programmatically integrate moderation into your content design process without manual reviews.

What is CE.SDK?

CreativeEditor SDK (CE.SDK) is a powerful, easy-to-use, and fully customizable design editor. With just a few lines of code, CE.SDK provides your application with template-based workflows and automation.

The role-specific editing UI focuses on keeping content adaptation dead simple: in Creator Mode, you have the liberty to create a design from scratch or customize existing templates. When creators are ready to share a design for others to use and customize in Adopter Mode (Default UI), creators may decide which elements can be changed. See CE.SDK in action or try it for free.

What is Sightengine?

Sightengine is a convenient tool for automatic content moderation. It detects and filters unwanted content in photos, videos, and live streams through fast, accurate, easy-to-integrate, privacy-compliant APIs.

Sightengine offers you APIs to get moderation results instantly and effectively. The API-centric approach allows you to scale to up to tens of millions of images per month. You can have a look at the Sightengine demo or try it for free.

Integrate Content Moderation into CE.SDK

Let us look at how to implement an image editor with content moderation for your users. For this, you will learn how to use CE.SDK and React to implement an image design component and use the CE.SDK APIs to programmatically moderate the content with Sightengine on a Node.js server. You can try out a live showcase of content moderation with CE.SDK.

With the following steps, you can easily achieve this result with CE.SDK, React, Sightengine, and Node.js.

1. Get Started with CE.SDK

Integrate CreativeEditor SDK with React by using the official documentation. As you will see in the final snippet, this takes only a few lines of code.

Now that you have a working CE.SDK instance, your users may unleash their creativity by designing templates and sharing them with others. The powerful APIs ensure control at every step of the editing process and allow programmatically performing operations, such as content moderation.

2. Build a Content Moderation API with Sightengine

Let’s move on to content moderation with Sightengine. Keep in mind that any backend technology will work, but here you will implement a content moderation API based on Sightengine in Node.js.

2.1 Getting Started with Sightengine

First, you need a Sightengine account. If you do not have one yet, you can create a free account. Fill out the form, click Sign Up, and receive a verification email in your inbox. Click on the verification link and activate your account.

Then, navigate to API Keys on your Sightengine Dashboard page and retrieve your API user and secret credentials. You will need them to interact with the Sightengine moderation APIs.

2.2 Prerequisites

Here is a list of all prerequisites for the backend application you are going to build:

You can add express and node-fetch to your project’s dependencies with the following command:

npm install --save express node-fetch

2.3 Setting up a Node.js Express Server

Follow this step-by-step guide to set up a Node.js Express development server. You will need this to perform an API call to Sightengine to moderate your content server-side. Any Sightengine API call involves your Sightengine user and secret pair of credentials, and you do not want to expose them.

You may also want to store the result of the moderation API in your database, as well as other data. For example, you might be interested in keeping track of how many times a particular user has violated the moderation rules to warn or ban them.

2.4 Implementing a Content Moderation API in your Node.js Server

You can now implement a content moderation API in your Node.js Express server as follows:

    const fetch = require("node-fetch")

    router.get('/content-moderation', function (req, res) {
      if (!req.query.url) {
            res.status(404).send("Please provide an image url")
            return
        }

        // defining the Sightengine API parameters
        const params = {
            url: req.query.url,
            models: "nudity,wad,offensive",
            api_user: "YOUR_SIGHTENGINE_USER",
            api_secret: "YOUR_SIGHTENGINE_SECRET",
        }

        // generating the Sightengine API URL
        const queryString = new URLSearchParams(params).toString();
        const requestOptions = {
            method: "GET",
        }
        const sightengineUrl = "https://api.sightengine.com/1.0/check.json?" + queryString

        // calling the Sightengine moderation API
        const externalRes = await fetch(SightengineUrl, requestOptions)
        // retrieving the moderation result from the Sightengine API
        const moderationResultText = await externalRes.text()

        res.append("Access-Control-Allow-Origin", ["*"])
        res.append("Access-Control-Allow-Methods", "GET")
        res.append("Access-Control-Allow-Headers", "Content-Type")

        // returning the Sightengine moderation result text
        res.status(200).send(text)
    })

Take care of replacing YOUR_SIGHTENGINE_USER and YOUR_SIGHTENGINE_SECRET with your Sightengine pair of credentials retrieved before.

Now, launch your Node.js server and reach the following URL below to use Sightengine to moderate the images specified in the URL query parameter:

    http://localhost:3000/content-moderation?url=<your_image_url>

3. Moderate Images Made with CE.SDK

We will now use CE.SDK and Sightengine to programmatically moderate user-generated images. Before we can dive in, check the following requirements.

3.1 Prerequisites

This is the list of all the prerequisites you need to build for the frontend demo application:

You can add the @cesdk/cesdk-js library to your project’s dependencies with the following command:

npm install --save @cesdk/cesdk-js

Also, the demo application will use other libraries, although they are not strictly required to achieve the goals of this tutorial and can be omitted here.

3.2 Getting Started

You can try out the demo application by cloning the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/content-moderation-with-cesdk-and-sightengine
cd content-moderation-with-cesdk-and-sightengine
npm i
npm start

Otherwise, keep following this tutorial and build the demo application for content moderation by yourself.

First, initialize a React project called content-moderation-with-cesdk-and-sightengine by employing the Create React App command below:

    npx content-moderation-with-cesdk-and-sightengine

Now, the content-moderation-with-cesdk-and-sightengine folder should have the following file structure:

content-moderation-with-cesdk-and-sightengine
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
   ├── App.css
   ├── App.js
   ├── App.test.js
   ├── index.css
   ├── index.js
   ├── logo.svg
   ├── reportWebVitals.js
   └── setupTests.js

Enter the content-moderation-with-cesdk-and-sightengine folder and start a local server with these two commands:

cd content-moderation-with-cesdk-and-sightengine
npm start

Visit http://localhost:3000/ in your browser, and you should now be seeing the default Create React App screen below:

3.3 Integrate the Moderation Check into your Design Process

Notice that the demo application you are about to implement is just a simplified version of the app you can find in this GitHub repository. This demo app integrates the Sightengine moderation functionality with the CE.SDK design capabilities.

Now, let’s move on to the main component where the integration takes place:

import CreativeEditorSDK from '@cesdk/cesdk-js';
import ValidationBox from '../ui/ValidationBox/ValidationBox';
import React, { useCallback, useEffect, useRef, useState } from 'react';
import { ReactComponent as RefreshIcon } from './refresh.svg';
import { moderateImages } from './moderationUtils';
import styles from './ImageModerator.module.css';
import classNames from 'classnames';

function selectBlock(cesdk, blockId) {
  // deselecting all blocks
  cesdk.engine.block
    .findAllSelected()
    .forEach((block) => cesdk.engine.block.setSelected(block, false));

  // selecting the block having the blockId passed as parameter
  cesdk.engine.block.setSelected(blockId, true);
}

const ImageModerator = () => {
  const cesdkContainerDiv = useRef(null);
  const cesdkRef = useRef(null);
  const [validationResults, setValidationResults] = useState([]);

  useEffect(() => {
    // initializing CE.SDK if it has not been initialized yet
    if (cesdkContainerDiv.current && !cesdkRef.current) {
      const config = {
        role: 'Adopter',
        theme: 'light',
        initialSceneURL: `${window.location.protocol + '//' + window.location.host}/cases/content-moderation/example.scene`,
        ui: {
          elements: {
            panels: {
              settings: true,
            },
            navigation: {
              action: {
                save: true,
              },
            },
          },
        },
        callbacks: {
          // calling this function when the
          // user presses the "Save" button
          onSave: () => runImageModerationCheck(),
        },
      };
      CreativeEditorSDK.init(cesdkContainerDiv.current, config).then(
        (instance) => {
          cesdkRef.current = instance;
          // running the moderation check on initialization
          runImageModerationCheck();
        }
      );
    }
    return () => {
      if (cesdkRef.current) {
        cesdkRef.current.dispose();
      }
    };
  }, [cesdkContainerDiv]);

  const runImageModerationCheck = useCallback(async () => {
    if (!cesdkRef.current) {
      return;
    }
    // retrieving the moderation results
    const validationResults = await moderateImages(cesdkRef.current);
    setValidationResults(validationResults);
  }, []);

  return (
    <div className={classNames(styles.wrapper, 'space-y-2')}>
      <div className={styles.header}>
        <div className={styles.headerDiv}>
          <div>
            <h3 className="h4" style={{ color: 'white' }}>
              Content Moderation
            </h3>
            <p className={styles.headerDivMessage}>
              Check images for compliance with your content guidelines before
              further processing and provide user feedback.
            </p>
          </div>
          <button
            onClick={() => runImageModerationCheck()}
            className={'button button--white space-x-2'}
          >
            <span>Validate Image Content</span>
            <RefreshIcon />
          </button>
        </div>
        <ValidationBox
          results={validationResults}
          onSelect={(blockId) => {
            selectBlock(cesdkRef.current, blockId);
          }}
        />
      </div>
      <div className={styles.cesdkWrapper}>
        <div ref={cesdkContainerDiv} className={styles.cesdk}></div>
      </div>
    </div>
  );
};

export default ImageModerator;

In the first useEffect() hook, CE.SDK is initialized, and a callback on the save action is registered. This will take care of calling the runImageModerationCheck() function when the user presses the CE.SDK Save button. For more information on how callbacks work in CE.SDK, see the official documentation on observing events.

runImageModerationCheck() retrieves the moderation results obtained from Sightengine and then stores it in the validationResults array variable. This is then passed to the ValidationBox component, which will show the results of the validation process to the end-user. None of this would be possible without the moderateImages() function from the moderationUtils.js file below:

export const moderateImages = async (cesdk) => {
  // retrieving all image blocks with a name
  const imageBlocksData = cesdk.engine.block
    .findByType('image')
    .map((blockId) => ({
      blockId,
      url: cesdk.engine.block.getString(blockId, 'image/imageFileURI'),
      blockType: cesdk.engine.block.getType(blockId),
      blockName: cesdk.engine.block.getName(blockId),
    }))
    .filter((block) => {
      return block.blockName !== '';
    });

  console.log('Moderation check launched...');

  // turning the moderation results on each image block into
  // a flat array
  const imagesWithValidity = await Promise.all(
    imageBlocksData.flatMap(async (imageBlockData) => {
      // retrieving the moderation results
      const imageModerationResults = await callSightengineModerationAPI(
        imageBlockData.url
      );

      console.log(imageBlockData.blockName, imageModerationResults);

      return imageModerationResults.flatMap((checkResult) => ({
        ...checkResult,
        ...imageBlockData,
      }));
    })
  );

  console.log('Moderation check completed');

  return imagesWithValidity.flat();
};

// a cache variable to avoid
// calling the Sightengine API when not necessary
const complianceCache = {};

const callSightengineModerationAPI = async (url) => {
  // if the url is already present in the cache, then
  // return the result previously retrieved
  if (complianceCache[url]) {
    return complianceCache[url];
  }

  // calling the Sightengine moderation API
  const response = await fetch(
    // your endpoint URL
    'https://europe-west3-img-ly.cloudfunctions.net/sightengineApiProxy?url=' +
      encodeURIComponent(url),
    {
      method: 'get',
      headers: {
        accept: 'application/json',
        'Accept-Language': 'en-US,en;q=0.8',
        'Content-Type': 'multipart/form-data;',
      },
    }
  );

  const results = await response.json();

  if (results.error) {
    console.error(results.error);
    return [];
  } else {
    const checkResults = [
      {
        name: 'Weapons',
        description: 'Handguns, rifles, machine guns, threatening knives...',
        state: percentageToState(results.weapon),
      },
      {
        name: 'Alcohol',
        description: 'Wine, beer, cocktails, champagne...',
        state: percentageToState(results.alcohol),
      },
      {
        name: 'Drugs',
        description: 'Cannabis, syringes, glass pipes, bongs, pills...',
        state: percentageToState(results.drugs),
      },
      {
        name: 'Nudity',
        description: 'Images that contain either raw nudity or partial nudity.',
        state: percentageToState(1 - results.nudity.safe),
      },
    ];

    // storing the Sightengine results in the cache
    complianceCache[url] = checkResults;
    return checkResults;
  }
};

const percentageToState = (percentage) => {
  // defining the custom moderation logic based on the
  // percentage returned by Sightengine
  if (percentage > 0.8) {
    return 'failed';
  } else if (percentage > 0.4) {
    return 'warning';
  } else {
    return 'success';
  }
};

Here, we use the CE.SDK reference variable to retrieve all images in the scene through the CE.SDK APIs. Then, these are converted into URLs and passed callSightengineModerationAPI() function, which takes care of calling the Sightengine moderation API, retrieving the results, using them to produce a new output, and returning it to the caller.

We used an IMG.LY endpoint here to evade putting together a backend application. In a real-world scenario, replace https://europe-west3-img-ly.cloudfunctions.net/sightengineApiProxy?url= with the URL of the /content-moderation endpoint of your backend application. Also, note that a cache system was set up to avoid calling the Sightengine API too many times and using up call credits.

This is what the demo application will look like after being launched and the moderation check has been executed:

As you can see, the Sightengine API correctly detects drugs and weapons. Next, remove the rightmost image containing a gun and click Save to re-run the moderation check.

Sightengine no longer detects weapons in the image, proving that the moderation process is working as expected.

Et voilà! You have learned to implement content moderation for user-generated images with CE.SDK and Sightengine.

Conclusion

A challenge of user-generated content is avoiding the risk of hosting and displaying offensive or outright illegal content. Inappropriate content affects your user community and brand reputation and may even cause legal action against you.

In this article, we used CE.SDK to initialize a fully-featured design editor in React. We employed its API to programmatically call a Node.js server using Sightengine to perform content moderation on images. This allows moderation of user-generated content automatically and prevents the upload of inappropriate content to your website.

We are only scratching the surface of what is possible with CE.SDK – you can implement any UI on top of its powerful image processing API and extend the core functionality with features such as placeholders and define validation rules to guide your users’ creation process.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.

Automation – IMG.LY Blog

Javascript Video Editing: Ultimate Guide for Developers and PMs

Video Editing Use Cases

Simple Video Editing

Video Annotation

Video Composition

Template-based Video Creation

Creative Automation for Video

Essential Video Editing Features

Adjustments

Composition

The Technology Landscape

Canvas API

WebGL

WebCodecs API

WebAssembly (Wasm)

MediaStream API

Web Audio API

Using These Technologies Together

Open Source Javascript Video Editing Libraries

FFmpeg.js

Remotion

Etro.js

Extensibility

Cross Platform Video Editing

Javascript Video Editing & Generative AI

Conclusion

IMG.LY Research: AI-based Generative Design Editing

Introduction to LLMs

Human vs. AI Workflows for Executing Design Tasks

Using Generative AI to Execute Design-Related Tasks

Composing the Input Text

Applying the Output Text

Issues faced

Conclusion

Appendix: Use-case Dependent Serialization of CE.SDK Designs

Appendix: Parsing and transforming LLM response

Further Research Topics

How Printers Can Leverage Creative Automation

What is Creative Automation?

Creative Automation for Print Service Providers and Print-on-Demand

How to Apply Creative Automation in Print

Creative Automation in Print with CE.SDK

Step 1: Organize your Photos

Step 2: Customize your Photo Book

Step 3

Conclusion

Variable Data Printing: What Is It & Why Do You Need It

How It Works

Common Applications of VDP

Marketing

Advertising

Customer Relationship Management

Why Use VDP

Benefits

Challenges

Example of Leveraging Variable Data Printing with CE.SDK

Creating Custom Thank-You Card

Conclusion

Personalize or Perish: Why Companies Need to Go Personal

What is Personalization?

Benefits of Personalization

Leveraging Personalization

Marketing

E-Commerce

Print Products

Personalization Challenges

Data Collection and Preparation

Creative Materials Production

Custom Prints

Scale Personalization with CE.SDK

Personalized Cross-Channel Content

How to Develop a Web2print Editor in a Timely Manner?

Conclusion

Content Moderation for User-Generated Designs with CE.SDK and Sightengine

Why You Should Moderate User-Generated Content

What is CE.SDK?

What is Sightengine?

Integrate Content Moderation into CE.SDK

1. Get Started with CE.SDK