Image Editing – IMG.LY Blog

IMG.LY Partners with Getty Images to Revolutionize Image Integration

Klaudia — Wed, 03 May 2023 13:07:31 GMT

We are thrilled to announce our latest partnership with Getty Images, the world’s leading provider of images, clips, and illustrations for visual communication. This collaboration has brought forth the Getty Images Library SDK - a bridge that seamlessly integrates image searching, licensing, and editing into your application. This revolutionary toolkit grants access to millions of stock images and enables your teams and users to browse, license, and edit images without ever leaving your application or website.

By integrating Getty Images Library SDK into your application, you can save precious resources, streamline your workflows, and ensure legal compliance with a broad range of licensed stock images at your fingertips. Imagine not having to switch between different platforms to search, license, and edit images – all while staying in your application.

Empower your Developers and Users

At IMG.LY, we are passionate about delivering powerful editing tools with an intuitive interface, providing a creative outlet for users of all skill levels. Teaming up with Getty Images, we strive to provide a toolkit that empowers content creators to communicate their message in the most efficient and effective way possible, while upholding Getty Images’ commitment to quality and diversity in visual images.

Designed by developers, for developers, the integration is quick and easy to set up. Once integrated, users can effortlessly browse through images, license the visuals they want, and make quick edits such as cropping or rotating images, all within the same application.

We are excited to share this news and look forward to working with Getty Images to bring tools and empower creative expression!

Get Started

Learn more about the Getty Images Library SDK, how to integrate it, or contact Getty Images to obtain a license key.

How to Draw on Images in React Native

Natalia — Thu, 20 Oct 2022 13:09:38 GMT

It has become a commonplace requirement for many applications to manipulate images in one way or another before users are ready to share them. Therefore, it is crucial to foresee end-user needs while working on an app involving any interaction with graphical content. Image annotation can be necessary in various cases, such as simple inspection apps, bug reporting software, or machine learning applications relying on manual user input.

In this article, we are shedding light on how to draw on images in React Native with the help of a simple HTML5

Enable the characteristics of the
Set the thickness and color of the brush instrument;
Define at what moment your brush needs to be activated and when the drawing process will finish;
Start drawing!

Let’s first clear up the purpose of

Canvas Integration for Images

<canvas> is an HTML component commonly used for creating and manipulating graphics, animation, and other visualization features. This element provides JavaScript APIs enabling image overlays and handling user input to make drawing possible. Moreover, while we have several methods for image manipulation with

<! -- HTML example of using canvas: -->
<canvas id="myCanvas" width="200" height="100"></canvas>

You might wonder why you cannot easily substitute <canvas> with some lines of HTML and CSS in your code. The answer lies on the surface: <canvas> creates a single flattened graphic rather than multiple components lying on top of each other (i.e., typically the output of HTML and CSS).

Now, let’s see how to implement this powerful element in your React Native application:

Drawing with Canvas

import React, { useEffect } from 'react';

// CanvasContext here is used for drawing onto the canvas
import { useCanvas } from './CanvasContext';

export function Canvas() {
  const { canvasRef, prepareCanvas, startDrawing, finishDrawing, draw } =
    useCanvas();

  useEffect(() => {
    prepareCanvas();
  }, []);

  return (
    <canvas
      onMouseDown={startDrawing}
      onMouseUp={finishDrawing}
      onMouseMove={draw}
      ref={canvasRef}
    />
  );
}

We start by enabling the characteristics of our <canvas> container, which are meant to render the CanvasContext component presented below.

The size of our future canvas is determined with prepareCanvas, which includes both the height (canvas.height) and width (canvas.width) of the HTML element. For the drawing process itself, it is necessary to assign not only the thickness and color of the brush instrument (context) but also take into account its activation (startDrawing with onMouseDown) and how the user can finish a stroke (finishDrawing with onMouseUp).

import React, { useContext, useRef, useState } from 'react';

// enabling drawing on the blank canvas
const CanvasContext = React.createContext();

export const CanvasProvider = ({ children }) => {
  const [isDrawing, setIsDrawing] = useState(false);
  const canvasRef = useRef(null);
  const contextRef = useRef(null);

  // defining width & height of the canvas
  const prepareCanvas = () => {
    const canvas = canvasRef.current;
    canvas.width = window.innerWidth * 2;
    canvas.height = window.innerHeight * 2;
    canvas.style.width = `${window.innerWidth}px`;
    canvas.style.height = `${window.innerHeight}px`;

    // defining the thickness and colour of our brush
    const context = canvas.getContext('2d');
    context.scale(2, 2);
    context.lineCap = 'round';
    context.strokeStyle = 'black';
    context.lineWidth = 5;
    contextRef.current = context;
  };

  const startDrawing = ({ nativeEvent }) => {
    const { offsetX, offsetY } = nativeEvent;
    contextRef.current.beginPath();
    contextRef.current.moveTo(offsetX, offsetY);
    setIsDrawing(true);
  };

  const finishDrawing = () => {
    contextRef.current.closePath();
    setIsDrawing(false);
  };

  const draw = ({ nativeEvent }) => {
    if (!isDrawing) {
      return;
    }
    const { offsetX, offsetY } = nativeEvent;
    contextRef.current.lineTo(offsetX, offsetY);
    contextRef.current.stroke();
  };

  // Once the canvas is cleared it return to the default colour
  const clearCanvas = () => {
    const canvas = canvasRef.current;
    const context = canvas.getContext('2d');
    context.fillStyle = 'white';
    context.fillRect(0, 0, canvas.width, canvas.height);
  };

  return (
    <CanvasContext.Provider
      value={{
        canvasRef,
        contextRef,
        prepareCanvas,
        startDrawing,
        finishDrawing,
        clearCanvas,
        draw,
      }}
    >
      {children}
    </CanvasContext.Provider>
  );
};

export const useCanvas = () => useContext(CanvasContext);

Drawbacks of the Canvas Method

Despite the canvas component offering powerful features, it is also quite demanding to use in its original form. For example, it may be problematic to adjust the properties of the brush to ensure that you can draw with smooth movements. Thus, implementing any solution for this issue would further increase the coding time and reduce the compatibility between different frameworks. Also, separately defining canvas and brush properties overcomplicates the code and requires advanced sources for successful integration into your React Native app. Although some temporary solutions aim to facilitate this process (such as react-native-image-draw or react-native-sketch-draw), none provides a permanent solution with clear documentation and guidance.

If you wonder about a magic door to avoid this coding nightmare, you can always choose a commercial, all-in-one solution like PhotoEditor SDK. Implementing this SDK only takes a few lines of code, saves time and expenses in building an app, and you can expect great support from the developers. This way, you can skirt the hassle of coding an editor and instead focus on creating a great product.

PhotoEditor SDK Integration for Drawing on Your Images

To get started integrating the PhotoEditor SDK in your React Native app, refer to this guide from the official documentation. You can then ensure comfortable image interaction for your users with the optimized brush tool, as in the example below.

Conclusion

This article aimed to examine image editing instruments suitable for React Native framework. The HTML <canvas> element could have big potential as it provides fundamental tools for creating graphics or animation. However, its application is time-consuming and requires comprehensive knowledge in the field. Therefore, if you want to guarantee a smooth and easy integration of image drawing tools for your app, consider using advanced software like PhotoEditor SDK.

Comparing the Top 5 Open Source JavaScript Image Manipulation Libraries

Mark — Tue, 27 Sep 2022 13:29:03 GMT

Image Manipulation Libraries (IML) are used to perform various manipulation functions on images: you can increase the brightness of an image, add saturation or filters, crop and resize, and more valuable features that will help you to do almost everything and turn your web browser into an Adobe Lightroom!
We have included libraries for every significant image manipulation task, from simple, low-level operations such as filters, brightness, and saturation to single-purpose libraries for cropping or image composition. To finish up, we wandered into the realm of testing images, since no image manipulation application is complete without a reliable way to test results.

Sharp

Sharp is a high-performance Node.js image processing library to resize different image formats such as JPEG, PNG, WebP, AVIF, SVG, and TIFF. The typical use case for this high-speed Node.js module is to convert large images in standard formats to smaller, web-friendly images.

Sharp is helpful only if you want to resize a giant file or a variety of them. If, on the other hand, you only want to resize a single small image, then you probably shouldn’t use it. Instead, ordinary HTML and vanilla JavaScript will be more beneficial. Sharp is taking full advantage of multiple CPU cores and L1/L2/L3 cache, allowing you to resize and compress your images much more quickly.

Use cases

I used it to resize a whole NFT Collection with a size of over 80Gb, which after compression came out at circa 10Gb. So again, if you would like to use it for resizing several large images, then it’s the best choice for you - you’ll do it in the fastest and most efficient way.

Example

You can install this library by using the command below:

npm install sharp

This is how you would resize an image using sharp:

const sharp = require('sharp');
const fs = require('fs');

sharp('yellow.png')
  .rotate(180)
  .resize(200)
  .toBuffer()
  .then((data) => {
    fs.writeFileSync('yellow.png', data);
  })
  .catch((err) => {
    console.log(err);
  });

Cropper.js

Cropper.js is another popular JavaScript library for image manipulation. You can use it to crop your images in all possible ways, change aspect ratio, rotate, zoom and work with canvas data. Cropper.js is the right choice for cropping without any extra features.

Use Cases

You can use its flexible API to create a custom image cropping UI in your web app allowing your users to adjust photos to the correct size and aspect ratio. It will be more efficient because it requires almost nothing, doesn’t have any useless features, and is very optimized for cropping!

Example

This library can be installed with the following command:

npm install cropperjs

import Cropper from 'cropperjs';

const image = document.getElementById('image');
const cropper = new Cropper(image, {
  aspectRatio: 16 / 9,
  crop(event) {
    console.log(event.detail.x);
    console.log(event.detail.y);
    console.log(event.detail.width);
    console.log(event.detail.height);
    console.log(event.detail.rotate);
    console.log(event.detail.scaleX);
    console.log(event.detail.scaleY);
  },
});

Check out the Cropper.js demo, click on any property that you want to apply to your image and see the result instantly.

For an in-depth guide on how to add image cropping to your React app, check out our guide on using react-image-crop.

Merge Images

Working with the canvas can be slightly tedious, especially when you need the canvas context to do relatively simple things like merging a few images. Merge Images abstracts all repetitive tasks into simple function calls. It’s a wrapper around the Canvas API, abstracting its low-level functions, which makes this specific task a lot easier. You can also create parameters like positioning, opacity, and several more. You can find them in the docs on GitHub.

Use cases

It’s a valuable library that can help with several tasks. For example, you can generate an NFT Collection where you can merge all components to have a complete set of variations. Or you can find this library helpful for personal use, for example, combining a few different images to generate a collage.

Example

Install the library itself:

npm install --save merge-images

Then you can use this code to generate a simple image:

//write this inside of your javascript file

import mergeImages from 'merge-images';

mergeImages(['/body.png', '/eyes.png', '/mouth.png']).then(
  (b64) => (document.querySelector('img').src = b64)
);

//And that would update the img element to show this image:

Here, I used it to merge several components to create a final variation (here I used Background, Character, Horns, Emotion, and Accessories).

LooksSame

LooksSame is a library for comparing images. If the two images you uploaded are duplicates, the library will let you know. All you have to do is provide a link to the pictures you would like compared. LooksSame is not strictly an image manipulation library, but is helpful for testing purposes.

Use cases

You can use this library and write code for automation and loop cycles that search a directory and compare each pair of your images for duplicates. Additionally, you can delete these duplicates and automate this process as well. It is a splendid library if you’re constantly working with ML algorithms that include large numbers of images.

Example

To start comparing images, you need just to install it by using this command:

npm i looks-same

Here, you can see how to test this library with Jest:

var looksSame = require('looks-same');

//Parameters can be paths to files or buffer with compressed png image

test('image1 and image2 are the same', async () => {
  expect(looksSame('image1.png', 'image2.png')).toBe(true);
});

//Result will be "image1 and image2 are the same" if they are the same

Jimp

Jimp stands for JavaScript Image Manipulation Program, that allows you to edit your image in almost every possible way. With it, you can invert your image, add some text, resize, use pixelation, clone images, blur them, invert colors, and several other cool features which will increase the image manipulation capabilities of your app.

Use cases

With Jimp, you can build your web application where you’ll be able to edit and manipulate images in almost any possible way. Crop, resize, rotate, and filter features let you create your own photo editor and add an interface on top of it. To streamline this process, you can try PhotoEditor SDK (PE.SDK). PE.SDK will provide you with a polished user interface and photo editing features, so you can focus on the application you are building instead.

Example

Here’s the JavaScript code to try this excellent library. There’s an async function in which you can see many types of properties that you can easily change. Such as Add Text, Resize Image, Blur the Image, etc. To use at least one of them, you’ll need to uncomment only part that you need, and your program is ready to run!

// Import dependencies
const Jimp = require('jimp');

(async function () {
  // Read the image
  const image = await Jimp.read('images/shapes.png'); // <http://www.example.com/path/to/lenna.jpg>

  // // Add text
  //  const font = await Jimp.loadFont(Jimp.FONT_SANS_16_WHITE); // bitmap fonts
  //  image.print(font, 0, 0, 'Hello world!'); // <https://github.com/libgdx/libgdx/wiki/Hiero>

  //  // Resize the image
  //  // Resize the image to 250 x 250
  //  image.resize(250, 250);

  //  // Resize the height to 250 and scale the width accordingly
  //  image.resize(Jimp.AUTO, 250);

  //  // Resize the width to 250 and scale the height accordingly
  //  image.resize(250, Jimp.AUTO);

  //  // Add a sepia wash to the image
  //  image.sepia();

  //  // Pixelation
  //  image.pixelate(5);
  //  image.pixelate(5, 50, 50, 190, 200); pixe,x, y, w, h

  //  // Clone
  //  const image2 = image.clone();

  //  // Blur the image
  //  image.gaussian(1);
  //  image.blur(1);

  //  // Inverts the image
  //  image.invert();

  //  // Set the brightness
  //  image.brightness( 0.5 ); // -1 to +1

  //  // Resize the image
  //  image.resize(256, 256);

  //  // Set the quality
  //  image.quality(100);

  //  // Convert to grayscale
  //  image.greyscale();

  // Save the image
  image.write('images/edited-shapes.png'); // writeAsync
})();

PS: Don’t forget to uncomment the effects which you want to use and save the file after. :)

As you can see here, I blurred everything and inverted the colors for the previous image that I showed you, getting the following result:

Conclusion

That’s it! Now you know the differences between five of the most popular image manipulation libraries for JavaScript. Some of them are useful for only one task, such as resizing images - others can do multiple tasks at once and help you with more processes. You have also met PE.SDK to work smarter and save time building photo editing solutions, and focus on building your great product instead.

Content Moderation for User-Generated Designs with CE.SDK and Sightengine

Antonello — Mon, 13 Jun 2022 14:07:54 GMT

Creating content and sharing it with others has become one of the core features offered by online services. As users publish their content to your website, you want to consider implementing a system to prevent inappropriate posts. Ideally, content will be programmatically moderated before it is uploaded on your platform. That is what you will learn in this article!

Follow this step-by-step tutorial and learn how to integrate content moderation with Sightengine in your design process built with CreativeEditor SDK in React. At the end of this tutorial, you will achieve the following result:

Why You Should Moderate User-Generated Content

User-generated content represents a large portion of online content, whether text or multimedia. Empowering users with tools and features to create high-quality content is excellent. However, it is a common concern that potentially inappropriate content on your website will be associated with your brand. Naturally, you will want to avoid this.

A recent example from the web-to-print industry serves as a cautionary tale: the German e-commerce company Spreadshirt neglected to moderate content on its t-shirt printing platform, leading to a wave of outrage on social media. Even though the company reacted swiftly and stated an apology, Spreadshirt could have avoided most of the damage if it had implemented a system that automatically flags potentially harmful content. Of course, a religious or cultural symbol is not inherently inappropriate, but the misuse is.

It is highly advisable to take precautions to prevent your company from facing similar adversity. A common problem, however, is the costly and time-consuming manual moderation, epecially when controlling multimedia files like images.

Luckily, you can use CE.SDK and Sightengine to programmatically integrate moderation into your content design process without manual reviews.

What is CE.SDK?

CreativeEditor SDK (CE.SDK) is a powerful, easy-to-use, and fully customizable design editor. With just a few lines of code, CE.SDK provides your application with template-based workflows and automation.

The role-specific editing UI focuses on keeping content adaptation dead simple: in Creator Mode, you have the liberty to create a design from scratch or customize existing templates. When creators are ready to share a design for others to use and customize in Adopter Mode (Default UI), creators may decide which elements can be changed. See CE.SDK in action or try it for free.

What is Sightengine?

Sightengine is a convenient tool for automatic content moderation. It detects and filters unwanted content in photos, videos, and live streams through fast, accurate, easy-to-integrate, privacy-compliant APIs.

Sightengine offers you APIs to get moderation results instantly and effectively. The API-centric approach allows you to scale to up to tens of millions of images per month. You can have a look at the Sightengine demo or try it for free.

Integrate Content Moderation into CE.SDK

Let us look at how to implement an image editor with content moderation for your users. For this, you will learn how to use CE.SDK and React to implement an image design component and use the CE.SDK APIs to programmatically moderate the content with Sightengine on a Node.js server. You can try out a live showcase of content moderation with CE.SDK.

With the following steps, you can easily achieve this result with CE.SDK, React, Sightengine, and Node.js.

1. Get Started with CE.SDK

Integrate CreativeEditor SDK with React by using the official documentation. As you will see in the final snippet, this takes only a few lines of code.

Now that you have a working CE.SDK instance, your users may unleash their creativity by designing templates and sharing them with others. The powerful APIs ensure control at every step of the editing process and allow programmatically performing operations, such as content moderation.

2. Build a Content Moderation API with Sightengine

Let’s move on to content moderation with Sightengine. Keep in mind that any backend technology will work, but here you will implement a content moderation API based on Sightengine in Node.js.

2.1 Getting Started with Sightengine

First, you need a Sightengine account. If you do not have one yet, you can create a free account. Fill out the form, click Sign Up, and receive a verification email in your inbox. Click on the verification link and activate your account.

Then, navigate to API Keys on your Sightengine Dashboard page and retrieve your API user and secret credentials. You will need them to interact with the Sightengine moderation APIs.

2.2 Prerequisites

Here is a list of all prerequisites for the backend application you are going to build:

You can add express and node-fetch to your project’s dependencies with the following command:

npm install --save express node-fetch

2.3 Setting up a Node.js Express Server

Follow this step-by-step guide to set up a Node.js Express development server. You will need this to perform an API call to Sightengine to moderate your content server-side. Any Sightengine API call involves your Sightengine user and secret pair of credentials, and you do not want to expose them.

You may also want to store the result of the moderation API in your database, as well as other data. For example, you might be interested in keeping track of how many times a particular user has violated the moderation rules to warn or ban them.

2.4 Implementing a Content Moderation API in your Node.js Server

You can now implement a content moderation API in your Node.js Express server as follows:

    const fetch = require("node-fetch")

    router.get('/content-moderation', function (req, res) {
      if (!req.query.url) {
            res.status(404).send("Please provide an image url")
            return
        }

        // defining the Sightengine API parameters
        const params = {
            url: req.query.url,
            models: "nudity,wad,offensive",
            api_user: "YOUR_SIGHTENGINE_USER",
            api_secret: "YOUR_SIGHTENGINE_SECRET",
        }

        // generating the Sightengine API URL
        const queryString = new URLSearchParams(params).toString();
        const requestOptions = {
            method: "GET",
        }
        const sightengineUrl = "https://api.sightengine.com/1.0/check.json?" + queryString

        // calling the Sightengine moderation API
        const externalRes = await fetch(SightengineUrl, requestOptions)
        // retrieving the moderation result from the Sightengine API
        const moderationResultText = await externalRes.text()

        res.append("Access-Control-Allow-Origin", ["*"])
        res.append("Access-Control-Allow-Methods", "GET")
        res.append("Access-Control-Allow-Headers", "Content-Type")

        // returning the Sightengine moderation result text
        res.status(200).send(text)
    })

Take care of replacing YOUR_SIGHTENGINE_USER and YOUR_SIGHTENGINE_SECRET with your Sightengine pair of credentials retrieved before.

Now, launch your Node.js server and reach the following URL below to use Sightengine to moderate the images specified in the URL query parameter:

    http://localhost:3000/content-moderation?url=<your_image_url>

3. Moderate Images Made with CE.SDK

We will now use CE.SDK and Sightengine to programmatically moderate user-generated images. Before we can dive in, check the following requirements.

3.1 Prerequisites

This is the list of all the prerequisites you need to build for the frontend demo application:

You can add the @cesdk/cesdk-js library to your project’s dependencies with the following command:

npm install --save @cesdk/cesdk-js

Also, the demo application will use other libraries, although they are not strictly required to achieve the goals of this tutorial and can be omitted here.

3.2 Getting Started

You can try out the demo application by cloning the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/content-moderation-with-cesdk-and-sightengine
cd content-moderation-with-cesdk-and-sightengine
npm i
npm start

Otherwise, keep following this tutorial and build the demo application for content moderation by yourself.

First, initialize a React project called content-moderation-with-cesdk-and-sightengine by employing the Create React App command below:

    npx content-moderation-with-cesdk-and-sightengine

Now, the content-moderation-with-cesdk-and-sightengine folder should have the following file structure:

content-moderation-with-cesdk-and-sightengine
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
   ├── App.css
   ├── App.js
   ├── App.test.js
   ├── index.css
   ├── index.js
   ├── logo.svg
   ├── reportWebVitals.js
   └── setupTests.js

Enter the content-moderation-with-cesdk-and-sightengine folder and start a local server with these two commands:

cd content-moderation-with-cesdk-and-sightengine
npm start

Visit http://localhost:3000/ in your browser, and you should now be seeing the default Create React App screen below:

3.3 Integrate the Moderation Check into your Design Process

Notice that the demo application you are about to implement is just a simplified version of the app you can find in this GitHub repository. This demo app integrates the Sightengine moderation functionality with the CE.SDK design capabilities.

Now, let’s move on to the main component where the integration takes place:

import CreativeEditorSDK from '@cesdk/cesdk-js';
import ValidationBox from '../ui/ValidationBox/ValidationBox';
import React, { useCallback, useEffect, useRef, useState } from 'react';
import { ReactComponent as RefreshIcon } from './refresh.svg';
import { moderateImages } from './moderationUtils';
import styles from './ImageModerator.module.css';
import classNames from 'classnames';

function selectBlock(cesdk, blockId) {
  // deselecting all blocks
  cesdk.engine.block
    .findAllSelected()
    .forEach((block) => cesdk.engine.block.setSelected(block, false));

  // selecting the block having the blockId passed as parameter
  cesdk.engine.block.setSelected(blockId, true);
}

const ImageModerator = () => {
  const cesdkContainerDiv = useRef(null);
  const cesdkRef = useRef(null);
  const [validationResults, setValidationResults] = useState([]);

  useEffect(() => {
    // initializing CE.SDK if it has not been initialized yet
    if (cesdkContainerDiv.current && !cesdkRef.current) {
      const config = {
        role: 'Adopter',
        theme: 'light',
        initialSceneURL: `${window.location.protocol + '//' + window.location.host}/cases/content-moderation/example.scene`,
        ui: {
          elements: {
            panels: {
              settings: true,
            },
            navigation: {
              action: {
                save: true,
              },
            },
          },
        },
        callbacks: {
          // calling this function when the
          // user presses the "Save" button
          onSave: () => runImageModerationCheck(),
        },
      };
      CreativeEditorSDK.init(cesdkContainerDiv.current, config).then(
        (instance) => {
          cesdkRef.current = instance;
          // running the moderation check on initialization
          runImageModerationCheck();
        }
      );
    }
    return () => {
      if (cesdkRef.current) {
        cesdkRef.current.dispose();
      }
    };
  }, [cesdkContainerDiv]);

  const runImageModerationCheck = useCallback(async () => {
    if (!cesdkRef.current) {
      return;
    }
    // retrieving the moderation results
    const validationResults = await moderateImages(cesdkRef.current);
    setValidationResults(validationResults);
  }, []);

  return (
    <div className={classNames(styles.wrapper, 'space-y-2')}>
      <div className={styles.header}>
        <div className={styles.headerDiv}>
          <div>
            <h3 className="h4" style={{ color: 'white' }}>
              Content Moderation
            </h3>
            <p className={styles.headerDivMessage}>
              Check images for compliance with your content guidelines before
              further processing and provide user feedback.
            </p>
          </div>
          <button
            onClick={() => runImageModerationCheck()}
            className={'button button--white space-x-2'}
          >
            <span>Validate Image Content</span>
            <RefreshIcon />
          </button>
        </div>
        <ValidationBox
          results={validationResults}
          onSelect={(blockId) => {
            selectBlock(cesdkRef.current, blockId);
          }}
        />
      </div>
      <div className={styles.cesdkWrapper}>
        <div ref={cesdkContainerDiv} className={styles.cesdk}></div>
      </div>
    </div>
  );
};

export default ImageModerator;

In the first useEffect() hook, CE.SDK is initialized, and a callback on the save action is registered. This will take care of calling the runImageModerationCheck() function when the user presses the CE.SDK Save button. For more information on how callbacks work in CE.SDK, see the official documentation on observing events.

runImageModerationCheck() retrieves the moderation results obtained from Sightengine and then stores it in the validationResults array variable. This is then passed to the ValidationBox component, which will show the results of the validation process to the end-user. None of this would be possible without the moderateImages() function from the moderationUtils.js file below:

export const moderateImages = async (cesdk) => {
  // retrieving all image blocks with a name
  const imageBlocksData = cesdk.engine.block
    .findByType('image')
    .map((blockId) => ({
      blockId,
      url: cesdk.engine.block.getString(blockId, 'image/imageFileURI'),
      blockType: cesdk.engine.block.getType(blockId),
      blockName: cesdk.engine.block.getName(blockId),
    }))
    .filter((block) => {
      return block.blockName !== '';
    });

  console.log('Moderation check launched...');

  // turning the moderation results on each image block into
  // a flat array
  const imagesWithValidity = await Promise.all(
    imageBlocksData.flatMap(async (imageBlockData) => {
      // retrieving the moderation results
      const imageModerationResults = await callSightengineModerationAPI(
        imageBlockData.url
      );

      console.log(imageBlockData.blockName, imageModerationResults);

      return imageModerationResults.flatMap((checkResult) => ({
        ...checkResult,
        ...imageBlockData,
      }));
    })
  );

  console.log('Moderation check completed');

  return imagesWithValidity.flat();
};

// a cache variable to avoid
// calling the Sightengine API when not necessary
const complianceCache = {};

const callSightengineModerationAPI = async (url) => {
  // if the url is already present in the cache, then
  // return the result previously retrieved
  if (complianceCache[url]) {
    return complianceCache[url];
  }

  // calling the Sightengine moderation API
  const response = await fetch(
    // your endpoint URL
    'https://europe-west3-img-ly.cloudfunctions.net/sightengineApiProxy?url=' +
      encodeURIComponent(url),
    {
      method: 'get',
      headers: {
        accept: 'application/json',
        'Accept-Language': 'en-US,en;q=0.8',
        'Content-Type': 'multipart/form-data;',
      },
    }
  );

  const results = await response.json();

  if (results.error) {
    console.error(results.error);
    return [];
  } else {
    const checkResults = [
      {
        name: 'Weapons',
        description: 'Handguns, rifles, machine guns, threatening knives...',
        state: percentageToState(results.weapon),
      },
      {
        name: 'Alcohol',
        description: 'Wine, beer, cocktails, champagne...',
        state: percentageToState(results.alcohol),
      },
      {
        name: 'Drugs',
        description: 'Cannabis, syringes, glass pipes, bongs, pills...',
        state: percentageToState(results.drugs),
      },
      {
        name: 'Nudity',
        description: 'Images that contain either raw nudity or partial nudity.',
        state: percentageToState(1 - results.nudity.safe),
      },
    ];

    // storing the Sightengine results in the cache
    complianceCache[url] = checkResults;
    return checkResults;
  }
};

const percentageToState = (percentage) => {
  // defining the custom moderation logic based on the
  // percentage returned by Sightengine
  if (percentage > 0.8) {
    return 'failed';
  } else if (percentage > 0.4) {
    return 'warning';
  } else {
    return 'success';
  }
};

Here, we use the CE.SDK reference variable to retrieve all images in the scene through the CE.SDK APIs. Then, these are converted into URLs and passed callSightengineModerationAPI() function, which takes care of calling the Sightengine moderation API, retrieving the results, using them to produce a new output, and returning it to the caller.

We used an IMG.LY endpoint here to evade putting together a backend application. In a real-world scenario, replace https://europe-west3-img-ly.cloudfunctions.net/sightengineApiProxy?url= with the URL of the /content-moderation endpoint of your backend application. Also, note that a cache system was set up to avoid calling the Sightengine API too many times and using up call credits.

This is what the demo application will look like after being launched and the moderation check has been executed:

As you can see, the Sightengine API correctly detects drugs and weapons. Next, remove the rightmost image containing a gun and click Save to re-run the moderation check.

Sightengine no longer detects weapons in the image, proving that the moderation process is working as expected.

Et voilà! You have learned to implement content moderation for user-generated images with CE.SDK and Sightengine.

Conclusion

A challenge of user-generated content is avoiding the risk of hosting and displaying offensive or outright illegal content. Inappropriate content affects your user community and brand reputation and may even cause legal action against you.

In this article, we used CE.SDK to initialize a fully-featured design editor in React. We employed its API to programmatically call a Node.js server using Sightengine to perform content moderation on images. This allows moderation of user-generated content automatically and prevents the upload of inappropriate content to your website.

We are only scratching the surface of what is possible with CE.SDK – you can implement any UI on top of its powerful image processing API and extend the core functionality with features such as placeholders and define validation rules to guide your users’ creation process.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out on Twitter with any questions, comments, or suggestions.

How To Add Overlays to a Video in React Native

Antonello — Mon, 28 Mar 2022 16:23:32 GMT

In this article, you will learn to add a text and image overlay to a video with JavaScript in React Native. All you need is React Native Video, the correct styling rules, and just a few lines of code.

By following this guide, you will easily achieve a logo and text overlay:

First, let’s learn how to implement a React Native component that allows you to add an overlay image and overlay text to a video.

What is React Native Video?

Mobile apps come with several native interfaces and features. This is what makes React Native different from React, which is based on browser rendering. So, <div> becomes <View>, <p> becomes <Text>, and so on. In particular, React Native supports only a limited amount of components. You can find the entire list here.

To this day, React Native does not have a native <Video> component. That is why react-native-video was born - a widely used library without considering alternatives. This 500kB package equips you with everything you need to get started with videos in React Native.

Prerequisites

Here is the list of all the prerequisites for the demo application you are going to build:

You can add react-native-video to your project’s dependencies by launching the following npm command:

npm install react-native-video

Or if you are a yarn user:

yarn add react-native-video

Then, follow this guide to learn how to link react-native-video to your project. This step is not required for Android users with React Native 0.60 and above.

Adding Overlay Image and Text to a Video in React Native

You can clone the GitHub repository supporting this article and launch the following commands to try the demo application:

git clone https://github.com/Tonel/how-to-add-overlays-to-videos-in-react-native-imgly
cd how-to-add-overlays-to-videos-in-react-native-imgly
npm i
npm start

The project is a React Native Create App project, and you can learn more about how it works and how to run it here.

Otherwise, keep following this step-by-step tutorial and learn how to build a React Native component for adding overlays to a video.

1. Initializing a React Native Project

Create React Native App is the officially supported and easiest way to create an empty and working React Native application. Install it globally with the following command:

npm install -g create-react-native-app

Now, you can initialize a new project called react-native-video-overlays-demo with the following command:

create-react-native-app react-native-video-overlays-demo

The react-native-video-overlays-demo folder should now contain a demo project with the following file structure:

react-native-video-overlays-demo
├── .buckconfig
├── .gitattributes
├── .gitignore
├── App.js
├── app.json
├── babel.config.js
├── index.js
├── metro.config.js
├── package.json
├── package-lock.json
├── yarn.lock
├── android
│   └── ...
└── ios
    └── ...

Enter the react-native-video-overlays-demo folder:

cd react-native-video-overlays-demo

Then launch the development server with one of the following commands:

for launching an Expo app

npm start

for Android development on your phone or an emulator

npm run android

for iOS development on your phone or an emulator

npm run ios

for react-native-web development

npm run web

2. Defining a Component to Add Overlays on a Video

Let’s see how to build a VideoWithOverlays component that allows you to add overlays such as text and an image to a video. You can implement it as follows:

import React, { useEffect, useState } from 'react';
import { Image, StyleSheet, Text, View } from 'react-native';
import Video from 'react-native-video';

export default function VideoWithOverlays({
  videoComponentProps,
  text,
  imageSrc,
}) {
  const {
    video,
    overlayText,
    videoWithOverlays,
    overlayTextView,
    overlayImageView,
    overlayImage,
  } = styles;

  const [height, setHeight] = useState(0);
  const [overlayHeight, setOverlayHeight] = useState(0);

  useEffect(() => {
    // setting the overlay height to 10px starting
    // from the bottom of the video
    setOverlayHeight(height / 2 + height / 3 - 10);
  }, [height]);

  return (
    <View style={videoWithOverlays}>
      <Video
        {...videoComponentProps}
        style={video}
        onLayout={(event) => {
          const { height } = event.nativeEvent.layout;
          setHeight(height);
        }}
      />
      <View style={{ ...overlayTextView, top: overlayHeight }}>
        {text && <Text style={overlayText}>{text}</Text>}
      </View>
      <View style={{ ...overlayImageView, top: overlayHeight }}>
        {imageSrc && (
          <Image style={overlayImage} source={imageSrc} resizeMode="contain" />
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  overlayTextView: {
    position: 'relative',
    alignItems: 'flex-end',
    justifyContent: 'flex-end',
    right: 5,
  },
  overlayImageView: {
    position: 'relative',
    alignItems: 'flex-start',
    justifyContent: 'flex-end',
    left: 5,
  },
  overlayImage: {
    position: 'absolute',
    height: 40,
  },
  overlayText: {
    fontSize: 25,
    fontWeight: 'bold',
    color: 'white',
  },
  video: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
    marginTop: '50%',
  },
  videoWithOverlays: {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
  },
});

The react-native-video Video component is used to play the video and its props are exposed thanks to the videoComponentProps prop. Then, after being placed into the layout, its height depending on the size of the device screen is stored in the height variable. This is used to calculate the height the overlays should be placed at in the useEffect hook. In detail, the overlays are placed at 10px from the bottom edge of the video.

The first internal View represents the optional text overlay. The overlay logic depends entirely on the custom overlayTextView and overlayText StyleSheet fields, representing the CSS classes responsible for displaying the text passed as a prop with text as an overlay. By default, the text overlay is placed in the bottom-right corner of the video.

The second internal View represents the optional image overlay. The overlay logic depends entirely on the custom overlayImageView and overlayImage StyleSheet fields, representing the CSS classes taking care of showing the image passed as a prop with imageSrc as an overlay. By default, the image overlay is placed in the bottom-left corner of the video. As you can see, both overlays are optional.

3. Putting It All Together

Let’s see the VideoWithOverlays component defined above in action. All you need to do is replace the App.js file with this:

import React from 'react';
import { StyleSheet, View } from 'react-native';
import VideoWithOverlays from './src/components/VideoWithOverlays/VideoWithOverlays';

export default function App() {
  // a static image stored into the assets folder
  const imgLy = require('./src/assets/images/IMG_LY.png');

  return (
    <View style={styles.container}>
      <VideoWithOverlays
        videoComponentProps={{
          // replace this free video source with your video
          source: {
            uri: 'https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4',
          },
        }}
        // overlay text
        text={'IMG.LY'}
        // overlay image
        imageSrc={imgLy}
      />
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#fff',
    alignItems: 'center',
    justifyContent: 'center',
  },
});

Here, <VideoWitOverlays> is initialized with a free video source retrieved from a public URL and two overlays. The text overlay states “IMG.LY” and it is placed in the bottom-right corner of the video. While the image overlay represents the IMG.LY logo and is placed in the bottom-left corner of the video. This is a screenshot of the VideoWitOverlays in action:

Final Considerations

Adding overlays elements to a video in React Native is not complex and takes just a few lines of code. On the other hand, the approach presented above is just a frontend trick and does not modify the video. You are just showing it behind text or an image, giving the illusion to the users that it is a single source of content.

If you wanted to let users modify their videos by adding overlays or other elements and then export them, this would involve complex logic and require an advanced UI. Building such a React Native component is a time-consuming task that represents a waste of energy. If that is your goal, you should take into consideration a complete and ready-to-use SDK solution, such as VideoEditorSDK. This would provide your users with the ability to add overlays and use many other cool features.

Add Overlays to a Video With React Native module for VideoEditor SDK

Visit this page from the official documentation to learn how to get started with VideoEditorSDK in React Native. Then, load your video and start editing it by adding overlays, text, and stickers. Follow the previous links to learn more about each feature.

Download our free mobile demo app from the App Store or Google Play, and try an extensive set of tools to edit videos.

Conclusion

In this tutorial, we learned how to define a React Native component that allows you to reproduce a video and add overlays. Specifically, we used react-native-video and defined everything required to place text or an image on your video. However, if you intend to export your edit, you will have to build a complex video editing application. In this scenario, you will want to consider a fully-featured, cutting-edge, and easy-to-adopt solution – such as VideoEditorSDK.
Looking for more video capabilities? Check out our solution for Short Video Creation, and Camera SDK!

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How To Add a Sticker to a Texture With Three.js

Antonello — Fri, 25 Feb 2022 15:10:58 GMT

Disclaimer: This article has been updated in June 2023 to reflect the changes introduced by Three.js r150

In this article, you will learn to add a sticker to a texture with WebGL. In detail, you will see how to use the Three.js JavaScript library to add an overlay image to a texture. Check out our guide to explore other image editing operations, such as resizing an image in React.

Dealing with 3D graphics used to be a challenging, complex, and time-consuming task. Thanks to libraries like three.js, it has become more accessible than ever. Follow this easy step-by-step how-to and achieve the following result:

Prerequisites

Here are all the prerequisites you need to meet to follow this tutorial and build the demo application shown above:

Explore the official documentation page to learn about how to install three.js.

What is Three.js?

As stated in the GitHub page of the project, the goal of the Three.js project is to create a lightweight 3D JavaScript library with a very low level of complexity. In other words, Three.js aims to make 3D graphics more accessible, especially considering that Three.js only requires a browser to run.

To be more specific, it allows you to create a 3D graphics project where you can modify elements in your browser and see visual feedback. The library is currently built on top of a WebGL renderer, but it also supports WebGPU, SVG, and CSS3D renderers.

If you are not familiar with these concepts, WebGL stands for “Web Graphics Library” and is a JavaScript API for rendering high-performance interactive 3D and 2D graphics. You can use it in many web browsers without external plugins or libraries.

As you can verify from the MDN compatibility page, browsers currently support the most popular WebGL features. However, keep in mind that the user’s device must meet hardware requirements for some features to work.

WebGL is undoubtedly a powerful tool but requires a lot of skill and involves a lot of boilerplate code. Here is where Three.js comes into play, making 3D development much easier.

Now, let’s see it in action!

Adding an Overlay Image to a Texture in WebGL

Clone the GitHub repository that supports this article with the following commands:

git clone https://github.com/Tonel/how-to-add-a-sticker-with-threejs-imgly

Additionally, try the demo application by launchingthe command below:

npm install
npm run start

Then, visit http://localhost:5000 in the browser.

Now that you know what the demo app looks like, keep following this tutorial and learn how to build it.

Specifically, this is the entire JavaScript code required to achieve the goal:

import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

// set up the scene, camera, and renderer
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(
  40,
  window.innerWidth / window.innerHeight,
  1,
  1000
);
camera.position.set(150, 100, 150);
new OrbitControls(camera, document.body);

// initialize the WebGL renderer
const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// add light to the scene to make the texture visible
const light = new THREE.AmbientLight(0xffffff);
scene.add(light);

// add the axes helper indicator to the scene
scene.add(new THREE.AxesHelper(20));

// load the base texture
const texture = new THREE.TextureLoader().load(
  'https://threejs.org/examples/textures/hardwood2_diffuse.jpg'
);

// create a new buffer geometry and place it
// to the horizontal plane
const textureGeometry = new THREE.PlaneGeometry(50, 50);
textureGeometry.rotateX(-90 * THREE.MathUtils.DEG2RAD);
// translate the plane to the center of the scene
textureGeometry.translate(25, 0, 25);

// create a MeshBasicMaterial with the base texture
const textureMaterial = new THREE.MeshBasicMaterial({ map: texture });

// create a mesh using the geometry and material
// and add it to the scene
const textureMesh = new THREE.Mesh(textureGeometry, textureMaterial);
scene.add(textureMesh);

// repeat the same logic to place the sticker image
// over the texture
const sticker = new THREE.TextureLoader().load(
  'https://i.imgur.com/IYh17Rv.png'
);

const stickerGeometry = new THREE.PlaneGeometry(20, 20);
stickerGeometry.rotateX(-90 * THREE.MathUtils.DEG2RAD);
stickerGeometry.translate(25, 0, 25);
const stickerMaterial = new THREE.MeshBasicMaterial({
  map: sticker,
  transparent: true,
});
// create a mesh using the image geometry and material
const imageMesh = new THREE.Mesh(stickerGeometry, stickerMaterial);
scene.add(imageMesh);

// adjust the position and rotation of the sticker mesh
// by setting the y-coordinate to 1
// in order for the sticker to rest on the texture
imageMesh.position.set(0, 1, 0);

// render the scene
function launchThreeJs() {
  requestAnimationFrame(launchThreeJs);
  renderer.render(scene, camera);
}

launchThreeJs();

First, the code imports the necessary components from the Three.js library. Next, it initialize a Scene and set up a PerspectiveCamera with OrbitsControl, allowing the user to move around. Then, it creates a WebGL renderer and sets its size to match the window’s dimensions. The scene is lit by an AmbientLight instance, which illuminates all objects in the scene equally.

The snippet loads a base texture using TextureLoader from the given URL, creates a PlaneGeometry, defines a mesh with some geometries, and then add the resulting element to the scene. This logic will take care of rendering the texture image in the scene.

To add the sticker to the texture, the same procude gets repeated. This time, Three.js is instructed to load a sticker image and position it over the original texture. Note that the position of the sticker mesh is adjusted by setting its y-coordinate to 1 to ensure the new image rests on top of the old one.

Et voilà! You just learned how to add a sticker to a WebGL texture with three.js!

Adding a Sticker With `PHOTOEDITORSDK`

PhotoEditorSDK uses WebGL as the main renderer. You can effortlessly add a sticker to your textures without writing any line of code. WebGL will be used behind the scene. Consequently, you do not need to know WebGL or code in GLSL to harness its power.

Adding a sticker to an image only takes a few seconds here. Just jump to this article from the official documentation and learn how to get started with PhotoEditorSDK in Vanilla JavaScript. Then, upload your texture image and a sticker with the Sticker feature. As shown below, you can position a sticker on top of the original image with a point-and-click UI:

Check out this feature on the PhotoEditorSDK demo page.

Conclusion

Today we learned how to add an overlay image to a WebGL texture in JavaScript with three.js. This tool makes it easier to deal with WebGL and 3D graphics. In particular, by using one of the many plugins supporting the Three.js project, we were able to add a sticker to a texture with just a bunch of lines of code.

3D graphics remains a complex topic, especially when it comes to developing in the OpenGL Shading Language. Three.js tries to hide this encumbrance as much as possible, but sometimes you still require it. If you want to avoid using WebGL directly to achieve your goal, consider an all-in-one and easy-to-use solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How To Scale, Crop, Flip, and Filter an Image in CSS

Antonello — Wed, 09 Feb 2022 19:41:38 GMT

In this article, you will learn how to perform popular image processing operations in CSS. That is possible thanks to the several features introduced by CSS3, which allow you to change how the images look to end-users. To learn instead about image manipulation operations for other frameworks, you can head on here: React or vanilla JS.

First, let’s delve into how image manipulation works in CSS. Then, let’s finally see the most common CSS image processing operations in action.

How CSS Image Manipulation Work

CSS does not allow you to deal with files, data formats, or data transformation. In other words, CSS cannot change an image on a deep level, yet with CSS, you can determine how a browser displays elements.

As a result, as opposed to what would happen when using an HTML canvas element, no pixel manipulation will be executed behind-the-scene. Therefore, your CSS rules won’t affect the file that represents your image. What changes is how your image is displayed frontend-side.

You can flip, blur, and crop an image in CSS, but when downloading the image with the right-click option, the image saved will always be a copy of the original image file. Read this page from MDN Web Docs, if you want to learn more about this phenomenon.

Prerequisites

The CSS properties you will see are widely based on CSS3 and are not supported by Internet Explorer. To verify if your target browsers support a CSS property, visit MDN Web Docs and explore the Browser compatibility section.

Image Processing in CSS

Let’s look at how to perform scaling, cropping, flipping, and filtering an image in CSS.

Scaling an image with `object-fit: scale-down`

The object-fit CSS property allows you to define how the content of an HTML <img> element should be resized to fit its container. In particular, scale-down acts as if none or contain were set, depending on which would result in a smaller image.

When it acts like none, the image will keep its original intrinsic size and not be scaled down. On the contrary, when it acts like contain the image will be scaled down to fit the container while maintaining its aspect ratio.

Test this CSS rule in the live demo below:

In particular, this is the CSS rule where the magic happens:

.manipulated-image {
  object-fit: scale-down;
  height: 100px;
}

Cropping an image with `object-fit` and `object-position`

By employing object-fit in conjunction with object-position, you can show a part of your image, just if it has been cropped. In detail, the object-position CSS property allows you to specify the positioning of your <img> element within its container box. This happens by passing this CSS property a position parameter containing from one to four values. This parameter is what defines the position of the <img> element.

By using object-fit to set how your <img> element should fit its container and by positioning it with object-position, you can show only a part of your image as shown in the live example below:

This is the CSS rule to look attention to:

.manipulated-image {
  object-fit: cover;
  object-position: -150px -25px;
}

Flipping an image with `transform`

The transform CSS property is a powerful tool that gives you the ability to rotate, scale, skew, or translate an<img> element. Specifically, you can flip an image by using the scaleX() and scaleY() transformation functions, or by adopting the rotateX() or rotateY() functions.

scaleX() and scaleY() define a transformation that resizes an HTML element along the x-axis and y-axis, respectively. Both accept a number representing the scaling factor as a parameter. Learn how to flip an image vertically with scaleX() in the example below:

In detail, this is the CSS rule to flip an image vertically:

.manipulated-image {
  transform: scaleX(-1);
}

Similarly, rotateX() and rotateY() define a transformation that rotates an HTML element without deforming it around the horizontal axis and vertical axis, respectively. Both require an angle representing the angle of rotation as a parameter. Now, let’s see how to flip an image vertically with rotateY() in the example below:

Specifically, this is what the CSS rule to flip an image vertically looks like:

.manipulated-image {
  transform: rotateY(180deg);
}

Filtering an image with `filter`

The filter CSS property allows you to apply visual effects on an <img> element. This property provides access to several built-in effects, such as blur(), grayscale(), contrast(), and many others. If required, you can also define your custom image filtering functions.

CSS filters are a powerful tool that is typically used to adjust how an image is displayed by the browser and shown to the end-user. Using them is really easy, as shown in the live demo below:

These are the CSS rules where the filtering operations are defined:

.blurred-image {
  filter: blur(2px);
}
.grayscale-image {
  filter: grayscale(80%);
}
.high-contrast-image {
  filter: contrast(150%);
}

CSS-Only Image Editor vs PhotoEditor SDK

Image manipulation in CSS is an easy and straightforward way to manipulate how an image is displayed with no effort. On the other hand, you cannot save or export the manipulated images with CSS. This means that it is not possible to implement a file image editor based solely on CSS.

Therefore, you can rely on CSS only if you are interested in changing how your images are rendered by the browser. Conversely, if you are looking for a way to manipulate your images and then export them as files, then you need a much more advanced tool. In this case, a commercial solution like IMG.LY’s Photo Editor SDK will provide you with a complete, fast, and easy-to-use image editor.

Conclusion

Today we learned how to use CSS to control how browsers display images. In detail, we delved into how to scale, crop, flip, and filter an image in CSS. These represent the most common image manipulation operations and allow you to transform your image CSS-side. As we have learned, these operations only determine how your browser renders an image. They do not change an image as a data structure.
This means that it is not possible to create an image editor that allows users to export manipulated images while using only CSS. So, if this is what you were looking for, you should consider adopting a fast, advanced, and ready-to-use solution – such as PhotoEditorSDK. And you can get right into the PhotoEditorSDK or VideoEditorSDK and its myriad integrations by starting with a free trial over here.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

How To Manipulate an Image With Jimp in React

Antonello — Wed, 26 Jan 2022 13:36:27 GMT

In this article, you will learn how to use jimp in React to perform the most common image processing operations. As you are about to learn, this popular JavaScript library allows you to effortlessly manipulate an image with a few lines of code.

What is Jimp?

Jimp stands for JavaScript Image Manipulation Program, and it is one of the most popular open-source image processing libraries, with more than one million weekly downloads. As stated on the official GitHub page, Jimp is a Promise-based library that relies entirely on Vanilla JavaScript. This means that it has no native or external dependencies, as you can verify here. This makes it more reliable and lightweight.

Jimp comes with several functions that allows you to transform your images in endless ways. Particularly, it includes the resize(), blur(), crop(), rotate() functions and many others. At the time of writing, these are image formats supported by Jimp:

Let’s now see how to use the Jimp transformation functions.

Prerequisites

This is the list of prerequisites you need to make the React applications using Jimp we are about to build works:

You can add jimp to your project’s dependencies by launching the following command:

npm install jimp

Now, you have everything required to start getting your hands dirty with Jimp in React.

Basic Image Processing in Jimp

All the Jimp usage examples presented below share the same logic. So, let’s address it immediately.

Specifically, this is what a basic JimpDemo component looks like:

import Jimp from 'jimp';
import { useEffect, useState } from 'react';

export function JimpDemo({ imageUrl }) {
  const [jimpImage, setJimpImage] = useState(undefined);
  const [image, setImage] = useState(undefined);
  const [transformedImage, setTransformedImage] = useState(undefined);

  // loading an image every time imageUrl changes
  useEffect(() => {
    const loadImage = async () => {
      // generating the Jimp data structure
      // loading an image from an URL
      const jimpImage = await Jimp.read(imageUrl);
      setJimpImage(jimpImage);

      // transforming jimpImage into its Base64 representation
      // and storing it
      const image = await jimpImage.getBase64Async(Jimp.MIME_JPEG);
      setImage(image);
    };

    loadImage();
  }, [imageUrl]);

  // generating the transformed image
  // as soon as the Jimp data structure is ready
  useEffect(() => {
    if (jimpImage) {
      const transformImage = async () => {
        // performing the Jimp image processing operation
        // on jimpImage...

        // e.g. jimpImage.crop(100, 100)

        // storing the transformed image
        // in Base64 format
        const transformedImage = await jimpImage.getBase64Async(Jimp.MIME_JPEG);
        setTransformedImage(transformedImage);
      };

      transformImage();
    }
  }, [jimpImage, height, width]);

  return image && jimpImage ? (
    <>
      <h1>Original Image</h1>
      <img className="originalImage" src={image} alt="Original" />
      <h1>Transformed Image</h1>
      <img
        className="transformedImage"
        src={transformedImage}
        alt="Transformed"
      />
    </>
  ) : (
    <>Loading...</>
  );
}

First, the imageUrl prop value storing the URL to the image you want to transform is used to generate a Jimp data structure. This happens in the first useEffect() function when launching Jimp.read(). This function takes the path to a file or a URL and returns a Promise with the Jimp data structure you call the manipulation functions on. Also, the Base64 representation of the original image is stored to be able to show it in the HTML <img> tag.

Keep in mind that the function passed to useEffect() should not be async. This is why an internal async function was defined and then called instead.

Then, the second useEffect() function is called as soon as jimpImage is initialized and takes care of performing the transformation function on the original image. Again, the Base64 representation of the transformed image is saved.

Finally, both the original and transformed images are rendered by the component. As you can imagine, what will change in the following examples will be the Jimp image transform function called, which might also require extra props.

Cropping an image

The Jimp crop() function crops an image at a given point to a given size.

jimpImage.crop(x, y, width, height)

x: the x coordinate to crop from
y: the y coordinate to crop from
width: the width of the crop region
height: the height of the crop region

See it in action in the live example below:

Resizing an image

The Jimp resize() function resizes an image to set width and height using a 2-pass bilinear algorithm.

jimpImage.resize(width, height);

width: the width to resize the image
height: the height to resize the image to

See it in action in the live example below:

Rotating an image

The Jimp rotate() function rotates an image clockwise by a number of degrees. Note that the width and height of the image will be resized accordingly by default.

jimpImage.rotate(degrees);

degrees: the number of degrees to rotate the image by

See it in action in the live example below:

Filtering an image

Although Jimp does not come with a single filtering function, it offers several functions to achieve the desired results. For example, you can implement an image filtering feature by harnessing the blur(), color(), dither(), normalize(), or threshold() functions.

Let’s see some of them in action in the live demo below:

Jimp vs PhotoEditor SDK

Undoubtedly, Jimp is a powerful tool. It’s an amazingly lightweight and simple library that offers so many image manipulations. On the other hand, achieving complex results requiring multiple manipulations to be performed on the end user’s needs could become challenging. In detail, this would require an advanced, robust, reliable, and easy-to-use UI that only a commercial solution such as IMG.LY’s PhotoEditor SDK can offer.

You could build such a UI yourself on top of Jimp, but this would take a lot of time and effort. This is especially true if you want to implement a general-purpose application, such as an image processing application. For limited and specific tasks without any user interaction, Jimp is the ideal solution.

Also, now you can take a look at the CreativeEditor SDK which is a more advanced API solution and obvious alternative to Jimp!

Conclusion

Today we learned about the jimp JavaScript library, its dependencies, how it works, and how to use it in React real-world scenarios. Specifically, this npm library comes with several image manipulation functions that can be used with a few lines of code. However, building a complex image editing application on top of it would require an entire UI, and you might want to avoid spending time on it. In this case, you should look into a more advanced, fully-featured, and ready-to-use solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

CE.SDK v1.3.0 Release

Daniel — Mon, 17 Jan 2022 15:07:02 GMT

After another six weeks of work, we are happy to ship another update of our CreativeEditor SDK (CE.SDK). This release is again packed with great features to enhance the developer experience and user experience. It includes:

New improved Default UI
Lockable Design & Placeholders
SVG Assets
Multi Selection
Nudging (moving elements with arrow and shift key)
Create scenes from an Image.

New Improved Default UI

That is the first release to introduce a new UI type for CE.SDK, which we call Default UI. It comes with a slick interface and focuses on fast design adoption. The Default UI removes the need for inspectors for many editing cases, thus leaving your users undistracted and focused on adapting or creating designs.

Check our official documentation to learn about configuring the UI Elements.

Lockable Design & Placeholders

In the previous version, CE.SDK introduced the roles “creator” and “adopter” – differentiating between template creation and adoption. In creator mode, you have the liberty to create a design from scratch or change existing designs.

When creators are ready to share a design, they define which design elements can be changed when opening the editor in the “adopter” role. Typically, creators will allow adopters to exchange content, but also the options to style or arrange an item can be controlled.

When the editor is opened in the “adopter” role (Default UI), items that are allowed to change will be highlighted and images will be shown with an overlay indicating that the content shall be replaced before exporting or saving the design.

Find more information about placeholders and constraints in our official documentation.

Dashboard

Creating and adopting designs is one part, but organizing designs is another. With this release, we also introduced a dashboard to create and organize your designs for your users. Reach out to us if you want to see a demo of the dashboard in action and understand.

SVG Assets

With this version, we introduced SVG as an asset-type alternative to JPEG and PNG image formats. This has the benefit that standard SVG Stickers can just be used or uploaded in every design. This has the major advantage that stickers will be crisp at any resolution while coming with a very low download size. Reducing the server load and data transfer for you and your users. In this turn, we also replaced all our example stickers with their SVG equivalents.

Multi Selection

Positioning multiple items at once has been a pain before this release, but not anymore. With the new multi-selection interaction, users can select two or more items at once. Just hold down the “shift” modifier key and start selecting multiple items. Once selected, they can be moved around simultaneously.

Nudging

Always wanted to fine-tune your design and pixel-perfectly align your content?

You can now use the arrow keys (⬅️, ➡️, ⬆️ ⬇️) to nudge the selected items to their correct position pixel by pixel. You can also hold down shift to nudge even further.

Create Scene from Image

Before this release, the CE.SDK could either start from an empty or a predefined scene. From now on, there is also an option to start the editor with any image. It will auto-create a page with the size of the given image and insert the image into this one, allowing a more content-driven creation process. Learn how to create a scene from an image in our documentation.

Minor changes

Besides the major features, there are a ton of small improvements. These include:

UI interface to rename items and pages to a user-defined name
Ability to choose between two UI types, “default” and “advanced”
Searching in image libraries now uses i18n translations
Ability to change the UI Font.

Feel free to send us any questions, or let us know what you think on Twitter.

Thank you for your time!

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How to Apply Custom Image Filters Using HTML5 Canvas in JavaScript

Antonello — Tue, 14 Sep 2021 15:25:56 GMT

In this article, you will learn how to apply custom filters to an image in JavaScript, without using any external library. As you are about to learn, this goal can be achieved by employing only the HTML5 <canvas> element.

Instagram has changed image processing forever. Since its market debut, users have become increasingly interested and accustomed to applying filters to their images and photos. This desirable, valuable, and requested feature might seem complex to implement, but it is not.

So, let’s see how to implement a feature for users to apply custom filters to an image with Vanilla JavaScript. Follow this step-by-step guide to achieve the following result:

Custom Filters With `<canvas>`

The HTML <canvas> element allows you to edit an image. It natively offers you the possibility to implement several image-processing features, such as implementing a brush. In other words, it is the only prerequisite required to apply custom filters to an image. All you need to do is load an image, turn it into an array of pixels by using the getImageData() function, and apply a transformation to change their value according to the result you want to achieve. Finally, you can either overwrite the original image or create a new one after applying the filter.

Now, let’s see everything required to implement such a cool feature.

Clone the GitHub repository supporting this tutorial by launching the following command:

    git clone https://github.com/Tonel/how-to-apply-custom-image-filters-in-javascript-img-ly

And try the application by opening the index.html file in your browser.

Otherwise, continue following this article to learn how to implement the same demo application step by step.

1. Implementing Simple Filters

Simple filters can be applied by changing the value of each pixel the image consists of. For example, you can implement a function to apply thresholding as follows:

function applyThreshold(sourceImageData, threshold = 127) {
  const src = sourceImageData.data;

  for (let i = 0; i < src.length; i += 4) {
    const r = src[i];
    const g = src[i + 1];
    const b = src[i + 2];

    // thresholding the current value
    const v = 0.2126 * r + 0.7152 * g + 0.0722 * b >= threshold ? 255 : 0;

    src[i] = src[i + 1] = src[i + 2] = v;
  }
  return sourceImageData;
}

This simple function receives the ImageData object relative to the image to be thresholded as a parameter. Then, its data property representing a one-dimensional array with the image data in the RGBA order is accessed. This contains integer values between 0 (black) and 255 (white). Finally, data is iterated and modified to apply the thresholding operation, before returning the updated ImageData object.

That is just an example, but with this approach, you may implement many other simple filters.

2. Implementing Convolution-Based Filters

Convolution is one of the most important concepts when it comes to image processing. Particularly, it allows you to apply a filter by taking the weighted sum of a rectangle of pixels from an image and use it as the output value pixel by pixel. The coefficients used to perform the weighted sum are defined through a matrix, which is called kernel. By changing the kernel, the final result will change accordingly. Some kernels are more popular than others and can be used for blurring, sharpening, and edge detection. You can find a list of these well-known kernels here.

function applyConvolution(sourceImageData, outputImageData, kernel) {
  const src = sourceImageData.data;
  const dst = outputImageData.data;

  const srcWidth = sourceImageData.width;
  const srcHeight = sourceImageData.height;

  const side = Math.round(Math.sqrt(kernel.length));
  const halfSide = Math.floor(side / 2);

  // padding the output by the convolution kernel
  const w = srcWidth;
  const h = srcHeight;

  // iterating through the output image pixels
  for (let y = 0; y < h; y++) {
    for (let x = 0; x < w; x++) {
      let r = 0,
        g = 0,
        b = 0,
        a = 0;

      // calculating the weighed sum of the source image pixels that
      // fall under the convolution kernel
      for (let cy = 0; cy < side; cy++) {
        for (let cx = 0; cx < side; cx++) {
          const scy = y + cy - halfSide;
          const scx = x + cx - halfSide;

          if (scy >= 0 && scy < srcHeight && scx >= 0 && scx < srcWidth) {
            let srcOffset = (scy * srcWidth + scx) * 4;
            let wt = kernel[cy * side + cx];
            r += src[srcOffset] * wt;
            g += src[srcOffset + 1] * wt;
            b += src[srcOffset + 2] * wt;
            a += src[srcOffset + 3] * wt;
          }
        }
      }

      const dstOffset = (y * w + x) * 4;

      dst[dstOffset] = r;
      dst[dstOffset + 1] = g;
      dst[dstOffset + 2] = b;
      dst[dstOffset + 3] = a;
    }
  }

  return outputImageData;
}

This function expects the source ImageData object as a parameter. Similarly, you must provide it with a blank ImageData object with the same size as the first one to be used as output, and a matrix representing the kernel. Then, the convolution algorithm is applied on the outputImageData object by iterating on the sourceImageData one.

Et voilà! Now, you have the building blocks required to implement a filtering feature in Vanilla JavaScript. Let’s see how in detail.

3. Filtering in Action

Let’s take a look at how to implement a simple application allowing users to upload an image, choose the filter to apply, and see the final result :

<!DOCTYPE html>
<html>
  <body>
    <h1>Image Resizer</h1>
    <p>
      Please, upload an image and apply the filter that best suits the image
    </p>
    <input id="upload" type="file" accept="image/*" />
    <div id="images" style="visibility: hidden;">
      <h2>Original image</h2>
      <img id="imageToFilter" style="max-width: 100%; height: 200px;" />
      <div style="margin-top: 5px;">
        <span>Filter: </span>
        <input type="radio" name="filterRadio" value="noFilter" checked />
        <label>No Filter</label>
        <input type="radio" name="filterRadio" value="threshold" />
        <label>Threshold</label>
        <input type="radio" name="filterRadio" value="sharpen" />
        <label>Sharpen</label>
        <input type="radio" name="filterRadio" value="blur" />
        <label>Blur</label>
      </div>
      <h2>Filtered image</h2>
      <img id="filteredImage" style="max-width: 100%;" />
    </div>
    <script src="src/index.js"></script>
  </body>
</html>

// hiding the div that will contain the images
const imagesDiv = document.querySelector('#images');
const fileInput = document.querySelector('#upload');

let imageToFilter = null;
const filteredImage = document.querySelector('#filteredImage');

// initializing the filter value
const filterElement = document.getElementsByName('filterRadio');

let filter;
filterElement.forEach((f) => {
  if (f.checked) filter = f.value;
});

// applying the selected filter
filterElement.forEach((f) => {
  f.onclick = () => {
    filter = f.value;
    filteredImage.src = filterImage(imageToFilter, filter);
  };
});

fileInput.addEventListener('change', async (e) => {
  const [file] = fileInput.files;

  // displaying the uploaded image
  imageToFilter = document.querySelector('#imageToFilter');
  imageToFilter.src = await fileToDataUri(file);

  // making the div containing the image visible
  imagesDiv.style.visibility = 'visible';

  // applying the defaul filter
  imageToFilter.addEventListener('load', () => {
    filteredImage.src = filterImage(imageToFilter, filter);
  });
  return false;
});

function fileToDataUri(field) {
  return new Promise((resolve) => {
    const reader = new FileReader();
    reader.addEventListener('load', () => {
      resolve(reader.result);
    });
    reader.readAsDataURL(field);
  });
}

function filterImage(imageToFilter, filter) {
  const canvas = document.createElement('canvas');

  const context = canvas.getContext('2d');

  const canvasWidth = imageToFilter.width;
  const canvasHeight = imageToFilter.height;

  canvas.width = canvasWidth;
  canvas.height = canvasHeight;

  context.drawImage(imageToFilter, 0, 0, canvasWidth, canvasHeight);

  const sourceImageData = context.getImageData(0, 0, canvasWidth, canvasHeight);
  const blankOutputImageData = context.createImageData(
    canvasWidth,
    canvasHeight
  );
  const outputImageData = applyFilter(
    sourceImageData,
    blankOutputImageData,
    filter
  );

  context.putImageData(outputImageData, 0, 0);

  return canvas.toDataURL();
}

function applyFilter(sourceImageData, outputImageData, filter) {
  if (filter === 'noFilter') {
    return sourceImageData;
  } else if (filter === 'threshold') {
    return applyThreshold(sourceImageData);
  } else if (filter === 'sharpen') {
    return applyConvolution(
      sourceImageData,
      outputImageData,
      [0, -1, 0, -1, 5, -1, 0, -1, 0]
    );
  } else if (filter === 'blur') {
    return applyConvolution(sourceImageData, outputImageData, [
      1 / 16,
      2 / 16,
      1 / 16,
      2 / 16,
      4 / 16,
      2 / 16,
      1 / 16,
      2 / 16,
      1 / 16,
    ]);
  } else {
    // fallback option
    return sourceImageData;
  }
}

function applyThreshold(sourceImageData, threshold = 127) {
  const src = sourceImageData.data;

  for (let i = 0; i < src.length; i += 4) {
    const r = src[i];
    const g = src[i + 1];
    const b = src[i + 2];

    // thresholding the current value
    const v = 0.2126 * r + 0.7152 * g + 0.0722 * b >= threshold ? 255 : 0;
    src[i] = src[i + 1] = src[i + 2] = v;
  }

  return sourceImageData;
}

function applyConvolution(sourceImageData, outputImageData, kernel) {
  const src = sourceImageData.data;
  const dst = outputImageData.data;

  const srcWidth = sourceImageData.width;
  const srcHeight = sourceImageData.height;

  const side = Math.round(Math.sqrt(kernel.length));
  const halfSide = Math.floor(side / 2);

  // padding the output by the convolution kernel
  const w = srcWidth;
  const h = srcHeight;

  // iterating through the output image pixels
  for (let y = 0; y < h; y++) {
    for (let x = 0; x < w; x++) {
      let r = 0,
        g = 0,
        b = 0,
        a = 0;

      // calculating the weighed sum of the source image pixels that
      // fall under the convolution kernel
      for (let cy = 0; cy < side; cy++) {
        for (let cx = 0; cx < side; cx++) {
          const scy = y + cy - halfSide;
          const scx = x + cx - halfSide;

          if (scy >= 0 && scy < srcHeight && scx >= 0 && scx < srcWidth) {
            let srcOffset = (scy * srcWidth + scx) * 4;
            let wt = kernel[cy * side + cx];
            r += src[srcOffset] * wt;
            g += src[srcOffset + 1] * wt;
            b += src[srcOffset + 2] * wt;
            a += src[srcOffset + 3] * wt;
          }
        }
      }

      const dstOffset = (y * w + x) * 4;
      dst[dstOffset] = r;
      dst[dstOffset + 1] = g;
      dst[dstOffset + 2] = b;
      dst[dstOffset + 3] = a;
    }
  }
  return outputImageData;
}

As you can see, the HTML page contains everything required to upload an image, and select a filter. There is also a no filter option, allowing users to restore the original image. Also, to better appreciate the effects of each filter, the original image is always shown as well.

From a technical point of view, what happens is that the input element is used to upload an optional image. Then, the uploaded image is passed to the applyFilter() function, which takes care of applying the filterselected in the HTML radio buttons by calling the right function. This can involve either a simple filter, or a convolution-based one. In this latter case, the appropriate kernel is passed to the applyConvolution() function as well. Then, the output ImageData object is used to show the resulting image by using the putImageData() function. Please, note that the example that was just implemented corresponds to the live demo you can find at the beginning of the article.

Final Considerations

Implementing image filtering is not a simple task and can easily be implemented with Vanilla JavaScript. On the other hand, the more filters you want to add, the more complicated things can get. In this tutorial, you just saw how to implement non-complex filters, but there are filtering techniques that require much more effort. Plus, the majority of them involve iterating through each pixel. As a result, on large images, this can turn into a time-consuming process. So, if you do not want to ruin the user experience, you might be forced to adapt academic and well-known algorithms to make them more efficient. Thus, implementing this feature efficiently can turn into a considerably complex task.

If you do not want to deal with such unavoidable optimizations, a commercial and all-in-one solution like PhotoEditorSDK should be taken into account. This way, you will not have to worry about accomplishing challenging and time-consuming tasks anymore. Also, whenever you need help, you can ask for support from the img.ly developers who built the SDK.

Filter an Image with `photoeditorsdk`

First, you should read this article from the official documentation on how to get started with PhotoEditorSDK in HTML and JavaScript. Then, by using Filters feature, you can start enjoying more than 60 high-quality filters. Moreover, each of them was optimized natively to be lightning fast. This way, you can minimize the processing time, and achieve the following result with a few clicks:

Conclusion

In this article, we looked at how to implement a feature to allows users to apply a filter on an image with JavaScript. Implementing such a feature can be entirely achieved by only using the HTML5 <canvas> element. However, making the image processing fast enough not to make users wait can become a problem. This is the real issue when dealing with such a feature, which inevitably involves iterating through each pixel of which an image is composed. Consequently, optimizing well-known filtering algorithms to avoid this issue is what makes implementing the feature challenging. So, if you do not want to deal with this issue, you can adopt an easy, complete, advanced, and all-in-one solution – such as PhotoEditorSDK.

Thanks for reading! We hope that you found this article helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

Inpainting: Removing Distracting Objects in High-Resolution Images

Vivien — Tue, 08 Dec 2020 15:03:28 GMT

Introduction

You may know this situation: You are out on a trip when suddenly a unique opportunity for a photograph appears, like a wild animal showing up or sun rays breaking through the rain clouds for a few seconds. Without hesitation, you grab your camera and capture the sight. Later you discover that a distracting object, like a road sign, is ruining your shot. Time for some cumbersome retouching.

Now, imagine you could erase the distracting object just by highlighting it. Wonderful! From the field of deep learning, a technique for image manipulation called Image Inpainting makes it possible. Image Inpainting aims to cut out undesired parts of an image and fills up missing information with plausible content of patterns, colors, and textures that match the surrounding.

Today we would like to share experiences that we have gained during the application of deep learning inpainting approaches. Furthermore, we’ll present some quality optimization steps that we have implemented to improve results addressing the transformation to high-resolution outputs. But let us start with a quick introduction: who are we and why are we concerned with these kinds of topics?

We are a small consortium consisting of the Bochumer Institute of Technology, a research institute aiming to transfer knowledge from academia into industry, and the company IMG.LY, a team of software engineers and designers developing creative tools like the PhotoEditor SDK and the UBQ engine. Together we are working in the EFRE.NRW funded research project KI Design that targets artificial intelligence (AI) and deep learning-based algorithms for image content analysis and modification, as well as a leveraging tool kit for aesthetic improvements.

Image Inpainting has been a viable technique in image processing for quite some time, even before “Artificial Intelligence” was on everyone’s lips. Common for most inpainting algorithms is that an area of an image is highlighted to be corrected. Many conventional algorithms then analyze the statistical distribution to fill the resulting gap by finding and using nearest neighbor patches. The most famous and state of the art approach of this method is the PatchMatch algorithm. It uses a fast, structured randomized search to identify the approximate nearest neighbor patches that will fill in the respective part of the image.

However, there are two drawbacks: first, regardless of the approximation, performance still might be an issue, and second, the results suffer from a lack of semantic understanding of the scene. Thus, research dived into new ideas and directions and tried the application and implementation of AI- and neural network-based approaches to solving these issues.‌‌‌‌ For us, the removal of annoying background content is a useful feature, as it improves the overall image aesthetic. Having this available on mobile devices would be particularly interesting. Due to performance limitations and ever-improving integrated cameras, a mobile solution requires a fast and lightweight model architecture as well as the ability to process high-resolution images.

Summarized, our expectation for an AI-based inpainting algorithm are:

removal of (manually highlighted) background objects/persons
feasibility to process high-resolution images
fast and lightweight network (applicable for smartphones)

The number of publications addressing this or similar requirements has increased enormously in recent years. After digging into the literature, we identified two promising approaches and tested them.

Testing and Comparing Model Architectures

These selected networks were based on the latest scientific findings and appeared to provide high-quality output. Both approaches have well-documented repositories – a special thank you to the authors for their great work (of repositories and papers as well)! The selected networks are:

Partial Convolutional Neural Networks (PCONV); [paper, github-repository]
Generative Multi-column Convolutional Neural Networks (GMCNN); [paper, github-repository]

You may be wondering why exactly we chose these models for comparison purposes, as the latest scientific findings sound a bit vague. Indeed, it is considerably difficult to identify the best fitting model architecture. As far as we know, there is no standardized validation method or data set. Most papers demonstrate their results on self-selected test images and further compare them with again self-selected approaches. The only option we had was to evaluate models that seemed reasonable to us. A validation method or standardized test set could be a valuable scientific contribution here. Let’s turn back to the selected models.‌‌‌‌ The PCONV network uses multiple convolutional layers and adds a partial convolutional layer. The key feature is that the convolution does not consider invalid pixels, indicated by an updating mask. This prevents the algorithm from picking up the color of the mask (typically the average color tone of the image) and transmit it into the reconstruction process.

The GMCNN – a GAN-based model – is built in a special architecture consisting of 3 networks: a generator, split up into three branches addressing different feature levels, a local and global discriminator, a VGG19 net calculating the implicit diversified Markov random field (ID-MRF), introduced in the paper. This ID-MRF serves as a loss term comparing generated content with nearest-neighbor patches of the ground truth image. While the interaction of all three networks is required in the training phase, only the generative network serves for testing and production. More details and figures regarding the model architecture are available in the official paper.

Due to the lack of standardized sets, we created our own test sets addressing different levels of complexity. This also included image data requiring an understanding of semantic structures. In our comparison, we paid special attention to ensuring the filled content was harmonious, and a possible artifact interspersion was reduced to a minimum. In particular, image artifacts could raise issues in terms of translation with respect to high-resolution information. Here is an example output of our tests:

In comparison, the inpainting result based on PCONV suffered from some blurred artifacts and erratically deviating shades, cf. Figure 1C, whereas the GMCNN-based result appeared to be more precise and plausible concerning the semantic context, cf. Figure 1D. You can see this clearly when you look at the grille door that was covered by a person. The GMCNN approach, cf Figure 1D, had recognized and respected the grid structure, while the PCONV overlayed this with a uniform (black) color tone. In consideration of all test data results, we decided to follow up with the GMCNN.

However, we would like to emphasize that this does not mean that one model architecture is better suited for Image Inpainting than the other. The used weights build-up of the PCONV architecture may achieve similar results with further training or different test sets.

What About High-Resolution Inpainting?

At the current state of the model, the processing of high-resolution images remains uncovered. Out of all the papers and repositories we found, even papers promising high-resolution often just targeted image sizes of 1024x1024 pixel at maximum. Our expectations were a resolution of substantially more than 2000x2000 pixels. A reason for this issue seemed to be the hardware demanding and time-consuming training phase when processing high-resolution images. ‌‌
Furthermore, the application of a high-resolution inpainting model could entail performance issues. These are not neglectable to us, as we are facing a prospective implementation on smartphones that can’t keep up with the power of a modern graphics card. Thus, an additional challenge is a high-quality transformation of low-resolution outputs to a high-resolution.

Apply Low-Resolution Inpainting Output to High-Resolution Images

‌‌The GMCNN model was trained with the Places dataset, formatted in a 512x680 resolution. Feeding in high-resolution images would exceed the training input size by far and further require information of feature dimensions that the model has never seen before. That could result in almost completely distorted reconstructions.

A straightforward solution is to downscale the high-resolution image before feeding it to the model and then resize the result up to the original image size conclusively. Due to the upscaling (e.g., via bicubic interpolation), the image details suffer from a loss of quality. Therefore a better approach is to take only the masked areas of the upscaled inpainting prediction and stitch it back into the original image. That prevents the loss of initially known details from the unmasked regions. For the maintained inpainting regions, the lack of image details, as well as the artifacts and distortions, pose a complex challenge that we aimed to overcome with the following approaches.‌‌‌‌

Shrinking-Mask-Approach

While the inpainted area mostly yields realistic-looking content for the more marginal regions, the performance decreases strongly towards the center, cf. Figure 3D. We especially noticed this behavior for larger masks. Conclusively a recursive inpainting procedure with an iteratively shrinking mask, cf. Figure 3E, seems to be a reasonable approach. With this concept, we try to improve the inpainting results in a progressive manner starting from the boundary to the center of the masked regions while utilizing the generated information of the preceding recursion, cf. Figure 3F.‌‌

To us, it was essential to have a dynamic method that allows the handling of all mask forms and sizes. Therefore, we decided to apply an erosion kernel to the original mask in a recursive fashion until it is fully eroded. The amount of shrunk masks determines the number of inpainting performed by the network.‌‌‌‌

Two-Step-Approach

While investigating and testing various quality optimization steps, we also fed high-resolution images into our model and discovered that the results for smaller masks were convincing. That led us to the hypothesis that not the resolution but rather the number of pixels to reconstruct seems to be the limiting factor. This finding served as the basis for our two-step-approach.‌‌‌‌

Briefly, the approach works as follows. In the first step, we perform inpainting on a downscaled high-resolution image while applying the original mask. In a second step, we transfer the model output of step one into a higher resolution and perform inpainting again. This time we apply a modified mask containing only small coherent mask regions, for which we exploit the provided higher resolution context information. ‌‌
In more detail, the first step is characterized as the baseline approach, cf. Figure 5: We scale the masked image down to the training resolution of 512x680 pixels and fill up the missing information.

Optionally, the shrinking-mask-approach can be applied in the first step.

In the second step, we quadruplicate the output of step 1 to a resolution of 1024x1360 pixels. To prevent the resolution loss for unmasked regions caused by this upscaling, we stitch the generated content into the same sized (downscaled) input image. The resulting image serves as the model input for step 2. ‌‌

To avoid/reduce image artifacts in the subsequent inpainting process, we modify the original mask to contain only the small mask regions and the boundaries of the large mask regions. In detail, we temporarily shrink the mask with an erosion kernel to ablate small mask segments and the marginal areas of larger mask sections, cf. Figure 6B. Finally, we calculate the difference between the original mask and the altered mask, resulting in our desired modified mask, cf. Figure 6C.

By re-inpainting, we double the resolution of the generated content for the small contiguous mask regions, cf. Figure 7A bottom right, as well as for the masked boundary areas, cf. Figure 7A upper left. Moreover, through the latter, we achieve smoothing of the intense decay in resolution between the unmasked regions and the generated content arising from step 1. Finally, we scale our image back to the original input resolution and stitch the generated content to the original image to maintain the original resolution for unmasked areas.

Conclusion

For us, it was impressive to see how AI-based inpainting can successfully and deceptively realistic fill in missing information. Not only the consideration of structural (semantic) content is an advantage compared to conventional approaches, but especially the decreased demand on required hardware. In our view, this opens up the opportunity to reach a much larger group of users of inpainting algorithms: in place of using powerful hardware and professional software, mobile devices could achieve small but decisive changes.

In summary, we have dealt with the application of high-resolution images, which is undoubtedly gaining in importance due to the ever-improving smartphone cameras. Processing high-resolution images entail an increasing number of pixels to “inpaint” and could further lead to quality as well as performance issues. Thus, we decided to improve the output of low-resolution networks and to provide them with more information to support a subsequent upscaling procedure.
We have implemented two different approaches, shrinking-mask and two-step-approach that can be applied independently or in a combined manner. It turned out that both methods subjectively increased the image quality. However, this comes along with higher computational demands, as models are applied multiple times.
Overall, we think that the combination of these two approaches will represent a good toolkit for AI-based high-resolution image inpainting. But we’ll keep an eye on the upcoming scientific developments.

A Photo and Video Editor for React Native Apps

Julia — Fri, 29 May 2020 10:30:01 GMT

How to integrate a Photo and Video Editor into your React Native App

This tutorial walks you through the integration process of PhotoEditor SDK and VideoEditor SDK into your React Native app for iOS and Android. You’ll learn how to use our React Native modules to facilitate the integration and to customize our editors. For this tutorial we presume that all the necessary development tools for building an iOS and Android app are met, so make sure to complete the official React Native CLI Quickstart guides for iOS and Android beforehand.

Please make sure to acquire the licenses for PhotoEditor SDK and VideoEditor SDK before integrating them.

https://www.youtube.com/embed/e9JiCMQKrJY?feature=oembed

In this tutorial, we’re going to show how to integrate PhotoEditor SDK and VideoEditor SDK for iOS and Android into your React Native app. Therefore, we created React Native modules for our products to simplify this process for you as much as possible. We’re going to use VideoEditor SDK’s README, which is in most parts identical to the PhotoEditor SDK’s README, and Visual Studio Code. So, let’s get started.

First, we create a React Native project with the name “Demo” based on the default template by using the command npx react-native init Demo. The project will now be initialized and automatically install all dependencies of the current React Native version. Afterwards, we can find the new React Native project ready to use in the folder “Demo”. So, we’ll speed this up a little.

We already prepared another folder with resources and assets that we want to integrate in our app. Here we chose an image, the required licenses for our PhotoEditor SDK and VideoEditor SDK for both target platforms, a video and two logos that we will later use to show how we can customize our editors. We copy these resources into the root of our project to make the resources accessible for our app.

Now, we switch to the folder of the “Demo” project*.* We can now copy and execute the command yarn add react-native-videoeditorsdk from the README to install the dependencies to the React Native module for our VideoEditor SDK … and to the React Native module for PhotoEditor SDK by issuing the command yarn add react-native-photoeditorsdk.

Now, we’re going to set up the dependencies for our native iOS libraries. We can simply copy the command cd ios && pod install && cd .. from the README and execute it to install all iOS dependencies. They include the native PhotoEditor SDK and VideoEditor SDK libraries that are required by our React Native modules.

And now, we set up the dependencies for our native Android libraries. The required steps that we will now take are described in detail in the README.

android {
  defaultConfig {
    multiDexEnabled true
  }
}

dependencies {
  implementation ‘androidx.multidex:multidex:2.0.1’
}

We copy the lines and add them at the end of our android/app/build.gradle file. Now we need to change the superclass of our MainApplication class to enable Multidex. Next, we add the img.ly repository and the plugin by copying the following lines and add them at the top of our android/build.gradle file located in our Android folder.

buildscript {
  repositories {
    jcenter()
    maven { url "https://plugins.gradle.org/m2/" }
    maven { url "https://artifactory.img.ly/artifactory/imgly" }
  }

  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.61"
    classpath 'ly.img.android.sdk:plugin:7.1.8'
  }
}

Now, we can configure our PhotoEditor SDK and VideoEditor SDK by opening the android/app/build.gradle file and add these lines under apply plugin: “com.android.application".

apply plugin: 'ly.img.android.sdk'
apply plugin: 'kotlin-android'

// Comment out the modules you don't need, to save size.
imglyConfig {
  modules {
    include 'ui:text'
    include 'ui:focus'
    include 'ui:frame'
    include 'ui:brush'
    include 'ui:filter'
    include 'ui:sticker'
    include 'ui:overlay'
    include 'ui:transform'
    include 'ui:adjustment'
    include 'ui:text-design'
    include 'ui:video-trim'

    // This module is big, remove the serializer if you don't need that feature.
    include 'backend:serializer'

    // Remove the asset packs you don't need, these are also big in size.
    include 'assets:font-basic'
    include 'assets:frame-basic'
    include 'assets:filter-basic'
    include 'assets:overlay-basic'
    include 'assets:sticker-shapes'
    include 'assets:sticker-emoticons'
  }
}

Getting back to our iOS version, we can now launch our demo project on iOS, which will currently look like a plain React Native project that we initialized with the first command.

The main difference to an off-the-shelf React Native project is that our React Native modules are installed and ready-to-use in the App.js file once the native projects are compiled. Then, it won’t be necessary to recompile the native projects for the remainder of this tutorial. We sped up the compilation a little and here we go — our React Native app is running on the iOS simulator. Now, we do the same for the Android version and wait until the project is compiled.

Now, the demo project launched on both platforms as we can see on the right on the iOS simulator at the top, and on the Android emulator at the bottom of the screen.

We’ve decided that we want to start our photo editor by pressing a button. So next, we’re going to actually customize our React Native app by adding this button. Therefore, we open the App.js file and import the Button component in order to create a button with the title “Edit a sample image”. For now, we leave the onPress function empty.

We save the App.js file to trigger a refresh of the running apps and immediately see the result on the right. The new button appears in both, the iOS and the Android app.

Now we create a second button with the title “Edit a sample video”. This will respectively start our video editor. And again, we save the App.js file and see the second button appear on the right side.

Next, we are going to add the code that actually opens our editors when we press the buttons. Visual Studio Code automatically imported the respective React Native PhotoEditor SDK module for us at the very top of the file, while writing the code that makes use of our SDK. We do the same for the VideoEditor SDK. We use the require function to make static assets available to our app. Here, we “require” our sample image and our sample video that we copied to the app’s folder in the beginning and pass them as the first argument to our openEditor functions. The first argument can also be a regular URI.

We save the App.js file again and now we can click the buttons to start our photo editor or video editor. There we go! We still see a watermark here. The reason for this watermark is that we haven’t unlocked our SDKs so far which we will do next.

We unlock both products with our licenses to get rid of the watermark. If not unlocked, the watermark will be on both the image and video previews as well as on the exported images and videos. To unlock the products, we use the unlockWithLicense function of each SDK. In total, we need four license files, one license file for each product and platform combination. The license files should be named pesdk_license and vesdk_license with platform-specific extensions .ios.json and .android.json. React Native will then automatically pick the right file for the corresponding platform. After this, the watermarks will be removed for PhotoEditor SDK and VideoEditor SDK on both platforms. And now you can also see it in the simulator — no watermarks anymore.

In the next step, we’re going to change the configuration of the editors. If no changes are made to the configuration, our default stickers are available with the editor. To customize them, we need to import the Configuration from either the PhotoEditor SDK or VideoEditor SDK. The configurations are compatible between both products. So, for this tutorial, we decided to use the VideoEditor SDK configuration which we are now using by adding Configuration to the react-native-videoeditorsdk imports. We decided that we want to add custom stickers to our editors. Therefore we define a non-default configuration to the sticker tool.

const configuration: Configuration = {
  sticker: {}
}

To customize the sticker assets, we need to define the sticker “categories” array. Here, we define a new category and name the identifier demo_sticker_category. These asset identifiers must always be unique. Next, we set a name for the category and we name it “Logos”.

Each category also requires a thumbnail image to be displayed in the editor. For the thumbnail, we use the React logo that we added to the folder of our app at the very beginning. Next, we define the items for this new sticker category. These items are the actual stickers that we can apply to the edited image or video. We now create a new sticker for the React logo. Therefore, we call the identifier demo_sticker_react and name it “React”. These sticker names won’t appear in the UI, but they are used for accessibility. Now, we need to define the actual image that should be used for that sticker. Here we use the React image again.

To create a second sticker, we can now copy and paste the code of the first sticker. We create a sticker with our img.ly logo and rename the identifier of the pasted code to demo_sticker_imgly. Accordingly, we set the name to “img.ly” and change the file to imgly.png.

In addition, we want to specify a non-default tint mode for our second sticker by using tintMode: TintMode.SOLID which enables us to change the color of the sticker. The TintMode type is automatically added to the VideoEditor SDK imports for us by Visual Studio Code. Now, that we completed our configuration, we need to pass it as the second argument to the openEditor functions in order to take effect.

We save the App.js file again to refresh the running apps and we can see the result live after starting a new editing session. Please note that you cannot alter the configuration of a running editor instance. You always need to start a new editing session to see configuration changes.

We want to use another feature of our SDKs which is called serialization. With the serialization feature, we can capture all image and video editing operations that are applied in the editor and export them. This allows us to import the editing operations in later sessions and continue editing. The serializations are compatible between both products as well. The input serialization is the third parameter of the openEditor functions of our SDKs and the output serialization is optionally part of their result type.

First, we check if the result is “null”. This is the case when a user clicks the “Discard” button in the editor and thus does not export an image or video. If the result is not ”null”, we know that the user exported an image or video. Then we can assign the exported serialization to the previously defined global serialization variable which will then be input to the next editing session. We copy the code and add it to the video editor as well to enable the serialization function here too.

Now, one thing is left to enable the actual serialization export in the configuration. The serialization export is disabled per default because not every user needs the serialization feature. Here, we enable it now and also change the export type to object. By doing so, the result type of the editor will contain the serialization as an object. Per default, the serialization is exported to a file and that file name is returned as part of the export result. Writing the serialization to a file is a reasonable default as serializations can be quite large, especially if large amounts of binary data for personal stickers are embedded.

Now, we can run the app on the simulator and use all the parameters that we configured in this tutorial. First, we can add our custom stickers, both the React logo and the img.ly logo. Here we can also change the colors which we enabled with the tint mode.

We can also use the text design tool to add a phrase to our image. Here we can pick different designs, so we’re trying a couple and place the text design fitting to the image and logo.

Next, we export our image with the serialization. With the serialization function enabled, it is now possible to import the editing operations into our video editor. This allows us to keep on editing because the serialization is compatible between both products. So here we can add further words to our text design. We can also put filters on our video. For example, we can choose the peach duo tone and increase the contrast a little. And here we go! We successfully integrated PhotoEditor SDK and VideoEditor SDK into our React Native app.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Why our SDK is not Photoshop or Lego.

Magnus — Sun, 23 Feb 2020 23:00:00 GMT

PhotoEditor SDK (Software Development Kit) or more general white-label editor solution might sound like big words. And of course, we’d love to tell you that we are a jack of all trades. However, while our standard is to provide you with the most sophisticated editor solution possible, there are challenges to be faced on that path. Now, while I’d be more than happy to talk about all the reasons in favor of using our SDKs, I’m convinced you’ll get the clearest image of whether PhotoEditor SDK is the right solution for you when I tell you what we can’t do. So, let’s jump right in.

We are no Photoshop

First of all, it has to be said that image editing is a quite flexible concept. It relates to everything that alters the appearance or content of an image. From applying a black & white filter and adding text to make your Instagram-stories more epic, over cutting out undesired aspects of an image like an ex-boyfriend or altering ratios up to the point where the image does not display what was originally captured, for sure, some Tinder profile pictures.

Now, tools like Photoshop are built for professionals to edit images and polish them in hours of detail work. In other words: it’s an extremely powerful solution you can do ANYTHING with. And while this is its main appeal, its complexity is also why non-designers who don’t want to spend a lot of time learning how to use it — simply don’t.

However, that isn’t to say that these users don’t have a need to edit images for different purposes. This is where we come into play, making image editing and design accessible for everyone at the tap of a finger. While our PhotoEditor SDK doesn’t do everything, it delivers the set of tools your users really need and want to see in your app — no matter if it’s a social network, a printing service or a marketing tool — at a very high quality.

We are no Lego either

I guess many of us remember the joy of playing with Lego when we were little toddlers. No matter how crazy the idea, we put it together brick by brick eventually ending up with a quite colorful but yet solid plane, car or spaceship — thanks to Lego’s incredible modularity.

Can you approach using PhotoEditor SDK in the same way? No.
Is that a bad thing? No, again — actually quite the opposite. While freedom of design is often viewed as one of the greatest goods to strive for, it adds tons of complexity when building applications. However, right at the top of the must-have list is a smooth and intuitive UX. Hence, not every part of the editor can just be re-attached to any other place within it. Our designers created a well thought out UI that makes operating the editor simple and enjoyable.

You can think of it as a Lego building instruction. Sure, you could try to assemble that recently bought Lego product without it, by trial and error. However, as you didn’t purchase just a handful of randomly selected bricks, I’m convinced you’d love to successfully complete that flashy Millennium Falcon to proudly place it somewhere in your living room. Therefore, following the steps of the enclosed booklet is similar to editing your way through our well thought out UI.

Also, back then Lego seemed compatible with almost every other toy we owned, or at least, we didn’t mind mixing them up. However, when it comes to coding, not every brick attaches to any other brick per default. For instance, there used to be only rudimentary demos for the integration of our SDKs with a handful of frameworks available. Thus, making the standard integration as explained in our docs the best way for the implementation. That is, until now.

With our React Native wrapper, we have released the first in a row of framework integrations for PhotoEditor SDK. Feel free to check it out here. In addition, you should read the corresponding blog article my colleague, Alex, published a few days ago. The good news — our development team doesn’t stop there and will add more wrappers during the coming months. So, stay tuned.

If you’re interested in implementing the editor into your app or website using a framework, you should let us know which framework you’re interested in.

So, what exactly can you expect from us?

In essence: an editor solution that provides you with…

**·** a well-balanced feature selection serving a wide variety of use cases,

**·** a degree of customizability ensuring that no one will ever notice that your editor is a third-party library, and

**·** ongoing maintenance and support at an easy to calculate price over time.

To sum things up, ideally, you already know what you expect from a third-party editor solution. So, instead of telling you that we are the best choice for your use-case, I’d like you to be the judge of that.

We are happy to discuss any and every question you might have. Also, as qualitative feedback is always appreciated, we’d love to talk to you. Don’t hesitate to reach out.

Until then, cheers and keep it up
Magnus

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

Image Editing – IMG.LY Blog

IMG.LY Partners with Getty Images to Revolutionize Image Integration

Empower your Developers and Users

Get Started

How to Draw on Images in React Native

Canvas Integration for Images

Drawing with Canvas

Drawbacks of the Canvas Method

PhotoEditor SDK Integration for Drawing on Your Images

Conclusion

Comparing the Top 5 Open Source JavaScript Image Manipulation Libraries

Sharp

Use cases

Example

Use Cases

Example

Merge Images

Use cases

Example

LooksSame

Use cases

Example

Jimp

Use cases

Example

Conclusion

Content Moderation for User-Generated Designs with CE.SDK and Sightengine

Why You Should Moderate User-Generated Content

What is CE.SDK?

What is Sightengine?

Integrate Content Moderation into CE.SDK

1. Get Started with CE.SDK

2. Build a Content Moderation API with Sightengine

2.1 Getting Started with Sightengine

2.2 Prerequisites

2.3 Setting up a Node.js Express Server

2.4 Implementing a Content Moderation API in your Node.js Server

3. Moderate Images Made with CE.SDK

3.1 Prerequisites

3.2 Getting Started

3.3 Integrate the Moderation Check into your Design Process

Conclusion

How To Add Overlays to a Video in React Native

What is React Native Video?

Prerequisites

Adding Overlay Image and Text to a Video in React Native

1. Initializing a React Native Project

2. Defining a Component to Add Overlays on a Video

3. Putting It All Together

Final Considerations

Add Overlays to a Video With React Native module for VideoEditor SDK

Conclusion

How To Add a Sticker to a Texture With Three.js

Prerequisites

What is Three.js?

Adding an Overlay Image to a Texture in WebGL

Adding a Sticker With PHOTOEDITORSDK

Conclusion

How To Scale, Crop, Flip, and Filter an Image in CSS

How CSS Image Manipulation Work

Prerequisites

Image Processing in CSS

Scaling an image with object-fit: scale-down

Cropping an image with object-fit and object-position

Flipping an image with transform

Filtering an image with filter

CSS-Only Image Editor vs PhotoEditor SDK

Conclusion

To stay in the loop with our latest articles and case studies, subscribe to our Newsletter.

How To Manipulate an Image With Jimp in React

What is Jimp?

Prerequisites

Basic Image Processing in Jimp

Cropping an image

Resizing an image

Rotating an image

Filtering an image

Jimp vs PhotoEditor SDK

Conclusion

CE.SDK v1.3.0 Release

Adding a Sticker With `PHOTOEDITORSDK`

Scaling an image with `object-fit: scale-down`

Cropping an image with `object-fit` and `object-position`

Flipping an image with `transform`

Filtering an image with `filter`

Custom Filters With `<canvas>`

Filter an Image with `photoeditorsdk`