Mark – IMG.LY Blog

Comparing the Top 5 Open Source JavaScript Image Manipulation Libraries

Mark — Tue, 27 Sep 2022 13:29:03 GMT

Image Manipulation Libraries (IML) are used to perform various manipulation functions on images: you can increase the brightness of an image, add saturation or filters, crop and resize, and more valuable features that will help you to do almost everything and turn your web browser into an Adobe Lightroom!
We have included libraries for every significant image manipulation task, from simple, low-level operations such as filters, brightness, and saturation to single-purpose libraries for cropping or image composition. To finish up, we wandered into the realm of testing images, since no image manipulation application is complete without a reliable way to test results.

Sharp

Sharp is a high-performance Node.js image processing library to resize different image formats such as JPEG, PNG, WebP, AVIF, SVG, and TIFF. The typical use case for this high-speed Node.js module is to convert large images in standard formats to smaller, web-friendly images.

Sharp is helpful only if you want to resize a giant file or a variety of them. If, on the other hand, you only want to resize a single small image, then you probably shouldn’t use it. Instead, ordinary HTML and vanilla JavaScript will be more beneficial. Sharp is taking full advantage of multiple CPU cores and L1/L2/L3 cache, allowing you to resize and compress your images much more quickly.

Use cases

I used it to resize a whole NFT Collection with a size of over 80Gb, which after compression came out at circa 10Gb. So again, if you would like to use it for resizing several large images, then it’s the best choice for you - you’ll do it in the fastest and most efficient way.

Example

You can install this library by using the command below:

npm install sharp

This is how you would resize an image using sharp:

const sharp = require('sharp');
const fs = require('fs');

sharp('yellow.png')
  .rotate(180)
  .resize(200)
  .toBuffer()
  .then((data) => {
    fs.writeFileSync('yellow.png', data);
  })
  .catch((err) => {
    console.log(err);
  });

Cropper.js

Cropper.js is another popular JavaScript library for image manipulation. You can use it to crop your images in all possible ways, change aspect ratio, rotate, zoom and work with canvas data. Cropper.js is the right choice for cropping without any extra features.

Use Cases

You can use its flexible API to create a custom image cropping UI in your web app allowing your users to adjust photos to the correct size and aspect ratio. It will be more efficient because it requires almost nothing, doesn’t have any useless features, and is very optimized for cropping!

Example

This library can be installed with the following command:

npm install cropperjs

import Cropper from 'cropperjs';

const image = document.getElementById('image');
const cropper = new Cropper(image, {
  aspectRatio: 16 / 9,
  crop(event) {
    console.log(event.detail.x);
    console.log(event.detail.y);
    console.log(event.detail.width);
    console.log(event.detail.height);
    console.log(event.detail.rotate);
    console.log(event.detail.scaleX);
    console.log(event.detail.scaleY);
  },
});

Check out the Cropper.js demo, click on any property that you want to apply to your image and see the result instantly.

For an in-depth guide on how to add image cropping to your React app, check out our guide on using react-image-crop.

Merge Images

Working with the canvas can be slightly tedious, especially when you need the canvas context to do relatively simple things like merging a few images. Merge Images abstracts all repetitive tasks into simple function calls. It’s a wrapper around the Canvas API, abstracting its low-level functions, which makes this specific task a lot easier. You can also create parameters like positioning, opacity, and several more. You can find them in the docs on GitHub.

Use cases

It’s a valuable library that can help with several tasks. For example, you can generate an NFT Collection where you can merge all components to have a complete set of variations. Or you can find this library helpful for personal use, for example, combining a few different images to generate a collage.

Example

Install the library itself:

npm install --save merge-images

Then you can use this code to generate a simple image:

//write this inside of your javascript file

import mergeImages from 'merge-images';

mergeImages(['/body.png', '/eyes.png', '/mouth.png']).then(
  (b64) => (document.querySelector('img').src = b64)
);

//And that would update the img element to show this image:

Here, I used it to merge several components to create a final variation (here I used Background, Character, Horns, Emotion, and Accessories).

LooksSame

LooksSame is a library for comparing images. If the two images you uploaded are duplicates, the library will let you know. All you have to do is provide a link to the pictures you would like compared. LooksSame is not strictly an image manipulation library, but is helpful for testing purposes.

Use cases

You can use this library and write code for automation and loop cycles that search a directory and compare each pair of your images for duplicates. Additionally, you can delete these duplicates and automate this process as well. It is a splendid library if you’re constantly working with ML algorithms that include large numbers of images.

Example

To start comparing images, you need just to install it by using this command:

npm i looks-same

Here, you can see how to test this library with Jest:

var looksSame = require('looks-same');

//Parameters can be paths to files or buffer with compressed png image

test('image1 and image2 are the same', async () => {
  expect(looksSame('image1.png', 'image2.png')).toBe(true);
});

//Result will be "image1 and image2 are the same" if they are the same

Jimp

Jimp stands for JavaScript Image Manipulation Program, that allows you to edit your image in almost every possible way. With it, you can invert your image, add some text, resize, use pixelation, clone images, blur them, invert colors, and several other cool features which will increase the image manipulation capabilities of your app.

Use cases

With Jimp, you can build your web application where you’ll be able to edit and manipulate images in almost any possible way. Crop, resize, rotate, and filter features let you create your own photo editor and add an interface on top of it. To streamline this process, you can try PhotoEditor SDK (PE.SDK). PE.SDK will provide you with a polished user interface and photo editing features, so you can focus on the application you are building instead.

Example

Here’s the JavaScript code to try this excellent library. There’s an async function in which you can see many types of properties that you can easily change. Such as Add Text, Resize Image, Blur the Image, etc. To use at least one of them, you’ll need to uncomment only part that you need, and your program is ready to run!

// Import dependencies
const Jimp = require('jimp');

(async function () {
  // Read the image
  const image = await Jimp.read('images/shapes.png'); // <http://www.example.com/path/to/lenna.jpg>

  // // Add text
  //  const font = await Jimp.loadFont(Jimp.FONT_SANS_16_WHITE); // bitmap fonts
  //  image.print(font, 0, 0, 'Hello world!'); // <https://github.com/libgdx/libgdx/wiki/Hiero>

  //  // Resize the image
  //  // Resize the image to 250 x 250
  //  image.resize(250, 250);

  //  // Resize the height to 250 and scale the width accordingly
  //  image.resize(Jimp.AUTO, 250);

  //  // Resize the width to 250 and scale the height accordingly
  //  image.resize(250, Jimp.AUTO);

  //  // Add a sepia wash to the image
  //  image.sepia();

  //  // Pixelation
  //  image.pixelate(5);
  //  image.pixelate(5, 50, 50, 190, 200); pixe,x, y, w, h

  //  // Clone
  //  const image2 = image.clone();

  //  // Blur the image
  //  image.gaussian(1);
  //  image.blur(1);

  //  // Inverts the image
  //  image.invert();

  //  // Set the brightness
  //  image.brightness( 0.5 ); // -1 to +1

  //  // Resize the image
  //  image.resize(256, 256);

  //  // Set the quality
  //  image.quality(100);

  //  // Convert to grayscale
  //  image.greyscale();

  // Save the image
  image.write('images/edited-shapes.png'); // writeAsync
})();

PS: Don’t forget to uncomment the effects which you want to use and save the file after. :)

As you can see here, I blurred everything and inverted the colors for the previous image that I showed you, getting the following result:

Conclusion

That’s it! Now you know the differences between five of the most popular image manipulation libraries for JavaScript. Some of them are useful for only one task, such as resizing images - others can do multiple tasks at once and help you with more processes. You have also met PE.SDK to work smarter and save time building photo editing solutions, and focus on building your great product instead.

How to Stream Videos Using Javascript and HTML5

Mark — Mon, 29 Aug 2022 20:36:00 GMT

If you would tell someone from the early 2000s what you’re about to do, they would be amazed and would give everything to understand how you did it. Because back then HTML was much simpler and didn’t even support video playback. So to stream your video inside of your browser you were forced to use some third-party services like Flash or Silverlight. Now we have HTML5 and new versions of JavaScript, and by combining these two new technologies you’ll end up with results that rival native applications. Streaming video inside of your web application nowadays is a breeze.

The `<video>` tag

You can easily start streaming your own video by using only one tag in HTML5. I’m talking about the <video> tag. It embeds a fully-fledged media player in the document and similar to the img tag accepts a path to the media we want to play inside the src attribute as well as accepting parameters for width and height.

In addition, there are video-specific properties such as whether the video should loop or autoplay and whether to display the default browser controls.

To handle the case of browsers not supporting the video tag you can provide fallback content inside the <video></video> tags.

Combining with JavaScript

You can also combine this simple video streaming in HTML with JavaScript and make it more professional and be able to control your streaming manually. When you get an element from the DOM of a video tag, the object that you get exposes various methods and attributes which can be used for manipulating media content. Firstly, we need to access the DOM and declare an object which will be used for our manipulations. Now we can use myVideo to programmatically pause, change the playback rate and current time of the video.

const myVideo = document.getElementById('#video1');

//pause the video
myVideo.pause();

//Change the speed of playback
myVideo.playbackRate = 3.0;

//Change current time
myVideo.currentTime = 4;

However, most videos that we see on the web today such as YouTube, Twitch, or any other social media display much more complex behaviors than just changing playback speed or pausing the video. On YouTube, you can change the quality of your video, add subtitles, or even add features like autoplay if you have several other videos queued up and don’t want to switch it manually.

All those services actually still use a video tag. But instead of streaming a video file through the “src” attribute inside of your tag, they use powerful tools such as Media Source Extensions or other adaptive streaming API that will help stream your video more efficiently.

What are Media Source Extensions (MSE)

Media Source Extensions is a W3C specification that allows JavaScript to send streams to media codecs within Web browsers that support HTML5 video and audio that most browsers implement today. In other words, it allows JavaScript to generate streams and facilitate a variety of use cases like adaptive streaming and time-shifting live streams. It’s quite an advanced solution if you want to make your video playback more customizable and professional. MSE gives us finer grained control over how much and how often content is fetched, and some control over memory usage details, such as when buffers are evicted.

Wrap your resource via the Media Source Extensions API instead of dealing with the URL directly (We’ll talk more closely about it a little bit later).

This specification was designed with the following goals:

Let JavaScript create a media stream regardless of how the media is fetched.
Leverage the browser cache as much as possible.
Provide requirements for byte stream format specifications.
Minimize the need for media parsing in JavaScript.

As I mentioned, we are still using HTML5 video tags, and even more surprising is that we’ll still use the “src” attribute. Only this time, instead of adding a link to the video, add a link to the MediaSource object.

If you’re a little bit confused, then don’t worry, I’ll explain everything. To enable this type of use case, the W3C has defined a static method URL.createObjectURL. You can use this API to create a URL that directly points to a JavaScript object created on the client, rather than actually pointing to an online resource.

You can see how it works in this example:

const videoTag = document.getElementById('my-video');

// creating the MediaSource, just with the "new" keyword, and the URL for it
const myMediaSource = new MediaSource();
const url = URL.createObjectURL(myMediaSource);

// attaching the MediaSource to the video tag
videoTag.src = url;

Now that you have a MediaSource, what should you do? That’s not the only MSE specification. It also defines another concept, Source Buffers.

Source Buffers

The SourceBuffer interface represents a chunk of media to be passed into an HTMLMediaElement and played via a MediaSource object. To put the video directly into the MediaSource for playback, you’ll need Source Buffers. MediaSource contains one or more instances of it. Each is associated with a content type, such as Audio, Video, or both of them at the same time. Source buffers are all associated with a single MediaSource and are used in JavaScript to add video data directly to HTML5 video tags. Separating video and audio allows managing them separately on the server. Doing so has several advantages such as control over your traffic and increasing performance. This is how it works:

// -- Create a MediaSource and attach it to the video (We already learned about that) --

const videoTag = document.getElementById('my-video');
const myMediaSource = new MediaSource();
const url = URL.createObjectURL(myMediaSource);
videoTag.src = url;

// 1. add source buffers

const audioSourceBuffer = myMediaSource.addSourceBuffer(
  'audio/mp4; codecs="mp4a.40.2"'
);
const videoSourceBuffer = myMediaSource.addSourceBuffer(
  'video/mp4; codecs="avc1.64001e"'
);

// 2. download and add our audio/video to the SourceBuffers

// for the audio SourceBuffer
fetch('<http://server.com/audio.mp4>')
  .then(function (response) {
    // The data has to be a JavaScript ArrayBuffer
    return response.arrayBuffer();
  })
  .then(function (audioData) {
    audioSourceBuffer.appendBuffer(audioData);
  });

// the same for the video SourceBuffer
fetch('<http://server.com/video.mp4>')
  .then(function (response) {
    // The data has to be a JavaScript ArrayBuffer
    return response.arrayBuffer();
  })
  .then(function (videoData) {
    videoSourceBuffer.appendBuffer(videoData);
  });

Adaptive Streaming

Many video players have an “auto quality” feature that automatically selects quality based on network speed. If you have a slow internet connection then you’ll end up watching a video in the lowest resolution (probably 360p or even lower), on the other hand, if the connection is good then the video will be in the highest resolution (probably 1080p or even higher). It depends on how many options the specific video has. In addition, your hardware capability is also taken into account.

On the server side, we create different segments for every quality we want to support. For example, we would put the following files on our server:

./video/
  ├── ./360p/
  |     ├── video_0.mp4
  |     ├── video_1.mp4
  |     └── video_2.mp4
	├── ./720p/
  |     ├── video_0.mp4
  |     ├── video_1.mp4
  |     └── video_2.mp4
  └── ./1080p/
        ├── video_0.mp4
        ├── video_1.mp4
        └── video_2.mp4

The web player then automatically selects the correct video file to load when the network or CPU state changes. This is entirely done in JavaScript. For video segments, it could, for example, look as follows:


/**
 * We first derive the desired quality from the bandwidth
 * Then pass it to the pushVideoSegment()
 * which fetches each segment with the desired quality in turn and returns a promise
**/

function pushVideoSegment(nb, wantedQuality) {
 const url = "<http://my-server/video/>" +
 wantedQuality + "/segment" + nb + ".mp4");
return fetch(url)
 .then((response) => response.arrayBuffer());
 .then(function(arrayBuffer) {
 videoSourceBuffer.appendBuffer(arrayBuffer);
 });
}

/**
 * Translate an estimated bandwidth to the right audio
 * quality as defined on server-side.
 * @param {number} bandwidth
 * @returns {string}
 */
function fromBandwidthToQuality(bandwidth) {
 return bandwidth > 320e3 ? "360p" : "720p";
}

// first estimate the bandwidth. Most often, this is based on
// the time it took to download the last segments
const bandwidth = estimateBandwidth();

const quality = fromBandwidthToQuality(bandwidth);

pushVideoSegment(0, quality)
 .then(() => pushVideoSegment(1, quality))
 .then(() => pushVideoSegment(2, quality));

Conclusion

Congratulations, you’ve learned multiple ways how to stream a video file into your browser. We have explored several tools that are helping you not just change some video parameters by using vanilla JS and HTML, and also Media Source Extensions (MSE) that are helping all developers around the world to build professional web players which you’re using yourself, such as YouTube, Twitch, TikTok and many others.

If you want to give your users the ability to also edit videos within your mobile application check out our Video Editor SDK offering features such as trimming, transforms, video composition and more. It’s fully customizable to match the look and feel of your app and the most performant solution on the market.

Looking for more video capabilities? Check out our solutions for Short Video Creation, and Camera SDK!

If you have questions, feedback or comments reach out to us on Twitter.

Mark – IMG.LY Blog

Comparing the Top 5 Open Source JavaScript Image Manipulation Libraries

Sharp

Use cases

Example

Use Cases

Example

Merge Images

Use cases

Example

LooksSame

Use cases

Example

Jimp

Use cases

Example

Conclusion

How to Stream Videos Using Javascript and HTML5

The <video> tag

Combining with JavaScript

What are Media Source Extensions (MSE)

This specification was designed with the following goals:

**Source Buffers**

**Adaptive Streaming**

Conclusion

The `<video>` tag

Source Buffers

Adaptive Streaming