Development – IMG.LY Blog

Cutting Through The Jungle: An In-depth Review of Cloud GPU Providers to Train Your AI Models in 2024

Walter — Mon, 22 Apr 2024 06:35:54 GMT

Navigating the World of AI Models Hosting

Here at IMG.LY, we recently dug into finding the best place to host AI models to support apps we’re dreaming up. We wanted to figure out if using cloud GPUs or going serverless would work better for us. As we were looking specifically for service providers to run Image Generation Workloads on, we focused on those that could be the best fit for that. Along the way, we picked up some cool insights and ran into a few hiccups. We think sharing our journey and the things we figured out could help you when you’re looking to deploy your own AI models.

First off, we’ll explain what cloud GPU and serverless hosting really mean. Then, we’ll chat about their good and not-so-good sides when it comes to hosting AI models. It’s super important to make sure whatever hosting you choose fits your model like a glove. We’ll talk about some tools we stumbled upon that could help with that. Next up, we’ll give you a peek at some of the providers we checked out and our thoughts on how they might fit with what we’re working on. We decided to skip over the big names like IBM, Google, and Amazon this time. We were curious about what the newer, smaller companies have to offer.

To wrap things up, we’ll share some final thoughts on all our research. Plus, we’ll throw in some tips and ideas you might want to think about when you’re doing your own digging. Whether you’re developing AI models or planning to host some of the well-known ones, we hope our adventure helps you nail down the perfect hosting solution for what you need. Ready to jump in?

Kinds of Cloud Hosting for AI Models

Cloud hosting has been around for as long as there has been a cloud. Though the server hardware is not at your location, earlier versions of cloud hosting required that your team learnt lots about server infrastructure. As things have evolved, providers now manage the infrastructure so that you can focus on your work. You can now host even just a single function in the cloud, if that’s what you need. In our research, we looked at general serverless hosting and at Cloud GPU AI providers.

Serverless Hosting

Serverless hosting can be defined as an architecture model that lets developers build and run applications and services without managing the servers they run on. The cloud provider manages things like security, provisioning, scaling, and connectivity.

In a serverless CPU-loads hosting the host provisions your services to the most appropriate and available hardware. However, with most of the providers of GPU loads you get to choose.

Serverless Pros:

Pay-per-compute model: you only pay for the compute time you consume.
Autoscaling: the provider will automatically scale up or down depending on load, from a few requests a day to thousands per second.
No server management: eliminates the need for developers to also understand server infrastructure. Often, just a Docker image holding an application is sufficient.

Serverless Cons:

Cold starts: instance deallocates after a certain idle time (enabling the great pay-per-compute model) so initial request after this can be noticeably slow.
Limited control over specifics: certain GPU hardware or even server hardware may be unavailable at times which can impact performance.
Limitations on time - there may be limitations on the execution time of functions, which can impact long-running processes.

Cloud GPU Hosting

Cloud GPU hosting provides access to GPU and TPU (Tensor Processing Unit) hardware that can perform the parallel operations essential for AI model training and inference. The provider allows users to configure specific hardware for their jobs.

With cloud GPU each service or model gets its own GPU while running. Your other services communicate with the model through an API.

Cloud GPU Pros:

High performance: GPUs are specifically designed to run AI models and other tasks like deep learning and complex simulations.
Full control of hardware: users can specify specific hardware configurations for their projects.
Persistent availability: resources are not deallocated, so there is no latency for provisioning for the first request.
Cost-effective experiments: the upfront cost of purchasing GPU hardware to experiment with different configurations is eliminated. Services are priced with a pay-as-you-go model.

Cloud GPU Cons:

Costs over time: costs do not go down during periods of low demand. Over time, costs can potentially surpass the cost of investing in local hardware.
Management overhead - managing and optimizing hardware configurations is not automatically part of the hosting. You’ve got to learn some server administration and manage security and upgrades.

Providers

It’s important to understand that this isn’t a ranking of the best providers or an endorsement. It’s what we discovered with some web searching, reviewing the available documentation, and tinkering with any demo or free tools and models the provider makes available. The list could easily have been different providers and we think some of the pros and cons and qualities would be the same. Hopefully, some of the questions we raise and the pros or cons we noticed in our research can help you to guide your research.

Our goal was to find potential hosts for various workflows with different models in a scalable manner. We want to be able to build applications around the workflows. Some of our, specific, requirements include:

Autoscaling, ideally out-of-the-box without the need for custom Kubernetes setup or similar technologies.
Minimal vendor lock-in.
Compatibility with various technologies (REST API, WebSocket, Webhooks, etc.).
Support for Windows Server.

With those disclaimers and caveats, here is a short summary of our research.

Provider	Best For
Runpod IO (Serverless)	Deploy AI models with GPU support and require customizable API interfaces.
Vast AI (Serverless)	Affordable GPU resources and a variety of GPU options for AI model training.
Paperspace (Serverless)	Flexible workflows and support for different stages of AI model development.
CoreWeave (Serverless)	Strong knowledge of Kubernetes and need autoscaling capabilities for AI workloads.
Modal (Serverless)	Comprehensive documentation and examples for deploying AI models in containers.
ComfyICU (Serverless)	Serverless infrastructure tailored for hosting ComfyUI applications.
Replicate (Serverless)	Easy-to-use API for executing AI tasks without managing infrastructure.
Genesis Cloud (Cloud GPU)	Sustainability and need scalable GPU instances for AI model training.
Fly IO (Cloud GPU)	To deploy complete applications with GPU support in a scalable environment.
Runpod IO (Cloud GPU)	GPU resources in various regions and require customizable Docker-based deployments.
Lamda Labs (Cloud GPU)	On-demand GPU resources for model training and inference tasks.
Together AI (Cloud GPU)	A platform for testing serverless models and occasional access to GPU clusters.

If you want to skip ahead to a specific part, here are the providers we will be diving into:

Serverless Providers
Runpod IO (Serverless)
Vast AI
Paperspace
Banana Dev
CoreWeave
Modal
ComfyICU
Replicate

GPU Cloud Providers
Genesis Cloud
Fly IO
Runpod IO (Cloud GPU)
Lamda Labs
Together AI

Serverless Providers

Runpod IO (Serverless)

Runpod IO

Concept:

A Docker image that includes the installation of Python + GPU packages, models, and ComfyUI.
Python/Go handlers act as an API interface to ComfyUI, which is vendor-specific, but can be wrapped in a more general API for reuse. For more information, see this article on hosting a ComfyUI workflow via API.

Pros:

Good documentation, including public GitHub repositories with examples.
Relatively large community for a new provider.
Compatibility with Windows Server.
Handlers allow for webhook and WebSocket-like communication for API feedback.
Network volume to store models/data and reduce cold start times.
Control over the number of workers and the ability to define persistently active workers.

Cons:

Availability of GPUs, especially in Europe, needs to be validated.
Handlers can only be written in Python and Go.

Open Questions:

General open questions regarding serverless infrastructure and AI inference tasks.

Conclusion:

The overall package seems very mature. The setup can largely be adopted from the GitHub examples. Good documentation and community support (notably on Reddit). The open questions regarding pricing and cold starts are typical for serverless infrastructure.

Vast AI

Concept:

Peer-to-Peer Sharing. Companies/organizations can rent out their unused GPUs.
A GPU Marketplace approach.

Pros:

Affordable prices through their peer-to-peer GPU sharing model.
A wide selection of different GPUs.
Good global availability of GPUs.
Ability to define autoscaler groups, allowing different workflows to scale differently.

Cons:

The autoscaler is currently only in beta mode.
Data privacy/security concerns when renting GPUs from anonymous providers.

Open Questions:

How will the autoscaler beta evolve?
Control over GPU providers: Can one allow only certain trusted providers (e.g., those based in the EU)?

Conclusion:

Even though the pricing is more affordable, there may be significant issues, in terms of security and data protection, as well as the fact that the autoscaler is still in the beta phase.

Paperspace

Concept:

The serverless approach (Workflows or Gradient) is still in beta Paperspace Gradient Workflows is based on Argo Workflows which utilizes Kubernetes.
A predefined API is available for communicating with workflows, as detailed in DigitalOcean’s documentation for Paperspace commands.

Pros:

The ability to use different machines (GPUs) at different stages of a workflow.
Provided by Digital Ocean, allows for general hosting customers to expand into GPU hosting without finding a new vendor.
Possible Windows support as outlined in DigitalOcean’s documentation on running Windows apps.

Cons:

Complex documentation: offers many features for various use cases (AI learning, data preparation, validation, and inference).
Vendor lock-in through a proprietary system: Gradient Workflows and YAML config are specific to Paperspace.
No real-time feedback over the API.

Open Questions:

Since it’s still in beta, how will the ecosystem continue to develop?
How extensive is the knowledge of Kubernetes required to implement autoscaling?

Conclusion:

It’s positive that it’s offered by Digital Ocean as they are a more mature company with general hosting experience. The approach seems very specific to Digital Ocean. Furthermore, it may require experience with Kubernetes.

Banana Dev

It has been excluded: Recently, they announced the termination of their serverless model as it was not cost-effective.

Learning from this: Currently, there are many new providers entering the market aiming to establish themselves as cloud GPU or serverless GPU providers. This highlights the importance of minimizing vendor lock-in.

CoreWeave

Concept:

Heavily based on Kubernetes.
- A Kubernetes file is created for setup; scaling and additional infrastructure are managed by Core Weave.

Pros:

Autoscaling by default with the possibility of scaling to zero.
Supports Windows.
Minimal vendor lock-in due to Kubernetes configuration.

Cons:

Strong dependency on Kubernetes, with the serverless setup based on KNative documentation.
Does not offer a handler API, etc., to communicate directly with ComfyUI.

Open Questions:

How complicated would it be to implement an API interface and resulting scaling to address the correct instances, etc.

Conclusion:

Good documentation and a close interface to Kubernetes. For a team with strong knowledge of Kubernetes, this could be a prime candidate.

Concept:

Container Setup: Containers are defined through Modal’s own container setup Modal custom container documentation.
- Docker images can also be used.
Modal-specific handlers to communicate with ComfyUI and other models.

Pros:

Supports webhooks and custom endpoints Modal webhooks documentation.
Focus on fast startups/cold starts.
Emphasis on AI inference tasks.
Comprehensive documentation with many examples.

Cons:

Vendor lock-in if Modal’s container setup is used.
Autoscaling and scaling configuration are not directly described.

Open Questions:

How exactly does the autoscaling work?

Assessment:

For us, this is a candidate for closer consideration. The container setup can be managed through Dockerfiles, and the API defined by Modal’s own interface.

ComfyICU

Concept:

Pure focus on ComfyUI, serverless infrastructure.
API interface for communication.

Pros:

Minimal setup effort.

Cons:

Limited control over the API.
Limited GPU resources.

Open Questions:

How does the autoscaling work, if it exists at all?
Community-based open source. What is the long-term support for this project?

Conclusion:

Potentially useful for testing or building a demo site, but probably not suitable for developing our commercial applications.

Replicate

Concept:

Execution of AI tasks/models in the cloud via an API.
No access to infrastructure, etc.

Pros:

Supports various languages: Node, Python, Swift.

Cons:

No control over the infrastructure, number of GPUs, or workers.
API rate limits.

Open Questions:

How can autoscaling be enabled?
Is it possible to create custom API endpoints, webhooks, websockets?

Conclusion:

For testing or as a demo for one’s own model, this can be a very good platform. However, as a standalone application interface, it doesn’t meet some of our core requirements.

GPU Cloud Providers

Genesis Cloud

Concept:

Focus on sustainability and renewable energy.
Scaling through instances as detailed in here.

Pros:

A REST API is available for managing instances.

Cons:

The availability of GPUs varies significantly by region.
Limited selection of GPUs.

Open Questions:

How quickly can new instances be scaled up or down?

Conclusion:

The use case for Genesis Cloud appears to be more suited for model training or tasks that require a significant amount of computing power for extended periods.

Fly IO

Concept:

Focus on the deployment of complete applications.
Also offers its own GPU servers.

Pros:

Docker File support with additional configuration via a TOML file.
Quick scaling of GPUs up or down facilitated by the launch process.

Cons:

Limited selection of GPUs, with only very large GPUs available.
Specifically tailored for Linux.

Open Questions:

How well does the launch system perform for relatively fast inference tasks?

Conclusion:

Since primarily large GPUs are available, the focus here also appears to be more on model training or other long-duration tasks. However, the launch system might also potentially be used for inference.

Runpod IO (Cloud GPU)

Runpod IO

Concept:

A wide range of GPUs available across various regions.
Base Docker images for popular tasks or support for custom Docker images.

Pros:

Many different data center regions.
A variety of CPUs available.
Simple setup via Docker images.

Cons:

No direct autoscaling (would need to use Runpod Serverless for that).
Despite a large selection of GPUs and many different data center locations, the availability of GPUs is not very high.

Open Questions:

Can autoscaling be implemented without using serverless?

Conclusion:

The setup can largely be adopted from the GitHub examples. There is good documentation and a community (much of it on Reddit). The availability of GPUs could become a problem, especially for smaller GPUs.

Lamda Labs

Concept:

On-demand cloud with a focus on model training and inference.
Similar concept to Runpod, offering a variety of GPUs.
- GPU availability is very limited.

Conclusion:

Runpod and Lambda Labs seem to have a similar approach and similar offerings. Runpod appears to have greater availability.

Together AI

Concept:

Offers an API and playground for testing serverless models.
Also offers GPU clusters but only upon request.

Conclusion:

We didn’t dig into the GPU clusters since information is available only upon request. Otherwise, in the API/serverless area, it appears to be similar to Replicate.

Established Providers

As we said in the introduction we did not examine the old, large providers like Google Cloud, AWS, Azure, Nvidia, etc., in detail. Rather, we focused on the new providers aiming specifically at the market segment of AI GPUs. With the older providers, we are more in the realm of cloud GPUs and less in serverless. Given the size of these providers and the wide range of market segments they cover, it can make sense to opt for them if one is already familiar with their architecture and documentation.

Google Cloud Platform (GCP)
AWS
Microsoft Azure
IBM Cloud
NVIDIA GPU Cloud (NGC)

Conclusion

Just as we saw that performance can vary wildly for different models, pricing can be similarly complex. When evaluating costs, consider factors like response times, the number of required workers, and potential charges for features like caching. Many providers offer detailed pricing guidelines on their websites, which can be crucial for ensuring you only pay for the computing power you truly need. Experimenting with performance of your model and applications during development will be helpful to make sure your hardware and pricing are both optimized for your application.

Another thing to consider is what kind of experience does your team already have? Most cloud GPU services provide tools like CLI or REST APIs to manage resources, which can be a steep learning curve if your team is not familiar with these technologies. Additionally, while serverless platforms may support multiple programming languages, compatibility with your team’s preferred language—be it JavaScript, Python, or Go—is essential. As exciting as it can be to learn new languages, it’s probably not the best use of your team’s time.

The size of files you’ll be moving between your model and the other parts of your project may also be a factor. Your users may not notice latency for models that communicate using text only. Text moves quickly from point to point in a network. However, if your model takes large image files as input or output, you may find that moving data between data centers is too slow. Then you’d want to focus on providers who can offer more general hosting in addition to cloud GPU hosting.

As we continue to research this for our own projects, we are thinking the best configuration for us is to use a cloud GPU exclusively for generation tasks and communicate with it via an API from our existing back end. We will have to experiment to see if we can have those functions geographically separate, or if we need to find one hosting company and one data center for both. As we learn more we may change our ideas, but that’s part of the fun of working in technology, things change. By using the higher-cost cloud GPU for as few tasks as possible, we’ll know we aren’t wasting compute power for things easily handled by a general CPU.

We hope this has given you some useful background and ideas as you research hosting options for your AI projects. Understanding the subtle differences between serverless and cloud GPU hosting can spark innovative ideas tailored to your needs. Perhaps some of the lesser-known providers we’ve explored might just be the perfect fit for your next project. As always, the dynamic nature of technology keeps us on our toes—ready to adapt and evolve. Happy hosting!

Thanks for reading. Join over 3000 specialists with powerful apps and subscribe to our newsletter. We keep you in the loop with brand-new features, early access, and updates.

How to Create Image Filters for iOS

Walter — Tue, 24 Aug 2021 08:09:37 GMT

In this tutorial, you will learn how to write a custom CIFilter in Metal and Swift. You can use this filter in any CoreImage pipeline.

Apple offers over 200 image filters in the CoreImage framework, but sometimes you need a little extra to make your images perfect. Apple provides two ways to create customize image filters. You can chain CIFilters together in a CIFilter subclass or wrap a CIKernel in a CIFilter subclass. Either method creates a filter that CoreImage can execute on the GPU. That makes the filter fast. CoreImage filters can filter live video without impacting the frame rate. Before iOS 11 the only way to write a custom CIKernel was to pass a string containing the code to the GPU at runtime using the OpenGL Shading Language. This had two drawbacks: it was difficult to find errors in the string of commands and the kernel was not compiled until runtime. This tutorial will show you how to add a custom filter with a Metal-based CIKernel to any Swift project in Xcode. The code samples in this tutorial use Xcode 12.5 and Swift 5.

Clone this repository for a sample project and example code that supports this tutorial. The repo also contains some other custom filters, demonstrating other aspects of Metal filters.

Adding Metal Support to an Xcode Project

The first step is to add a Metal source code file to your project.

Find the “Metal File” template, select it and click Next. Name the file and save it. At compile time, Xcode combines all of the .metal files in your project into a single .metallib file. So, organize your Metal code into lots of files or just one file as you prefer. The last step before you start writing code is to add compiler flags for CIKernel objects.

In the Build Settings, find the other Metal Compiler Flags and add an entry of -fcikernel.

Then find the Other Metal Linker Flags and add an entry of -cikernel.

Important note: any project without .metal files hides the Metal Compiler and Linker sections from build Settings.

Setting Up a Metal File for CoreImage Kernels

Metal is a general-purpose technology for writing code that will execute on the GPU. The compiler flags tell Metal that you will be writing code to work with CoreImage. With a Metal source file in your project and the compiler flags set, you’re finally ready to write some code! Open the Metal file you created above. It should be empty except for

#include <metal_stdlib>
using namespace metal;

Below these lines, import the CoreImage headers by adding:

#include <CoreImage/CoreImage.h>

and add a stub for your filter code with:

extern "C" {
  namespace coreimage {
    // KERNEL GOES HERE
  }
}

Prefix your kernel code with extern "C" and the CoreImage namespace. You write kernel code in the Metal Shader Language, which is a variation of C. If you only know Swift, you should be able to rely on Xcode’s code-completion and symbol lookup to help you as you’re learning to write Metal code. In addition to the general language reference, Apple provides a Metal Shading Language for Core Image Kernels document.

The example below creates a CIColorKernel. This kernel is optimized for changing the color value of each pixel in an image. It will change each pixel to a grayscale version of itself. A filter applying his kernel will send the color value of each pixel of the image to the kernel.

float4 grayscaleFilterKernel(sample_t s) { //1
  float gray = (s.r + s.g + s.b) / 3;      //2
  return float4(gray, gray, gray, s.a);    //3
}

Here is how this kernel works:

The filter provides the color of the current pixel to the kernel as a sample_t type. The kernel will return the new color for that pixel as a float4 type. The sample_t type is equivalent to a float4 type and is only named differently because of historical convention. A float4 type contains four float values. In the case of color, they are the red, green, blue and alpha values for the pixel. Each float has a value between zero and 1.
Calculate a grayscale version of the pixel by averaging the three color channels.
Create a new float4 with the averaged value for the three color channels and the original alpha value for the fourth.

CoreImage also provides a CIWarpKernel class that is optimized for changing the position of each pixel and a CIBlendKernel for blending two images. For more complex tasks, CoreImage also provides a general CIKernel. Apple suggests that chaining together optimized kernels in a pipeline is usually more efficient than trying to write a larger, general kernel. A color-optimized kernel only has access to the color of the current pixel. A transform-optimized kernel can only affect the position of the current pixel. A general kernel can impact color and position as well as read other values from the source image.

Wrapping a `CIKernel` with a `CIFilter`

Now you’ll make a CIFilter subclass as the interface between the Metal code and your Swift code. Every CIFilter has an outputImage variable that returns a CIImage. Generally, input variables for a filter begin with input. The grayscale filter you are making has a single input and a single output. Create a new Swift file in your project and title it “GrayscaleFilter.swift”. Then be sure to import CoreImage by adding

import CoreImage

to the top of the file. Then create the filter as a subclass of CIFilter.

class GrayscaleFilter: CIFilter {

Next, create a kernel variable as either static or lazy, so that it only gets created one time, regardless of how often it’s called.

static var kernel: CIColorKernel = { () -> CIColorKernel in
    let url = Bundle.main.url(forResource: "default",
                            withExtension: "metallib")! //1
    let data = try! Data(contentsOf: url)
    return try! CIColorKernel(functionName: "grayscaleFilterKernel",
                      fromMetalLibraryData: data) //2
}()

Look in the bundle for the compiled metal code. The metal code will have a filename of “default”.
The .metalib may contain many kernels so use the one called “grayscaleFilterKernel”

Now add a variable to hold the input image.

var inputImage: CIImage?

Finally override the outputImage variable.

override var outputImage: CIImage? {
    guard let inputImage = inputImage else { return .none }
    return GrayscaleFilter.kernel.apply(extent: inputImage.extent,
                                   roiCallback: { (index, rect) -> CGRect in
                                                  return rect
                                  }, arguments: [inputImage])
}

The apply function of the kernel takes an extent argument. For a CIImage the extent property is the width and height of the image. The function has an roiCallback that allows you to specify what part of the image the kernel uses for each calculation. For a color filter, you generally just pass back the rect. Finally, pass in any arguments as an array. Notice that there is not any type checking here. It is up to you to pass in the correct types in the correct order.

Using the Filter

With the Metal code wrapped in a CIFilter you can now apply the filter images the same as using one of the built-in filters. Create an instance of the filter, then assign the inputImage variable a CIImage and display the outputImage. Your code might look something like this:

let filter = GrayscaleFilter()
filter.inputImage = image

displayView.image = UIImage(ciImage: (filter.outputImage ?? image) ?? CIImage())

and it can render a grayscale version of an image, like in this example.

Going Further

You can write custom filters and chain them to the built-in filters to invent new effects. Many of the math formulas for effects are available on the Internet. Most are easy to translate into the Metal Shader Language to make them perfect for your application. Writing an entire image or video editing application to showcase your filters is a much larger task. Using an SDK like the IMG.LY PhotoEditorSDK or VideoEditorSDK can help you provide an app that has the features that users will expect in addition to your cool filters.

To add your filter to the PhotoEditorSDK you first wrap your CIFilter with an Effect. For a basic filter, you only need to override the newEffectFilter variable.

import PhotoEditorSDK

class GrayscaleEffect: Effect {
  override var newEffectFilter: CIFilter? {
      GrayscaleFilter()
  }
}

Then add the filter to the Effect.all array when configuring the SDK, using something like

Effect.all = [NoEffect(), GrayscaleEffect(identifier: "grayscaleFilter", displayName: "Best Gray")]

Now your filter appears the same as the other 60+ filters that ship with the SDK.

It applies to live camera previews as well as still images.

Wrapping Up

In this tutorial, you learned how to configure Xcode for custom Metal code. You also learned how to create a color filter kernel in Metal and wrap it in a CIFilter. By combining your filters with Apple’s filters in pipelines, you can create unique effects for your next app. Further, using an SDK such as PhotoEditorSDK or VideoEditorSDK allows you to showcase your filters in a full-featured image or video editing application. The PhotoEditor SDK for iOS documentation will give you deeper look into all other image adjustments including, ofcourse, the image filtering we discussed above.

Thanks for reading! We hope that you found this tutorial helpful. Feel free to reach out to us on Twitter with any questions, comments, or suggestions.

A Photo and Video Editor for React Native Apps

Julia — Fri, 29 May 2020 10:30:01 GMT

How to integrate a Photo and Video Editor into your React Native App

This tutorial walks you through the integration process of PhotoEditor SDK and VideoEditor SDK into your React Native app for iOS and Android. You’ll learn how to use our React Native modules to facilitate the integration and to customize our editors. For this tutorial we presume that all the necessary development tools for building an iOS and Android app are met, so make sure to complete the official React Native CLI Quickstart guides for iOS and Android beforehand.

Please make sure to acquire the licenses for PhotoEditor SDK and VideoEditor SDK before integrating them.

https://www.youtube.com/embed/e9JiCMQKrJY?feature=oembed

In this tutorial, we’re going to show how to integrate PhotoEditor SDK and VideoEditor SDK for iOS and Android into your React Native app. Therefore, we created React Native modules for our products to simplify this process for you as much as possible. We’re going to use VideoEditor SDK’s README, which is in most parts identical to the PhotoEditor SDK’s README, and Visual Studio Code. So, let’s get started.

First, we create a React Native project with the name “Demo” based on the default template by using the command npx react-native init Demo. The project will now be initialized and automatically install all dependencies of the current React Native version. Afterwards, we can find the new React Native project ready to use in the folder “Demo”. So, we’ll speed this up a little.

We already prepared another folder with resources and assets that we want to integrate in our app. Here we chose an image, the required licenses for our PhotoEditor SDK and VideoEditor SDK for both target platforms, a video and two logos that we will later use to show how we can customize our editors. We copy these resources into the root of our project to make the resources accessible for our app.

Now, we switch to the folder of the “Demo” project*.* We can now copy and execute the command yarn add react-native-videoeditorsdk from the README to install the dependencies to the React Native module for our VideoEditor SDK … and to the React Native module for PhotoEditor SDK by issuing the command yarn add react-native-photoeditorsdk.

Now, we’re going to set up the dependencies for our native iOS libraries. We can simply copy the command cd ios && pod install && cd .. from the README and execute it to install all iOS dependencies. They include the native PhotoEditor SDK and VideoEditor SDK libraries that are required by our React Native modules.

And now, we set up the dependencies for our native Android libraries. The required steps that we will now take are described in detail in the README.

android {
  defaultConfig {
    multiDexEnabled true
  }
}

dependencies {
  implementation ‘androidx.multidex:multidex:2.0.1’
}

We copy the lines and add them at the end of our android/app/build.gradle file. Now we need to change the superclass of our MainApplication class to enable Multidex. Next, we add the img.ly repository and the plugin by copying the following lines and add them at the top of our android/build.gradle file located in our Android folder.

buildscript {
  repositories {
    jcenter()
    maven { url "https://plugins.gradle.org/m2/" }
    maven { url "https://artifactory.img.ly/artifactory/imgly" }
  }

  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.61"
    classpath 'ly.img.android.sdk:plugin:7.1.8'
  }
}

Now, we can configure our PhotoEditor SDK and VideoEditor SDK by opening the android/app/build.gradle file and add these lines under apply plugin: “com.android.application".

apply plugin: 'ly.img.android.sdk'
apply plugin: 'kotlin-android'

// Comment out the modules you don't need, to save size.
imglyConfig {
  modules {
    include 'ui:text'
    include 'ui:focus'
    include 'ui:frame'
    include 'ui:brush'
    include 'ui:filter'
    include 'ui:sticker'
    include 'ui:overlay'
    include 'ui:transform'
    include 'ui:adjustment'
    include 'ui:text-design'
    include 'ui:video-trim'

    // This module is big, remove the serializer if you don't need that feature.
    include 'backend:serializer'

    // Remove the asset packs you don't need, these are also big in size.
    include 'assets:font-basic'
    include 'assets:frame-basic'
    include 'assets:filter-basic'
    include 'assets:overlay-basic'
    include 'assets:sticker-shapes'
    include 'assets:sticker-emoticons'
  }
}

Getting back to our iOS version, we can now launch our demo project on iOS, which will currently look like a plain React Native project that we initialized with the first command.

The main difference to an off-the-shelf React Native project is that our React Native modules are installed and ready-to-use in the App.js file once the native projects are compiled. Then, it won’t be necessary to recompile the native projects for the remainder of this tutorial. We sped up the compilation a little and here we go — our React Native app is running on the iOS simulator. Now, we do the same for the Android version and wait until the project is compiled.

Now, the demo project launched on both platforms as we can see on the right on the iOS simulator at the top, and on the Android emulator at the bottom of the screen.

We’ve decided that we want to start our photo editor by pressing a button. So next, we’re going to actually customize our React Native app by adding this button. Therefore, we open the App.js file and import the Button component in order to create a button with the title “Edit a sample image”. For now, we leave the onPress function empty.

We save the App.js file to trigger a refresh of the running apps and immediately see the result on the right. The new button appears in both, the iOS and the Android app.

Now we create a second button with the title “Edit a sample video”. This will respectively start our video editor. And again, we save the App.js file and see the second button appear on the right side.

Next, we are going to add the code that actually opens our editors when we press the buttons. Visual Studio Code automatically imported the respective React Native PhotoEditor SDK module for us at the very top of the file, while writing the code that makes use of our SDK. We do the same for the VideoEditor SDK. We use the require function to make static assets available to our app. Here, we “require” our sample image and our sample video that we copied to the app’s folder in the beginning and pass them as the first argument to our openEditor functions. The first argument can also be a regular URI.

We save the App.js file again and now we can click the buttons to start our photo editor or video editor. There we go! We still see a watermark here. The reason for this watermark is that we haven’t unlocked our SDKs so far which we will do next.

We unlock both products with our licenses to get rid of the watermark. If not unlocked, the watermark will be on both the image and video previews as well as on the exported images and videos. To unlock the products, we use the unlockWithLicense function of each SDK. In total, we need four license files, one license file for each product and platform combination. The license files should be named pesdk_license and vesdk_license with platform-specific extensions .ios.json and .android.json. React Native will then automatically pick the right file for the corresponding platform. After this, the watermarks will be removed for PhotoEditor SDK and VideoEditor SDK on both platforms. And now you can also see it in the simulator — no watermarks anymore.

In the next step, we’re going to change the configuration of the editors. If no changes are made to the configuration, our default stickers are available with the editor. To customize them, we need to import the Configuration from either the PhotoEditor SDK or VideoEditor SDK. The configurations are compatible between both products. So, for this tutorial, we decided to use the VideoEditor SDK configuration which we are now using by adding Configuration to the react-native-videoeditorsdk imports. We decided that we want to add custom stickers to our editors. Therefore we define a non-default configuration to the sticker tool.

const configuration: Configuration = {
  sticker: {}
}

To customize the sticker assets, we need to define the sticker “categories” array. Here, we define a new category and name the identifier demo_sticker_category. These asset identifiers must always be unique. Next, we set a name for the category and we name it “Logos”.

Each category also requires a thumbnail image to be displayed in the editor. For the thumbnail, we use the React logo that we added to the folder of our app at the very beginning. Next, we define the items for this new sticker category. These items are the actual stickers that we can apply to the edited image or video. We now create a new sticker for the React logo. Therefore, we call the identifier demo_sticker_react and name it “React”. These sticker names won’t appear in the UI, but they are used for accessibility. Now, we need to define the actual image that should be used for that sticker. Here we use the React image again.

To create a second sticker, we can now copy and paste the code of the first sticker. We create a sticker with our img.ly logo and rename the identifier of the pasted code to demo_sticker_imgly. Accordingly, we set the name to “img.ly” and change the file to imgly.png.

In addition, we want to specify a non-default tint mode for our second sticker by using tintMode: TintMode.SOLID which enables us to change the color of the sticker. The TintMode type is automatically added to the VideoEditor SDK imports for us by Visual Studio Code. Now, that we completed our configuration, we need to pass it as the second argument to the openEditor functions in order to take effect.

We save the App.js file again to refresh the running apps and we can see the result live after starting a new editing session. Please note that you cannot alter the configuration of a running editor instance. You always need to start a new editing session to see configuration changes.

We want to use another feature of our SDKs which is called serialization. With the serialization feature, we can capture all image and video editing operations that are applied in the editor and export them. This allows us to import the editing operations in later sessions and continue editing. The serializations are compatible between both products as well. The input serialization is the third parameter of the openEditor functions of our SDKs and the output serialization is optionally part of their result type.

First, we check if the result is “null”. This is the case when a user clicks the “Discard” button in the editor and thus does not export an image or video. If the result is not ”null”, we know that the user exported an image or video. Then we can assign the exported serialization to the previously defined global serialization variable which will then be input to the next editing session. We copy the code and add it to the video editor as well to enable the serialization function here too.

Now, one thing is left to enable the actual serialization export in the configuration. The serialization export is disabled per default because not every user needs the serialization feature. Here, we enable it now and also change the export type to object. By doing so, the result type of the editor will contain the serialization as an object. Per default, the serialization is exported to a file and that file name is returned as part of the export result. Writing the serialization to a file is a reasonable default as serializations can be quite large, especially if large amounts of binary data for personal stickers are embedded.

Now, we can run the app on the simulator and use all the parameters that we configured in this tutorial. First, we can add our custom stickers, both the React logo and the img.ly logo. Here we can also change the colors which we enabled with the tint mode.

We can also use the text design tool to add a phrase to our image. Here we can pick different designs, so we’re trying a couple and place the text design fitting to the image and logo.

Next, we export our image with the serialization. With the serialization function enabled, it is now possible to import the editing operations into our video editor. This allows us to keep on editing because the serialization is compatible between both products. So here we can add further words to our text design. We can also put filters on our video. For example, we can choose the peach duo tone and increase the contrast a little. And here we go! We successfully integrated PhotoEditor SDK and VideoEditor SDK into our React Native app.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

Designing a Photo Editor

Ludwig — Mon, 10 Apr 2017 12:30:00 GMT

Part 1: Design as a team

One designer on your team, in outright conviction that the current 2677FD can not be the right choice of blue, changes the color slightly to a more saturated 2676FB, unaware he again forgot to turn off flux. The next day Lisa is infuriated. The color she loved so much now looks terrible on her display. Aziz is furious. He was working on illustrations for the product website, nitpicking the right colors based on the original blue. Oh noes, what a mess…

Heard or seen that before? Bear with me, you´re not alone: Even though we are just a small team of three designers, the journey we had redesigning our product, the PhotoEditor SDK, left us with numerous insights about design collaboration. Most of those lessons were learned the hard way and many of the following suggestions are techniques that matured over the last year. The methods discussed below can be applied to any team, be it two or fifty designers. So, whether your team also struggles with the inability of Sketch to allow for real collaboration or if you’re just curious about how we dealt with the struggle, read on my fellow designer!

Use style guides early on

Right after the first screen is considered finished, sit together as a team of designers and discuss colors (don’t let the developers get wind of it). Schedule the whole thing so you have enough time to get finished while the sun is still out. Otherwise you will look at the interface the next day and have a really hard time reading the ten percent opacity labels.

The naming of colors/styles is crucial, they help to communicate with the developers and your fellow designers (we’ll get to that) and keeping a clean state of used styles.

As you can see, naming styles with a predefined set of simple rules, facilitates finding the style you are searching for. Especially since the colors that come into question for the given element are right by your selection. The prefix indicates the type of color: s for shadow, n for neutral, p for primary …

Why start counting at 100? Sketch doesn’t order styles by their absolute value, thus 10 comes before 1. This also gives you the flexibility to define several color ranges for one color type.

If your team defines their neutral colors in a dark spectrum, bright, neutral colors can then be defined as a detached unit by starting from the next hundreds.

Use Grids

I am from a really cozy Bavarian village, miles away from any accumulation of at least 12 houses (at least it felt that way in my youth, where the fastest transport we could use was the new lawn mower from the neighbours). To visit my friends in the next village, walking across an area of meadows, cow shit and finally woodland was the fastest path to take. There was no real pathway so everyone took a slightly different route to uber between the villages, which was just fine during the summer. However, come winter, not so much.

Every year it took us a few weeks, a lot wet socks and thus angry mom’s, to settle for one path. Ultimately, one of us took the courage to create the path of truth by preparing a trail with really tiny steps. By this time the snow usually hit the 30 cm mark.

I encountered a similar situation 10 years later. I started in a team that did give zero fucks about any type of layout or grid. Basically, every member took his own route. Switching out a specific component was often accompanied by the change of multiple components on the project since the perception of equal spacing between each one wasn’t given anymore.

Nowadays, I use grids everywhere, no matter how small the project, alone or in a team — and if you don’t already, I encourage you to start now! It makes it easier to let your creation feel coherent, speeds up the design process and helps any possible future collaborator to navigate trough and comprehend your design philosophy.

**Tip**: choose an easy to reach shortcut for show grid and show rulers or make a macro to show and hide both (mine is one of the mouse buttons).

For the photo editor we chose a primary grid of 8pt and a secondary one of 4pt. All components are aligned on the primary grid and fonts on the secondary. Designing isolated components based on the grid helps to keep things evenly spaced, even if components get switched out or rearranged.

We used a lot of different icon dimensions across all platforms (iOS, Android, HTML5), to guarantee a coherent look across them. It was essential to specify a layout and predefined rules for them as well. I will discuss the creation of icons in a future post, where I will also delve into finding a style that is unique and fits your project. So stay tuned!

If you haven’t started yet, create grids from the start on in every component of the project! It will probably get your feet wet at first, but your moms will be happy, I promise.

Keep your symbols organized

Keeping all the components ordered and close to each other is, again, great for consistency. After defining a component once in the page view, you will most likely return to the symbols page for future refinements. Now you can easily compare the component to every other used across the app. It will also make you slightly more liked by developers, something you should strive for as a designer at every time (to increase the chances of your next masonry grid not to get waived off as: “to time consuming to implement in the current release — and the release after”). Framing the cooperation between the two parties as smooth as possible is paramount at all steps of the product cycle.

And also, defining grids on both, the parent artboard and the components. This way pixel imperfections and inconsistencies will be visible immediately.

Also now, all elements inside a component share similar rules about placement and margins. They are easily interchangeable and allow for fast prototyping and iteration even at the final design stage.

One of the hardest things to accomplish while working day in day out on a project like this was keeping everything neatly organized while more and more components were created and added from different parties. In the sprints before an upcoming release, we often were sloppy keeping the symbols page organized, which in turn did cost us a lot of time later on. As a solution, we created a plugin, that lints your symbols page on safe.

What’s next?

The project is far from being finished and we will most likely change things here and there, building upon the methods discussed above. We are on an exciting journey crafting, designing, redefining a creative editor that is already used by hundreds of customers with millions of users. So expect more insights and learnings from our team in future articles!

Thanks for reading! To stay in the loop, subscribe to our Newsletter.