Technology – IMG.LY Blog

My digital journey to #nfts

Vanessa — Fri, 26 Mar 2021 11:07:26 GMT

The Beginning

In my first years as a graphic designer, my world consisted mainly of paper, books and magazines. To give you an example, my bachelor and master thesis were both printed books (Yes - I am that old). So i was born analog, but i wanted to become more digital.

In the last three years, I have done my own personal digitalization and transformed from the analog design world into a digital one to the point that last weekend I created and launched my first crypto art - an NFT.

But First – What is an NFT?

I have read about NFTs before from time to time. Then, two weeks ago, the internet went wild about the news that a well-known New York gallery sold a digital painting worth $69 million for the first time. And this made me thinking a bit more about the subject again and I decided to join the hype. Let’s get rich!

But what is a NFT and how does it work? NFT stands for Non-Fungible Token which is secured by the Ethereum blockchain. 1 ETH is a fungible token – like bitcoin – a digital currency that is exchangeable. However, in contrast to ETH, a NFT can represent ownership of unique items. These unique items can be digital art like an image or a video, or any other digital asset. Within the process of creating an NFT, the digital asset will get connected to the token - which is called “minting”. Even though that image can easily be copied or recreated, the NFT can prove to a third-party that a specific person or entity owns this token. This makes digital art verifiable and unique.

What excites me so much about NFTs is that they make it possible to give digital art a real value. In addition it gives digital artists the opportunity to be perceived as artists in the same way as their colleagues who paint with brush and paint on canvas. So this means that digital art can finally step up to it’s physical counterpart. The beginning of a whole new era of art.

Currently it feels like that one can basically mint everything to an NFT and people go crazy about it. Why? Because people always liked to collect special and rare things like stamps, pokemon cards, oil paintings and so on - the list is basically endless.

Going digital

Circling back to my own story about transforming from an analog world to a digital one, the first step was to make my work more digital. Three years ago I gave up being self-employment and started working as an art director at IMG.LY. We do develop software for photo and video editing. Our first product was the Photoeditor SDK later followed by our Videoeditor SDK. However, last year we started defining our goals and begin the journey to transform our editor into a completely new tool that combines photo editing with automatic layouting. Automated and simplified processes make it possible for designer to focus more on the development of cool concepts instead of struggling with the technology itself.

Some time ago - while testing the functions of our tool, I created a series of pixel-art images. In my opinion this pixel-art style fits the topic of NFTs quite well, so that I created a small series with three matching images, which got the name “Life behind Pixels”.

One of these pictures was supposed to be my first NFT, but I had to realize that my digital transformation still has some gaps that I need to fill. There are various marketplaces that function like galleries, where one can upload their digital art for everyone to see and be verified as an official artist. I chose Rarible because it has a rather small community compared to the big ones like e.g. OpenSea and it is possible for everyone to join. Other marketplaces tend to select artists they work with based on the number of followers and how well-known they are in the community.

If you want to create an NFT – so called “minting” it, you have to connect your digital wallet to one of these platforms and pay a fee (gas) of about $60 in Ethereum. Unfortunately I don’t have a wallet or even digital money, so what can I do? I asked my boyfriend who works as firmware engineer in the 9elements cyber security department and who was of course already deep in the crypto universe and bang: My first NFT was minted. Now I’m even more fired up.

And although this NFT topic is on everyone’s lips in the digital scene right now, I recently noticed in a clubhouse talk with other designers that this hasn’t arrived everywhere yet.

And now what?

First we are curious to see what happens with our first NFT and we are trying to find out what else is possible with it. Instead of attaching unique information to a simple image file and making it unique, there is a way more potential behind it: NFTs can be minted in a way that allows them to react to their environment using their integrated code. For example they can show a different state in the morning, at noon and in the evening or their appearance changes due to the number of attached wallets.

Right now we’re looking at everything there is to know about NFT and blockchain to develop more crypto art and be part of the new era. Our plan is to create a new piece of crypto art every month. I’m trying to figure out what’s hot in crypto design right now. What do we need to stand out and be visually and technically unique?

And what are the options if you don’t have a friend working in the tech area?I think besides a monthly NFT, there could also be a monthly blogpost ;)

Also check out Instagram and Twitter for daily updates!
https://www.instagram.com/ellly_pix https://twitter.com/ellly_pix

A more technical deep dive can be found here.

Despite the fact that I am super thrilled about NFTs and that I finally generated my own, there is also a lot of controversial discussion going on about the climate impact of NFTs and the energy consumed to mint digital assets. If you like to follow up with this discussion, I put together some interesting links where you can also find some alternatives to the “minting” process:

Calculate the CO2 Footprint of your Ethereum address: https://carbon.fyi/
h ttps://www.greenpeace.org.uk/news/the-biggest-problem-with-carbon-offsetting-is-that-it-doesnt-really-work/
https://memoakten.medium.com/the-unreasonable-ecological-cost-of-cryptoart-2221d3eb2053
https://www.hicetnunc.xyz/
https://xtz.news/latest-tezos-news/getting-started-as-an-nft-artist-on-tezos-using-hicetnunc/

How to integrate a Photo Editor into your iOS App

Felix — Wed, 10 Jul 2019 00:00:00 GMT

This tutorial is going to walk you through the integration process of the PhotoEditor SDK for iOS. You’ll learn how to setup the editor in seven minutes using CocoaPods. We created similar tutorials for Android and HTML5, so make sure to check those out as well.(Please make sure to get a trial license for the PhotoEditor SDK before integrating it.)

https://www.youtube.com/embed/AHzD4Mnm-NA?feature=oembed

Transcript

In this tutorial, we’re going to show you how to integrate the PhotoEditor SDK into your iOS app. We’re going to use Xcode and the PhotoEditor SDK’s documentation. So, first of all, we navigate to the “Getting Started” guide for iOS. We’re going to install the SDK using CocoaPods, because it is the recommended way to initialize this dependency.

We create a new Xcode project and select the “Single View App” template. Then, we assign a name to our project. For this tutorial, we’ll go with “PhotoEditorApp.” Here, it is essential that the given bundle identifier is identical to the one you used to request your license with. We decide to save our project files to the documents folder and create the new project.

Now, we change the iOS Deployment Target to version 9.0 as the SDK supports all iOS versions from 9.0 upwards, and thus, our app will be available for the highest possible number of devices. After that, we insert our previously downloaded license file into our app by simply dragging and dropping the file into our app’s folder. Here, it’s important that “Copy items if needed” is checked so that our license file will be copied to the appropriate place in our project instead of just being referenced.

Now, we can close our Xcode project. We open our folder structure and see that our project has been created in the documents folder. Now, we need the terminal to install the PhotoEditor SDK via CocoaPods. We type cd which is short for “change directory”, to change the directory of our source files. We can copy the project’s path via drag and drop. We now instruct CocoaPods to prepare Xcode for the use with CocoaPods by typing pod init. After that, CocoaPods creates the file “podfile.” We open the podfile in a text editor by double-clicking.

Here, we uncomment the second line and thus define that our project shall run from iOS version 9.0 upwards. Next, we copy this line (pod 'PhotoEditorSDK') from the PhotoEditor SDK’s documentation and paste it under # Pods for PhotoEditorApp to create a dependency between our project and the PhotoEditor SDK. We save the podfile and close the text editor. Now, we return to the terminal and instruct CocoaPods to install all dependencies by typing pod install and executing our command with the return key.

Here we can see that CocoaPods advises us to open our Xcode project with the xcworkspace file only. Furthermore, we see that there’s now a new folder called “pods.” Ideally, you wouldn’t check that folder into your source control system as it potentially contains large files, which can result in some issues. Github, for example, has a limit for the maximum file size, which the PhotoEditor SDK would exceed. So, it would be best if every user that works on the project activates pod install to install the dependencies for each processing machine.

Now, we can open the project file to incorporate our license into the file AppDelegate.swift. We copy this line from our documentation and paste it under the method UIApplication, didfinishLaunchingWithOptions.

if let licenseURL = Bundle.main.url(forResource: "license", withExtension: "") {
    PESDK.unlockWithLicense(at: licenseURL)
  }

It is important that we name our license file exactly like we did when we incorporated it into our project. In our case, that would be “ios_license” instead of “license.” Also, we can see that there is an error message, because we haven’t imported the PhotoEditor SDK yet. We simply type **import** PhotoEditorSDK, and the issue is resolved. Now our project is prepared so that we can use the PhotoEditor SDK with an active license.

Next, we’re going to prepare our app, so that it actually uses the PESDK. We configure our app with the Main.storyboard by opening the assistantview and creating a new button by just dragging and dropping it on the canvas. We’re going to use Alignment Constraints so that the button will always be displayed in the middle of the screen no matter the form factor of the device in use.

We’re going to name our button after our app and link it from our storyboard with our source code on the right-hand side via drag and drop. Now, we’re going to set a name for our method that is going to be called once the button is pressed. For this tutorial, we’ll go with startPhotoEditor.

We’ve decided that we want to start the PhotoEditor SDK with a CameraViewController. So, we copy the respective code from our documentation and paste it in the body of our method under startPhotoEditor.

let cameraViewController = CameraViewController()
present(cameraViewController, animated: true, completion: nil)

When we save the file, we get yet another error message that the camera can’t be found because we haven’t imported the PhotoEditor SDK for this source file either. Once we’ve done that, the error message disappears.

As we want our editor to start in camera mode, we have to create some entries in the Info.plist file that will allow us to use our device’s camera and grant access to our device’s image library. That is a restriction by Apple. If we didn’t create these entries, we wouldn’t be able to use our app as it would crash as soon as we open the camera. All necessary entries can be found under “Privacy.” First, we need “Camera Usage Description.” Here, we have to give a reason why we want access to our user’s camera. Said reason should be as meaningful as possible. For this tutorial, we’ll go with “We use your camera to take photos for editing.” We also want to allow the import of photos from the device’s library, so we add the identifier “Photo Library Usage Description” and type “We use your photos so that you can edit them” as description.

Now, we can run the app in the simulator. If we wanted to run the app on a physical device, we’d have to provide an Apple iOS developer account under “Team” so that the app can be signed because only registered developers are allowed to install apps on physical devices. However, we’ll go with the simulator.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

How to integrate a Photo Editor into your Website

Felix — Wed, 10 Jul 2019 00:00:00 GMT

This tutorial is going to walk you through the integration process of the PhotoEditor SDK into your website. You’ll learn how to setup the HTML5 editor in four minutes. We created similar tutorials for iOS and Android, so make sure to check those out as well.(Please make sure to get a trial license for the PhotoEditor SDK before integrating it.)

https://www.youtube.com/embed/9ZfEN3LMJU8?feature=oembed

Transcript

In this tutorial, we’re going to show you how to integrate the PhotoEditor SDK for HTML5 into your website. We’re going to use Visual Studio Code and the PhotoEditor SDK’s documentation. So let’s get to it.

We unzip the file and copy the assets, css and js folder. Those contain all the elements our editor needs to run properly. We paste the folders into our project folder. For this demo, we want to open a sample image with the editor, so we copy that into our folder as well.

Here, we already opened our project’s folder with Visual Studio Code. Next, we’re going to create an HTML file that we’re going to use to build our site. We copy this code for the <head> section and paste it into our index.html file.

<head>
  <!-- React Dependencies for the SDK UI -->
  <script src="js/vendor/react.production.min.js"></script>
  <script src="js/vendor/react-dom.production.min.js"></script>
  <!-- PhotoEditor SDK-->
  <script src="js/PhotoEditorSDK.min.js"></script>
  <!-- PhotoEditor SDK UI -->
  <script src="js/PhotoEditorSDK.UI.DesktopUI.min.js"></script>
  <link rel="stylesheet" href="css/PhotoEditorSDK.UI.DesktopUI.min.css" />
</head>

Here’s where we link to the JavaScript files from our js folder. Please note that if you store the folder somewhere else, you have to adjust these paths accordingly. Also, this is where the css file for our desktop UI is loaded.

Next, we’re going to add a <body> to our site. Then we’ll need a <div> where our editor is going to be displayed. For that, we can simply copy the snippet from the documentation.

<div id="editor"></div>

The <div>’s ID is set to editor so we can address it via JavaScript. Now, we copy this JavaScript snippet from the documentation so that the editor loads properly.

<script>
  window.onload = function () {
    var image = new Image()
    image.onload = function () {
        var container = document.getElementById('editor')
        var editor = new PhotoEditorSDK.UI.DesktopUI({
        container: container,
        // Please replace this with your license: https://www.photoeditorsdk.com/dashboard/subscriptions
        license: '{"owner":"Imgly Inc.","version":"2.1", ...}',
        editor: {
          image: image
        },
        assets: {
          // This should be the absolute path to your `assets` directory
          baseUrl: '/assets'
        }
      })
    }
    // image.crossOrigin = 'Anonymous'  // Setup CORS accordingly if needed
    image.src = './example.jpg'
  }
</script>

We have already prepared our license file here. It is important to copy the entire content of the license file and paste it after license. Also, it is important to insert the license as a string and not as a JavaScript object.

Please make sure that under baseUrl you have entered the correct path to your assets folder.

Under image.src we load our image. Please note that if you don’t load the image locally but from another source, say a server like an AWS S3 bucket, you have to setup CORS properly. We dedicated a whole section in our documentation to this issue, so make sure to check that out. However, as we’re going to load the image locally, we don’t have to mind that line.

Finally, it is important that we create the editor after the image is loaded so that we can pass the image to the editor.

Alright, we’re set. Now we can test everything by running a server. We’ll go with the SimpleHTTPServer.

As we can see, everything runs properly. Thanks a lot for watching and see you next time.

That was all that is necessary to get the PhotoEditor SDK up and running. For your convenience, here’s the whole source code of the HTML file:

<!DOCTYPE html>
<html>
  <head>
    <!-- React Dependencies for the SDK UI -->
    <script src="js/vendor/react.production.min.js"></script>
    <script src="js/vendor/react-dom.production.min.js"></script>
    <!-- PhotoEditor SDK-->
    <script src="js/PhotoEditorSDK.min.js"></script>
    <!-- PhotoEditor SDK UI -->
    <script src="js/PhotoEditorSDK.UI.DesktopUI.min.js"></script>
    <link rel="stylesheet" href="css/PhotoEditorSDK.UI.DesktopUI.min.css" />
  </head>

  <body>
    <div id="editor"></div>
    <script>
      window.onload = function () {
        var image = new Image()
        image.onload = function () {
            var container = document.getElementById('editor')
            var editor = new PhotoEditorSDK.UI.DesktopUI({
            container: container,
            // Please replace this with your license: https://www.photoeditorsdk.com/dashboard/subscriptions
            license: '{"owner":"Imgly Inc.","version":"2.1", ...}',
            editor: {
              image: image
            },
            assets: {
              // This should be the absolute path to your `assets` directory
              baseUrl: '/assets'
            }
          })
        }
        // image.crossOrigin = 'Anonymous'  // Setup CORS accordingly if needed
        image.src = './example.jpg'
      }
    </script>
  </body>
</html>

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

How to integrate a Photo Editor into your Android app

Felix — Tue, 11 Dec 2018 00:00:00 GMT

As developer friendliness is paramount to us, we created a short video tutorial to guide you through our integration process. We created similar tutorials for iOS and HTML5, so make sure to check those out as well.(Please make sure to get a trial license for the PhotoEditor SDK before integrating it.)

https://www.youtube.com/embed/FpmASFvgUdM?feature=oembed

Video Transcript

Today we’re going to show you how to integrate the PhotoEditor SDK into your Android app. We’re going to need Android Studio and the PhotoEditor SDK’s documentation to do so.

First, we’re going to launch a new Android Studio project. Here, we can choose a name for our application. Then, we’re going to insert our company domain and our package name. Inserting a package name is very important as you need a license that is issued for your specific package name to run the PhotoEditor SDK.

We’re also going to include Kotlin support, as every modern Android app should do that.

We then choose Phone and Tablet and API 16 because this is the oldest API that the SDK currently supports. Please note, that this is eventually going to change.

We then choose Empty Activity. Here you can set a name. For this tutorial, however, we’re going to leave it as it is.

Now, a new Android Studio project is created.

For the sake of clarity, we’re going to use the project view.

First, we have to include the license. To do so, we open the PhotoeditorApp folder and navigate through the subfolders app, src and main. In this subfolder, we create yet another folder called assets. We then simply paste our license into it.

We head over to the documentation, find the section for Android and make sure that the latest version is selected. As of the time of this recording that is version 6 or V6. Then we navigate to the “Getting Started” section. Here you can also look up how to include the license. We head over to the project’s build.gradle file and paste this line which is the path to our SDK repository under buildscript and repositories.

maven { url 'https://artifactory.img.ly/artifactory/imgly' }

We then copy this line and paste it under dependencies, and thus, we have linked the SDK.

classpath 'ly.img.android.pesdk:plugin:6.0.8'

Next, we’re going to apply the PESDK plugin. So, we find the app’s build.gradle folder and paste this line under the other plugins.

apply plugin: 'ly.img.android.pesdk'

Now, we’re going to insert the PhotoEditor SDK’s configuration that contains all tools. We copy this block and paste it under the PESDK plugin. It is important that the license naming here matches the name of the license file we previously imported.

pesdkConfig {

    licencePath "LICENSE" // Name of the Licence file in the asset folder

    // If you use another supportLibVersion ('com.android.support'), change this version here to update your own supportLibVersion
    supportLibVersion "27.1.1"

    // Define the modules you need
    modules {
        // Add all the backend modules you need
        include 'ly.img.android.pesdk.operation:text'
        include 'ly.img.android.pesdk.operation:frame'
        include 'ly.img.android.pesdk.operation:focus'
        include 'ly.img.android.pesdk.operation:brush'
        include 'ly.img.android.pesdk.operation:camera'
        include 'ly.img.android.pesdk.operation:filter'
        include 'ly.img.android.pesdk.operation:sticker'
        include 'ly.img.android.pesdk.operation:overlay'
        include 'ly.img.android.pesdk.operation:adjustment'
        include 'ly.img.android.pesdk.operation:text-design'
        include 'ly.img.android.pesdk.operation:abstract-sticker'

        // Add all the UI modules you need
        include 'ly.img.android.pesdk.ui.mobile_ui:core'
        include 'ly.img.android.pesdk.ui.mobile_ui:text'
        include 'ly.img.android.pesdk.ui.mobile_ui:focus'
        include 'ly.img.android.pesdk.ui.mobile_ui:frame'
        include 'ly.img.android.pesdk.ui.mobile_ui:brush'
        include 'ly.img.android.pesdk.ui.mobile_ui:filter'
        include 'ly.img.android.pesdk.ui.mobile_ui:camera'
        include 'ly.img.android.pesdk.ui.mobile_ui:sticker'
        include 'ly.img.android.pesdk.ui.mobile_ui:overlay'
        include 'ly.img.android.pesdk.ui.mobile_ui:transform'
        include 'ly.img.android.pesdk.ui.mobile_ui:adjustment'
        include 'ly.img.android.pesdk.ui.mobile_ui:text-design'


        // Add the serializer if you want
        include 'ly.img.android.pesdk:serializer'

        // Add asset packs if you want
        include 'ly.img.android.pesdk.assets:font-basic'
        include 'ly.img.android.pesdk.assets:font-text-design'
        include 'ly.img.android.pesdk.assets:frame-basic'
        include 'ly.img.android.pesdk.assets:filter-basic'
        include 'ly.img.android.pesdk.assets:overlay-basic'
        include 'ly.img.android.pesdk.assets:sticker-shapes'
        include 'ly.img.android.pesdk.assets:sticker-emoticons'
    }
}

Then, we copy the buildToolsVersion und paste it in the android category. And for the app to compile in the correct Java version, we copy this code block and paste it at the end of the android category.

compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

We hit “Sync Now” to save our settings. That often takes a while, so we sped it up for you.

We want to build a simple application that just consists of a single button that starts the photo editor. So first, we create a button. To do so, we navigate to the app folder and open the subfolders src, res and layout to find the activity_main (or whichever way you decided to name it). We click on “Text” to edit the file and add a button. For height and width, we enter wrap_content. Now, we set an ID for the button and call it sdk_button. We then set a display text for the button. For this tutorial, we’ll use the string app_name, but you, of course, can choose whichever string you like.

We return to the app folder and find the MainActivity in the subfolders src and java. We head over to the documentation, where we are presented with two options. We can either start the editor in camera mode or in the device’s gallery. We’ll go with the camera mode. As we’re writing our MainActivity in Kotlin, we choose the Kotlin code example and paste it in the MainActivity. We can overwrite the onCreate as it is also included in the code example.

companion object {
        const val PESDK_RESULT = 1
    }

    // Important permission request for Android 6.0 and above, don't forget to add this!
    override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<String>, grantResults: IntArray) {
        PermissionRequest.onRequestPermissionsResult(requestCode, permissions, grantResults)
        super.onRequestPermissionsResult(requestCode, permissions, grantResults)
    }

    override fun permissionGranted() {}

    override fun permissionDenied() {
        /* TODO: The Permission was rejected by the user. The Editor was not opened,
         * Show a hint to the user and try again. */
    }

    // Create an empty new SettingsList and apply the changes on this reference.
    private fun createPesdkSettingsList() = SettingsList().apply {

        // If you include our asset packs and use our UI you also have to add them to the UI,
        // otherwise, they are only available for the backend
        // See the specific feature sections of our guides if you want to know how to add your own Assets.

        getSettingsModel(UiConfigFilter::class.java).apply {
            setFilterList(FilterPackBasic.getFilterPack())
        }

        getSettingsModel(UiConfigText::class.java).apply {
            setFontList(FontPackBasic.getFontPack())
        }

        getSettingsModel(UiConfigFrame::class.java).apply {
            setFrameList(FramePackBasic.getFramePack())
        }

        getSettingsModel(UiConfigOverlay::class.java).apply {
            setOverlayList(OverlayPackBasic.getOverlayPack())
        }

        getSettingsModel(UiConfigSticker::class.java).apply {
            setStickerLists(
              StickerPackEmoticons.getStickerCategory(),
              StickerPackShapes.getStickerCategory()
            )
        }

        // Set custom camera image export settings
        getSettingsModel(CameraSettings::class.java).apply {
            setExportDir(Directory.DCIM, "SomeFolderName")
            setExportPrefix("camera_")
        }

        // Set custom editor image export settings
        getSettingsModel(EditorSaveSettings::class.java).apply{
            setExportDir(Directory.DCIM, "SomeFolderName")
            setExportPrefix("result_")
            savePolicy = EditorSaveSettings.SavePolicy.RETURN_ALWAYS_ONLY_OUTPUT
        }
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        openCamera()
    }

    private fun openCamera() {
        val settingsList = createPesdkSettingsList()

        CameraPreviewBuilder(this)
          .setSettingsList(settingsList)
          .startActivityForResult(this, PESDK_RESULT)
    }

    override fun onActivityResult(requestCode: Int, resultCode: Int, data: android.content.Intent) {
        super.onActivityResult(requestCode, resultCode, data)

        if (resultCode == Activity.RESULT_OK && requestCode == PESDK_RESULT) { // Editor has saved an Image.

            val resultURI = data.getParcelableExtra<Uri?>(ImgLyIntent.RESULT_IMAGE_URI)?.also {
                // Scan result uri to show it up in the Gallery
                sendBroadcast(Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE).setData(it))
            }

            val sourceURI = data.getParcelableExtra<Uri?>(ImgLyIntent.SOURCE_IMAGE_URI)?.also {
                // Scan source uri to show it up in the Gallery
                sendBroadcast(Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE).setData(it))
            }

            Log.i("PESDK", "Source image is located here $sourceURI")
            Log.i("PESDK", "Result image is located here $resultURI")

            // TODO: Do something with the result image

            // OPTIONAL: read the latest state to save it as a serialisation
            val lastState = data.getParcelableExtra<SettingsList>(ImgLyIntent.SETTINGS_LIST)
            try {
                PESDKFileWriter(lastState).writeJson(File(
                  Environment.getExternalStorageDirectory(),
                  "serialisationReadyToReadWithPESDKFileReader.json"
                ))
            } catch (e: IOException) { e.printStackTrace() }

        } else if (resultCode == Activity.RESULT_CANCELED && requestCode == PESDK_RESULT) {

            // Editor was canceled
            val sourceURI = data.getParcelableExtra<Uri?>(ImgLyIntent.SOURCE_IMAGE_URI)
            // TODO: Do something...
        }
    }

Right now, we are presented with quite some error warnings, but that’s no problem at all because we still need to implement the interface, as you can see here: PermissionRequest.Response. Here, we must choose the dedicated ly.img.android.pesdk.ui.utils and add .response at the end.

After that, we’re down to only two errors because we still need to integrate the correct directory. Again, we choose the dedicated ly.img.android.pesdk.backend.model.constant. And thus all errors are resolved.

If we were to start the app now it would automatically run the editor in camera mode. But we don’t want the editor to start automatically, hence the button we previously added. So, in order to change that, we scroll to the onCreate method and as you can see here, the app would instantly open the camera.

We create a new variable that we’re going to call sdkButton and set it equal with findViewbyId. We then set the ID to sdk_button, set an OnClickListener and paste openCamera beneath it.

And, we have a working little app that contains the PhotoEditor SDK. You could take a deeper dive into PhotoEditor SDK integration in other environments and frameworks in the official documentation put together by developers themselves.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

From 2D to 3D Photo Editing

Malte — Tue, 26 Jun 2018 00:00:00 GMT

Last November, we released Portrait, an iOS app that helps create amazing, stylized selfies and portraits instantly.

With over a million downloads and many more portrait images created, we feel that the idea and vision of Portrait was more than confirmed. The central component of Portrait is an AI that is trained to clip portraits from the background, a technique we are eager to further improve and refine. In fact, Portrait helped us to explore a novel technique for image editing, as we were able to leverage a new powerful data set in photography: depth data.

We began feeding our AI models with the depth data from the iPhone Xs TrueDepth camera and had one goal in mind: to infer depth information for portrait imagery, or bringing three-dimensionality into a two-dimensional photo. Along the way, we created a new architecture concept, that allows performance and memory improvements through modularizing and reusing neural networks.

In the following article, we’d like to present some of our results along with the insights we made.

The New Cool: Depth Data

The usage of depth data in image editing initially became available with the iPhone 7 Plus when Apple introduced ‘Portrait Mode’. By combining a depth map and face detection, the devices are able to blur our distant objects and backgrounds, mimicking a ‘bokeh’ or depth of field effect, which is well known from DSLRs cameras.

While the actual implementation varies, all major manufacturers nowadays offer a similar mode by incorporating depth data into their image editing pipeline. This is either achieved through the conventional dual or even triple camera on the back of a phone, dual-pixel offset calculations combined with machine learning or dedicated sensors like Apples TrueDepth module. In fact, for a modern flagship phone, some sort of depth based portrait mode is almost a commodity.

From a developers perspective, things look a little different: Depth data became a first-class citizen throughout the iOS APIs in iOS 11 and such data is now easily accessible on supported devices. Android users obviously have access to depth data as well, either by utilizing multiple cameras or by Googles dual-pixel based machine learning approach, seen in the newer Pixel 2 phones. But contrary to iOS, Android doesn’t yet offer a common developer interface to access such data. In fact, developers aren’t able to access any of the depth information Google or other manufacturers collected within their camera apps. This means developers would either need to implement the algorithm to infer depth from two images themselves or try to rebuild Googles sophisticated machine learning powered system. Neither of these options is practical and probably not even possible given the usual limitations to camera APIs.

So although being quite common, depth data isn’t as easily accessible for developers as one might think. Right now you’re out of luck on Android, dependent on hardware on iOS and even then limited to the 1.000$ flagship if you’re interested in depth for images taken with the front camera. And last but not least, across all devices and platforms, there is no way for you to generate a depth map for an existing image.

Deep Possibilities

Despite the restrictions, we decided to first explore the power of depth for image editing, as depth data provides many new exciting creative possibilities:

If we have a depth map for a given image, our editing possibilities are increased dramatically. Instead of a 2D image, a flat plane of color values, we suddenly have a depth value for each individual pixel, which translates into a 3D landscape highlighting distinct objects in the foreground and a clear indication of background.

Depth-aware Editing

Instead of relying on color and texture differences to determine fore- and background, one could literally edit these regions individually. This allows adjustments like darkening the background while lightening the foreground, which makes portraits ‘pop’. If we’d be able to generate a high-resolution depth map, we could easily replace the AI currently used in Portrait and would allow even more sophisticated creatives. Thanks to the new APIs, there are already some awesome iOS apps available that specialize in depth based editing. One famous example is Darkroom with their “depth-aware filters”:

Depth of Field Effects

As a depth of field or bokeh effect was the initial motivation for Apple to incorporate depth sensing technology, it is one of the most obvious applications. Depth is crucial for such an effect, as the amount of bluriness of any given region directly depends on its distance to the camera lens.

3D Asset Placement

As mentioned above, a depth map gives us a 3D understanding of the image. We’re able to tell if subject A is positioned in front of or behind subject B. This allows placement of digital assets like stickers or text in a ‘depth-aware’ fashion, but could also be used to apply ‘intelligent’ depth of field, e.g. a bokeh effect that ensures all faces are in focus.

Enter Deep Learning

Motivated by the possibilities enabled by depth maps, we were wondering if we could bring this magic to any type of portrait image. We consulted existing literature on depth inference and found various papers¹ and articles on the topic, some of which even presented results that seemed sufficient for our use cases. In our case, we didn’t need accurate, as in ‘this pixel is 30cm in front of the camera’, results, but we were only interested in getting the general distance relations correct. For us, knowing that region A was slightly behind but definitely way in front of region B was enough to generate a visually pleasing effect and by constraining our domain to portrait imagery, we were able to further reduce the tasks complexity.

Given our experience with deep learning and our current focus on introducing machine learning powered features to the PhotoEditor SDK, we immediately decided to tackle the new challenge with deep learning or more specifically convolutional neural networks. Having a huge dataset of image and depth map pairs available, made this choice even easier. We stuck to a system similar to our previous segmentation model but decided to put more emphasis on allowing the reuse of individual parts, as this would come in handy when adding additional features in the future. To achieve this, we created a new modularized neural network approach named Hydra, which will be presented in an upcoming blog post.

During development, we followed our tried and tested workflow of starting with a complex custom model, which is then tweaked and refined to match our performance requirements while maintaining the prediction quality we need. Once that was done, we had a fast and small model, trained on thousands of iPhone front camera selfies and capable of inferring high fidelity depth maps from a plain RGB image in under a second.

The Prototype

After creating a small model capable of inferring depth maps for any given portrait image, we immediately wanted to evaluate its performance in a ‘real-world’ environment. We decided to build a prototype that applies a depth of field effect to a portrait image, by using the model and its outputs. With our long-term goal of deploying the model to iOS, Android and the web in mind, we built the prototype using TensorFlowJS to explore this newly released library. Our browser demo consists of a minimal ‘Hydra’ implementation with individual modules, one for extracting features and one for the actual depth inference, which can both be executed individually.

While being optimized for performance and memory footprint, the trained weights of the model still add up to ~18MB, which we will improve by further fine-tuning or even applying pruning or quantization. Once the models are loaded, all further processing happens on the device though, so you may try out all the samples without worrying about your data plan.

Results

Seeing our vision come to life was quite a stunning experience. Suddenly our browser was able to perform a complex depth of field effect without the need for special hardware, manual annotations or anything else apart from our image. And the best part was manually moving the focal plane through the image, either by sliding or tapping on different regions. Although being trained on ‘just’ selfies the model handles turned heads, silhouettes and multiple people pretty well and isn’t as restricted to its domain as we initially expected.

And while our initial prototype is still weighing in at ~18MB, we’re certain to slim that down further in order to use the model in production. Performance wise we were very impressed with the TensorFlowJS inference speed. Even though everything is happening on the client side and is therefore dependent on the clients hardware, we saw inference speed below one second right of the bat and those greatly improved after the initial run, as the resources were already allocated. While not being immediately helpful for the depth inference part, this allowed us to further confirm our theory behind Hydra: Re-running inference once the necessary resources on the machine have been allocated greatly increases performance and might even allow real-time performance after an initial setup-time.

To summarise, we’re definitely eager to further explore the use of depth data in image editing and think we have found a way to overcome the access restrictions on different platforms and hardware with our custom model. Combined with our new Hydra approach we can see lots of potential features that will delight both our users and customers and we will keep you updated right here.

(1)
The papers we extracted most knowledge for our use case from were:
“Depth Map Prediction from a Single Image using a Multi-Scale Deep Network” (arXiv)
“Deeper Depth Prediction with Fully Convolutional Residual Networks” (arXiv)

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

The Photograph that Will Not Vanish.

Felix — Mon, 22 May 2017 00:00:00 GMT

The HP Sprocket paves the way for a whole new way of experiencing mobile photography. While pictures nowadays are either ephemeral or get stored away in digital vaults, HP breathes new life into material photographs by making mobile printing available to anyone. For this piece, we sat together with Carem Pereira, SCRUM Master for the sprocket Android team in Brazil, to recap the development of HP’s portable gem.

The HP Sprocket is a beautifully small and portable printer that instantly prints two by three-inch color photos — no cartridge or ink required. Peel the back of the photos and you can even stick them onto anything you like. “Our message is that printing can be easy and fun on the go. Snap, print, and play. You can take the sprocket anywhere, easily print photos on the spot and share them with your friends” says Carem Pereira. But creating a straightforward printing experience was only half the battle for the sprocket team, as Pereira explains: “From the first day on, editing features were planned to be a central part of the sprocket’s core experience. We wanted to give our users the ability to personalize and customize their snapshots before printing or sharing them.”

“The PhotoEditor SDK absolutely saved us a lot of time.”

Consequently, HP wanted to incorporate these essential features into their free sprocket companion app that connects the user’s mobile device to the printer via Bluetooth and allows for the management and printing of the pictures. “We’ve been working on a very tight schedule and had only three months until the release of our first MVP mid-September 2016”, Pereira explains, “in the beginning, we wanted to implement the editing features ourselves, but by the time we started the estimates, we realized that we’d eventually not be able to meet our goals. So, we were looking for a third-party solution.”

“One of our team members was already working with the SDK for another project, the HP Print Bot,” Pereira continues, “so, we compared PhotoEditor SDK with other solutions and found that it would be the best fit for us since it provides all the features that are crucial for our use case and we were already familiar with it. Also, it was of great importance to us that the look of the editor in sprocket matched the rest of the app and we saw that it would be very easy to accomplish that with the PhotoEditor SDK.”

The PhotoEditor SDK also leaves full control over content assets like stickers, fonts and filters, a feature critical to the HP team: “We want to stay relevant to our customers and one of the examples where we accomplish that is with our assets, like Stickers and Frames that are tailored for specific times and holidays. We have releases every two weeks that contain a new set of assets,” Pereira explains.

“It’s amazing how just a little gesture like a photograph can make such a difference in someone’s life!”

The HP Sprocket became an instant success. “The sprocket continues to be one of the great highlights for HP in consumer printing this year. The customers love using the printer, and we are already in millions of prints, we even sold out worldwide during the holidays last year, which was a great surprise for us,” says Carem Pereira.

But it’s not only sales figures that define its impact, as the story of Dom Russell and Seb Trevaskis exemplifies: The two Physiotherapy graduates brought the HP Sprocket to a Vietnamese orphanage where they educated, designed and implemented individualized rehabilitation programs for children who suffered from mental and physical disabilities due to the Vietnam War herbicide Agent Orange. The sprocket brought happiness to the orphanage and created everlasting memories for the children, as Dom Russell explains: “The sprocket was awesome, and the kids loved it! They would stick the photos on all their favorite possessions. Some would even just stare at them for hours as it may have been the first time they’d seen a physical copy of themselves. I think those photos will stay with them for life seeing the way they reacted when it got printed and the way they treated them as well. It’s amazing how just a little gesture like a photograph can make such a difference in someone’s life!”

Needless to say that sprocket was also an instant hit at our office. Playing around with the app that contains our editor was truly amazing and really let us grasp its potential. Using the sprocket on many different occasions gave us not only a hands-on experience of what our SDK is capable of but also of what needs further improvement or even what’s missing. Meanwhile our office is plastered with sprocket prints.

“PhotoEditor SDK was an essential asset for making things work with a top-notch standard.”

“We continue to get great feedback from our users on how straightforward and easy our experience is. And again, easy, that is the key,” Pereira says, “the PhotoEditor SDK was a vital and essential asset for making things work on time and with a top-notch standard. That was paramount to us,” Carem Pereira concludes. We couldn’t be happier about this.

Thanks for reading! To stay in the loop, subscribe to our Newsletter.