Creativity – IMG.LY Blog

What is Visual Prompting?

Jan — Tue, 29 Jul 2025 13:06:11 GMT

A New Paradigm for Creative AI, Built by IMG.LY

To say it’s trite to refer to the impact of AI in this or that domain as disruptive or groundbreaking would be an understatement. Yet, few areas have been as profoundly affected as the creative process. With just a text prompt, anyone can produce stunning images, remix visual styles, and explore design possibilities at a scale and speed never seen before. AI has inserted itself so quickly into this process that its gone from curious novelty to an essential part of the creator toolchain.

The more serious adoption we see, however, the more key limitations of today’s AI tooling come into focus: the prompt itself.

Text alone, for all its expressive power, struggles to capture the essence of visual intent. Most creative work doesn’t begin with a sentence it begins with a sketch, a layout, a mood board, or an arrangement of elements. Visual ideas are shared by pointing, placing, showing.

At IMG.LY, we have begun to think about better ways to direct AI for visual generation, the term we use is Visual Prompting.

Visual Prompting: the practice of composing a visual scene or layout as input for a generative model.

Instead of describing what you want with paragraphs of text, you show it directly using a canvas of images, text, spatial cues, and annotations. This visual composition then becomes the prompt for the AI to generate new content in return. It’s a more natural, intuitive, and powerful way to collaborate with AI, especially when integrated directly into the creative process.

Problem: the Chat Disconnect

The current generation of AI tools has largely been shaped by language-first interfaces. Whether it’s ChatGPT for writing or Midjourney for image generation, the assumption is the same: the user will type a descriptive prompt, and the AI will generate a result based on it.

But when it comes to design, this workflow quickly runs into friction. Visual ideas are inherently spatial and non-linear. Trying to express layout, balance, mood, or specific spatial relationships through text can feel like trying to describe a painting over the phone. It’s possible but unnecessarily cumbersome.

A designer might want to:

Indicate that a certain area in the image should be blue.
Replace a background with a texture sample.
Position a character precisely in a composition.
Annotate which parts of a scene to preserve or modify.

All of these are difficult to express fluently in text. But they’re effortless in a visual interface. The truth is: an image is worth more than a thousand words when prompting an image.

What Is Visual Prompting?

Visual Prompting is a multimodal approach to generative AI, where the input to the model is not just text, but a full visual composition: images, text, annotations, and layout.

Rather than prompting AI in isolation, the user builds their intent on a canvas. This might include:

Reference images that communicate mood or style.
Text blocks indicating desired copy or instructions.
Annotations pointing to specific areas with notes like “make this glow” or “replace this object.”
Spatial composition: where elements are arranged meaningfully to convey intent.

The visual prompt is then interpreted by a multimodal model such as OpenAI’s gpt-image-1 to generate new visual content that reflects not only the textual description, but also the visual context.

How Visual Prompting Works in CE.SDK

About time for an example. As part of our recent AI released we demoed how to use OpenAIs gpt-image-1 model to build visual prompting into CreativeEditor SDK (CE.SDK).

Here’s what the process looks like inside CE.SDK:

Compose Visually: The user creates a layout with reference content, uploaded images, icons, color schemes, design elements, placeholder text, and annotations. This composition represents the “prompt” in visual form.
Add AI Layers: With a single click, the user can trigger image generation using CE.SDK’s built-in AI plugin. The plugin sends the visual context (alongside any optional text input) to a multimodal model capable of interpreting both.
Refine and Iterate: Users can adjust the layout, reposition elements, change annotations, or layer in new references, then prompt again. Because the canvas is interactive and editable, the feedback loop is tight.
Build Up Complexity: Over time, users can layer generated images with manually designed components or other generated outputs, creating rich compositions that blend AI creativity with human direction.

This workflow turns the traditional prompt/response cycle into a conversation between the designer and the model, with the canvas acting as the shared language.

Who Is Visual Prompting For?

The use cases for Visual Prompting extend across industries:

Creative teams can go from reference to generation in seconds, iterating visually instead of wrangling prompts.
Marketing teams can generate regionalized or personalized creative variants from a shared layout.
Product designers can prototype in context, turning layouts into realistic screens without leaving the editor.
Storytellers and content creators can use annotated sketches to generate detailed illustrations or scene variations.
E-commerce platforms can give sellers the power to visually customize their brand materials with AI assistance.

In every case, Visual Prompting replaces friction with flow and text-based prompting with something more expressive, more reliable, and more fun.

Built for This: Multimodal Models and CE.SDK’s Plugin System

Visual Prompting is only possible because of two parallel advancements:

Multimodal AI models, such as OpenAI’s gpt-image-1, that can interpret both images and text, understand spatial relationships, and respond to annotated cues.
A flexible, composable editor SDK like CE.SDK, which enables the construction of visual prompts on a live canvas, and makes it easy to integrate AI models directly into the design flow.

Our SDK was built from the ground up to support AI-first creative workflows. Its plugin architecture allows you to add any model or API, image generation, video generation, captioning, text rewriting and use it natively inside the editor without the need to switch tools or copy/paste.

Generative AI’s full potential is only unlocked when it is embedded directly into the tools creatives use not siloed in chatbots or separate interfaces. Visual Prompting allows that embedding to go even deeper, aligning the mode of input (visual) with the desired output (visual).

Explore It Yourself

🎨 Try out Visual Prompting in our AI Editor demo
📘 Learn How to Integrate AI into CE.SDK
💬 Contact Us to Bring Visual Prompting to Your Product

My digital journey to #nfts

Vanessa — Fri, 26 Mar 2021 11:07:26 GMT

The Beginning

In my first years as a graphic designer, my world consisted mainly of paper, books and magazines. To give you an example, my bachelor and master thesis were both printed books (Yes - I am that old). So i was born analog, but i wanted to become more digital.

In the last three years, I have done my own personal digitalization and transformed from the analog design world into a digital one to the point that last weekend I created and launched my first crypto art - an NFT.

But First – What is an NFT?

I have read about NFTs before from time to time. Then, two weeks ago, the internet went wild about the news that a well-known New York gallery sold a digital painting worth $69 million for the first time. And this made me thinking a bit more about the subject again and I decided to join the hype. Let’s get rich!

But what is a NFT and how does it work? NFT stands for Non-Fungible Token which is secured by the Ethereum blockchain. 1 ETH is a fungible token – like bitcoin – a digital currency that is exchangeable. However, in contrast to ETH, a NFT can represent ownership of unique items. These unique items can be digital art like an image or a video, or any other digital asset. Within the process of creating an NFT, the digital asset will get connected to the token - which is called “minting”. Even though that image can easily be copied or recreated, the NFT can prove to a third-party that a specific person or entity owns this token. This makes digital art verifiable and unique.

What excites me so much about NFTs is that they make it possible to give digital art a real value. In addition it gives digital artists the opportunity to be perceived as artists in the same way as their colleagues who paint with brush and paint on canvas. So this means that digital art can finally step up to it’s physical counterpart. The beginning of a whole new era of art.

Currently it feels like that one can basically mint everything to an NFT and people go crazy about it. Why? Because people always liked to collect special and rare things like stamps, pokemon cards, oil paintings and so on - the list is basically endless.

Going digital

Circling back to my own story about transforming from an analog world to a digital one, the first step was to make my work more digital. Three years ago I gave up being self-employment and started working as an art director at IMG.LY. We do develop software for photo and video editing. Our first product was the Photoeditor SDK later followed by our Videoeditor SDK. However, last year we started defining our goals and begin the journey to transform our editor into a completely new tool that combines photo editing with automatic layouting. Automated and simplified processes make it possible for designer to focus more on the development of cool concepts instead of struggling with the technology itself.

Some time ago - while testing the functions of our tool, I created a series of pixel-art images. In my opinion this pixel-art style fits the topic of NFTs quite well, so that I created a small series with three matching images, which got the name “Life behind Pixels”.

One of these pictures was supposed to be my first NFT, but I had to realize that my digital transformation still has some gaps that I need to fill. There are various marketplaces that function like galleries, where one can upload their digital art for everyone to see and be verified as an official artist. I chose Rarible because it has a rather small community compared to the big ones like e.g. OpenSea and it is possible for everyone to join. Other marketplaces tend to select artists they work with based on the number of followers and how well-known they are in the community.

If you want to create an NFT – so called “minting” it, you have to connect your digital wallet to one of these platforms and pay a fee (gas) of about $60 in Ethereum. Unfortunately I don’t have a wallet or even digital money, so what can I do? I asked my boyfriend who works as firmware engineer in the 9elements cyber security department and who was of course already deep in the crypto universe and bang: My first NFT was minted. Now I’m even more fired up.

And although this NFT topic is on everyone’s lips in the digital scene right now, I recently noticed in a clubhouse talk with other designers that this hasn’t arrived everywhere yet.

And now what?

First we are curious to see what happens with our first NFT and we are trying to find out what else is possible with it. Instead of attaching unique information to a simple image file and making it unique, there is a way more potential behind it: NFTs can be minted in a way that allows them to react to their environment using their integrated code. For example they can show a different state in the morning, at noon and in the evening or their appearance changes due to the number of attached wallets.

Right now we’re looking at everything there is to know about NFT and blockchain to develop more crypto art and be part of the new era. Our plan is to create a new piece of crypto art every month. I’m trying to figure out what’s hot in crypto design right now. What do we need to stand out and be visually and technically unique?

And what are the options if you don’t have a friend working in the tech area?I think besides a monthly NFT, there could also be a monthly blogpost ;)

Also check out Instagram and Twitter for daily updates!
https://www.instagram.com/ellly_pix https://twitter.com/ellly_pix

A more technical deep dive can be found here.

Despite the fact that I am super thrilled about NFTs and that I finally generated my own, there is also a lot of controversial discussion going on about the climate impact of NFTs and the energy consumed to mint digital assets. If you like to follow up with this discussion, I put together some interesting links where you can also find some alternatives to the “minting” process:

Calculate the CO2 Footprint of your Ethereum address: https://carbon.fyi/
h ttps://www.greenpeace.org.uk/news/the-biggest-problem-with-carbon-offsetting-is-that-it-doesnt-really-work/
https://memoakten.medium.com/the-unreasonable-ecological-cost-of-cryptoart-2221d3eb2053
https://www.hicetnunc.xyz/
https://xtz.news/latest-tezos-news/getting-started-as-an-nft-artist-on-tezos-using-hicetnunc/

On the Democratization of Design

Daniel — Mon, 24 Jun 2019 00:00:00 GMT

Over the last century we have experienced the democratization of various domains of our lives:

Key drivers have been technological advancements and lower barriers to leveraging those technologies. In turn, technology became more accessible to more people resulting in products, services, and tools that improve our lives every day.

“Computers are a bicycle for the mind” — Steve Jobs

Tools, such as computers, allow us to finish more tasks in a shorter amount of time and with fewer resources. Even more importantly, said tools also rid us from boring and repetitive work, thus enabling us to shift our attention to exciting and more creative tasks, paving the way for us to experiment and create rather than repeat.

Evolve Design with better Tools

Proper visual design attracts, engages, and persuades; it helps convey stories and facilitates communication.

Nowadays, design in all its facets like advertising, social stories, product presentations, or merchandise is omnipresent in our daily lives. As there is no arguing that appealing and beautiful design has an impact on us and our surrounding, consequently there is also a growing demand for good design and easy-to-use tools.

“Use a picture. It’s worth a thousand words!” — Arthur Brisbamne

Although the importance of design is unquestioned, creating appealing designs remains hard for most of us, and there are many reasons for that.

First of all, design tools are still challenging to master, and while they empower professional designers to create stunning assets, they remain a closed book for most people.

Secondly, even if we leave most of the creational part to professionals, the average user is still unable to use existing designs as a basis to create new assets or refine and update small details. Even for the slightest changes, one has to get familiar with the inner workings of sophisticated design tools.

Also, when it comes to collaboration, the process with most tools is far from frictionless, to put it mildly. Still, most designs get shared as Photoshop, Sketch, or PowerPoint files, and there is no tool or platform that grants unhindered access for multiple stakeholders with varying degrees of experience with design solutions.

As today’s design processes regularly involve multiple stakeholders like designers, marketers, copywriters, or product engineers, one has to acknowledge that prevalent design solutions simply lack features and workflows that reflect daily routines.

Make Design Accessible through Technology

As we pointed out, the overall access to good visual design is still very limited, but with the growing need for compelling creatives in virtually every aspect of the digital and non-digital realm this shouldn’t be and it doesn’t have to. We believe in the democratization of design.

Better tools enable the democratization of design!

We believe that:

We need to overhaul design processes and methods to cope with the ever growing design needs and the ever-rising number of stakeholders.
Achieving elegant design must be made simple.
Design can be made accessible through technology.

The question remains: How do we achieve that?

We came to the conclusion that there are four key elements design tools should have that will make design more accessible:

Collaboration,
building blocks of intelligent design components,
automation of tedious design tasks, and
bringing together Editing, Compositing, and Layouting.

Collaboration

We hold as an opinion that design tools have to facilitate the collaboration between professionals and non-professionals, thus allowing an effective and frictionless design creation and adaption process. This includes direct collaboration on a design as well as the exchange of whole design templates, design components, and presets.

While some solutions already offer templates for content creation, we think that design components as a more modular approach are going to be a necessity to simplify the creation of new complex assets for everyone.

Building Blocks of Intelligent Design Components

If Lego™ taught us anything, then it is that creativity can flow when you base your work on a solid foundation.

That said, one has to admit that while the basic elements of design such as text, shapes, or images can be creatively combined in millions of ways to create stunning designs, they also can be used in a trillion ways to create underwhelming output.

Similar to modern UI design, complex building blocks will play a central role in the visual design process, allowing creators to be more efficient by removing repetitive tasks from the workflow and also allowing average users to create and adopt complex designs quickly. Intelligent design components will shift the focus to the composition by guiding the user in carefully chosen constraints.

Automation of Tedious Design Tasks

Speaking of intelligent building blocks, we believe that said blocks must not be static but adaptive to fit into an effective and frictionless design process. That may include the automation of certain design tasks like text layout or even abstracting away the complexity of image editing.

We envision that with the help of technology and specifically artificial intelligence, recurring and taunting tasks like for example, the separation of a picture into fore- and background can become totally interaction- and hassle-free. Even automatically choosing the right adjustments based on the image content will be possible.

Altogether, intelligent design components are a necessity and will be the next step in design automation, fostering a flexible and fail-safe adaption and creation process and thus ensuring that even without professional knowledge designs will be created faster and look better.

Bringing together Editing, Composition, and Layouting

Photo editing and layouting are both central to the design process. That is why, in a holistic approach, combining both will yield exciting new possibilities, improved visual consistency, and a reduced amount of friction while going back and forth between layout and photo editing.

Closing words

To sum it up: To democratize design, we need intelligent design tools and ecosystems that allow for the cooperation between all stakeholders, as well as a community that is open to share and learn from one another by building and sharing reusable and intelligent design components.

Rise above the Noise

Eray — Tue, 01 May 2018 22:00:00 GMT

A few years back, a close friend came by with a presentation she had prepared for a marketing campaign. Overenergized by this career-boosting assignment, she had nothing else on her mind for weeks, barely reachable and always stressed out. When she finally popped out of her little bubble, you could feel how proud she was, presenting me her latest draft with a gleam in her eyes. She knew that I’m a very visual guy, so she asked for my opinion.

While I could see the enormous amount of work she put into research, structure and storytelling, most of its brilliance was shaded by its design — or better, non-existing design. Oversaturated images, logos placed maliciously in the middle of nowhere, no concept of harmonic paddings, margins or color palettes. As a consequence, the visual presentation was an emotional flatline, to put it mildly. Slide after slide, I was in a constant struggle of ’should I tell her’ or not.

If you’re a designer, you might be familiar with this feeling. Your trained eyes can be a true pain in the ass, often triggering discomfort whenever something is unbalanced, or, let’s say, lacks a certain design attitude. In the digital world, with each ad, presentation, website or user interface you come across, there lurks an urge to set things straight, like with a crooked frame on the wall. While some might argue that this can quickly end in an unhealthy compulsiveness, to some degree this is important. Good design is an essential ingredient to the effectiveness of the underlying medium. After all, your design is the packaging for your content, for your message, and as such, it deserves equal attention. When done right, its true power unfolds with emotions; it can make people feel excited, and most importantly, **it can create desirability**. In the end, it’s the perfect tool to maximise the impact of your message and let your work truly stand out. But heck, enough with design philosophy.

Back at my friend’s computer, I suggested a few visual changes to the presentation, asking her to choose other fonts, pick different colours, change some pictures, and add more consistency to the layout. She did most of the changes herself following instructions and simple questions, which gave her a sense of ownership over the design.

Looking back, it was two things that were really frustrating me. On the one hand it’s my conviction that a marketeer should know the basic principles of design. You’re responsible for communication, and design is an important tool to shape it and helps to rise above the noise. On the other hand, it was also the tools she’s been using for the presentation and assets that were responsible for that mess. Tools made for designers, so a designer can craft the most beautiful visuals, seem powerless in the hands of a novice. Even business tools like Powerpoint do so little to educate and empower their users to create good design. Time after time, I’ve seen how marketeers, coders, product managers and business managers across industries had to rely on design departments or just struggled with their own mediocre design skills.

Who could have known that this frequent observation would help us pave the way for our own product. In our early days, when we built the first versions of the PhotoEditor SDK, we focused on helping developers to add basic photo editing functions to their products. With non-destructive editing and handling of multiple elements of the canvas we expanded our editor beyond photography — to a design tool.

Once we had all the essential functions in place, we started looking at every tool, from text over brush to adjustments from different angles. The most essential question was: how can this tool deliver beautiful output to those who lack design expertise? To those who don’t really know what character spacing is, or to the ones that don’t really understand what clarity does to an image and what it’s good for.

We believe in the vast potential of democratising design, and digital design starts with an editor.

Our journey towards that goal has just started. In April, we have launched a novel text design tool that makes text layouting a breeze. Our app Portrait showcases how we can generate beautifully designed portraits instantly and has already been downloaded over a Million times. There is much more to come, with tons of ideas, prototypes and data in our backpack.

Of course, making our vision happen is tremendously challenging from a UI perspective, and again, it is the quality of the design that will make the difference. As editing is a process that spans platforms, use cases and mediums — I consider this job a true boss fight for every designer. A huge challenge with a huge accomplishment in return, provided that we’re able to pull it off.

So, if you happen to be a designer, and feel the itching in your fingertips, just shoot us an email. Let’s start a conversation, and once you join our team let’s shape the future of design.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

Text ❤ Design

Felix — Thu, 12 Apr 2018 00:00:00 GMT

Introducing the Text Design Tool

There is power in words just as there is in design. In another advance in making good design more accessible to everyone, today, we are proud to give you a sneak peek at an upcoming product we created in addition to the PhotoEditor SDK: The Text Design Tool merges input text with typography, creating stunning designs for a multitude of use-cases.

While most of us are accustomed to the use of photo editing and enhancement tools, results often lack the desired appeal when it comes to typography and layout. Simply because most people don’t know about character or line spacing and other rules of typesetting and let’s be honest, why should they? But as well-designed text layout can add a lot to its expressiveness, we wanted to equip everyone with a tool that automates typesetting and layout, similar to apps like Typorama.

To transform our vision into a user-friendly tool, we first created recipes and set layout rules for different combinations of fonts with decoration elements. The Text Design Tool currently holds 16 layout designs for various use-cases. We worked close with the designers Tommi Gutscher and Ramona Schratt to conceptualize the designs and their different flavors. Through simple keyboard entries, you can populate the designs with your own words. The tool then lays out your text according to our recipes upon a single tap. You can then fine-tune your creative by choosing from 15 different text colors or by using the randomize functionality that shuffles the fonts, alignments and decorations until the creative lives up to your vision. You can even create a mask from your text that lets the background image shine through.

But that is not the end of the road, as, for the future, we will work with a wide range of artists and designers to expand our tool with a broad spectrum of new and exciting designs that can be applied to a variety of use-cases.

The Text Design Tool is currently available for iOS only. Versions for Android and HTML5 are going to be released in the next weeks.

Download our Photo Editor App for iOS to get a hands-on experience with the Text Design Tool.

**Thanks for reading! To stay in the loop, subscribe to our Newsletter.**

When Creativity meets A.I.

Eray — Thu, 16 Nov 2017 23:00:00 GMT

A new generation of A.I. algorithms, propelled by rising computational power, new hardware, and a shift in paradigms made its first notable impact in the creative world: The works of Gatys et al. and Krizhevsky et al. have not only gathered considerable public attention but have helped apps like Prisma to be adapted and used by millions. I strongly believe that this is merely the beginning. **With the help of machine learning, we will fine-tune, simplify, and automate creative processes and ultimately empower new techniques for design and content creation.**

We’ve been following this topic for quite some time now and have spent considerable effort in researching the opportunities of deep learning for our PhotoEditorSDK. After more than a year of research and development, today, we’re finally bringing one of our apps to beta. **Portrait** combines supervised deep learning with the visual power of our SDK. In a nutshell, Portrait makes creating beautifully designed portrait images as easy as taking a selfie. You turn your selfies into movie poster-like portraits, with styles ranging from double-exposure photography to stencil art. One may consider it as the next iteration of what Apple and Google recently brought to market with their new camera features.

We’ve now come a long way and gained invaluable insights on our journey so far. Not only did we get our hands dirty with countless training sessions and refinements to the neural net, but our first hand experience also helped to set expectation management right and to dismantle hype from substance. Most notably, it changed our product shaping process, making it more important than ever to foster strong ties between the product stakeholders and to share a common vision and goal everybody can get behind.

In the following I’d like to share the story of how we built the app and closed the gaps between roles of the stakeholders within this process.

Preface: Before Neural Networks were the Hot New Thing

My journey begins over ten years ago, while I was graduating in neuroscience. Back then, the idea of A.I. was just a vague promise. Artificial Neural Networks were too small, computers lacked the necessary power, and the results were certainly nice, but still too weak to compete with other traditional algorithms. Research felt stuck in tiny little specializations without really following a broader vision. Dazzled by its impracticability, my interest in Neural Networks slowly began to fade.

It took research on Neural Networks another six years to get back on my radar. At that time, I was leading several product developments at 9elements. When I learned about the work of DeepMind (now Google) I had a genuine feeling that this time, A.I. was ready for the limelight.

As we were in the course of building a library for image editing and computer vision — the PhotoEditorSDK, we realized how much neural nets could also affect the creative space, given its ability to abstract and formalize rules. What if there was a machine that could reproduce the common and dull tasks you have to do as an art director within a second? What if designers could get rid of repetitive and tedious activities that interrupt their creative flow?

But this topic isn’t something you’d learn in a week, obviously. Still, innovations cannot happen if you’re not willing to take a risk, so we decided to invest considerable time and resources into this technology.

From a product management’s perspective, this process is actually an anti-pattern: Usually, you wouldn’t want to start by finding the right purpose for a technology, instead you’d find the right technology for a purpose. I still believe that this is essentially the right approach, but sometimes you have to abandon your best practices and take a swim in uncharted waters. Consequently, we asked Malte, one of our iOS engineers, to spearhead our research and take a deep dive into this topic. We decided to start off with image segmentation as the first process that we wanted to optimize through machine learning. Masking and clipping sometimes can be a tedious tasks, and ultimately we wanted to reduce this process that can take several minutes to a single click.

Chapter 1: The Machine Engineer

Malte, who is a diligent engineer and — how convenient — a passionate photographer, started investigating some approaches that focused on image segmentation. You can read more about his journey in his article. Although he experimented with various neural networks and post-processing techniques, the resulting masks sometimes lacked the desired accuracy and wouldn’t have matched a user’s expectations. This was a first expected insight. As we want to deliver ready-to-use products to our customers, that don’t need any complex tweaking, this was something we had to fix. Our problems originated mostly from our rather ambitious goal to segment any type of object within an image. It would have required to train with vast data and to scale up the number of filters in our network. However, due to our on-device constraint, this would have killed our carefully crafted performance.

Therefore, we shifted this generalist approach to a specialized network for images of a certain domain that the model can be applied to. In hindsight, this seems quite obvious, as our rather small model would have never been able to cope with the amount of variations existing in ‘the real world’ anyway. So, we went back to the drawing board and started discussing which domain to focus on. That’s where we got suck; we struggled to find an obvious trend in our customers’ use cases or known photography platforms.

It was actually during his summer holiday, when Malte had the flash of genius. At a stop-over in Singapore, he noticed how the city was flooded with selfie-stick wielding tourists. The sheer amount of selfies taken at any public place in Singapore left him astonished and he realised that he just found the right domain. Selfies, and portraits in general, felt like an infinite datasource and prime use case for our image segmentation algorithm. Back home, we decided to focus on selfies and portrait-like photography.

Malte started searching for portrait datasets and found a collection of roughly 2000 portrait images collected from Flickr. Those were a great starting point and after a few training runs, he already reached satisfactory results, as the model was now capable to capture all available variations. At that point, we had a system at our hands that was able to segment portrait or selfie images in real-time on the device you’re capturing them with. This seemed like a great opportunity, but we didn’t want to stop just there. Releasing a prototype that can free a selfie from its background is nice, but doesn’t feel like something that would truly showcase how AI can make a difference in our creative process.

Chapter 2: The Art Director

This is where our Art Director Tommi, a renowned graphic artist and former sprayer, stepped in to explore what can be done with **a selfie, an accurate alpha mask and the image editing features from our PhotoEditor SDK**. When Tommi took the lead, I asked him to draft a vision, a creative direction for our app that combines all the tools and possibilities at our disposal.

Together, we started exploring portrait trends and unique imagery that would help us find a direction for our showcase. Soon, the walls of our meeting rooms and offices were plastered with inspirational works on portrait photography of all different kinds and styles. This visual catalogue kept inspiring us, although we weren’t sure on which style to settle in the end. It was when we could hardly find any more free spots on our walls and after looking at them for countless times that the idea struck:

What if we could enable users to turn their portrait to what we saw on these walls? And this, without actually having the design expertise they would normally be required to do so.

Instead of brooding over a completely new form of portraits, we could take all these styles and instantly realize them with the technology we had. From that point on, we flipped our process upside down. Instead of thinking about what the technology is capable of, or identifying a problem worth solving, we aimed for the creative output that we wanted our app to produce. While we started our venture with a technology, we now had visual results that we could work towards. The main question shifted from “What is our technology capable of?” to “How can we achieve this visual output with our technology?”

Tommi designed five lead graphics, so our team of engineers and designers could grasp what we ultimately wanted to achieve, using only a selfie and the features of our SDK.

Act 3: Closing ranks

With such a clear vision for our app, we started separating the wheat from the chaff, categorizing the portraits and understanding which operations of our SDK we had to combine, assemble and enhance to create these visuals.

What followed was a remarkable interplay across multiple stakeholders of our team. While we were always very vocal proponents of building strong relationship between product stakeholders, **the introduction of the AI layer actually glued our team further together**.

Our designers started to embrace the engineering perspective, playfully identifying both opportunities and constraints through the tech layer. At the same time, our engineers embraced the design vision and formalized it into code. Let me give you some examples:

While thinking of the UI, we understood that the transformation of a selfie into a graphical artwork required an immediate feedback for the user, so they can find a pose that works best with the respective artwork. Consequently, we optimized our networks for real-time processing, a true challenge that needed strong expertise in both iOS engineering and neural net architecture.

Our designs and recipes in turn had to be tweaked to gracefully allow for errors of our AI, because an error rate of 3% can still produce undesired artefacts and mask inaccuracies. We did that by using techniques that beautifully fringed edges of the portrait.

Altogether, the close cooperation, as well as countless meetings, feedback loops, and the continuous fine tuning of the code and underlying recipes is what brings us here.

All of this wouldn’t have happened if we hadn’t took the risk to invest in a rising technology in the first place. And all of this wouldn’t have been possible if it wasn’t for the exemplary cooperation between all the stakeholders. Portrait is a showcase of how technology can inspire and tie a team together. This, in the end, is absolutely necessary if we want to achieve the leaps we expect with AI. If you want to impact the creative space by introducing an AI layer to it, your engineers have to think like designers, or at least deeply understand their work.

The Road Ahead

Portrait is a first showcase and one step of many in our venture to wire several AI aspects deep into our SDK. On our journey, we’ve identified many more opportunities where we can help broader audiences to make creative work and design more accessible. Of course, we will also improve our models and networks with better and more data, always keeping in mind the aesthetic and visual output we’d like to achieve. We’ll keep you posted on our updates and next ventures into this exciting new era.

If you liked what you read, I’d encourage you to check out Portrait and our PhotoEditor SDK!

Thanks to my co-authors Malte & Felix!

Thanks for reading! To stay in the loop, subscribe to our Newsletter.

The IMG.LY Photo Roll

Felix — Thu, 21 Sep 2017 00:00:00 GMT

After today, we at IMG.LY are going to divert some of our energy to a new endeavour, the img.ly Photo Roll. We feel deeply rooted in the open-source movement and we know that the internet and most of the tools that we use today wouldn’t be nearly the same if it wasn’t for the awesome and creative work of countless people that shared their assets for free. Still today we heavily rely on open-source technology and creative work to give us guidance and inspire us to create new and unseen things.

Working on our product PhotoEditor SDK for the last two years gave us a lot of useful and valuable insights about photography and the various possibilities to create exceptional visual content. As some of us are enthusiastic photographers, it only seems logical to join the community of photographers and creative folks that enrich the internet with extraordinary and free content every single day. Since we have always gratefully used the work of other people, we would like to give something back to the community. Therefore, we are going to start sharing pictures we took around the globe via our Unsplash and Instagram account. Please feel free to download, use, modify and share them at your will. The photos that we’re going to post over the following weeks and months will also be available in our photo roll that ships with the PhotoEditor SDK.

We certainly hope that you’re going to make use of them. And we’d be happy to hear from you what you created with our pictures, and we’d love to see them in a new guise. After all, their real value only shows if someone is inspired by and can get creative with them.

Thanks to all the people that contribute to our initiative: Malte Baumann, Tommi Gutscher, Niklas Priddat, Cem Selcuk and Eray Basar. Below we have already compiled a few pictures that we are going to post over the next few days. Stay tuned.

Cheers!
Your friends at IMG.LY

Thanks for reading! To stay in the loop, subscribe to our Newsletter.