Sascha – IMG.LY Blog

How to Load Stripe Data into Google BigQuery

Sascha — Thu, 18 Jul 2024 10:04:26 GMT

At IMG.LY, we recognize that leveraging data is essential for driving innovation and growth. To optimize our data for reporting, we consolidate multiple data sources, including Stripe billing and financial data, into Google BigQuery.

IMG.LY is the leading provider of creative editing SDKs for video, photo, and design templates. While this article may not directly relate to media creation, we believe in empowering developers through knowledge sharing. Let’s dive in.

Until now, we’ve relied on Fivetran to fetch our data from Stripe and store it in Google BigQuery. Fivetran uses Stripe’s API, calling each endpoint, iterating over all resources, and storing the results in BigQuery (or any other supported data warehouse). While this generally works well, issues can arise. For instance, we sometimes create Stripe Subscriptions using inline pricing with the price_data parameter. This generates a new Price object in Stripe on-the-fly and immediately sets it to active: false. Consequently, the Price object is not returned by Stripe API’s price endpoint, leading to missing data in our warehouse. Although Fivetran’s support was exceptional in resolving this issue within a day, it highlighted a potential flaw in relying solely on ETL services for data extraction.

Recently, Stripe introduced Data Pipeline, its own service for transferring Stripe data into a data warehouse. This ensures complete, reliable data without needing a third-party service to read Stripe’s API. Additionally, you can receive test environment data and access several tables not available via the API. For a comprehensive summary of the available data, refer to Stripe’s official data schema.

Currently, Stripe supports only Snowflake and Amazon Redshift as data warehouses. However, they’ve recently added the option to deliver data as Parquet files into Google Cloud Storage (GCS). The next step for us was to import this data into Google BigQuery.

Setting Up Stripe Data Pipeline with Google Cloud Storage

Stripe is renowned for its excellent developer experience, and this beta feature is no exception. Enabling it within the Stripe Dashboard is quick, and the documentation is straightforward. After following the instructions and enabling the feature, it takes a while for data to appear in GCS. Once available, a complete data dump is provided every 6 hours, structured as follows:

At the root level, Stripe creates a folder representing the date and time of the latest transfer, e.g., 2024071600 (YYYYMMDDHH), representing the 12 am push on July 16, 2024.
One level deeper, there are two folders: livemode and testmode, representing live and test data, respectively.
Each folder contains one folder per data table, e.g., subscriptions or invoices. Additionally, a coreapi_SUCCESS file indicates successful data transfer to your GCS bucket and readiness for consumption.
Within the table folders are several Parquet files containing the actual data for each table.

Loading the Data from Google Cloud Storage into Google BigQuery

There are multiple ways to transfer data from GCS to BigQuery. We opted for the following approach:

Using Google Cloud Scheduler to publish a message to Google Pub/Sub every 6 hours at 1 am, 7 am, 1 pm, and 7 pm.
Creating a Google Cloud Function that listens for new messages on the above Pub/Sub topic. When a message is received, it triggers a Node.js script that loads the most recent data from GCS into BigQuery and deletes it from GCS.

Let’s delve into the details.

Create a Google Cloud Scheduler Job

First, create a new Cloud Scheduler job here with the following configuration:

Name: Choose a name for this job.
Region: The region is not crucial for this task; we used europe-west3 since most of our services are in Germany.
Frequency: We want the job to run every 6 hours at 1 am, 7 am, 1 pm, and 7 pm. Stripe publishes data every 6 hours, but it takes time to transfer it to GCS. We chose 1 hour later than Stripe’s push time, so our value is 0 1,7,13,19 * * *.
Timezone: Choose ‘Coordinated Universal Time (UTC)’.
Target type: Choose ‘Pub/Sub’.
Select a Cloud Pub/Sub topic: Select or create a new Pub/Sub topic using the default configuration. This is used to trigger the Cloud Function.
Message body: For this task, we don’t look at the contents of the message, as such the content of this value doesn’t matter. We opted for a simple load string.

Finally, click ‘Create’ to set up the scheduler. Now, a message is published to the selected Pub/Sub topic every 6 hours. Next, we need to respond to this message.

Create a Google Cloud Function

Create a Google Cloud Function triggered by Pub/Sub here with the following configuration:

Environment: Choose ‘2nd gen’.
Function name: Choose a name for this Cloud function.
Region: Select the region for the function, typically europe-west3 for our services.
Trigger type: Choose ‘Cloud Pub/Sub’.
Cloud Pub/Sub topic: Select the Pub/Sub topic created in the previous step.

Adjust the ‘Runtime, build, connections and security settings’ based on your Cloud setup and the required processing power for Stripe data. Generally, the following settings work well:

Memory allocated: ‘512 MiB’
CPU: ‘1’
Timeout: ‘540’
Minimum number of instances: ‘0’ (to ensure the function shuts down when not in use)
Maximum number of instances: ‘1’
Service account: Use or create a service account with permissions to access the GCS bucket where Stripe data is stored and the BigQuery datasets to load the data.
Ingress settings: Choose ‘Allow internal traffic only’.

Click ‘Next’ to provide the function’s code. Select:

Runtime: ‘Node.js 20’
Source code: ‘Inline Editor’
Entry point: loadStripeData

In the package.json, add the BigQuery and Cloud Storage Node.js packages:

In the index.js, add the following code:

This script does the following:

For each environment (live and test), it searches for the latest folder containing a coreapi_SUCCESS file.
For each table, it groups all related Parquet files and loads them into the BigQuery table using WRITE_TRUNCATE, which overwrites existing data. Note that the location is specified as EU, matching our BigQuery dataset and GCS bucket location. Adjust this parameter if your data is elsewhere.
If all files for an environment are loaded without errors, the files are deleted from GCS. This step is optional; if you prefer to keep a backup, you can omit this part.

Click ‘Deploy’ to deploy your Cloud function.

Create BigQuery Datasets

The final step is to create two datasets in Google BigQuery. Open Google BigQuery, click on the three dots next to your project’s name, and select ‘Create dataset’. Enter a name and choose a location matching your GCS bucket’s location. Repeat this process for the test dataset.

Conclusion

Following these steps will ensure your Stripe data is imported into BigQuery and automatically updated every 6 hours. However, as Data Pipeline for GCS is still in beta, there are some limitations. For example, the schema of the Parquet files lacks type annotations for timestamps, so all timestamps in BigQuery are represented as INTEGER instead of TIMESTAMP.

Additionally, some tables, such as subscription_item_change_events, are not currently transferred when syncing with Google Cloud Storage, although this issue is expected to be resolved soon. Meanwhile, we continue to use Fivetran in conjunction with the above method to sync Stripe data to Google BigQuery and plan to fully migrate to Data Pipeline once it exits the beta phase.

Thank you for reading!

3,000+ creative professionals gain exclusive access and hear of our releases first—subscribe to our newsletter and never miss out.

PE.SDK and VE.SDK 10 for Android and 11 for iOS Release

Sascha — Wed, 04 May 2022 12:46:21 GMT

We are thrilled to announce the release of PhotoEditor SDK and VideoEditor SDK 10 for Android and 11 for iOS. This new major version is packed with features and improvements.

This release is adding:

Custom Watermarks
Background Removal for Photos
Background Removal for Stickers
Custom Sticker Libraries
Giphy Integration
And More Updates for Android and iOS

Custom Watermarks

With our new support for custom watermarks, you can add your company logo or any other image to photo and video exports. You can place your watermark in either corner or the center of your photo and video. Additionally, you can specify the size of the watermark and determine its distance from the outline if you choose to place it near a corner. Your users will not be able to remove nor modify the watermark.

For more details on Custom Watermarks, take a look at our documentation for Android or iOS.

Background Removal for Photos

You have unwaveringly requested this feature: we are excited to offer the automatic removal of backgrounds for static content! Background Removal is fully running on-device and currently specializes in images containing a person only. PE.SDK enables this feature exclusively when it detects a person in the photo.

The removed background will turn transparent if your selected output supports transparency, such as PNG. In other cases, such as JPG, your background removal will be tinted black.

This feature is available on Android and iOS 15.0 and higher only. Easily enable Background Removal for Photos with our Android guide or iOS guide.

Background Removal for Stickers

A more common use case for automatic background removal is with stickers, so we’ve also added this functionality. Select a sticker including a person and tap the Remove BG button. Simple!

This feature is available on Android and iOS 15.0 and higher only. Easily enable Background Removal for Stickers with our Android documentation or iOS documentation.

Custom Sticker Libraries

We wanted to make it easier for you to provide your users with more sticker tool content. So far, you were able to create multiple sticker categories with different stickers that could either be part of your app or stored on a server. Additionally, you could enable our personal stickers feature on iOS or Android), giving your users the option to select stickers from their camera roll.

We have now added more ways to provide your users with stickers.

Android

You can now create sticker categories that do not present our sticker selection interface, but will instead load any arbitrary Fragment. That gives you complete control over the sticker selection screen, and you can use it to load and display any content. When the user selects any of your stickers, all you have to do is pass it back to the SDK with a callback. Get started with your Custom Sticker Library with our Android documentation.

iOS

You can now create sticker categories that do not present our sticker selection interface, but will instead load any arbitrary view controller that conforms to the StickerCollection protocol. That gives you complete control over the sticker selection screen, and you can use it to load and display any content. When the user selects any of your stickers, simply pass it back to the SDK using a delegate method.
Additionally, we’ve created a new StickerProviderCategory that you can pass any object conforming to the StickerProvider protocol. This category provides most of the features required to interact with sticker content providers, such as search and pagination. You can use this category to very quickly integrate any sticker service.

Find more information on Custom Sticker Libraries in the iOS documentation.

GIPHY Integration

This online database and search engine for funny GIFs and beautiful animations is a staple in many social applications. Easily integrate GIPHY into your app:

Android

Using our new Custom Sticker Library, we’ve also created an integration for GIPHY on Android. All you have to do is head over to GIPHY, create your API key, pass it to your GiphySettings and create a GiphyStickerCategoryItem. Your users will then immediately have access to all the content GIPHY has to offer.

For more details, visit our documentation on adding GIPHY for Android.

iOS

Using the new StickerProviderCategory, we added GIPHY support on iOS. All you have to do is head over to GIPHY, create your API key, and pass this to the GIPHY sticker category. Your users will then immediately have access to all the content GIPHY has to offer.

For more details, visit our documentation here.

More Updates for Android

Additionally, we have:

Increased the minimum API level for PhotoEditor SDK from 16 (Android 4) to 21 (Android 5), which ensures support for 98% of all phones in use.
Added support for headless video exporting.
Increased video playback stability.
Fixed a number of issues with degraded audio quality in videos.
And added the ability to choose a theme at runtime.

More Updates for iOS

Additionally, we have:

Increased our deployment target from iOS 9 to iOS 13.
Migrated all usage of OpenGL to Metal.
Added a better result API.
Added support for SwiftUI.
Enhanced our video export to display a progress indicator and the option to cancel the export.

Bringing Wide Color to PhotoEditor SDK

Sascha — Tue, 10 Jan 2017 00:00:00 GMT

Inspired by Mike Krieger’s great post on supporting wide color images in Instagram, I decided to challenge myself by introducing the same wide color support into our PhotoEditor SDK within a day.

Mike did an excellent job describing everything that is needed to support wide color images so I am not going to reiterate his explanation. Instead, here I’m going to share two findings I made in the process.

Image Export

Mike suggests that in order to preserve the color space while converting an UIImage into a JPEG, the UIImageJPEGRepresentation(_:_:) method has to be replaced with the new UIGraphicsImageRenderer.jpegData(withCompressionQuality:actions:) method. Checking the color profile of the generated JPEG by UIImageJPEGRepresentation(_:) I noticed that it seemed unnecessary and resulting in more work than needed. To verify this, I started Hopper and took a look at the internals of the new method. Here’s what it basically does:

It passes actions to UIGraphicsRenderer.runDrawingActions(_:completionActions:).
It obtains the resulting image using UIGraphicsImageRendererContext.currentImage.
It passes that image to UIImageJPEGRepresentation(_:_:).

As you can see, it makes use of UIImageJPEGRepresentation(_:_:) internally too, so there really is no need to go the extra mile. If you are actually doing any drawing within the actions block using wide color UIColors it absolutely makes sense to use Mike’s apoproach. But if you only want to convert an UIImage into a JPEG image, using the old method is far easier.

Core Image

We are using CIImage, CIFilter and CIContext for most of our processing. In our live preview we use a CIContext to directly render a CIImage into a GLKView and I’ve run into exactly the same problem as Mike has — namely not being able to get that view to be color space-aware. Using the offscreen buffer workaround that he suggested was going to be too much work though and also I didn’t want to sacrifice any performance.

After doing some research and digging further into the GLKit and Core Image assembly I found a nice solution. It looks like in order to render wide color images into OpenGL, OpenGLES 3 and a GLKViewDrawableColorFormat (that is currently not defined as an enum case) has to be used. The following code works like a charm for me:

let api: EAGLRenderingAPI
let colorFormat: GLKViewDrawableColorFormat

if #available(iOS 10.0, *) {
  if UIScreen.main.traitCollection.displayGamut == .P3 {
    api = .openGLES3
    colorFormat = GLKViewDrawableColorFormat(rawValue: 10)!
  } else {
    api = .openGLES2
    colorFormat = .RGBA8888
  }
} else {
  api = .openGLES2
  colorFormat = .RGBA8888
}

let context = EAGLContext(api: api)
let previewView = GLKView(frame: CGRect.zero, context: context!)
previewView.delegate = self
previewView.drawableColorFormat = colorFormat

This code creates a GLKView that is backed by a CAEAGLLayer with the following drawableProperties set:

[
 "EAGLDrawablePropertyRetained": 0,
 "EAGLDrawablePropertyColorFormat": EAGLColorFormatRGBA_XR10_64BPP
]

Unfortunately EAGLColorFormatRGBA_XR10_64BPP also doesn’t seem to be documented at this point, but my guess is that setting this color format on Instagram’s EAGLView would also solve their problem of making the view color space-aware.

Thanks to Core Image we can just switch to OpenGLES 3 without having to do any other modifications to our code. However, I am currently not sure if this is considered a private API use. I would greatly appreciate if anyone could shed some light on this.

I have not yet been able to test this on any device other than the iPhone 7 Plus, but I will update this post if I run into any problems with future tests.

Final Thoughts

Getting an app to support wide color images is definitely worth it and not as difficult as I had imagined. Unfortunately, it currently looks like iOS is still missing (public) support for wide color in some parts of its frameworks. I hope that Apple addresses these problems soon. Nevertheless we’re optimistic that we’ll be able to include wide color image support in the next release of our SDK. Feel free to check out our demo app in the App Store which we will update as soon as possible.

Sascha – IMG.LY Blog

How to Load Stripe Data into Google BigQuery

Setting Up Stripe Data Pipeline with Google Cloud Storage

Loading the Data from Google Cloud Storage into Google BigQuery

Create a Google Cloud Scheduler Job

Create a Google Cloud Function

Create BigQuery Datasets

Conclusion

PE.SDK and VE.SDK 10 for Android and 11 for iOS Release

Custom Watermarks

Background Removal for Photos

Background Removal for Stickers

Custom Sticker Libraries

Android

iOS

GIPHY Integration

Android

iOS

More Updates for Android

More Updates for iOS

Thanks for reading! Let us know what you think on Twitter, or stay in the loop with our Newsletter.

Bringing Wide Color to PhotoEditor SDK

Image Export

Core Image

Final Thoughts