Getting Started

This page describes how to get started with the Data Push API in a simple way using Postman.

The Data Push API provides an easy way to transfer data into Celonis. For pushing data, Celonis currently uses micro-batching. Instead of either inserting huge chunks all at once or inserting record by record, data meant to be pushed into the platform is structured into these so-called "micro-batches". As an exchange format, the commonly used Parquet or CSV file formata are used. The API provides an easy way of creating, executing, and monitoring micro-batch jobs/Data Push jobs. Once the data is available in the system, you can transform, analyze and work with it as you would with a Celonis-provided extractor.

How it works

  1. Creating a Data Push job defines which Data Pool, Data Connection target table data should be pushed into.
  2. You send data to Celonis by adding chunks, either Parquet (preferred) or CSV to a Data Push Job.
  3. Submit a Data Push job by calling the execute Data Push Job endpoint.
  4. Check the status of the Data Push Job by calling the list Data Push Jobs endpoint.

API Limits:

  • There is a 1 GB data limit on each data chunk. In general, splitting data into chunks smaller than 1 GB makes sense. For a push job as a whole which can contain multiple chunks, there is no data limit.

The API to submit a push job is rate limited on a team basis. These limits are fixed to guarantee platform stability for all of our customers:

  • 10 requests/second
  • 250 requests/hour - The interval starts at the full hour (UTC based).

Prerequisites

  1. You need to have a Celonis team.
  2. You need an Application or API key for authentication before starting to initiate a Data Push Job.

Before starting

Determine the URL for your team

First you need to determine the URL for your team and cluster.

  • The base URL for the Celonis API is https://<team>.<cluster>.celonis.cloud/data-ingestion/api

To find the team and the cluster, please check the URL you use to access the Celonis EMS and retrieve the team and cluster from it.

For example, the base URL for dev team in US-2 cluster would be:
https://dev.us-2.celonis.cloud/data-ingestion/api

You'll also need the Data Pool ID, for any further operations related to Data Push API. The Data Pool ID is displayed in the URL when you open a Data Pool in your Celonis team.

Install a REST client

Since the Data Push API is a REST API, you will need to have a way to interact with it. There are multiple ways to do this, but to make things simple, we will use Postman, which is a very popular and free REST client.

If you already have a REST client, you can ignore this section.

Otherwise, go to the Postman Download page and download it for your Operating System. Once it's downloaded, install it.

Once installed, it should look like something like this:

Postman

To start using it, create an account or click on "Skip and go to the app".

Authentication

Each request to the API must be authenticated with a Celonis Application or API key. There are two ways of doing this:

Using an API key.

You can find out how to create an API key by following our User API Keys guide.

The Celonis API uses Bearer Token Authentication for verifying consumer access. The credentials must be sent in an Authorization header in the HTTP request. Credentials sent in the URL or body of the request will be ignored.

To authenticate using Bearer Token Authentication:

  1. Create the token in the EMS:
    MDg5MGVkNDktNjMwZC00ODdiLTkyNGItMjNmMzMxNjRmM2IwOkhNUVRMUis4SGh6NHhBY21Vck9GaWdkem5rYzBrb3p0N056WUM0bGlqczMM
  2. Include the string in the HTTP Authorization header formatted like this:
    Authorization: Bearer MDg5MGVkNDktNjMwZC00ODdiLTkyNGItMjNmMzMxNjRmM2IwOkhNUVRMUis4SGh6NHhBY21Vck9GaWdkem5rYzBrb3p0N056WUM0bGlqczMM

Using an Application API key (user independent one).

You can find out how to create an Application by following our Application API Keys guide.

To authenticate using AppKey Authentication:

  1. Create the Application in the EMS:
    YjNiZDI4MWUtYzAyNS00NTJjLTk1MjAtZjE4ZGEzMjM5MzIzOlVTRzVhNis2djMzaUtBbFNoNVlFMDBsSFA5OEJ2MDA2c0REK3N2OWIzVHdm
  2. Include the string in the HTTP Authorization header formatted like this:
    Authorization: AppKey YjNiZDI4MWUtYzAyNS00NTJjLTk1MjAtZjE4ZGEzMjM5MzIzOlVTRzVhNis2djMzaUtBbFNoNVlFMDBsSFA5OEJ2MDA2c0REK3N2OWIzVHdm

Authorization

The newly created Application Key is required to have necessary permission to be used for any Data Push API related operations. There are two methods to provide these permissions:

  1. On the "Admin & Settings -> Permissions" page, add the permissions by editing the "Data-Integration" tab.
  2. Add permissions for a particular Data Pool, by clinking the three dots to the right of that Data Pool.

Consuming the Data Push API

Once you have completed the previous steps, you can start making requests to the API. This example will use Postman for that.

  1. Go to File > New... and select "HTTP Request". Once you do this, you should see something like this:

Postman empty request

  1. Enter the request URL next to the Send button. For this URL use the one from the "Determine the URL for your team" section. In our example, the URL is https://dev.us-2.celonis.cloud/data-ingestion/api .
  2. Now you need to set your API or Application Key in order to authenticate our requests. To do this, click on the "Headers" tab beneath the URL you just entered. In key, type "Authorization" and for value, enter:
  3. Bearer <YOUR_KEY> if you have an API Key
  4. AppKey <YOUR_KEY> if you have an Application Key

In the following example, we have an Application Key: Postman authorization

Creating a new Data Push Job

Sample URL: https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/ A POST request should be made to create a Data Push Job, which will create a new Data Push Job and provide a unique ID for this newly created job as a result. This ID should be used for the remaining operations performed with regards to this particular Data Push Job. The request body must provide values for the mandatory fields. type : DELTA/REPLACE fileType : PARQUET/CSV targetTable : dataPoolId :

Postman create DataPushJob request

Upserting chunks to a Data Push Job

Once the Data Push Job has been created, use its ID to upsert chunks/files. Multiple files can be uploaded in the context of a single Data Push job. We support PARQUET and CSV file as chunks to be uploaded. URL : https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9/chunks/upserted

Postman chunk upsert request

Executing a Data Push Job

The POST request to execute the Data Push job will insert the data provided in the chunks of the Data Push job one by one. After the successful execution of a Data Push job, you will be able to see new records in the table. URL : https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9

Postman execute request

Getting a Data Push Job

GET request (with URL : https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9) can be used to get the details of a particular Data Push job.

Postman get data-push-job request

Cancel a Data Push Job

A Data Push job can be canceled by a POST request (with URL GET request (with URL : https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9/cancel), if the Data Push job is not in a DONE/CANCELED/ERROR status.

Postman Collection for reference