Getting Started

The Data Push API provides an easy way to transfer data into Celonis. For pushing data, Celonis currently uses micro-batching. Instead of either inserting huge chunks all at once or inserting record by record, data meant to be pushed into the platform is structured into these so-called "micro-batches". As an exchange format, the commonly used Parquet or CSV file formata are used. The API provides an easy way of creating, executing, and monitoring micro-batch jobs/Data Push jobs. Once the data is available in the system, you can transform, analyze and work with it as you would with a Celonis-provided extractor.

1. Creating a Data Push job defines which Data Pool, Data Connection target table data should be pushed into.
2. You send data to the Celonis Platform by adding chunks, either Parquet (preferred) or CSV to a Data Push job.
3. Submit a Data Push job by calling the execute Data Push Job endpoint.
4. Check the status of the Data Push Job by calling the list Data Push Jobs endpoint.

There is a 1 GB data limit on each data chunk. In general, splitting data into chunks smaller than 1 GB makes sense. For a push job as a whole which can contain multiple chunks, there is no data limit.

The API to submit a push job is rate limited on a team basis. These limits are fixed to guarantee platform stability for all of our customers:

• 10 requests/second
• 250 requests/hour - The interval starts at the full hour (UTC based).

First you need to determine the URL for your team and cluster.

• The base URL for the Celonis API is: https://<team>.<cluster>.celonis.cloud/data-ingestion/api

To find the team and the cluster, please check the URL you use to access the Celonis Platform and retrieve the team and cluster from it.

For example, the base URL for dev team in US-2 cluster would be: https://dev.us-2.celonis.cloud/data-ingestion/api

You'll also need the Data Pool ID, for any further operations related to Data Push API. The Data Pool ID is displayed in the URL when you open a Data Pool in your Celonis team.

Since the Data Push API is a REST API, you will need to have a way to interact with it. There are multiple ways to do this, but to make things simple, we will use Postman, which is a very popular and free REST client.

If you already have a REST client, you can ignore this section.

Otherwise, go to the Postman Download page and download it for your Operating System. Once it's downloaded, install it.

Once installed, it should look like something like this:

To start using it, create an account or click on "Skip and go to the app".

OAuth 2.0 is the recommended solution for authenticating Celonis APIs. OAuth uses scopes to manage access to resources, which means that the OAuth clients can only access APIs allowed by the scopes they are configured to use.

See this page for more information about using OAuth 2.0 with your Celonis APIs.

The Celonis APIs formerly allowed the use of either a Bearer Token Authentication or AppKey to verify consumer access. OAuth 2.0 is the recommended solution and both the API Key and AppKey options are no longer being supported. Refer to this page for more information on migrating from either of these solutions to OAuth 2.0.

Once you have completed the previous steps, you can start making requests to the API. This example will use Postman for that.

1. Go to File > New... and select "HTTP Request". Once you do this, you should see something like this:

2. Enter the request URL next to the Send button. For this URL use the one from the "Determine the URL for your team" section. In our example, the URL is https://dev.us-2.celonis.cloud/data-ingestion/api.

3. Now you need to set your API or Application Key in order to authenticate our requests. To do this, click on the "Headers" tab beneath the URL you just entered.

4. In Key, enter "Authorization" and for Value, enter:

   • Bearer <YOUR_KEY> if you have an API Key
   • AppKey <YOUR_KEY> if you have an Application Key

The following example uses an Application Key:

Sample URL: https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/

A POST request should be made to create a Data Push Job, which will create a new Data Push Job and provide a unique ID for this newly created job as a result. This ID should be used for the remaining operations performed with regards to this particular Data Push Job. The request body must provide values for the mandatory fields.

• type: DELTA/REPLACE
• fileType: PARQUET/CSV
• targetTable: Desired TableName in DataLake
• dataPoolId: dataPoolId

Once the Data Push Job has been created, use its ID to upsert chunks/files. Multiple files can be uploaded in the context of a single Data Push job. We support PARQUET and CSV file as chunks to be uploaded.

URL: https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9/chunks/upserted

The POST request to execute the Data Push job will insert the data provided in the chunks of the Data Push job one by one. After the successful execution of a Data Push job, you will be able to see new records in the table.

URL: https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9

GET request (with URL: https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9) can be used to get the details of a particular Data Push job.

A Data Push job can be canceled by a POST request (with URL GET request, with URL: https://dev.us-2.celonis.cloud/data-ingestion/api/v1/data-push/bea4fb85-5d27-4336-b4db-b83207e8a03a/jobs/00e7706a-e2f7-49be-a7be-6098628774d9/cancel), if the Data Push job is not in a DONE/CANCELED/ERROR status.

Postman Collection for reference.