Using the Data Push API
The Data Push API takes advantage of micro-batching to transfer data into the Celonis Platform. This method involves creating, executing, and monitoring multiple smaller transfers of data, removing the need to either transfer huge chunks of data or inserting records individually. After the data is available in the platform, you can transform, analyze and work with it as you would with a Celonis-provided extractor.
For all available Data Push API references, see: Celonis Developer Documentation - Data Push API.
The following diagram provides an overview of how the data push API process works:
Create data push: This stage defines which data pool, data connection, and in which table your data should be pushed. You can also define a table schema and primary keys for delta loading when creating a data push Job.
Push chunk: You then push the data to the platform in chunks, using either parquet (preferred) or a .CSV file.
Submit push job: This stage is asynchronous.
List push jobs: You can then view and monitor the status of your push jobs by listing the push jobs.
Data push authentication
If you would like to connect and use the API, you can use one of the following authentication options:
OAuth client (recommended): ** Create an OAuth client in **Admin & Settings - Applications. The OAuth client needs to have the scope ‘integration.data-pools:data_push’ or ‘integration.data-pools’ to use the Data Push API. In addition, the client needs to be granted Data Push API permission on the corresponding data pool.
**API key: ** These are created within an individual user profile in your Celonis Platform, with the key’s permissions mirroring those of the user who created them. For security reasons, an API key is only displayed at the time it is created. Therefore you must create a new key if you no longer have access to any you create.
To create an API key, see: /document/preview/1774509#UUID-ae0c550f-ce88-ced3-67e4-5227a6941a70.
**Application key: ** Creating application keys enables you to give access and permissions to any applications or requests you create. Once created, an application key must then be granted the necessary permissions within your Celonis Platform (as by default, application keys are created without permissions).
To create an application key, see: /document/preview/1771003#UUID-f9e4f66a-2672-c740-e6c4-89c481528fc4.
Data push API limits
There is a 1 GB data limit on each data chunk, however there are no limits for the number of chunks you can push or their combined size.
The API to submit a push job is rate limited on a Celonis Platform team basis:
- 10 requests/second
- 250 requests/hour - The interval starts at the full hour (UTC based).
Data pool ID
Take the ID of the Data Pool which you want to populate with data. You can find it in the URL after entering your Data Pool in Data Integration.
If you also want the data grouped together with a specific Data Connection, note the ID of that data connection by taking it from the URL after clicking on it.