Skip to main content
Version: 8.9 (unreleased)

Camunda backup creation (Elasticsearch/OpenSearch)

Back up your Camunda 8 Self-Managed components and cluster.

Prerequisites​

The following prerequisites are required before you can create a backup.

PrerequisiteDescription
Set up a snapshot repository in the secondary datastore.

Configure a snapshot repository on the datastore itself:

Note: For Elasticsearch configuration with the Camunda Helm chart on AWS EKS using IRSA, see configuration example.

Configure component backup storage.

Configure the backup storage for the following components. This is also important for restoring a backup.

Configure Zeebe backup storage.Configure the backup storage for Zeebe. This is required regardless of your secondary storage choice. See Zeebe backup configuration.
note

You should keep the backup storage of the components configured at all times to ease the backup and restore process and avoid unnecessary restarts.

tip

You can use the same backup storage location for both Elasticsearch / OpenSearch snapshots and Zeebe partition backups, as long as different paths are configured:

  • Set the basePath for Zeebe.
  • Set the base_path for Elasticsearch / OpenSearch.

To learn more about how to configure these settings, refer to the prerequisites linked documentation above.

Considerations​

Why you should use backup and restore​

note

The Camunda 8.8 release introduces breaking changes for Operate and Tasklist.

note

If the Camunda applications cannot access Elasticsearch with cluster-level privileges, run the backup of Operate and Tasklist indices (steps 2 and 4 from the backup procedure) as a standalone application separate from the main application. For details, see the standalone backup application for Elasticsearch or OpenSearch.

The Camunda 8 components like the Orchestration Cluster and Optimize store data in various formats and across multiple indices in Elasticsearch or OpenSearch. Because of this distributed and interdependent architecture, creating a consistent and reliable backup requires coordination between the components.

For example, using Elasticsearch or OpenSearch's native snapshot capabilities directly does not produce a coherent backup. This is because Operate, Tasklist, and Optimize each manage their data across multiple indices, which cannot be reliably captured together without involvement from the components that understand their structure. For this reason, backups must be initiated through each component individually, using their built-in backup functionality.

The same principle applies to Zeebe. Backups must be scheduled through Zeebe to ensure a consistent snapshot of all partition data. Simply taking a disk-level snapshot of each Zeebe broker is not enough, as the brokers operate independently and data may not be aligned across them at the time of the snapshot. Since disk-level backups are not synchronized, this can lead to inconsistencies and invalid recovery points.

A complete backup of a Camunda 8 cluster using Elasticsearch or OpenSearch includes:

  • Backups of Web Applications (Operate, Tasklist), and Optimize (triggered through their APIs).
  • Backup of indices from Elasticsearch/OpenSearch containing exported Zeebe records.
  • A Zeebe broker partition backup (triggered through its API).

Because the data across these systems is interdependent, all components must be backed up as part of the same backup window. Backups taken independently at different times may not align and could result in an unreliable restore point.

warning

To ensure a consistent backup, you must follow the process outlined in this guide. Deviating from it can result in undetected data loss, as there is no reliable method to verify cross-component data integrity after backup.

Following the documented procedure results in a hot backup, meaning that:

  • Zeebe continues to process and export data.
  • Web Applications (Operate, Tasklist), and Optimize remain fully operational during the backup process.

This ensures high availability while preserving the integrity of the data snapshot.

Additional considerations​

A backup x of Camunda 8 using Elasticsearch or OpenSearch consists of backup x of Zeebe, backup x of Optimize, and backup x of Web Applications (Operate, Tasklist). The backup ID must be an integer and greater than the previous backups.

Optimize is not part of the Web Applications backup API and needs to be executed separately to successfully make a backup. Depending on your deployment configuration, you may not have Optimize deployed. It is safe to ignore the backup instructions for Optimize if it is not deployed.

breaking change

As of Camunda 8.8, the indexPrefix of Operate and Takslist must match. By default it is set to "". If overriden, it must set consistently across Operate and Tasklist.

breaking change

As of Camunda 8.8, configuring Operate and Tasklist with different repository names will potentially create multiple backups in different repositories.

breaking changes

As of Camunda 8.8, the /actuator endpoints for backups have been moved to /actuator/backupHistory (Web Applications) and /actuator/backupRuntime (Zeebe). The previous /actuator/backups endpoint is still active only if the applications are deployed standalone (each application is running in its own process).

About the backup process​

To create a backup you must complete the following back up process.

You can also optionally back up your Web Modeler data.

before you begin
  • To create a consistent backup, you must complete the backing in the outlined order.
  • You must complete the prerequisites before creating a backup.

Back up process​

Example API endpoint definition​

note

This will heavily depend on your setup, the following examples are based on examples given in the Management API in Kubernetes using either active port-forwarding or overwrite of the local curl command.

As noted in the Management API section, this API is typically not publicly exposed. Therefore, you will need to access it directly using any means available within your environment.

# only export the BACKUP_ID once as it has to stay consistent throughout the backup procedure
export BACKUP_ID=$(date +%s) # unix timestamp as unique always increasing ID

export ELASTIC_SNAPSHOT_REPOSITORY="camunda" # the name of your snapshot repository
export ELASTIC_ENDPOINT="http://localhost:9200"

export OPENSEARCH_SNAPSHOT_REPOSITORY="camunda" # the name of your snapshot repository
export OPENSEARCH_ENDPOINT="" # highly dependent on your environment

export ORCHESTRATION_CLUSTER_MANAGEMENT_API="http://localhost:9600"
export OPTIMIZE_MANAGEMENT_API="http://localhost:9620"

1. Soft pause exporting in Zeebe​

This step uses the management API.

This will continue exporting records, but not delete those records (log compaction) from Zeebe. This makes the backup a hot backup, as covered in the why you should use backup and restore.

curl -XPOST "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/exporting/pause?soft=true"
Example output
note

Yes, 204 is the expected result and indicates a successful soft pause.

{
   "body":null,
   "status":204,
   "contentType":null
}

2. Start a backup x of the web applications (Operate / Tasklist)​

This step uses the web applications management backup API.

curl -XPOST "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/backupHistory" \
   -H "Content-Type: application/json" \
   -d "{\"backupId\": $BACKUP_ID}"
Example output

{
   "scheduledSnapshots":[
      "camunda_webapps_1748937221_8.8.0_part_1_of_5",
      "camunda_webapps_1748937221_8.8.0_part_2_of_5",
      "camunda_webapps_1748937221_8.8.0_part_3_of_5",
      "camunda_webapps_1748937221_8.8.0_part_4_of_5",
      "camunda_webapps_1748937221_8.8.0_part_5_of_5"
   ]
}

3. Start a backup x of Optimize​

This step uses the Optimize management backup API.

curl -XPOST "$OPTIMIZE_MANAGEMENT_API/actuator/backups" \
   -H "Content-Type: application/json" \
   -d "{\"backupId\": $BACKUP_ID}"
Example output
{
   "message":"Backup creation for ID 1748937221 has been scheduled. Use the GET API to monitor completion of backup process"
}

4. Wait for backup x of the web applications (Operate / Tasklist) to complete​

This step uses the the web applications management backup API.

curl -s "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/backupHistory/$BACKUP_ID"
Example output
{
   "backupId":1748937221,
   "state":"COMPLETED",
   "failureReason":null,
   "details":[
      {
         "snapshotName":"camunda_webapps_1748937221_8.8.0_part_1_of_5",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:55:15.685+0000",
         "failures":[

         ]
      },
      {
         "snapshotName":"camunda_webapps_1748937221_8.8.0_part_2_of_5",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:55:16.288+0000",
         "failures":[

         ]
      },
      {
         "snapshotName":"camunda_webapps_1748937221_8.8.0_part_3_of_5",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:55:17.092+0000",
         "failures":[

         ]
      },
      {
         "snapshotName":"camunda_webapps_1748937221_8.8.0_part_4_of_5",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:55:17.293+0000",
         "failures":[

         ]
      },
      {
         "snapshotName":"camunda_webapps_1748937221_8.8.0_part_5_of_5",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:55:18.298+0000",
         "failures":[

         ]
      }
   ]
}

Alternatively as a one-line to wait until the state is COMPLETED using a while loop and jq to parse the response JSON.

while [[ "$(curl -s "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/backupHistory/$BACKUP_ID" | jq -r .state)" != "COMPLETED" ]]; do echo "Waiting..."; sleep 5; done; echo "Finished backup with ID $BACKUP_ID"

5. Wait for backup x of Optimize to complete​

This step uses the Optimize management backup API.

curl -s "$OPTIMIZE_MANAGEMENT_API/actuator/backups/$BACKUP_ID"
Example output
{
   "backupId":1748937221,
   "failureReason":null,
   "state":"COMPLETED",
   "details":[
      {
         "snapshotName":"camunda_optimize_1748937221_8.8.0_part_1_of_2",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:53:54.389+0000",
         "failures":[

         ]
      },
      {
         "snapshotName":"camunda_optimize_1748937221_8.8.0_part_2_of_2",
         "state":"SUCCESS",
         "startTime":"2025-06-03T07:53:54.389+0000",
         "failures":[

         ]
      }
   ]
}

Alternatively as a one-line to wait until the state is COMPLETED using a while loop and jq to parse the response JSON.

while [[ "$(curl -s "$OPTIMIZE_MANAGEMENT_API/actuator/backups/$BACKUP_ID" | jq -r .state)" != "COMPLETED" ]]; do echo "Waiting..."; sleep 5; done; echo "Finished backup with ID $BACKUP_ID"

6. Create a backup x of the exported Zeebe indices in Elasticsearch/OpenSearch​

You can create this backup using the respective Snapshots API.

By default, the old Elasticsearch or OpenSearch exporter creates indices with the prefix zeebe-record. If you configured a different prefix in the exporter, use that prefix instead.

This remains relevant if you run Optimize, which still relies on the former exporters.

The following uses the Elasticsearch snapshot API to create a snapshot.

curl -XPUT "$ELASTIC_ENDPOINT/_snapshot/$ELASTIC_SNAPSHOT_REPOSITORY/camunda_zeebe_records_backup_$BACKUP_ID?wait_for_completion=true" \
-H 'Content-Type: application/json' \
-d '{
      "indices": "zeebe-record*",
      "feature_states": ["none"]
      }'
Example output
{
   "snapshot":{
      "snapshot":"camunda_zeebe_records_backup_1748937221",
      "uuid":"1p_HdzKeTZ-zY-SN1LJ9VQ",
      "repository":"camunda",
      "version_id":8521000,
      "version":"8.18.0",
      "indices":[
         "zeebe-record_process_8.8.0_2025-06-03",
         "zeebe-record_job_8.8.0_2025-06-03",
         "zeebe-record_process-instance_8.8.0_2025-06-03",
         "zeebe-record_deployment_8.8.0_2025-06-03"
      ],
      "data_streams":[

      ],
      "include_global_state":true,
      "state":"SUCCESS",
      "start_time":"2025-06-03T08:05:10.633Z",
      "start_time_in_millis":1748937910633,
      "end_time":"2025-06-03T08:05:11.336Z",
      "end_time_in_millis":1748937911336,
      "duration_in_millis":603,
      "failures":[

      ],
      "shards":{
         "total":8,
         "failed":0,
         "successful":8
      },
      "feature_states":[

      ]
   }
}

7. Wait for backup x of the exported Zeebe indices to complete before proceeding​

Using ?wait_for_completion=true in the previous call, as outlined, ensures that the request only returns once the backup has completed. However, to double-check that the backup completed successfully, you can perform the following verification:

The following uses the Elasticsearch snapshot API to get the snapshot status.

curl "$ELASTIC_ENDPOINT/_snapshot/$ELASTIC_SNAPSHOT_REPOSITORY/camunda_zeebe_records_backup_$BACKUP_ID/_status"
Example output
{
   "snapshots":[
      {
         "snapshot":"camunda_zeebe_records_backup_1748937221",
         "repository":"camunda",
         "uuid":"1p_HdzKeTZ-zY-SN1LJ9VQ",
         "state":"SUCCESS",
         "include_global_state":true,
         "shards_stats":{
            "initializing":0,
            "started":0,
            "finalizing":0,
            "done":8,
            "failed":0,
            "total":8
         },
         "stats":{
            "incremental":{
               "file_count":0,
               "size_in_bytes":0
            },
            "total":{
               "file_count":8,
               "size_in_bytes":0
            },
            "start_time_in_millis":1748937910633,
            "time_in_millis":0
         },
         "indices":{
            "zeebe-record_process_8.8.0_2025-06-03",
            "zeebe-record_job_8.8.0_2025-06-03",
            "zeebe-record_process-instance_8.8.0_2025-06-03",
            "zeebe-record_deployment_8.8.0_2025-06-03"
         }
      }
   ]
}

8. Create a backup x of the Zeebe broker partitions​

This step uses the Zeebe management backup API.

curl -XPOST "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/backupRuntime" \
   -H "Content-Type: application/json" \
   -d "{\"backupId\": $BACKUP_ID}"
Example output
{
   "message":"A backup with id 1748937221 has been scheduled. Use GET actuator/backups/1748937221 to monitor the status."
}

9. Wait for backup x of Zeebe to complete before proceeding​

This step uses the Zeebe management backup API.

curl "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/backupRuntime/$BACKUP_ID"
Example output
{
   "backupId":1748937221,
   "state":"COMPLETED",
   "details":[
      {
         "partitionId":1,
         "state":"COMPLETED",
         "createdAt":"2025-06-03T08:06:06.246997293Z",
         "lastUpdatedAt":"2025-06-03T08:06:10.408893628Z",
         "checkpointPosition":1,
         "brokerVersion":"8.8.0"
      }
   ]
}

Alternatively as a one-line to wait until the state is COMPLETED using a while loop and jq to parse the response JSON.

while [[ "$(curl -s "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/backupRuntime/$BACKUP_ID" | jq -r .state)" != "COMPLETED" ]]; do echo "Waiting..."; sleep 5; done; echo "Finished backup with ID $BACKUP_ID"

10. Resume exporting in Zeebe using the management API​

curl -XPOST "$ORCHESTRATION_CLUSTER_MANAGEMENT_API/actuator/exporting/resume"
Example output
note

Yes, 204 is the expected result and indicates a successful resume.

{
   "body":null,
   "status":204,
   "contentType":null
}
warning

If any of the steps above fail, you might have to restart with a new backup ID. Ensure Zeebe exporting is resumed if the backup process force quits in the middle of the process.

(Optional) Back up Web Modeler data​

To create a Web Modeler data backup, refer to the official PostgreSQL documentation to back up the database that Web Modeler uses.

For example, to create a backup of the database using pg_dumpall, use the following command:

pg_dumpall -U <DATABASE_USER> -h <DATABASE_HOST> -p <DATABASE_PORT> -f dump.psql --quote-all-identifiers
Password: <DATABASE_PASSWORD>

pg_dumpall might ask multiple times for the same password. The database will be dumped into dump.psql.

note

Database dumps created with pg_dumpall/pg_dump can only be restored into a database with the same or later version of PostgreSQL, see PostgreSQL documentation.

info

You can restore a Web Modeler data backup.

Cleaning up backups​

Depending on your company’s backup policies (for example, retention periods and number of backups to keep) you should consider regularly cleaning up your old backups to reduce storage costs and efficiently manage resources.

You can use the delete backup APIs for each component to remove the associated resources from the configured backup storage. You will have to provide the same backup ID for all calls to remove it from all backup stores.

For Zeebe, you would also have to remove the separately backed up zeebe-record index snapshot using the Elasticsearch / OpenSearch API directly.

Primary storage retention​

note

This only affects the primary storage and does not interfere with Elasticsearch/OpenSearch backups.

With Camunda 8.9 you now have the option to enable a retention mechanism over primary storage (Zeebe's) backups. This will periodically delete backups from the configured blob storage based on the preconfigured retention window. Learn more about configuring backup retention here.