Backup and restore

note

The Camunda 8.8 release introduces breaking changes for Operate and Tasklist.

You can use the backup feature of Camunda 8 Self-Managed to regularly back up the state of all of its components (Zeebe, Operate, Tasklist, and Optimize) without any downtime (except Web Modeler, see the Web Modeler backup and restore documentation). In case of failures that lead to data loss, you can recover the cluster from a backup.

A backup of a Camunda 8 cluster consists of a backup of Zeebe, Operate, Tasklist, Optimize, and exported Zeebe records in Elasticsearch. Since the data of these applications are dependent on each other, it is important that the backup is consistent across all components. The backups of individual components taken independently may not form a consistent recovery point. Therefore, you must take the backup of a Camunda 8 cluster as a whole. To ensure a consistent backup, follow the process described below.

Configure backup store

To take backups, you must first configure backup storage.

Operate, Tasklist, and Optimize use Elasticsearch as backend storage and use the snapshot feature of Elasticsearch for backing up their state. Therefore, you must configure a snapshot repository in Elasticsearch.

Zeebe stores its backup to an external storage and must be configured before the cluster is started. Refer to Zeebe backup configuration for additional information.

Optimize uses its own backup process and needs to be executed separately to successfully make a backup. Depending on your deployment configuration, you may not have Optimize deployed. It is safe to ignore the backup instructions for Optimize if it is not deployed.

Backup process

The backup of each component and the backup of a Camunda 8 cluster is identified by an ID. This means a backup x of Camunda 8 consists of backup x of Zeebe, backup x of the web applications (Operate and Tasklist), and backup x of Optimize. The backup ID must be an integer and greater than the previous backups.

note

We recommend using the timestamp as the backup ID.

To back up a Camunda 8 cluster, execute the following sequential steps:

1. Soft pause exporting in Zeebe. See the Zeebe management API.
2. Trigger a backup x of the web applications. See how to take a web application backup.
3. Trigger a backup x of Optimize. See how to take an Optimize backup
4. Wait until the backup x of web applications is complete. See how to monitor a web application backup.
5. Wait until the backup x of Optimize is complete. See how to monitor an Optimize backup.
6. Take a backup x of the exported Zeebe records in Elasticsearch using the Elasticsearch Snapshots API.

PUT /_snapshot/my_repository/camunda_zeebe_records_backup_x
{
   "indices": "zeebe-record*",
   "feature_states": ["none"]
}

By default, the indices are prefixed with zeebe-record. If you have configured a different prefix when configuring Elasticsearch exporter in Zeebe, use this instead.

7. Wait until the backup x of the exported Zeebe records is complete before proceeding.
8. Take a backup x of Zeebe. See how to take a Zeebe backup.
9. Wait until the backup x of Zeebe is completed before proceeding. See how to monitor a Zeebe backup.
10. Resume exporting in Zeebe. See Zeebe management API.

note

If any of the steps above fail, you may have to restart with a new backup id. Ensure exporting is resumed if the backup process force quits in the middle of the process.

Restore

To restore a Camunda 8 cluster from a backup, all components must be restored from their backup corresponding to the same backup id:

1. Start Zeebe, Operate, Tasklist, and Optimize. (To ensure templates/aliases etc. are created)
2. Confirm proper configuration (such as shards, replicas count, etc.)
3. Stop Operate, Tasklist, and Optimize.
4. Delete all indices.
5. Restore the state of the web applications and Optimize.
6. Restore zeebe-records* indices from Elasticsearch snapshot.
7. Restore Zeebe.
8. Start Zeebe, Operate, Tasklist, and Optimize.