For the complete documentation index, see llms.txt.
Skip to main content
Version: 8.10 (unreleased)

Restore a backup with the Restore Application (RDBMS)

Restore Zeebe partition data with the legacy Restore Application, a standalone app that runs on each broker node while all Camunda components are stopped, when using a relational database management system (RDBMS) as secondary storage.

This page is part of the RDBMS restore procedure. With Camunda 8.10 and later, you can use the Restore API instead, which does not require restarting the brokers. To compare the two, see choosing a restore approach.

After you ensure all prerequisites are met, the procedure consists of the following steps:

  1. Stop all Camunda components.
  2. Restore the RDBMS using your database vendor's native tools.
  3. Restore Zeebe from its primary storage backup using one of the restore options.
  4. Start all Camunda 8 components.

Prerequisites​

The following prerequisites are required before you can restore a backup:

PrerequisiteDescription
Camunda versionBackups can be restored using the same Camunda version they were created with, or up to one minor version newer. For example, a backup taken with 8.9.x can be restored with 8.9.x or 8.10.x.
Backup availableAt least one Zeebe primary storage backup is available in the configured blob store. See Create a backup.
Backup storageZeebe is configured with the same backup storage as outlined in the prerequisites.

1. Stop all Camunda components​

It is critical that no Camunda components (Zeebe, Operate, Tasklist, Optimize, Connectors) are running during the restore. Running components may propagate an incorrect cluster configuration, potentially disrupting cluster communication and data consistency.

2. Restore the RDBMS​

Restore the RDBMS from its backup using your database vendor's native tools. The restored database must contain the entire Camunda schema.

Skip this step if the RDBMS was already restored another way, for example as part of a wider disaster recovery procedure.

Complete this step before you restore Zeebe. Each restore option below reads the exporter position from the restored RDBMS to determine which primary storage backup to restore from, so the RDBMS must already be in its target state.

3. Restore Zeebe from its primary storage backup​

Camunda provides a standalone restore application that must be run on each node where a Zeebe Broker will be running. This is a Spring Boot application similar to the broker and can run using the binary provided as part of the distribution. The app can be configured the same way a broker is configured — via environment variables or using the configuration file located in config/application.yaml.

warning

Persistent volumes or disks must not contain any pre-existing data before restoring Zeebe. If data exists from a previous deployment, it must be cleared first. On physical tenant enabled environments, only the data directory for the specific tenant must be cleared.

warning

When restoring, provide the same configuration (node id, data directory, cluster size, and replication count) as the broker that will be running on this node. The partition count must be the same as in the backup.

The number of partitions backed up is also visible via the backup management API. If brokers were dynamically scaled between backup and restore, this is not an issue — as long as the partition count remains unchanged.

Restore options​

There are four restore options. In all cases, the restore app reads the exporter position from the restored RDBMS to ensure consistency between primary and secondary storage.

note

--backupId is mutually exclusive with --from/--to. Specifying both will result in error.

This is the recommended restore option. No additional parameters are required — the restore application automatically determines the best backup to use.

The restore app reads the exporter position from the restored RDBMS for each partition and identifies the most recent backup taken before that position. It then applies all subsequent backups in the range, restoring up to the latest available backup.

orchestration:
  enabled: true
  env:
    - name: SPRING_PROFILES_ACTIVE
      value: "restore"
    - name: ZEEBE_RESTORE
      value: "true"
    - name: CAMUNDA_DATA_PRIMARYSTORAGE_BACKUP_STORE
      value: "S3" # or GCS, AZURE, FILESYSTEM
    # Rest of the backup store configuration (bucket, region, etc.)
    - name: CAMUNDA_DATA_SECONDARY_STORAGE_TYPE
      value: "rdbms"
    # Rest of the RDBMS configuration (URL, username, password)

connectors:
  enabled: false
optimize:
  enabled: false

Kubernetes-specific behavior​

When restoring in Kubernetes using the official Camunda Helm chart, there are specific behaviors to be aware of.

Alternative startup override

An alternative approach to overwriting the startup behavior to restore the partitions:

orchestration:
  enabled: true
  command:
    - "/usr/local/camunda/bin/restore"
  env:
    - name: SPRING_PROFILES_ACTIVE
      value: "restore"
  # all the envs related to the backup store as above

The application exits after restore and Kubernetes restarts the pod, which appears as CrashLoopBackOff. This is expected behavior. The restore application does not restore state again once partitions are already restored to persistent disk.

After removing the temporary restore command, or unsetting ZEEBE_RESTORE and the related restore environment variables to restore Zeebe's default behavior, you may optionally restart the StatefulSet to ensure the changes take effect immediately. This can be done by scaling the StatefulSet down and back up, or by deleting the pods so they are recreated with the newly deployed revision.

tip

In Kubernetes, Zeebe runs as a StatefulSet, which is intended for long-running, persistent applications. Because StatefulSet pods are restarted automatically, restore-mode pods can appear in CrashLoopBackOff after a successful restore. Observe Zeebe Broker logs during restore. If a pod has already restarted, use --previous to view logs from the completed restore run:

kubectl logs <zeebe-pod-name> --previous

The restore app will not import or overwrite data again, but you may miss the first successful run if you are not observing logs actively.

Restoring a cluster with multiple Physical Tenants​

The Restore Application supports restoring multiple Physical Tenants, allowing you to restore one or more tenants without affecting the others on the node. It still must run on all brokers while the cluster is offline. By default, the default tenant is always selected as a restore target unless explicitly overridden.

To specify a single tenant to restore, use the ZEEBE_RESTORE_TENANT_ID environment variable or the corresponding CLI argument, --tenantId. Provide the rest of the restore options as you would for a normal restore.

warning

If restoring a single tenant, ensure that only the data directory for that tenant is cleared before starting the restore.

To perform a cluster-wide restore among all Physical Tenants simultaneously, use the ZEEBE_RESTORE_ALL_TENANTS environment variable or the corresponding CLI argument, --allTenants. Provide the rest of the restore options as you would for a normal restore.

During a cluster-wide restore, you can provide argument overrides for individual tenants. With Helm values, supply the overrides through extraConfiguration and include the overrides file in the Spring additional locations by setting the spring.config.additional-location property to point to the restore-overrides.yaml file.

orchestration:
  env:
    - name: SPRING_PROFILES_ACTIVE
      value: "restore"
    - name: ZEEBE_RESTORE
      value: "true"
    - name: ZEEBE_RESTORE_ALL_TENANTS
      value: "true"
    - name: ZEEBE_RESTORE_FROM_TIMESTAMP
      value: "<TIMESTAMP>"
    - name: ZEEBE_RESTORE_TO_TIMESTAMP
      value: "<TIMESTAMP>"

  extraConfiguration:
    - file: restore-overrides.yaml
      content: |
        override:
          tenanta:
            from: "<TIMESTAMP>"
            to: "<TIMESTAMP>"
          tenantb:
            backupId: [32]

Restore success or failure​

If restore was successful, the app exits with the log message Successfully restored broker from backup.

However, the restore will fail if:

  • There is no valid backup matching the secondary storage (the exporter position exceeds all available backups).
  • There is no valid backup within the specified time range.
  • The backup store is not configured correctly.
  • The configured data directory is not empty.
  • There is a gap in the backup range needed for restore (missing backups between the range start and the required checkpoint).
  • The exporter position in the RDBMS is missing for one or more partitions (when using RDBMS-aware restore).
  • Due to any other unexpected errors.

If the restore fails, you can re-run the application after fixing the root cause.

Data directory is not empty​

If the data directory is not empty, the restore will fail with an error message:

Broker's data directory /usr/local/camunda/data is not empty. Aborting restore to avoid overwriting data. Please restart with a clean directory

On some filesystems, the data directory may contain special files and folders that can't or shouldn't be deleted. In such cases, the restore application can be configured to ignore the presence of these files and folders. The configuration option zeebe.restore.ignoreFilesInTarget takes a list of file and folder names to ignore. By default, it ignores the lost+found folder found on ext4 filesystems. To also ignore .snapshot folders, set zeebe.restore.ignoreFilesInTarget: [".snapshot", "lost+found"] or the equivalent environment variable ZEEBE_RESTORE_IGNOREFILESINTARGET=".snapshot,lost+found".

4. Start all Camunda 8 components​

After both primary and secondary storage are restored, start all Camunda components. Ensure all components are configured to use the restored database instance and that the configuration matches the original deployment.

note

After starting the components, monitor the logs for any errors or warnings. Components will reconcile their state with the restored data, which may take some time depending on the size of the data. When using RDBMS-aware or time range restore, Zeebe re-exports events from the backup's checkpoint position up to its current state, bringing the RDBMS up to date.

(Optional) Restoring Optimize data​

If you previously backed up Optimize data, restore it independently using the standalone Optimize restore procedure. Optimize can be restored while the Orchestration Cluster restore is in progress or after it completes; the restore procedures are independent.

See back up and restore Optimize independently for the complete procedure.

(Optional) Restoring Camunda Hub data​

If you previously backed up Camunda Hub data, restore it using the same database tools.

See back up and restore Camunda Hub data for the complete procedure.