Version: 8.4

IAM roles for service accounts

IAM roles for service accounts (IRSA) is a way within AWS to authenticate workloads in Amazon EKS (Kubernetes), for example, to execute signed requests against AWS services. This is a replacement for basic auth and is generally considered a best practice by AWS.

The following considers the managed services by AWS and provided examples are in Terraform syntax.

Aurora PostgreSQL

Aurora PostgreSQL is a managed AWS PostgreSQL–compatible service.

Setup

When using the Terraform provider of AWS with the resource aws_rds_cluster to create a new rational database (RDS) or Aurora cluster, supply the argument iam_database_authentication_enabled = true to enable the IAM roles functionality. See the AWS documentation for availability and limitations.

AWS policy

An AWS policy (later assigned to a role) is required to allow assuming a database user within a managed database. See the AWS documentation for policy details.

Create the policy via Terraform using the aws_iam_policy.

resource "aws_iam_policy" "rds_policy" {
  name = "rds-policy"

  policy = jsonencode({
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:region:account-id:dbuser:DbiResourceId/db-user-name"
         ]
      }
   ]
})
}

IAM to Kubernetes mapping

To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful.

module "aurora_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "aurora-role"

  role_policy_arns = {
    policy = aws_iam_policy.rds_policy.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
      namespace_service_accounts = ["aurora-namespace:aurora-serviceaccount"]
    }
  }
}

These two Terraform snippets allow the service account aurora-serviceaccount within the aurora-namespace to assume the user db-user-name within the database DbiResourceId. The output of the module aurora_role has the output iam_role_arn to annotate a service account to make use of the mapping.

Annotate the service account with the iam_role_arn output of the aurora_role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
  name: aurora-serviceaccount
  namespace: aurora-namespace

Database configuration

The setup required on the Aurora PostgreSQL side is to create the user and assign the required permissions to it. The following is an example when connected to the PostgreSQL database, and can also be realized by using a Terraform PostgreSQL Provider. See the AWS documentation for reference concerning Aurora specific configurations.

# create user and grant rds_iam role, which requires the user to login via IAM authentication over password
CREATE USER "db-user-name";
GRANT rds_iam TO "db-user-name";

# create some database and grant the user all privileges to it
CREATE DATABASE "some-db";
GRANT ALL privileges on database "some-db" to "db-user-name";

Keycloak

caution

IAM Roles for Service Accounts can only be implemented with Keycloak 21 onwards. This may require you to adjust the version used in the Camunda Helm Chart.

From Keycloak versions 21+, the default JDBC driver can be overwritten, allowing use of a custom wrapper like the aws-advanced-jdbc-wrapper to utilize the features of IRSA. This is a wrapper around the default JDBC driver, but takes care of signing the requests.

Furthermore, the official Keycloak documentation also provides detailed instructions for utilizing Amazon Aurora PostgreSQL.

A custom Keycloak container image containing necessary configurations is conveniently accessible on Docker Hub at camunda/keycloak. This image, built upon the base image bitnami/keycloak, incorporates the required wrapper for seamless integration.

Container image sources

The sources of the Camunda Keycloak images can be found on GitHub. In this repository, the aws-advanced-jdbc-wrapper is assembled in the Dockerfile.

Maintenance of these images is based on the upstream Bitnami Keycloak images, ensuring they are always up-to-date with the latest Keycloak releases. The lifecycle details for Keycloak can be found on endoflife.date.

Keycloak image configuration

Bitnami Keycloak container image configuration is available at hub.docker.com/bitnami/keycloak.

Kubernetes configuration

As an example, configure the following environment variables to enable IRSA:

# The AWS wrapper is not capable of XA transactions
- name: KEYCLOAK_EXTRA_ARGS
  value: "--db-driver=software.amazon.jdbc.Driver --transaction-xa-enabled=false --log-level=INFO,software.amazon.jdbc:INFO"

# Enable the AWS IAM plugin
- name: KEYCLOAK_JDBC_PARAMS
  value: "wrapperPlugins=iam"
- name: KEYCLOAK_JDBC_DRIVER
  value: "aws-wrapper:postgresql"

# Configure database
- name: KEYCLOAK_DATABASE_USER
  value: db-user-name
- name: KEYCLOAK_DATABASE_NAME
  value: db-name
- name: KEYCLOAK_DATABASE_HOST
  value: db-host
- name: KEYCLOAK_DATABASE_PORT
  value: 5432

# Ref: https://www.keycloak.org/server/configuration-metrics
- name: KEYCLOAK_ENABLE_STATISTICS
  value: "true"

# Needed to see if Keycloak is healthy: https://www.keycloak.org/server/health
- name: KEYCLOAK_ENABLE_HEALTH_ENDPOINTS
  value: "true"

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Helm chart

For a Helm-based deployment, you can directly configure these settings using Helm values. Below is an example of how you can incorporate these settings into your Helm chart deployment:

identity:
  keycloak:
    postgresql:
      enabled: false
    image: docker.io/camunda/keycloak:23 # use a supported and updated version listed at https://hub.docker.com/r/camunda/keycloak/tags
    extraEnvVars:
      - name: KEYCLOAK_EXTRA_ARGS
        value: "--db-driver=software.amazon.jdbc.Driver --transaction-xa-enabled=false --log-level=INFO,software.amazon.jdbc:INFO"
      - name: KEYCLOAK_JDBC_PARAMS
        value: "wrapperPlugins=iam"
      - name: KEYCLOAK_JDBC_DRIVER
        value: "aws-wrapper:postgresql"
    externalDatabase:
      host: "aurora.rds.your.domain"
      port: 5432
      user: keycloak
      database: keycloak

note

For additional details, refer to the Camunda 8 Helm deployment documentation.

Web Modeler

As the Web Modeler REST API uses PostgreSQL, configure the restapi to use IRSA with Amazon Aurora PostgreSQL. Check the Web Modeler database configuration for more details. Web Modeler already comes fitted with the aws-advanced-jdbc-wrapper within the Docker image.

Kubernetes configuration

As an example, configure the following environment variables

- name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
  value: software.amazon.jdbc.Driver
- name: SPRING_DATASOURCE_URL
  value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: SPRING_DATASOURCE_USERNAME
  value: db-user-name

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Identity

Since Identity uses PostgreSQL, configure identity to use IRSA with Amazon Aurora PostgreSQL. Check the Identity database configuration for more details. Identity already comes fitted with the aws-advanced-jdbc-wrapper within the Docker image.

Kubernetes configuration

As an example, configure the following environment variables

- name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
  value: software.amazon.jdbc.Driver
- name: SPRING_DATASOURCE_URL
  value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: SPRING_DATASOURCE_USERNAME
  value: db-user-name

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

OpenSearch

AWS OpenSearch is a managed OpenSearch service provided by AWS, which is a distributed search and analytics engine built on Apache Lucene.

note

As of the 8.4 release, Zeebe, Operate, and Tasklist are now compatible with Amazon OpenSearch 2.5.x. Note that using Amazon OpenSearch requires setting up a new Camunda installation. A migration from previous versions or Elasticsearch environments is currently not supported.

Setup

For OpenSearch, the most common use case is the use of fine-grained access control.

When using the Terraform provider of AWS with the resource opensearch_domain to create a new OpenSearch cluster, supply the arguments:

advanced_security_options.enabled = true
advanced_security_options.anonymous_auth_enabled = false to activate fine-grained access control.

Without fine-grained access control, anonymous access is enabled and would be sufficient to supply an IAM role with the right policy to allow access. In our case, we'll have a look at fine-grained access control and the use without it can be derived from this more complex example.

AWS Policy

An AWS policy, which later is assigned to a role, is required to allow general access to OpenSearch. See the AWS documentation for the explanation of the policy.

Create the policy via Terraform using the aws_iam_policy.

resource "aws_iam_policy" "opensearch_policy" {
  name = "opensearch_policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
        "es:DescribeElasticsearchDomains",
        "es:DescribeElasticsearchInstanceTypeLimits",
        "es:DescribeReservedElasticsearchInstanceOfferings",
        "es:DescribeReservedElasticsearchInstances",
        "es:GetCompatibleElasticsearchVersions",
        "es:ListDomainNames",
        "es:ListElasticsearchInstanceTypes",
        "es:ListElasticsearchVersions",
        "es:DescribeElasticsearchDomain",
        "es:DescribeElasticsearchDomainConfig",
        "es:ESHttpGet",
        "es:ESHttpHead",
        "es:GetUpgradeHistory",
        "es:GetUpgradeStatus",
        "es:ListTags",
        "es:AddTags",
        "es:RemoveTags",
        "es:ESHttpDelete",
        "es:ESHttpPost",
        "es:ESHttpPut"
        ],
        "Resource" : [
            "arn:aws:es:region:account-id:domain/test-domain/*"
        ]
    }
    ]
  })
}

IAM to Kubernetes mapping

To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful:

module "opensearch_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = "opensearch-role"

  role_policy_arns = {
    policy = aws_iam_policy.opensearch_policy.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
      namespace_service_accounts = ["opensearch-namespace:opensearch-serviceaccount"]
    }
  }
}

These two Terraform snippets will allow the service account opensearch-serviceaccount within the opensearch-namespace to generally access the AWS OpenSearch service for the test-domain cluster.

The output of the module opensearch_role has the output iam_role_arn to annotate a service account to use the mapping.

Annotate the service account with the iam_role_arn output of the opensearch_role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
  name: opensearch-serviceaccount
  namespace: opensearch-namespace

This step is required to be repeated for Tasklist and Zeebe, to grant their service accounts access to OpenSearch.

Database configuration

This setup is sufficient for OpenSearch clusters without fine-grained access control.

Fine-grained access control adds another layer of security to OpenSearch, requiring you to add a mapping between the IAM role and the internal OpenSearch role. Visit the AWS documentation on fine-grained access control.

There are different ways to configure the mapping within OpenSearch:

Via a Terraform module in case your OpenSearch instance is exposed.
Via the OpenSearch dashboard.
Via the REST API.

The important part is assigning the iam_role_arn of the previously created opensearch_role to an internal role within OpenSearch. For example, all_access on the OpenSearch side is a good candidate, or if required, extra roles can be created with more restrictive access.

Operate

Configure Operate to use the feature set of IRSA for the OpenSearch Exporter. Check the Operate OpenSearch configuration.

Kubernetes configuration

As an example, configure the following environment variables:

- name: CAMUNDA_OPERATE_OPENSEARCH_URL
  value: https://test-domain.region.es.amazonaws.com
- name: CAMUNDA_OPERATE_ZEEBEOPENSEARCH_URL
  value: https://test-domain.region.es.amazonaws.com
- name: CAMUNDA_OPERATE_DATABASE
  value: opensearch

Where the value is whatever the endpoint of your OpenSearch cluster is.

note

AWS OpenSearch listens on port 443 opposed to the usual port 9200.

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Tasklist

Configure Tasklist to use the feature set of IRSA for the OpenSearch Exporter. Check the Tasklist OpenSearch configuration.

Kubernetes configuration

As an example, configure the following environment variables:

- name: CAMUNDA_TASKLIST_OPENSEARCH_URL
  value: https://test-domain.region.es.amazonaws.com
- name: CAMUNDA_TASKLIST_ZEEBEOPENSEARCH_URL
  value: https://test-domain.region.es.amazonaws.com
- name: CAMUNDA_TASKLIST_DATABASE
  value: opensearch

Where the value is whatever the endpoint of your OpenSearch cluster is.

note

AWS OpenSearch listens on port 443 opposed to the usual port 9200.

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Zeebe

Configure Zeebe to use the feature set of IRSA for the OpenSearch Exporter. Check the Zeebe OpenSearch exporter configuration.

Kubernetes configuration

As an example, configure the following environment variables:

- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_AWS_ENABLED
  value: "true"
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_CLASSNAME
  value: io.camunda.zeebe.exporter.opensearch.OpensearchExporter
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_URL
  value: https://test-domain.region.es.amazonaws.com
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_BULK_SIZE
  value: "1"
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_INDEX_PROCESSMESSAGESUBSCRIPTION
  value: "true"

note

AWS OpenSearch listens on port 443 opposed to the usual port 9200.

note

Don't forget to set the serviceAccountName of the deployment/statefulset to the created service account with the IRSA annotation.

Troubleshooting

Instance Metadata Service (IMDS)

Instance Metadata Service is a default fallback for the AWS SDK due to the default credentials provider chain. Within the context of Amazon EKS, it means a pod will automatically assume the role of a node. This can hide many problems, including whether IRSA was set up correctly or not, since it will fall back to IMDS in case of failure and hide the actual error.

Thus, if nothing within your cluster relies on the implicit node role, we recommend disabling it by defining in Terraform the http_put_response_hop_limit, for example.

Using a Terraform module like the Amazon EKS module, one can define the following to decrease the default value of two to one, which results in pods not being allowed to assume the role of the node anymore.

eks_managed_node_group_defaults {
    metadata_options = {
        http_put_response_hop_limit = 1
    }
}

Overall, this will disable the role assumption of the node for the Kubernetes pod. Depending on the resulting error within Operate, Zeebe, and Web-Modeler, you'll get a clearer error, which is helpful to debug the error more easily.

Aurora PostgreSQL​

Setup​

AWS policy​

IAM to Kubernetes mapping​

Database configuration​

Keycloak​

Container image sources​

Keycloak image configuration​

Kubernetes configuration​

Helm chart​

Web Modeler​

Kubernetes configuration​

Identity​

Kubernetes configuration​

OpenSearch​

Setup​

AWS Policy​

IAM to Kubernetes mapping​

Database configuration​

Operate​

Kubernetes configuration​

Tasklist​

Kubernetes configuration​

Zeebe​

Kubernetes configuration​

Troubleshooting​

Instance Metadata Service (IMDS)​

Aurora PostgreSQL

Setup

AWS policy

IAM to Kubernetes mapping

Database configuration

Keycloak

Container image sources

Keycloak image configuration

Kubernetes configuration

Helm chart

Web Modeler

Kubernetes configuration

Identity

Kubernetes configuration

OpenSearch

Setup

AWS Policy

IAM to Kubernetes mapping

Database configuration

Operate

Kubernetes configuration

Tasklist

Kubernetes configuration

Zeebe

Kubernetes configuration

Troubleshooting

Instance Metadata Service (IMDS)