IAM roles for service accounts
IAM roles for service accounts (IRSA) is a way within AWS to authenticate workloads in Amazon EKS (Kubernetes), for example, to execute signed requests against AWS services. This is a replacement for basic auth and is generally considered a best practice by AWS.
The following considers the managed services by AWS and provided examples are in Terraform syntax.
Aurora PostgreSQL
Aurora PostgreSQL is a managed AWS PostgreSQL–compatible service.
Setup
When using the Terraform provider of AWS with the resource aws_rds_cluster to create a new rational database (RDS) or Aurora cluster, supply the argument iam_database_authentication_enabled = true
to enable the IAM roles functionality. See the AWS documentation for availability and limitations.
AWS policy
An AWS policy (later assigned to a role) is required to allow assuming a database user within a managed database. See the AWS documentation for policy details.
Create the policy via Terraform using the aws_iam_policy.
resource "aws_iam_policy" "rds_policy" {
name = "rds-policy"
policy = jsonencode({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:region:account-id:dbuser:DbiResourceId/db-user-name"
]
}
]
})
}
IAM to Kubernetes mapping
To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful.
module "aurora_role" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
role_name = "aurora-role"
role_policy_arns = {
policy = aws_iam_policy.rds_policy.arn
}
oidc_providers = {
main = {
provider_arn = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
namespace_service_accounts = ["aurora-namespace:aurora-serviceaccount"]
}
}
}
These two Terraform snippets allow the service account aurora-serviceaccount
within the aurora-namespace
to assume the user db-user-name
within the database DbiResourceId
.
The output of the module aurora_role
has the output iam_role_arn
to annotate a service account to make use of the mapping.
Annotate the service account with the iam_role_arn
output of the aurora_role
.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
name: aurora-serviceaccount
namespace: aurora-namespace
Database configuration
The setup required on the Aurora PostgreSQL side is to create the user and assign the required permissions to it. The following is an example when connected to the PostgreSQL database, and can also be realized by using a Terraform PostgreSQL Provider. See the AWS documentation for reference concerning Aurora specific configurations.
# create user and grant rds_iam role, which requires the user to login via IAM authentication over password
CREATE USER "db-user-name";
GRANT rds_iam TO "db-user-name";
# create some database and grant the user all privileges to it
CREATE DATABASE "some-db";
GRANT ALL privileges on database "some-db" to "db-user-name";
Keycloak
IAM Roles for Service Accounts can only be implemented with Keycloak 21 onwards. This may require you to adjust the version used in the Camunda Helm Chart.
For Keycloak versions 21+, the default JDBC driver can be overwritten, allowing use of a custom wrapper like the aws-advanced-jdbc-wrapper to utilize the features of IRSA. This is a wrapper around the default JDBC driver, but takes care of signing the requests.
The following example uses the mentioned aws-advanced-jdbc-wrapper
. Additionally, see the Keycloak documentation on overwriting the default JDBC driver.
A custom Docker image is required as there's currently no upstream image with all the configurations required.
Dependencies
Required are the following software.amazon.awssdk
artifacts for the aws-advanced-jdbc-wrapper
to work:
- regions
- rds
- aws-core
- sdk-core
- sts
- auth
- http-client-spi
- profiles
- endpoints-spi
- protocol-core
- aws-json-protocol
- json-utils
- aws-query-protocol
- metrics-spi
- third-party-jackson-core
- utils
The wrapper itself is available on GitHub.
Example usage
The following will use Gradle to retrieve the artifacts from Maven Central and build the final KeyCloak image with AWS IRSA support. This only requires Docker
on your machine as everything is done within Docker by using multi-stage builds.
Create a file called build.gradle
with the following content:
apply plugin: 'groovy'
repositories {
mavenCentral()
}
def jdbcversion = '2.2.2' // set to latest version of aws-advanced-jdbc-wrapper package
def awsSdkVersion = '2.20.107' // set to latest version of software.amazon.awssdk
dependencies {
implementation group: 'software.amazon.jdbc', name: 'aws-advanced-jdbc-wrapper', version: jdbcversion
implementation group: 'software.amazon.awssdk', name: 'apache-client', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'auth', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'aws-core', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'aws-json-protocol', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'aws-query-protocol', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'endpoints-spi', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'http-client-spi', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'json-utils', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'metrics-spi', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'profiles', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'protocol-core', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'rds', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'regions', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'sdk-core', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'sts', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'third-party-jackson-core', version: awsSdkVersion
implementation group: 'software.amazon.awssdk', name: 'utils', version: awsSdkVersion
}
task copyDependencies(type: Copy) {
from configurations.runtimeClasspath
into "lib"
}
Create a file called Dockerfile
with the following content in the same directory:
FROM gradle:jdk17-focal as lib
WORKDIR /home/gradle
COPY build.gradle /home/gradle
RUN gradle copyDependencies
FROM keycloak/keycloak:21.1 as builder
# Enable health and metrics support
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
# Configure a database vendor
ENV KC_DB=postgres
COPY --from=lib /home/gradle/lib /opt/keycloak/providers
WORKDIR /opt/keycloak
RUN /opt/keycloak/bin/kc.sh build
FROM keycloak/keycloak:21.1
COPY --from=builder /opt/keycloak/ /opt/keycloak/
ENV KC_DB=postgres
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
Run the following command or variation with docker buildx
to create and publish the required image.
docker build . --no-cache
Kubernetes configuration
As an example, configure the following environment variables
- name: KC_DB_DRIVER
value: software.amazon.jdbc.Driver
- name: KC_DB_URL
value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: KC_DB_USERNAME
value: db-user-name
# The AWS wrapper is not capable of XA transactions
- name: KC_TRANSACTION_XA_ENABLED
value: false
Don't forget to set the serviceAccountName
of the deployment/statefulset to the created service account with the IRSA annotation.
Web Modeler
As the Web Modeler REST API uses PostgreSQL, configure the restapi
to use IRSA with Amazon Aurora PostgreSQL. Check the Web Modeler database configuration for more details.
Web Modeler already comes fitted with the aws-advanced-jdbc-wrapper within the Docker image.
Kubernetes configuration
As an example, configure the following environment variables
- name: SPRING_DATASOURCE_DRIVER_CLASS_NAME
value: software.amazon.jdbc.Driver
- name: SPRING_DATASOURCE_URL
value: jdbc:aws-wrapper:postgresql://[DB_HOST]:[DB_PORT]/[DB_NAME]?wrapperPlugins=iam
- name: SPRING_DATASOURCE_USERNAME
value: db-user-name
Don't forget to set the serviceAccountName
of the deployment/statefulset to the created service account with the IRSA annotation.
OpenSearch
AWS OpenSearch is a managed OpenSearch service provided by AWS, which is a distributed search and analytics engine built on Apache Lucene.
Setup
For OpenSearch, the most common use case is the use of fine-grained access control
.
When using the Terraform provider of AWS with the resource opensearch_domain to create a new OpenSearch cluster, supply the argument advanced_security_options.enabled = true
and set advanced_security_options.anonymous_auth_enabled = false
to activate fine-grained access control
.
Without fine-grained access control
, anonymous access is enabled and would be sufficient to supply an IAM role with the right policy to allow access. In our case, we'll have a look at fine-grained access control
and the use without it can be derived from this more complex example.
AWS Policy
An AWS policy, which later is assigned to a role, is required to allow general access to OpenSearch. See the AWS documentation for the explanation of the policy.
Create the policy via Terraform using the aws_iam_policy.
resource "aws_iam_policy" "opensearch_policy" {
name = "opensearch_policy"
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"es:DescribeElasticsearchDomains",
"es:DescribeElasticsearchInstanceTypeLimits",
"es:DescribeReservedElasticsearchInstanceOfferings",
"es:DescribeReservedElasticsearchInstances",
"es:GetCompatibleElasticsearchVersions",
"es:ListDomainNames",
"es:ListElasticsearchInstanceTypes",
"es:ListElasticsearchVersions",
"es:DescribeElasticsearchDomain",
"es:DescribeElasticsearchDomainConfig",
"es:ESHttpGet",
"es:ESHttpHead",
"es:GetUpgradeHistory",
"es:GetUpgradeStatus",
"es:ListTags",
"es:AddTags",
"es:RemoveTags",
"es:ESHttpDelete",
"es:ESHttpPost",
"es:ESHttpPut"
],
"Resource" : [
"arn:aws:es:region:account-id:domain/test-domain/*"
]
}
]
})
}
IAM to Kubernetes mapping
To assign the policy to a role for the IAM role to service account mapping in Amazon EKS, a Terraform module like iam-role-for-service-accounts-eks is helpful:
module "opensearch_role" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
role_name = "opensearch-role"
role_policy_arns = {
policy = aws_iam_policy.opensearch_policy.arn
}
oidc_providers = {
main = {
provider_arn = "arn:aws:iam::account-id:oidc-provider/oidc.eks.region.amazonaws.com/id/eks-id"
namespace_service_accounts = ["opensearch-namespace:opensearch-serviceaccount"]
}
}
}
These two Terraform snippets will allow the service account opensearch-serviceaccount
within the opensearch-namespace
to generally access the AWS OpenSearch service for the test-domain
cluster.
The output of the module opensearch_role
has the output iam_role_arn
to annotate a service account to use the mapping.
Annotate the service account with the iam_role_arn
output of the opensearch_role
.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::account-id:role/role-name
name: opensearch-serviceaccount
namespace: opensearch-namespace
Database configuration
This setup is sufficient for OpenSearch clusters without fine-grained access control
.
Fine-grained access control
adds another layer of security to OpenSearch, requiring you to add a mapping between the IAM role and the internal OpenSearch role. Visit the AWS documentation on fine-grained access control
.
There are different ways to configure the mapping within OpenSearch:
- Via a Terraform module in case your OpenSearch instance is exposed.
- Via the OpenSearch dashboard.
- Via the REST API.
The important part is assigning the iam_role_arn
of the previously created opensearch_role
to an internal role within OpenSearch. For example, all_access
on the OpenSearch side is a good candidate, or if required, extra roles can be created with more restrictive access.
Operate
Configure Operate to use the feature set of IRSA for the OpenSearch Exporter. Check the Operate OpenSearch configuration.
Kubernetes configuration
As an example, configure the following environment variables:
- name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
value: https://test-domain.region.es.amazonaws.com
- name: CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_URL
value: https://test-domain.region.es.amazonaws.com
Where the value is whatever the endpoint of your OpenSearch cluster is.
AWS OpenSearch listens on port 443 opposed to the usual port 9200.
Don't forget to set the serviceAccountName
of the deployment/statefulset to the created service account with the IRSA annotation.
Zeebe
Configure Zeebe to use the feature set of IRSA for the OpenSearch Exporter. Check the Zeebe OpenSearch exporter configuration.
Kubernetes configuration
As an example, configure the following environment variables:
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_AWS_ENABLED
value: true
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_CLASSNAME
value: io.camunda.zeebe.exporter.opensearch.OpensearchExporter
- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_URL
value: https://test-domain.region.es.amazonaws.com
Don't forget to set the serviceAccountName
of the deployment/statefulset to the created service account with the IRSA annotation.
Troubleshooting
Versions used
This page was created based on the following versions available and may work with newer releases of mentioned software.
Software | Version |
---|---|
AWS Aurora PostgreSQL | 13 / 14 / 15 |
AWS JDBC Driver Wrapper | 2.2.2 |
AWS OpenSearch | 2.5 |
AWS SDK Dependencies | 2.20.x |
KeyCloak | 21.x |
Terraform AWS Provider | 5.9.0 |
Terraform Amazon EKS Module | 19.15.3 |
Terraform IAM Roles Module | 5.28.0 |
Terraform PostgreSQL Provider | 1.20.0 |
Instance Metadata Service (IMDS)
Instance Metadata Service is a default fallback for the AWS SDK. Within the context of Amazon EKS, it means a pod will automatically assume the role of a node. This can hide many problems, including whether IRSA was set up correctly or not, since it will fall back to IMDS in case of failure and hide the actual error.
Thus, if nothing within your cluster relies on the implicit node role, we recommend disabling it by defining in Terraform the http_put_response_hop_limit
, for example.
Using a Terraform module like the Amazon EKS module, one can define the following to decrease the default value of two to one, which results in pods not being allowed to assume the role of the node anymore.
eks_managed_node_group_defaults {
metadata_options = {
http_put_response_hop_limit = 1
}
}
Overall, this will disable the role assumption of the node for the Kubernetes pod. Depending on the resulting error within Operate, Zeebe, and Web-Modeler, you'll get a clearer error, which is helpful to debug the error more easily.