Amazon MSK with AWS IAM integration

Amazon Managed Streaming for Apache Kafka(MSK) for Apache kafka now supports integration with AWS Identity and Access Management (IAM) for authentication and authorization . This enables IAM policies to control the access , reading and writing of data . In this article I will put a brief about different authentication mechanism Amazon MSK support and what are the benefits of using IAM .We will set up a Amazon MSK cluster with IAM and access the MSK cluster with EC2 machine with IAM profile using the kafka binaries .

Before we start with the AWS IAM, let us look at the authentication mechanism Amazon MSK supports :

Lets have a quick review of the authentication process above :

PlainText authentication useful for training purpose . With mTLS its a good mechanism but it comes up with own way of implementation creation of certificate distribution etc. , SASL/SCRAM is a secured way of communication with user id and password but the credential need to be given to the client. and stored in plan text in the client.

AWS IAM authentication is a standard authentication in AWS , using IAM we can Authenticate : validate the identities Authorization : access management using policies and Audit : compliance requirement

We will use the following scenario to create a kafka cluster with IAM integration , will have EC2 machine with kafka binaries , Security group to connect to kafka cluster and IAM role : The below diagram depict the scenario and components will be used in the example :

Creating a Amazon MSK Cluster with AWS IAM :

If you are creating the MSK from the console then select the below options of IAM role

Security setting

If you are using cloud-formation you can add the below ClientAuthentication properties to enable the IAM

MSKCluster:
  Type: AWS::MSK::Clsuter
  Properties:
    ClientAuthentication:
      Sasl:
        Iam:
          Enabled: true
        Scram:
          Enabled: true

We need to create a Security Group allowing port 9098 which will be used with the client machine to connect the MSK cluster for IAM authentication

Create a role which will have the policy to create topic, read and write data to the topic, I have given a sample policy which can create a topic , write and read permission

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:CreateTopic"
      ],
      "Resource": [
        "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid/*",
        "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid/topic-iam/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": [
        "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid/*",
        "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid/topic-iam/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:ReadData"
      ],
      "Resource": [
        "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid/*",
        "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid/topic-iam/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:AlterGroup",
        "kafka-cluster:DescribeGroup"
      ],
      "Resource": [
        "arn:aws:kafka:region:account-id:cluster/cluster-name/*",
       ]
    }
  ]
}

Setting the Client Machine to Produce & Consumer:

Create a credential profile called msk_kafka on the EC2 , this will be used to connect the cluster.

[msk_kafka]
role_arn = arn:aws:iam::AccountId:role/producer-role
credential_source = Ec2InstanceMetadata

we will be using kafka binaries , you can follow these steps to set up the client machine :

Install the java client on your EC2 : “sudo yum install java-1.8.0”

Download the apache kafka:

“wget https://archive.apache.org/dist/kafka/2.2.1/kafka_2.12-2.2.1.tgz”

Extract the file where the the file from above downloaded : “tar -xzf kafka_2.12–2.2.1.tgz”

Download the jar from this location : aws-msk-iam-auth-1.1.1-all.jar which will give the class to work with the AWS IAM client . I have used the aws-msk-iam-auth-1.1.1-all.jar make sure you check the jar version before the export .

export CLASSPATH=/home/ec2-user/aws-msk-iam-auth-1.1.1-all.jar

Go to the kafka_2.12–2.2.1 directory.

Create kafka client configuration name “client.properties” with the following details :

security.protocol = SASL_SSL
sasl.mechanism = AWS_MSK_IAM
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName=”msk_kafka”;
sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler

Create a topic using the following command: (replace broker-address with the actual broker address from the Amazon MSK cluster)

bin/kafka-topics.sh --bootsstrap-server brokeraddress:9098 -- create --replication-factor 3 --partitions 3 --topic topic-iam --command-config client.properties

Write data to the topic :

bin/kafka-console-producer.sh --broker-list brokeraddress:9098 --producer.config client.properties --topic topic-iam  

Read data from topic :

bin/kafka-console-consumer.sh --bootsstrap-server brokeraddress:9098 --topic topic-iam --from-beginning --consumer.config client.properties 

Summary : IAM access control are not applied on Zookeper . The IAM authorizer class aws-msk-iam-auth can be used with the java client to connect MSK cluster using IAM .