Setting Up Kafka MSK with EC2: Challenges and Solutions

Amazon Managed Streaming for Apache Kafka (MSK) simplifies the deployment of Kafka clusters, but integrating it with EC2 instances can come with challenges. In this blog, we’ll walk through the common issues faced while setting up Kafka MSK with EC2 and how we resolved them.

Problem 1: Authentication Failed with SASL Mechanism SCRAM-SHA-512

Error:

Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512

Solution:

• The issue was due to an incorrect secret configuration in AWS Secrets Manager.

• We created a properly formatted secret with the prefix AmazonMSK_.

• We associated the secret with the Kafka cluster using:

  aws kafka batch-associate-scram-secret \
    --cluster-arn <cluster-arn> \
    --secret-arn-list <correct-secret-arn> \
    --region <region>
  

• Verification via aws kafka list-scram-secrets confirmed the secret association.

Problem 2: Kafka Topics Not Visible After Authentication

Error:

kafka-topics.sh --list returned only system topics (__consumer_offsets, __amazon_msk_canary)

Solution:

• After setting up the correct authentication, we confirmed topic creation using:

  kafka-topics.sh --create --topic biometric-attendance \
    --bootstrap-server <broker-url>:9096 \
    --partitions 3 \
    --replication-factor 2 \
    --command-config client.properties
  

• Listing the topics again confirmed the presence of biometric-attendance.

Problem 3: Connectivity Issues from Windows PowerShell

Error:

Test-NetConnection -ComputerName <broker-url> -Port 9096
WARNING: TCP connect failed

Solution:

• Since MSK brokers are within a VPC, we needed an EC2 instance in the same VPC to act as a bridge.

• We used an SSH tunnel from Windows to connect securely:

  ssh -i <key-file.pem> -L 9096:<broker-private-ip>:9096 ec2-user@<ec2-public-ip>
  

• This allowed our local machine to communicate with Kafka inside the private network.

Problem 4: Kafka Producer Failing with Connection Timeout

Error:

Disconnecting from node -1 due to socket connection setup timeout.
Bootstrap broker <broker-url>:9096 disconnected

Solution:

• Ensured security.protocol=SASL_SSL and sasl.mechanism=SCRAM-SHA-512 in client.properties.

• Adjusted bootstrap.servers in the producer config:

  bootstrap.servers=<broker-url>:9096
  

• Restarted the Kafka producer, which successfully established the connection.

Conclusion

Setting up Kafka MSK with EC2 required careful authentication setup, proper network configurations, and ensuring that the producer and consumer were correctly configured. By resolving these issues step by step, we achieved a fully functional Kafka setup.

For more such insights, visit our website: www.strinosoft.com