Kafka SSL Client Authentication in Multi-Tenancy Architecture

Apache Kafka is the key product for not only messaging transformations but also real-time data processing, in addition to many other use cases. Architectures hosted inside the cloud claim to be secure in terms of communication and providing general security. But when it comes to the multiple client/consumer communication from a server/producer, Kafka provides in-built support for SSL as well as user-based authentication. In the below article, we will set up such an authentication mechanism step-by-step.

The solution is divided into three parts: 

1. SSL support for one or more brokers: Generate the key and the certificate for each machine in the cluster. You can use Java's KeyTool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.
2. Kafka Configurations (We used Kafka 2.11-2.3.0).
3. Running the whole set up.

Instructions to Install This Use Case

SSL support for one or more brokers. We will use Java's key tool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.

We are going to use one Kafka server and two clients (consumers). Also, here, we are using self-signed certificates. Otherwise, we need to have TrustStore and KeyStore JKSs for each server.

Points to note: 

1. Please create a folder for creating and keeping all cert files.
2. Please provide identical details and passwords for all. In my case I have used:

Plain Text

1

issuer=C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = 

Only the CN for client2 I have given a different user for testing purposes. Please generate your certificate carefully, else there will be a problem in the next part.

Shell

1

keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

2

3

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

4

5

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert 

6

7

keytool -keystore kafka.client1.truststore.jks -alias CARoot -import -file ca-cert

8

9

keytool -keystore kafka.client2.truststore.jks -alias CARoot -import -file ca-cert

10

11

keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file

12

13

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial 

14

15

keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert

16

17

keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

18

19

keytool -keystore kafka.client1.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

20

21

keytool -keystore kafka.client1.keystore.jks -alias localhost -certreq -file cert-file

22

23

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial

24

25

keytool -keystore kafka.client1.keystore.jks -alias CARoot -import -file ca-cert

26

27

keytool -keystore kafka.client1.keystore.jks -alias localhost -import -file cert-signed

28

29

keytool -keystore kafka.client2.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

30

31

keytool -keystore kafka.client2.keystore.jks -alias localhost -certreq -file cert-file

32

33

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial 

34

35

keytool -keystore kafka.client2.keystore.jks -alias CARoot -import -file ca-cert

36

37

keytool -keystore kafka.client2.keystore.jks -alias localhost -import -file cert-signed

Once everything is generated, you can see the generated files:

Plain Text

1

λ ls

2

3

ca-cert ca-key cert-signed kafka.client1.truststore.jks kafka.client2.truststore.jks kafka.server.truststore.jks ca-cert.srl cert-file kafka.client1.keystore.jks kafka.client2.keystore.jks kafka.server.keystore.jks

Kafka Configuration

Chang the server.properties file with below lines:

Properties files

1

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093

Also add:

Properties files

1

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

2

ssl.endpoint.identification.algorithm=

3

ssl.keymanager.algorithm=SunX509

4

ssl.keystore.location=<p>/kafka.server.keystore.jks

5

ssl.keystore.password=<p>

6

ssl.keystore.type=JKS

7

ssl.protocol=TLS

8

ssl.trustmanager.algorithm=PKIX

9

ssl.truststore.location=<p>/kafka.server.truststore.jks

10

ssl.truststore.password=<p>

11

ssl.truststore.type=JKS

12

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

13

allow.everyone.if.no.acl.found=true

14

ssl.client.auth=required

Then, we can create the necessary new files: client-ssl.properties, client-ssl1.properties, and client-ssl2.properties inside kafka_2.11-2.3.0\config.

Properties files

1

security.protocol=SSL

2

ssl.truststore.location=<p>/kafka.client1.truststore.jks

3

ssl.truststore.password=<p>

4

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

5

ssl.truststore.type=JKS

6

ssl.endpoint.identification.algorithm=

7

ssl.keystore.location=<p>/kafka.client1.keystore.jks

8

ssl.keystore.password=<p>

9

ssl.key.password=<p>

Properties files

1

security.protocol=SSL

2

ssl.truststore.location=<p>/kafka.client1.truststore.jks

3

ssl.truststore.password=<p>

4

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

5

ssl.truststore.type=JKS

6

ssl.endpoint.identification.algorithm=

7

ssl.keystore.location=<p>/kafka.client1.keystore.jks

8

ssl.keystore.password=<p>

9

ssl.key.password=<p>

Properties files

1

security.protocol=SSL

2

ssl.truststore.location=<p>/kafka.client2.truststore.jks

3

ssl.truststore.password=<p>

4

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

5

ssl.truststore.type=JKS

6

ssl.endpoint.identification.algorithm=

7

ssl.keystore.location=<p>/kafka.client2.keystore.jks

8

ssl.keystore.password=<p>

9

ssl.key.password=<p>

First, run Kafka and ZooKeeper:

Shell

1

 Zkserver

2

3

 .\bin\windows\kafka-server-start.bat .\config\server.properties

Then, open a new terminal and create a new topic:

Shell

1

.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --topic test1 --partitions 1 --replication-factor 1

2

3

.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181

After this, check the created certificate:

Shell

1

openssl s_client -debug -connect localhost:9093 -tls1

It will return the following details at the end. It means your certificate is generated properly.

Shell

1

Wtmz24ChQdgNcygKXLq1AHgDetoHz57hrx5f75/gh31nDdgHpv4xKyO40TSIH+8v

2

3

PqgvvrogH0lgLCwsJfqwPEJbWZjL6pvLsBfPB8NMICMXpL50ZA==

4

5

-----END CERTIFICATE-----

6

7

subject=C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c

8

9

issuer=C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c@technaura.com

10

11

 ---

12

13

Acceptable client certificate CA names

14

15

C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c@technaura.com

16

17

Client Certificate Types: ECDSA sign, RSA sign, DSA sign

We will run the set up for three different scenarios, i.e. without authentication, only server-side authentication, server, and client-side authentication.

Running the Whole Setup

The command for producing using console producer:

.\bin\windows\kafka-console-producer.bat --broker-list <broker host:port> --topic <topic-name> --producer.config config\<config file>

The command for consuming using console consumer:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server <server host:port> --topic <topic-name> --consumer.config config\<config file>