The SSL support in librdkafka is completely configuration based, no new APIs are introduced, this means that any existing applications dynamically linked with librdkafka will get automatic SSL support by upgrading only the library.

This page serves as a simple HOWTO guide.

Also see Deploying SSL for Kafka

• Apache Kafka brokers: version 0.9.0.0 or later.

• Make sure the openssl and libssl-dev packages are installed.

• Reconfigure and rebuild librdkafka (./configure --reconfigure && make). Verify that WITH_SSL is set to 1 in config.h

• Create a convenient ssl directory where you execute the commands to create certificates and keys.

• All key and keystore passwords are abcdefgh. See gen-ssl-certs.sh for how to change this.

There is a script in librdkafka's tests/ directory called gen-ssl-certs.sh that automates the certificate and key generation steps outlined in the above link. It will be used throughout this HOWTO so make sure the script is in your $PATH (or equivalent).

If you dont have a proper CA certificate you can generate your own for testing.

gen-ssl-certs.sh ca ca-cert <the_ca_CN>

For each broker (let $BROKER be broker hostname), do:

gen-ssl-certs.sh -k server ca-cert broker_${BROKER}_ ${BROKER}

This is only needed if you want to authenticate clients on the broker.

The generated keys are standard OpenSSL PEM keys usable by librdkafka and any OpenSSL based client (and probably others as well).

For each client (let $CLIENT be client name), do:

gen-ssl-certs.sh client ca-cert client_${CLIENT}_ ${CLIENT}

This is only needed if you want to use the official Java clients that uses a Java keystore instead of standard PEM keys.

For each client (let $CLIENT be client name), do:

gen-ssl-certs.sh -k client ca-cert client_${CLIENT}_ ${CLIENT}

For each broker copy its keystore files (broker_${BROKER}_*.jks) to the broker node and add the following to the broker's server.properties configuration file (replace filenames as needed):

# SSL
ssl.protocol = TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type = JKS
ssl.keystore.location = broker_???????_server.keystore.jks
ssl.keystore.password = abcdefgh
ssl.key.password = abcdefgh
ssl.truststore.type = JKS
ssl.truststore.location = broker_????????_server.truststore.jks
ssl.truststore.password = abcdefgh
# To require authentication of clients use "require", else "none" or "request"
ssl.client.auth = required

Restart the brokers and monitor the log output to see that the configuration was accepted.

For each client copy its key files (client_${CLIENT}_*) and the public CA-cert to the client node and configure your librdkafka application with the following properties:

metadata.broker.list=at_least_one_of_the_brokers
security.protocol=ssl
# CA certificate file for verifying the broker's certificate.
ssl.ca.location=ca-cert
# Client's certificate
ssl.certificate.location=client_?????_client.pem
# Client's key
ssl.key.location=client_?????_client.key
# Key password, if any.
ssl.key.password=abcdefgh