SSL / TLS setup for ZooKeeper

We have a ZooKeeper server and a java client application that connects to it. SSL has appeared in ZooKeeper since version 3.5.: https://issues.apache.org/jira/browse/ZOOKEEPER-2125

Server Tuning

There are several types of ZooKeeper server distribution. We’ll cover the standalone version as well as the Apache Kafka version. We need a configured server for start. The setup process is described in the article: here.

At the root of the distribution, create a folder called ssl for storing keystore and trustore.

Next, you need to add the following keys to the file with server settings zoo.cfg:

secureClientPort=<ssl_port>
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.keyStore.location=<zk_distr_path>/ssl/<server_keystore_file>.jks
ssl.keyStore.password=<server_keystore_pass>
ssl.trustStore.location=<zk_distr_path>/ssl/<server_truststore_file>.jks
ssl.trustStore.password=<server_truststore_pass>

Truststore is required. How to determine the path to the standard cacerts storage is described in this article. If the trustore name does not contain an extension, you must specify it explicitly:

ssl.trustStore.type = JKS

ZooKeeper CLI setup

Let’s duplicate the script <zk_distr_path>/bin/zkCli.sh or <zk_distr_path>/bin/zkCli.cmd and name it zkCliSSL. The last lines contain an enumeration of the parameters “-Dzookeeper.log.file = ${ZOO_LOG_FILE}”, etc.

We need to modify the script using this example. For the correct transfer to the next line at the end of each record, you must put the “\” symbol.

"$JAVA" \
"-Dzookeeper.log.dir=${ZOO_LOG_DIR}" \
"-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
"-Dzookeeper.log.file=${ZOO_LOG_FILE}" \
"-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty" \
"-Dzookeeper.client.secure=true" \
"-Dzookeeper.ssl.keyStore.location=<zk_distr_path>/ssl/<client_keystore>.jks" \
"-Dzookeeper.ssl.keyStore.password=<client_keystore_pass>" \
"-Dzookeeper.ssl.trustStore.location=<zk_distr_path>/ssl/<client_truststore>.jks" \
"-Dzookeeper.ssl.trustStore.password=<client_truststore_pass>" \
-cp "$CLASSPATH" $CLIENT_JVMFLAGS $JVMFLAGS \
org.apache.zookeeper.ZooKeeperMain "$@"

Let’s connect to the server to check the settings. We use the sh or cmd extension depending on the OS:

./bin/zkCliSSL.sh -server <zk_server_host>:<ssl_port>

If the ZooKeeper CLI does not connect to the server, check that the hostname matches the Common Name field of the certificate being used. Information on generating certificates can be found in this article. If re-issuing certificates is problematic, there is a workaround for test environments. It is described in this article.

ZooKeeper Java client setup

One way to customize the client is to set system properties via the System.setProperty(…) method:

System.setProperty("zookeeper.clientCnxnSocket", "org.apache.zookeeper.ClientCnxnSocketNetty");
System.setProperty("zookeeper.ssl.keyStore.location", "<zk_distr_path>/ssl/<client_keystore>.jks");
System.setProperty("zookeeper.ssl.keyStore.password", "<client_keystore_pass>");
System.setProperty("zookeeper.ssl.trustStore.location", "<zk_distr_path>/ssl/<client_truststore>.jks");
System.setProperty("zookeeper.ssl.trustStore.password", "<client_truststore_pass>");
System.setProperty("zookeeper.client.secure", "true")

This will force all ZooKeeper instances created in the Java application to use SSL with the specified settings.

If you need differentiated settings for different instances, you can use the ZKClientConfig class. It also has a setProperty method. It is necessary by analogy to set all the settings keys and then pass the configured object of this class to the client’s constructor:

ZooKeeper zooKeeper = new ZooKeeper("<connString>", <session_timeout>, <watcher>, <zk_client_config>)

The composition and order of parameters may vary from version to version.

Difference between distributions

Standalone

Server settings file: <zk_distr_path>/conf/zoo.cfg

CLI startup script: <zk_distr_path>/bin/zkCli.sh

As part of Apache Kafka

Server settings file: <zk_distr_path>/etc/kafka/zookeeper.properties.

It’s easier to make the CLI startup script by ourselves:

<zk_distr_path>/bin/kafka-run-class <ssl_keys_like_-Dzookeeper.client.secure> org.apache.zookeeper.ZooKeeperMain -server <zk_server_host>:<ssl_port>

If you still have any questions, feel free to ask me in the comments under this article, or write me on promark33@gmail.com.