Apache ZooKeeper Configuration Management

Apache ZooKeeper as a Centralized Configuration Storage

If you're new to ZooKeeper then you're strongly suggested to take a look at its design concepts and architecture in the official documentation section. The documentation would be the best if you're already familiar with ZooKeeper's fundamentals and looking for a starting point to develop real-world applications using Apache ZooKeeper.

As you could find out from documentation and articles about ZooKeeper, it acts the best as a coordination service for distributed applications. In our concept, ZooKeeper is planned to be used as a centralized configuration data storage. However, despite the fact that ZooKeeper has data model that looks like a UNIX OS file system (ZNode can be interpreted as a "directory" that can have data associated with it), there are several constraints, which limits using ZooKeeper as an ordinary file system:

• Default ZNode size limit is 1MB. It can be increased by changing jute.maxbuffer Java system property for all of communicating ZooKeeper servers and clients. However, it is strongly discouraged to do that because it may cause performance drop and seriously disrupt operation of the whole system or even cause its malfunction. And another thing to remember - one megabyte limitation involves not only ZNode value, but also its key and a list of child node names.
• All the data is stored in the memory and duplicated on each server in ZooKeeper ensemble. This fact must be taken into account during the planning phase of the system development. Server machine should have enough RAM and correct JVM heap settings (max. 3/4 of the total amount of memory) to be able to operate properly for the most severe design conditions. Disk swapping would degrade ZooKeeper performance significantly.
• Each write operation is flushed to the disk. Thereby, operations that require extra low latencies or that perform intensive writes of large amount of data are wrong for ZooKeeper. Also, be aware that write speeds can not be increased by adding new instances to the ensemble; on the contrary, one can observe a slight performance drop on write operations.

On the other hand, there are several advantages of using ZooKeeper for our use case:

• All required configuration data is in centralized storage, that will help to avoid issues with data integrity. Meanwhile, ZooKeeper won't be a "single point of failure", because it allows running several instances in ensemble. It can survive failures as long as a majority of servers are active.
• Centralized configuration storage, which is always online, allows maintaining dynamic configuration of the distributed system. This results in an ability to adjust system settings without its restart or even in an ability to perform system auto tuning depending on values of environment metrics. For example, re-adjust settings of specific system component when it runs under high load.
• Flexible control of access to the specific ZNode and its child ZNodes. For instance, this allows to restrict access of employee or service to specific environments/services, protect config entry from unapproved changes by setting read-only access to it, etc.
• Out-of-the-box scalable and reliable tool for synchronizing various runtime system metrics.

You know, that creating complex all-purpose system that can do "almost everything" often leads to worse outcomes compared to the set of specialized tools, and even if you're sure in the final result the development process become very resource consuming and results in a poor ROI. Considering all the pros and cons, we have to strictly define the boundaries of our system:

• Individual configuration entries should have maximum size of KBytes: numeric and text constants, xml/json configurations, etc.; large binary resources have to be avoided.
• System components should avoid time-critical operations with variables and constants at runtime.
• ZooKeeper doesn't track changes to specific nodes itself, so you need to have separate backup/version control systems.

Example

Let's build an application that will show basic examples of communicating with ZooKeeper ensemble to coordinate configuration information. System will consist of one running instance of Apache Zookeeper and simple HTTP service, that use remote configuration to initialize itself on startup and serve several dummy requests. Structure of the system as well as its functionality could be easily extended according to your needs due to great scalability of the ZooKeeper-based solutions.

Structure of Configuration Data

The structure of the ZooKeeper storage of system configuration data is equivalent to file system structure of any UNIX-like operation system. In this case, we have several root paths for different kind of initial and runtime configuration data.

Root structure of the storage of configuration constants is the following:

• /system/<environment>/ - child ZNodes in this path should store global configuration constants which are common for all services and nodes of the system for a target environment.
• /system/<environment>/<service>/ - child ZNodes in this path should store service-specific configuration constants for a target environment.

Below is description for used placeholders:

• <environment> means environment identifier: dev, test, prod, etc.
• <service> is a name of a service (system component).

Technology Stack

The following software is used to build and run the example:

• Apache ZooKeeper (3.4.6): A high-performance coordination service for distributed applications.
• Scala (2.10.4): A general purpose programming language, which smoothly integrates features of object-oriented and functional programming paradigms.
• Apache Curator (2.7.0): A high level API library that simplifies using Apache ZooKeeper.
• Akka HTTP/Spray (1.3.2): An open-source toolkit for building REST/HTTP-based integration layers on top of Scala and Akka.
• Akka (2.3.8): An asynchronous event-driven middleware framework for building high performance and reliable distributed applications.
• SBT (0.13.5): An interactive build tool.

Initializing ZooKeeper Client

The first step to get remote configuration data from ZooKeeper in our HTTP Service is creating a client. We use capabilities of Curator for this purposes. It is pretty intelligent high-level API that adds many features that are built on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations.

Below is the code that initializes a client.

/**
 * Creates ZooKeeper remote configuration client.
 *
 * Initializes connection to the ZooKeeper ensemble.
 *
 * @param service service name
 * @param environment system environment
 * @param connectionString connection string; default value is taken from local configuration
 * @param connectionTimeout connection timeout; default value is taken from local configuration
 * @param sessionTimeout session timeout; default value is taken from local configuration
 * @param retryPolicy connection retry policy; default policy retries specified number of times with increasing sleep time between retries
 * @param authScheme authentication scheme; null by default
 * @param authData authentication data bytes; null by default
 * @return client instance
 */
def initZooKeeperClient(service: String, environment: String = "dev", connectionString: String = DefaultConnectionString, connectionTimeout: Int = DefaultConnectionTimeout, sessionTimeout: Int = DefaultSessionTimeout, retryPolicy: RetryPolicy = DefaultRetryPolicy, authScheme: String = null, authData: Array[Byte] = null): CuratorFramework = {
  val lookupClient = CuratorFrameworkFactory.builder()
    .connectString(connectionString)
    .retryPolicy(new retry.RetryOneTime(RetryInterval))
    .buildTemp(LookupClientTimeout, TimeUnit.MILLISECONDS)
  val serviceConfigPath = "/%s/%s".format(environment, service)
  try {
    lookupClient.inTransaction().check().forPath(serviceConfigPath).and().commit()
  } catch {
    case ke: KeeperException => {
      throw new MissingResourceException(
        "Remote configuration for %s service in %s environment is unavailable: %s - %s."
          .format(service, environment, ke.code(), ke.getMessage),
        "ZNode", serviceConfigPath)
    }
  }
  val client = CuratorFrameworkFactory.builder()
    .connectString(connectionString)
    .connectionTimeoutMs(connectionTimeout)
    .sessionTimeoutMs(sessionTimeout)
    .retryPolicy(retryPolicy)
    .authorization(