zookeeper kazoo Basic Usage

http://kazoo.readthedocs.org/en/latest/basic_usage.html

Basic Usage

Connection Handling

To begin using Kazoo, a KazooClient object must be created and a connection established:

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

By default, the client will connect to a local Zookeeper server on the default port (2181). You should make sure Zookeeper is actually running there first, or the start command will be waiting until its default timeout.

Once connected, the client will attempt to stay connected regardless of intermittent connection loss or Zookeeper session expiration. The client can be instructed to drop a connection by calling stop:

zk.stop()

Logging Setup

If logging is not setup for your application, you can get following message:

No handlers could be found for logger "kazoo.client"

To avoid this issue you can at the very minimum do the following:

import logging
logging.basicConfig()

Read Python’s logging tutorial for more details.

Listening for Connection Events

It can be useful to know when the connection has been dropped, restored, or when the Zookeeper session has expired. To simplify this process Kazoo uses a state system and lets you register listener functions to be called when the state changes.

from kazoo.client import KazooState

def my_listener(state):
    if state == KazooState.LOST:
        # Register somewhere that the session was lost
    elif state == KazooState.SUSPENDED:
        # Handle being disconnected from Zookeeper
    else:
        # Handle being connected/reconnected to Zookeeper

zk.add_listener(my_listener)

When using the kazoo.recipe.lock.Lock or creating ephemeral nodes, its highly recommended to add a state listener so that your program can properly deal with connection interruptions or a Zookeeper session loss.

Understanding Kazoo States

The KazooState object represents several states the client transitions through. The current state of the client can always be determined by viewing the state property. The possible states are:

• LOST
• CONNECTED
• SUSPENDED

When a KazooClient instance is first created, it is in the LOST state. After a connection is established it transitions to the CONNECTED state. If any connection issues come up or if it needs to connect to a different Zookeeper cluster node, it will transition to SUSPENDED to let you know that commands cannot currently be run. The connection will also be lost if the Zookeeper node is no longer part of the quorum, resulting in a SUSPENDED state.

Upon re-establishing a connection the client could transition to LOST if the session has expired, orCONNECTED if the session is still valid.

Note

These states should be monitored using a listener as described previously so that the client behaves properly depending on the state of the connection.

When a connection transitions to SUSPENDED, if the client is performing an action that requires agreement with other systems (using the Lock recipe for example), it should pause what it’s doing. When the connection has been re-established the client can continue depending on if the state isLOST or transitions directly to CONNECTED again.

When a connection transitions to LOST, any ephemeral nodes that have been created will be removed by Zookeeper. This affects all recipes that create ephemeral nodes, such as the Lock recipe. Lock’s will need to be re-acquired after the state transitions to CONNECTED again. This transition occurs when a session expires or when you stop the clients connection.

Valid State Transitions

• LOST -> CONNECTED

  New connection, or previously lost one becoming connected.

• CONNECTED -> SUSPENDED

  Connection loss to server occurred on a connection.

• CONNECTED -> LOST

  Only occurs if invalid authentication credentials are provided after the connection was established.

• SUSPENDED -> LOST

  Connection resumed to server, but then lost as the session was expired.

• SUSPENDED -> CONNECTED

  Connection that was lost has been restored.

Read-Only Connections

New in version 0.6.

Zookeeper 3.4 and above supports a read-only mode. This mode must be turned on for the servers in the Zookeeper cluster for the client to utilize it. To use this mode with Kazoo, the KazooClient should be called with the read_only option set to True. This will let the client connect to a Zookeeper node that has gone read-only, and the client will continue to scan for other nodes that are read-write.

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181', read_only=True)
zk.start()

A new attribute on KeeperState has been added, CONNECTED_RO. The connection states above are still valid, however upon CONNECTED, you will need to check the clients non- simplified state to see if the connection is CONNECTED_RO. For example:

from kazoo.client import KazooState
from kazoo.client import KeeperState

@zk.add_listener
def watch_for_ro(state):
    if state == KazooState.CONNECTED:
        if zk.client_state == KeeperState.CONNECTED_RO:
            print("Read only mode!")
        else:
            print("Read/Write mode!")

It’s important to note that a KazooState is passed in to the listener but the read-only information is only available by comparing the non-simplified client state to the KeeperState object.

Warning

A client using read-only mode should not use any of the recipes.

Zookeeper CRUD

Zookeeper includes several functions for creating, reading, updating, and deleting Zookeeper nodes (called znodes or nodes here). Kazoo adds several convenience methods and a more Pythonic API.

Creating Nodes

Methods:

• ensure_path()
• create()

ensure_path() will recursively create the node and any nodes in the path necessary along the way, but can not set the data for the node, only the ACL.

create() creates a node and can set the data on the node along with a watch function. It requires the path to it to exist first, unless the makepath option is set to True.

# Ensure a path, create if necessary
zk.ensure_path("/my/favorite")

# Create a node with data
zk.create("/my/favorite/node", b"a value")

Reading Data

Methods:

• exists()
• get()
• get_children()

exists() checks to see if a node exists.

get() fetches the data of the node along with detailed node information in a ZnodeStat structure.

get_children() gets a list of the children of a given node.

# Determine if a node exists
if zk.exists("/my/favorite"):
    # Do something

# Print the version of a node and its data
data, stat = zk.get("/my/favorite")
print("Version: %s, data: %s" % (stat.version, data.decode("utf-8")))

# List the children
children = zk.get_children("/my/favorite")
print("There are %s children with names %s" % (len(children), children))

Updating Data

Methods:

• set()

set() updates the data for a given node. A version for the node can be supplied, which will be required to match before updating the data, or a BadVersionError will be raised instead of updating.

zk.set("/my/favorite", b"some data")

Deleting Nodes

Methods:

• delete()

delete() deletes a node, and can optionally recursively delete all children of the node as well. A version can be supplied when deleting a node which will be required to match the version of the node before deleting it or a BadVersionError will be raised instead of deleting.

zk.delete("/my/favorite/node", recursive=True)

Retrying Commands

Connections to Zookeeper may get interrupted if the Zookeeper server goes down or becomes unreachable. By default, kazoo does not retry commands, so these failures will result in an exception being raised. To assist with failures kazoo comes with a retry() helper that will retry a function should one of the Zookeeper connection exceptions get raised.

Example:

result = zk.retry(zk.get, "/path/to/node")

Some commands may have unique behavior that doesn’t warrant automatic retries on a per command basis. For example, if one creates a node a connection might be lost before the command returns successfully but the node actually got created. This results in akazoo.exceptions.NodeExistsError being raised when it runs again. A similar unique situation arises when a node is created with ephemeral and sequence options set, documented here on the Zookeeper site.

Since the retry() method takes a function to call and its arguments, a function that runs multiple Zookeeper commands could be passed to it so that the entire function will be retried if the connection is lost.

This snippet from the lock implementation shows how it uses retry to re-run the function acquiring a lock, and checks to see if it was already created to handle this condition:

# kazoo.recipe.lock snippet

def acquire(self):
    """Acquire the mutex, blocking until it is obtained"""
    try:
        self.client.retry(self._inner_acquire)
        self.is_acquired = True
    except KazooException:
        # if we did ultimately fail, attempt to clean up
        self._best_effort_cleanup()
        self.cancelled = False
        raise

def _inner_acquire(self):
    self.wake_event.clear()

    # make sure our election parent node exists
    if not self.assured_path:
        self.client.ensure_path(self.path)

    node = None
    if self.create_tried:
        node = self._find_node()
    else:
        self.create_tried = True

    if not node:
        node = self.client.create(self.create_path, self.data,
            ephemeral=True, sequence=True)
        # strip off path to node
        node = node[len(self.path) + 1:]

create_tried records whether it has tried to create the node already in the event the connection is lost before the node name is returned.

Custom Retries

Sometimes you may wish to have specific retry policies for a command or set of commands that differs from the retry() method. You can manually create a KazooRetry instance with the specific retry policy you prefer:

from kazoo.retry import KazooRetry

kr = KazooRetry(max_tries=3, ignore_expire=False)
result = kr(client.get, "/some/path")

This will retry the client.get command up to 3 times, and raise a session expiration if it occurs. You can also make an instance with the default behavior that ignores session expiration during a retry.

Watchers

Kazoo can set watch functions on a node that can be triggered either when the node has changed or when the children of the node change. This change to the node or children can also be the node or its children being deleted.

Watchers can be set in two different ways, the first is the style that Zookeeper supports by default for one-time watch events. These watch functions will be called once by kazoo, and do not receive session events, unlike the native Zookeeper watches. Using this style requires the watch function to be passed to one of these methods:

• get()
• get_children()
• exists()

A watch function passed to get() or exists() will be called when the data on the node changes or the node itself is deleted. It will be passed a WatchedEvent instance.

def my_func(event):
    # check to see what the children are now

# Call my_func when the children change
children = zk.get_children("/my/favorite/node", watch=my_func)

Kazoo includes a higher level API that watches for data and children modifications that’s easier to use as it doesn’t require re-setting the watch every time the event is triggered. It also passes in the data and ZnodeStat when watching a node or the list of children when watching a nodes children. Watch functions registered with this API will be called immediately and every time there’s a change, or until the function returns False. If allow_session_lost is set to True, then the function will no longer be called if the session is lost.

The following methods provide this functionality:

• ChildrenWatch
• DataWatch

These classes are available directly on the KazooClient instance and don’t require the client object to be passed in when used in this manner. The instance returned by instantiating either of the classes can be called directly allowing them to be used as decorators:

@zk.ChildrenWatch("/my/favorite/node")
def watch_children(children):
    print("Children are now: %s" % children)
# Above function called immediately, and from then on

@zk.DataWatch("/my/favorite")
def watch_node(data, stat):
    print("Version: %s, data: %s" % (stat.version, data.decode("utf-8")))

Transactions

New in version 0.6.

Zookeeper 3.4 and above supports the sending of multiple commands at once that will be committed as a single atomic unit. Either they will all succeed or they will all fail. The result of a transaction will be a list of the success/failure results for each command in the transaction.

transaction = zk.transaction()
transaction.check('/node/a', version=3)
transaction.create('/node/b', b"a value")
results = transaction.commit()

The transaction() method returns a TransactionRequest instance. It’s methods may be called to queue commands to be completed in the transaction. When the transaction is ready to be sent, thecommit() method on it is called.

In the example above, there’s a command not available unless a transaction is being used, check. This can check nodes for a specific version, which could be used to make the transaction fail if a node doesn’t match a version that it should be at. In this case the node /node/a must be at version 3 or /node/b will not be created.