etcd-conductor: orchestrate your etcd cluster deployment and management

July 8th, 2015 by bostjan

This is a brief guide to a tool called etcd-conductor, which can be used to install, deploy, bootstrap and manage a cluster of etcd nodes. The initial spark for its development came from a frustration that followed brief experiments to manage etcd cluster with Puppet. It was cumbersome. And slow. And impossible to do elegantly. Did I say already it was slow? Controlled rolling restart when reconfiguring whole cluster? Yeah right.

Let me state very clearly that I have nothing against Puppet. I actually use it a lot, because I find it very convenient for keeping infrastructure up to date, in a centrally controlled manner. When deploying applications, Puppet takes care of the base stack (OS, daemons, interpreters, etc), while some interactive deployment tool does the application-level parts.

Etcd is a bit different beast: it just makes sense to manage it more like an application than as a system service. Sometimes you need to do things instantly, like bootstrapping new cluster, or adding new node to existing cluster, or removing (failed) node from a cluster. The last two actions are a multi-step procedures, and those steps need to be executed in specified order. Which quite hard to do with Puppet, as the order of catalog runs across all cluster nodes is not exactly predetermined.

Then there is another thing that was considered at design phase: using etcd-conductor as git submodule in a git repository, where one keeps track of his etcd cluster configuration. This last sentence will deserve its own post one day, but let me explain this briefly in the following paragraph.


Git submodules vs keeping up with code and data changes

The latest trend in application and systems development favours git as a tool for tracking configuration and code changes. Many tools are only distributed as git clones. Which is fine, until you start thinking about keeping your data versioned too. Tools usually require you to adjust certain configuration files within repository and whatnot. What options do you then have?

  1. Clone tool, adjust configuration and commit configuration to the very same git repo. Obviously a flawed approach, as down the timeline you will have to do the merging when upgrading your tool to newer versions (or SHA-1 hashes, to be more precise:). This is cumbersome at best.
  2. Create new repo, import tool, import configuration, commit. Equally bad as #1, as you have to reimport your tool manually when upgrading.
  3. Use tool’s git clone as a git submodule in repository where you track configation data chages. Obviously this seems the only sane approach if git-based configuration tracking flow is our objective.

…back to etcd-conductor – How to easily bootstrap a new etcd cluster

So, with git-submodule-thingie out of the way, these are the steps to start using etcd-conductor for managing your etcd cluster.

This is basically all it takes to set up a new etcd cluster when using etcd-conductor. Have fun!

Adding a new node to existing cluster

Procedure for adding a new node is a three-step process:

  1. Inform existing cluster of new node that is about to join it
  2. Start new node and point it to existing cluster members
  3. Reconfigure all nodes in cluster and do a rolling restart

But with etcd-conductor you only need to do two things:

  1. Add data about your new node to inventory file (hosts)
  2. Run ./bin/add-new-node NEW_NODE_NAME

Removing node from cluster

Procedure for removing node is a three-step process too:

  1. Inform the cluster about node that is to leave it
  2. Stop the node and destroy it’s data
  3. Reconfigure all nodes in cluster and do a rolling restart

With etcd-conductor you only need to do one thing:

  1. Run ./bin/remove-node NODE_NAME

That is it. Have fun!

Leave a Reply