Clustered VM Testing How-To

Recently I’ve been testing clusters of VMs running on my local host.

I thought that there must be a standard way to test multi-node VM setups, but asking around at work, and on github yielded no answers.

So I came up with my own solution, which I outline here.

ShutItFile

A ShutItFile is a superset of a Dockerfile that allows straighforward automation of automation tasks.

Here’s an example of a ShutItFile that manipulates two VMs, and tests network connectivity between them.

It creates two machines (machine1 and machine2) and logs into them in turn using the ‘VAGRANT_LOGIN’ directive. On each machine it installs python, sets up a simple python http server which serves the text: ‘Hi from machine1’ (from machine1) or ‘Hi from machine2’ from machine2.

It then tests that the output matches expectation from both machines using the ‘ASSERT_OUTPUT’ directive.

To demonstrate the ‘testing’ nature of the ShutItFile, a ‘PAUSE_POINT’ directive is included, which drops you into the run with a terminal, and a deliberately wrong ‘ASSERT_OUTPUT’ directive is included to show what happens when a test fails (and the terminal is interactive). This makes debugging a _lot_ easier.

DELIVERY bash

# Set up trivial webserver on machine1
VAGRANT_LOGIN machine1
INSTALL python
# Add file 
RUN echo 'hi from machine1' > /root/index.html
RUN nohup python -m SimpleHTTPServer 80 &
VAGRANT_LOGOUT

# Set up trivial webserver on machine2
VAGRANT_LOGIN machine2
INSTALL python
RUN echo 'hi from machine2' > /root/index.html
RUN nohup python -m SimpleHTTPServer 80 &
VAGRANT_LOGOUT

# Test machine2 from machine1
VAGRANT_LOGIN machine1
INSTALL python
RUN curl machine2
ASSERT_OUTPUT hi from machine2
VAGRANT_LOGOUT

# Test machine1 from machine2
VAGRANT_LOGIN machine2
INSTALL python
RUN curl machine1
ASSERT_OUTPUT hi from machine1
VAGRANT_LOGOUT

# Example debug
VAGRANT_LOGIN machine1
INSTALL python
PAUSE_POINT 'Have a look around, debug away'
# Trigger a 'failure'
RUN curl machine2
ASSERT_OUTPUT will never happen
VAGRANT_LOGOUT

To run this ShutItFile (which we call here ‘ShutItFile.sf’), you run like this:

# Install shutit
pip install shutit
shutit skeleton --shutitfile ShutItFile.sf \
    --name /tmp/shutitfile_build \
    --domain twovm.twovm \
    --delivery bash\
    --pattern vagrant \
    --vagrant_num_machines 2\
    --vagrant_machine_prefix machine

This code for this example is available here.

Video

There’s a video of the above run here:

Create Your Own

If you want to create your own multinode test:

pip install shutit  #use sudo if needed, --upgrade if upgrading
shutit skeleton

Follow the instructions, choosing ‘shutitfile’ as the pattern, and ‘vagrant’ as the delivery method, eg:

$  shutit skeleton

# Input a name for this module.
# Default: /space/git/shutitfile/examples/vagrant/simple_two_machine/shutit_sabers


# Input a ShutIt pattern.
Default: bash

bash:              a shell script
docker:            a docker image build
vagrant:           a vagrant setup
docker_tutorial:   a docker-based tutorial
shutitfile:        a shutitfile-based project (can be docker, bash, vagrant)

shutitfile


# Input a delivery method from: bash, docker, vagrant.
# Default: ' + default_delivery + '

docker:      build within a docker image
bash:        run commands directly within bash
vagrant:     build an n-node vagrant cluster

vagrant
# ShutIt Started... 
# Loading configs...
# Run:
cd /space/git/shutitfile/examples/vagrant/simple_two_machine/shutit_sabers && ./run.sh
# to run.
# Or
# cd /space/git/shutitfile/examples/vagrant/simple_two_machine/shutit_sabers && ./run.sh -c
# to run while choosing modules to build.

and follow the commands given (at the place in bold above) to run.

Initially you are given empty ShutItFiles. You could start by adding the commands from the example here.

A cheatsheet for the various ShutItFile commands is available here.

Watch me do this here.

Real-world Usage

As an example of real-world usage, this technique is being used to regression test Chef recipes used to provision OpenShift.

The Chef scripts are here, and the regression tests are here.