One of ZFS's features we use a lot at our company is the ability to send/recv snapshots via stout/stdin. This enables sys admins to stream snapshots of a filesystem to another pool on a local or remote server. The latter can be achieved by piping the snapshot using common tools like ssh or netcat.

Since a couple of months we started using CEPH's RBD (Rados Block Device) to supply images for some of our virtual machines. CEPH's architecture allows images to be striped and replicated throughout multiple servers (using a clever algorithm I'm not going to elaborate on right now). But even with this fault tolerant architecture, it feels safe to create an off-site backup of our RBD images. This can be accomplished using the new RBD import/export feature which works almost like ZFS's send/recv.

I stumbled upon this URL:

Inspired by scuttlemonkey's article I wanted to create a script which would scan all images in a specified pool (SOURCEPOOL) and replicate those images to another pool (DESTPOOL).

First, the script needs to create a snapshot of the image in the source pool. Then it checks whether the image exists on the destination pool. If it's the first time the script is running (for this image), a new (empty) image gets created on the remote pool and filled with the first diff (which happens to be the complete image data the first time). In the second run (the next day) a new snapshot gets created of the local image. Then the image at the remote pool gets checked again to see if yesterdays snapshot exists (should be) and receives the delta data between yesterdays and todays snapshot of the local image. This is done using the --from-snap flag. Finally, the changed extents between the --from-snap and the current snap get pulled out as a json string and compared against the changed extents of the same snapshots on the destination pool using MD5.

This process is being repeated for all images in the pool. By running this script day after day, a history of snapshots will be created on both the source and destination pool.

Limitations: you should never mount or alter anything on the images in the destination pool as the changed extents check will fail (like it should). Also, this script assumes it is being run every day and every run has to be completed within the current day. You should therefore schedule this script as early as possible. For instance, at a minute past midnight.

Note: a useful addition to this script would be the ability to purge old snapshots on the remote (and perhaps local pool) so that the images don't grow too large. Another addition would be the ability to detect the last snapshot instead of assuming this script is being run every day.


#!/bin/bash

SOURCEPOOL="my-sourcepool"
DESTPOOL="my-destpool"
DESTHOST="root@xxx.xxx.xxx.xxx"

#what is today's date?

TODAY=`date +"%Y-%m-%d"`
YESTERDAY=`date +"%Y-%m-%d" --date="1 days ago"`

#list all images in the pool

IMAGES=`rbd ls $SOURCEPOOL`

for LOCAL_IMAGE in $IMAGES; do

#check whether remote host/pool has image

if [[ -z $(ssh $DESTHOST rbd ls $DESTPOOL | grep $LOCAL_IMAGE) ]]; then
echo "info: image does not exist in remote pool. creating new image"

#todo: check succesful creation

`ssh $DESTHOST rbd create $DESTPOOL/$LOCAL_IMAGE -s 1`
fi

#create today's snapshot

if [[ -z $(rbd snap ls $SOURCEPOOL/$LOCAL_IMAGE | grep $TODAY) ]]; then
echo "info: creating snapshot $SOURCEPOOL/$LOCAL_IMAGE@$TODAY"
`rbd snap create $SOURCEPOOL/$LOCAL_IMAGE@$TODAY`
else
echo "warning: source image $SOURCEPOOL/$LOCAL_IMAGE@$TODAY already exists"
fi

# check whether to do a init or a full

if [[ -z $(ssh $DESTHOST rbd snap ls $DESTPOOL/$LOCAL_IMAGE) ]]; then
echo "info: no snapshots found for $DESTPOOL/$LOCAL_IMAGE doing init"
`rbd export-diff $SOURCEPOOL/$LOCAL_IMAGE@$TODAY - | ssh $DESTHOST rbd import-diff - $DESTPOOL/$LOCAL_IMAGE`
else
echo "info: found previous snapshots for $DESTPOOL/$LOCAL_IMAGE doing diff"

#check yesterday's snapshot exists at remote pool

if [[ -z $(ssh $DESTHOST rbd snap ls $DESTPOOL/$LOCAL_IMAGE | grep $YESTERDAY) ]]; then
echo "error: --from-snap $LOCAL_IMAGE@$YESTERDAY does not exist on remote pool"
exit 1
fi
#check todays's snapshot already exists at remote pool

if [[ -z $(ssh $DESTHOST rbd snap ls $DESTPOOL/$LOCAL_IMAGE | grep $TODAY) ]]; then
`rbd export-diff --from-snap $YESTERDAY $SOURCEPOOL/$LOCAL_IMAGE@$TODAY - | ssh $DESTHOST rbd import-diff - $DESTPOOL/$LOCAL_IMAGE`

#comparing changed extents between source and destination

SOURCE_HASH=`rbd diff --from-snap $YESTERDAY $SOURCEPOOL/$LOCAL_IMAGE@$TODAY --format json | md5sum | cut -d ' ' -f 1`
DEST_HASH=`ssh $DESTHOST rbd diff --from-snap $YESTERDAY $DESTPOOL/$LOCAL_IMAGE@$TODAY --format json | md5sum | cut -d ' ' -f 1`

if [ $SOURCE_HASH == $DEST_HASH ]; then
echo "info: changed extents hash check ok"
else
echo "error: changed extents hash on source and destination don't match: $SOURCE_HASH not equals $DEST_HASH"
fi
else
echo "error: snapshot $DESTPOOL/$LOCAL_IMAGE@$TODAY already exists, skipping"
exit 1
fi
fi

done