A python script to backup/restore the docker containers / volumes.
The idea :
- backup will backup the metadata from a container and its volumes
- restore will recreate a container from the saved metadata and restore its volumes
####Requires Python && Docker-py python package.
apt-get install python-pip
pip install docker-py texttable
##How to use Running backuper.py with -h will produce :
usage: backuper.py [-h] {list,backup,restore} ...
backup/restore/list a container and its volumes
positional arguments:
{list,backup,restore}
sub-command help :
list Lists the volumes of the container
backup Backups a container to a tar file
restore Restore a container from tar backup
optional arguments:
-h, --help show this help message and exit
You can then get help for the sub commands via
./backuper.py <subcommand> -h
Help for backup :
usage: backuper.py backup [-h] [-p] [-i INCLUDEVOLUMES]
[-s Absolute_Storage_Path]
container
positional arguments:
container Name of the container
optional arguments:
-h, --help show this help message and exit
-p, --pausecontainer Should we stop the source container before
extracting/saving its volumes and restart it after
backup (useful for files to be closed prior the
backup)if docker is >=1.3, the pause/unpause system
will be usedif docker is <1.3, a stop/restart command
will be used (breaking links)
-i INCLUDEVOLUMES, --includevolumes INCLUDEVOLUMES
include volumes in backup (without this option only
backups in /var/lib/docker/vfs on host are backed up.
The syntax is a string of elements that will be
matched against all volumes/bindings. Elements are
seperated by a coma ', ' and can be regex
-s Absolute_Storage_Path, --storage Absolute_Storage_Path
where to store/restore data, defaults to current path
(for BACKUP running inside a container, this parameter
isn't used)
Help for list :
$ ./backuper.py list -h
usage: backuper.py list [-h] container
positional arguments:
container Name of the container
optional arguments:
-h, --help show this help message and exit
Help for restore :
$ ./backuper.py restore -h
usage: backuper.py restore [-h] [-d destcontainername]
[-s Absolute_Storage_Path] [-r]
container
positional arguments:
container Name of the container
optional arguments:
-h, --help show this help message and exit
-d destcontainername, --destcontainer destcontainername
name of the restored container, defaults to source
container name
-s Absolute_Storage_Path, --storage Absolute_Storage_Path
where to store/restore data, defaults to current path
(for BACKUP running inside a container, this parameter
isn't used)
-r, --restoreinplace if the backed up container had mounted (bound)
directories on host, should we restore these bindings
AND the data in it (overwriting data on host maybe)
./backuper.py list containername
This command will list all the volumes/mount points/boindings of containername
./backuper.py backup containername --storage /tmp
This command will save the metadata and volumes as a tar file named : /tmp/containername.tar
./backuper.py backup containername --storage /tmp --pausecontainer
This command will save the metadata and volumes as a tar file named : /tmp/containername.tar
Additionnaly, the source container will be stopped (check KNOWN BUGS below about pausing) before backup and restarted afterwards
./backuper.py restore containername --storage /tmp --destcontainer newone
This command will restore the container containername and its volumes as a new container named newone from the tar file named : /tmp/containername.tar
First, you need to build it :
docker build -t docker-backuper .
Once done, can can backup using :
docker run -t -i --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
--volumes-from <container>
-v /tmp:/backup \
acaranta/docker-backuper \
backup <container> --pausecontainer
- The .tar backups will be stored in /backup ... which you can bind to any dir on your docker host.
- In this mode, the
--storageoption is ignored as the data will be stored in the bound directory/backup. - The container's volumes to be backed up are mounted using the --volumes-from option
- if added, the option
--pausecontainerwill stop the container to backup, and restart it afterwards (see KNOWN BUGS below about pausing)
Then you can restore using :
docker run -t -i --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v <TarStoragePath>:/backup \
acaranta/docker-backuper \
restore <container> --destname <newcontainer> --storage <TarStoragePath>
- The .tar backups will be Fetched in the argument passed as
--storageand which also has to be bound using-voption to/backup. It works differently from the backup, because for the restore, a container is launched on the docker host with the data storage dir mounted directly in order to read the tar files, it therefore need the/backupbinding AND the --storage argument both pointing towards the same path. - the container will be restored under the name
<newcontainer>
##FULL EXAMPLE Let's imagine we want a mysql container and we inject some data :
$ docker run -p 3306:3306 -e MYSQL_ROOT_PASSWORD=pouet -v /var/mytest:/justanexample --name mysqlsrv -d mysql
$ #Wait a moment (few seconds) for mysql to be available ;)
$ mysql -h 127.0.0.1 -uroot -ppouet -e "create database mytestdb ;"
$ echo "CREATE TABLE myTable (id mediumint(8) unsigned NOT NULL auto_increment,naming varchar(255) default NULL, phone varchar(100) default NULL, PRIMARY KEY (id)) AUTO_INCREMENT=1;
INSERT INTO myTable (naming,phone) VALUES (\"Brittany Z. Casey\",\"03 38 23 30 49\"),(\"Brandon C. Brady\",\"04 31 19 78 68\"),(\"Celeste I. Fletcher\",\"08 05 11 34 95\"),(\"Jael X. Mueller\",\"04 02 92 34 51\"),(\"Jonah T. Short\",\"09 07 74 41 37\"),(\"Natalie P. Hayes\",\"06 95 60 81 42\"),(\"Idola T. Mason\",\"06 49 92 54 92\"),(\"Indigo D. Chase\",\"04 77 59 48 16\"),(\"David K. Craig\",\"02 98 39 69 02\"),(\"Byron K. Sanders\",\"01 07 76 41 74\");
INSERT INTO myTable (naming,phone) VALUES (\"Brody D. Fitzpatrick\",\"06 60 39 08 84\"),(\"Grady B. Kline\",\"01 11 62 76 30\"),(\"Tanek O. Nelson\",\"04 34 16 80 50\"),(\"Ralph N. Reeves\",\"06 68 22 14 15\"),(\"Finn F. Castro\",\"06 17 23 14 90\"),(\"Imelda Z. Kemp\",\"02 37 63 50 46\"),(\"Denise A. Mcguire\",\"08 16 40 59 90\"),(\"Kellie U. Pierce\",\"02 63 15 98 34\"),(\"Rhiannon P. Holman\",\"05 60 21 05 15\"),(\"Quail A. Copeland\",\"04 26 63 57 14\");
" | mysql -h 127.0.0.1 -uroot -ppouet mytestdb
$ mysql -h 127.0.0.1 -uroot -ppouet mytestdb -e "select * from myTable"
Then we check what volumes can be backed up :
$ docker run -t -i --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
acaranta/docker-backuper \
list mysqlsrv
Volumes on container mysqlsrv ...
+-------------------------------------------------------+-------------------------------------------------------+
| Mount point (in container) | Bound to (on docker host) |
+=======================================================+=======================================================+
| /justanexample | /var/mytest |
+-------------------------------------------------------+-------------------------------------------------------+
| /var/lib/mysql | /var/lib/docker/vfs/dir/aad993c1c9dbb9d3e31a9e7c48a31 |
| | 6b88bc607e3416b57deaaf44ca5cd1c887a |
+-------------------------------------------------------+-------------------------------------------------------+
The result show that there are 2 volumes that can be backed up.
One of them volume resides in docker's /var/lib/docker/vfs/dir/ folder.
By default, launching a backup without specifying anything more would include this volume only.
However let's imagine we want both volumes included ... We would use the --includevolumes option :
--includevolumes "/justanexample,/var/lib/mysql"--includevolumes ".*"--includevolumes "justan,vfs"--includevolumes "exam,mytest"All of these would match both volumes !
Then we backup this container :
$ docker run -t -i --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
--volumes-from mysqlsrv \
-v /tmp:/backup \
acaranta/docker-backuper \
backup mysqlsrv --pausecontainer --includevolumes ".*"
We can remove completely the test container :
$ docker rm -f mysqlsrv
And restore from backup :
$ docker run -t -i --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/backup \
acaranta/docker-backuper \
restore mysqlsrv --destcontainer mysqlnewsrv --storage /tmp
And retest :
$ mysql -h 127.0.0.1 -uroot -ppouet mytestdb -e "select * from myTable"
##IMPORTANT NOTES & KNOWN "BUGS" ###In Place restoring and bound volumes if a container was launched with a boud volume, ie :
docker run -d -v /srv/docker-external-volumes/registry:/mnt/registry \
-e STORAGE_PATH=/mnt/registry/storage \
-e SQLALCHEMY_INDEX_DATABASE=sqlite:////mnt/registry/db/dbreg.sqlite \
-p 5000:5000 --name my_registry registry
(here /srv/docker-external-volumes/registry --> /mnt/registry)).
The restore will create a new volumes under /var/lib/docker/vfs unless option --restoreinplace is added.
In this case, the restore WILL take place in the bound path on host ... aka it will overwrite (it data is present) the contents of /srv/docker-external-volumes/registry !!!
###Docker UNIX Socket and not TCP socket
Why is the docker client access statically set to use the docker socket /var/run/docker.sock instead of gibing the choice to use either unix socket or TCP socket ?
That is not a mistake, that is a choice I made : to be clear the backup/restore NEEDS to access the docker host's filesystem.
You could in theory do that while using local tcp socket to the docker API ... but allowing this could generate (esay) mistake by running the backup/restore distantly ... while trying to work on local Filesystem ... and err ... BAM ! ;)
So until I find a better way, it should stay this way ;)
###Restarting the container during backup
If you use the --pausecontainer container option during backup, bear in mind that
there are two ways it can behave :
- you are using Docker version <1.3 : the
--pausecontaineroption will stop/restart the container... Therefore containers linked to the backed up container will have to be recreated in order to regenerate links !!! Dynamic links are a strongly wished feature in docker (moby/moby#3155). But for the time being, restart or stop/start breaks them. You have been warned ;) - you are using Docker version >=1.3 : the
--pausecontaineroption will use the pause/unpause commands to hang the container, therefore not breaking links AFAIT(ested) ;)
##TODO
- add a way to nicely name the tar files ?
- add a way to timestamp the tar files and let the user choose different restore points ?
- Review all the metadata parameters that still needs to be restored (ie : ro or rw volumes)
##DISCLAIMER Please TEST your backup/restore procedure, your data, etc ... this is provided as-is and does not garantee anything ! ;)
Based on the code from docker-volume-backup : https://github.com/paimpozhil/docker-volume-backup