Replace lftp's call to 'rmdir' by calls to 'rm -rf'.

[brioche.git] / README

.. HTML version generated with rst2html -t README > README.html

.. |date| date:: %b %e, %Y

Brioche Backup
~~~~~~~~~~~~~~

:Author:    Amand Tihon
:Contact:   <amand.tihon@alrj.org>
:Version:   1.2
:Date:      |date|
:Copyright: GNU GPL, see copyright file.

.. sectnum::

.. contents:: Table of contents
   :backlinks: none


Abstract
========

Brioche is yet another backup shell script. Its main features are

- Full and differential backups
- LVM snapshots
- Xen oriented (somehow)
- Upload to a distant FTP server.


Rationale
=========

Large numbers of backup solutions are freely available today, but when playing
with incremental or differential backups, most of them rely on some filesystem
capabilities, like *hard-linking*. If the only remote location available to
store the backups is an FTP server, this is not a solution.
Brioche relies on GNU tar's ``--listed-incremental`` option to create *real*
differential [#diff]_ archives.

A second feature that makes Brioche interresting is the ability to use LVM
snapshots. In the case of a Xen setup where the domUs use logical volumes as
partitions, Brioche is able to backup everything from the dom0.


.. [#diff] Each differential backup is based upon the last *full* backup, unlike
      incremental, which are based upon the last (full or incremental) backup.


Getting Brioche
===============

You can download the tarball from http://www.alrj.org/projects/brioche
or get the latest development version with the following git command: ::

  git clone http://git.alrj.org/git/brioche.git

A gitweb interface is also available at http://git.alrj.org/


Requirements
============

Brioche relies on a few easily available free software :

- In any case, Brioche will require GNU tar, which is able to deal with
  incremental backups. Don't even try it with any other tar implementation.
  Tested with version 1.16.
- For LVM snapshots, lvm2 will obviously be needed.
- FTP backups, if they're used, will require lftp.

See the `References`_ section for links to the aforementioned softwares.


Installation
============

Copy the three files ``brioche``, ``brioche.conf`` and ``briochetab`` where
you like, and set the ``CONFIG_FILE`` variable in the ``brioche`` script
accordingly. If needed, run ``chmod +x /path/to/brioche``. In a typical setup,
the ``brioche`` script will be put in ``/usr/local/bin`` with the other two
files under ``/etc``.


Using Brioche
=============

Configuration
-------------

Edit the file ``brioche.conf`` to suit your needs. Each option is commented
inline and will be detailed here.

BACKUPTAB
  Full absolute path to the briochetab file. This file describes which logical
  volumes and partitions must be backed up. Its format is explained in the
  `Defining backups`_ section.

MAILTO
  When the job is done, Brioche will send a summary of the operations by email.
  You can set it to any value that your ``mail`` command understands.

.. _REPODIR:

REPODIR
  Brioche will store all the generated archives in this directory.Note that
  before doing a full backup, Brioche will move all previous archives into an
  "undo" subdirectory, which will be removed only if the backup is successful.
  Make sure there's enough free space on the device where REPODIR is located.

USAGE_WARN
  The report email will include a warning if the space used on REPODIR goes
  beyond the given threshold. The value must be an integer.

COMPRESS
  This directive allows to specify the compression method to apply to the
  archives. Possible values are "none", "gz", "bz2" and "lzma".
  **Warning** : lzma may not be available with older versions of GNU tar.

TAR_OPTS
  Additionnal options that you may want to pass to tar. A typical value could
  be ``"--one-file-system -S"``. The first option will skip all other
  mointpoints (very usefull if you have /dev, /proc, /sys or REPODIR mounted
  under a device that must be archived). The second one will try to deal with
  sparse files.

SNAPSHOT_MOUNTPOINT
  Sets the directory where the temporary LVM snapshots must be mounted.

SNAPSHOT_NAME
  The name to use for the snapshot volumes.

SNAPSHOT_SIZE
  Set the size of the snapshot volume. The same suffix than for lvcreate(8)
  are available.

USE_GPG
  When set to "yes", Brioche will encrypt the archives with GnuPG.
  See the `Encryption with GnuPG`_ section below for a detailed explanation on
  how to use GnuPG with Brioche.

GPG_KEY
  The identifier of the public GnuPG key to use when encrypting the archives.
  This is the key that will be needed in case of restore.

GPG_PASSPHRASE
  The passphrase for the GnuPG private key used to encrypt the archives.

USE_FTP
  If set to "yes", Brioche will upload the backups on an FTP server.
  See the `Using FTP`_ section for more information about this feature.

FTP_HOST
  The address of the FTP server.

FTP_DIR
  The base directory on the FTP server under which all the archives will be
  stored. Brioche will never touch anything that is not below this directory.

FTP_KEEP
  Tells Brioche to keep a certain amount of older runs on the FTP. A *run* is
  a full backup plus all its subsequent differential backups. See the `Using
  FTP`_ section for a more detailed explanation.


Defining backups
----------------

The backups are defined in the file ``birochetab``. Here is a typical example
for a Xen config where cottman is the dom0 and syrtis, kadarin, valeron are
domUs: ::

  # Partition or LV           Snapshot    Host name     Volume name
  # ---------------------------------------------------------------
  /                           no          cottman       root
  /usr                        no          cottman       usr

  /dev/vg00/valeron-root      yes         valeron       root

  /dev/vg00/kadarin-root      yes         kadarin       root
  /dev/vg00/kadarin-home      yes         kadarin       home

  /dev/vg00/syrtis-root       yes         syrtis        root
  /dev/vg00/syrtis-home       yes         syrtis        home
  /dev/vg00/syrtis-usr        yes         syrtis        usr
  /dev/vg00/syrtis-var        yes         syrtis        var

Blank lines, or lines beginning with # are ignored.

The first column defines the directory or logical volume to backup. In this
example,the first two lines are plain directories, while the other ones point
to LVM devices.

The second column specify if the backup should be taken from an LVM snapshot or
not. It **must** be set to "no" for the backup of a directory and to "yes" for
the backup of a logical volume.

The last two columns are more or less cosmetic, and define where the archive
files will be stored, and how they'll be named. The destination directory will
be created under the REPODIR_, and its name will be the value on the third
column. Inside this directory, archive files will be named from the value given
in the fourth column. Grouping by domUs' hostnames is only a suggestion, it can
be completely different and adapted to suit your needs.

For instance, a full and a differential backup for the host valeron of the
previous example would lead to the following structure: ::

  user:/REPODIR$ ls -l valeron/
  total 356544
  -rw-r--r-- 1 root root 363545613 Jan  3 03:10 root.full.20090103.tar.bz2
  -rw-r--r-- 1 root root    504722 Jan  3 03:10 root.full.snar
  -rw-r--r-- 1 root root    504725 Jan  4 04:06 root.incr.20090104.snar
  -rw-r--r-- 1 root root    160542 Jan  4 04:06 root.incr.20090104.tar.bz2

+--------------------------------------------------------------------------+
| **WARNING !**                                                            |
|                                                                          |
| In case your REPODIR_ is not on a distinct device, it will be included   |
| in the backup if you've included the device in your ``briochetab``.      |
|                                                                          |
| To avoid this issue, you can either exclude it explicitely by adding an  |
| "``--exclude=...``" option in TAR_OPTS, or simply specify in your        |
| ``briochetab`` file which directories need to be archived.               |
+--------------------------------------------------------------------------+


Encryption with GnuPG
---------------------


Using FTP
---------

With the help of lftp, Brioche is able to store an history of backups on an FTP
server. This is mainly usefull when no other distant repository is available.
If possible, consider using an CIFS, NFS, sshfs or any other kind of remote
mountpoint for your REPODIR_.

The archives present in the local REPODIR_ will be mirrored to the FTP server
after each backup, be it a full or a differential one.

Since there is no way to hide the credentials if they are passed to lftp on the
command line, the authentication relies on your ``.netrc`` file. See ``man
netrc(5)`` for more information. In the home directory of the user that runs
Brioche (typically root's), create the ``.netrc`` file with the following
lines: ::

  machine ftp.example.com
  login username
  password SikRet

Don't forget to secure it with ``chmod 600 .netrc`` or lftp will refuse to use
it. The machine name must match the FTP_HOST configuration directive in
``brioche.conf``.

On the FTP server, Brioche will keep a configurable amount of *runs*.
Each *run* consists of a full backup and all the differential backups that are
based on it. Before doing a full backup, Brioche will rotate the
*runs* and keep only the configured number of older backups. The current
backups can always be found under ``/FTP_DIR/hostname/latest/``. Older ones
will be under ``/FTP_DIR/hostname/run-X/`` with *X* equal to 1 for the previous
run, 2 for the one before and so forth, up to the value of FTP_KEEP.

Here's what happens during the rotation:

- the oldest run is removed
- all the ``run-X/`` directories are shifted (``run-3/`` becomes ``run-4/``,
  etc)
- the ``latest/`` directory is renamed to ``run-1/``
- a new, empty, ``latest/`` directory is created, ready to accept the new files.



Running Brioche
---------------

Brioche understands the following arguments:

-f, --full  Do a full backup (by default, brioche will try to do a
            differential).
-h, --help  Show a very limited help.

When everything is ready, execute the ``brioche`` script as root. The
script is pretty verbose, so don't panic if you see lots of lines scrolling in
your terminal. By default, Brioche will try to make differential backups, but
will gracefully fall back and do a full backup if none is available.

If all is fine, it can be added in the system crontab. Here's a suggestion for
weekly full backup on Sunday, with differential during the weekdays: ::

  # Daily incremental backup
  30  3  *  *  1-6  /usr/local/bin/brioche > /var/log/backup.`dow`.log 2>&1
  # Weekly full backup on Sunday
  30  3  *  *  0  /usr/local/bin/brioche -f > /var/log/backup.`dow`.log 2>&1

The output of the script will be saved in /var/log/backup.DOW.log with DOW
being the abbreviated day of the week (see man date(1) for the format).
The ``dow`` bash script is provided alongside Brioche.


Bug reporting
=============

There's no bugtracker for this project, your bug reports should be sent to the
author : Amand Tihon <amand.tihon@alrj.org>. Please include as much information
as possible in your report.



References
==========

- GNU tar documentation : http://www.gnu.org/software/tar/manual/
- LVM documentation and links : http://sourceware.org/lvm2/
- lftp homepage : http://lftp.yar.ru/