1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
6 <meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
7 <title>Brioche Backup</title>
8 <meta name="author" content="Amand Tihon" />
9 <meta name="date" content="Jan 9, 2009" />
10 <meta name="copyright" content="GNU GPL, see copyright file." />
11 <style type="text/css">
14 :Author: David Goodger (goodger@python.org)
15 :Id: $Id: html4css1.css 5196 2007-06-03 20:25:28Z wiemann $
16 :Copyright: This stylesheet has been placed in the public domain.
18 Default cascading style sheet for the HTML output of Docutils.
20 See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
21 customize this style sheet.
24 /* used to remove borders from tables and images */
25 .borderless, table.borderless td, table.borderless th {
28 table.borderless td, table.borderless th {
29 /* Override padding for "table.docutils td" with "! important".
30 The right padding separates the table cells. */
31 padding: 0 0.5em 0 0 ! important }
34 /* Override more specific margin styles with "! important". */
35 margin-top: 0 ! important }
37 .last, .with-subtitle {
38 margin-bottom: 0 ! important }
44 text-decoration: none ;
51 margin-bottom: 0.5em }
53 /* Uncomment (and remove this text!) to get bold-faced definition list terms
61 div.abstract p.topic-title {
65 div.admonition, div.attention, div.caution, div.danger, div.error,
66 div.hint, div.important, div.note, div.tip, div.warning {
68 border: medium outset ;
71 div.admonition p.admonition-title, div.hint p.admonition-title,
72 div.important p.admonition-title, div.note p.admonition-title,
73 div.tip p.admonition-title {
75 font-family: sans-serif }
77 div.attention p.admonition-title, div.caution p.admonition-title,
78 div.danger p.admonition-title, div.error p.admonition-title,
79 div.warning p.admonition-title {
82 font-family: sans-serif }
84 /* Uncomment (and remove this text!) to get reduced vertical space in
86 div.compound .compound-first, div.compound .compound-middle {
87 margin-bottom: 0.5em }
89 div.compound .compound-last, div.compound .compound-middle {
98 div.dedication p.topic-title {
106 div.footer, div.header {
115 div.line-block div.line-block {
121 margin: 0 0 0.5em 1em ;
122 border: medium outset ;
124 background-color: #ffffee ;
129 div.sidebar p.rubric {
130 font-family: sans-serif ;
133 div.system-messages {
136 div.system-messages h1 {
140 border: medium outset ;
143 div.system-message p.system-message-title {
150 h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
151 h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
169 ol.simple, ul.simple {
173 list-style: decimal }
176 list-style: lower-alpha }
179 list-style: upper-alpha }
182 list-style: lower-roman }
185 list-style: upper-roman }
199 white-space: nowrap }
208 font-family: sans-serif ;
213 font-family: sans-serif ;
225 pre.literal-block, pre.doctest-block {
230 font-family: sans-serif ;
231 font-style: oblique }
233 span.classifier-delimiter {
234 font-family: sans-serif ;
238 font-family: sans-serif }
241 white-space: nowrap }
249 span.section-subtitle {
250 /* font-size relative to parent (h1..h6 element) */
254 border-left: solid 1px gray;
262 margin-bottom: 0.5em }
265 border-left: solid 1px black;
268 table.docutils td, table.docutils th,
269 table.docinfo td, table.docinfo th {
270 padding-left: 0.5em ;
271 padding-right: 0.5em ;
272 vertical-align: top }
274 table.docutils th.field-name, table.docinfo th.docinfo-name {
277 white-space: nowrap ;
280 h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
281 h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
285 list-style-type: none }
290 <div class="document" id="brioche-backup">
291 <h1 class="title">Brioche Backup</h1>
292 <table class="docinfo" frame="void" rules="none">
293 <col class="docinfo-name" />
294 <col class="docinfo-content" />
296 <tr><th class="docinfo-name">Author:</th>
297 <td>Amand Tihon</td></tr>
298 <tr><th class="docinfo-name">Contact:</th>
299 <td><<a class="reference external" href="mailto:amand.tihon@alrj.org">amand.tihon@alrj.org</a>></td></tr>
300 <tr><th class="docinfo-name">Version:</th>
302 <tr><th class="docinfo-name">Date:</th>
303 <td>Jan 9, 2009</td></tr>
304 <tr><th class="docinfo-name">Copyright:</th>
305 <td>GNU GPL, see copyright file.</td></tr>
308 <!-- HTML version generated with rst2html -t README > README.html -->
309 <div class="contents topic" id="table-of-contents">
310 <p class="topic-title first">Table of contents</p>
311 <ul class="auto-toc simple">
312 <li><a class="reference internal" href="#abstract" id="id2">1 Abstract</a></li>
313 <li><a class="reference internal" href="#rationale" id="id3">2 Rationale</a></li>
314 <li><a class="reference internal" href="#getting-brioche" id="id4">3 Getting Brioche</a></li>
315 <li><a class="reference internal" href="#requirements" id="id5">4 Requirements</a></li>
316 <li><a class="reference internal" href="#installation" id="id6">5 Installation</a></li>
317 <li><a class="reference internal" href="#using-brioche" id="id7">6 Using Brioche</a><ul class="auto-toc">
318 <li><a class="reference internal" href="#configuration" id="id8">6.1 Configuration</a></li>
319 <li><a class="reference internal" href="#defining-backups" id="id9">6.2 Defining backups</a></li>
320 <li><a class="reference internal" href="#encryption-with-gnupg" id="id10">6.3 Encryption with GnuPG</a></li>
321 <li><a class="reference internal" href="#using-ftp" id="id11">6.4 Using FTP</a></li>
322 <li><a class="reference internal" href="#running-brioche" id="id12">6.5 Running Brioche</a></li>
325 <li><a class="reference internal" href="#bug-reporting" id="id13">7 Bug reporting</a></li>
326 <li><a class="reference internal" href="#references" id="id14">8 References</a></li>
329 <div class="section" id="abstract">
330 <h1>1 Abstract</h1>
331 <p>Brioche is yet another backup shell script. Its main features are</p>
333 <li>Full and differential backups</li>
334 <li>LVM snapshots</li>
335 <li>Xen oriented (somehow)</li>
336 <li>Upload to a distant FTP server.</li>
339 <div class="section" id="rationale">
340 <h1>2 Rationale</h1>
341 <p>Large numbers of backup solutions are freely available today, but when playing
342 with incremental or differential backups, most of them rely on some filesystem
343 capabilities, like <em>hard-linking</em>. If the only remote location available to
344 store the backups is an FTP server, this is not a solution.
345 Brioche relies on GNU tar's <tt class="docutils literal"><span class="pre">--listed-incremental</span></tt> option to create <em>real</em>
346 differential <a class="footnote-reference" href="#diff" id="id1">[1]</a> archives.</p>
347 <p>A second feature that makes Brioche interresting is the ability to use LVM
348 snapshots. In the case of a Xen setup where the domUs use logical volumes as
349 partitions, Brioche is able to backup everything from the dom0.</p>
350 <table class="docutils footnote" frame="void" id="diff" rules="none">
351 <colgroup><col class="label" /><col /></colgroup>
353 <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Each differential backup is based upon the last <em>full</em> backup, unlike
354 incremental, which are based upon the last (full or incremental) backup.</td></tr>
358 <div class="section" id="getting-brioche">
359 <h1>3 Getting Brioche</h1>
360 <p>You can download the tarball from <a class="reference external" href="http://www.alrj.org/projects/brioche">http://www.alrj.org/projects/brioche</a>
361 or get the latest development version with the following git command:</p>
362 <pre class="literal-block">
363 git clone http://git.alrj.org/git/brioche.git
365 <p>A gitweb interface is also available at <a class="reference external" href="http://git.alrj.org/">http://git.alrj.org/</a></p>
367 <div class="section" id="requirements">
368 <h1>4 Requirements</h1>
369 <p>Brioche relies on a few easily available free software :</p>
371 <li>In any case, Brioche will require GNU tar, which is able to deal with
372 incremental backups. Don't even try it with any other tar implementation.
373 Tested with version 1.16.</li>
374 <li>For LVM snapshots, lvm2 will obviously be needed.</li>
375 <li>FTP backups, if they're used, will require lftp.</li>
377 <p>See the <a class="reference internal" href="#references">References</a> section for links to the aforementioned softwares.</p>
379 <div class="section" id="installation">
380 <h1>5 Installation</h1>
381 <p>Copy the three files <tt class="docutils literal"><span class="pre">brioche</span></tt>, <tt class="docutils literal"><span class="pre">brioche.conf</span></tt> and <tt class="docutils literal"><span class="pre">briochetab</span></tt> where
382 you like, and set the <tt class="docutils literal"><span class="pre">CONFIG_FILE</span></tt> variable in the <tt class="docutils literal"><span class="pre">brioche</span></tt> script
383 accordingly. If needed, run <tt class="docutils literal"><span class="pre">chmod</span> <span class="pre">+x</span> <span class="pre">/path/to/brioche</span></tt>. In a typical setup,
384 the <tt class="docutils literal"><span class="pre">brioche</span></tt> script will be put in <tt class="docutils literal"><span class="pre">/usr/local/bin</span></tt> with the other two
385 files under <tt class="docutils literal"><span class="pre">/etc</span></tt>.</p>
387 <div class="section" id="using-brioche">
388 <h1>6 Using Brioche</h1>
389 <div class="section" id="configuration">
390 <h2>6.1 Configuration</h2>
391 <p>Edit the file <tt class="docutils literal"><span class="pre">brioche.conf</span></tt> to suit your needs. Each option is commented
392 inline and will be detailed here.</p>
393 <dl class="docutils">
395 <dd>Full absolute path to the briochetab file. This file describes which logical
396 volumes and partitions must be backed up. Its format is explained in the
397 <a class="reference internal" href="#defining-backups">Defining backups</a> section.</dd>
399 <dd>When the job is done, Brioche will send a summary of the operations by email.
400 You can set it to any value that your <tt class="docutils literal"><span class="pre">mail</span></tt> command understands.</dd>
402 <dl class="docutils" id="repodir">
404 <dd>Brioche will store all the generated archives in this directory.Note that
405 before doing a full backup, Brioche will move all previous archives into an
406 "undo" subdirectory, which will be removed only if the backup is successful.
407 Make sure there's enough free space on the device where REPODIR is located.</dd>
409 <dd>The report email will include a warning if the space used on REPODIR goes
410 beyond the given threshold. The value must be an integer.</dd>
412 <dd>This directive allows to specify the compression method to apply to the
413 archives. Possible values are "none", "gz", "bz2" and "lzma".
414 <strong>Warning</strong> : lzma may not be available with older versions of GNU tar.</dd>
416 <dd>Additionnal options that you may want to pass to tar. A typical value could
417 be <tt class="docutils literal"><span class="pre">"--one-file-system</span> <span class="pre">-S"</span></tt>. The first option will skip all other
418 mointpoints (very usefull if you have /dev, /proc, /sys or REPODIR mounted
419 under a device that must be archived). The second one will try to deal with
421 <dt>SNAPSHOT_MOUNTPOINT</dt>
422 <dd>Sets the directory where the temporary LVM snapshots must be mounted.</dd>
423 <dt>SNAPSHOT_NAME</dt>
424 <dd>The name to use for the snapshot volumes.</dd>
425 <dt>SNAPSHOT_SIZE</dt>
426 <dd>Set the size of the snapshot volume. The same suffix than for lvcreate(8)
429 <dd>When set to "yes", Brioche will encrypt the archives with GnuPG.
430 See the <a class="reference internal" href="#encryption-with-gnupg">Encryption with GnuPG</a> section below for a detailed explanation on
431 how to use GnuPG with Brioche.</dd>
433 <dd>The identifier of the public GnuPG key to use when encrypting the archives.
434 This is the key that will be needed in case of restore.</dd>
435 <dt>GPG_PASSPHRASE</dt>
436 <dd>The passphrase for the GnuPG private key used to encrypt the archives.</dd>
438 <dd>If set to "yes", Brioche will upload the backups on an FTP server.
439 See the <a class="reference internal" href="#using-ftp">Using FTP</a> section for more information about this feature.</dd>
441 <dd>The address of the FTP server.</dd>
443 <dd>The base directory on the FTP server under which all the archives will be
444 stored. Brioche will never touch anything that is not below this directory.</dd>
446 <dd>Tells Brioche to keep a certain amount of older runs on the FTP. A <em>run</em> is
447 a full backup plus all its subsequent differential backups. See the <a class="reference internal" href="#using-ftp">Using
448 FTP</a> section for a more detailed explanation.</dd>
451 <div class="section" id="defining-backups">
452 <h2>6.2 Defining backups</h2>
453 <p>The backups are defined in the file <tt class="docutils literal"><span class="pre">birochetab</span></tt>. Here is a typical example
454 for a Xen config where cottman is the dom0 and syrtis, kadarin, valeron are
456 <pre class="literal-block">
457 # Partition or LV Snapshot Host name Volume name
458 # ---------------------------------------------------------------
462 /dev/vg00/valeron-root yes valeron root
464 /dev/vg00/kadarin-root yes kadarin root
465 /dev/vg00/kadarin-home yes kadarin home
467 /dev/vg00/syrtis-root yes syrtis root
468 /dev/vg00/syrtis-home yes syrtis home
469 /dev/vg00/syrtis-usr yes syrtis usr
470 /dev/vg00/syrtis-var yes syrtis var
472 <p>Blank lines, or lines beginning with # are ignored.</p>
473 <p>The first column defines the directory or logical volume to backup. In this
474 example,the first two lines are plain directories, while the other ones point
476 <p>The second column specify if the backup should be taken from an LVM snapshot or
477 not. It <strong>must</strong> be set to "no" for the backup of a directory and to "yes" for
478 the backup of a logical volume.</p>
479 <p>The last two columns are more or less cosmetic, and define where the archive
480 files will be stored, and how they'll be named. The destination directory will
481 be created under the <a class="reference internal" href="#repodir">REPODIR</a>, and its name will be the value on the third
482 column. Inside this directory, archive files will be named from the value given
483 in the fourth column. Grouping by domUs' hostnames is only a suggestion, it can
484 be completely different and adapted to suit your needs.</p>
485 <p>For instance, a full and a differential backup for the host valeron of the
486 previous example would lead to the following structure:</p>
487 <pre class="literal-block">
488 user:/REPODIR$ ls -l valeron/
490 -rw-r--r-- 1 root root 363545613 Jan 3 03:10 root.full.20090103.tar.bz2
491 -rw-r--r-- 1 root root 504722 Jan 3 03:10 root.full.snar
492 -rw-r--r-- 1 root root 504725 Jan 4 04:06 root.incr.20090104.snar
493 -rw-r--r-- 1 root root 160542 Jan 4 04:06 root.incr.20090104.tar.bz2
495 <table border="1" class="docutils">
500 <tr><td><p class="first"><strong>WARNING !</strong></p>
501 <p>In case your <a class="reference internal" href="#repodir">REPODIR</a> is not on a distinct device, it will be included
502 in the backup if you've included the device in your <tt class="docutils literal"><span class="pre">briochetab</span></tt>.</p>
503 <p class="last">To avoid this issue, you can either exclude it explicitely by adding an
504 "<tt class="docutils literal"><span class="pre">--exclude=...</span></tt>" option in TAR_OPTS, or simply specify in your
505 <tt class="docutils literal"><span class="pre">briochetab</span></tt> file which directories need to be archived.</p>
511 <div class="section" id="encryption-with-gnupg">
512 <h2>6.3 Encryption with GnuPG</h2>
514 <div class="section" id="using-ftp">
515 <h2>6.4 Using FTP</h2>
516 <p>With the help of lftp, Brioche is able to store an history of backups on an FTP
517 server. This is mainly usefull when no other distant repository is available.
518 If possible, consider using an CIFS, NFS, sshfs or any other kind of remote
519 mountpoint for your <a class="reference internal" href="#repodir">REPODIR</a>.</p>
520 <p>The archives present in the local <a class="reference internal" href="#repodir">REPODIR</a> will be mirrored to the FTP server
521 after each backup, be it a full or a differential one.</p>
522 <p>Since there is no way to hide the credentials if they are passed to lftp on the
523 command line, the authentication relies on your <tt class="docutils literal"><span class="pre">.netrc</span></tt> file. See <tt class="docutils literal"><span class="pre">man</span>
524 <span class="pre">netrc(5)</span></tt> for more information. In the home directory of the user that runs
525 Brioche (typically root's), create the <tt class="docutils literal"><span class="pre">.netrc</span></tt> file with the following
527 <pre class="literal-block">
528 machine ftp.example.com
532 <p>Don't forget to secure it with <tt class="docutils literal"><span class="pre">chmod</span> <span class="pre">600</span> <span class="pre">.netrc</span></tt> or lftp will refuse to use
533 it. The machine name must match the FTP_HOST configuration directive in
534 <tt class="docutils literal"><span class="pre">brioche.conf</span></tt>.</p>
535 <p>On the FTP server, Brioche will keep a configurable amount of <em>runs</em>.
536 Each <em>run</em> consists of a full backup and all the differential backups that are
537 based on it. Before doing a full backup, Brioche will rotate the
538 <em>runs</em> and keep only the configured number of older backups. The current
539 backups can always be found under <tt class="docutils literal"><span class="pre">/FTP_DIR/hostname/latest/</span></tt>. Older ones
540 will be under <tt class="docutils literal"><span class="pre">/FTP_DIR/hostname/run-X/</span></tt> with <em>X</em> equal to 1 for the previous
541 run, 2 for the one before and so forth, up to the value of FTP_KEEP.</p>
542 <p>Here's what happens during the rotation:</p>
544 <li>the oldest run is removed</li>
545 <li>all the <tt class="docutils literal"><span class="pre">run-X/</span></tt> directories are shifted (<tt class="docutils literal"><span class="pre">run-3/</span></tt> becomes <tt class="docutils literal"><span class="pre">run-4/</span></tt>,
547 <li>the <tt class="docutils literal"><span class="pre">latest/</span></tt> directory is renamed to <tt class="docutils literal"><span class="pre">run-1/</span></tt></li>
548 <li>a new, empty, <tt class="docutils literal"><span class="pre">latest/</span></tt> directory is created, ready to accept the new files.</li>
551 <div class="section" id="running-brioche">
552 <h2>6.5 Running Brioche</h2>
553 <p>Brioche understands the following arguments:</p>
554 <table class="docutils option-list" frame="void" rules="none">
555 <col class="option" />
556 <col class="description" />
558 <tr><td class="option-group">
559 <kbd><span class="option">-f</span>, <span class="option">--full</span></kbd></td>
560 <td>Do a full backup (by default, brioche will try to do a
561 differential).</td></tr>
562 <tr><td class="option-group">
563 <kbd><span class="option">-h</span>, <span class="option">--help</span></kbd></td>
564 <td>Show a very limited help.</td></tr>
567 <p>When everything is ready, execute the <tt class="docutils literal"><span class="pre">brioche</span></tt> script as root. The
568 script is pretty verbose, so don't panic if you see lots of lines scrolling in
569 your terminal. By default, Brioche will try to make differential backups, but
570 will gracefully fall back and do a full backup if none is available.</p>
571 <p>If all is fine, it can be added in the system crontab. Here's a suggestion for
572 weekly full backup on Sunday, with differential during the weekdays:</p>
573 <pre class="literal-block">
574 # Daily incremental backup
575 30 3 * * 1-6 /usr/local/bin/brioche > /var/log/backup.`dow`.log 2>&1
576 # Weekly full backup on Sunday
577 30 3 * * 0 /usr/local/bin/brioche -f > /var/log/backup.`dow`.log 2>&1
579 <p>The output of the script will be saved in /var/log/backup.DOW.log with DOW
580 being the abbreviated day of the week (see man date(1) for the format).
581 The <tt class="docutils literal"><span class="pre">dow</span></tt> bash script is provided alongside Brioche.</p>
584 <div class="section" id="bug-reporting">
585 <h1>7 Bug reporting</h1>
586 <p>There's no bugtracker for this project, your bug reports should be sent to the
587 author : Amand Tihon <<a class="reference external" href="mailto:amand.tihon@alrj.org">amand.tihon@alrj.org</a>>. Please include as much information
588 as possible in your report.</p>
590 <div class="section" id="references">
591 <h1>8 References</h1>
593 <li>GNU tar documentation : <a class="reference external" href="http://www.gnu.org/software/tar/manual/">http://www.gnu.org/software/tar/manual/</a></li>
594 <li>LVM documentation and links : <a class="reference external" href="http://sourceware.org/lvm2/">http://sourceware.org/lvm2/</a></li>
595 <li>lftp homepage : <a class="reference external" href="http://lftp.yar.ru/">http://lftp.yar.ru/</a></li>
600 <hr class="footer" />
601 Generated on: 2009-01-09 14:26 UTC.