1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
6 <meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
7 <title>Brioche Backup</title>
8 <meta name="author" content="Amand Tihon" />
9 <meta name="date" content="Jan 5th, 2008" />
10 <meta name="copyright" content="GNU GPL, see copyright file." />
11 <style type="text/css">
14 :Author: David Goodger (goodger@python.org)
15 :Id: $Id: html4css1.css 5196 2007-06-03 20:25:28Z wiemann $
16 :Copyright: This stylesheet has been placed in the public domain.
18 Default cascading style sheet for the HTML output of Docutils.
20 See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
21 customize this style sheet.
24 /* used to remove borders from tables and images */
25 .borderless, table.borderless td, table.borderless th {
28 table.borderless td, table.borderless th {
29 /* Override padding for "table.docutils td" with "! important".
30 The right padding separates the table cells. */
31 padding: 0 0.5em 0 0 ! important }
34 /* Override more specific margin styles with "! important". */
35 margin-top: 0 ! important }
37 .last, .with-subtitle {
38 margin-bottom: 0 ! important }
44 text-decoration: none ;
51 margin-bottom: 0.5em }
53 /* Uncomment (and remove this text!) to get bold-faced definition list terms
61 div.abstract p.topic-title {
65 div.admonition, div.attention, div.caution, div.danger, div.error,
66 div.hint, div.important, div.note, div.tip, div.warning {
68 border: medium outset ;
71 div.admonition p.admonition-title, div.hint p.admonition-title,
72 div.important p.admonition-title, div.note p.admonition-title,
73 div.tip p.admonition-title {
75 font-family: sans-serif }
77 div.attention p.admonition-title, div.caution p.admonition-title,
78 div.danger p.admonition-title, div.error p.admonition-title,
79 div.warning p.admonition-title {
82 font-family: sans-serif }
84 /* Uncomment (and remove this text!) to get reduced vertical space in
86 div.compound .compound-first, div.compound .compound-middle {
87 margin-bottom: 0.5em }
89 div.compound .compound-last, div.compound .compound-middle {
98 div.dedication p.topic-title {
106 div.footer, div.header {
115 div.line-block div.line-block {
121 margin: 0 0 0.5em 1em ;
122 border: medium outset ;
124 background-color: #ffffee ;
129 div.sidebar p.rubric {
130 font-family: sans-serif ;
133 div.system-messages {
136 div.system-messages h1 {
140 border: medium outset ;
143 div.system-message p.system-message-title {
150 h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
151 h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
169 ol.simple, ul.simple {
173 list-style: decimal }
176 list-style: lower-alpha }
179 list-style: upper-alpha }
182 list-style: lower-roman }
185 list-style: upper-roman }
199 white-space: nowrap }
208 font-family: sans-serif ;
213 font-family: sans-serif ;
225 pre.literal-block, pre.doctest-block {
230 font-family: sans-serif ;
231 font-style: oblique }
233 span.classifier-delimiter {
234 font-family: sans-serif ;
238 font-family: sans-serif }
241 white-space: nowrap }
249 span.section-subtitle {
250 /* font-size relative to parent (h1..h6 element) */
254 border-left: solid 1px gray;
262 margin-bottom: 0.5em }
265 border-left: solid 1px black;
268 table.docutils td, table.docutils th,
269 table.docinfo td, table.docinfo th {
270 padding-left: 0.5em ;
271 padding-right: 0.5em ;
272 vertical-align: top }
274 table.docutils th.field-name, table.docinfo th.docinfo-name {
277 white-space: nowrap ;
280 h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
281 h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
285 list-style-type: none }
290 <div class="document" id="brioche-backup">
291 <h1 class="title">Brioche Backup</h1>
292 <table class="docinfo" frame="void" rules="none">
293 <col class="docinfo-name" />
294 <col class="docinfo-content" />
296 <tr><th class="docinfo-name">Author:</th>
297 <td>Amand Tihon</td></tr>
298 <tr><th class="docinfo-name">Contact:</th>
299 <td><<a class="reference external" href="mailto:amand.tihon@alrj.org">amand.tihon@alrj.org</a>></td></tr>
300 <tr><th class="docinfo-name">Version:</th>
302 <tr><th class="docinfo-name">Date:</th>
303 <td>Jan 5th, 2008</td></tr>
304 <tr><th class="docinfo-name">Copyright:</th>
305 <td>GNU GPL, see copyright file.</td></tr>
308 <!-- HTML version generated with rst2html -t README > README.html -->
309 <div class="contents topic" id="table-of-contents">
310 <p class="topic-title first">Table of contents</p>
311 <ul class="auto-toc simple">
312 <li><a class="reference internal" href="#abstract" id="id2">1 Abstract</a></li>
313 <li><a class="reference internal" href="#rationale" id="id3">2 Rationale</a></li>
314 <li><a class="reference internal" href="#getting-brioche" id="id4">3 Getting Brioche</a></li>
315 <li><a class="reference internal" href="#requirements" id="id5">4 Requirements</a></li>
316 <li><a class="reference internal" href="#installation" id="id6">5 Installation</a></li>
317 <li><a class="reference internal" href="#using-brioche" id="id7">6 Using Brioche</a><ul class="auto-toc">
318 <li><a class="reference internal" href="#configuration" id="id8">6.1 Configuration</a></li>
319 <li><a class="reference internal" href="#defining-backups" id="id9">6.2 Defining backups</a></li>
320 <li><a class="reference internal" href="#using-ftp" id="id10">6.3 Using FTP</a></li>
321 <li><a class="reference internal" href="#running-brioche" id="id11">6.4 Running Brioche</a></li>
324 <li><a class="reference internal" href="#bug-reporting" id="id12">7 Bug reporting</a></li>
325 <li><a class="reference internal" href="#references" id="id13">8 References</a></li>
328 <div class="section" id="abstract">
329 <h1>1 Abstract</h1>
330 <p>Brioche is yet another backup shell script. Its main features are</p>
332 <li>Full and differential backups</li>
333 <li>LVM snapshots</li>
334 <li>Xen oriented (somehow)</li>
335 <li>Upload to a distant FTP server.</li>
338 <div class="section" id="rationale">
339 <h1>2 Rationale</h1>
340 <p>Large numbers of backup solutions are freely available today, but when playing
341 with incremental or differential backups, most of them rely on some filesystem
342 capabilities, like <em>hard-linking</em>. If the only remote location available to
343 store the backups is an FTP server, this is not a solution.
344 Brioche relies on GNU tar's <tt class="docutils literal"><span class="pre">--listed-incremental</span></tt> option to create <em>real</em>
345 differential <a class="footnote-reference" href="#diff" id="id1">[1]</a> archives.</p>
346 <p>A second feature that makes Brioche interresting is the ability to use LVM
347 snapshots. In the case of a Xen setup where the domUs use logical volumes as
348 partitions, Brioche is able to backup everything from the dom0.</p>
349 <table class="docutils footnote" frame="void" id="diff" rules="none">
350 <colgroup><col class="label" /><col /></colgroup>
352 <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Each differential backup is based upon the last <em>full</em> backup, unlike
353 incremental, which are based upon the last (full or incremental) backup.</td></tr>
357 <div class="section" id="getting-brioche">
358 <h1>3 Getting Brioche</h1>
359 <p>You can download the tarball from <a class="reference external" href="http://www.alrj.org/projects/brioche">http://www.alrj.org/projects/brioche</a>
360 or get the latest development version with the following git command:</p>
361 <pre class="literal-block">
362 git clone http://git.alrj.org/git/brioche.git
364 <p>A gitweb interface is also available at <a class="reference external" href="http://git.alrj.org/">http://git.alrj.org/</a></p>
366 <div class="section" id="requirements">
367 <h1>4 Requirements</h1>
368 <p>Brioche relies on a few easily available free software :</p>
370 <li>In any case, Brioche will require GNU tar, which is able to deal with
371 incremental backups. Don't even try it with any other tar implementation.
372 Tested with version 1.16.</li>
373 <li>For LVM snapshots, lvm2 will obviously be needed.</li>
374 <li>FTP backups, if they're used, will require lftp.</li>
376 <p>See the <a class="reference internal" href="#references">References</a> section for links to the aforementioned softwares.</p>
378 <div class="section" id="installation">
379 <h1>5 Installation</h1>
380 <p>Copy the three files <tt class="docutils literal"><span class="pre">brioche</span></tt>, <tt class="docutils literal"><span class="pre">brioche.conf</span></tt> and <tt class="docutils literal"><span class="pre">briochetab</span></tt> where
381 you like, and set the <tt class="docutils literal"><span class="pre">CONFIG_FILE</span></tt> variable in the <tt class="docutils literal"><span class="pre">brioche</span></tt> script
382 accordingly. If needed, run <tt class="docutils literal"><span class="pre">chmod</span> <span class="pre">+x</span> <span class="pre">/path/to/brioche</span></tt>. In a typical setup,
383 the <tt class="docutils literal"><span class="pre">brioche</span></tt> script will be put in <tt class="docutils literal"><span class="pre">/usr/local/bin</span></tt> with the other two
384 files under <tt class="docutils literal"><span class="pre">/etc</span></tt>.</p>
386 <div class="section" id="using-brioche">
387 <h1>6 Using Brioche</h1>
388 <div class="section" id="configuration">
389 <h2>6.1 Configuration</h2>
390 <p>Edit the file <tt class="docutils literal"><span class="pre">brioche.conf</span></tt> to suit your needs. Each option is commented
391 inline and will be detailed here.</p>
392 <dl class="docutils">
394 <dd>Full absolute path to the briochetab file. This file describes which logical
395 volumes and partitions must be backed up. Its format is explained in the
396 <a class="reference internal" href="#defining-backups">Defining backups</a> section.</dd>
398 <dd>When the job is done, Brioche will send a summary of the operations by email.
399 You can set it to any value that your <tt class="docutils literal"><span class="pre">mail</span></tt> command understands.</dd>
401 <dl class="docutils" id="repodir">
403 <dd>Brioche will store all the generated archives in this directory.Note that
404 before doing a full backup, Brioche will move all previous archives into an
405 "undo" subdirectory, which will be removed only if the backup is successful.
406 Make sure there's enough free space on the device where REPODIR is located.</dd>
408 <dd>The report email will include a warning if the space used on REPODIR goes
409 beyond the given threshold. The value must be an integer.</dd>
411 <dd>This directive allows to specify the compression method to apply to the
412 archives. Possible values are "gz", "bz2" and "lzma". <strong>Warning</strong> : lzma may
413 not be available with older versions of GNU tar.</dd>
415 <dd>Additionnal options that you may want to pass to tar. A typical value could
416 be <tt class="docutils literal"><span class="pre">"--one-file-system</span> <span class="pre">-S"</span></tt>. The first option will skip all other
417 mointpoints (very usefull if you have /dev, /proc, /sys or REPODIR mounted
418 under a device that must be archived). The second one will try to deal with
420 <dt>SNAPSHOT_MOUNTPOINT</dt>
421 <dd>Sets the directory where the temporary LVM snapshots must be mounted.</dd>
422 <dt>SNAPSHOT_NAME</dt>
423 <dd>The name to use for the snapshot volumes.</dd>
424 <dt>SNAPSHOT_SIZE</dt>
425 <dd>Set the size of the snapshot volume. The same suffix than for lvcreate(8)
428 <dd>If set to "yes", Brioche will upload the backups on an FTP server.
429 See the <a class="reference internal" href="#using-ftp">Using FTP</a> section for more information about this feature.</dd>
431 <dd>The address of the FTP server.</dd>
433 <dd>The base directory on the FTP server under which all the archives will be
434 stored. Brioche will never touch anything that is not below this directory.</dd>
436 <dd>Tells Brioche to keep a certain amount of older runs on the FTP. A <em>run</em> is
437 a full backup plus all its subsequent differential backups. See the <a class="reference internal" href="#using-ftp">Using
438 FTP</a> section for a more detailed explanation.</dd>
441 <div class="section" id="defining-backups">
442 <h2>6.2 Defining backups</h2>
443 <p>The backups are defined in the file <tt class="docutils literal"><span class="pre">birochetab</span></tt>. Here is a typical example
444 for a Xen config where cottman is the dom0 and syrtis, kadarin, valeron are
446 <pre class="literal-block">
447 # Partition or LV Snapshot Host name Volume name
448 # ---------------------------------------------------------------
452 /dev/vg00/valeron-root yes valeron root
454 /dev/vg00/kadarin-root yes kadarin root
455 /dev/vg00/kadarin-home yes kadarin home
457 /dev/vg00/syrtis-root yes syrtis root
458 /dev/vg00/syrtis-home yes syrtis home
459 /dev/vg00/syrtis-usr yes syrtis usr
460 /dev/vg00/syrtis-var yes syrtis var
462 <p>Blank lines, or lines beginning with # are ignored.</p>
463 <p>The first column defines the directory or logical volume to backup. In this
464 example,the first two lines are plain directories, while the other ones point
466 <p>The second column specify if the backup should be taken from an LVM snapshot or
467 not. It <strong>must</strong> be set to "no" for the backup of a directory and to "yes" for
468 the backup of a logical volume.</p>
469 <p>The last two columns are more or less cosmetic, and define where the archive
470 files will be stored, and how they'll be named. The destination directory will
471 be created under the <a class="reference internal" href="#repodir">REPODIR</a>, and its name will be the value on the third
472 column. Inside this directory, archive files will be named from the value given
473 in the fourth column. Grouping by domUs' hostnames is only a suggestion, it can
474 be completely different and adapted to suit your needs.</p>
475 <p>For instance, a full and a differential backup for the host valeron of the
476 previous example would lead to the following structure:</p>
477 <pre class="literal-block">
478 user:/REPODIR$ ls -l valeron/
480 -rw-r--r-- 1 root root 363545613 Jan 3 03:10 root.full.20090103.tar.bz2
481 -rw-r--r-- 1 root root 504722 Jan 3 03:10 root.full.snar
482 -rw-r--r-- 1 root root 504725 Jan 4 04:06 root.incr.20090104.snar
483 -rw-r--r-- 1 root root 160542 Jan 4 04:06 root.incr.20090104.tar.bz2
485 <table border="1" class="docutils">
490 <tr><td><p class="first"><strong>WARNING !</strong></p>
491 <p>In case your <a class="reference internal" href="#repodir">REPODIR</a> is not on a distinct device, it will be included
492 in the backup if you've included the device in your <tt class="docutils literal"><span class="pre">briochetab</span></tt>.</p>
493 <p class="last">To avoid this issue, you can either exclude it explicitely by adding an
494 "<tt class="docutils literal"><span class="pre">--exclude=...</span></tt>" option in TAR_OPTS, or simply specify in your
495 <tt class="docutils literal"><span class="pre">briochetab</span></tt> file which directories need to be archived.</p>
501 <div class="section" id="using-ftp">
502 <h2>6.3 Using FTP</h2>
503 <p>With the help of lftp, Brioche is able to store an history of backups on an FTP
504 server. This is mainly usefull when no other distant repository is available.
505 If possible, consider using an CIFS, NFS, sshfs or any other kind of remote
506 mountpoint for your <a class="reference internal" href="#repodir">REPODIR</a>.</p>
507 <p>The archives present in the local <a class="reference internal" href="#repodir">REPODIR</a> will be mirrored to the FTP server
508 after each backup, be it a full or a differential one.</p>
509 <p>Since there is no way to hide the credentials if they are passed to lftp on the
510 command line, the authentication relies on your <tt class="docutils literal"><span class="pre">.netrc</span></tt> file. See <tt class="docutils literal"><span class="pre">man</span>
511 <span class="pre">netrc(5)</span></tt> for more information. In the home directory of the user that runs
512 Brioche (typically root's), create the <tt class="docutils literal"><span class="pre">.netrc</span></tt> file with the following
514 <pre class="literal-block">
515 machine ftp.example.com
519 <p>Don't forget to secure it with <tt class="docutils literal"><span class="pre">chmod</span> <span class="pre">600</span> <span class="pre">.netrc</span></tt> or lftp will refuse to use
520 it. The machine name must match the FTP_HOST configuration directive in
521 <tt class="docutils literal"><span class="pre">brioche.conf</span></tt>.</p>
522 <p>On the FTP server, Brioche will keep a configurable amount of <em>runs</em>.
523 Each <em>run</em> consists of a full backup and all the differential backups that are
524 based on it. Before doing a full backup, Brioche will rotate the
525 <em>runs</em> and keep only the configured number of older backups. The current
526 backups can always be found under <tt class="docutils literal"><span class="pre">/FTP_DIR/hostname/latest/</span></tt>. Older ones
527 will be under <tt class="docutils literal"><span class="pre">/FTP_DIR/hostname/run-X/</span></tt> with <em>X</em> equal to 1 for the previous
528 run, 2 for the one before and so forth, up to the value of FTP_KEEP.</p>
529 <p>Here's what happens during the rotation:</p>
531 <li>the oldest run is removed</li>
532 <li>all the <tt class="docutils literal"><span class="pre">run-X/</span></tt> directories are shifted (<tt class="docutils literal"><span class="pre">run-3/</span></tt> becomes <tt class="docutils literal"><span class="pre">run-4/</span></tt>,
534 <li>the <tt class="docutils literal"><span class="pre">latest/</span></tt> directory is renamed to <tt class="docutils literal"><span class="pre">run-1/</span></tt></li>
535 <li>a new, empty, <tt class="docutils literal"><span class="pre">latest/</span></tt> directory is created, ready to accept the new files.</li>
538 <div class="section" id="running-brioche">
539 <h2>6.4 Running Brioche</h2>
540 <p>Brioche understands the following arguments:</p>
541 <table class="docutils option-list" frame="void" rules="none">
542 <col class="option" />
543 <col class="description" />
545 <tr><td class="option-group">
546 <kbd><span class="option">-f</span>, <span class="option">--full</span></kbd></td>
547 <td>Do a full backup (by default, brioche will try to do a
548 differential).</td></tr>
549 <tr><td class="option-group">
550 <kbd><span class="option">-h</span>, <span class="option">--help</span></kbd></td>
551 <td>Show a very limited help.</td></tr>
554 <p>When everything is ready, execute the <tt class="docutils literal"><span class="pre">brioche</span></tt> script as root. The
555 script is pretty verbose, so don't panic if you see lots of lines scrolling in
556 your terminal. By default, Brioche will try to make differential backups, but
557 will gracefully fall back and do a full backup if none is available.</p>
558 <p>If all is fine, it can be added in the system crontab. Here's a suggestion for
559 weekly full backup on Sunday, with differential during the weekdays:</p>
560 <pre class="literal-block">
561 # Daily incremental backup
562 30 3 * * 1-6 /usr/local/bin/brioche > /var/log/backup.`date "+%a"`.log 2>&1
563 # Weekly full backup on Sunday
564 30 3 * * 0 /usr/local/bin/brioche -f > /var/log/backup.`date "+%a"`.log 2>&1
566 <p>The output of the script will be saved in /var/log/backup.DOW.log with DOW
567 being the abbreviated day of the week (see man date(1) for the format).</p>
570 <div class="section" id="bug-reporting">
571 <h1>7 Bug reporting</h1>
572 <p>There's no bugtracker for this project, your bug reports should be sent to the
573 author : Amand Tihon <<a class="reference external" href="mailto:amand.tihon@alrj.org">amand.tihon@alrj.org</a>>. Please include as much information
574 as possible in your report.</p>
576 <div class="section" id="references">
577 <h1>8 References</h1>
579 <li>GNU tar documentation : <a class="reference external" href="http://www.gnu.org/software/tar/manual/">http://www.gnu.org/software/tar/manual/</a></li>
580 <li>LVM documentation and links : <a class="reference external" href="http://sourceware.org/lvm2/">http://sourceware.org/lvm2/</a></li>
581 <li>lftp homepage : <a class="reference external" href="http://lftp.yar.ru/">http://lftp.yar.ru/</a></li>
586 <hr class="footer" />
587 Generated on: 2009-01-06 15:56 UTC.