# scdbackup-0.9.2
http://scdbackup.webframe.org scdbackup@gmx.net Thomas Schmitt
http://scdbackup.sourceforge.net
Purpose:
backup large amounts of data on one or more CDs, DVDs, or BDs,
simple (therefore no excuse not to do the backup),
no special tool needed for reading the backup
Formats:
ISO 9660 file systems (readable after : mount )
afio archives (readable by : afio ... /dev/cdrom )
star archives (readable by : star ... </dev/cdrom )
Special features:
Breaks the 650 MB / 4.7 GB / 25 GB limits (more than one CD / DVD / BD)
Information script on each media tells where a certain file may be found
Interrupted backups can be resumed at the start of the failed piece
Checksums appended to media
Optional multi copy backups provide ability to repair damaged volumes
Incremental backups with an arbitrary number of levels
Restore utility searches most recent backuped file versions
Commands for CD:
scdbackup_home ... backup of the user's $HOME directory. ISO 9660 .
scdbackup ........ command for defining own backup collections and for
spontaneous data copies. ISO 9660. Example:
scdbackup /home /home2=/usr/home -not /home/guest
Backup /home and /usr/home . But use /home2 on CD as
name for /usr/home and omit subdir /home/guest .
scdbackup_afio ... like scdbackup but with afio archives rather than ISO 9660.
This may also produce star-Archives.
scdbackup_sys .... backup of system directories as afio archives (or star).
scdbackup_verify . test a resulting CD for correct readability.
scdbackup_blank .. blank a CD-RW to make it writable again or to make
unreadable the previously stored data.
scdbackup_askme .. read a backup configuration or content list, perform
fast search and eventually guide user through restore from
ISO 9660 backups with multiple volumes and levels.
scdbackup_clear .. release all locks (in case one has survived the end
of a script).
For DVD/BD according commands are available. They all begin with "sdvdbackup" :
sdvdbackup_home, sdvdbackup, sdvdbackup_afio, sdvdbackup_sys,
sdvdbackup_verify, sdvdbackup_blank, sdvdbackup_askme, sdvdbackup_clear
With BD (Blu-ray) of nominal "25 GB" use option: -bd
For "50GB" use: -bd_dl
Software requirements (see appendix LINKS for addresses):
cc
A C language compiler including linker and other usual companions
is needed for installing scdbackup from source but not for its
usage.
/bin/sh
compatible to GNU Bourne-Again SHell (bash) or Korn Shell (ksh)
including usual programs like: cat, date, du, echo, ls, ...
see hints for working around shell compatibility problems in
script change_shell_to_use .
gzip
data compression program (for incremental backup management)
Required for writing CD and formatting ISO-Filesystems:
xorriso
with program xorriso based on libisofs and libburn for CD/DVD/BD
or cdrtools
with programs cdrecord and mkisofs
or cdrkit
with programs wodim and genisoimage
Additionally advised for writing DVD or BD if not xorriso is used anyway:
dvd+rw-tools
with programs growisofs, dvd+rw-mediainfo, dvd+rw-format .
or cdrskin
with program cdrskin based on libburn
are preferred over the burn programs of contemporary cdrtools or cdrkit
which provide less comfortable DVD write services but may be used
via scdbackup, nevertheless. Their usability with BD is unclear.
Additionally required for system backups, compressed or encrypted backups:
afio
Program afio is a well tested and solid archiver.
or star
Program star is an archiver which can make very complete
backups including ACLs.
This program system is constantly tested on Intel/AMD Linux systems.
Ports to other usable systems are appreciated.
Table of Content:
- Planning (of usage and installation)
- Installation (at first for a normal desktop user)
- Superuser Planning (of an installation used by the superuser)
- Usage
Backup Commands
ISO 9660 Filesystems
afio Archives (or star archives)
Restartability
Verifying CDs and DVDs
Blanking CD-RW or DVD-RW , Erasing data from media
Incremental Backups
Backup Information Scripts
Backup Information+Restore Tool
Access Permissions (of the scdbackup installation)
- Inner details - Appendice
DIRECTORIES Directories of a scdbackup installation
CONFIGURATION Configuration files and variables
WORKERS Operational components of scdbackup (the software)
RECOVERY Precautions for and recovery from disasters
AUTOMAT Issues of unattented backup runs
VERIFY Format of checksum records, tags, block checksum lists
REDUNDANCY Repairable multi copy backups
CRYPTO Encrypted afio or star backups
ATTRLIST Recording and restoring ACLs and Extended File Attributes
LINKS Links to supporting software
ISO ISO 9660 formatter programs
CD CD burn programs
DVD DVD and BD burn programs
DVD-TYPES Usage and peculiarties of DVD and BD
IDE Using consumer grade burners with Linux
NET Burning via a remote scdbackup installation over network
TUNING Options to make scdbackup faster or leaner
INODE Standard method of incremental backup and alternatives
SNAPSHOT How to deal with filesystem snapshots
WRITER Interface specifications for write adapter programs
-----------------------------------------------------------------------
Planning
Burning of CD or DVD on Linux relies on applications which format and record
the data. The most commonly readable format is ISO 9660.
scdbackup's configuration will request some parameters from you. So you have
to find out the appropriate answers and eventually prepare resources. The
non-trivial parameters and eventual preparations are described in the following
text up to paragraph "Installation".
scdbackup needs an external formatter program and external burn programs.
Supported ISO 9660 formatters: xorriso, genisoimage, mkisofs
Supported CD burn programs: cdrskin, cdrecord, xorriso, wodim
Supported DVD burn programs: growisofs, cdrskin, xorriso, cdrecord, wodim
The configuration scripts of scdbackup will check for available programs
with above names and ask the user to choose one of them.
If your ISO formatted disks shall be read with long names by common MS-Windows
systems then you will have to add "-J" to the name of the ISO 9660 formatter
program (like "xorriso -J").
Without "-J" the files will appear on MS-Windows with truncated names but
will bear their original content.
If there occur problems with using the recorder then ask the sysadmin to read
paragraph "Permission to use the recorder" and ask for appropriate remedy.
Instead of a burn program it is possible to use a scdbackup installation on
a remote computer via SSH. (See appendix NET.)
For CD
When configuring for CD you will have to enter a "Address of CD recorder".
If in that moment you do not see a list of addresses like the following,
or if you recorder does not show up, then there is a permission problem and
the superuser will have to interfere.
Either
------------------- SCSI devices. To be used like 0,0,0
4,0,0 400) 'PHILIPS ' 'SPD3300L ' 'H5S2' Removable CD-ROM
------------------- end of SCSI device list
------------------- ATA devices. To be used like ATA:0,0,0
0,0,0 0) 'HL-DT-ST' 'DVDRAM GSA-4082B' 'A201' Removable CD-ROM
------------------- end of ATA device list
or
----------------------------------------------------- devices
/dev/sr0 : 'PHILIPS ' 'SPD3300L'
/dev/hda : 'HL-DT-ST' 'DVDRAM GSA-4082B'
----------------------------------------------------- end of device list
In this example the address is "ATA:0,0,0" resp. "/dev/hda" for the LG GSA
burner and "4,0,0" resp "/dev/sr0" for the PHILIPS burner.
With question "Speed ?" you should enter the maximum value among the available
media. Slower media will be handled properly by the burn programs.
For DVD
Configuration for DVD traditionally depends on operational commands growisofs,
dvd+rw-mediainfo and dvd+rw-format from package dvd+rw-tools.
Nevertheless, modern versions of above CD burn programs are also able to write
to DVD media.
Programs wodim and cdrecord are inferior for the purposes of sdvdbackup. They
do not allow to write a stream of data of which the exact length cannot be
predicted.
growisofs has the best experience with uncommon media types. A disadvantage
is that the package dvd+rw-tools contains no program to list accessible drives.
So one eventually has to know the DVD writer's device file address.
/dev/dvd may or may not be right.
If your sysadmin doesn't know (after reading Appendix DVD) and your system
management software doesn't tell, then close the drive trays and try commands
like eject /dev/sr0 or eject /dev/dvd . If it reacts, then that address is
worth a try.
The other burn programs will allow the configuration script to list the
accessible drives as it is done for CD configuration.
With growisofs "Speed ?" will be "any" for good media and "1" for poorly
performing disks.
For the other burn programs numbers like "2" or "4" are appropriate.
You will get asked for a "Large file split directory" and a "Buffer file name".
You will only need that if you got single files larger than approximately
600 MB (or 2 GB minus 8 kB with DVD) or if your computer system and burner
are too slow and too picky to avoid misburns due to lowspeed data transfer.
If you do not expect the need for said "split directory" or "Buffer file"
then skip now to "Installation".
Disk buffer
Split directory and buffer file may be two buffer objects which each can occupy
as much space on your hard disk as fits on a single media (i.e. 650 MB resp.
4.7 GB). Therefore it is possible that you cannot afford both or even a single
one of them.
Splitting of files is needed with DVD for files of size near 2 GB or larger.
That is because a lot of reader software handles files properly only if they
are a bit smaller than 2 GB. With CD you will need splitting if you want to
backup files that do not fit on a single media.
ISO formatter program xorriso can perform file splitting on the fly. Eventually
enter "-xorriso_cut_out" when asked for "Large file split directory".
The other ISO formatter programs mkisofs and genisoimage, as well as the
archive formatters afio and star need a split directory.
Eventually one can combine "-xorriso_cut_out" with a split directory for
archive formatters. Like: "-xorriso_cut_out:/my/archiver/split_directory".
If there are oversized files and if no splitting is enabled, then those
files are excluded from backups, unless this security feature has been
disabled explicitely by input "-allow_large_files".
The buffer file is eventually used for assembling the data of a backup volume
on disk before they are handed over to the burner without putting a high
load on the system. This may be necessary if the system is heavily loaded or
if a compressed archive format is to be written and if the burner hardware got
no protection against buffer underruns (Burnproof, Smartburn, etc.).
Decision criteria:
If you got oversized files and want to backup them, you will absolutely need
splitting. If your formatter is not xorriso >= 0.1.2 this means you need a
split directory.
The buffer file should not be necessary if you got modern hardware.
(It is useful for advanced purposes, nevertheless.)
After you decided to have a split directory, find a suitable filesystem and
create a new empty directory there.
For a buffer file you would just have to choose a name in an existing directory
on a suitable filesystem. Suitable here means : with enough free space.
You must have separate directories resp. files for CD and DVD unless you
only got one single burner which is in charge for both media types. I strongly
recommend to use different names in any case.
Set the access permissions of the newly created split directories to rwx
for the interested user(s). (In most frigthening case: chmod a+rwx )
Make sure the directory of an eventual bufferfile offers rwx-permissions
to all those users.
Each single scdbackup installation (i.e. for security reasons : each user)
should use own addresses for their buffer directories and buffer files.
------------------------------------------------------------------------------
Installation
To achieve a clear security situation each user should have an own separate
scdbackup installation.
Steps 1 and 2 must be done by the particular user who shall use that
installation. This is not necessarily the superuser but rather the owner
of the backup data.
The superuser may follow these steps for the installation of an own,
very private backup system. But superuser authority will elsewise only
be needed for a few optional activities.
If programs cdrecord and/or growisofs are installed properly on the system
and may be used by the installing user for burning, then there is no
urgent need for superuser actions at all.
Each of the steps may be repeated to restore original software
and permission settings or to edit the configuration.
--- Step 1 :
Copy the archive to a directory where you can create a sub directory
scdbackup-0.9.2, go there, unpack the archive and go to the sub directory
inst newly created by this. For example use $HOME/scdbackup :
mkdir $HOME/scdbackup
cd $HOME/scdbackup
tar xzf $HOME/downloaded_stuff/scdbackup-0.9.2.tar.gz
cd scdbackup-0.9.2/inst
If the superuser is doing this, it is also absolutely necessary to do
chown root ../../scdbackup-0.9.2/*/* ../../scdbackup-0.9.2/* \
../../scdbackup-0.9.2
and best to issue an appropriate chgrp command, too.
--- Step 1b (only needed when upgrading from an older version) :
If you already had installed another version of scdbackup, you may now
copy some history data and the old configuration as a template into
the new directory :
./FETCH_CONFIGURATION
This refuses to overwrite any files so it only makes sense in a freshly
unpacked directory. Any users afterwards are still configured to use their
old installation.
If you maintain incremental backup configurations and suffered from long
planning times you may be interested in reading the first paragraph of
appendix INODE below.
Only if you programmed own applications which make use of inner parts of
scdbackup previous to version 0.8.5, only then you should now read appendix
DIRECTORIES.
The mere use of commands scdbackup* or sdvdbackup* within your programs
will continue to work without ./COMPATIBILITY_LINKS, of course.
Unless the user id number is 0 the following scripts write the files
.scdbackup_dir and .scdbackup_lang into the user's $HOME-Directory.
They eventually write file .scdbackup_userid into the user's $HOME-Directory.
--- Step 2a if you got a CD writer (for DVD writer only, see Step 2b below) :
While the CD-Recorder is not busy, run :
./CONFIGURE_CD
Several questions have to be answered:
What ISO 9660 formatter program to use ? (... found programs ...)
What CD burn program to use ? (... found programs ...)
Address of CD recorder ? (... found addresses ...)
Speed ? (e.g. 2)
Automatically erase CD before writing ? (yj1/n0)
Large file split directory ? ( "-" = exclude large files)
Buffer file name ? ( or "-" or "-afio_compress" )
Estimated capacity for a compressed afio CD ? (eg. 1024m)
Exclusion list of directories for system backup ?
( With other shells than bash : Your user id number ? )
If no disk buffer (see above) is needed then answer the question
"Large file split directory" by a single "-" or if advised by the message
text: "-xorriso_cut_out".
"Buffer file name" should be "-afio_compress" for true burnproof hardware
and media, else enter "-" or a filename.
The "Exclusion list of directories for system backup" is only of interest
if you are superuser right now. As normal user just press the Enter key.
--- Step 2b if you got a DVD writer (works with or without Step 2a) :
If neither command growisofs nor cdrskin nor wodim nor cdrecord is not found
by your shell then you will have to install dvd+rw-tools as described in
Appendix DVD.
While the DVD writer is not busy, run :
./CONFIGURE_DVD
Several questions have to be answered:
What ISO 9660 formatter program to use ? (... found programs ...)
What DVD burn program to use ? (... found programs ...)
Address of DVD recorder ? (... found addresses ...)
Speed ? (e.g. 2)
Automatically erase DVD before writing ? (yj1/n0)
Large file split directory ? ( "-" = exclude large files)
Buffer file name ? ( or "-" or "-afio_compress" )
Estimated capacity for a compressed afio DVD ? (e.g. 8000m)
Exclusion list of directories for system backup ?
( Non-bash users also may be prompted for a unique user id number )
If no disk buffer (see above) is needed then answer the question
"Large file split directory" by a single "-" or if advised by the message
test: "-xorriso_cut_out".
"Buffer file name" should be "-afio_compress" for true burnproof hardware
and media, else enter "-" or a filename.
The "Exclusion list of directories for system backup" is only of interest
if you are superuser right now. As normal user just press the Enter key.
--- Step 2c. (Only with private installation for superuser)
For his own private installation of scdbackup, the superuser has to run :
./ADD_USER
Make no mistake : only if you are already superuser now and only with the
installation which shall do the superuser's backup jobs and where no other
user is permitted.
After Configuration
Permission settings after a first configuration will not allow others to
use your scdbackup installation. This is for your security. Stay with it.
The commands of scdbackup are now accessible in subdirectory ./cmd
of $HOME/scdbackup/scdbackup-0.9.2 .
Make a first test :
$HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home -version
should reply "scdbackup 0.9.2 2009.12.13.060857" (timestamp varies).
You may now add $HOME/scdbackup/scdbackup-0.9.2/cmd to your PATH variable :
export PATH="$PATH":"$HOME/scdbackup/scdbackup-0.9.2/cmd"
and then execute commands like
scdbackup_home -version
There are startup scripts like .profile or .bashrc where you may add
this directory path permanently. Ask your sysadmin.
If this is an upgrade from scdbackup-0.8 or older then the superuser will
have to remove the old commands in /usr/bin (or wherever they are) or to
overwrite them as described in the next paragraph.
If this is your first encounter with scdbackup then i advise you to next
read "Usage" or just try
$HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home
(without "-version").
The eventual superuser activities can wait for an hour - if burn program and
CD/DVD/BD writing device are in proper state, that is.
Superuser Part of Installation
The superuser may decide to copy the scdbackup commands to a public command
directory rather than to teach his users how to permanently set their PATH.
These command scripts are quite small and merely direct the users to their
own scdbackup installations. Their only hazard is to get tricked into a fake
scdbackup installation by the user's own environment variables SCDBACKUP_DIR
or HOME or by the user's own configuration file $HOME/.scdbackup_dir .
Regular installations of other users should repel such an attempt, anyway.
With upgrades from scdbackup-0.8 or older, the superuser has to make sure
that the old installed commands get removed or overwritten by the new ones.
Commands "which scdbackup" and "which scdbackup_sys" should tell where they
are located.
A mistrusting superuser may decide to install the commands manually
following this template (copy commands to /usr/local/bin and /usr/sbin ):
target=/usr/local/bin
chmod a-w ../scdbackup-0.9.2/cmd/*
cp ../scdbackup-0.9.2/cmd/* "$target"
chown root "$target"/scdbackup* "$target"/sdvdbackup*
chmod a+rx,u+w,go-w "$target"/scdbackup* "$target"/sdvdbackup*
chmod go-x "$target"/scdbackup_sys "$target"/sdvdbackup_sys
(have a look into /usr/local/sbin whether there is s*dbackup_sys to be removed)
But there is a script which does this more comfortably, reliably and avoids
any "*" in the commands.
If the superuser intends to use scdbackup for own backup runs, then there is
no reason to particularly mistrust the following script. (Bug reports welcome)
--- Step 3 (optional, see possible reasons above)
Become superuser now and run (within directory scdbackup-0.9.2/inst)
./SUPERUSER_CONFIGURE
This script copies the command scripts scdbackup* to a public command directory
like /usr/bin or /usr/local/bin if CD writing is actually configured.
If DVD writing has been configured, then the commands sdvdbackup* are copied
to that directory.
You will be asked for the public command directory where to install the scripts
Install directory ? (e.g. /usr/bin , "-" = do not install)
The user should afterwards check whether
scdbackup_home -version
yields the same result as in the first test :
$HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home -version
There is no need to repeat step 3 with further user's installations.
Permission to use the recorder:
Depending on the burn program used, the permission to use a CD/DVD recorder
has to be achieved in different ways:
cdrecord (package cdrtools) Needs to be run by the superuser or to be owned
by the superuser and treated with chmod u+s
growisofs (package dvd+rw-tools) The user needs rw-permission for /dev/srN
resp. /dev/hdX. Uses device addresses but no
SCSI numbers. chmod u+s is deprecated.
wodim (package cdrkit) Either to be used like cdrecord or to be used
under normal user id with rw-permissions on
appropriate /dev/scdN resp. /dev/hdX files.
cdrskin (packages cdrskin The user needs rw-permission for /dev/srN resp.
or libburn) /dev/hdX. The superuser should perform
cdrskin --devices
to learn about the device files for chmod +rw
Application of chmod u+s is strongly deprecated.
xorriso (packages xorriso Same preconditions as with cdrskin. Uses device
or libisoburn) addresses but no SCSI numbers. xorriso is a
burn program as well as an ISO 9660 formatter.
If the superuser does not have an own scdbackup installation yet, then this
would nearly be the time to start -as superuser- with step 1 in order
to get one.
But first read this :
Superuser Planning
The superuser will most likely use format afio unless there are user data
to take care of, which clearly benefit from ISO 9660 format. By help of script
star_as_afio_wrapper one may alternatively employ the archive format of
program star. (See example in appendix WORKERS.)
The idea is to have a rather slim system backup, which is made with
scdbackup_sys
Be aware that scdbackup_sys does not create a recovery system but
just records the files (even quite special ones) in an afio archive.
This conserves the system configuration, but for re-installing you
need to do a base installation of Linux first.
The base installation has to be compatible with the system you made
the archive from. So make copies of your Linux CDs to have a compatible
system at hand when needed.
You may also use program star as intended by its author to make a full backup
of the system partitions. This will not need copies of the originally
installed Linux distribution media.
See appendix RECOVERY for details of disaster precautions and recovery.
Alternatively you may choose to use a disaster recovery system like mkCDrec
( http://mkcdrec.ota.be/project/index.html ) which tries to save you as quick
as possible from the consequences of big accidents.
Nevertheless, if you cannot get the old hardware running again it might be
impossible to use system backups of any kind flatly. Then you have to install
a completely new operating system and may use the old system backup only for
as source of configuration files.
I myself never lost a complete system but had occasions when an older
reference copy of the system files came in very handy.
User data which rely on exact file attributes or which are likely to
change during the runtime of the backup should not be stored as ISO 9660
if mkisofs or genisoimage is used as ISO formatter program.
Modern versions of program xorriso can record all file attributes including
ACLs but are still more prone to file size change mishaps than sequential
archive formats.
So for at least for agile data, one should better use scdbackup_afio .
Usually there are several kinds of data on a Linux disk, which should be
backuped:
Persistent system objects, often with special attributes and permissions
Installed application packages (often below /opt)
User created or collected data (often below /home)
Other kinds of data do not need backup or would even make problems :
Volatile system objects (e.g. /proc)
Temporary files (e.g. /tmp or $HOME/.netscape/cache)
Removable filesystems (/mnt /floppy /cdrom)
During installation, you will be asked for "Exclusion list of directories
for system backup".
This list should contain all directories under / , which do not contain
persistent system data.
Example (have a look in your own / directory for more candidates) :
/opt /home /mnt /floppy /cdrom /cdrw /tmp /proc /sys
All other subdirectories of the root directory will be covered by the system
backup. Among the excluded directories, only /opt and /home need backup.
If there is only one real user who has not got a giant amount of data, a
combined backup of /opt and /home is appropriate. Since my Netscape cache is
quite large, i exclude it.
scdbackup_afio /home /opt -not /home/thomas/.netscape/cache
If there are very large amounts of data, it will be convenient to have separate
backups. These backups can be run with different frequency. /opt will not need
to be backuped as often as /home/projects .
scdbackup_afio /opt
scdbackup_afio /home -not /home/projects /home/thomas/.netscape/cache
scdbackup /home/projects
Real users (like "thomas", "guest1", "guest2") should do separate backups with
scdbackup_home
or
scdbackup_afio $HOME
Exclude their $HOME directories from the general data backup
scdbackup_afio /home \
-not /home/thomas /home/guest1 /home/guest2
Next you should make a time schedule, when to do which backup. It is generally
a good idea to have at least three older backups. From time to time a permanent
backup could be made on CD-R in order to be kept forever.
On the other hand ... be aware what traces you leave for future archeologists.
Permission settings after an installation will not allow others to
use a scdbackup installation, unless these restrictions are lifted
by ./SET_PERMISSIONS (see below "Access Permissions") or unless
laxer settings have been fetched during ./FETCH_CONFIGURATION.
Interested and authorized users then go to the installation directory
and perform ./ADD_USER .
Now go and -as superuser- begin with step 1 of the installation procedure.
If the superuser did not use scdbackup before , execute as step 1b :
./SUPERUSER_FETCH_CONF /home/thomas/scdbackup/scdbackup-0.9.2
to get the previously made settings of user "thomas" as templates for
own ./CONFIGURE_CD resp ./CONFIGURE_DVD . Else
./FETCH_CONFIGURATION
is ok.
In any case : check what you confirm with ./CONFIGURE_CD and ./CONFIGURE_DVD .
Take care not to allow others to access your eventual disk buffer objects.
Best is to have none, if one can live without them.
-----------------------------------------------------------------------
Usage
scdbackup offers different formats for the data on media.
The data format preferable for user data files is in most cases ISO 9660.
The media are mountable and the files may be used directly by any program that
can handle files which offer no write access.
This format is also most likely to be accessible on other operating systems.
In the most simple case, command scdbackup_home is sufficient.
For arbitrary data locations use command scdbackup .
As said, this is ok for directories trees and data files of a single user,
where write permissions are general and simple.
mkisofs with version numbers below 2.0 will not reliably cope with symbolic
links.
Independently of eventual weaknesses of the ISO formatter program there may
also arise problems with the filesystem drivers. Not all of them show the
w-permissions of directories correctly.
Another disadvantage is that scdbackup does not offer data compression
together with ISO format.
For such purposes, scdbackup offers the format of the reliable archiver afio
or the format of Joerg Schilling's backup program star.
The commands scdbackup_afio and scdbackup_sys produce these formats.
Backup Commands
The backup creating command scripts of the scdbackup-system all accept
the options described by
cd_backup_planer -help | less
but usually it is only necessary to provide some directory or file names.
See also doc/examples.html for commented usage examples.
ISO 9660 Filesystems
These filesystems can be mounted on devices like /dev/cdrom or /dev/dvd .
They also can be exchanged with other systems like Windows or Mac.
scdbackup_home a command that backups the user's $HOME directory on CD
except the Netscape cache. No arguments needed.
sdvdbackup_home DVD version of above.
scdbackup a command that backups the files and directories
given as arguments by the user.
These objects may be renamed by giving the target name
before '=' and the source name.
Example: rename /usr/home (on disk) to /home2 (on CD)
/home2=/usr/home
Subdirecories and files may be excluded by writing them
behind option -not. Example: exclude three directories
-not /home/guest /home/kindergarden /usr/home/guest
Complete example:
scdbackup /home /home2=/usr/home -not /home/guest
Backup /home and /usr/home . But use /home2 on CD as
name for /usr/home and omit subdir /home/guest
sdvdbackup DVD version of above.
Directory access permissions may be restored by the compressed shell script
/added_by_scdbackup/DIR_PERMS.sh.gz
on the last volume of such a backup. (See also examples.html#restore_incr )
This may be necessary because directory access permissions on a mounted CD
may not be the same as they were during the backup.
afio Archives
afio archives can be read with the CD-ROM or DVD drive while the media is
inserted but not mounted.
Check Archive:
afio -tv /dev/cdrom
Re-install (*beware*, only if you need):
cd /
afio -ivZ /dev/cdrom
Example : Extract X11 configuration files to a harmless place
No need to be superuser for that !
mkdir $HOME/system_reference_copy
cd $HOME/system_reference_copy
afio -ivZ -y 'etc/X11/XF86Config*' /dev/cdrom
Now you may compare them with those of your current system
diff etc/X11/XF86Config /etc/X11/XF86Config | less
Read for details : man afio .
By help of script star_as_afio_wrapper one may alternatively create archives
in the format of program star, provided that program is installed. For an
example see appendix WORKERS.
scdbackup_afio like scdbackup but with afio archives rather than
ISO 9660 file systems. If there is a buffer file or
if compression was set during configuration then
the archives contain compressed files.
scdbackup_afio /home -not /home/guest
sdvdbackup_afio DVD version of above.
scdbackup_sys a command that backups the root directory as afio
archives. Some well known non-system directories and
some trash directories are excluded. (My apologies
to all those who work hard on programs that fill in
the "trash" :) Arguments can be given to include or
exclude other directories.
The exclusion list is set by CONFIGURE . Example:
/cdrom /cdrw /floppy /home /mnt /opt /proc /tmp
Example (to be executed as superuser):
include /opt (which usually is excluded)
but omit /home2 and /usr/trash
scdbackup_sys /opt -not /home2 /usr/trash
sdvdbackup_sys DVD version of above.
If the size of the resulting afio archives exceeds 650 MB then they are
split into pieces of 650 MB and stored on several CDs which form a multi
volume archive. Such archives have to be read from their first CD up to
the one which contains the desired data. If a CD is missing then all
following ones may be unreadable. DVD split size is 4480 MB.
The main advantage of compressed multi volume archives is their most effective
use of CD space. Single volume compressed archives usually need quite a lot of
spare space.
The ability to create multi volume archives allows you to risc larger single
volume ones. Worst case is an additional CD with a few MB on it.
(You may adjust the input size by -max_size number )
It is convenient to have a copy of the program bin/raedchen (e.g. on floppy)
to be able to concatenate the parts of such an archive.
Check:
raedchen -quiet -pipe /dev/cdrom 650m "cat" \
-post_pipe_command "eject /dev/cdrom" | afio -tv -
Restore:
raedchen -quiet -pipe /dev/cdrom 650m "cat" \
-post_pipe_command "eject /dev/cdrom" | afio -ivZ -
This is necessary because the CD returns some more bytes than have been
written. The false extra bytes spoil at least the file which is stored
at the end of the first and the beginning of the second CD.
With DVD use :
raedchen -quiet -pipe /dev/dvd 4480m "cat" \
-post_pipe_command "eject /dev/dvd" | afio ...
If you are completely without scdbackup, then you will have to use a shell
loop for concatenation. Like this ( 332800 = 650 MB / 2 kB ) :
( while test -n 1
do
dd if=/dev/cdrom bs=2048 count=332800 || exit 0
eject /dev/cdrom
echo 'Next media please' >&2
read dummy
done ) | afio ...
There is an appropriate option to make the verify commands handle such
overflown media. (see below)
The actual CD overflow size for afio is set by configuration file
scdbackup_media_cap_value or shell variable SCDBACKUP_MEDIA_CAP .
For DVD it is set in sdvdbackup_media_cap_value resp. SDVDBACKUP_MEDIA_CAP.
If you change the defaults 650m / 4480m take care to record the new size
on paper or on the raedchen floppy.
Restartability
All the above backup commands can be executed with first argument -resume and
an optional number as second argument. They then restart an existing backup
script at the backup piece with the given number. A dash (-) preceding the
number will cause only this particular volume to be created and the script to
end afterwards.
If the number is missing (or the second argument is "AUTO") then the script
resumes at the piece depicted by the status file ({skriptname}.pieceno).
scdbackup , scdbackup_sys and scdbackup_afio additionaly recognize these
options if one of them is given as first argument :
-prepare_only prepares for -resume but does not create CDs
-last_volume_count prints number of volumes of most recent backup
See script outer_loop for an example how to use them.
Verifying CDs and DVDs
First of all : the verification command described here does *not* ensure that
the backup contains what you *want* to backup. Wether you and scdbackup did
it right has to be proven by looking at the meaning of the content i.e.
whether the byte size is plausible and whether expected filenames are on
CD (locatable by ASKME).
This verification nevertheless ensures that the data stream has been transfered
to the CD successfully and can be read without alteration. (There is only
a very very remote chance of 1 to 3e38 that an alteration will be unnoticed.)
Therefore it checks the CD recorder, the CD media and the CD reading device.
In order to be verificable, a CD must have been created by version 0.8 -or
more modern- of scdbackup with standard content of file
scdbackup_make_checksum_value .
With these settings the commands scdbackup, scdbackup_home, scdbackup_sys and
scdbackup_afio record the length and a 128 Bit MD5 checksum of every backup
volume that is written. This record is stored in a recordfile on your disk
and may be used later to checkread the CD.
Beginning with version 0.8.3 the record is also stored as a checksum tag
at the end of the volume itself.
scdbackup_verify Reads raw data from a CD and checks whether these are
the same as the ones handed over to cdrecord. It reads
CD data from /dev/cdrom and compares them with the most
recent volume record.
scdbackup_verify -auto
tries to gather records from any source and checks whether
one of them matches or obviously indicates a mismatch.
Other data sources (CD devices) or older records may be
used by options to scdbackup_verify. E.g.:
scdbackup_verify /dev/cdrom -auto
For backups made with scdbackup 0.8 to 0.8.2 you need to
keep the record list. Backup it.
See output of scdbackup_verify -help resp. file
scdbackup_verify_help for a description of options,
the recordfile's name and more examples.
The command reads the whole CD and reports its result by
a text on stderr as well as by its exit value:
* "OK ..." and an exit value of 0 if the verification
process was successful.
* Various complaints in CAPITAL LETTERS and exit value 1
if verification fails (i.e. damaged or confused CD).
sdvdbackup_verify DVD version of above. Uses a different recordfile !
The record list usually is stored in file logs/backup_log resp.
logs/backup_dvd_log below scdbackup's installation directory. With normal
installation settings you can only read that list if you are the same user as
the one who wrote the backup.
The list's address may be permanently changed in files
scdbackup_checksum_list_value resp. sdvdbackup_checksum_list_value
and temporarily overriden by exporting shell variable
SCDBACKUP_CHECKSUM_LIST resp. SDVDBACKUP_CHECKSUM_LIST .
Checksum computation and checksum verification are based on cd_backup_planer
options -filter_md5 , -compare_checksum and -search_md5 . The format of
checksum records and tags is described in appendix VERIFY below.
For multi volume afio archives (see above) you will have to give
scdbackup_verify the media capacity as it was set when the backup was
written. Like :
scdbackup_verify /dev/cdrom -auto - - 650m
sdvdbackup_verify /dev/dvd -auto - - 4480m
If verification fails directly after recording then you may redo the affected
volume by a -resume run with the appropriate volume number. Like
sdvdbackup -resume 3
One should verify stored backups in regular intervals to get aware of damage
or deterioration.
scdbackup_verify -auto
If verification fails with such a check then you should consider to replace
all volumes of the whole backup by a new backup.
If this is not possible then you may make several tries to copy the media
image to a reliable disk and check it there in the hope that the copy attempt
delivered the correct image by incident (this does happen with small damages).
dd if=/dev/cdrom of=/tmp/hopefully_sound_image_file bs=2048
scdbackup_verify /tmp/hopefully_sound_image_file -auto
If successful, copy the image to new media:
cat /tmp/hopefully_sound_image_file | scdbackup -pipe_to_media
With DVD one should eventuellay truncate the media image to the known size plus
three extra MB for eventual checksum tags or blocklists. Like:
dd if=/dev/cdrom of=/tmp/hopefully_sound_image_file bs=1M count=1718
after
sdvdbackup_verify /dev/cdrom -auto_end
reported a match or mismatch after "1715m bytes".
If the payload size is not known from the previously failed attempts to verify
the media, then one should curb the output:
dd if=/tmp/hopefully_sound_image_file bs=1M count=1718 | \
sdvdbackup -pipe_to_media
Be aware that some old filesystems may be unable to take a file of 4200 MB.
If you expect your backups to be not reproducible during the intended storage
period (i.e. if they are an outsourced data archive rather than a backup for
a life disk), then you should consider to record them with block checksum
lists and as several identical copies. See appendix REDUNDANCY.
Blanking CD-RW or DVD-RW , Erasing data from media
If a CD-RW has been used and shall be re-used, it is necessary to blank it.
The same is true with DVD-RW under certain circumstances. If blanking comes
into effect it damages the data on the media, of course.
If automatic blanking is configured as "1" (or "y") then with each backup
the media get treated according to their automatically recognized type and
state. Usually classification and recognition of media types is reliable.
If nevertheless the type cannot be recognized or if it is not suitable for
the backup command, then the backup aborts. It may be resumed after the media
is brought to writeable state manually and automatic blanking is temporarily
disabled:
export SCDBACKUP_BLANKEN=0
scdbackup -resume
Not only in case of emergency but also with recognizable media it may be
worthwile to do blanking separately from backup writing.
Depending on the blank options, on type of the burner, and on the writer
software blanking lasts some time. So it is convenient to have some writeable
media ready.
Also one might sometimes want to apply unusual blanking options.
With DVD+RW there is no blanking required for making media writeable again.
Nevertheless, the "shred" option can be very helpful to ensure your privacy
when reusing media.
DVD-RW media for use with programs growisofs or cdrskin should be blanked
"force" once to bring them into a mode with the misleading name
"Restricted Overwrite".
After that, growisofs and cdrskin will behave with them like with DVD+RW.
DVD-RW for use with cdrecord need to be blanked before re-use just like CD-RW.
The same is true for growisofs DVD-RW media which were written unformatted
from unused state or have been de-formatted to mode "Sequential".
"shred" is a blanking option which does not necessarily make media re-usable.
It rather causes the media to be filled with random data. After that, the
previously stored data are unreadable by common means.
scdbackup_blank asks you to insert the CD-RW and press 'Enter'.
If an argument is given, this is used as blanking option.
Usual options are "fast" or "all". See output of option
blank=help
of the burn program (cdrecord, wodim, or cdrskin).
Alternatively, there is option "shred" (see above).
Without argument blanking is set to the option in file
scdbackup_blanken_value. "1" or "y" mean option "fast".
If no other clue is found option "fast" will be used.
Note: If a blanking directive blank=... is part of file
scdbackup_cdrecord_value then no other blanking option
will work.
sdvdbackup_blank DVD version of above.
Its behavior depends on the media type and on the writer
program (growisofs or cdrskin vs. cdrecord or wodim).
For DVD+RW only option "shred" does anything senseful.
For cdrecord, wodim and DVD-RW : see scdbackup_blank
For growisofs, cdrskin and DVD-RW : Options "all", "full"
and "sequential" lead to mode "Sequential". All other
options lead to convenient mode "Restricted Overwrite".
Both commands try to find out the type of the media in the drive. This depends
much on output of programs dvd+rw-mediainfo, cdrecord, wodim or cdrskin which
is not explicitely mentioned in their documentation.
If this classification fails to yield a usable media type, the user is asked
to confirm one of the types "CD-RW", "DVD-RW", "DVD+RW". The classification
attempt as well as the eventual user inquiry can be avoided by the third
argument to the program. Like : sdvdbackup_blank all - DVD-RW
CAUTION : Best is not to use these commands with one-time writable media !
E.g. if a DVD-R is not recognized properly or declared by the user to be a
DVD+RW, then option "shred" will use it up.
After a small backup on DVD it is easy to read old remains from larger
backups until they really get overwritten. Therefore before re-use of
confidential media apply sdvdbackup_blank shred . Do it multiple times
if privacy is a big issue.
The "shred" option tells what will happen next and require another Enter
keypress as confirmation. Then it eventually applies appropriate blanking.
It writes the amount of random data given by the second argument to the
program. E.g.:
sdvdbackup_blank shred 4.7e9
If that second argument is missing then the configuration settings in file
scdbackup_media_cap_value resp. file sdvdbackup_media_cap_value apply.
Usually the preset size of 4480m bytes should be sufficient to hit all
databytes on a DVD. Nevertheless you may derive a size from the maximum size
given by growisofs during write. Subtract 200 (e.g.: shred 4700372792 ).
Remains of backups made with version 0.8.3 or newer will get reported
by command scdbackup_verify -scan_print . Like :
- 1_1 A40907.115444 4699824167 7ec3be0efef3f27dd7ec90216d2ee3b4
which tells that backup data may be stored up to byte 4699824167.
Blanking option "shred" produces a checksum record with name "shredded"
and therefore together with the appropriate s*backup_verify command is
suitable as a quality check for the media.
A large organisation, which presumes to have many enemies, prescribes
additional runs with a character and its complement. It also demands
verification of the random write result. The said organisation does not allow
any erasing of "TOP SECRET" media but demands their complete physical
destruction.
Although i am in doubt whether the complement achieves the desired effect with
CD/DVD media, there are two blanking options "character" and "complement"
which perform the prescribed runs. A compliant procedure would be :
export SCDBACKUP_BLANK_CHARACTER="U"
sdvdbackup_blank character 4.7e9
sdvdbackup_blank complement 4.7e9
sdvdbackup_blank shred 4.7e9
sdvdbackup_verify
The character written is a capital "U" (= 01010101). By variable
SCDBACKUP_BLANK_CHARACTER a different normal letter or digit may be chosen.
Better do not use any special characters.
Incremental Backups
With larger amounts of data it becomes annoying or even impossible to
backup the old data at each new run. To avoid this (within certain
limitations) one may use an incremental backup scheme.
Such a backup consists of a base backup which might be quite old but
contains a full snapshot of all files without regarding their age.
On top of the base backup there are update levels which provide more
and more short-term updates. This saves a lot of time and CDs but
provides similar security as a set of full backups. Invest some of
the savings into increased backup frequency and also keep outdated
backup CDs a bit longer. This is to compensate the media redundancy
which is reduced with incremental backups.
It has to be emphasized that correct restoring of a multi-level incremental
backup is much more complicated than with a full backup.
Consider your potential future situation, when you might really need the full
content of your backup. Weigh whether avoiding the repeated workload of a
full backup really justifies the additional workload in the future crisis
situation. Begin with incremental backups if you have to leave out full
backups because you cannot afford them.
Initially a backup configuration needs to be created. It records
all parameters of the backup, keeps track of the times and states
of the backup area when backups have been made.
Also the creation process starts the initial base backup which is
called -level 0 .
Choose a configuration name that can be used to create a new directory.
You will need some disk space there for the state records (usually
below 1% of the backup area size).
Let us assume you choose $HOME/my_backup :
$ scdbackup ...the usual arguments for your backup ... \
-conf_dir $HOME/my_backup -level -create_configuration
After that creation, only the name of the configuration and the
desired update level have to be given with the scdbackup command.
Next day one could execute this command to get a backup of the
files that are new or changed since the -level 0 backup :
$ scdbackup -conf_dir $HOME/my_backup -level 1
Options -conf_dir and -level are explained in output of
cd_backup_planer -help . There also is an example of a scheme.
Several independant configurations may be defined. Distinct configurations
do not interfere even if they share data sources.
Not only scdbackup but also scdbackup_home and scdbackup_afio may be
used for defining and running incremental backups.
The state records in the configuration directory consume some disk space
although they get compressed by gzip. You should watch the size of file
$HOME/my_backup/level_0/content_file_list.gz
during the -level 0 run and abort it if you cannot afford the space needed.
While watching the size do not get worried by the output delay of the gzip
pipe. Any substantial size change will get visible in time.
If you cannot afford the disk space, you may disable state recording by
using option -disable_safe_leveling together with -level -create_configuration.
But be aware that file movements and deletions cannot be detected that way
and will therefore not be included in the higher backup levels. Such a
backup is still a good source for recovering from limited data accidents
but it may be unable to accurately reconstruct the most recent state of
your backup area.
The default settings of scdbackup configure a method which depends on the
traditional behavior of inode numbers (POSIX "file serial numbers"). If your
filesystem turns out to be non-compliant with tradition then you will get
suspiciously large backups at higher levels. Please inform me about the type
of the affected filesystem and refer to appendix INODE for usage of the old
checksum based method.
If you ever need to re-install an incremental backup with multiple levels
you will begin with copying level 0 first (e.g. by use of cp -dR ).
Then you will remove the files which had vanished when level 1 was made.
Then you will copy level 1.
This is repeated with all higher level backups.
For the removal job, there is a compressed shell script
/added_by_scdbackup/VANISHED.sh.gz
on the last volume of each backup level. Don't run it for fun !
(It is interesting to have a look at its content, nevertheless.)
See examples.html for an example restore procedure.
More detail information about state recording can be found in the output
of cd_backup_planer -help . Look for "re-install an incremental backup"
and also refer to the program options mentioned there.
The decision whether to put a file object into an incremental backup depends on
the following properties : mtime (display by ls -l), ctime (by ls -lc),
eventually file content checksum (if enabled and some ancestor directories have
young time stamps), eventually the inode and device numbers (if enabled),
an economy consideration which eventually includes whole directories if
there are many small files to be included.
Backup Information Scripts
Each backup CD contains a shell script named ASKME which can tell on which one
a given fileaddress may have ended up in the backup. Since it only knows the
fileaddresses given to the formatter program, this script cannot always tell
for sure whether a file was actually present at the time of the backup.
But it can tell the CD where one has to look.
ASKME only needs an bash or ksh shell but no installation of scdbackup.
See text askme_help for usage information.
In case of incremental backups the information scripts of all older backup
levels are also present on each CD. Because they can become very large and
hardly usable, they are by default truncated to stubs which can just identify
backup date, level and volume. To be usable for searching, the backup
configuration has to be created with option
-info_script -full
or all level runs have to be done using this option.
For incremental backups with default stubs, refer to scdbackup_askme in the
next paragraph.
With ISO 9660 formatted backups, the script is located in the root directory
of the backup volume.
With archive formatted backups (afio, star) it is the first item of a backup
volume. Its name and address vary with the installation settings at backup
time. One may obtain the address within the afio archive by
afio -t /dev/cdrom 2>/dev/null | head -1
which should reply something like
u/thomas/scdbackup/scdbackup-0.9.2/tmp/ASKME_1000.1.z
(omit 2>/dev/null in insatisfying reply.)
The script should run on nearly any shell that is around but it has the
disadvantage to be relatively slow. With multilevel backups of large disks
one may have to wait minutes for an answer.
Backup Information+Restore Tool
Command scdbackup_askme provides a binary program which can read the ASKME
scripts and perform the searches much faster. Also it can search for text
pieces (like fgrep) or for regular expressions (like grep).
The main advantage of scdbackup_askme, nevertheless, is its ability to read
content lists and vanish lists from incremental configuration directories or
from the last volume of each backup level. Beginning with scdbackup version
0.8.7, non-incremental ISO 9660 backups too have a content list on the lasti
volume.
These lists hold more detailed information about the backup content than the
ASKME scripts. They are able to surely tell whether a particular file was
backuped (whereas ASKME can often only tell about a directory above the file's
address). The content lists are able to tell the address of a file on the
backup media (whereas ASKME can only tell the original address on disk).
Based on content lists it is possible to have scdbackup_askme perform restore
operations from ISO 9660 backup volumes.
Both commands scdbackup_askme and sdvdbackup_askme may be pre-configured by
startup files. See below "Startup Files for Information+Restore Tool".
Examples (see text doc/scdbackup_askme_help for options and more information)
Using the level ASKME scripts from backup media (ISO format):
scdbackup_askme /cdrom/ASKME -levelall -search_grep '/trees/.*oak'
scdbackup_askme /cdrom/ASKME -levelall -search_sh '/home/*/trees/*oak*'
Using an incremental backup configuration on disk
scdbackup_askme $HOME/my_backup -levelall -dialog
After the backup state is read, the user will get prompted for options or
search texts. Caused by option -search_sh those search texts will be
compared as shell parser patterns against the source addresses. In this mode
there is a current working directory which is addressed by search texts which
do not begin with '/'. The special file names '.' and '..' are recognized.
-pwd
-cd:/home/*/trees
*oak*
*/*oak*
../*oak*
-search_shpat
*trees/*oak*
-search_shname
*oak*
-page:20
*oak*
-search_shpat makes '/' match *,? or []. This allows to freely search for file
name snippets.
-search_shname looks into any directory and compares the single file names with
the search text. Like this shell command: find / -name '*oak*'
-page:20 causes output to pause after each 20 lines. The user gets asked for
pressing Enter. Input @ suppresses this paging temporarily. input @@ suppresses
output, Input @@@ aborts. Any other input is taken as option or search text and
aborts the current search.
Independend of the search mode there are options -ls and -ls_d which allow
a shell pattern search with abbreviated output. They are similar to shell
commands ls and ls -d :
-ls:..
-cd:..
-ls_d:*tree*
All searches may be performed on the target addresses, too.
-search_target
For the following examples switch back to shell parser search mode
-search_sh
To see the current settings
-status
To see all possible settings
-status:long
To see the help text
-help
To restore single files or whole directory trees from ISO 9660 backups,
restoring first has to enabled by setting a -restore_mode . This one protects
existing files on the disk and allows to continue trying after restore problems
-restore_mode:overwrite_off:on_error_ignore
The program further needs to know from where to read the data and it is
convenient to have it operate the according drive via programs eject and mount
-restore_from:/cdrom
-media_mode:load_eject:mount_umount
The original data are protected. So, in order to achieve a full restore result,
the restore gets directed to a copy location
-restore_skip:/home/thomas
-restore_to:/home/test/thomas
Now directory tree /home/thomas/gifs gets restored to that copy location
-restore:/home/thomas/gifs
The user will get prompted for the needed media. It is possible to skip some
if they are missing, but of course that will result in an incomplete restore.
The result will be a directory tree /home/test/thomas/gifs .
Above operations may as well be performed using incremental information from
backup media (ISO format). They are stored on the last volume of each level.
scdbackup_askme /cdrom/added_by_scdbackup -levelall \
-media_mode:load_eject:mount_unount -dialog
The user will get prompted for those last volumes. User input "@@@" confirmed
by user input "y" indicates that no more levels are available.
Then dialog starts as with the previous example.
Restore runs which have been aborted for some reason may be resumed at the
last restore position by:
scdbackup_askme -resume
One may also depict an explicit start level
scdbackup_askme -resume:1
and additionally an explicit start volume
scdbackup_askme -resume:1:4
Option -resume does also work in dialog but only for restore operations
which have been performed within the same dialog session, or for those which
were skillfully prepared by options -resume_last_* and -resume_opt_text.
Aborted -resume runs may also be resumed by -resume.
If the abort left a file object on disk which is neither in its pre-restore
state nor restored correctly, then option -resume refuses work and demands
the affected fileobject to be handled first.
In dialog mode one may enter "go on" after the file is checked, and eventually
repaired or deleted. In non-dialog mode the program aborts until the offending
command -resume_last_dirty in file
$(scdbackup -where tmp)/askme_resume_state_"$UID"_dirty
is removed or set to an empty file address.
File objects recorded as "dirty" are questionable in any way. The directories
in the path of such a file may been left with too generous permissions for the
owner or too sparse permissions for anybody. One should check and correct.
(Owners usually have rwx on their directories so that is rarely a problem.)
scdbackup_askme has the appeal of a command line shell but it should be not
too hard to create a graphical frontend for it. The file doc/saskme_frontend.c
contains example code covering the task of forking scdbackup_askme and to
perform dialog with this forked process. (No GUI code included.)
Access Permissions
Initially only the installing user and the superuser have permission to
run backups by help of a scdbackup installation. In hostile or even just
weakly controlled environments one should unconditionally stay with these
settings and eventually install independend copies of scdbackup for
additional users.
In the case of a strictly personal computer it may seem tolerable to
have the superuser using the scdbackup installation of the desktop user.
But one should be aware, that entertainment components of web browsers and
other programs impose a risk that the desktop user might get overtaken
by intruders.
Nevertheless it may be desirable to have a single scdbackup installation
which can be shared by several users. In this case the superuser should install
but *absolutely not use* that installation. In case of permission scheme
"group_only", the superuser has to set the desired group membership of the
installation (e.g. by chgrp -R ).
The script ./SET_PERMISSIONS allows the installing user to loosen the
access restrictions. The script expects as only argument one of the following
permission scheme names :
owner_only this is the default described above
group_only grants group members the same rights as the installing user
anybody allows any user to run backups but tries to keep them
from spoiling the configuration. Not really safe.
custom prepared name for a custom permission scheme:
edit SET_PERMISSIONS and search "Change this according to"
unprotected as the name says : totally unprotected and totally unsafe
Those settings are permanent and get restored whenever ./CONFIGURE_CD or
./CONFIGURE_DVD are run.
Restore the most secure settings by entering the scdbackup installation
directory and executing :
./SET_PERMISSIONS owner_only
-----------------------------------------------------------------------------
Inner details - Appendice
See table of content at the beginning ot this text.
-----------------------------------------------------------------------------
Appendix DIRECTORIES :
The components of the scdbackup system are stored in several sub directories of
the installation directory. Especially those files which need to be written
during daily operation of scdbackup or during configuration are separated
from those files which usually need not to be changed.
After unpacking the installation directory scdbackup-0.9.2 contains these
data files :
TIMESTAMP the timestamp of the tar archive
cd_get_variables reads the configuration for CD
dvd_get_variables reads the configuration for DVD
scdbackup_switcher dispatcher script for user commands and language choice
A few symbolic links refer to files in the the ./doc directory :
COPYING LIESMICH README
These directories are created below scdbackup-0.9.2 :
aux a few parts which do not fit into any other category
bin binary programs, resulting from C sources
cmd user accessible commands
conf configuration files and custom intermediate scripts
doc documentation
inst configuration scripts
logs list files with checksum records
scripts worker scripts of the scdbackup system (for bash or ksh)
src C sources which lead to the programs in ./bin
tmp files created during daily usage
For using scdbackup at least the directories ./logs and ./tmp have to provide
write permissions. There are configuration files which allow to redirect them
to other addresses outside of scdbackup's installation directory.
Therefore it is generally advised not to address subdirectories of scdbackup
directly but to determine their actual address by $(scdbackup -where xyz) .
See also the examples throughout this text.
Older applications which directly use inner parts of scdbackup will not find
those programs and scripts any more. Applications which restricted themselves
to commands scdbackup* and sdvdbackup* and did not access configuration
files directly are not affected.
To ease the transition of affected applications there is a script
./COMPATIBILITY_LINKS in directory ./inst which creates a lot of symbolic
links within directory scdbackup-0.9.2 . This makes available all programs
which in older documentation have been advertised for use.
Eventually missing links may be created by hand like this :
cd $(scdbackup -where)
ln -s $(scdbackup -where scripts)/cd_afio_command cd_afio_command
Nevertheless i advise you to migrate eventual applications to the new directory
structure.
For mere safety reasons, a normal user's scdbackup installation should
*not* be FHS compliant. FHS mainly addresses system-global software.
To get a scdbackup installation of the superuser conformant to FHS, you would
have to scatter it over your system so it matches the following settings
Configuration file : Content :
$HOME/.scdbackup_dir /opt/scdbackup-0.9.2
/opt/scdbackup-0.9.2/scdbackup_conf_dir /etc/opt/scdbackup-0.9.2
/etc/opt/scdbackup-0.9.2/scdbackup_tmp_dir_value /var/opt/scdbackup-0.9.2/tmp
/etc/opt/scdbackup-0.9.2/scdbackup_logs_dir_value /var/opt/scdbackup-0.9.2/logs
See below appendix CONFIGURATION for the meaning of these files.
In order to obey the FHS imperative "Programs to be invoked by users must be
located in the directory /opt/<package>/bin" one may also move the content of
the ./cmd directory to ./bin .
-----------------------------------------------------------------------------
Appendix CONFIGURATION :
These three configuration files are used by both CD and DVD subsystem :
$HOME/.scdbackup_dir the address of the scdbackup-directory
(such a file is created by ./ADD_USER)
May be overridden by shell variable SCDBACKUP_DIR
$HOME/.scdbackup_lang the language to be used in dialog. Currently only
en (english) and de (german) are supported.
Any other content will default to en .
May be overridden by shell variable SCDBACKUP_LANG
$HOME/.scdbackup_userid only needed if shell variable EUID is not set
automatically by the shell. This file may contain
a unique decimal number to distinguish this user
from any others of the system. The configuration
scripts will prompt for a number and create such
a file if they detect $EUID empty.
Override this file or EUID by SCDBACKUP_USERID .
(Note that this does not give you any other user's
permissions or privileges.)
The following configuration files get created by ./CONFIGURE_CD in directory
./conf
(DVD configuration files are listed further below)
This address may be overridden by the content of file
$(scdbackup -where)/scdbackup_conf_dir
resp. variable SCDBACKUP_CONF_DIR. Its actual address may be queried by
scdbackup -where conf
Each of the files *_value may be overridden by a shell variable. This
variable has the same name as the filename's beginning but is written in
capital letters. But note that an empty variable does not override anything.
Example:
Override files scdbackup_scsi_adr_value , scdbackup_speed_value and
scdbackup_buffer_value to use the cruelly fast IDE writer which chokes
my system if i use a file buffer at speed 12 :
export SCDBACKUP_SCSI_ADR="1,0,0"
export SCDBACKUP_SPEED="12"
export SCDBACKUP_BUFFER="-"
scdbackup_scsi_adr_value the address of the recorder
(e.g. 3,0 , ATA:1,0,0 , /dev/hdc)
scdbackup_speed_value the recording speed (e.g. 12)
scdbackup_blanken_value whether to blank the CD before writing:
"0"=no , "1"=yes (mode will be "fast")
or the blank= option as for cdrecord
"all" , "fast"
or to perform blanking during the write run rather
than in a separate blank run :
"inline_all" , "inline_fast"
The text may contain all three arguments for
scdbackup_blank. Like: all - CD-RW
scdbackup_use_prog_value for CONFIGURE_CD to remember the answer to the
question which creates the next four files:
scdbackup_mkisofs_value name and options of the ISO 9660 formatter program
(e.g. mkisofs -J -l -L -R)
scdbackup_path_list_value eventually contains cd_backup_planer options for
the use of the -path-list feature of ISO formatters.
Among others there is the address of the temporary
list file which will buffer the path-list.
Note that the text "mkisofs_path_list" gets expanded
to "mkisofs_path_list"_$EUID and that "=EUID=" gets
converted to $EUID. (This does only apply to reading
the file but not to evaluating SCDBACKUP_PATH_LIST)
scdbackup_iso_filetypes_value eventually contains options for cd_backup_planer
which may allow other file types than "data" and
"dir" to be handed over to the ISO formatter. Older
mkisofs aborts on symbolic links, though.
scdbackup_cdrecord_value name and fixed options of the CD recording program
(e.g. cdrecord -v )
scdbackup_cdrecord_opt_value additional options of the CD recording program
to be used with runs that write data but not with
the separate blanking run. (e.g. -fs=8m -eject)
scdbackup_split_dir_value for CONFIGURE_CD to remember the answer to the
question which creates the following two files :
scdbackup_large_file_iso_value options for cd_backup_planer concerning
handling of large files when planning an ISO
format backup.
Note that the text "=EUID=" gets converted to
the userid number. (This does only apply to reading
the file but not to $SCDBACKUP_LARGE_FILE_ISO)
scdbackup_large_file_afio_value
As scdbackup_large_file_iso_value with afio format.
Does not apply to scdbackup_sys.
scdbackup_afio_value name of a program call-compatible to afio plus
eventual fixed options for writing. The program must
recognize option "-Z" for compression and "-" as
target name for stdout. It has to expect the
addresses of input files at stdin. (see: man afio)
Default value is : afio -o
scdbackup_buffer_input_value for CONFIGURE_CD to remember the answer to the
question which creates the following two files:
scdbackup_buffer_value name of the buffer file or - (e.g. /buf/isofs)
scdbackup_afio_compress_value whether to enable compression with afio :
0 = no, 1 = yes, - = compress if buffer file is used
(default is - )
scdbackup_z_cap_value estimated capacity for compressed afio. If empty
or 0 then cd_dir_afiosize is used.
scdbackup_sys_excl_value contains a list of directories to be excluded from
the system backup.
scdbackup_t_lock_value whether to use the locking protocol:
1 = yes, store lockfiles in scbackup directory
/path = yes, store lockfiles in path (e.g. /tmp )
anything else means no locking
scdbackup_bin_sbin_root_value parent directory for installation of commands
scdbackup_is_configured indicates by content "yes" that scdbackup was
configured sucessfully for CD.
Further configuration files are not adjustable by ./CONFIGURE_CD but may
be changed or overridden to customize scdbackup's behavior :
scdbackup_use_shell_value contains the path of a bash compatible interpreter.
scdbackup_media_cap_value maximum capacity of media used with afio archives.
If an afio archive exceeds this size then it is
cut on the first CD and continued at a new one.
(set initially to "650m" by CONFIGURE_CD)
This size is also used for blank option "shred".
scdbackup_max_size_value default volume size if option -max_size is omitted.
(640m if file is missing and variable not set)
scdbackup_blank_speed_value the speed for blanking CD-RWs (e.g. 4) in case
it differs from recording speed. (This file is not
created by ./CONFIGURE_CD and usually not needed.
Before my CRW8424S deceased completely it was able
to blank with speed 4 but to write only with 2.)
scdbackup_make_checksum_value contains options for generating MD5 checksums.
This file is initially created by CONFIGURE_CD but
its content may be set to a single "-" to prevent
checksum computation.
scdbackup_checksum_list_value address of the file containing checksum records.
(if missing : .../scdbackup-0.9.2/logs/backup_log )
scdbackup_verify_adr_value default for data_source with scdbackup_verify
(if missing : /dev/cdrom )
scdbackup_keep_buffer_value whether to keep the eventual buffer file with ISO
image or afio archive to survive until the next
volume gets written. Content "1" means yes.
scdbackup_options_with_iso_value additional options for cd_backup_planer
which shall take effect with any backup that
produces ISO 9660 file systems.
scdbackup_options_with_afio_value additional options for cd_backup_planer
which shall take effect with any backup that
produces afio archives.
The following configuration files get created by ./CONFIGURE_DVD in directory
./conf
(CD configuration files are listed above)
This address may be overridden by the content of file
$(scdbackup -where)/scdbackup_conf_dir
Like with CD configuration each of the files *_value may be overridden by a
shell variable.
E.g.: Override files sdvdbackup_dev_adr_value and sdvdbackup_speed_value :
export SDVDBACKUP_SCSI_ADR="/dev/sr2"
export SDVDBACKUP_SPEED="1"
sdvdbackup_dev_adr_value the device address of the recorder (e.g. /dev/dvd)
Eventually used with growisofs.
sdvdbackup_scsi_adr_value the SCSI address of the recorder (e.g. 3,0)
Eventually used with cdrecord, wodim, cdrskin.
sdvdbackup_speed_value the recording speed (e.g. any, 1)
sdvdbackup_blanken_value whether to blank the DVD before writing:
"0"=no , "1"=yes (mode will be "fast")
or the blank= option for sdvdbackup_blank :
"all" , "fast"
The text may contain all three arguments for
scdbackup_blank. Like: all - DVD-RW
sdvdbackup_use_prog_value for CONFIGURE_DVD to remember the answer to the
question which creates the next two files:
sdvdbackup_mkisofs_value name and options of the ISO 9660 formatter program.
(e.g. mkisofs -J -l -L -R)
sdvdbackup_path_list_value eventually contains cd_backup_planer options for
the use of the -path-list feature of ISO formatters.
Among others there is the address of the temporary
list file which will buffer the path-list.
Note that the text "mkisofs_path_list" gets expanded
to "mkisofs_path_list"_$EUID and that "=EUID=" gets
converted to $EUID. (This does only apply to reading
the file but not to evaluating SCDBACKUP_PATH_LIST)
sdvdbackup_iso_filetypes_value eventually contains options for cd_backup_planer
which may allow other file types than "data" and
"dir" to be handed over to the ISO formatter. Older
mkisofs aborts on symbolic links, though.
Default : -result_filetypes data:dir:arg_redirect
sdvdbackup_cdrecord_value name and fixed options of the DVD recording program
(e.g.
/u/test/scdbackup-0.9.2/scripts/growisofs_wrapper )
sdvdbackup_cdrecord_opt_value additional options of the DVD recording program
to be used with runs that write data.
(e.g. -final_eject bufsize=64m)
sdvdbackup_stream_rec_value additional option to control slow checkreading
with DVD-RAM, BD-RE and formatted BD-R. Will only
get into effect if the DVD recording program name
begins by 'xorriso -as cdrecord' or 'cdrskin '.
sdvdbackup_split_dir_value for CONFIGURE_DVD to remember the answer to the
question which creates the following two files:
sdvdbackup_large_file_iso_value options for cd_backup_planer concerning
handling of large files when planning an ISO
format backup.
Note that the text "=EUID=" gets converted to
the userid number. (This does only apply to reading
the file but not to $SDVDBACKUP_LARGE_FILE_ISO)
sdvdbackup_large_file_afio_value
As sdvdbackup_large_file_iso_value with afio format.
Does not apply to sdvdbackup_sys.
sdvbackup_afio_value name of a program call-compatible to afio and
eventual fixed options for writing.
Default value is : afio -o
(See above scdbackup_afio_value for more details)
sdvdbackup_buffer_input_value for CONFIGURE_DVD to remember the answer to the
question which creates the following two files :
sdvdbackup_buffer_value name of the buffer file or - (e.g. /buf/isofs_dvd)
sdvdbackup_afio_compress_value whether to enable compression with afio :
0 = no, 1 = yes, - = compress if buffer file is used
(default is - )
sdvdbackup_z_cap_value estimated capacity for compressed afio. If empty
or 0 then cd_dir_afiosize is used.
sdvdbackup_sys_excl_value contains a list of directories to be excluded from
the system backup.
sdvdbackup_t_lock_value whether to use the locking protocol :
1 = yes, store lockfiles in scbackup directory
/path = yes, store lockfiles in path (e.g. /tmp )
anything else means no locking
sdvdbackup_bin_sbin_root_value parent directory for installation of commands
sdvdbackup_is_configured indicates by content "yes" that scdbackup was
configured sucessfully for DVD.
Further configuration files are not adjustable by ./CONFIGURE_CD but may
be changed or overridden to customize scdbackup's behavior :
sdvdbackup_use_shell_value contains the path of a bash compatible interpreter.
sdvdbackup_media_cap_value maximum capacity of media used with afio archives.
If an afio archive exceeds this size then it is
cut on the first DVD and continued at a new one.
(set initially to "4480m" by CONFIGURE_DVD)
This size is also used for erasing (blanking) DVD.
sdvdbackup_max_size_value default volume size if option -max_size is omitted.
(4250m if file is missing and variable not set)
sdvdbackup_make_checksum_value contains options for generating MD5 checksums.
This file is initially created by CONFIGURE_DVD but
its content may be set to a single "-" or made
empty to prevent checksum computation.
sdvdbackup_checksum_list_value address of the file containing checksum records
(default: .../scdbackup-0.9.2/logs/backup_dvd_log )
sdvdbackup_verify_adr_value default for device address with sdvdbackup_verify
(if missing : sdvdbackup_dev_adr_value )
sdvdbackup_keep_buffer_value whether to keep the eventual buffer file with ISO
image or afio archive to survive until the next
volume gets written. Content "1" means yes.
sdvdbackup_options_with_iso_value additional options for cd_backup_planer
which shall take effect with any backup that
produces ISO 9660 file systems.
sdvdbackup_options_with_afio_value additional options for cd_backup_planer
which shall take effect with any backup that
produces afio archives.
This file may be created by hand before running ./CONFIGURE_DVD :
sdvdbackup_use_prodvd_value whether to configure for cdrecord (file content "1")
or for growisofs (any other content).
This file preserves the name of the permission scheme for ./SET_PERMISSIONS .
It applies to both, CD and DVD permissions :
scdbackup_set_permissions_value one of the names mentioned in paragraph
"Access Permissions" above.
The files to redirect the directories which need w-permission for daily usage
apply to both CD and DVD :
scdbackup_tmp_dir_value may contain the address of a directory which will
be used instead of the ./tmp directory within
scdbackup's installation directory.
scdbackup_logs_dir_value may contain the address of a directory which will
be used instead of the ./logs directory.
Redirection of ./conf, ./tmp, or ./logs is only recommended for situations
where scdbackup is installed on a read-only filesystem and the writeable
directories need to be outsourced to a rw-filesystem.
Other (expert) applications are: sharing logs between several scdbackup
version installations of the same user, setting up an FHS compliant
installation (FHS with scdbackup is safe only if no other than the superuser
is allowed to use it !).
One may view the effective configuration of a scdbackup environment by
scdbackup -show_configuration | less
sdvdbackup -show_configuration | less
The burn scripts scripts/cd_backup_command , scripts/cd_afio_command ,
scripts/cd_pipe_command allow the use of three intermediate scripts or
programs. If you provide executable code under one of the following file names
within the ./conf directory, it will run at the given occasion.
Each of the runs gets at least eight arguments :
$1 Address of buffer file ("-" or "" if on-the-fly)
$2 SCSI address used by cdrecord or device address used by growisofs
$3 Caller ("iso9660" for scdbackup, "afio" for scdbackup_afio)
$4 Volume number
$5 Total number of volumes
$6 scdbackup installation directory
$7 cdrecord program used (with options like -v)
(contains e.g. .../growisofs_wrapper if running for DVD)
$8 formatter program used (with several options)
These program runs are prepared :
conf/volume_prompt_script
before the user gets opportunity to press Enter.
conf/volume_preburn_script
after the eventual recorder lock is refreshed and
before the eventual blank run.
conf/volume_afterburn_script
after the run of the writer program ended and
before the eventual buffer file gets removed.
This script gets additional arguments:
$9 Address of the eventual checksum list
${10} Recordname of the record written to this list
${11} Timestamp of record
${12} Size
${13} Checksum (superstition brings bad luck. U.Eco)
Make sure your shell can evaluate ${10} or use shift
to access double digit parameters (after copying all
single digit parameters needed).
conf/volume_abortedburn_script
runs instead of volume_afterburn_script if the
writer program indicates failure and the whole
backup run is to be aborted.
There may also be scripts specialized for a particular media type. If both
- the general script and the specialized script - are present, then the
generalized script gets executed first and the specialized script gets
excecuted after that. Both scripts get the same arguments.
Specialized scripts for CD are :
conf/volume_prompt_script_cd
conf/volume_preburn_script_cd
conf/volume_afterburn_script_cd
conf/volume_abortedburn_script_cd
For DVD :
conf/volume_prompt_script_dvd
conf/volume_preburn_script_dvd
conf/volume_afterburn_script_dvd
conf/volume_abortedburn_script_dvd
Example :
Put your backup into a set of hard disk files rather than onto a set of media.
For that prepare a writer which directs the volume stream into a disk file :
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_data_to /home/archive_dir/next_volume"
Create an afterburn script which renames the disk file to a filename like
"A50922.132751_volume_1_7" :
echo '#!/bin/sh' > $(scdbackup -where conf)/volume_afterburn_script
echo 'mv /home/archive_dir/next_volume /home/archive_dir/"${11}"_volume_"$4"_"$5"' >> $(scdbackup -where conf)/volume_afterburn_script
chmod u+x $(scdbackup -where conf)/volume_afterburn_script
Now run your backup and have "yes" answer any media prompt
yes | scdbackup_afio -max_size 1g /opt /home
Do not forget to remove the afterburn script
rm $(scdbackup -where conf)/volume_afterburn_script
unset SCDBACKUP_CDRECORD
Startup Files for Information+Restore Tool
Commands scdbackup_askme and sdvdbackup_askme may execute the lines of a
startup file as options before any arguments or dialog input lines are
executed. The files must be readable but should not be executable.
Each line may be any option which is allowed for the commands. Because the
startup lines are read as first options they eventually will be in effect
when the info source is read in.
Empty lines and lines which begin with '#' are ignored.
There is no way to set the info source within the startup file. It has always
to be the first argument of the commands.
conf/scdbackup_askme_rc is read by command scdbackup_askme
conf/sdvdbackup_askme_rc is read by command sdvdbackup_askme
-----------------------------------------------------------------------------
Appendix WORKERS :
The commands use a program called bin/cd_backup_planer as central component.
cd_backup_planer splits a list of directories into pieces which then each
fit on CD-R or other media. It generates a script with backup commands.
The lists of files or directories are determined according to maximum
capacities in terms of total size and number of items. Oversized directories
are split automatically. Oversized files are skipped unless a
"Large file split directory" was defined during configuration.
Also the program is able to generate a shell script that can print several
informations about the backup. This script then is included in each piece
of the backup.
Read text
doc/cd_backup_planer_help
or execute
$(scdbackup -where bin)/cd_backup_planer -help | less
to get a description of the program and its options.
To avoid problems while running mkisofs the program performs a collision test
of target names and a readability test of source names. Collisions are
circumvented by automatical renaming. Problematic filenames are stated
explicitely. Unreadable files and directories may be excluded automatically.
The command scripts (e.g. /usr/bin/scdbackup) merely determine the location
of the scdbackup system and then start a master script there :
scdbackup_switcher finds out what language to use in dialog. It reports
the command name, the scdbackup version and the language
code on stderr. Then it starts one of the worker scripts.
If the only argument of the command is one of the
following, scdbackup_switcher prints a reply to stdout
and ends rather than starting the program :
-version "scdbackup" versioncode timestamp
-which the address of the effective worker script
-where the address of the effective scdbackup
installation directory. An optional second
argument may request the actual address of
a scdbackup subdirectory in case it is
redirected from its usual place. Like :
bin conf doc logs scripts tmp
(see also its usage throughout this text)
-show_configuration a list of variable names and
contents. See appendix CONFIGURATION.
-show_configuration_exported a list of export commands
for the shell. Content is quoted.
The scripts use two common shell code snippets to read the configuration
settings :
cd_get_variables reads the variables which where set with CONFIGURE_CD .
(to be executed inline via dot)
dvd_get_variables reads the variables which where set with CONFIGURE_DVD
(and uses them to override CONFIGURE_CD's settings
in cd_get_variables).
The following files are stored in directory ./scripts .
scdbackup_switcher gets ordered to start one of the following scripts :
backup_iso_en plans and performs backups in ISO 9660 format.
It is used by scdbackup, scdbackup_home,
sdvdbackup, sdvdbackup_home
backup_afio_en plans and performs backups in archive format.
It is used by scdbackup_afio, scdbackup_sys,
sdvdbackup_afio, sdvdbackup_sys
blank_en blanks a single media and/or overwrites it with
random bytes. (scdbackup_blank, sdvdbackup_blank)
verify_en verifies a single media against recorded checksums.
(scdbackup_verify, sdvdbackup_verify)
askme_en starts the binary information program.
(scdbackup_askme, sdvdbackup_askme)
clear_en releases all possibly open locks of a scdbackup
installation.
show_confguration performs option -show_configuration (see above).
Common code snippets which make the command scripts less redundant :
handle_existing_script performs options like -resume and other activities
which deal with the backup script that resulted from
the previous run.
handle_planer_failure is called if the run of cd_backup_planer fails.
install_backup_script installs the new resulting backup script and makes
it ready for execution.
Size determination is crucial for planning the distribution of backup data.
Besides the -internal size measure there may be used du -sk or scripts
which simulate the intended backup format :
cd_dir_isosize Deprecated, is able consume enormous virtual memory :
can be used instead of du -sk to estimate more
accurately the size which a directory will have in
ISO 9660 filesystem.
cd_dir_afiosize can be used to determine the size of a (possibly
compressed) afio archive made from a particular
directory. This is very slow since the archive is
actually generated, counted and discarded.
Several scripts scripts which refer to writing of media and pseudo-media :
cd_backup_command is able to write a single piece to a CD or DVD writer
as ISO 9660 file system.
cd_afio_command is able to write a single piece as afio archive
to a CD or DVD writer. If a buffer file is used then the
archive contains compressed files (afio option Z).
For CDs written on the fly this isn't suitable
since the output rate of a compressor can be very
low (i.e. the compression rate can be very high).
cd_pipe_command directs stdin to a single CD or DVD without any further
formatting. I.e. data already have to be formatted.
This is used with scdbackup -pipe_to_media .
If the first argument is -multi_volume then multi volume
rules as with scdbackup_afio apply. Use this mode for
piping large archives or dumps onto several media.
cd_blank_command blanks a CD-RW using the configuration of scdbackup
and eventually starts shredder_command.
sdvd_blank_command warns of shredder_command and starts it for DVD.
shredder_command overwrites a media with random data.
afio_sencrypt a script which produces an encrypted and eventually gzip
compressed afio archive. Encryption is done by sencrypt.
See http://scdbackup.webframe.org/README_sencrypt
See also appendix CRYPTO below.
One of three modes is selectable by the first argument.
Depending on the mode a second argument may be needed :
-each_file_by keyphrase_file
afio-traditional encoding of each single file
yields a rugged archive but is quite slow.
The key phrase has to be stored in a file and the
second argument has to be its address. In case of
compression the first 10 bytes of each file stay
unencrypted to avoid attacks via known cleartext.
Unpack (uncompressed) :
afio -i -P 'sencrypt' -Q '-d' \
-Q '-f' -Q "$keyphrase_file" -Z /dev/cdrom
Unpack (compressed) :
afio -i -P 'sencrypt' -Q '-d' -Q '-g' \
-Q '-f' -Q "$keyphrase_file" -Z /dev/cdrom
-whole_archive_by keyphrase_file
encryption of the resulting archive as a whole.
Again the key phrase has to be stored in a file.
With uncompressed archives the first 322 bytes
stay unencrypted. With compressed ones it is 162.
Unpack (uncompressed) :
sencrypt -d -s 322 -f "$keyphrase_file" \
</dev/cdrom | afio -i -
Unpack (compressed) :
sencrypt -d -g -s 162 -f "$keyphrase_file" \
</dev/cdrom | afio -i -
-whole_archive
encryption of the whole archive. The key phrase is
requested at /dev/tty. The first 322 resp. 162
bytes of the archive stay unencrypted.
Unpack (uncompressed):
sencrypt -d -s 322 </dev/cdrom | afio -i -
Unpack (compressed) :
sencrypt -d -g -s 162 </dev/cdrom | afio -i -
If the next argument is -Z then compression is enabled.
This -Z is eventually provided by scdbackup if
compression is configured (s*_afio_compress_value).
Any further arguments are forwarded to afio -o.
Variable SCDBACKUP_SENCRYPT_OPTIONS may be used to
override the preset sencrypt options. E.g. to avoid any
unencrypted head (and invite known-plaintext-attacks):
export SCDBACKUP_SENCRYPT_OPTIONS="-s 0"
star_as_afio_wrapper a script which replaces usage of program afio by usage of
program star. This will then produce star-archives rather
than afio archives. About encryption see also appendix
CRYPTO and http://scdbackup.webframe.org/README_sencrypt
Options recognized (-Z is given by scdbackup as needed):
-Z compress by help of star -z
-acl record ACL information by star artype=exustar -acl
artype=archive_type_of_star
choose a particular archive type and thus disable
exact -dump mode of star. E.g. artype=gnutar
See option "artype=" in man page of star.
This option may be overridden by -acl.
-encrypt
encrypt the whole archive by sencrypt and a
prompted key phrase.
Unpack (uncompressed):
sencrypt -d -s 752 </dev/cdrom | star -x
Unpack (compressed):
sencrypt -d -g -s 162 </dev/cdrom | star -x
-encrypt_by keyphrase_file
encrypt the whole archive by sencrypt and a key
phrase stored in a file.
Unpack (uncompressed):
sencrypt -d -s 752 -f "$keyphrase_file" \
</dev/cdrom | star -x
Unpack (compressed):
sencrypt -d -g -s 162 -f "$keyphrase_file" \
</dev/cdrom | star -x
-fifo
use star option -fifo despite it obscures errors
errctl=filename
finely adjustable error handling instructions for
star (see man page: option "errctl=")
-help explain options, predict unpack command, abort.
Any other arguments are ignored. No whitespace is allowed
in filename arguments (they have to be a single word).
Variable SCDBACKUP_SENCRYPT_OPTIONS applies (see above).
I got my copy of star from
http://cdrecord.berlios.de/old/private/star.html
compiled it and installed the binary in /usr/bin.
Online man page of star
http://cdrecord.berlios.de/old/private/man/star.html
Usage example (exustar allows files larger than 2 GB):
export SDVDBACKUP_AFIO_COMPRESS="1"
export SDVDBACKUP_LARGE_FILE_AFIO=" "
export SDVDBACKUP_AFIO="$(scdbackup -where scripts)/star_as_afio_wrapper -acl"
sdvdbackup_afio ...
Applies to sdvdbackup_sys, too.
List archive's table of content:
gunzip </dev/cdrom | star -tv
growisofs_wrapper a script which replaces the call of cdrecord in
cd_backup_command and cd_afio_command. It calls program
growisofs.
It recognizes the arguments -final_eject and
-no_final_eject which control disk tray behavior after
growisofs did its unload-reload cycle.
For other options see output of:
$(sdvdbackup -where scripts)/growisofs_wrapper -help
pipe_data_to a script which may be used rather than cdrecord or
growisofs_wrapper to direct the backup to an arbitrary
file address. It ignores any but its first and its last
argument. The first one is the target address
("-" = stdout), the last one is the the data source and
will be provided by scdbackup. If it is not "-" (= stdin)
and not the address of an existing usable file, the
script will print a notice on stderr and end with exit
value 0. Example (write to a 250 MB QIC tape device) :
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_data_to /dev/rmt0"
scdbackup_afio -max_size 240m ...
pipe_to_raedchen a script like pipe_data_to which uses raedchen as filter
in order to show a progress counter or to limit troughput
or to ease i/o buffer load. Mandatory arguments :
$1 = the target like with pipe_data_to.
$2 = the -step value for raedchen (0=quiet).
$3 = the -baudlimit value for raedchen (0=unlimited).
$4 = the flush option for raedchen (-flush or -).
Example (write to DVD-RAM sr0 without clogging disk I/O):
export SDVDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_raedchen /dev/sr0 1m 0 -flush"
pipe_to_cdrecord_sao a script which circumvents the need to have a buffer
file with cdrecord in -sao mode (e.g. with DVD). This is
done by announcing the media_cap size to cdrecord and
padding shorter volumes up to that maximum size.
The script expects as first argument the address of the
cdrecord-compatible program which it shall use in -sao
mode. The last argument may either be "-" for stdin or
the address of a readable file. Most other options are
handed over to that program and are followed by a last
argument "-".
For an example see below Appendix DVD, cdrecord.
pipe_to_ssh a script like pipe_data_to which forwards the input to a
remote installation of scdbackup via SSH. The first five
arguments are not ignored but define the remote service :
hostname login with_passwd remote_command remote_option
Example (for writing DVD on host ts2 via user thomas) :
.../pipe_to_ssh ts2 thomas 1 sdvdbackup -pipe_to_media
The value "1" of with_passwd indicates that the login
will ask for a password. This triggers a 15 second wait
period and a (non-password) user prompt before ssh is
executed. See also appendix NET.
pipe_to_null a script which only reports its start and then ends
unless its last argument is "-". In that case it pipes
stdin to /dev/null and ends only when stdin is closed.
It ignores any other arguments and therefore is suitable
to replace cdrecord or growisofs_wrapper when no media
shall be burned.
noop like pipe_to_null but ends immediately in any case.
mxn_wrapper a writer script that multiplies and/or permutes backup
volumes.Its arguments form three groups separated by ".":
type inner_cp media_cp [second_buf permutation_keys] .
writer_program [and options] .
[ignored arguments] buffer_file
It only works with buffer file. The backup volume gets
copied inner_cp times on each media. A number of media_cp
media is filled with this multiple volume. Writing is
done by the writer program specified in the second group
of arguments. In the last group only the last argument
is of interest. For [second_buf permutation_keys] see
appendix REDUNDANCY below.
type is either "DVD" or "CD".
ask_user performs user interaction when a new media is required.
Does not apply to archive overrun handling.
return_wrapper a script which catches the exit value of the ISO
formatter program and writes it to a file in ./tmp
In case of a crowded multi-user system one may activate a locking protocol
for recorders and result scripts which constitute pending backup runs.
The protocol prevents concurrent access to these resources.
To activate it, write "1" into fileis conf/*backup_t_lock_value or variables
*BACKUP_T_LOCK. Content "0" disables the protocol.
Two special pieces of shell code are executed inline by the example scripts
to control the locking protocol:
cd_lock_skript locks the result script
Expects the following variables to be set:
$t_lock ("1"=locking protocol activated)
$skript (name of result script)
$bdir (name of scdbackup directory)
$slock_time (lifetime of locking)
Sets:
$s_id (id of the lock)
a trap which at end of the script releases the
lock on the result script.
cd_lock_skript_and_cdrecorder refreshes the lock on the result script and
locks the CD recorder.
Expects the following variables to be set:
all which are expected by cd_lock_skript
$s_id (id from cd_lock_skript)
Sets:
$c_id (id of the CD recorder lock)
a trap which at end of the script releases the
lock on result script and CD recorder.
The locking protocol is implemented by the program temp_lock and several
scripts. The example scripts first lock their result scripts. After the result
script is generated, the CD recorder gets locked. Locking is implemented by
data which are stored in lockfiles.
Lockfiles have the name *.lock and guard the following objects :
${prefix}automatically_created*.lock backup scripts with user number and
"-cd" or "_dvd" indicating media
${prefix}currently_blanking.lock indicates an eraser run
${prefix}cdrecorder.lock all burner drives (together)
The text ${prefix} is either the address of the scdbackup installation's ./tmp
directory or ${path}/scdbackup_ if another directory ${path} has been
configured to coordinate locking.
lock filename seconds [factor] tries to obtain a lock on filename.lock .
The lock has a maximum lifetime given by seconds/factor
seconds may be a bc expression. It is important to
memorize the standard output of this command. It is
needed for releasing a lock.
Example:
id=$(lock cdrecorder 1800+7200 2)
locks for at most 1800+7200/2 = 5400 seconds
release filename id releases the lock on filename.lock . id is the text which
has been printed by lock when the lock was obtained.
Example (release the lock of the example above):
release cdrecorder $id
refresh filename id seconds [factor] prolongs the lifetime of a lock if
it is less than the given time (seconds/factor). Also
a check is made whether the lock is still valid. (If
invalid then the example scripts abort or try to
regain the lock.)
lock_state prints the active lock files
In directory ./bin there are several programs which get generated by
inst/COMPILE from C source files located in ./src .
cd_backup_planer the central planning program. It also fulfills a lot
of other tasks. See doc/cd_backup_planer_help .
raedchen counts bytes and shows the count from time to time
on standard error. A spinning bar is shown like
on a booting SUN 4 workstation.
Also it divides and concatenates large archives
if necessary.
(see: ./raedchen -help )
saskme performs accelerated evaluation of ASKME files.
temp_lock manipulates temporary locks in special lock files and
gives informations about their state.
(See temp_lock -help )
Workers and binaries produce backup scripts in scdbackup's tmp directory.
These scripts are started for performing the backup resp. restarted for
resuming an interrupted backup. Other files in tmp hold the program state
of scdbackup_askme to be used when resuming an interrupted restore.
The file names are composed by the particular commands as follows
scdbackup , scdbackup_home :
automatically_created_backup_script_${userid}${dvd}
scdbackup_afio :
automatically_created_afio_backup_script_${userid}${dvd}
scdbackup_sys :
automatically_created_sys_backup_script_${userid}${dvd}
scdbackup_askme :
askme_resume_state_${userid}${dvd} ....... full program setup
askme_resume_state_${userid}${dvd}_opt ... restore option
askme_resume_state_${userid}${dvd}_lv .... level, volume, resume start
askme_resume_state_${userid}${dvd}_src ... position within volume
askme_resume_state_${userid}${dvd}_dirty . evtl. questionable File object
${userid} is either the system userid number or the number set by
$HOME/.scdbackup_userid
${dvd} is empty for CD backups and "_dvd" for DVD backups.
Each script may be accompanied by a file ${scriptname}.pieceno which records
the current volume number of the ongoing backup. This number is used if a
backup shall be resumed and no explicit volume number is given option -resume.
The file ${scriptname}_emerging hosts the emerging script content during the
planning phase of the backup. It is later renamed to ${scriptname} if the
planning process was successfull. A very small placeholder file remains.
This procedure shall protect resumable backup scripts from being erased by
inadverted starts of of the planning process cause by a misspelled option
-resume as long as the option begins with "-".
Configuration is done by a hierachy of scripts which are located in ./inst .
Intended for execution by the user :
CONFIGURE (yet quite unnecessary) master script
CONFIGURE_CD main script for CD configuration
CONFIGURE_DVD same for DVD
SUPERUSER_CONFIGURE to be run by the superuser after configuration is done.
Installs scdbackup*_en scripts as scdbackup* commands and
sdvdbackup*_en scripts as sdvdbackup* commands.
ADD_USER to be run by other interested users
FETCH_CONFIGURATION copies previous configuration to new installation
MIGRATE_CONF_DIR adapts a configuration directory of an incremental backup
to the new scdbackup directory (rarely necessary, though).
DIET Reduces the size of the scdbackup installation without
damaging its backup functionality.
Without any arguments it removes some configuration
scripts needed only for initial configuration, the
complete german language version, most of the website
files, and the delivery scripts. It compiles the binaries
to minimal size and then removes their source code.
Optional argument -remove_inst removes all configuration
scripts, optional argument -remove_doc removes all
documentation files. Optional argument -radical does both.
SET_DEFAULT_HELPER supplement for scdbackup-*-x86-rip to quickly choose the
ISO 9660 formatter and the burn program together with
roughly suitable default options. See appendice ISO, CD,
and DVD for more qualified methods to make these choices
in persistent scdbackup installations.
For a help text run ./SET_DEFAULT_HELPER without arguments.
Non user executable :
CONFIGURE_VERSION sets variable scdbackup_conf_version
CONFIGURE_FUNCTIONS some functions common to both CONFIGURE_CD and _DVD :
evaluate_local_mkisofs() , write_mkisofs_config()
ask_text() , ask_text_vf() , write_value()
default_setting() , remove_file() , normalize_input()
check_for_tmp() , write_large_file_options() ,
write_buffer_conf()
COMPILE inline code produces the binaries from C source
PROMPT_USERID asks for a user id number if $EUID is empty
-----------------------------------------------------------------------
Appendix RECOVERY
If you need to restore files after a minor mishap then you may use the
programs which are appropriate for the backup format. Namely
mount , cp -a for ISO 9660 filesystems (for incremental ISO 9660 backups
scdbackup_askme is a nice alternative to cp -a)
afio -i for afio archives
star -x for star archives
For details see the appropriate man pages.
For instructions about restore of incremental backups see
doc/examples.html#restore_incr
The situation becomes more complicated if your whole system passed the way.
You will then have to replace eventually faulty hardware and to start from
scratch. There are several approaches which depend on your preparations and
on the particular situation after the disaster.
With all those approaches, the backups of user data and optional software
packages will be restored after the system setup is completed.
There are two main situations after a disaster :
- Your hardware can be revived resp. replaced by sufficiently compatible
components. Then you may strive for a flat restore of your system.
- New hardware has to be installed which would have overextended your
sysadmin skills if it had replaced the old hardware in a regular way
without disaster.
Then you have to install a new operating system and the system backup is
usable mainly as source for configuration files or custom system add-ons.
This illustrates the importance of a _lean_ system backup. Try to put
anything non system dependend into other backups. (See "Superuser Planning")
There are two preparation strategies supported by scdbackup :
- scdbackup_sys which records the non-volatile file objects of the system
- complete star backups of the system partitions
It can hardly be a mistake to make backups of both kinds.
There are several disaster recovery systems available which might serve
the same purpose. You should consider to use them too. The more choice you
have in case of disaster the better are your chances to recover seamlessly.
For any disaster recovery you have to record your partition layout. E.g. print
the output of
set -x ; hostname ; date ; fdisk -l ; mount ; df ; cat /etc/fstab ; set +x
on paper and store the sheets with the backups.
Only a sufficiently similar layout of the system partitions and identical
addresses of the data partitions will allow you a flat recovery of the system.
For making use of a backup made with scdbackup_sys you will need installation
media of your operating system. Copies can be made like :
dd if=/dev/cdrom of=/somewhere/bufferfile
dd if=/somewhere/bufferfile | scdbackup -pipe_to_media
resp. for DVD :
dd if=/dev/dvd of=/somewhere/bufferfile
dd if=/somewhere/bufferfile | sdvdbackup -pipe_to_media
rm /somewhere/bufferfile
Usually scdbackup_sys produces one or more media with single afio archives.
Each of them is to be unpacked by afio -i . If a media overflow had occured
at backup time then you will have to concatenate the two media contents as
shown below for star or above in paragraph "afio Archives". (It pays off to
redo an overflown system backup with smaller -max_size rather than to accept
coupled media pairs.)
With a set of scdbackup_sys media, you have these two options :
- Installation of the old operating system to the old or to sufficiently
compatible hardware.
Install the operating system in a minimal version but with all desired
partitions and filesystems. Then boot a rescue system and copy the system
backup over the minimal system installation. A suitable rescue system for
x86-Linux computers is RIP. It contains all extraction software mentioned
above.
Use the rescue systems's bootloader (e.g. GRUB) to boot the restored system,
and then run the install procedure of the system's actual bootloader
(e.g. lilo) to restore the original boot facilities.
You will also have to make the necessary configuration changes to acommodate
your new hardware. If booting would not be possible with the old
configuration, then you may try to change it from the rescue system or by
your Linux' installation tool.
- Installation of a new operating system.
Configure and tune it as near to your needs as you can. If you need
configuration parameters, refer to the configuration files in the system
backup. It is convenient to unpack it in some temporary subtree of the
new filesystem.
The system backup may also be the source of files which you cannot find in
the user data backups because they are not under /home or /opt.
If the system backup was made as star archives then the procedure stays the
same. Nevertheless star offers a more complete system backup if you allow it
to take care of the whole system partition.
To make use of a star level=0 backup of the system partitions you have to run
system backups by this command pipeline rather than by scdbackup_sys :
star -c -xdev -acl -link-dirs level=0 -C / . \
| gzip | sdvdbackup -pipe_to_media -multi_volume
Consider seriously to do this in single user mode or from a running rescue
system like RIP with scdbackup additionally installed on it (see below).
Be prepared to insert additional media if one DVD is not enough to take
the compressed archive. In that case you will have to practice concatenation
without help of scdbackup. Like this command for two media :
( dd if=/dev/hdc bs=2048 count=2293760
echo "insert next DVD" >&2 ; read dummy
dd if=/dev/hdc bs=2048 count=2293760 ) | gunzip | star -t -v
The number 2293760 is for DVD (4480 MB). With CD use 332800 (650 MB).
With a single DVD life is simpler :
gunzip </dev/dvd | star -t -v
(For real restore use star option -x rather than -t )
If /boot or /usr are separate filesystems then one should make such a backup
with those partitions too :
star -c -xdev -acl -link-dirs level=0 -C /boot . \
| gzip | scdbackup -pipe_to_media -multi_volume
star -c i-xdev -acl -link-dirs level=0 -C /usr . \
| gzip | scdbackup -pipe_to_media -multi_volume
This allows :
- Flat restore of a complete star backup which demands sufficiently compatible
hardware. If the disk was replaced or the partitioning was lost, then you
have to partition it first and install filesystems of the appropriate type.
This can either be done by a consumer Linux distribution's installation tool
or by a rescue system.
Then a rescue system is booted which can extract star archives (x86 use RIP).
Again, you will have to adapt the restored system to eventual hardware
changes.
Use the rescue systems's bootloader (e.g. GRUB) to boot the restored system,
and then run the install procedure of the system's actual bootloader
(e.g. lilo) to restore the original boot facilities.
- Installation and configuration of a new operating system and use of the
star archive as source for mainly configuration files like above with
scdbackup_sys.
If you have Intel/AMD/x86 hardware, then get an ISO 9660 image of RIP from:
http://www.tux.org/pub/people/kent-robotti/looplinux/rip/
and burn it to CD like :
dd bs=2048 if=/somewhere/RIPLinux-2.9.iso | scdbackup -pipe_to_media
Next make an emergency drill: insert the RIP CD, reboot, choose RIP in the
GRUB menu which will appear, login as "root" (no password requested), and
check your archives for readability by
afio -tv /dev/hdc
resp.
gunzip </dev/hdc | star -t -v
At this occasion you should also record the MBR of your hard disk.
See (when RIP is up) :
/usr/doc/RIP/backup-mbr.txt
There is a matching binary-only version of scdbackup-0.8.8 for RIP at
http://scdbackup.webframe.org/scdbackup-0.8.8-x86-rip.tgz
which you may download and put on some suitable media. It will help you with
unpacking encrypted backups, damaged high-redundancy backups, multivolume
archives. You may also make after-disaster backups of still readable data
partitions on your hard disks.
Just unpack after booting RIP
( cd / ; tar xzf .../scdbackup-0.8.8-x86-rip.tgz )
Most features described in this README should run out of the box. The only
mandatory configuration effort is to find out the addresses of the DVD
and/or CD burner. (Try: eject /dev/hdX , cdrecord dev=ATA -scanbus )
Without further configuration you may just set the configuration variables to
that address. Like
export SCDBACKUP_SCSI_ADR="/dev/hdd"
scdbackup /=/mnt
or
export SCDBACKUP_SCSI_ADR="ATA:1,0,0"
scdbackup /=/mnt
or for DVD
export SDVDBACKUP_DEV_ADR="/dev/hdd"
sdvdbackup /=/mnt
For verification you will give the device address explicitely and avoid the
awful behavior of the cdrom driver at end of disk :
scdbackup_verify /dev/hdd -auto_end
sdvdbackup_verify /dev/hdd -auto_end
You may run the configuration scripts, too
cd /opt/scdbackup/inst
./CONFIGURE_CD
./CONFIGURE_DVD
See also text http://scdbackup.webframe.org/scdbackup-x86-rip.txt
If you do not use x86-Linux, then you should look for a standalone-tool which
can do the following for your hardware and operating system :
- mount the desired filesystem types in read-write mode
- unpack afio resp. star archives
- for flat star restore : format disks and filesystems
Alternatively you can try to perform the unpacking in the single user mode of
a preliminarily installed operating system.
-----------------------------------------------------------------------
Appendix AUTOMAT
Although scdbackup is a software which is intended to be used in scripts,
its normal usage model is not for unattended operation. This is due to the
fact that with common burner hardware it is necessary to insert a media and
to confirm this preparation by an input line at the control terminal of
the backup run.
The input lines are buffered, so it is no problem to give such an OK for
the first volume in advance. Just press the Enter key again immediately
after you started scdbackup.
This does not help, of course, if there is no control terminal, because
scdbackup gets started as a background process (by cron, window managers, etc).
In such a case you will have to provide media management. You are lucky if
you only have to expect a single volume.
The input requests of scdbackup expect as answer a newline character at
stdin. Other (printable) characters are ignored. So you may avoid the input
requests by a pipe like this :
yes | scdbackup ...
A filter which answers automatically once and then waits for user input :
( echo ; while read dummy ; do echo ; done ) | scdbackup ...
If there is more than one volume then some kind of media change has to be
performed before the next volume can be written. To trigger this, you may
provide intermediate scripts as described in Appendix CONFIGURATION :
conf/volume_prompt_script
conf/volume_preburn_script
conf/volume_afterburn_script
conf/volume_abortedburn_script
You will have to decide what action to perform at which occasion. If you
want to send mail to a human operator and wait for a reply, then do this
in conf/volume_prompt_script.
You possibly want to control the whole backup run by own means. See script
outer_loop for an example how to break down the run into single media steps.
A note about cron :
Cron runs are known to happen in a very sparse shell environment.
scdbackup relies on the existence of environment variables $HOME and $EUID
(if it was present during configuration).
Therefore it is a good idea to wrap scdbackup in a script which ensures those
two variables are set properly. One should also not rely on the preset variable
$PATH but replace its content by the text that is replied by
echo $PATH
in an interactive shell session of that user.
Example script for user "root" :
#!/bin/sh
test "$HOME" = /root || export HOME=/root
test "$EUID" = "" && export EUID=0
export PATH="/usr/sbin:/bin:/usr/bin:/sbin:/usr/X11R6/bin"
yes | sdvdbackup -pacifier -off ...your.arguments...
-----------------------------------------------------------------------
Appendix VERIFY
The commands s*backup_verify compare the byte stream from a backup volume
with checksum records. These records may be obtained from a record list file
or from the end of the backup volume itself. The latter is possible only
with backups written by scdbackup 0.8.3 or younger.
Checksum computation, recording and comparison is usually done by program
cd_backup_planer. This appendix specifies the checksum data formats on disk and
backup media as well as redundancy and permutation formats.
Checksum Records
Checksum Tags
Block Checksum Lists
Multiplied Volumes
Permuted Volumes
xorriso Data File Checksums
Checksum Records
A checksum record consists of four words and a newline character :
name timestamp size md5
name may consist of up to 80 characters.
"1_1"
timestamp is of format YYMMDD.hhmmss with "A0" = 2000 , "A1" = 2001 ,
"B0" = 2010 , ... like: A40904.230615 .
size is a decimal number and may become quite large.
Expect more than 32 bit. Like
"4529208064"
md5 is a MD5 checksum compatible with programs like md5sum.
It is represented as printable lowercase hex number. Like
"4e64fdbb31227575b91a66776e13aa24"
The record list is a plain text file on disk with one record per line.
One may remove old lines or even the whole file content if the old records
become annoying. Also one may copy lines of the list file and use them for
verifying pre-0.8.3 CDs or DVDs on other computer systems.
The recordname is composed of the shell variable SCDBACKUP_CHECKSUM_NAME ,
the volume number and the total number of volumes. Since above variable usually
is empty, the recordname usually looks like "1_3" (1 of 3).
Checksum Tags
To make a checksum record recognizeable in the course of an arbitrarily
formatted stream it is wrapped in a checksum tag. This tag is supposed to have
a very low probability to be mistaken within the stream. This is achieved
by high inner redundancy and by the demand that the tag must tell its own
exact position in the stream (i.e. the tag producer has counted the
stream up to the tag's position). So there is a certain systematic shield
against accidential appearence of false but credible tags but the record list
on hard disk is still more secure against false verify results.
A tag consists of these components and a newline character :
version_id pos_in_stream record_len checksum_rec rec_md5
version_id is currently "scdbackup_checksum_tag_v0.1".
pos_in_stream is a decimal number which has to be identical to the byte count
up to the first character of version_id. Expect more than 32 bit.
record_len is a decimal number which gives the length of checksum_rec.
checksum_rec is a four component record as described above but without
the newline character.
rec_md5 is the MD5 checksum (hex) of checksum_rec as a whole.
The following options to commands s*backup_verify select the various search
modes. Use them as described in the help texts for option -auto :
-auto read whole checksum record list and scan for checksum tags in
the stream of the data which are to be verified.
-auto_end like -auto. End on first list match or on first credible tag.
-scan scan but do not read checksum record list from file.
-scan_end like -scan. End on first valid checksum tag, whether it matches
or not.
-scan_print like -scan. Print a list of found valid checksum tags.
This will eventually show tags of older backups.
-sorted like -auto but do not scan for checksum tags (as old -auto did).
-sorted_end like -sorted, with end after first match.
Tags get recognized only if they immediately follow the data which their
checksum record is about. I.e. only if pos_in_stream equals size .
This restriction is imposed by the scanner implementation, not by the format.
Checksum tags are produced by cd_backup_planer option -filter_checksum_padsize
and are scanned for by cd_backup_planer option -scan_for_checksum together
with -search_md5.
The configuration and the check commands use these options automatically.
If program xorriso is used for ISO 9660 formatting, and if it is modern enough
to offer option -scdbackup_tag, and if the default settings for checksum
production are in effect, then no cd_backup_planer filter is used. Instead
xorriso will compute the checksum record, append it to its output as a tag
and append it to the record list on hard disk, just as the filter would have
done.
Block Checksum Lists
Appended to a checksum tag there may be a block checksum list which begins
directly after that tag's newline character.
It comprises of the following lines and a binary part which may be unprintable.
A run of
bin/cd_backup_planer -filter_checksum_blocksize 64k -filter_md5
filtered a data stream of 229208064 bytes and produced this list :
scdbackup checksum block list v0.1
blocksize=65536
complete_blocks=3497
remainder=28672
[3498 binary block checksums of 16 byte each]
[one final newline character after the 3498*16 bytes]
md5_of_list=42eae294fe0a2e96c924b363e561c85a
chained tag follows
... meta checksum tags and meta block checksum lists ..
end of list
The part starting at "chained tag follows" up to (but not including)
"end of list" may be missing.
The printable lines got newline characters at their ends.
The header part tells that this is a block checksum list, that blocksize is 64k
and that there were 3497*64k+28672 bytes in the stream. Because remainder is
not 0 there are complete_blocks+1 binary block checksums.
The checksum md5_of_list is computed from the first line "scdbackup ch..."
up to (but not including) the newline after the binary data. A list is usable
only if its checksum still matches its content.
If the line after md5_of_list is "chained tag follows" then another checksum
tag and a block checksum list are appended which cover the previous tag and
list. These meta tags and lists can be used to recover a damaged block list.
But if md5_of_list confirms the validity of the list then any eventual meta
lists are not needed.
If the list ends with "end of list" then it is not covered by further checksum
data.
Multiplied Volumes
A backup volume may be multiplied in order to have several copies of it on the
same media (see below Appendix REDUNDANCY).
A sequence of multiplied volumes is divided by simple tags of the form
version_id pos_in_stream \n
version_id currently is "scdbackup volume multiplication mark v0.1".
pos_in_stream is a decimal number text which gives the position of the
tag within the data stream.
The blank and the newline character at the end are important.
Permuted Volumes
A backup volume may be permuted in order to expose different data blocks to
potential systematic errors (see below Appendix REDUNDANCY, Permuted Volumes).
Such permuted volumes are not usable unless they get unpacked according to
the information stored in their header blocks.
A run of
bin/cd_backup_planer -permute_blocks 4 $buffer_file 64k 0 0
with a $buffer_file of 105182569 bytes produced this header
scdbackup block permutation header v0.1
blocksize=65536
complete_blocks=1604
remainder=62825
permutation_key=4
datastart=1
telling that blocksize is 64k, total size is 1604*64k+62825, permutation is
number 4 (first and second half of volume interlaced block by block).
datastart tells that the permuted blocks begin at block number 1 (= byte 64k).
The reverse permutations are defined for a random access data source and
sequential data output by the following C-language function.
Its return value is 1 on success , 0 on failure.
#define _FILE_OFFSET_BITS=64
#define _LARGEFILE_SOURCE
int Unpermute(blocksize,complete_blocks,remainder,datastart,permutation_key,
fp_in,fp_out)
/* Parameters obtained from file header or printed paper */
double blocksize,complete_blocks,remainder,datastart;
int permutation_key;
/* These two file pointers may not refer to the same file object */
/* fp_in is for random access reading, fp_out is for sequential writing */
FILE *fp_in,*fp_out;
{
double x,idx,n,q1,q2,q3,to_read;
int mode,interlaced,ret;
/* adjust this value to your needs (or implement a read-write loop) */
#define Unpermute_max_blocksizE 1024*1024
char buffer[Unpermute_max_blocksizE];
if(blocksize>Unpermute_max_blocksizE)
return(0);
/* The chain of n permuted blocks is divided into quarters */
/* 0 <= q1 <= q2 <= q3 <= n */
n= complete_blocks+(remainder>0);
q2= floor(n/2.0);
q1= floor(q2/2.0);
q3= floor((q2+n)/2.0);
/* To produce an unpermuted output stream,
iterate over the number of data blocks */
for(x= 0.0; x<n ; x++) {
/* compute the blocknumber idx where to read block x */
mode= permutation_key&3;
interlaced= !!(permutation_key&4);
idx= x;
if(mode==0) { /* identical mapping */
;
} else if(mode==1) { /* swap halves */
if(idx<q2)
idx= idx+n-q2;
else
idx= idx-q2;
} else if(mode==2) { /* mirror */
idx= n-1.0-idx;
} else if(mode==3) { /* swap quarters */
if(idx<q1)
idx= idx+q2-q1;
else if(idx<q2)
idx= idx-q1;
else if(idx<q3)
idx= idx+n-q3;
else
idx= idx-q3+q2;
}
if(interlaced) {
if(idx<q2)
idx= idx*2.0+1.0;
else
idx= 2.0*(idx-q2);
}
idx+= datastart;
/* Read and write that block */
if(x==n-1 && remainder>0)
to_read= remainder;
else
to_read= blocksize;
ret= fseeko(fp_in,((off_t) blocksize)*((off_t) idx),SEEK_SET);
if(ret<0)
return(0);
ret= fread(buffer,(size_t) to_read, (size_t) 1,fp_in);
if(ret<=0)
return(0);
fwrite(buffer,(size_t) to_read, (size_t) 1,fp_out);
if(ret<=0)
return(0);
}
return(1);
}
Be aware that double mantissae are about 50 bit only. So this code is limited
to 1 petablocks. I decided to use data type double for the index computation
in order to achieve maximum C-language portability of that part. In case that
64 bit fseeko() and type off_t are not available for i/o addressing you may
have to find own ways to exceed the 31 bit limit of fseek().
xorriso Checksums
If program xorriso is used for ISO 9660 formatting, and if it is modern enough
to offer option -md5, then it will generate own MD5 checksums.
The superblock and the directory tree of the ISO image will get an MD5 sum
each. Damages of those entities will be detected by xorriso when it performs
-dev or -indev after -md5 "on".
Each data file will get a MD5 sum which. After the directory tree has
successfully been loaded they can be verified by xorriso option -check_md5.
-----------------------------------------------------------------------
Appendix REDUNDANCY
English language note :
This text uses the terms "redundancy" and "redundant" for information objects
which contain a certain degree of inner repetition. The other english language
meaning of these terms (like: superfluous, not needed, void) is never intended
throughout this text.
Redundant Backup Concepts
Producing Repairable Backup Volumes
Recovering Damaged Repairable Backup Volumes
Recovering Damaged Block Checksum Lists
Permuted Volumes
Repairable Backup Volumes for Long Term Backups
Even if the backup media verify after writing, they are still prone to
deterioration. If your backups are to be read after many years they might turn
out to be damaged.
The data of a completely unreadable backup media are lost, of course. But in
the case of partially unreadable media you will wish to have means to evaluate
the damage and to try a repair.
For that, you will have to prepare already at backup time. The overhead will
be substantial, so the precautions described here are not standard with
scdbackup.
Although scdbackup provides all necessary means to extract and recover data
from redundant backups, it may be wise to print out above appendix VERIFY
and store it together with the backup. It contains the definitions of the
various data formats used by scdbackup's redundant backup facility.
DISCLAIMER :
If you have to rely heavily on repairability of backups, then you *MUST* test
the whole processing chain described here whether it works for you. That means
you have to produce a redundant backup, copy one of the media images to disk,
cat </dev/cdrom >/tmp/recovering_image
damage it by help of a command like
$(scdbackup -where bin)/cd_backup_planer \
-write_to_file /tmp/recovering_image 10m string "spoiling the content"
and then try to recover it from your media by using the recovery tools of
scdbackup as described below. For example by :
$(scdbackup -where bin)/cd_backup_planer -process_blocklist \
mend /tmp/recovering_image 0 /dev/cdrom /tmp/recovering_image
You need to learn this now, as long as the emergency situation is only virtual.
A successful repair will be confirmed by :
scdbackup_verify /tmp/recovering_image -auto
Redundant Backup Concepts
A redundant backup consists of several identical copies of each backup
volume and of a block checksum list attached to each copy which helps to
identify eventually damaged parts of the volume. It contains MD5 checksums of
chunks of the total data. A block checksum list gets generated and attached
to the volume by a run of cd_backup_planer -filter_checksum_blocksize .
Such a list may be used to combine several partly damaged copies of a backup
volume to gain an undamaged volume.
The appropriate place to make use of option -filter_checksum_blocksize is
the configuration variable SCDBACKUP_MAKE_CHECKSUM which usually contains
a line with options like "-filter_checksum_padsize 1 -filter_md5".
For the following examples, the backup will be made with this setting :
export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/cd_ -filter_checksum_blocksize 64k -filter_md5"
Of course the directory for the list files has to exist :
mkdir $HOME/blocklist_dir
By this, a checksum tag is appended to the backup volume, as always. After that
tag there is appended a block checksum list which has about 0.025 % the size
of the backup data.
By -blocklist_fileadr the list also gets stored in a separate disk file :
$HOME/blocklist_dir/cd_1_1_A41123.194219_229208064
provided that directory exists.
Copies of those files should be stored at safe places and their names should
be printed and stored together with the backup volumes. Just in case the
lists appended to the volumes become all unusable and unrepairable.
If the block checksum list itself occupies more than a single block of data
then it gets appended an own checksum tag and again a block checksum list.
This is repeated until the size is smaller than a single block.
To prevent endless appending, this feature is disabled with blocks smaller
than 1k.
Separate block checksum list files which are covered by appended checksum data
may be treated by the recovery tools much like ordinary data.
So it might pay off to have lots of copies of them spread everywhere.
Producing Repairable Backup Volumes
In order to recover damaged backup data you should have at least two, better
three or more formerly identical copies of the media. The more copies you have,
the larger is the chance to collect a complete undamaged volume.
At backup time several copies of each volume can be achieved by writer
script mxn_wrapper . It not only can make copies on several media but also
is able to write several copies to a single media (provided they fit).
This is done by a run of cd_backup_planer with option -multiply_volume .
The first copy is directly usable as a normal volume. Any further copies
may be retrieved by cd_backup_planer with option -extract_multiplied .
The script mxn_wrapper does only work if a buffer file is used.
The wrapper expects three groups of arguments, separated by two arguments
which consist of a single dot ".". The first two groups set up mxn_wrapper.
They have to be provided by you. The last group is provided by scdbackup.
The first group describes the media type, the multiplication and eventual the
permutation of the backup volume :
type inner_copies media_copies [second_buffer permutation_keys] .
type may be "CD" or "DVD" selecting scdbackup or sdvdbackup for
writing via option -pipe_to_media . If in doubt, use type "CD".
inner_copies gives the number of volume copies to be written on each single
media. Of course, you have to reduce the size of the volumes by
an according factor using option -max_size .
media_copies gives the number of media which shall be filled with identical
copies of the media image resulting from the inner_copies
multiplication.
The options second_buffer and permutation keys may be omitted. They are
explained below, after the introduction of permuted volumes.
The second group describes the writer program to be used for writing a media
image to the media. This should be the settings given by
scdbackup -show_configuration | fgrep CDRECORD=
resp.
sdvdbackup -show_configuration | fgrep CDRECORD=
The answer will be something like
SCDBACKUP_CDRECORD=cdrecord -v
resp.
SDVDBACKUP_CDRECORD=.../scdbackup-0.9.2/scripts/growisofs_wrapper
The last argument of the third group gives the name of the buffer file to read.
It is appended together with the other members of group 3 by scdbackup's
scripts which believe to communicate with cdrecord. (see appendix WRITER)
Examples :
You may want to verify the media from these examples by scdbackup_verify -auto
in order to see both checksums: the one of the actual backup volume and the one
of the write process started by mxn_wrapper.
Make 2 identical copies of each CD by using standard cdrecord. The address of
the buffer file is /cdbuffer/iso_buffer_cd :
mxn_options="CD 1 2 . cdrecord -v ."
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/mxn_wrapper $mxn_options"
export SCDBACKUP_BUFFER="/cdbuffer/iso_buffer_cd"
export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/cd_ -filter_checksum_blocksize 64k -filter_md5"
scdbackup ...usual.arguments.to.scdbackup...
The same for DVD
mxn_options="DVD 1 2 . $(scdbackup -where scripts)/growisofs_wrapper ."
export SDVDBACKUP_CDRECORD="$(sdvdbackup -where scripts)/mxn_wrapper $mxn_options"
export SDVDBACKUP_BUFFER="/dvdbuffer/iso_buffer_dvd"
export SDVDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/dvd_ -filter_checksum_blocksize 64k -filter_md5"
sdvbackup ...usual.arguments.to.sdvdbackup...
Of each volume make 2 copies per media, and 3 media copies. The volumes shall
have about 315 MB of size :
mxn_options="CD 2 3 . cdrecord -v ."
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/mxn_wrapper $mxn_options"
export SCDBACKUP_BUFFER="/cdbuffer/iso_buffer_cd"
export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/cd_ -filter_checksum_blocksize 64k -filter_md5"
scdbackup -max_size 315m ...usual.arguments.to.scdbackup...
Put 6 identical CD sized volume copies on a single DVD :
mxn_options="DVD 6 1 . $(scdbackup -where scripts)/growisofs_wrapper ."
export SDVDBACKUP_CDRECORD="$(sdvdbackup -where scripts)/mxn_wrapper $mxn_options"
export SDVDBACKUP_BUFFER="/dvdbuffer/iso_buffer_dvd"
export SDVDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/dvd_ -filter_checksum_blocksize 64k -filter_md5"
sdvdbackup -max_size 640m ...usual.arguments.to.sdvdbackup...
Additional copies of existing and verified backups can be made by
scdbackup -pipe_to_media . The original backup volume has to be verified, of
course, so you do not copy damaged data and maybe even certify them by a
block checksum list.
If the backups already got appended block checksum lists it is advised to copy
them without any further checksum attachments :
unset SCDBACKUP_CDRECORD SCDBACKUP_BUFFER
export SCDBACKUP_MAKE_CHECKSUM=" "
If the backups do not yet contain block checksum lists, one should add some :
export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/ -filter_checksum_blocksize 64k -filter_md5"
In both cases, copy the media to disk, verify and then copy to other media
cat /dev/cdrom >/cdbuffer/my_buffer_file
scdbackup_verify /cdbuffer/my_buffer_file -auto
cat /cdbuffer/my_buffer_file | scdbackup -pipe_to_media
The same works with DVD and sdvdbackup -pipe_to_media .
The copied media have to verify with the same checksum record as the originals.
They also have to verify with their possible new checksum record which
eventually covers the original data as well as the original checksum appendice.
Be aware that each copy process from media eventually adds a new tail of
padding (~ 300 kB).
Recovering Damaged Repairable Backup Volumes
The prerequisite of any repair attempt is a valid block checksum list. It is
the base for the decision whether a particular block is useable or not. The
various repair attempts may be repeated - but it is essential to use the same
block checksum list each time.
One may strip a valid block checksum list together with its master tag from
a backup volume by
volume=/dev/cdrom
skip_bytes=0
$(scdbackup -where bin)/cd_backup_planer \
-blocklist_fileadr /tmp/scdbackup_ \
-scan_for_checksum -standalone -scan_for_checksum -blocklist \
-scan_for_checksum -pacifier -scan_for_checksum -first \
-scan_for_checksum -source:$volume \
-scan_for_checksum -skip:$skip_bytes \
-search_md5 - - - - >/dev/null
With above example this will produce a file of 14 kB named
/tmp/scdbackup_1_1_A41123.194219_229208064
which is equivalent to the data verification part of a separate block checksum
list file stored at backup time. If there are several valid blocklists then
they all will get stored with names which are derived from name, date and
byte count of their checksum tag.
If the backup volume is damaged, you will see the usual verifier warnings.
One may speed up extraction if a minimum estimation of the volume size is
known. In above case, it is safe to ignore the first 200 MB
skip_bytes=200m
$(scdbackup -where bin)/cd_backup_planer \
...
This causes the extractor to skip this size and eventually existent lists.
It will warn that any valid tags do not match the truncated data stream.
If skip_bytes is too large it might miss the most recent tag and extract one
which stems from an older overwritten backup.
So if you are unsure about a minimum volume size estimation, or if you want
to verify the volume during extraction, or if you want to copy the volume to
disk rather than to /dev/null, then stay with skip_bytes=0 .
If you are unlucky, the list itself is not to find or damaged. So backup
your lists which are stored on disk. In case only damaged copies are available,
see "Recovering Damaged Block Checksum Lists" below.
For an example of the recovery process we use the blocklist extracted above :
blocklist=/tmp/scdbackup_1_1_A41123.194219_229208064
We could also use the backup volume itself, but that would have to scan the
whole volume for the list and therefore would need quite some time.
We could use the list $HOME/blocklist_dir/cd_1_1_A41123.194219_229208064 which
was stored on hard disk by the backup run.
The check facility for such block checksum lists
$(scdbackup -where bin)/cd_backup_planer -process_blocklist ...
offers several methods to deal with damaged backup data :
report prints list parameters and summary to stderr and the numbers of
damaged blocks to stdout. Last line to stdout is always "@".
$(scdbackup -where bin)/cd_backup_planer -process_blocklist \
report $blocklist 0 /dev/cdrom - >/tmp/damaged_block_list
For the following recovery methods it is important to distinguish sequential
data streams (stdin, stdout, ...) from random access file objects (disk files,
block devices, ...). The kind of available data source and the desired type
of the result will lead to the choice of the appropriate method. Random access
files are also usable as sequential data streams - but not vice versa.
collect copies undamaged blocks from a data stream to a random access file.
$(scdbackup -where bin)/cd_backup_planer -process_blocklist \
collect $blocklist 0 /dev/cdrom /tmp/recovering_image
mend checks the blocks in a file with random read-write access and tries
to replace the damaged ones by their counterparts in another random
access file.
$(scdbackup -where bin)/cd_backup_planer -process_blocklist \
mend $blocklist 0 /dev/cdrom /tmp/recovering_image
This method shall read media which do not produce sequential data
streams of sufficient length. In extreme cases a tool like dd_rescue
might help to access the data survivors. In less extreme cases the
optional interval addressing mechanism of "mend" may help. You may
process from a certain block number up to the end (e.g. mend:10000)
or an interval (e.g. mend:0:9999 processes the first 10000 blocks).
You may process the blocks in reversed order (e.g. mend:end:0).
Method "mend" may also be the fastest way to repair a volume which
does not contain many damaged blocks.
merge reads two sequential data streams and replaces damaged blocks of
the one stream by undamaged blocks of the other. Output is a
sequential stream on stdout.
$(scdbackup -where bin)/cd_backup_planer -process_blocklist \
merge $blocklist 0 /dev/cdrom /tmp/recovering_image \
>/tmp/next_recovering_image
The recovery methods can be repeated with all copies available and will
hopefully yield an undamaged backup volume.
Method "merge" will report whether the checked volume is completely repaired
or still needs more recovery. Method "mend" will be able to do so only if all
blocks have been processed. The results of method "collect" may be complete
even if all "collect" runs reported errors. A run of
-process_blocklist report $blocklist 0 /tmp/recovering_image - >/dev/null
shows the current state (>/dev/null to not get flooded by block numbers).
All four methods return an exit value of 0 if they are sure that the result
is completely repaired and a non zero exit value else.
This example reads three CDs simultaneously. One of them remotely via ssh :
ssh ts2 -l thomas cat /dev/cdrom | \
$(scdbackup -where bin)/cd_backup_planer -pacifier -off \
-process_blocklist merge $blocklist 0 - /dev/cdrom | \
$(scdbackup -where bin)/cd_backup_planer \
-process_blocklist merge $blocklist 0 - /dev/cdrecorder \
>/tmp/recovering_image
This extracts both copies from a multi-copy CD and drops usable blocks to disk
file /tmp/recovering_image .
Read the first copy which is directly accessible :
$(scdbackup -where bin)/cd_backup_planer \
-process_blocklist collect $blocklist 0 /dev/cdrom /tmp/recovering_image
Now skip 1 copy and deliver the next one to the "collect" method :
$(scdbackup -where bin)/cd_backup_planer \
-extract_multiplied /dev/cdrom 1 | \
$(scdbackup -where bin)/cd_backup_planer -pacifier -off \
-process_blocklist collect $blocklist 0 - /tmp/recovering_image
This tries to squeeze data from the hinder end of a bad spot by processing
the blocks in reversed order.
$(scdbackup -where bin)/cd_backup_planer \
-process_blocklist mend:end:0 $blocklist 0 /dev/cdrom /tmp/recovering_image
If this for example yields the error message
While reading block 55376 of 66994 :
Error 5 : Input/output error
one could next try to skip the bad block and begin reading at number 55375
$(scdbackup -where bin)/cd_backup_planer \
-process_blocklist mend:55375:0 $blocklist 0 \
/dev/cdrom /tmp/recovering_image
If mend or collect do not yield a perfect result after all media are processed
then it may be rewarding to repeat them. Just in case some of the read errors
are volatile and the media are willing to produce some correct block from time
to time.
Possibly one would try to bang against aborting error spots several times from
both sides. From above :
$(scdbackup -where bin)/cd_backup_planer \
-process_blocklist mend:55400:0 $blocklist 0 \
/dev/cdrom /tmp/recovering_image
and from below
$(scdbackup -where bin)/cd_backup_planer \
-process_blocklist mend:55300:end $blocklist 0 \
/dev/cdrom /tmp/recovering_image
Recovering Damaged Block Checksum Lists
For the above examples one needs a valid block checksum list. If there are only
copies that refuse to load, one may try to recover them. This will work only
if blocksize was at least 1k and the lists are larger than one block.
A damaged tag and block checksum list may be read from a backup volume by
metablocklist=/tmp/metablocklist
$(scdbackup -where bin)/cd_backup_planer \
-extract_tag_tail 4m \
</dev/cdrom >$metablocklist
cp $metablocklist /tmp/recovering_blocklist
In that state it may serve as source of its own block checksum list and one
can try to gain undamaged blocks from other damaged volumes by this pipe :
$(scdbackup -where bin)/cd_backup_planer \
-extract_tag_tail 4m \
</dev/cdrom | \
$(scdbackup -where bin)/cd_backup_planer -pacifier -off \
-process_blocklist collect $metablocklist 0 - /tmp/recovering_blocklist
This pipe has to be repeated until all blocks of the list are repaired.
After recovery, you will have to alter such an extracted tag tail to make it
a usable block list. In the first line set to 0 the decimal number which begins
after the text "scdbackup_checksum_tag_v0.1" and which gives the size of the
preceeding backup data. Use a binary safe editor like vim.
For example change in /tmp/recovering_blocklist :
scdbackup_checksum_tag_v0.1 229208064 60 1_1 A41123.194219 229208064 4e64...
to
scdbackup_checksum_tag_v0.1 000000000 60 1_1 A41123.194219 229208064 4e64...
This will spoil the metablocklist validity of recovering_blocklist again.
So do not run more of above list recovery but use it now for volume recovery :
blocklist=/tmp/recovering_blocklist
... see above :"Recovering Damaged Repairable Backup Volumes" ...
Alternatively one may prepend the 229208064 bytes of possibly damaged backup
volume data
dd if=/dev/cdrom of=/tmp/recovering_image bs=1 count=229208064
cat /tmp/recovering_blocklist >>/tmp/recovering_image
and craft a damaged volume with valid block checksum list.
If $metablocklist is a damaged list file which was stored at backup time on
disk, then one will have to skip 1 checksum tag in order to reach the desired
meta block list data :
...
-process_blocklist collect $metablocklist 1 - /tmp/recovering_blocklist
In that case you will not have to set the first tag's position number to 0
because it already has that value.
Permuted Volumes
The above provisions should be able to recover from random errors if the number
of copies can cope with the density of errors. Nevertheless, there may be
systematic error scenarios which increase the probability that all copies of
a backup data block get hit. E.g. if the media tends to deteriorate at the
end of its data content.
A possible precaution against this menace are permuted backup volume images.
They are neither mountable nor extractable in that state but they will
expose different data blocks to said systematic errors than readable volumes
will do. The permutation can be reverted in order to get the volume readable
or at least usable as source of repair blocks.
There are 8 permutation schemes defined.
0 ... blocks stay in sequence (but shifted by one or more blocks)
1 ... swap both halves of the file
2 ... mirror sequence of blocks (i.e. revert sequence)
3 ... swap 1st and 2nd quarter, sawp 3rd and 4th
+4 ... additionally interlace resulting 1st and 2nd half
See cd_backup_planer -help and above appendix VERIFY.
It is worthwhile to use different schemes for several reserve copies.
At least have one of {1,3} and one of {4,6,7}.
Record the permutation parameters (as shown on stderr) independendly
of the permuted files. Although the permuted files are supposed to contain
those parameters in their header section, be prepared for this section
to be damaged.
Permuted backup volumes may be produced by mxn_wrapper.
The first argument group of mxn_wrapper allows to set a permutation key for
each of the media copies. The argument sequence is :
type inner_copies media_copies [second_buffer permutation_keys] .
The first three are explained above ("Producing Repairable Backup Volumes").
The other argument apply to permutation :
second_buffer gives the name of a second buffer file of media size. Such
a file is needed if inner_copies is larger than 1 together
with permutation keys. It is ignored if inner_copies is 1.
Use "-" in that case.
permutation_keys is a list of keys for cd_backup_planer -permute_blocks.
"-" means no permutation, "0" to "7" choose one of the defined
permutation schemes (of which none is readable without reverse
permutation !). blocksize is always 64k.
If the number of key arguments is less than media_copies then
unpermuted media will get written before the permuted ones.
Example :
Put 2 identical volume copies on 4 DVD. Permute the last 2 DVD . Because
the 2 on-media copies are quite invariant under permutation key 1, i choose
keys 4 (interlace) and 7 (swap quarters and interlace) :
bscripts=$(scdbackup -where scripts)
mxn_options="DVD 2 4 /my_second_disk/multiply_buffer 4 7 ."
mxn_options="$mxn_options $bscripts/growisofs_wrapper ."
export SDVDBACKUP_CDRECORD="$bscripts/mxn_wrapper $mxn_options"
export SDVDBACKUP_BUFFER="/dvdbuffer/iso_buffer_dvd"
export SDVDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/dvd_ -filter_checksum_blocksize 64k -filter_md5"
sdvdbackup -max_size 2125m ...usual.arguments.to.scdbackup...
In general, file permutation and reverse permutation can be done from a disk
file or block device by option -permute_blocks of bin/cd_backup_planer.
Examples :
Restore the original sequence of a permuted backup in a buffer file
$(scdbackup -where bin)/cd_backup_planer \
-permute_blocks restore /dev/cdrom 0 0 0 >/cdbuffer/my_buffer_file
Directly drop undamaged blocks of the second volume copy on the media into
a recovering backup volume image in the /tmp directory
$(scdbackup -where bin)/cd_backup_planer \
-permute_blocks restore /dev/cdrom 0 0 0 | \
$(scdbackup -where bin)/cd_backup_planer -pacifier -off \
-extract_multiplied - 1 | \
$(scdbackup -where bin)/cd_backup_planer -pacifier -off \
-process_blocklist collect $blocklist 0 - /tmp/recovering_image
Produce quite a mixed-up collection of reserve blocks from a buffer file
which contains a verified, readable backup volume image :
$(scdbackup -where bin)/cd_backup_planer \
-permute_blocks 7 /cdbuffer/my_buffer_file 256k 0 0 | \
scdbackup -pipe_to_media
Of course, this media will only verify with its newly computed checksum and
not with the checksum of the backup volume.
Verify a permuted volume on-the-fly :
$(scdbackup -where bin)/cd_backup_planer \
-permute_blocks restore /dev/cdrom 0 0 0 | \
scdbackup_verify - -auto
In case of a damaged header section of the permutated file, you will have to
give the necessary parameters as arguments :
$(scdbackup -where bin)/cd_backup_planer \
-permute_blocks -7 /dev/cdrom 256k $blockcount 1 >/cdbuffer/my_buffer_file
A permutation scheme which causes less disk rattling is swapping of image
halves :
... -permute_blocks 1 /cdbuffer/my_buffer_file 256k 0 0 ...
-----------------------------------------------------------------------
Appendix CRYPTO
It is advisable to store your backups far away from the computer with the
original data. This imposes the problem that it is hard to ensure that nobody
gets access to the content of those backups.
The appropriate answer to this problem is encryption.
But you have to test regularly whether you still know how to unpack such an
encrypted backup. When in need then you will find nobody else who will be able
to do that for you. See paragraph "Testing" below.
With contemporary ISO formatter programs it is not possible to encrypt only
the content of the files and not the structure of the ISO filesystem.
A completely encrypted ISO filesystem would lose its advantages over a
sequential archive format, though. So it appears natural to combine encryption
with afio rather than with ISO 9660.
The script afio_sencrypt provides the opportunity to produce encrypted and
eventually compressed afio archives. Encryption is done by program sencrypt.
See
http://scdbackup.webframe.org/README_sencrypt
Usage of vanilla afio may be replaced by usage of afio_sencrypt via the
configuration files s*backup_afio_value resp. variables S*BACKUP_AFIO.
For encryption you will have to choose a key phrase. Decryption will only be
possible if you can provide the identical key phrase again.
The phrase should contain many characters. Only the first 1000+ are used but it
does not harm to provide more. To be suitable for all possible input ways, the
phrase should not contain newline characters or characters with special meaning
to the shell. Use letters, numbers, space, period, comma, underscore, minus.
If it is not intended to memorize them in human mind, then use a mix of more or
less random characters. Else use a phrase which is long but easy to remember.
(But not your favorite movie quote which is known to all your friends !)
Next you will have to decide whether you want an unencrypted afio archive
holding encrypted files or whether you want to encrypt the archive as a whole.
The first method is quite slow but produces the usual rugged afio archives
which can be read even if the start of the archive data is missing. The file
names and their attributes (like access permissions) will stay visible in
cleartext.
The second method is faster. It hides names and attributes, but a damaged
archive will be unreadable (unless repaired by methods described in REDUNDANCY).
With this second method you got the option to use an ad-hoc key phrase which
does not have to be stored in a file.
You may also encrypt archives made by program star, but only the method
of whole archive encryption is available. At the end of each afio oriented
paragraph, there will be appropriate examples with star_as_afio_wrapper.
Encryption
Usually you will first establish a well protected keyphrase file :
mkdir $HOME/keyfiles
chmod u+rwx,go-rwx $HOME/keyfiles
touch $HOME/keyfiles/dvd_key_phrase
chmod u+rw,go-rw $HOME/keyfiles/dvd_key_phrase
vi $HOME/keyfiles/dvd_key_phrase
You will either have to memorize the keyphrase perfectly or you will have
to make an emergency copy on external media. Like :
cp $HOME/keyfiles/dvd_key_phrase /media/usbstick/dvd_key_phrase
Then you set variable SCDBACKUP_AFIO resp. SDVDBACKUP_AFIO to the appropriate
call of afio_sencrypt or star_as_afio_wrapper .
For unencrypted afio-archives containing encrypted files
export SDVDBACKUP_AFIO="$(sdvdbackup -where scripts)/afio_sencrypt -each_file_by $HOME/keyfiles/dvd_key_phrase"
For a completely encrypted afio-archive
export SDVDBACKUP_AFIO="$(sdvdbackup -where scripts)/afio_sencrypt -whole_archive_by $HOME/keyfiles/dvd_key_phrase"
If you prefer to enter the key phrase manually
export SDVDBACKUP_AFIO="$(sdvdbackup -where scripts)/afio_sencrypt -whole_archive"
Key phrases for encryption are asked twice and have to be identical both times.
For completely encrypted star-archives
export SDVDBACKUP_AFIO="$(scdbackup -where scripts)/star_as_afio_wrapper -encrypt_by $HOME/keyfiles/dvd_key_phrase"
For completely encrypted star-archives with manual key prompt
export SDVDBACKUP_AFIO="$(scdbackup -where scripts)/star_as_afio_wrapper -encrypt"
Then run your usual scdbackup afio command
sdvbackup_afio ...usual.arguments...
and memorize on paper the appropriate unpacking command from the list below !
Do not rely on availability or accurateness of a future README file.
Unpacking
To unpack the archive, you need to know what method was used and whether
compression was enabled (sdvdbackup -show_configuration | grep AFIO_COMPRESS).
Uncompressed unencrypted afio-archives containing encrypted files
afio -i -P 'sencrypt' -Q '-d' \
-Q '-f' -Q "$HOME/keyfiles/dvd_key_phrase" -Z /dev/cdrom
Compressed unencrypted afio-archives containing encrypted files
afio -i -P 'sencrypt' -Q '-d' -Q '-g' \
-Q '-f' -Q "$HOME/keyfiles/dvd_key_phrase" -Z /dev/cdrom
Uncompressed encrypted afio-archives
sencrypt -d -s 322 -f "$HOME/keyfiles/dvd_key_phrase" </dev/cdrom | \
afio -i -
(In the past -s 72 was used. Try that only if -s 322 fails.)
Compressed encrypted afio-archives
sencrypt -d -g -s 162 -f "$HOME/keyfiles/dvd_key_phrase" </dev/cdrom | \
afio -i -
(In the past -s 59 was used. Try that only if -s 162 fails.)
Uncompressed encrypted star-archives
sencrypt -d -s 752 -f "$HOME/keyfiles/dvd_key_phrase" </dev/cdrom | \
star -x
Compressed encrypted star-archives
sencrypt -d -g -s 162 -f "$HOME/keyfiles/dvd_key_phrase" </dev/cdrom | \
star -x
To get the passphrase prompted at /dev/tty rather than being read from a file,
leave out option -f "$HOME/keyfiles/dvd_key_phrase" . (Only suitable for
archives which where encrypted as a whole.)
For decryption key phrases are asked only once.
Of course, you may use afio -tv (list content) rather than afio -i (unpack)
or you may use option -y to extract single files :
sencrypt -d -g -s 162 </dev/cdrom | afio -i -y 'etc/X11/XF86Config*' -
See also man afio (resp. man star).
Testing
Test whether you are able to reproduce the keyphrase correctly.
Do this by creating a second key file without using the original one:
touch $HOME/keyfiles/test_key_phrase
chmod u+rw,go-rw $HOME/keyfiles/test_key_phrase
cat /media/usbstick/dvd_key_phrase >$HOME/keyfiles/test_key_phrase
or
vi $HOME/keyfiles/test_key_phrase
or omit the -f "$HOME..." arguments in the following examples to be prompted
for the key phrase at /dev/tty.
For testing decryptability of an unencrypted archive with encrypted files, you
will have to extract a file and check whether it is readable.
Uncompressed :
mkdir $HOME/test_dir
cd $HOME/test_dir
afio -i -P 'sencrypt' -Q '-d' \
-Q '-f' -Q "$HOME/keyfiles/test_key_phrase" -Z \
-v -y '*ASKME*' /dev/cdrom
view .../*ASKME*
Compressed :
...
afio -i -P 'sencrypt' -Q '-d' -Q '-g' \
...
For testing decryptability of an archive which is encrypted as a whole, it is
sufficient to test whether afio resp. star can produce a table of content.
Uncompressed afio-archive :
sencrypt -d -s 322 -f "$HOME/keyfiles/test_key_phrase" </dev/cdrom | \
afio -tv -
Compressed afio-archive :
sencrypt -d -g -s 162 -f "$HOME/keyfiles/test_key_phrase" </dev/cdrom | \
afio -tv -
(In the past -s 72 resp. -s 59 have been used)
Uncompressed encrypted star-archive :
sencrypt -d -s 752 -f "$HOME/keyfiles/test_key_phrase" </dev/cdrom | \
star -tv
Compressed encrypted star-archive :
sencrypt -d -g -s 162 -f "$HOME/keyfiles/test_key_phrase" </dev/cdrom | \
star -tv
-----------------------------------------------------------------------
Appendix ATTRLIST
There are two families of file attributes which are extensions to the
traditional Unix properties : ACLs and Extended Attributes. They are available
only with certain filesystem types and may demand special options of
command mount. See : man 5 acl, man 5 attr, man 8 mount (options "acl", "xfs"
and "user_xattr").
Neither afio nor ISO 9660 are able to store those attributes together
with the file items. So if they are present in the backup area then they have
to be recorded separately in lists suitable for the utilities setfacl and
setfattr. See : man 1 setfacl, man 1 setfattr
For this, utilities getfacl resp. getfattr need to be present at backup time.
The archive format exustar of program star is able to record and restore ACLs.
If the script star_as_afio_wrapper is used with option -acl then there should
be no need to take the precautions described here. (see appendix WORKERS)
You will need a recent version of star to read those archives, though.
scdbackup offers this capability independantly of the backup format via
the following options :
-facl_list_adr ....... records traditional permissions and eventual ACLs
for the target addresses of the backup content
-faclsrc_list_adr .... same for source address
-fattr_list_adr ...... records eventual Extended Attributes, target addresses
-fattrsrc_list_adr ... same for source address
For details see : $(scdbackup -where bin)/cd_backup_planer -help | less
Example :
scdbackup -facl_list_adr /added_by_scdbackup/facl.gz=facl.gz \
-fattr_list_adr /added_by_scdbackup/fattr.gz=fattr.gz \
...usual.arguments...
With above command the last volume of the backup will contain the files
/added_by_scdbackup/facl.gz
/added_by_scdbackup/fattr.gz
With format afio a target address must be present but the list files will
always be stored under their source address. The facl and fattr list files
are stored near the end of the last volume, possibly followed by some other
list files. To get an overview of the last 10 items of the volume, execute :
afio -t /dev/cdrom | tail
The reply (which needs some time) should contain two line likes these
u/thomas/scdbackup/scdbackup-0.9.2/tmp/facl.gz
u/thomas/scdbackup/scdbackup-0.9.2/tmp/fattr.gz
You may also set a directory address of your own, but this has to be done by
source address and therefore the list file will emerge on your disk at that
address.
scdbackup_afio -facl_list_adr /dummy=$HOME/facl.gz ...
Within the lists, file addresses are recorded relative to the /-directories
of the volumes.
After copying the backup volumes to disk, e.g. by
mkdir /home/restore
cp -a /cdrom/* /home/restore
you will have to apply the lists to the resulting tree :
cd /home/restore
gunzip <added_by_scdbackup/facl.gz | setfacl --restore=-
gunzip <added_by_scdbackup/fattr.gz | setfattr --restore=-
Of course, the file system hosting /home/restore has to support ACLs to allow
the success of setfacl if the ACL is not actually a set of traditional Unix
permissions. setfattr needs Extended File Attributes in any case.
-----------------------------------------------------------------------
Appendix LINKS
Program (from package) URL
cdrecord (cdrtools) ftp://ftp.berlios.de/pub/cdrecord/alpha
wodim (cdrkit) http://debburn.alioth.debian.org/
growisofs (dvd+rw-tools) http://fy.chalmers.se/~appro/linux/DVD+RW/tools
dvd+rw-format ( " ) http://fy.chalmers.se/~appro/linux/DVD+RW/tools
dvd+rw-mediainfo ( " ) http://fy.chalmers.se/~appro/linux/DVD+RW/tools
cdrskin http://scdbackup.webframe.org/cdrskin_eng.html
http://scdbackup.sourceforge.net/cdrskin_eng.html
mkisofs (cdrtools) ftp://ftp.berlios.de/pub/cdrecord
genisoimage (cdrkit) http://debburn.alioth.debian.org/
xorriso http://scdbackup.webframe.org/xorriso_eng.html
http://scdbackup.sourceforge.net/xorriso_eng.html
afio http://members.brabant.chello.nl/~k.holtman/afio-2.5.tgz
star http://cdrecord.berlios.de/old/private/star.html
sencrypt http://scdbackup.webframe.org/README_sencrypt
RIP Rescue Linux http://www.tux.org/pub/people/kent-robotti/looplinux/rip
scdbackup for RIP http://scdbackup.webframe.org/scdbackup-x86-rip.txt
http://scdbackup.webframe.org/scdbackup-0.8.8-x86-rip.tgz
Home of libisofs, libburn, libisoburn which serve in cdrskin and xorriso:
libburnia project http://libburnia-project.org
-----------------------------------------------------------------------
Appendix ISO :
The scripts ./CONFIGURE_CD and ./CONFIGURE_DVD try to find and evaluate
suitable programs for producing ISO 9660 filesystem images. There are several
program names builtin for this check:
mkisofs
Production of ISO 9660 filesystem images tradionally depends on Joerg
Schilling's mkisofs from package cdrtools. mkisofs is supposed to be part
of any older Linux distribution.
Source of current cdrtools is best fetched from:
ftp://ftp.berlios.de/pub/cdrecord/alpha
genisoimage
genisoimage is the mkisofs fork within package cdrkit. Sometimes it is
distributed as "mkisofs".
It is quite modern and fully sufficient for the needs of scdbackup.
Package cdrkit can be obtained from:
http://debburn.alioth.debian.org/
xorriso
xorriso is an integrated ISO 9660 Rock Ridge multi-session writing tool.
Its ISO 9660 formatting capabilities are based on libisofs from
libburnia-project.org. The higher application levels are written by me.
xorriso is well suited for the needs of scdbackup towards a formatter program.
Obtain xorriso-*.tar.gz from:
http://scdbackup.sourceforge.net/xorriso_eng.html
xorriso has been declared first choice as formatter because it can cut the
pieces of split large files without needing a large disk buffer.
It handles the attributes of split directories more appropriately for
scdbackup's needs.
Another advantage is its ability to go on with its work even after quite
dramatic failures happened. Its exit value triggers a failure reaction of
scdbackup if a backuped file either vanished during backup or changed its
content in a way that cannot be recorded properly.
So in this case, the volume production stops and the user is urged to redo
the problematic volume. The user may well choose to go on with the next
volume number instead.
It is possible to avoid this interrupt of the backup volume processing by
setting a higher -return_with threshold:
export SDVDBACKUP_MKISOFS="xorriso -return_with FATAL 32 -abort_on FATAL -as mkisofs -graft-points"
export SCDBACKUP_MKISOFS="$SDVDBACKUP_MKISOFS"
xorriso is the only program which can serve both roles: ISO 9660 formatter
and CD/DVD writer program. See next two appendice CD and DVD.
Selection during Configuration
./CONFIGURE_* tries to find the above programs and evaluates which might be
best suited.
The list of found programs becomes part of the user prompt:
What ISO 9660 formatter program to use ? ( xorriso genisoimage mkisofs )
If no ISO formatter was set before, then the first list member will be offered
as default and thus can be chosen by an empty input line.
The normal ranking is
1st xorriso >= 0.1.2 (If providing option -as mkisofs)
2nd genisoimage (This surely provides modern mkisofs features)
3rd mkisofs (There have been some not-so-good versions long ago)
The default choice may also be set by variables before the run of
./CONFIGURE_CD resp. ./CONFIGURE_DVD.
Select xorriso
export SCDBACKUP_USE_GENISOIMAGE="xorriso"
export SDVDBACKUP_USE_GENISOIMAGE="xorriso"
Select mkisofs
export SCDBACKUP_USE_GENISOIMAGE="mkisofs"
export SDVDBACKUP_USE_GENISOIMAGE="mkisofs"
Select genisoimage
export SCDBACKUP_USE_GENISOIMAGE="genisoimage"
export SDVDBACKUP_USE_GENISOIMAGE="genisoimage"
Temporary Selection
The use of the formatter program is controlled by variable SCDBACKUP_MKISOFS
for CD:
export SCDBACKUP_MKISOFS="xorriso -return_with MISHAP 32 -abort_on FATAL -as mkisofs -graft-points"
export SCDBACKUP_MKISOFS="genisomage -l -allow-leading-dots -R -D -graft-points"
export SCDBACKUP_MKISOFS="mkisofs -l -allow-leading-dots -R -D -graft-points"
and with the same contents by SDVDBACKUP_MKISOFS for DVD.
-----------------------------------------------------------------------
Appendix CD :
The script ./CONFIGURE_CD tries to find and evaluate suitable programs for
burning CD media. There are several program names builtin for this check:
cdrecord
The CD subsystem traditionally depends on Joerg Schilling's cdrecord which
comes in package cdrtools together with mkisofs. cdrecord or a near clone
of it is supposed to be part of any older Linux distribution.
Source of current cdrtools is best fetched from:
ftp://ftp.berlios.de/pub/cdrecord/alpha
The usable addresses have undergone some evolution following the changes in
hardware customs. Real SCSI CD recorders have become rare. But USB and SATA
appear as pseudo-SCSI devices in Linux. On older Linux kernels the module
ide-scsi makes them appear like SCSI devices. Newer kernels offer them as
pseudo hard disks /dev/hdX.
The following table tries to give you an overview over Linux kernels and
possible addresses for CD burners :
Kernel SCSI,USB,SATA IDE + ide-scsi IDE (Joerg) IDE (Linus)
2.2 0,0,0 0,0,0 -unknown- -not usable-
2.4 0,0,0 0,0,0 ATA:1,0,0 -not usable-
2.6 0,0,0 -deprecated- ATA:1,0,0 /dev/hdc
The Joerg-style addresses know other prefixes like "REMOTE:". Run
cdrecord dev=HELP
to get a list. Read the prefix from lines "Transp. layer ind.:".
Use prefix without ":" to learn about the according devices. Like :
cdrecord dev=ATA -scanbus
Note: ATA does not work while the recorder is guarded by ide-scsi.
wodim
wodim is the cdrecord fork within package cdrkit.
It stems from cdrecord 2.01.01 and therefore is fully suitable for scdbackup.
Package cdrkit can be obtained from:
http://debburn.alioth.debian.org/
A valuable feature of wodim is that it needs no superuser authority but only
rw-permission for /dev/sgN resp. /dev/hdX. It might nevertheless be a bit
of guesswork to find out the /dev/sgN addresses which to chmod +rw.
cdrskin
For scdbackup on Linux kernel 2.4 and above, cdrskin is a complete alternative
to cdrecord. It is based on the library libburn from libburnia-project.org.
Its greatest appeal is that it is written by me. :))
Make sure to have version cdrskin-0.2.6 or newer in order to get the features
needed by scdbackup. With Linux kernel 2.4 the burner device must be under
control of ide-scsi emulation. With more modern kernels it shouldn't.
cdrskin is provided as source code and as Linux-x86-binaries at
http://scdbackup.webframe.org/cdrskin_eng.html
http://scdbackup.sourceforge.net/cdrskin_eng.html
To configure scdbackup for the use of cdrskin, the superuser should first
ensure that the CD devices have proper permissions set. Normal users will only
see devices which offer rw-permissions.
As superuser:
cdrskin --devices
The reply will be something like
0 dev='/dev/sr0' rwrwr- : 'TEAC' 'CD-ROM CD-532S'
1 dev='/dev/sr1' rwrw-- : 'LITE-ON' 'LTR-48125S'
from which you may derive a command which allows full access for everybody
(you may be more restrictive, if you want):
chmod a+rw /dev/sr0 /dev/sr1
xorriso
xorriso is able to perform single session burn runs in the way as needed by
scdbackup. Its writing capabilities are based on libburn from
libburnia-project.org. See also above at the end of Appendix ISO.
You need at least xorriso-0.1.2 for the "-as cdrecord" emulation
Although it is sufficient, its appearance is not very similar to cdrecord,
wodim or cdrskin. It does not understand the Bus,Lun,Target addresses of
cdrecord tradition but only /dev/... .
A run of
xorriso -devices
will show a list of available drive addresses.
This deviation of traditions earns xorriso the last place in the automatic
selection list. In general it is less flexible than cdrskin which uses the
same foundation software.
Selection during Configuration
./CONFIGURE_CD tries to find the above programs and evaluates which might be
best suited.
The list of found programs becomes part of the user prompt:
What CD burn program to use ? ( cdrskin cdrecord xorriso wodim )
If no CD burn program was set before, then the first list member will be
offered as default and thus can be chosen by an empty input line.
The normal ranking is
1st cdrskin >= 0.2.6 (In case of trouble, i can give support for this)
2nd cdrecord (Not easy to tell what gets found under that name)
3rd xorriso >= 0.1.2 (In case of trouble, i can give support for this)
4th wodim (In summer 2008 there seems to be no support for wodim)
As long as no CD burn program is set, it matters whether a program actually can
detect a CD drive to which it would be able to write. So a properly installed
cdrecord might outrank a cdrskin which suffers from missing rw-permissions.
The default choice may also be set by variables before the run of
./CONFIGURE_CD resp. ./CONFIGURE_DVD.
Select cdrecord
export SCDBACKUP_USE_CDRSKIN="cdrecord"
Select wodim
export SCDBACKUP_USE_CDRSKIN="wodim"
Select cdrskin
export SCDBACKUP_USE_CDRSKIN="cdrskin"
Select xorriso
export SCDBACKUP_USE_CDRSKIN="xorriso"
Then run
./CONFIGURE_CD
To trigger the full initial examination of programs:
export SCDBACKUP_USE_CDRSKIN="default"
./CONFIGURE_CD
Temporary Selection
For switching temporarily between modern versions of the programs it should be
enough to set one out of this list:
export SCDBACKUP_CDRECORD="cdrecord -v"
export SCDBACKUP_CDRECORD="wodim -v"
export SCDBACKUP_CDRECORD="cdrskin -v"
# Important: xorriso cannot handle cdrecord style addresses like "0,0,0".
export SCDBACKUP_CDRECORD="xorriso -as cdrecord -v"
to set a suitable drive address
export SCDBACKUP_SCSI_ADR="/dev/sr0"
and then to run the backup command
scdbackup ...
-----------------------------------------------------------------------
Appendix DVD :
The script ./CONFIGURE_DVD tries to find and evaluate suitable programs for
burning DVD media. There are several program names builtin for this check:
growisofs
The DVD subsystem traditionally depends on Andy Polyakov's dvd+rw-tools and
especially on his program growisofs.
The source of dvd+rw-tools is best fetched from
http://fy.chalmers.se/~appro/linux/DVD+RW/tools/
See also below: "How to get and install growisofs".
For BD (Blu-ray) media, you need at least growisofs-7.1.
cdrskin
For scdbackup on Linux kernel 2.4 and above and except some exotic media,
cdrskin is a complete alternative to growisofs, dvd+rw-mediainfo and
dvd+rw-format. It is based on the library libburn from libburnia.pykix.org.
Make sure to have version cdrskin-0.3.4 or newer in order to get all DVD
features needed by sdvdbackup. Older versions will not get selected
automatically. With Linux kernel 2.4 the burner device must be under control
of ide-scsi emulation. With more modern kernels it shouldn't.
cdrskin is provided as source code and as Linux-x86-binaries at
http://scdbackup.webframe.org/cdrskin_eng.html
http://scdbackup.sourceforge.net/cdrskin_eng.html
See above in Appendix CD, paragraph "cdrskin" for necessary drive preparation.
For BD (Blu-ray) media, you need at least cdrskin-0.6.0.
xorriso
xorriso is able to perform single session burn runs on the same media as
cdrskin with which it shares the fundamental burn software.
libburnia-project.org. See also above Appendix ISO.
Both programs are about equivalent from the view of sdvdbackup. Because
cdrskin accepts a larger set of drive addresses, it is slightely preferrable
over xorriso.
For BD (Blu-ray) media, you need at least xorriso-0.3.2.
wodim
wodim is the cdrecord fork within package cdrkit.
It stems from cdrecord 2.01.01 and has own DVD capabilities. See "Restrictions
with cdrecord and wodim".
Package cdrkit can be obtained from:
http://debburn.alioth.debian.org/
cdrecord
As of cdrtools-2.01.01a09, DVD burning is included in the source code version.
In the past there were time limited binaries "cdrecord-ProDVD" which are
obsoleted by the current cdrtools source releases.
You may obtain 2.01.01aXY releases from:
ftp://ftp.berlios.de/pub/cdrecord/alpha
Restrictions with cdrecord and wodim
Regrettably both, cdrecord(-ProDVD) and wodim, only offer DAO mode for DVD
which implies that without a buffer file on disk every DVD has to be padded
up to the maximum size.
Another disadvantage of both is that they do not offer convenient mode
"Restricted Overwrite" with DVD-RW media.
It has not been tested whether they work properly with BD (Blu-ray) media.
Selection during Configuration
./CONFIGURE_DVD tries to find the above programs and evaluates which might be
best suited.
The list of found programs becomes part of the user prompt:
What DVD burn program to use ? ( growisofs cdrskin cdrecord xorriso wodim )
If no CD burn program was set before, then the first list member will be
offered as default and thus can be chosen by an empty input line.
The normal ranking is
1st xorriso >= 0.1.8 (In case of trouble, i can give support for this)
2nd cdrskin >= 0.4.6 (In case of trouble, i can give support for this)
3rd growisofs (Well tested with sdvdbackup)
4th cdrskin >= 0.3.4 (Does not know stream_recording= for BD)
5th xorriso >= 0.1.2 (Does not know stream_recording= for BD)
6th wodim (In summer 2008 there seems to be no support for wodim)
7th cdrecord(-ProDVD)(Must issue "This copy of cdrecord is licensed for:"
or "Cdrecord-ProDVD")
As long as no DVD burn program is set,it matters whether a program actually can
detect a CD drive to which it would be able to write. So a properly installed
cdrecord might outrank a cdrskin which suffers from missing rw-permissions.
The default choice may also be set by variables before the run of
Select growisofs
export SDVDBACKUP_USE_PRODVD="growisofs"
Select cdrskin
export SDVDBACKUP_USE_PRODVD="cdrskin"
Select xorriso
export SDVDBACKUP_USE_PRODVD="xorriso"
Select cdrecord
export SDVDBACKUP_USE_PRODVD="cdrecord"
Select wodim
export SDVDBACKUP_USE_PRODVD="wodim"
Then run
./CONFIGURE_DVD
To trigger the full initial examination of programs:
export SDVDBACKUP_USE_PRODVD="default"
./CONFIGURE_DVD
Temporary Selection
Switching temporarily between those is a bit more complicated, because not only
the address of the burn program has to be changed, but also several companion
variables:
-------------------------------------------------------------------------------
Select growisofs temporarily
-------------------------------------------------------------------------------
export SDVDBACKUP_USE_PRODVD="0"
export SDVDBACKUP_CDRECORD="$(sdvdbackup -where scripts)/growisofs_wrapper"
export SDVDBACKUP_CDRECORD_OPT="-final_eject fs=16m"
export SDVDBACKUP_DEV_ADR="/dev/sr0"
-------------------------------------------------------------------------------
Select cdrskin temporarily
-------------------------------------------------------------------------------
export SDVDBACKUP_USE_PRODVD="cdrskin"
export SDVDBACKUP_CDRECORD="cdrskin -v"
export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject -data -tao assert_write_lba=0 fifo_start_at=0 driveropts=burnfree"
export SDVDBACKUP_SCSI_ADR="0,0,0"
or
export SDVDBACKUP_SCSI_ADR="/dev/sr0"
(But not SDVDBACKUP_DEV_ADR)
-------------------------------------------------------------------------------
Select xorriso temporarily
-------------------------------------------------------------------------------
export SDVDBACKUP_USE_PRODVD="xorriso"
export SDVDBACKUP_CDRECORD="xorriso -as cdrecord -v"
export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject"
export SDVDBACKUP_SCSI_ADR="/dev/sr0"
(But not SDVDBACKUP_DEV_ADR)
-------------------------------------------------------------------------------
Select cdrecord temporarily
-------------------------------------------------------------------------------
export SDVDBACKUP_USE_PRODVD="1"
export SDVDBACKUP_SCSI_ADR="0,0,0"
export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject driveropts=burnfree -data -sao"
Without buffer file (i.e. on-the-fly)
export SDVDBACKUP_BUFFER="-"
export SDVDBACKUP_MEDIA_CAP=4480m
export SDVDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_cdrecord_sao cdrecord"
With buffer file
export SDVDBACKUP_BUFFER="/dvdbuffer/isofs_dvd"
export SDVDBACKUP_CDRECORD="cdrecord -v"
-------------------------------------------------------------------------------
Select wodim temporarily
-------------------------------------------------------------------------------
export SDVDBACKUP_USE_PRODVD="wodim"
export SDVDBACKUP_SCSI_ADR="0,0,0"
export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject driveropts=burnfree -data -sao"
Without buffer file (i.e. on-the-fly)
export SDVDBACKUP_BUFFER="-"
export SDVDBACKUP_MEDIA_CAP=4480m
export SDVDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_cdrecord_sao wodim"
With buffer file
export SDVDBACKUP_BUFFER="/dvdbuffer/isofs_dvd"
export SDVDBACKUP_CDRECORD="wodim -v"
-------------------------------------------------------------------------------
How to get and install growisofs
Get the source package from :
http://fy.chalmers.se/~appro/linux/DVD+RW/tools/
and install according to the instructions given.
Meanwhile binary versions of dvd+rw-tools are part of Linux distributions.
It is a good idea to read the tutorial of Andy Polyakov's DVD software :
http://fy.chalmers.se/~appro/linux/DVD+RW/
because growisofs is useful independendly of scdbackup.
Error codes (like "SK=5h/ASC=21h/ACQ=00h") can be looked up at
http://fy.chalmers.se/~appro/linux/DVD+RW/keys.txt
In case you find no other source for dvd+rw-tools there is provided a
(outdated) copy under
http://scdbackup.webframe.org/dvd+rw-tools-5.19-1.4.9.7.tar.gz
Untar, enter the directory dvd+rw-tools-5.19.4.9.7, and execute make .
Then copy the programs to a directory where they can be executed by all
interested users. E.g. :
cp growisofs dvd+rw-format dvd+rw-mediainfo /usr/bin
Find out the SCSI device address of your DVD recorder. In my case it
was /dev/sr0 . You may count the entries of "Type: CD-ROM" in the output
of
cat /proc/scsi/scsi
until you reach the entry with your recorder. The first entry will be
/dev/sr0 , the next one /dev/sr1 and so on.
With Linux kernel version below 2.5 the EIDE DVD recorder has to be
emulated as an SCSI device as described by Andy's tutorial (or by
README appendix IDE where an ATAPI CD-RW gets attached as an SCSI device).
With newer kernels and recent versions of growisofs it is possible to
use IDE device addresses directly without SCSI emulation.
Make sure that the device offers read+write permissions to all interested
users. If it is not readable you will get slightely confusing messages
from growisofs.
-----------------------------------------------------------------------
Appendix DVD-TYPES (as of june 2008) :
There are many types of DVD media and BD (Blu-ray) media.
In general, media and writer firmware have to match properly. It is not
easy to determine this match in advance. Generally it is a good idea to
stick with the media list of the writer's manufacturer. Nevertheless, there
is no guarantee that those media will work or that others won't.
All my DVD burners achieve good results with 4x DVD+RW media.
My BD burner works fine with several media brands except the BD-RE disc which
was provided by its manufacturer. That one is totally unusable.
The reusal peculiarities of DVD+RW and DVD-RW are dealt automatically by
sdvdbackup if you answer "y" to ./CONFIGURE_DVD's question
"Automatically erase DVD before writing ? (y/n)".
This automat recognizes DVD+R, DVD-R, BD-R for which it omits erasing.
DVD+R, DVD+R DL and DVD-R
Each of these media can be used only once. sdvdbackup writes one backup on it
and then it cannot write to that disc again.
With DVD+R DL you need quite recent versions of the DVD burn programs.
Use sdvdbackup Option -dvd_dl to announce its unusual media size.
DVD+RW
DVD+RW seem to be the least problematic media. They are writeable and
re-writeable without any further precautions. The writer programs growisofs,
cdrskin, cdrecord and wodim can deal with them and re-use them without
further precautions.
I tested Fuji Film 4xDVD+RW, TDK DVD+RW47C, Tevion/Platinum 4x, Octron 4x.
They all are reported by dvd+rw-mediainfo as Media ID: "RICOHJPN/W11".
Meanwhile there appeared "RITEK/004" which work well, too.
Under Linux kernel 2.6 it is possible to use DVD+RW like DVD-RAM.
The restricted number of re-writes makes it not suitable for an rw-mounted
filesystem, though. The "tape device" method described with DVD-RAM works fine.
DVD-RW
A DVD-RW is in one of several media states which automatically select a certain
write behavior. Additionally to growisofs one needs program dvd+rw-format from
the same package dvd+rw-tools. cdrskin and xorriso are able to do the necessary
formatting and blanking by themselves. cdrecord and wodim can blank by
themselves.
- Restricted Overwrite media may be overwritten by sdvdbackup
and will stay in state Restricted Overwrite.
Media initially get into that state by :
dvd+rw-format -force /dev/...
cdrskin -v dev=... blank=format_overwrite
xorriso -outdev ... -format as_needed
Or via
sdvdbackup_blank
The burn programs cdrecord and wodim can neither
format DVD-RW nor write to formatted DVD-RW.
Less appealing with sdvdbackup are the sequential states.
- Sequential Blank .... media may be overwritten by sdvdbackup
and will then be in state Sequential Nonblank.
Blank is the state of previously unused media.
Written sequential DVD-RW can be in one of the two following states:
- Sequential Closed media is not suitable for sdvdbackup. It may be erased
and converted into state Sequential Blank by :
dvd+rw-format -blank=full /dev/...
cdrskin -v dev=... blank=all
xorriso -outdev ... -blank all
cdrecord -v dev=... blank=all
wodim -v dev=... blank=all
Or via:
sdvdbackup_blank all
- Sequential Appendable media is not suitable for sdvdbackup. growisofs and
cdrskin would be willing to add more data, but
sdvdbackup is not prepared to do so.
The same blank commands apply as with Closed media.
A special state can cause trouble and therefor it is better to avoid it:
- Sequential Fast Blank media can only be written in DAO mode. This demands a
buffer file or script pipe_to_cdrecord_sao as well as
modification of eventual cdrskin options.
In fact, this state is only suitable for burn programs
cdrecord and wodim, which have DAO as only write mode.
A DVD-RW gets into this state by:
cdrecord -v dev=... blank=fast
wodim -v dev=... blank=fast
For use with sdvdbackup via growisofs, cdrskin, or xorriso i advise state
"Restricted Overwrite" despite its misleading name.
This state is stable in itself and makes a DVD-RW behave much like a DVD+RW.
It may be necessary or desirable, though, to use the Sequential write mode.
If so, you will have to bring the media into state Sequential Blank before
each reusal. This lasts as long as a full 4.7 GB write run, if you do not want
to get "Fast Blank" media which is unsuitable for sdvdbackup together with
the better DVD burn programs.
With cdrecord and wodim there is only one possible write mode (-sao) and the
media can be in the sequential states.
DVD-RAM
growisofs, xorriso and cdrskin can use DVD-RAM like DVD+RW. See above.
DVD-RAM is supported directly at least by Linux kernels 2.4 or higher.
There is no media specific formatting needed (unlike DVD-RW).
You may install a filesystem on DVD-RAM like on a hard disk partition and
mount it for read-write filesystem operations. If so, sdvdbackup can make
use of it in the same way it could make use of a hard disk file system.
If the DVD-RAM is *not mounted* sdvdbackup can use it like a tape device
with quite similar behavior as a DVD+RW. This is achieved via script
pipe_to_raedchen which may replace growisofs_wrapper.
With reporting in megabytes steps :
export SDVDBACKUP_CDRECORD=\
"$(scdbackup -where scripts)/pipe_to_raedchen /dev/sr0 1m 0 -flush"
Without intermediate messages :
export SDVDBACKUP_CDRECORD=\
"$(scdbackup -where scripts)/pipe_to_raedchen /dev/sr0 0 0 -flush"
The fact that the slow DVD-RAM media share i/o buffer space with the system
hard disk is not very helpful for system performance during DVD-RAM writing.
Therefore the somewhat clumsy -flush option of raedchen has to be used.
The write and read performance of DVD-RAM was much lower than those of
DVD-RW or DVD+RW. There have been failed verifications, too.
Be aware that the usable capacity of a DVD-RAM is less than 4.7e9 bytes.
My tests with Panasonic LM-AF120U media yielded raw 4368m and with a
vanilla ext2 file system 4076m.
sdvdbackup option -dvd_ram adapts to the size of raw DVD-RAM media.
By default DVD-RAM is checkreading while writing. This reduces speed to half
or even less. The drive reaction in experiments with bad media has not been
convincing. The drives just need longer to abort a bad burn.
Programs xorriso and cdrskin can avoid this checkreading by option
stream_recording=.
Be aware that DVD-RAM might not be readable in older DVD-ROM drives.
BD-RE
BD-RE are much like very large DVD-RAM. They are sold unformatted and thus
have to get formatted first. This is done automatically by the BD burn
programs or may be done explicitely:
dvd+rw-format /dev/...
cdrskin -v dev=... blank=as_needed
xorriso -outdev ... -format as_needed
You need quite recent burn program versions for handling BD-RE.
The usable size depends on the individual formatting. Usually it is 22.5g
for media with nominal "25 GB".
sdvdbackup* commands can deal with the media.
Use their option -bd to announce "25 GB" media and option -bd_dl for "50 GB".
BD-RE by default perform Defect Management which shall detect small groups of
bad media blocks and replace them by reserve blocks. This usually slows down
writing speed by a factor of 2 or 3.
The burn programs cdrskin and xorriso are able to circumvent this mechanism
and to achieve full nominal writing speed of the media. This is controlled by
file sdvdbackup_stream_recording_value resp. variable SDVDBACKUP_STREAM_REC,
and may be configured together with the input of speed.
BD-R
BD-R can be used only once, like CD-R, DVD+R. They are sold unformatted and
may be used in that state. Unformatted BD-R do not perform Defect Management
and can be written with full nominal speed.
growisofs automatically formats them before first use.
The burn programs cdrskin and xorriso use unformatted BD-R if given.
BD-R may also get formatted explicitely:
dvd+rw-format /dev/...
cdrskin -v dev=... blank=format_if_needed
xorriso -outdev ... -format as_needed
sdvdbackup option -bd announces "25 GB" media. Option -bd_dl is for "50 GB".
To exploit the full capacity of unformatted BD-R you may specify a slightly
larger size by -max_size 22.2g resp. -max_size 44.4g .
Credits:
Some of the test media have been donated by Mike Evans, MFaya, Thomas Weber.
-----------------------------------------------------------------------
Appendix IDE (as of june 2008) :
This text applies to CD or DVD writers which are connected to ATA controllers
under Linux kernel 2.4. It does not apply to drives which appear as SCSI
device files /dev/sgN and /dev/srM unconditionally. I.e. those connected
via genuine SCSI or USB.
With Linux kernel 2.6 the ide-scsi emulation has fallen into disgrace.
Modern versions of cdrecord can access ATAPI devices directly via addresses
like ATA:0,1,0 or ATAPI:0,1,0 . Use
cdrecord dev=ATA -scanbus
wodim dev=ATA -scanbus
cdrskin dev=ATA -scanbus
to get an overview of available devices and their addresses. Text
ftp://ftp.berlios.de/pub/cdrecord/README.ATAPI
tells more. (ATA seems to be preferrable over ATAPI.)
Although cdrecord issues complaints, addresses like "/dev/hdc" work with
kernel 2.6 for all three programs. Such files may be listed by:
wodim --devices
cdrskin --devices
xorriso -devices
growisofs (for DVD) on kernel 2.6 accepts /dev/hdX files as of at least
dvd+rw-tools-5.17.4.8.6 . On kernel 2.4 it is /dev/srN, which might occur
on 2.6 systems too.
During installation of a modern consumer distribution with Linux kernel 2.4
an IDE writer should be recognized automatically and ide-scsi emulation should
get set up properly. If there are no real SCSI devices present, the addresses
should be 0,0,0 for CD and /dev/sr0 for DVD. Then you are lucky and already
done.
If not (or if you are no consumer), you will have to do some sysadmin work.
With a Linux kernel 2.4 or below, following text applies :
How my IDE CD writer came to SCSI address 1,0,0
My Yamaha SCSI writer deceased in 2003 and i was forced to purchase an IDE
writer (with ridiculous high speed for an incredibly cheap price).
Now i needed to present it to cdrecord as another SCSI device.
I did this with SuSE 7.2 using LILO as bootloader. With other systems
this might be completely different. Refer to the manuals of your Linux
distribution.
The IDE writer (actually an ATAPI writer) is master at secondary IDE.
Therefore in Linux : hdc
LILO : File /etc/lilo.conf
I added the kernel option "hdc=ide-scsi" to the append line. Like:
append="...existing.options... hdc=ide-scsi"
There was no old append line. So it looks now as this :
append="hdc=ide-scsi"
Then i ran the command lilo as superuser.
SuSE 7.2 : File /etc/modules.conf
There was a line :
alias scsi_hostadapter off
which i changed according to the manuals
alias scsi_hostadapter ide-scsi
My SCSI CD-ROM still works :))
SuSE 7.2 : File /etc/init.d/boot.local
Since the module did not load automatically, i added to the boot script:
/sbin/modprobe ide-scsi
After a reboot, cdrecord -scanbus listed a second SCSI bus with a single
device at address 1,0,0 .
The new CD reading device became /dev/sr1 like with the old SCSI writer.
That possibly depends on the SCSI addresses of old CD-ROM and writer.
So you may have to correct /etc/fstab and softlinks in /dev
(like /dev/cdrom and /dev/cdrecorder) if sr1 and sr0 get swapped
by the new SCSI-Adress 1,0,0.
Not a good idea was to attach the IDE writer as slave hdb of the hda disk.
Transactions like blanking or fixating blocked the disk for minutes.
Also disk performance was awful during data writing.
I therefore transplanted my disk hdc to hdb and attached the writer
to an IDE controller of its own.
That new writer was a LITE-ON LTR-48125S (40x12x40) which shows convincing
performance as well as a fine protection against buffer underrun. No special
options needed with cdrecord_prog (an old cdrecord-1.6). So it should
also work with other CD recording software based on cdrecord.
With newer versions of cdrecord you may need to enable -driveropts=burnfree
during the run of ./CONFIGURE (which will notify you about this feature).
Meanwhile i am using a LG GSA-4082B DVD writer with SuSE 9.0 which during
installation got nicely recognized and SCSI-emulated with address 0,0,0
resp. /dev/sr0 .
It can be used as /dev/hdc resp. ATA:0,0,0 on a kernel 2.6 based RIP-Linux
system.
-----------------------------------------------------------------------
Appendix NET :
scdbackup mainly is designed only to backup data on the computer with the
CD burner. What to do with a network of computers ?
If you want to backup data accessible via NFS or other file systems, you
may run scdbackup on the computer with the CD burner and give it those NFS
addresses. But there still is the issue of backups where you need to be local
superuser and there also may be the need to backup the file systems directly
without the additional filter of a network file system.
A more general approach which uses SSH is presented here. It consists of
normal scdbackup installations on each of the hosts for each user who wants to
do backups. All formatting is done locally on these hosts and only the data
stream is transported over the network to the remote burner.
On the host with the burner drive there is a user identity with a scdbackup
installation properly configured for the burner. This user identy has to offer
SSH login to the scdbackup user on the burnerless machine in the network.
The burner configurations of the login user will be in effect. I.e. locking,
blanking, buffering and checksum handling will mainly be controlled by the
burner host installation. Nevertheless the configured activities around burning
will be performed on the client host according to its configuration, too.
So a remote backup will get two checksum tags appended. The first one verifies
the remote data transfer and the second one verifies the burner success.
Verification option -auto is advised.
If the burner is vulnerable to misburns caused by low data transfer, it is
advised to configure a buffer file for that user. Network traffic may become
slow due to external influences.
After these preconditions are established and tested, the scdbackup user
needs to do an own installation on the burnerless host. As address for
the burner one may enter "-" (rather than "0,0,0" or "/dev/sr0"). The archiver
program afio and an ISO formatter are needed to do the backup formatting.
The role of the burn program will be fulfilled by scripts/pipe_to_ssh .
On the burnerless machine, enter as "burn program" a string of the form
username@hostname:passwordflag
username and hostname are the usual ssh parameters. password_flag is either 0
or 1, depending on whether the SSH server asks for a password on login.
If a password request is expected, there will be a sleep period of 15 seconds
and an input request at /dev/tty (not stdin). So in that case you will be
forced to attend the backup and press Enter once per media.
If you have concerns about a SSH shell account without password then consider
a specialized account without shell interpreter but rather a script which
starts scdbackup -pipe_to_media resp. sdvdbackup -pipe_to_media .
That will require different users for CD and DVD, of course.
The appropriate remote_command and remote_option for pipe_to_ssh would be
"-" each. Like : .../pipe_to_ssh ts4 burndvd 0 - -
Other scdbackup tasks like verifying or blanking should be performed on the
machine which got the appropriate drive. Usually the human operator will
sit at the machine with the burner and use a SSH login to start the scdbackup
runs on the burnerless hosts (which then use SSH to forward their data stream).
There is no way to temporarily override the remode scdbackup settings.
You will have to configure several login users to switch between usage models.
Examples of configuration on the burnerless machine:
To use the scdbackup installation of user thomas on host ts2 where the SSH
server is expected not to ask for a password, check in advance whether this
will work with SSH :
ssh ts2 -l thomas scdbackup -version
Run ./CONFIGURE_CD and answer the question
"What CD burn program to use ? ( ... )"
by input
thomas@ts2:0
Additionally disable local blanking to avoid (harmless) error messages.
"Automatically erase ... before writing ? (yj1/n0)"
by input
n
Server ts4 asks thomas for a password on login. So use passwordflag value 1:
thomas@ts4:1
Again, disable blanking.
One may temporarily redirect backups to host ts2, user thomas and use its
configured CD burner ("scdbackup" "-pipe_to_media") :
export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_ssh ts2 thomas 0 scdbackup -pipe_to_media" \
export SCDBACKUP_BLANKEN=0
-----------------------------------------------------------------------
Appendix TUNING :
Optimized Compilation versus Debuggable Compilation
The binary programs of scdbackup are generated with optimization for low CPU
load and no support for debugger programs.
In case that the compiler option -O2 is suspicicous or debugging is desired,
one may do (while the burners are not busy):
cd $(scdbackup -where inst)
export SCDBACKUP_COMPILE_MODE="-debuggable"
./CONFIGURE_CD
and answer the usual questions by empty input lines.
It has to be stressed that usually the runtime of scdbackup isn't determined
by its cpu load but by disk performance and/or burner speed.
Optimizations after Compiling
Depending on size and structure of the backup area, scdbackup's planning phase
can last a while. With nicely shaped file trees, the speed is mostly restricted
by filesystem performance.
Besides the speed there is the memory consumption of the planning run which
can cause problems. Although scdbackup tries to economize on memory it
eventually has to model large trees and directories which inherently need their
space.
In some special situations, nevertheless, scdbackup's implementation is to
blame for long planning runs or for exaggerated use of memory. This is due to
the fact that the balance between run time and memory consumption is tuned
for average trees on normal Linux systems.
If both, run time and memory are a problem then you should consider to divide
the backup area into several ones and to backup each of them with a frequency
which is appropriate to its importance and agility.
As long as there are still reserves of either run time or memory, it might help
to change the balance.
Composition Memory
Some backup areas contain very large trees which have to be split at deep
subdirectory levels in order to obtain digestible pieces. In this situation
it is repeatedly necessary to determine the sizes of subtrees by rescanning the
appropriate part of the filesystem. This can be a big waste of time because it
has to be redone for every directory level before the split depth is reached.
Option -composition_memory allows to record subtrees up to a particular
depth and with. So the sizes of these subtrees can be reused and there is no
need to access the filesystem again. Regrettably this can use up substantial
memory, so this buffering is quite restricted by default.
If the processing after "beginning to compose backup volumes" lasts very
long after reporting "... splitting /some/directory", then try a run with
-composition_memory 5 1000 64m \
-pacifier -timestamp -pacifier -composition_memory
The number 5 sets the depth of the provisionary buffer (5 directory levels).
The number 1000 sets the width, i.e. the maximum size of a directory's file
list so that it is allowed to be buffered. The number 64m allows to use at most
64 MB of memory for that buffering.
Watch consumption of memory during the first test run. Wether the limit is
obeyed in fact, depends much on the system's dynamic memory management.
Watch for pacifier messages with the same directories again after the
announcement "... splitting /some/directory". Sufficient buffering will
reduce these repetions resp. will prevent them at all if the parameters
are generous enough (and if you can affort that generosity).
The pacifier line will have the format
seconds directory_count bufferstate emerging_tree_size address
bufferstate is composed of 3 numbers
current_depth:number_of_buffered_directories:used_buffer_bytes
If recording is disabled due to the limits, current_depth is displaid as "X".
directoy_count is usually a single number. With large directories there
may be a file counter appended after a "/".
address is at or above the current crawler position and may be truncated in
order not to exceed the total message lenght of 79 character.
The pacifier messages emerge on stderr. You may direct them to a log file by
scdbackup ... 2>&1 | tee -i /tmp/scdbackup_run.log
and later view them (in a more readable layout) by
$(scdbackup -where bin)/cd_backup_planer -debug_stderr_log \
</tmp/scdbackup_run.log | less
If the situation is appropriate and the width is sufficient, then you may
expect a speedup by a factor of (depth+1).
Sometimes it can be of help to increase width and decrease depth. That depends
much on the shape of the tree. Make experiments. (For daily use, you may
decide not to use the additional -pacifier fields which are ordered above.)
Fast Composition
With -composition_mode fast the backup volumes are composed simultaneously
together with the file check. An eventual content list gets written in parts
as soon as a volume is completely composed. This saves several possible passes
over the backup area and should speed up the planning process substantially
if RAM cannot buffer the needed information for subsequent passes.
The main disadvantage is the fact that no economy considerations can be made
at that time. This leads to quite large automatically_created_backup_scripts
and in extreme cases may even waste media space if -max_items is exceeded
before -max_size.
Fast composition should be considered if increased composition memory did
not yield sufficient acceleration within the limits of available RAM.
Tree Hash Size
If the backup area contains directories with very many file items it can
last a long time to perform -load_old_tree and the CPU load during the
incremental file check can be quite high.
This can be tuned by option -tree_hashsize which controls the size of an
acceleration vector and the existence of hash lists in the directories of
the loaded old_tree model. The default of 20 lists is well suited for normal
sized directories of 20 to 1000 files. Directories with several ten thousand
files may slow down the program substantially whereas trees with very many
directories may consume too much memory.
Use for speeding up an extremely large flat directory:
-tree_hashsize 200
(beware of memory consumption, though)
Use for maximum memory reduction
-tree_hashsize 0
(beware of CPU load and run time, though).
If the size is larger than 0, each file item in the old_tree will use extra
sizeof(int *) bytes of memory (= 4 bytes on 32-bit Linux).
Each directory will use additional size*sizeof(int) bytes (= size*4 bytes on
32-bit Linux).
-----------------------------------------------------------------------
Appendix INODE :
The inode based method of incremental backups is usually much faster than the
old CRC based method. It is supposed to work on any reasonable filesystem type
so my advice is to switch old incremental configurations to the new method
and to read the further paragraphs of this appendix if unexpected results
appear on the first test runs with level 1 or above.
How to switch from the old method to the new one
The method of incremental backup is set in file
specific_arguments
within the configuration directory. This file was written by the last run
with -level -create_configuration resp. -level -update_configuration and the
new method will be automatically enabled with the next run of those options.
It might be more convenient to edit the file specific_arguments and to
replace the sub options ":crc:" resp. ":md5:" by ":inode:". So it looks like :
-changetest_options
type:size:inode:adr
-list_output_format
type:inode:size:adr
You are free to change this setting before any -level 0 backup. Higher levels
depend on the recorded informations of older levels.
So do run a -level 0 !
Inode Method of Incremental Backup Planning
A major problem with incremental backups on Linux is the fact that with all
filesystem types known to me, an address change of a file does not necessarily
update its timestamp. The number of occasions where this happens varies with
Linux version and filesystem type but all of them are prone in some way.
Thus a mere time comparison (-younger_than) would not recognize name changes
where an old file replaced another old file.
At restore time such a failure would result in a completely outdated content
attached to the affected filename.
The normal settings after -level -create_configuration or
-level -update_configuration cover this risk by checking the inode number and
the device number of any old file against the file's recorded numbers.
(See man 2 stat : dev_t st_dev, ino_t st_ino)
This method assumes that these two numbers which are associated to a particular
file address will change whenever a content change occurs that does not
update one of the file's timestamps (e.g. content replacement by name change).
Device numbers aren't the optimal choice to identify a filesystem but there
is no other traditional file object property which could be used instead.
Inodes are actually an inner implementation detail of certain filesystems.
Their ID-numbers have successfully claimed their place as unique identifiers
for file objects, though. They are mentioned as "file serial numbers" in the
POSIX specs. Nevertheless i do not know any clear statement within POSIX which
guarantees the persistence of that number as long as the file stays unchanged.
The inode method works only if one may assume a persistent association of
device-inode pair and the file object as understood by user and name space.
The currently known cases where this assumption fails are not overly common:
- A change in the disc adapter, whether physically or just by changing the
logical attachment to the system, may change the device number.
Vast parts of the filesystems will then be recognized as "changed" and
included in the backup although they are unchanged.
With some bad luck, a newly attached disk inherited the old device number
and a file on it does incidentially have the same type, size and inode number
as a file with the same name on the old disk. This file will be missing in an
incremental backup - with possibly dire consequences.
So after fiddling with your disks, go back to backup level 0.
- If the system assigns device numbers at boot time in a varying way, then
the fullfeatured method of the inode check is unusable.
This should become obvious by much too fat backup at -level 1 or higher.
You may work around this problem by adding to the -changetest_options
:ignore_device: . But do this only if you are sure that no directories
migrate from one filesystem to the other.
(To my knowledge this would happen only if you move mount points.
:ignore_device: is safe in any way if your backup only covers a single
filesystem resp. a part of it. Run command df for a filesystem overview.)
- If the inode numbers are assigned in a varying way at mount time
- which seems to be halfway covered by the tradition - then no kind of
inode check is usable.
I found indications in the web that VFAT and SMBFS are acting this way.
Personally i use the inode method for my backups of ext2 and reiserfs on a
vanilla SuSE 9.0 system.
The Old Checksum Method
scdbackup-0.8.4 and older used a slow method which records and compares content
checksums rather than system specific id numbers. It may still be used if
the inode numbers of one of your filesystems are not persistent enough.
To set up an incremental backup configuration for use of the old method, you
may add approprate -changetest_options and -list_output_format options :
scdbackup -changetest_options type:size:md5:adr \
-list_output_format type:size:md5:adr \
-conf_dir ... -level -create_configuration \
...adresses.of.backup.area...
Instead of "md5" you may also use "crc" to get the smaller CRC-32 checksums.
Why can't i just trust my ctime ?
Depending on version and type of your filesystem, the timestamp "ctime" will
get updated on rename. But even then there is still the possibility that a
file is renamed implicitely by renaming one of its parent directories.
Start situation (as listed by command ls -lcd ) :
drwxr-xr-x 1 thomas thomas 4096 Oct 6 08:46 a
drwxr-xr-x 1 thomas thomas 4096 Oct 6 08:46 b
-rw-r--r-- 1 thomas thomas 115 Oct 6 08:46 a/x
-rw-r--r-- 1 thomas thomas 115 Oct 6 08:46 b/x
On Oct 11 i perform
mv a away
mv b a
and get
drwxr-xr-x 1 thomas thomas 4096 Oct 11 14:01 away
drwxr-xr-x 1 thomas thomas 4096 Oct 11 14:01 a
-rw-r--r-- 1 thomas thomas 115 Oct 6 08:46 away/x (former a/x)
-rw-r--r-- 1 thomas thomas 115 Oct 6 08:46 a/x (former b/x)
Although my ext2 filesystem did update the timestamps of the moved directories
there was no update of the timestamps of their files.
None of the criteria besides checksum or inode number can detect that file
a/x is not the same as five days before. This is because a/x did exist before,
was a data file and did contain exactly the same number of bytes.
This precondition is not very likely but it is not impossible.
Having said this, here is the way to be daring and to rely entirely on
timestamps, type and size of the files. Change
specific_arguments
in the configuration directory of the incremental backup (-conf_dir).
So it looks like :
-changetest_options
type:size:adr
-list_output_format
type:size:adr
-----------------------------------------------------------------------
Appendix SNAPSHOT :
The following text is not based on own experience but on reading documentation.
Special thanks to Joerg Schilling and the manpage of his program star.
Be aware that snapshots will reduce the time windows for inconsistent file
states to the outmost minimum but that there remains a substantial risk of
recording such states. So with agile data base files or with problematic
system trees you will possibly have to do the backup from Single User Mode
or from a rescue system like RIP (see appendix RECOVERY).
The description of LVM promises snapshots which represent a frozen copy of
a filesystem while normal operations are going on undisturbed. Although this
does not solve all potential inconsistency problems of a live filesystem,
a snapshot is *the* preferrable source for a backup of a running system, if
that feature is available.
About LVM snapshots, see :
http://www.tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html
http://www.tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html
from where i learned about
# lvcreate -L592M -s -n dbbackup /dev/ops/databases
...
lvcreate -- logical volume "/dev/ops/dbbackup" successfully created
(Caution: the backup example is very optimistic about the inner consistency of
a database file in a snapshot inadverted to the database server.)
Several operating systems offer builtin support for snapshots (e.g. by fssnap).
The only drawback of a snapshot known to me is a pitfall with incremental
backups, which MUST BE AVOIDED by all means:
In the time between snapshot creation and start of scdbackup there may
happen changes with the life copy of the filesystem. If scdbackup records its
own start time as criterion for the next (future) level, then it will
exclude those changes from that future level.
Although the affected timespan will usually be quite small, this can still
have very dire consequences when restoring the backup.
Therefore you MUST use scdbackup option -level_timestamp to set the criterion
date to a point in time NOT LATER THAN THE SNAPSHOT.
Examples :
... halt programs and services which might write to the backup area and
therefore risk to get snapshot in an inconsistent state ...
echo > /tmp/scdbackup_snapstamp
... the commands for creating the snapshot ...
... resume programs and services which have been halted ...
scdbackup -level_timestamp /tmp/scdbackup_snapstamp -level 1
or
...
timestamp=$(date '+%m%d%H%M%Y.%S')
...
scdbackup -level_timestamp "$timestamp" -level 1
or just date it back one hour :
...
scdbackup -level_timestamp -1h -level ...
-----------------------------------------------------------------------
Appendix WRITER :
The content of parameter *_cdrecord_value has to point to a program which
in some aspects has to be compatible with cdrecord.
This program will be started by scdbackup for performing several tasks
and will get described these tasks by arguments suitable for cdrecord.
So the most general demand is that it must tolerate any cdrecord option
without aborting and without doing inappropriate things.
In addition it may expect own arguments, which preceed the cdrecord arguments
and set specific configuration parameters of the writer program. It has to be
able to recognize where its arguments end and where those for cdrecord begin.
This is achieved in most cases by prescribing a strictly fixed number of own
arguments.
The own arguments are set together with the program's address in
*_cdrecord_value . In scdbackup's view they fixely belong to that program.
A very simple writer will only evaluate its own few parameter arguments
and skip all others up to the last one.
If this last argument is "-", then the writer will take its input from stdin.
If the last argument is the address of an existing file, then it will read
that file up to EOF.
In any other case it will do nothing and return exit value 0.
Examples for this kind of writer are pipe_data_to and pipe_to_ssh .
A writer may offer further capabilities by the following options, like
it is done by growisofs_wrapper :
-atip retrieve media type data and print DVD type as "book type:"
CD types as " Is erasable" resp. " Is not erasable".
blank=type blank a DVD-RW disk (type= "fast" or "all")
-eject eject after work
To be recognized by scdbackup, these must be announced in the output of option
-help print help text and exit
in a way that combined stdout and stderr match these grep expressions :
.*-atip.*retrieve
.*blank=type
.*-eject.*after.*work
Such a writer has to be able to fulfill all blanking operation requests in a
senseful manner. growisofs_wrapper employs dvd+rw-mediainfo and dvd+rw-format.
Other writers may well just discard such requests silently. It depends much on
the needs of their target media.
If the particular capabilities are not offered in the writer program's -help
text then scdbackup will not request them but try to work around that lack
and eventually even abort if this lack is fatal (e.g. automatic blanking with
no fixed media type while writer offers blank=type but not -atip).
The most complete writers, of course, are modern cdrecord and cdrskin
or scripts which finally hand over all unknown arguments to said programs.
An example is pipe_to_cdrecord_sao , which manipulates some arguments as
well as the data input of a cdrecord run.
Nevertheless, the growisofs_wrapper class above does cover all current needs
of scdbackup. It is more compatible than old cdrecord_prog (no -atip).
A writer may reply a version text by
-version if only argument : print a version text
which should match grep expressiom
[Cc]drecord
only if -atip, blank and -eject are supported properly.