# scdbackup-0.9.2 http://scdbackup.webframe.org scdbackup@gmx.net Thomas Schmitt http://scdbackup.sourceforge.net Purpose: backup large amounts of data on one or more CDs, DVDs, or BDs, simple (therefore no excuse not to do the backup), no special tool needed for reading the backup Formats: ISO 9660 file systems (readable after : mount ) afio archives (readable by : afio ... /dev/cdrom ) star archives (readable by : star ... = 0.1.2 this means you need a split directory. The buffer file should not be necessary if you got modern hardware. (It is useful for advanced purposes, nevertheless.) After you decided to have a split directory, find a suitable filesystem and create a new empty directory there. For a buffer file you would just have to choose a name in an existing directory on a suitable filesystem. Suitable here means : with enough free space. You must have separate directories resp. files for CD and DVD unless you only got one single burner which is in charge for both media types. I strongly recommend to use different names in any case. Set the access permissions of the newly created split directories to rwx for the interested user(s). (In most frigthening case: chmod a+rwx ) Make sure the directory of an eventual bufferfile offers rwx-permissions to all those users. Each single scdbackup installation (i.e. for security reasons : each user) should use own addresses for their buffer directories and buffer files. ------------------------------------------------------------------------------ Installation To achieve a clear security situation each user should have an own separate scdbackup installation. Steps 1 and 2 must be done by the particular user who shall use that installation. This is not necessarily the superuser but rather the owner of the backup data. The superuser may follow these steps for the installation of an own, very private backup system. But superuser authority will elsewise only be needed for a few optional activities. If programs cdrecord and/or growisofs are installed properly on the system and may be used by the installing user for burning, then there is no urgent need for superuser actions at all. Each of the steps may be repeated to restore original software and permission settings or to edit the configuration. --- Step 1 : Copy the archive to a directory where you can create a sub directory scdbackup-0.9.2, go there, unpack the archive and go to the sub directory inst newly created by this. For example use $HOME/scdbackup : mkdir $HOME/scdbackup cd $HOME/scdbackup tar xzf $HOME/downloaded_stuff/scdbackup-0.9.2.tar.gz cd scdbackup-0.9.2/inst If the superuser is doing this, it is also absolutely necessary to do chown root ../../scdbackup-0.9.2/*/* ../../scdbackup-0.9.2/* \ ../../scdbackup-0.9.2 and best to issue an appropriate chgrp command, too. --- Step 1b (only needed when upgrading from an older version) : If you already had installed another version of scdbackup, you may now copy some history data and the old configuration as a template into the new directory : ./FETCH_CONFIGURATION This refuses to overwrite any files so it only makes sense in a freshly unpacked directory. Any users afterwards are still configured to use their old installation. If you maintain incremental backup configurations and suffered from long planning times you may be interested in reading the first paragraph of appendix INODE below. Only if you programmed own applications which make use of inner parts of scdbackup previous to version 0.8.5, only then you should now read appendix DIRECTORIES. The mere use of commands scdbackup* or sdvdbackup* within your programs will continue to work without ./COMPATIBILITY_LINKS, of course. Unless the user id number is 0 the following scripts write the files .scdbackup_dir and .scdbackup_lang into the user's $HOME-Directory. They eventually write file .scdbackup_userid into the user's $HOME-Directory. --- Step 2a if you got a CD writer (for DVD writer only, see Step 2b below) : While the CD-Recorder is not busy, run : ./CONFIGURE_CD Several questions have to be answered: What ISO 9660 formatter program to use ? (... found programs ...) What CD burn program to use ? (... found programs ...) Address of CD recorder ? (... found addresses ...) Speed ? (e.g. 2) Automatically erase CD before writing ? (yj1/n0) Large file split directory ? ( "-" = exclude large files) Buffer file name ? ( or "-" or "-afio_compress" ) Estimated capacity for a compressed afio CD ? (eg. 1024m) Exclusion list of directories for system backup ? ( With other shells than bash : Your user id number ? ) If no disk buffer (see above) is needed then answer the question "Large file split directory" by a single "-" or if advised by the message text: "-xorriso_cut_out". "Buffer file name" should be "-afio_compress" for true burnproof hardware and media, else enter "-" or a filename. The "Exclusion list of directories for system backup" is only of interest if you are superuser right now. As normal user just press the Enter key. --- Step 2b if you got a DVD writer (works with or without Step 2a) : If neither command growisofs nor cdrskin nor wodim nor cdrecord is not found by your shell then you will have to install dvd+rw-tools as described in Appendix DVD. While the DVD writer is not busy, run : ./CONFIGURE_DVD Several questions have to be answered: What ISO 9660 formatter program to use ? (... found programs ...) What DVD burn program to use ? (... found programs ...) Address of DVD recorder ? (... found addresses ...) Speed ? (e.g. 2) Automatically erase DVD before writing ? (yj1/n0) Large file split directory ? ( "-" = exclude large files) Buffer file name ? ( or "-" or "-afio_compress" ) Estimated capacity for a compressed afio DVD ? (e.g. 8000m) Exclusion list of directories for system backup ? ( Non-bash users also may be prompted for a unique user id number ) If no disk buffer (see above) is needed then answer the question "Large file split directory" by a single "-" or if advised by the message test: "-xorriso_cut_out". "Buffer file name" should be "-afio_compress" for true burnproof hardware and media, else enter "-" or a filename. The "Exclusion list of directories for system backup" is only of interest if you are superuser right now. As normal user just press the Enter key. --- Step 2c. (Only with private installation for superuser) For his own private installation of scdbackup, the superuser has to run : ./ADD_USER Make no mistake : only if you are already superuser now and only with the installation which shall do the superuser's backup jobs and where no other user is permitted. After Configuration Permission settings after a first configuration will not allow others to use your scdbackup installation. This is for your security. Stay with it. The commands of scdbackup are now accessible in subdirectory ./cmd of $HOME/scdbackup/scdbackup-0.9.2 . Make a first test : $HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home -version should reply "scdbackup 0.9.2 2009.12.13.060857" (timestamp varies). You may now add $HOME/scdbackup/scdbackup-0.9.2/cmd to your PATH variable : export PATH="$PATH":"$HOME/scdbackup/scdbackup-0.9.2/cmd" and then execute commands like scdbackup_home -version There are startup scripts like .profile or .bashrc where you may add this directory path permanently. Ask your sysadmin. If this is an upgrade from scdbackup-0.8 or older then the superuser will have to remove the old commands in /usr/bin (or wherever they are) or to overwrite them as described in the next paragraph. If this is your first encounter with scdbackup then i advise you to next read "Usage" or just try $HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home (without "-version"). The eventual superuser activities can wait for an hour - if burn program and CD/DVD/BD writing device are in proper state, that is. Superuser Part of Installation The superuser may decide to copy the scdbackup commands to a public command directory rather than to teach his users how to permanently set their PATH. These command scripts are quite small and merely direct the users to their own scdbackup installations. Their only hazard is to get tricked into a fake scdbackup installation by the user's own environment variables SCDBACKUP_DIR or HOME or by the user's own configuration file $HOME/.scdbackup_dir . Regular installations of other users should repel such an attempt, anyway. With upgrades from scdbackup-0.8 or older, the superuser has to make sure that the old installed commands get removed or overwritten by the new ones. Commands "which scdbackup" and "which scdbackup_sys" should tell where they are located. A mistrusting superuser may decide to install the commands manually following this template (copy commands to /usr/local/bin and /usr/sbin ): target=/usr/local/bin chmod a-w ../scdbackup-0.9.2/cmd/* cp ../scdbackup-0.9.2/cmd/* "$target" chown root "$target"/scdbackup* "$target"/sdvdbackup* chmod a+rx,u+w,go-w "$target"/scdbackup* "$target"/sdvdbackup* chmod go-x "$target"/scdbackup_sys "$target"/sdvdbackup_sys (have a look into /usr/local/sbin whether there is s*dbackup_sys to be removed) But there is a script which does this more comfortably, reliably and avoids any "*" in the commands. If the superuser intends to use scdbackup for own backup runs, then there is no reason to particularly mistrust the following script. (Bug reports welcome) --- Step 3 (optional, see possible reasons above) Become superuser now and run (within directory scdbackup-0.9.2/inst) ./SUPERUSER_CONFIGURE This script copies the command scripts scdbackup* to a public command directory like /usr/bin or /usr/local/bin if CD writing is actually configured. If DVD writing has been configured, then the commands sdvdbackup* are copied to that directory. You will be asked for the public command directory where to install the scripts Install directory ? (e.g. /usr/bin , "-" = do not install) The user should afterwards check whether scdbackup_home -version yields the same result as in the first test : $HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home -version There is no need to repeat step 3 with further user's installations. Permission to use the recorder: Depending on the burn program used, the permission to use a CD/DVD recorder has to be achieved in different ways: cdrecord (package cdrtools) Needs to be run by the superuser or to be owned by the superuser and treated with chmod u+s growisofs (package dvd+rw-tools) The user needs rw-permission for /dev/srN resp. /dev/hdX. Uses device addresses but no SCSI numbers. chmod u+s is deprecated. wodim (package cdrkit) Either to be used like cdrecord or to be used under normal user id with rw-permissions on appropriate /dev/scdN resp. /dev/hdX files. cdrskin (packages cdrskin The user needs rw-permission for /dev/srN resp. or libburn) /dev/hdX. The superuser should perform cdrskin --devices to learn about the device files for chmod +rw Application of chmod u+s is strongly deprecated. xorriso (packages xorriso Same preconditions as with cdrskin. Uses device or libisoburn) addresses but no SCSI numbers. xorriso is a burn program as well as an ISO 9660 formatter. If the superuser does not have an own scdbackup installation yet, then this would nearly be the time to start -as superuser- with step 1 in order to get one. But first read this : Superuser Planning The superuser will most likely use format afio unless there are user data to take care of, which clearly benefit from ISO 9660 format. By help of script star_as_afio_wrapper one may alternatively employ the archive format of program star. (See example in appendix WORKERS.) The idea is to have a rather slim system backup, which is made with scdbackup_sys Be aware that scdbackup_sys does not create a recovery system but just records the files (even quite special ones) in an afio archive. This conserves the system configuration, but for re-installing you need to do a base installation of Linux first. The base installation has to be compatible with the system you made the archive from. So make copies of your Linux CDs to have a compatible system at hand when needed. You may also use program star as intended by its author to make a full backup of the system partitions. This will not need copies of the originally installed Linux distribution media. See appendix RECOVERY for details of disaster precautions and recovery. Alternatively you may choose to use a disaster recovery system like mkCDrec ( http://mkcdrec.ota.be/project/index.html ) which tries to save you as quick as possible from the consequences of big accidents. Nevertheless, if you cannot get the old hardware running again it might be impossible to use system backups of any kind flatly. Then you have to install a completely new operating system and may use the old system backup only for as source of configuration files. I myself never lost a complete system but had occasions when an older reference copy of the system files came in very handy. User data which rely on exact file attributes or which are likely to change during the runtime of the backup should not be stored as ISO 9660 if mkisofs or genisoimage is used as ISO formatter program. Modern versions of program xorriso can record all file attributes including ACLs but are still more prone to file size change mishaps than sequential archive formats. So for at least for agile data, one should better use scdbackup_afio . Usually there are several kinds of data on a Linux disk, which should be backuped: Persistent system objects, often with special attributes and permissions Installed application packages (often below /opt) User created or collected data (often below /home) Other kinds of data do not need backup or would even make problems : Volatile system objects (e.g. /proc) Temporary files (e.g. /tmp or $HOME/.netscape/cache) Removable filesystems (/mnt /floppy /cdrom) During installation, you will be asked for "Exclusion list of directories for system backup". This list should contain all directories under / , which do not contain persistent system data. Example (have a look in your own / directory for more candidates) : /opt /home /mnt /floppy /cdrom /cdrw /tmp /proc /sys All other subdirectories of the root directory will be covered by the system backup. Among the excluded directories, only /opt and /home need backup. If there is only one real user who has not got a giant amount of data, a combined backup of /opt and /home is appropriate. Since my Netscape cache is quite large, i exclude it. scdbackup_afio /home /opt -not /home/thomas/.netscape/cache If there are very large amounts of data, it will be convenient to have separate backups. These backups can be run with different frequency. /opt will not need to be backuped as often as /home/projects . scdbackup_afio /opt scdbackup_afio /home -not /home/projects /home/thomas/.netscape/cache scdbackup /home/projects Real users (like "thomas", "guest1", "guest2") should do separate backups with scdbackup_home or scdbackup_afio $HOME Exclude their $HOME directories from the general data backup scdbackup_afio /home \ -not /home/thomas /home/guest1 /home/guest2 Next you should make a time schedule, when to do which backup. It is generally a good idea to have at least three older backups. From time to time a permanent backup could be made on CD-R in order to be kept forever. On the other hand ... be aware what traces you leave for future archeologists. Permission settings after an installation will not allow others to use a scdbackup installation, unless these restrictions are lifted by ./SET_PERMISSIONS (see below "Access Permissions") or unless laxer settings have been fetched during ./FETCH_CONFIGURATION. Interested and authorized users then go to the installation directory and perform ./ADD_USER . Now go and -as superuser- begin with step 1 of the installation procedure. If the superuser did not use scdbackup before , execute as step 1b : ./SUPERUSER_FETCH_CONF /home/thomas/scdbackup/scdbackup-0.9.2 to get the previously made settings of user "thomas" as templates for own ./CONFIGURE_CD resp ./CONFIGURE_DVD . Else ./FETCH_CONFIGURATION is ok. In any case : check what you confirm with ./CONFIGURE_CD and ./CONFIGURE_DVD . Take care not to allow others to access your eventual disk buffer objects. Best is to have none, if one can live without them. ----------------------------------------------------------------------- Usage scdbackup offers different formats for the data on media. The data format preferable for user data files is in most cases ISO 9660. The media are mountable and the files may be used directly by any program that can handle files which offer no write access. This format is also most likely to be accessible on other operating systems. In the most simple case, command scdbackup_home is sufficient. For arbitrary data locations use command scdbackup . As said, this is ok for directories trees and data files of a single user, where write permissions are general and simple. mkisofs with version numbers below 2.0 will not reliably cope with symbolic links. Independently of eventual weaknesses of the ISO formatter program there may also arise problems with the filesystem drivers. Not all of them show the w-permissions of directories correctly. Another disadvantage is that scdbackup does not offer data compression together with ISO format. For such purposes, scdbackup offers the format of the reliable archiver afio or the format of Joerg Schilling's backup program star. The commands scdbackup_afio and scdbackup_sys produce these formats. Backup Commands The backup creating command scripts of the scdbackup-system all accept the options described by cd_backup_planer -help | less but usually it is only necessary to provide some directory or file names. See also doc/examples.html for commented usage examples. ISO 9660 Filesystems These filesystems can be mounted on devices like /dev/cdrom or /dev/dvd . They also can be exchanged with other systems like Windows or Mac. scdbackup_home a command that backups the user's $HOME directory on CD except the Netscape cache. No arguments needed. sdvdbackup_home DVD version of above. scdbackup a command that backups the files and directories given as arguments by the user. These objects may be renamed by giving the target name before '=' and the source name. Example: rename /usr/home (on disk) to /home2 (on CD) /home2=/usr/home Subdirecories and files may be excluded by writing them behind option -not. Example: exclude three directories -not /home/guest /home/kindergarden /usr/home/guest Complete example: scdbackup /home /home2=/usr/home -not /home/guest Backup /home and /usr/home . But use /home2 on CD as name for /usr/home and omit subdir /home/guest sdvdbackup DVD version of above. Directory access permissions may be restored by the compressed shell script /added_by_scdbackup/DIR_PERMS.sh.gz on the last volume of such a backup. (See also examples.html#restore_incr ) This may be necessary because directory access permissions on a mounted CD may not be the same as they were during the backup. afio Archives afio archives can be read with the CD-ROM or DVD drive while the media is inserted but not mounted. Check Archive: afio -tv /dev/cdrom Re-install (*beware*, only if you need): cd / afio -ivZ /dev/cdrom Example : Extract X11 configuration files to a harmless place No need to be superuser for that ! mkdir $HOME/system_reference_copy cd $HOME/system_reference_copy afio -ivZ -y 'etc/X11/XF86Config*' /dev/cdrom Now you may compare them with those of your current system diff etc/X11/XF86Config /etc/X11/XF86Config | less Read for details : man afio . By help of script star_as_afio_wrapper one may alternatively create archives in the format of program star, provided that program is installed. For an example see appendix WORKERS. scdbackup_afio like scdbackup but with afio archives rather than ISO 9660 file systems. If there is a buffer file or if compression was set during configuration then the archives contain compressed files. scdbackup_afio /home -not /home/guest sdvdbackup_afio DVD version of above. scdbackup_sys a command that backups the root directory as afio archives. Some well known non-system directories and some trash directories are excluded. (My apologies to all those who work hard on programs that fill in the "trash" :) Arguments can be given to include or exclude other directories. The exclusion list is set by CONFIGURE . Example: /cdrom /cdrw /floppy /home /mnt /opt /proc /tmp Example (to be executed as superuser): include /opt (which usually is excluded) but omit /home2 and /usr/trash scdbackup_sys /opt -not /home2 /usr/trash sdvdbackup_sys DVD version of above. If the size of the resulting afio archives exceeds 650 MB then they are split into pieces of 650 MB and stored on several CDs which form a multi volume archive. Such archives have to be read from their first CD up to the one which contains the desired data. If a CD is missing then all following ones may be unreadable. DVD split size is 4480 MB. The main advantage of compressed multi volume archives is their most effective use of CD space. Single volume compressed archives usually need quite a lot of spare space. The ability to create multi volume archives allows you to risc larger single volume ones. Worst case is an additional CD with a few MB on it. (You may adjust the input size by -max_size number ) It is convenient to have a copy of the program bin/raedchen (e.g. on floppy) to be able to concatenate the parts of such an archive. Check: raedchen -quiet -pipe /dev/cdrom 650m "cat" \ -post_pipe_command "eject /dev/cdrom" | afio -tv - Restore: raedchen -quiet -pipe /dev/cdrom 650m "cat" \ -post_pipe_command "eject /dev/cdrom" | afio -ivZ - This is necessary because the CD returns some more bytes than have been written. The false extra bytes spoil at least the file which is stored at the end of the first and the beginning of the second CD. With DVD use : raedchen -quiet -pipe /dev/dvd 4480m "cat" \ -post_pipe_command "eject /dev/dvd" | afio ... If you are completely without scdbackup, then you will have to use a shell loop for concatenation. Like this ( 332800 = 650 MB / 2 kB ) : ( while test -n 1 do dd if=/dev/cdrom bs=2048 count=332800 || exit 0 eject /dev/cdrom echo 'Next media please' >&2 read dummy done ) | afio ... There is an appropriate option to make the verify commands handle such overflown media. (see below) The actual CD overflow size for afio is set by configuration file scdbackup_media_cap_value or shell variable SCDBACKUP_MEDIA_CAP . For DVD it is set in sdvdbackup_media_cap_value resp. SDVDBACKUP_MEDIA_CAP. If you change the defaults 650m / 4480m take care to record the new size on paper or on the raedchen floppy. Restartability All the above backup commands can be executed with first argument -resume and an optional number as second argument. They then restart an existing backup script at the backup piece with the given number. A dash (-) preceding the number will cause only this particular volume to be created and the script to end afterwards. If the number is missing (or the second argument is "AUTO") then the script resumes at the piece depicted by the status file ({skriptname}.pieceno). scdbackup , scdbackup_sys and scdbackup_afio additionaly recognize these options if one of them is given as first argument : -prepare_only prepares for -resume but does not create CDs -last_volume_count prints number of volumes of most recent backup See script outer_loop for an example how to use them. Verifying CDs and DVDs First of all : the verification command described here does *not* ensure that the backup contains what you *want* to backup. Wether you and scdbackup did it right has to be proven by looking at the meaning of the content i.e. whether the byte size is plausible and whether expected filenames are on CD (locatable by ASKME). This verification nevertheless ensures that the data stream has been transfered to the CD successfully and can be read without alteration. (There is only a very very remote chance of 1 to 3e38 that an alteration will be unnoticed.) Therefore it checks the CD recorder, the CD media and the CD reading device. In order to be verificable, a CD must have been created by version 0.8 -or more modern- of scdbackup with standard content of file scdbackup_make_checksum_value . With these settings the commands scdbackup, scdbackup_home, scdbackup_sys and scdbackup_afio record the length and a 128 Bit MD5 checksum of every backup volume that is written. This record is stored in a recordfile on your disk and may be used later to checkread the CD. Beginning with version 0.8.3 the record is also stored as a checksum tag at the end of the volume itself. scdbackup_verify Reads raw data from a CD and checks whether these are the same as the ones handed over to cdrecord. It reads CD data from /dev/cdrom and compares them with the most recent volume record. scdbackup_verify -auto tries to gather records from any source and checks whether one of them matches or obviously indicates a mismatch. Other data sources (CD devices) or older records may be used by options to scdbackup_verify. E.g.: scdbackup_verify /dev/cdrom -auto For backups made with scdbackup 0.8 to 0.8.2 you need to keep the record list. Backup it. See output of scdbackup_verify -help resp. file scdbackup_verify_help for a description of options, the recordfile's name and more examples. The command reads the whole CD and reports its result by a text on stderr as well as by its exit value: * "OK ..." and an exit value of 0 if the verification process was successful. * Various complaints in CAPITAL LETTERS and exit value 1 if verification fails (i.e. damaged or confused CD). sdvdbackup_verify DVD version of above. Uses a different recordfile ! The record list usually is stored in file logs/backup_log resp. logs/backup_dvd_log below scdbackup's installation directory. With normal installation settings you can only read that list if you are the same user as the one who wrote the backup. The list's address may be permanently changed in files scdbackup_checksum_list_value resp. sdvdbackup_checksum_list_value and temporarily overriden by exporting shell variable SCDBACKUP_CHECKSUM_LIST resp. SDVDBACKUP_CHECKSUM_LIST . Checksum computation and checksum verification are based on cd_backup_planer options -filter_md5 , -compare_checksum and -search_md5 . The format of checksum records and tags is described in appendix VERIFY below. For multi volume afio archives (see above) you will have to give scdbackup_verify the media capacity as it was set when the backup was written. Like : scdbackup_verify /dev/cdrom -auto - - 650m sdvdbackup_verify /dev/dvd -auto - - 4480m If verification fails directly after recording then you may redo the affected volume by a -resume run with the appropriate volume number. Like sdvdbackup -resume 3 One should verify stored backups in regular intervals to get aware of damage or deterioration. scdbackup_verify -auto If verification fails with such a check then you should consider to replace all volumes of the whole backup by a new backup. If this is not possible then you may make several tries to copy the media image to a reliable disk and check it there in the hope that the copy attempt delivered the correct image by incident (this does happen with small damages). dd if=/dev/cdrom of=/tmp/hopefully_sound_image_file bs=2048 scdbackup_verify /tmp/hopefully_sound_image_file -auto If successful, copy the image to new media: cat /tmp/hopefully_sound_image_file | scdbackup -pipe_to_media With DVD one should eventuellay truncate the media image to the known size plus three extra MB for eventual checksum tags or blocklists. Like: dd if=/dev/cdrom of=/tmp/hopefully_sound_image_file bs=1M count=1718 after sdvdbackup_verify /dev/cdrom -auto_end reported a match or mismatch after "1715m bytes". If the payload size is not known from the previously failed attempts to verify the media, then one should curb the output: dd if=/tmp/hopefully_sound_image_file bs=1M count=1718 | \ sdvdbackup -pipe_to_media Be aware that some old filesystems may be unable to take a file of 4200 MB. If you expect your backups to be not reproducible during the intended storage period (i.e. if they are an outsourced data archive rather than a backup for a life disk), then you should consider to record them with block checksum lists and as several identical copies. See appendix REDUNDANCY. Blanking CD-RW or DVD-RW , Erasing data from media If a CD-RW has been used and shall be re-used, it is necessary to blank it. The same is true with DVD-RW under certain circumstances. If blanking comes into effect it damages the data on the media, of course. If automatic blanking is configured as "1" (or "y") then with each backup the media get treated according to their automatically recognized type and state. Usually classification and recognition of media types is reliable. If nevertheless the type cannot be recognized or if it is not suitable for the backup command, then the backup aborts. It may be resumed after the media is brought to writeable state manually and automatic blanking is temporarily disabled: export SCDBACKUP_BLANKEN=0 scdbackup -resume Not only in case of emergency but also with recognizable media it may be worthwile to do blanking separately from backup writing. Depending on the blank options, on type of the burner, and on the writer software blanking lasts some time. So it is convenient to have some writeable media ready. Also one might sometimes want to apply unusual blanking options. With DVD+RW there is no blanking required for making media writeable again. Nevertheless, the "shred" option can be very helpful to ensure your privacy when reusing media. DVD-RW media for use with programs growisofs or cdrskin should be blanked "force" once to bring them into a mode with the misleading name "Restricted Overwrite". After that, growisofs and cdrskin will behave with them like with DVD+RW. DVD-RW for use with cdrecord need to be blanked before re-use just like CD-RW. The same is true for growisofs DVD-RW media which were written unformatted from unused state or have been de-formatted to mode "Sequential". "shred" is a blanking option which does not necessarily make media re-usable. It rather causes the media to be filled with random data. After that, the previously stored data are unreadable by common means. scdbackup_blank asks you to insert the CD-RW and press 'Enter'. If an argument is given, this is used as blanking option. Usual options are "fast" or "all". See output of option blank=help of the burn program (cdrecord, wodim, or cdrskin). Alternatively, there is option "shred" (see above). Without argument blanking is set to the option in file scdbackup_blanken_value. "1" or "y" mean option "fast". If no other clue is found option "fast" will be used. Note: If a blanking directive blank=... is part of file scdbackup_cdrecord_value then no other blanking option will work. sdvdbackup_blank DVD version of above. Its behavior depends on the media type and on the writer program (growisofs or cdrskin vs. cdrecord or wodim). For DVD+RW only option "shred" does anything senseful. For cdrecord, wodim and DVD-RW : see scdbackup_blank For growisofs, cdrskin and DVD-RW : Options "all", "full" and "sequential" lead to mode "Sequential". All other options lead to convenient mode "Restricted Overwrite". Both commands try to find out the type of the media in the drive. This depends much on output of programs dvd+rw-mediainfo, cdrecord, wodim or cdrskin which is not explicitely mentioned in their documentation. If this classification fails to yield a usable media type, the user is asked to confirm one of the types "CD-RW", "DVD-RW", "DVD+RW". The classification attempt as well as the eventual user inquiry can be avoided by the third argument to the program. Like : sdvdbackup_blank all - DVD-RW CAUTION : Best is not to use these commands with one-time writable media ! E.g. if a DVD-R is not recognized properly or declared by the user to be a DVD+RW, then option "shred" will use it up. After a small backup on DVD it is easy to read old remains from larger backups until they really get overwritten. Therefore before re-use of confidential media apply sdvdbackup_blank shred . Do it multiple times if privacy is a big issue. The "shred" option tells what will happen next and require another Enter keypress as confirmation. Then it eventually applies appropriate blanking. It writes the amount of random data given by the second argument to the program. E.g.: sdvdbackup_blank shred 4.7e9 If that second argument is missing then the configuration settings in file scdbackup_media_cap_value resp. file sdvdbackup_media_cap_value apply. Usually the preset size of 4480m bytes should be sufficient to hit all databytes on a DVD. Nevertheless you may derive a size from the maximum size given by growisofs during write. Subtract 200 (e.g.: shred 4700372792 ). Remains of backups made with version 0.8.3 or newer will get reported by command scdbackup_verify -scan_print . Like : - 1_1 A40907.115444 4699824167 7ec3be0efef3f27dd7ec90216d2ee3b4 which tells that backup data may be stored up to byte 4699824167. Blanking option "shred" produces a checksum record with name "shredded" and therefore together with the appropriate s*backup_verify command is suitable as a quality check for the media. A large organisation, which presumes to have many enemies, prescribes additional runs with a character and its complement. It also demands verification of the random write result. The said organisation does not allow any erasing of "TOP SECRET" media but demands their complete physical destruction. Although i am in doubt whether the complement achieves the desired effect with CD/DVD media, there are two blanking options "character" and "complement" which perform the prescribed runs. A compliant procedure would be : export SCDBACKUP_BLANK_CHARACTER="U" sdvdbackup_blank character 4.7e9 sdvdbackup_blank complement 4.7e9 sdvdbackup_blank shred 4.7e9 sdvdbackup_verify The character written is a capital "U" (= 01010101). By variable SCDBACKUP_BLANK_CHARACTER a different normal letter or digit may be chosen. Better do not use any special characters. Incremental Backups With larger amounts of data it becomes annoying or even impossible to backup the old data at each new run. To avoid this (within certain limitations) one may use an incremental backup scheme. Such a backup consists of a base backup which might be quite old but contains a full snapshot of all files without regarding their age. On top of the base backup there are update levels which provide more and more short-term updates. This saves a lot of time and CDs but provides similar security as a set of full backups. Invest some of the savings into increased backup frequency and also keep outdated backup CDs a bit longer. This is to compensate the media redundancy which is reduced with incremental backups. It has to be emphasized that correct restoring of a multi-level incremental backup is much more complicated than with a full backup. Consider your potential future situation, when you might really need the full content of your backup. Weigh whether avoiding the repeated workload of a full backup really justifies the additional workload in the future crisis situation. Begin with incremental backups if you have to leave out full backups because you cannot afford them. Initially a backup configuration needs to be created. It records all parameters of the backup, keeps track of the times and states of the backup area when backups have been made. Also the creation process starts the initial base backup which is called -level 0 . Choose a configuration name that can be used to create a new directory. You will need some disk space there for the state records (usually below 1% of the backup area size). Let us assume you choose $HOME/my_backup : $ scdbackup ...the usual arguments for your backup ... \ -conf_dir $HOME/my_backup -level -create_configuration After that creation, only the name of the configuration and the desired update level have to be given with the scdbackup command. Next day one could execute this command to get a backup of the files that are new or changed since the -level 0 backup : $ scdbackup -conf_dir $HOME/my_backup -level 1 Options -conf_dir and -level are explained in output of cd_backup_planer -help . There also is an example of a scheme. Several independant configurations may be defined. Distinct configurations do not interfere even if they share data sources. Not only scdbackup but also scdbackup_home and scdbackup_afio may be used for defining and running incremental backups. The state records in the configuration directory consume some disk space although they get compressed by gzip. You should watch the size of file $HOME/my_backup/level_0/content_file_list.gz during the -level 0 run and abort it if you cannot afford the space needed. While watching the size do not get worried by the output delay of the gzip pipe. Any substantial size change will get visible in time. If you cannot afford the disk space, you may disable state recording by using option -disable_safe_leveling together with -level -create_configuration. But be aware that file movements and deletions cannot be detected that way and will therefore not be included in the higher backup levels. Such a backup is still a good source for recovering from limited data accidents but it may be unable to accurately reconstruct the most recent state of your backup area. The default settings of scdbackup configure a method which depends on the traditional behavior of inode numbers (POSIX "file serial numbers"). If your filesystem turns out to be non-compliant with tradition then you will get suspiciously large backups at higher levels. Please inform me about the type of the affected filesystem and refer to appendix INODE for usage of the old checksum based method. If you ever need to re-install an incremental backup with multiple levels you will begin with copying level 0 first (e.g. by use of cp -dR ). Then you will remove the files which had vanished when level 1 was made. Then you will copy level 1. This is repeated with all higher level backups. For the removal job, there is a compressed shell script /added_by_scdbackup/VANISHED.sh.gz on the last volume of each backup level. Don't run it for fun ! (It is interesting to have a look at its content, nevertheless.) See examples.html for an example restore procedure. More detail information about state recording can be found in the output of cd_backup_planer -help . Look for "re-install an incremental backup" and also refer to the program options mentioned there. The decision whether to put a file object into an incremental backup depends on the following properties : mtime (display by ls -l), ctime (by ls -lc), eventually file content checksum (if enabled and some ancestor directories have young time stamps), eventually the inode and device numbers (if enabled), an economy consideration which eventually includes whole directories if there are many small files to be included. Backup Information Scripts Each backup CD contains a shell script named ASKME which can tell on which one a given fileaddress may have ended up in the backup. Since it only knows the fileaddresses given to the formatter program, this script cannot always tell for sure whether a file was actually present at the time of the backup. But it can tell the CD where one has to look. ASKME only needs an bash or ksh shell but no installation of scdbackup. See text askme_help for usage information. In case of incremental backups the information scripts of all older backup levels are also present on each CD. Because they can become very large and hardly usable, they are by default truncated to stubs which can just identify backup date, level and volume. To be usable for searching, the backup configuration has to be created with option -info_script -full or all level runs have to be done using this option. For incremental backups with default stubs, refer to scdbackup_askme in the next paragraph. With ISO 9660 formatted backups, the script is located in the root directory of the backup volume. With archive formatted backups (afio, star) it is the first item of a backup volume. Its name and address vary with the installation settings at backup time. One may obtain the address within the afio archive by afio -t /dev/cdrom 2>/dev/null | head -1 which should reply something like u/thomas/scdbackup/scdbackup-0.9.2/tmp/ASKME_1000.1.z (omit 2>/dev/null in insatisfying reply.) The script should run on nearly any shell that is around but it has the disadvantage to be relatively slow. With multilevel backups of large disks one may have to wait minutes for an answer. Backup Information+Restore Tool Command scdbackup_askme provides a binary program which can read the ASKME scripts and perform the searches much faster. Also it can search for text pieces (like fgrep) or for regular expressions (like grep). The main advantage of scdbackup_askme, nevertheless, is its ability to read content lists and vanish lists from incremental configuration directories or from the last volume of each backup level. Beginning with scdbackup version 0.8.7, non-incremental ISO 9660 backups too have a content list on the lasti volume. These lists hold more detailed information about the backup content than the ASKME scripts. They are able to surely tell whether a particular file was backuped (whereas ASKME can often only tell about a directory above the file's address). The content lists are able to tell the address of a file on the backup media (whereas ASKME can only tell the original address on disk). Based on content lists it is possible to have scdbackup_askme perform restore operations from ISO 9660 backup volumes. Both commands scdbackup_askme and sdvdbackup_askme may be pre-configured by startup files. See below "Startup Files for Information+Restore Tool". Examples (see text doc/scdbackup_askme_help for options and more information) Using the level ASKME scripts from backup media (ISO format): scdbackup_askme /cdrom/ASKME -levelall -search_grep '/trees/.*oak' scdbackup_askme /cdrom/ASKME -levelall -search_sh '/home/*/trees/*oak*' Using an incremental backup configuration on disk scdbackup_askme $HOME/my_backup -levelall -dialog After the backup state is read, the user will get prompted for options or search texts. Caused by option -search_sh those search texts will be compared as shell parser patterns against the source addresses. In this mode there is a current working directory which is addressed by search texts which do not begin with '/'. The special file names '.' and '..' are recognized. -pwd -cd:/home/*/trees *oak* */*oak* ../*oak* -search_shpat *trees/*oak* -search_shname *oak* -page:20 *oak* -search_shpat makes '/' match *,? or []. This allows to freely search for file name snippets. -search_shname looks into any directory and compares the single file names with the search text. Like this shell command: find / -name '*oak*' -page:20 causes output to pause after each 20 lines. The user gets asked for pressing Enter. Input @ suppresses this paging temporarily. input @@ suppresses output, Input @@@ aborts. Any other input is taken as option or search text and aborts the current search. Independend of the search mode there are options -ls and -ls_d which allow a shell pattern search with abbreviated output. They are similar to shell commands ls and ls -d : -ls:.. -cd:.. -ls_d:*tree* All searches may be performed on the target addresses, too. -search_target For the following examples switch back to shell parser search mode -search_sh To see the current settings -status To see all possible settings -status:long To see the help text -help To restore single files or whole directory trees from ISO 9660 backups, restoring first has to enabled by setting a -restore_mode . This one protects existing files on the disk and allows to continue trying after restore problems -restore_mode:overwrite_off:on_error_ignore The program further needs to know from where to read the data and it is convenient to have it operate the according drive via programs eject and mount -restore_from:/cdrom -media_mode:load_eject:mount_umount The original data are protected. So, in order to achieve a full restore result, the restore gets directed to a copy location -restore_skip:/home/thomas -restore_to:/home/test/thomas Now directory tree /home/thomas/gifs gets restored to that copy location -restore:/home/thomas/gifs The user will get prompted for the needed media. It is possible to skip some if they are missing, but of course that will result in an incomplete restore. The result will be a directory tree /home/test/thomas/gifs . Above operations may as well be performed using incremental information from backup media (ISO format). They are stored on the last volume of each level. scdbackup_askme /cdrom/added_by_scdbackup -levelall \ -media_mode:load_eject:mount_unount -dialog The user will get prompted for those last volumes. User input "@@@" confirmed by user input "y" indicates that no more levels are available. Then dialog starts as with the previous example. Restore runs which have been aborted for some reason may be resumed at the last restore position by: scdbackup_askme -resume One may also depict an explicit start level scdbackup_askme -resume:1 and additionally an explicit start volume scdbackup_askme -resume:1:4 Option -resume does also work in dialog but only for restore operations which have been performed within the same dialog session, or for those which were skillfully prepared by options -resume_last_* and -resume_opt_text. Aborted -resume runs may also be resumed by -resume. If the abort left a file object on disk which is neither in its pre-restore state nor restored correctly, then option -resume refuses work and demands the affected fileobject to be handled first. In dialog mode one may enter "go on" after the file is checked, and eventually repaired or deleted. In non-dialog mode the program aborts until the offending command -resume_last_dirty in file $(scdbackup -where tmp)/askme_resume_state_"$UID"_dirty is removed or set to an empty file address. File objects recorded as "dirty" are questionable in any way. The directories in the path of such a file may been left with too generous permissions for the owner or too sparse permissions for anybody. One should check and correct. (Owners usually have rwx on their directories so that is rarely a problem.) scdbackup_askme has the appeal of a command line shell but it should be not too hard to create a graphical frontend for it. The file doc/saskme_frontend.c contains example code covering the task of forking scdbackup_askme and to perform dialog with this forked process. (No GUI code included.) Access Permissions Initially only the installing user and the superuser have permission to run backups by help of a scdbackup installation. In hostile or even just weakly controlled environments one should unconditionally stay with these settings and eventually install independend copies of scdbackup for additional users. In the case of a strictly personal computer it may seem tolerable to have the superuser using the scdbackup installation of the desktop user. But one should be aware, that entertainment components of web browsers and other programs impose a risk that the desktop user might get overtaken by intruders. Nevertheless it may be desirable to have a single scdbackup installation which can be shared by several users. In this case the superuser should install but *absolutely not use* that installation. In case of permission scheme "group_only", the superuser has to set the desired group membership of the installation (e.g. by chgrp -R ). The script ./SET_PERMISSIONS allows the installing user to loosen the access restrictions. The script expects as only argument one of the following permission scheme names : owner_only this is the default described above group_only grants group members the same rights as the installing user anybody allows any user to run backups but tries to keep them from spoiling the configuration. Not really safe. custom prepared name for a custom permission scheme: edit SET_PERMISSIONS and search "Change this according to" unprotected as the name says : totally unprotected and totally unsafe Those settings are permanent and get restored whenever ./CONFIGURE_CD or ./CONFIGURE_DVD are run. Restore the most secure settings by entering the scdbackup installation directory and executing : ./SET_PERMISSIONS owner_only ----------------------------------------------------------------------------- Inner details - Appendice See table of content at the beginning ot this text. ----------------------------------------------------------------------------- Appendix DIRECTORIES : The components of the scdbackup system are stored in several sub directories of the installation directory. Especially those files which need to be written during daily operation of scdbackup or during configuration are separated from those files which usually need not to be changed. After unpacking the installation directory scdbackup-0.9.2 contains these data files : TIMESTAMP the timestamp of the tar archive cd_get_variables reads the configuration for CD dvd_get_variables reads the configuration for DVD scdbackup_switcher dispatcher script for user commands and language choice A few symbolic links refer to files in the the ./doc directory : COPYING LIESMICH README These directories are created below scdbackup-0.9.2 : aux a few parts which do not fit into any other category bin binary programs, resulting from C sources cmd user accessible commands conf configuration files and custom intermediate scripts doc documentation inst configuration scripts logs list files with checksum records scripts worker scripts of the scdbackup system (for bash or ksh) src C sources which lead to the programs in ./bin tmp files created during daily usage For using scdbackup at least the directories ./logs and ./tmp have to provide write permissions. There are configuration files which allow to redirect them to other addresses outside of scdbackup's installation directory. Therefore it is generally advised not to address subdirectories of scdbackup directly but to determine their actual address by $(scdbackup -where xyz) . See also the examples throughout this text. Older applications which directly use inner parts of scdbackup will not find those programs and scripts any more. Applications which restricted themselves to commands scdbackup* and sdvdbackup* and did not access configuration files directly are not affected. To ease the transition of affected applications there is a script ./COMPATIBILITY_LINKS in directory ./inst which creates a lot of symbolic links within directory scdbackup-0.9.2 . This makes available all programs which in older documentation have been advertised for use. Eventually missing links may be created by hand like this : cd $(scdbackup -where) ln -s $(scdbackup -where scripts)/cd_afio_command cd_afio_command Nevertheless i advise you to migrate eventual applications to the new directory structure. For mere safety reasons, a normal user's scdbackup installation should *not* be FHS compliant. FHS mainly addresses system-global software. To get a scdbackup installation of the superuser conformant to FHS, you would have to scatter it over your system so it matches the following settings Configuration file : Content : $HOME/.scdbackup_dir /opt/scdbackup-0.9.2 /opt/scdbackup-0.9.2/scdbackup_conf_dir /etc/opt/scdbackup-0.9.2 /etc/opt/scdbackup-0.9.2/scdbackup_tmp_dir_value /var/opt/scdbackup-0.9.2/tmp /etc/opt/scdbackup-0.9.2/scdbackup_logs_dir_value /var/opt/scdbackup-0.9.2/logs See below appendix CONFIGURATION for the meaning of these files. In order to obey the FHS imperative "Programs to be invoked by users must be located in the directory /opt//bin" one may also move the content of the ./cmd directory to ./bin . ----------------------------------------------------------------------------- Appendix CONFIGURATION : These three configuration files are used by both CD and DVD subsystem : $HOME/.scdbackup_dir the address of the scdbackup-directory (such a file is created by ./ADD_USER) May be overridden by shell variable SCDBACKUP_DIR $HOME/.scdbackup_lang the language to be used in dialog. Currently only en (english) and de (german) are supported. Any other content will default to en . May be overridden by shell variable SCDBACKUP_LANG $HOME/.scdbackup_userid only needed if shell variable EUID is not set automatically by the shell. This file may contain a unique decimal number to distinguish this user from any others of the system. The configuration scripts will prompt for a number and create such a file if they detect $EUID empty. Override this file or EUID by SCDBACKUP_USERID . (Note that this does not give you any other user's permissions or privileges.) The following configuration files get created by ./CONFIGURE_CD in directory ./conf (DVD configuration files are listed further below) This address may be overridden by the content of file $(scdbackup -where)/scdbackup_conf_dir resp. variable SCDBACKUP_CONF_DIR. Its actual address may be queried by scdbackup -where conf Each of the files *_value may be overridden by a shell variable. This variable has the same name as the filename's beginning but is written in capital letters. But note that an empty variable does not override anything. Example: Override files scdbackup_scsi_adr_value , scdbackup_speed_value and scdbackup_buffer_value to use the cruelly fast IDE writer which chokes my system if i use a file buffer at speed 12 : export SCDBACKUP_SCSI_ADR="1,0,0" export SCDBACKUP_SPEED="12" export SCDBACKUP_BUFFER="-" scdbackup_scsi_adr_value the address of the recorder (e.g. 3,0 , ATA:1,0,0 , /dev/hdc) scdbackup_speed_value the recording speed (e.g. 12) scdbackup_blanken_value whether to blank the CD before writing: "0"=no , "1"=yes (mode will be "fast") or the blank= option as for cdrecord "all" , "fast" or to perform blanking during the write run rather than in a separate blank run : "inline_all" , "inline_fast" The text may contain all three arguments for scdbackup_blank. Like: all - CD-RW scdbackup_use_prog_value for CONFIGURE_CD to remember the answer to the question which creates the next four files: scdbackup_mkisofs_value name and options of the ISO 9660 formatter program (e.g. mkisofs -J -l -L -R) scdbackup_path_list_value eventually contains cd_backup_planer options for the use of the -path-list feature of ISO formatters. Among others there is the address of the temporary list file which will buffer the path-list. Note that the text "mkisofs_path_list" gets expanded to "mkisofs_path_list"_$EUID and that "=EUID=" gets converted to $EUID. (This does only apply to reading the file but not to evaluating SCDBACKUP_PATH_LIST) scdbackup_iso_filetypes_value eventually contains options for cd_backup_planer which may allow other file types than "data" and "dir" to be handed over to the ISO formatter. Older mkisofs aborts on symbolic links, though. scdbackup_cdrecord_value name and fixed options of the CD recording program (e.g. cdrecord -v ) scdbackup_cdrecord_opt_value additional options of the CD recording program to be used with runs that write data but not with the separate blanking run. (e.g. -fs=8m -eject) scdbackup_split_dir_value for CONFIGURE_CD to remember the answer to the question which creates the following two files : scdbackup_large_file_iso_value options for cd_backup_planer concerning handling of large files when planning an ISO format backup. Note that the text "=EUID=" gets converted to the userid number. (This does only apply to reading the file but not to $SCDBACKUP_LARGE_FILE_ISO) scdbackup_large_file_afio_value As scdbackup_large_file_iso_value with afio format. Does not apply to scdbackup_sys. scdbackup_afio_value name of a program call-compatible to afio plus eventual fixed options for writing. The program must recognize option "-Z" for compression and "-" as target name for stdout. It has to expect the addresses of input files at stdin. (see: man afio) Default value is : afio -o scdbackup_buffer_input_value for CONFIGURE_CD to remember the answer to the question which creates the following two files: scdbackup_buffer_value name of the buffer file or - (e.g. /buf/isofs) scdbackup_afio_compress_value whether to enable compression with afio : 0 = no, 1 = yes, - = compress if buffer file is used (default is - ) scdbackup_z_cap_value estimated capacity for compressed afio. If empty or 0 then cd_dir_afiosize is used. scdbackup_sys_excl_value contains a list of directories to be excluded from the system backup. scdbackup_t_lock_value whether to use the locking protocol: 1 = yes, store lockfiles in scbackup directory /path = yes, store lockfiles in path (e.g. /tmp ) anything else means no locking scdbackup_bin_sbin_root_value parent directory for installation of commands scdbackup_is_configured indicates by content "yes" that scdbackup was configured sucessfully for CD. Further configuration files are not adjustable by ./CONFIGURE_CD but may be changed or overridden to customize scdbackup's behavior : scdbackup_use_shell_value contains the path of a bash compatible interpreter. scdbackup_media_cap_value maximum capacity of media used with afio archives. If an afio archive exceeds this size then it is cut on the first CD and continued at a new one. (set initially to "650m" by CONFIGURE_CD) This size is also used for blank option "shred". scdbackup_max_size_value default volume size if option -max_size is omitted. (640m if file is missing and variable not set) scdbackup_blank_speed_value the speed for blanking CD-RWs (e.g. 4) in case it differs from recording speed. (This file is not created by ./CONFIGURE_CD and usually not needed. Before my CRW8424S deceased completely it was able to blank with speed 4 but to write only with 2.) scdbackup_make_checksum_value contains options for generating MD5 checksums. This file is initially created by CONFIGURE_CD but its content may be set to a single "-" to prevent checksum computation. scdbackup_checksum_list_value address of the file containing checksum records. (if missing : .../scdbackup-0.9.2/logs/backup_log ) scdbackup_verify_adr_value default for data_source with scdbackup_verify (if missing : /dev/cdrom ) scdbackup_keep_buffer_value whether to keep the eventual buffer file with ISO image or afio archive to survive until the next volume gets written. Content "1" means yes. scdbackup_options_with_iso_value additional options for cd_backup_planer which shall take effect with any backup that produces ISO 9660 file systems. scdbackup_options_with_afio_value additional options for cd_backup_planer which shall take effect with any backup that produces afio archives. The following configuration files get created by ./CONFIGURE_DVD in directory ./conf (CD configuration files are listed above) This address may be overridden by the content of file $(scdbackup -where)/scdbackup_conf_dir Like with CD configuration each of the files *_value may be overridden by a shell variable. E.g.: Override files sdvdbackup_dev_adr_value and sdvdbackup_speed_value : export SDVDBACKUP_SCSI_ADR="/dev/sr2" export SDVDBACKUP_SPEED="1" sdvdbackup_dev_adr_value the device address of the recorder (e.g. /dev/dvd) Eventually used with growisofs. sdvdbackup_scsi_adr_value the SCSI address of the recorder (e.g. 3,0) Eventually used with cdrecord, wodim, cdrskin. sdvdbackup_speed_value the recording speed (e.g. any, 1) sdvdbackup_blanken_value whether to blank the DVD before writing: "0"=no , "1"=yes (mode will be "fast") or the blank= option for sdvdbackup_blank : "all" , "fast" The text may contain all three arguments for scdbackup_blank. Like: all - DVD-RW sdvdbackup_use_prog_value for CONFIGURE_DVD to remember the answer to the question which creates the next two files: sdvdbackup_mkisofs_value name and options of the ISO 9660 formatter program. (e.g. mkisofs -J -l -L -R) sdvdbackup_path_list_value eventually contains cd_backup_planer options for the use of the -path-list feature of ISO formatters. Among others there is the address of the temporary list file which will buffer the path-list. Note that the text "mkisofs_path_list" gets expanded to "mkisofs_path_list"_$EUID and that "=EUID=" gets converted to $EUID. (This does only apply to reading the file but not to evaluating SCDBACKUP_PATH_LIST) sdvdbackup_iso_filetypes_value eventually contains options for cd_backup_planer which may allow other file types than "data" and "dir" to be handed over to the ISO formatter. Older mkisofs aborts on symbolic links, though. Default : -result_filetypes data:dir:arg_redirect sdvdbackup_cdrecord_value name and fixed options of the DVD recording program (e.g. /u/test/scdbackup-0.9.2/scripts/growisofs_wrapper ) sdvdbackup_cdrecord_opt_value additional options of the DVD recording program to be used with runs that write data. (e.g. -final_eject bufsize=64m) sdvdbackup_stream_rec_value additional option to control slow checkreading with DVD-RAM, BD-RE and formatted BD-R. Will only get into effect if the DVD recording program name begins by 'xorriso -as cdrecord' or 'cdrskin '. sdvdbackup_split_dir_value for CONFIGURE_DVD to remember the answer to the question which creates the following two files: sdvdbackup_large_file_iso_value options for cd_backup_planer concerning handling of large files when planning an ISO format backup. Note that the text "=EUID=" gets converted to the userid number. (This does only apply to reading the file but not to $SDVDBACKUP_LARGE_FILE_ISO) sdvdbackup_large_file_afio_value As sdvdbackup_large_file_iso_value with afio format. Does not apply to sdvdbackup_sys. sdvbackup_afio_value name of a program call-compatible to afio and eventual fixed options for writing. Default value is : afio -o (See above scdbackup_afio_value for more details) sdvdbackup_buffer_input_value for CONFIGURE_DVD to remember the answer to the question which creates the following two files : sdvdbackup_buffer_value name of the buffer file or - (e.g. /buf/isofs_dvd) sdvdbackup_afio_compress_value whether to enable compression with afio : 0 = no, 1 = yes, - = compress if buffer file is used (default is - ) sdvdbackup_z_cap_value estimated capacity for compressed afio. If empty or 0 then cd_dir_afiosize is used. sdvdbackup_sys_excl_value contains a list of directories to be excluded from the system backup. sdvdbackup_t_lock_value whether to use the locking protocol : 1 = yes, store lockfiles in scbackup directory /path = yes, store lockfiles in path (e.g. /tmp ) anything else means no locking sdvdbackup_bin_sbin_root_value parent directory for installation of commands sdvdbackup_is_configured indicates by content "yes" that scdbackup was configured sucessfully for DVD. Further configuration files are not adjustable by ./CONFIGURE_CD but may be changed or overridden to customize scdbackup's behavior : sdvdbackup_use_shell_value contains the path of a bash compatible interpreter. sdvdbackup_media_cap_value maximum capacity of media used with afio archives. If an afio archive exceeds this size then it is cut on the first DVD and continued at a new one. (set initially to "4480m" by CONFIGURE_DVD) This size is also used for erasing (blanking) DVD. sdvdbackup_max_size_value default volume size if option -max_size is omitted. (4250m if file is missing and variable not set) sdvdbackup_make_checksum_value contains options for generating MD5 checksums. This file is initially created by CONFIGURE_DVD but its content may be set to a single "-" or made empty to prevent checksum computation. sdvdbackup_checksum_list_value address of the file containing checksum records (default: .../scdbackup-0.9.2/logs/backup_dvd_log ) sdvdbackup_verify_adr_value default for device address with sdvdbackup_verify (if missing : sdvdbackup_dev_adr_value ) sdvdbackup_keep_buffer_value whether to keep the eventual buffer file with ISO image or afio archive to survive until the next volume gets written. Content "1" means yes. sdvdbackup_options_with_iso_value additional options for cd_backup_planer which shall take effect with any backup that produces ISO 9660 file systems. sdvdbackup_options_with_afio_value additional options for cd_backup_planer which shall take effect with any backup that produces afio archives. This file may be created by hand before running ./CONFIGURE_DVD : sdvdbackup_use_prodvd_value whether to configure for cdrecord (file content "1") or for growisofs (any other content). This file preserves the name of the permission scheme for ./SET_PERMISSIONS . It applies to both, CD and DVD permissions : scdbackup_set_permissions_value one of the names mentioned in paragraph "Access Permissions" above. The files to redirect the directories which need w-permission for daily usage apply to both CD and DVD : scdbackup_tmp_dir_value may contain the address of a directory which will be used instead of the ./tmp directory within scdbackup's installation directory. scdbackup_logs_dir_value may contain the address of a directory which will be used instead of the ./logs directory. Redirection of ./conf, ./tmp, or ./logs is only recommended for situations where scdbackup is installed on a read-only filesystem and the writeable directories need to be outsourced to a rw-filesystem. Other (expert) applications are: sharing logs between several scdbackup version installations of the same user, setting up an FHS compliant installation (FHS with scdbackup is safe only if no other than the superuser is allowed to use it !). One may view the effective configuration of a scdbackup environment by scdbackup -show_configuration | less sdvdbackup -show_configuration | less The burn scripts scripts/cd_backup_command , scripts/cd_afio_command , scripts/cd_pipe_command allow the use of three intermediate scripts or programs. If you provide executable code under one of the following file names within the ./conf directory, it will run at the given occasion. Each of the runs gets at least eight arguments : $1 Address of buffer file ("-" or "" if on-the-fly) $2 SCSI address used by cdrecord or device address used by growisofs $3 Caller ("iso9660" for scdbackup, "afio" for scdbackup_afio) $4 Volume number $5 Total number of volumes $6 scdbackup installation directory $7 cdrecord program used (with options like -v) (contains e.g. .../growisofs_wrapper if running for DVD) $8 formatter program used (with several options) These program runs are prepared : conf/volume_prompt_script before the user gets opportunity to press Enter. conf/volume_preburn_script after the eventual recorder lock is refreshed and before the eventual blank run. conf/volume_afterburn_script after the run of the writer program ended and before the eventual buffer file gets removed. This script gets additional arguments: $9 Address of the eventual checksum list ${10} Recordname of the record written to this list ${11} Timestamp of record ${12} Size ${13} Checksum (superstition brings bad luck. U.Eco) Make sure your shell can evaluate ${10} or use shift to access double digit parameters (after copying all single digit parameters needed). conf/volume_abortedburn_script runs instead of volume_afterburn_script if the writer program indicates failure and the whole backup run is to be aborted. There may also be scripts specialized for a particular media type. If both - the general script and the specialized script - are present, then the generalized script gets executed first and the specialized script gets excecuted after that. Both scripts get the same arguments. Specialized scripts for CD are : conf/volume_prompt_script_cd conf/volume_preburn_script_cd conf/volume_afterburn_script_cd conf/volume_abortedburn_script_cd For DVD : conf/volume_prompt_script_dvd conf/volume_preburn_script_dvd conf/volume_afterburn_script_dvd conf/volume_abortedburn_script_dvd Example : Put your backup into a set of hard disk files rather than onto a set of media. For that prepare a writer which directs the volume stream into a disk file : export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_data_to /home/archive_dir/next_volume" Create an afterburn script which renames the disk file to a filename like "A50922.132751_volume_1_7" : echo '#!/bin/sh' > $(scdbackup -where conf)/volume_afterburn_script echo 'mv /home/archive_dir/next_volume /home/archive_dir/"${11}"_volume_"$4"_"$5"' >> $(scdbackup -where conf)/volume_afterburn_script chmod u+x $(scdbackup -where conf)/volume_afterburn_script Now run your backup and have "yes" answer any media prompt yes | scdbackup_afio -max_size 1g /opt /home Do not forget to remove the afterburn script rm $(scdbackup -where conf)/volume_afterburn_script unset SCDBACKUP_CDRECORD Startup Files for Information+Restore Tool Commands scdbackup_askme and sdvdbackup_askme may execute the lines of a startup file as options before any arguments or dialog input lines are executed. The files must be readable but should not be executable. Each line may be any option which is allowed for the commands. Because the startup lines are read as first options they eventually will be in effect when the info source is read in. Empty lines and lines which begin with '#' are ignored. There is no way to set the info source within the startup file. It has always to be the first argument of the commands. conf/scdbackup_askme_rc is read by command scdbackup_askme conf/sdvdbackup_askme_rc is read by command sdvdbackup_askme ----------------------------------------------------------------------------- Appendix WORKERS : The commands use a program called bin/cd_backup_planer as central component. cd_backup_planer splits a list of directories into pieces which then each fit on CD-R or other media. It generates a script with backup commands. The lists of files or directories are determined according to maximum capacities in terms of total size and number of items. Oversized directories are split automatically. Oversized files are skipped unless a "Large file split directory" was defined during configuration. Also the program is able to generate a shell script that can print several informations about the backup. This script then is included in each piece of the backup. Read text doc/cd_backup_planer_help or execute $(scdbackup -where bin)/cd_backup_planer -help | less to get a description of the program and its options. To avoid problems while running mkisofs the program performs a collision test of target names and a readability test of source names. Collisions are circumvented by automatical renaming. Problematic filenames are stated explicitely. Unreadable files and directories may be excluded automatically. The command scripts (e.g. /usr/bin/scdbackup) merely determine the location of the scdbackup system and then start a master script there : scdbackup_switcher finds out what language to use in dialog. It reports the command name, the scdbackup version and the language code on stderr. Then it starts one of the worker scripts. If the only argument of the command is one of the following, scdbackup_switcher prints a reply to stdout and ends rather than starting the program : -version "scdbackup" versioncode timestamp -which the address of the effective worker script -where the address of the effective scdbackup installation directory. An optional second argument may request the actual address of a scdbackup subdirectory in case it is redirected from its usual place. Like : bin conf doc logs scripts tmp (see also its usage throughout this text) -show_configuration a list of variable names and contents. See appendix CONFIGURATION. -show_configuration_exported a list of export commands for the shell. Content is quoted. The scripts use two common shell code snippets to read the configuration settings : cd_get_variables reads the variables which where set with CONFIGURE_CD . (to be executed inline via dot) dvd_get_variables reads the variables which where set with CONFIGURE_DVD (and uses them to override CONFIGURE_CD's settings in cd_get_variables). The following files are stored in directory ./scripts . scdbackup_switcher gets ordered to start one of the following scripts : backup_iso_en plans and performs backups in ISO 9660 format. It is used by scdbackup, scdbackup_home, sdvdbackup, sdvdbackup_home backup_afio_en plans and performs backups in archive format. It is used by scdbackup_afio, scdbackup_sys, sdvdbackup_afio, sdvdbackup_sys blank_en blanks a single media and/or overwrites it with random bytes. (scdbackup_blank, sdvdbackup_blank) verify_en verifies a single media against recorded checksums. (scdbackup_verify, sdvdbackup_verify) askme_en starts the binary information program. (scdbackup_askme, sdvdbackup_askme) clear_en releases all possibly open locks of a scdbackup installation. show_confguration performs option -show_configuration (see above). Common code snippets which make the command scripts less redundant : handle_existing_script performs options like -resume and other activities which deal with the backup script that resulted from the previous run. handle_planer_failure is called if the run of cd_backup_planer fails. install_backup_script installs the new resulting backup script and makes it ready for execution. Size determination is crucial for planning the distribution of backup data. Besides the -internal size measure there may be used du -sk or scripts which simulate the intended backup format : cd_dir_isosize Deprecated, is able consume enormous virtual memory : can be used instead of du -sk to estimate more accurately the size which a directory will have in ISO 9660 filesystem. cd_dir_afiosize can be used to determine the size of a (possibly compressed) afio archive made from a particular directory. This is very slow since the archive is actually generated, counted and discarded. Several scripts scripts which refer to writing of media and pseudo-media : cd_backup_command is able to write a single piece to a CD or DVD writer as ISO 9660 file system. cd_afio_command is able to write a single piece as afio archive to a CD or DVD writer. If a buffer file is used then the archive contains compressed files (afio option Z). For CDs written on the fly this isn't suitable since the output rate of a compressor can be very low (i.e. the compression rate can be very high). cd_pipe_command directs stdin to a single CD or DVD without any further formatting. I.e. data already have to be formatted. This is used with scdbackup -pipe_to_media . If the first argument is -multi_volume then multi volume rules as with scdbackup_afio apply. Use this mode for piping large archives or dumps onto several media. cd_blank_command blanks a CD-RW using the configuration of scdbackup and eventually starts shredder_command. sdvd_blank_command warns of shredder_command and starts it for DVD. shredder_command overwrites a media with random data. afio_sencrypt a script which produces an encrypted and eventually gzip compressed afio archive. Encryption is done by sencrypt. See http://scdbackup.webframe.org/README_sencrypt See also appendix CRYPTO below. One of three modes is selectable by the first argument. Depending on the mode a second argument may be needed : -each_file_by keyphrase_file afio-traditional encoding of each single file yields a rugged archive but is quite slow. The key phrase has to be stored in a file and the second argument has to be its address. In case of compression the first 10 bytes of each file stay unencrypted to avoid attacks via known cleartext. Unpack (uncompressed) : afio -i -P 'sencrypt' -Q '-d' \ -Q '-f' -Q "$keyphrase_file" -Z /dev/cdrom Unpack (compressed) : afio -i -P 'sencrypt' -Q '-d' -Q '-g' \ -Q '-f' -Q "$keyphrase_file" -Z /dev/cdrom -whole_archive_by keyphrase_file encryption of the resulting archive as a whole. Again the key phrase has to be stored in a file. With uncompressed archives the first 322 bytes stay unencrypted. With compressed ones it is 162. Unpack (uncompressed) : sencrypt -d -s 322 -f "$keyphrase_file" \ &2 ; read dummy dd if=/dev/hdc bs=2048 count=2293760 ) | gunzip | star -t -v The number 2293760 is for DVD (4480 MB). With CD use 332800 (650 MB). With a single DVD life is simpler : gunzip Unpermute_max_blocksizE) return(0); /* The chain of n permuted blocks is divided into quarters */ /* 0 <= q1 <= q2 <= q3 <= n */ n= complete_blocks+(remainder>0); q2= floor(n/2.0); q1= floor(q2/2.0); q3= floor((q2+n)/2.0); /* To produce an unpermuted output stream, iterate over the number of data blocks */ for(x= 0.0; x0) to_read= remainder; else to_read= blocksize; ret= fseeko(fp_in,((off_t) blocksize)*((off_t) idx),SEEK_SET); if(ret<0) return(0); ret= fread(buffer,(size_t) to_read, (size_t) 1,fp_in); if(ret<=0) return(0); fwrite(buffer,(size_t) to_read, (size_t) 1,fp_out); if(ret<=0) return(0); } return(1); } Be aware that double mantissae are about 50 bit only. So this code is limited to 1 petablocks. I decided to use data type double for the index computation in order to achieve maximum C-language portability of that part. In case that 64 bit fseeko() and type off_t are not available for i/o addressing you may have to find own ways to exceed the 31 bit limit of fseek(). xorriso Checksums If program xorriso is used for ISO 9660 formatting, and if it is modern enough to offer option -md5, then it will generate own MD5 checksums. The superblock and the directory tree of the ISO image will get an MD5 sum each. Damages of those entities will be detected by xorriso when it performs -dev or -indev after -md5 "on". Each data file will get a MD5 sum which. After the directory tree has successfully been loaded they can be verified by xorriso option -check_md5. ----------------------------------------------------------------------- Appendix REDUNDANCY English language note : This text uses the terms "redundancy" and "redundant" for information objects which contain a certain degree of inner repetition. The other english language meaning of these terms (like: superfluous, not needed, void) is never intended throughout this text. Redundant Backup Concepts Producing Repairable Backup Volumes Recovering Damaged Repairable Backup Volumes Recovering Damaged Block Checksum Lists Permuted Volumes Repairable Backup Volumes for Long Term Backups Even if the backup media verify after writing, they are still prone to deterioration. If your backups are to be read after many years they might turn out to be damaged. The data of a completely unreadable backup media are lost, of course. But in the case of partially unreadable media you will wish to have means to evaluate the damage and to try a repair. For that, you will have to prepare already at backup time. The overhead will be substantial, so the precautions described here are not standard with scdbackup. Although scdbackup provides all necessary means to extract and recover data from redundant backups, it may be wise to print out above appendix VERIFY and store it together with the backup. It contains the definitions of the various data formats used by scdbackup's redundant backup facility. DISCLAIMER : If you have to rely heavily on repairability of backups, then you *MUST* test the whole processing chain described here whether it works for you. That means you have to produce a redundant backup, copy one of the media images to disk, cat /tmp/recovering_image damage it by help of a command like $(scdbackup -where bin)/cd_backup_planer \ -write_to_file /tmp/recovering_image 10m string "spoiling the content" and then try to recover it from your media by using the recovery tools of scdbackup as described below. For example by : $(scdbackup -where bin)/cd_backup_planer -process_blocklist \ mend /tmp/recovering_image 0 /dev/cdrom /tmp/recovering_image You need to learn this now, as long as the emergency situation is only virtual. A successful repair will be confirmed by : scdbackup_verify /tmp/recovering_image -auto Redundant Backup Concepts A redundant backup consists of several identical copies of each backup volume and of a block checksum list attached to each copy which helps to identify eventually damaged parts of the volume. It contains MD5 checksums of chunks of the total data. A block checksum list gets generated and attached to the volume by a run of cd_backup_planer -filter_checksum_blocksize . Such a list may be used to combine several partly damaged copies of a backup volume to gain an undamaged volume. The appropriate place to make use of option -filter_checksum_blocksize is the configuration variable SCDBACKUP_MAKE_CHECKSUM which usually contains a line with options like "-filter_checksum_padsize 1 -filter_md5". For the following examples, the backup will be made with this setting : export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/cd_ -filter_checksum_blocksize 64k -filter_md5" Of course the directory for the list files has to exist : mkdir $HOME/blocklist_dir By this, a checksum tag is appended to the backup volume, as always. After that tag there is appended a block checksum list which has about 0.025 % the size of the backup data. By -blocklist_fileadr the list also gets stored in a separate disk file : $HOME/blocklist_dir/cd_1_1_A41123.194219_229208064 provided that directory exists. Copies of those files should be stored at safe places and their names should be printed and stored together with the backup volumes. Just in case the lists appended to the volumes become all unusable and unrepairable. If the block checksum list itself occupies more than a single block of data then it gets appended an own checksum tag and again a block checksum list. This is repeated until the size is smaller than a single block. To prevent endless appending, this feature is disabled with blocks smaller than 1k. Separate block checksum list files which are covered by appended checksum data may be treated by the recovery tools much like ordinary data. So it might pay off to have lots of copies of them spread everywhere. Producing Repairable Backup Volumes In order to recover damaged backup data you should have at least two, better three or more formerly identical copies of the media. The more copies you have, the larger is the chance to collect a complete undamaged volume. At backup time several copies of each volume can be achieved by writer script mxn_wrapper . It not only can make copies on several media but also is able to write several copies to a single media (provided they fit). This is done by a run of cd_backup_planer with option -multiply_volume . The first copy is directly usable as a normal volume. Any further copies may be retrieved by cd_backup_planer with option -extract_multiplied . The script mxn_wrapper does only work if a buffer file is used. The wrapper expects three groups of arguments, separated by two arguments which consist of a single dot ".". The first two groups set up mxn_wrapper. They have to be provided by you. The last group is provided by scdbackup. The first group describes the media type, the multiplication and eventual the permutation of the backup volume : type inner_copies media_copies [second_buffer permutation_keys] . type may be "CD" or "DVD" selecting scdbackup or sdvdbackup for writing via option -pipe_to_media . If in doubt, use type "CD". inner_copies gives the number of volume copies to be written on each single media. Of course, you have to reduce the size of the volumes by an according factor using option -max_size . media_copies gives the number of media which shall be filled with identical copies of the media image resulting from the inner_copies multiplication. The options second_buffer and permutation keys may be omitted. They are explained below, after the introduction of permuted volumes. The second group describes the writer program to be used for writing a media image to the media. This should be the settings given by scdbackup -show_configuration | fgrep CDRECORD= resp. sdvdbackup -show_configuration | fgrep CDRECORD= The answer will be something like SCDBACKUP_CDRECORD=cdrecord -v resp. SDVDBACKUP_CDRECORD=.../scdbackup-0.9.2/scripts/growisofs_wrapper The last argument of the third group gives the name of the buffer file to read. It is appended together with the other members of group 3 by scdbackup's scripts which believe to communicate with cdrecord. (see appendix WRITER) Examples : You may want to verify the media from these examples by scdbackup_verify -auto in order to see both checksums: the one of the actual backup volume and the one of the write process started by mxn_wrapper. Make 2 identical copies of each CD by using standard cdrecord. The address of the buffer file is /cdbuffer/iso_buffer_cd : mxn_options="CD 1 2 . cdrecord -v ." export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/mxn_wrapper $mxn_options" export SCDBACKUP_BUFFER="/cdbuffer/iso_buffer_cd" export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/cd_ -filter_checksum_blocksize 64k -filter_md5" scdbackup ...usual.arguments.to.scdbackup... The same for DVD mxn_options="DVD 1 2 . $(scdbackup -where scripts)/growisofs_wrapper ." export SDVDBACKUP_CDRECORD="$(sdvdbackup -where scripts)/mxn_wrapper $mxn_options" export SDVDBACKUP_BUFFER="/dvdbuffer/iso_buffer_dvd" export SDVDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/dvd_ -filter_checksum_blocksize 64k -filter_md5" sdvbackup ...usual.arguments.to.sdvdbackup... Of each volume make 2 copies per media, and 3 media copies. The volumes shall have about 315 MB of size : mxn_options="CD 2 3 . cdrecord -v ." export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/mxn_wrapper $mxn_options" export SCDBACKUP_BUFFER="/cdbuffer/iso_buffer_cd" export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/cd_ -filter_checksum_blocksize 64k -filter_md5" scdbackup -max_size 315m ...usual.arguments.to.scdbackup... Put 6 identical CD sized volume copies on a single DVD : mxn_options="DVD 6 1 . $(scdbackup -where scripts)/growisofs_wrapper ." export SDVDBACKUP_CDRECORD="$(sdvdbackup -where scripts)/mxn_wrapper $mxn_options" export SDVDBACKUP_BUFFER="/dvdbuffer/iso_buffer_dvd" export SDVDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/dvd_ -filter_checksum_blocksize 64k -filter_md5" sdvdbackup -max_size 640m ...usual.arguments.to.sdvdbackup... Additional copies of existing and verified backups can be made by scdbackup -pipe_to_media . The original backup volume has to be verified, of course, so you do not copy damaged data and maybe even certify them by a block checksum list. If the backups already got appended block checksum lists it is advised to copy them without any further checksum attachments : unset SCDBACKUP_CDRECORD SCDBACKUP_BUFFER export SCDBACKUP_MAKE_CHECKSUM=" " If the backups do not yet contain block checksum lists, one should add some : export SCDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/ -filter_checksum_blocksize 64k -filter_md5" In both cases, copy the media to disk, verify and then copy to other media cat /dev/cdrom >/cdbuffer/my_buffer_file scdbackup_verify /cdbuffer/my_buffer_file -auto cat /cdbuffer/my_buffer_file | scdbackup -pipe_to_media The same works with DVD and sdvdbackup -pipe_to_media . The copied media have to verify with the same checksum record as the originals. They also have to verify with their possible new checksum record which eventually covers the original data as well as the original checksum appendice. Be aware that each copy process from media eventually adds a new tail of padding (~ 300 kB). Recovering Damaged Repairable Backup Volumes The prerequisite of any repair attempt is a valid block checksum list. It is the base for the decision whether a particular block is useable or not. The various repair attempts may be repeated - but it is essential to use the same block checksum list each time. One may strip a valid block checksum list together with its master tag from a backup volume by volume=/dev/cdrom skip_bytes=0 $(scdbackup -where bin)/cd_backup_planer \ -blocklist_fileadr /tmp/scdbackup_ \ -scan_for_checksum -standalone -scan_for_checksum -blocklist \ -scan_for_checksum -pacifier -scan_for_checksum -first \ -scan_for_checksum -source:$volume \ -scan_for_checksum -skip:$skip_bytes \ -search_md5 - - - - >/dev/null With above example this will produce a file of 14 kB named /tmp/scdbackup_1_1_A41123.194219_229208064 which is equivalent to the data verification part of a separate block checksum list file stored at backup time. If there are several valid blocklists then they all will get stored with names which are derived from name, date and byte count of their checksum tag. If the backup volume is damaged, you will see the usual verifier warnings. One may speed up extraction if a minimum estimation of the volume size is known. In above case, it is safe to ignore the first 200 MB skip_bytes=200m $(scdbackup -where bin)/cd_backup_planer \ ... This causes the extractor to skip this size and eventually existent lists. It will warn that any valid tags do not match the truncated data stream. If skip_bytes is too large it might miss the most recent tag and extract one which stems from an older overwritten backup. So if you are unsure about a minimum volume size estimation, or if you want to verify the volume during extraction, or if you want to copy the volume to disk rather than to /dev/null, then stay with skip_bytes=0 . If you are unlucky, the list itself is not to find or damaged. So backup your lists which are stored on disk. In case only damaged copies are available, see "Recovering Damaged Block Checksum Lists" below. For an example of the recovery process we use the blocklist extracted above : blocklist=/tmp/scdbackup_1_1_A41123.194219_229208064 We could also use the backup volume itself, but that would have to scan the whole volume for the list and therefore would need quite some time. We could use the list $HOME/blocklist_dir/cd_1_1_A41123.194219_229208064 which was stored on hard disk by the backup run. The check facility for such block checksum lists $(scdbackup -where bin)/cd_backup_planer -process_blocklist ... offers several methods to deal with damaged backup data : report prints list parameters and summary to stderr and the numbers of damaged blocks to stdout. Last line to stdout is always "@". $(scdbackup -where bin)/cd_backup_planer -process_blocklist \ report $blocklist 0 /dev/cdrom - >/tmp/damaged_block_list For the following recovery methods it is important to distinguish sequential data streams (stdin, stdout, ...) from random access file objects (disk files, block devices, ...). The kind of available data source and the desired type of the result will lead to the choice of the appropriate method. Random access files are also usable as sequential data streams - but not vice versa. collect copies undamaged blocks from a data stream to a random access file. $(scdbackup -where bin)/cd_backup_planer -process_blocklist \ collect $blocklist 0 /dev/cdrom /tmp/recovering_image mend checks the blocks in a file with random read-write access and tries to replace the damaged ones by their counterparts in another random access file. $(scdbackup -where bin)/cd_backup_planer -process_blocklist \ mend $blocklist 0 /dev/cdrom /tmp/recovering_image This method shall read media which do not produce sequential data streams of sufficient length. In extreme cases a tool like dd_rescue might help to access the data survivors. In less extreme cases the optional interval addressing mechanism of "mend" may help. You may process from a certain block number up to the end (e.g. mend:10000) or an interval (e.g. mend:0:9999 processes the first 10000 blocks). You may process the blocks in reversed order (e.g. mend:end:0). Method "mend" may also be the fastest way to repair a volume which does not contain many damaged blocks. merge reads two sequential data streams and replaces damaged blocks of the one stream by undamaged blocks of the other. Output is a sequential stream on stdout. $(scdbackup -where bin)/cd_backup_planer -process_blocklist \ merge $blocklist 0 /dev/cdrom /tmp/recovering_image \ >/tmp/next_recovering_image The recovery methods can be repeated with all copies available and will hopefully yield an undamaged backup volume. Method "merge" will report whether the checked volume is completely repaired or still needs more recovery. Method "mend" will be able to do so only if all blocks have been processed. The results of method "collect" may be complete even if all "collect" runs reported errors. A run of -process_blocklist report $blocklist 0 /tmp/recovering_image - >/dev/null shows the current state (>/dev/null to not get flooded by block numbers). All four methods return an exit value of 0 if they are sure that the result is completely repaired and a non zero exit value else. This example reads three CDs simultaneously. One of them remotely via ssh : ssh ts2 -l thomas cat /dev/cdrom | \ $(scdbackup -where bin)/cd_backup_planer -pacifier -off \ -process_blocklist merge $blocklist 0 - /dev/cdrom | \ $(scdbackup -where bin)/cd_backup_planer \ -process_blocklist merge $blocklist 0 - /dev/cdrecorder \ >/tmp/recovering_image This extracts both copies from a multi-copy CD and drops usable blocks to disk file /tmp/recovering_image . Read the first copy which is directly accessible : $(scdbackup -where bin)/cd_backup_planer \ -process_blocklist collect $blocklist 0 /dev/cdrom /tmp/recovering_image Now skip 1 copy and deliver the next one to the "collect" method : $(scdbackup -where bin)/cd_backup_planer \ -extract_multiplied /dev/cdrom 1 | \ $(scdbackup -where bin)/cd_backup_planer -pacifier -off \ -process_blocklist collect $blocklist 0 - /tmp/recovering_image This tries to squeeze data from the hinder end of a bad spot by processing the blocks in reversed order. $(scdbackup -where bin)/cd_backup_planer \ -process_blocklist mend:end:0 $blocklist 0 /dev/cdrom /tmp/recovering_image If this for example yields the error message While reading block 55376 of 66994 : Error 5 : Input/output error one could next try to skip the bad block and begin reading at number 55375 $(scdbackup -where bin)/cd_backup_planer \ -process_blocklist mend:55375:0 $blocklist 0 \ /dev/cdrom /tmp/recovering_image If mend or collect do not yield a perfect result after all media are processed then it may be rewarding to repeat them. Just in case some of the read errors are volatile and the media are willing to produce some correct block from time to time. Possibly one would try to bang against aborting error spots several times from both sides. From above : $(scdbackup -where bin)/cd_backup_planer \ -process_blocklist mend:55400:0 $blocklist 0 \ /dev/cdrom /tmp/recovering_image and from below $(scdbackup -where bin)/cd_backup_planer \ -process_blocklist mend:55300:end $blocklist 0 \ /dev/cdrom /tmp/recovering_image Recovering Damaged Block Checksum Lists For the above examples one needs a valid block checksum list. If there are only copies that refuse to load, one may try to recover them. This will work only if blocksize was at least 1k and the lists are larger than one block. A damaged tag and block checksum list may be read from a backup volume by metablocklist=/tmp/metablocklist $(scdbackup -where bin)/cd_backup_planer \ -extract_tag_tail 4m \ $metablocklist cp $metablocklist /tmp/recovering_blocklist In that state it may serve as source of its own block checksum list and one can try to gain undamaged blocks from other damaged volumes by this pipe : $(scdbackup -where bin)/cd_backup_planer \ -extract_tag_tail 4m \ >/tmp/recovering_image and craft a damaged volume with valid block checksum list. If $metablocklist is a damaged list file which was stored at backup time on disk, then one will have to skip 1 checksum tag in order to reach the desired meta block list data : ... -process_blocklist collect $metablocklist 1 - /tmp/recovering_blocklist In that case you will not have to set the first tag's position number to 0 because it already has that value. Permuted Volumes The above provisions should be able to recover from random errors if the number of copies can cope with the density of errors. Nevertheless, there may be systematic error scenarios which increase the probability that all copies of a backup data block get hit. E.g. if the media tends to deteriorate at the end of its data content. A possible precaution against this menace are permuted backup volume images. They are neither mountable nor extractable in that state but they will expose different data blocks to said systematic errors than readable volumes will do. The permutation can be reverted in order to get the volume readable or at least usable as source of repair blocks. There are 8 permutation schemes defined. 0 ... blocks stay in sequence (but shifted by one or more blocks) 1 ... swap both halves of the file 2 ... mirror sequence of blocks (i.e. revert sequence) 3 ... swap 1st and 2nd quarter, sawp 3rd and 4th +4 ... additionally interlace resulting 1st and 2nd half See cd_backup_planer -help and above appendix VERIFY. It is worthwhile to use different schemes for several reserve copies. At least have one of {1,3} and one of {4,6,7}. Record the permutation parameters (as shown on stderr) independendly of the permuted files. Although the permuted files are supposed to contain those parameters in their header section, be prepared for this section to be damaged. Permuted backup volumes may be produced by mxn_wrapper. The first argument group of mxn_wrapper allows to set a permutation key for each of the media copies. The argument sequence is : type inner_copies media_copies [second_buffer permutation_keys] . The first three are explained above ("Producing Repairable Backup Volumes"). The other argument apply to permutation : second_buffer gives the name of a second buffer file of media size. Such a file is needed if inner_copies is larger than 1 together with permutation keys. It is ignored if inner_copies is 1. Use "-" in that case. permutation_keys is a list of keys for cd_backup_planer -permute_blocks. "-" means no permutation, "0" to "7" choose one of the defined permutation schemes (of which none is readable without reverse permutation !). blocksize is always 64k. If the number of key arguments is less than media_copies then unpermuted media will get written before the permuted ones. Example : Put 2 identical volume copies on 4 DVD. Permute the last 2 DVD . Because the 2 on-media copies are quite invariant under permutation key 1, i choose keys 4 (interlace) and 7 (swap quarters and interlace) : bscripts=$(scdbackup -where scripts) mxn_options="DVD 2 4 /my_second_disk/multiply_buffer 4 7 ." mxn_options="$mxn_options $bscripts/growisofs_wrapper ." export SDVDBACKUP_CDRECORD="$bscripts/mxn_wrapper $mxn_options" export SDVDBACKUP_BUFFER="/dvdbuffer/iso_buffer_dvd" export SDVDBACKUP_MAKE_CHECKSUM="-blocklist_fileadr $HOME/blocklist_dir/dvd_ -filter_checksum_blocksize 64k -filter_md5" sdvdbackup -max_size 2125m ...usual.arguments.to.scdbackup... In general, file permutation and reverse permutation can be done from a disk file or block device by option -permute_blocks of bin/cd_backup_planer. Examples : Restore the original sequence of a permuted backup in a buffer file $(scdbackup -where bin)/cd_backup_planer \ -permute_blocks restore /dev/cdrom 0 0 0 >/cdbuffer/my_buffer_file Directly drop undamaged blocks of the second volume copy on the media into a recovering backup volume image in the /tmp directory $(scdbackup -where bin)/cd_backup_planer \ -permute_blocks restore /dev/cdrom 0 0 0 | \ $(scdbackup -where bin)/cd_backup_planer -pacifier -off \ -extract_multiplied - 1 | \ $(scdbackup -where bin)/cd_backup_planer -pacifier -off \ -process_blocklist collect $blocklist 0 - /tmp/recovering_image Produce quite a mixed-up collection of reserve blocks from a buffer file which contains a verified, readable backup volume image : $(scdbackup -where bin)/cd_backup_planer \ -permute_blocks 7 /cdbuffer/my_buffer_file 256k 0 0 | \ scdbackup -pipe_to_media Of course, this media will only verify with its newly computed checksum and not with the checksum of the backup volume. Verify a permuted volume on-the-fly : $(scdbackup -where bin)/cd_backup_planer \ -permute_blocks restore /dev/cdrom 0 0 0 | \ scdbackup_verify - -auto In case of a damaged header section of the permutated file, you will have to give the necessary parameters as arguments : $(scdbackup -where bin)/cd_backup_planer \ -permute_blocks -7 /dev/cdrom 256k $blockcount 1 >/cdbuffer/my_buffer_file A permutation scheme which causes less disk rattling is swapping of image halves : ... -permute_blocks 1 /cdbuffer/my_buffer_file 256k 0 0 ... ----------------------------------------------------------------------- Appendix CRYPTO It is advisable to store your backups far away from the computer with the original data. This imposes the problem that it is hard to ensure that nobody gets access to the content of those backups. The appropriate answer to this problem is encryption. But you have to test regularly whether you still know how to unpack such an encrypted backup. When in need then you will find nobody else who will be able to do that for you. See paragraph "Testing" below. With contemporary ISO formatter programs it is not possible to encrypt only the content of the files and not the structure of the ISO filesystem. A completely encrypted ISO filesystem would lose its advantages over a sequential archive format, though. So it appears natural to combine encryption with afio rather than with ISO 9660. The script afio_sencrypt provides the opportunity to produce encrypted and eventually compressed afio archives. Encryption is done by program sencrypt. See http://scdbackup.webframe.org/README_sencrypt Usage of vanilla afio may be replaced by usage of afio_sencrypt via the configuration files s*backup_afio_value resp. variables S*BACKUP_AFIO. For encryption you will have to choose a key phrase. Decryption will only be possible if you can provide the identical key phrase again. The phrase should contain many characters. Only the first 1000+ are used but it does not harm to provide more. To be suitable for all possible input ways, the phrase should not contain newline characters or characters with special meaning to the shell. Use letters, numbers, space, period, comma, underscore, minus. If it is not intended to memorize them in human mind, then use a mix of more or less random characters. Else use a phrase which is long but easy to remember. (But not your favorite movie quote which is known to all your friends !) Next you will have to decide whether you want an unencrypted afio archive holding encrypted files or whether you want to encrypt the archive as a whole. The first method is quite slow but produces the usual rugged afio archives which can be read even if the start of the archive data is missing. The file names and their attributes (like access permissions) will stay visible in cleartext. The second method is faster. It hides names and attributes, but a damaged archive will be unreadable (unless repaired by methods described in REDUNDANCY). With this second method you got the option to use an ad-hoc key phrase which does not have to be stored in a file. You may also encrypt archives made by program star, but only the method of whole archive encryption is available. At the end of each afio oriented paragraph, there will be appropriate examples with star_as_afio_wrapper. Encryption Usually you will first establish a well protected keyphrase file : mkdir $HOME/keyfiles chmod u+rwx,go-rwx $HOME/keyfiles touch $HOME/keyfiles/dvd_key_phrase chmod u+rw,go-rw $HOME/keyfiles/dvd_key_phrase vi $HOME/keyfiles/dvd_key_phrase You will either have to memorize the keyphrase perfectly or you will have to make an emergency copy on external media. Like : cp $HOME/keyfiles/dvd_key_phrase /media/usbstick/dvd_key_phrase Then you set variable SCDBACKUP_AFIO resp. SDVDBACKUP_AFIO to the appropriate call of afio_sencrypt or star_as_afio_wrapper . For unencrypted afio-archives containing encrypted files export SDVDBACKUP_AFIO="$(sdvdbackup -where scripts)/afio_sencrypt -each_file_by $HOME/keyfiles/dvd_key_phrase" For a completely encrypted afio-archive export SDVDBACKUP_AFIO="$(sdvdbackup -where scripts)/afio_sencrypt -whole_archive_by $HOME/keyfiles/dvd_key_phrase" If you prefer to enter the key phrase manually export SDVDBACKUP_AFIO="$(sdvdbackup -where scripts)/afio_sencrypt -whole_archive" Key phrases for encryption are asked twice and have to be identical both times. For completely encrypted star-archives export SDVDBACKUP_AFIO="$(scdbackup -where scripts)/star_as_afio_wrapper -encrypt_by $HOME/keyfiles/dvd_key_phrase" For completely encrypted star-archives with manual key prompt export SDVDBACKUP_AFIO="$(scdbackup -where scripts)/star_as_afio_wrapper -encrypt" Then run your usual scdbackup afio command sdvbackup_afio ...usual.arguments... and memorize on paper the appropriate unpacking command from the list below ! Do not rely on availability or accurateness of a future README file. Unpacking To unpack the archive, you need to know what method was used and whether compression was enabled (sdvdbackup -show_configuration | grep AFIO_COMPRESS). Uncompressed unencrypted afio-archives containing encrypted files afio -i -P 'sencrypt' -Q '-d' \ -Q '-f' -Q "$HOME/keyfiles/dvd_key_phrase" -Z /dev/cdrom Compressed unencrypted afio-archives containing encrypted files afio -i -P 'sencrypt' -Q '-d' -Q '-g' \ -Q '-f' -Q "$HOME/keyfiles/dvd_key_phrase" -Z /dev/cdrom Uncompressed encrypted afio-archives sencrypt -d -s 322 -f "$HOME/keyfiles/dvd_key_phrase" $HOME/keyfiles/test_key_phrase or vi $HOME/keyfiles/test_key_phrase or omit the -f "$HOME..." arguments in the following examples to be prompted for the key phrase at /dev/tty. For testing decryptability of an unencrypted archive with encrypted files, you will have to extract a file and check whether it is readable. Uncompressed : mkdir $HOME/test_dir cd $HOME/test_dir afio -i -P 'sencrypt' -Q '-d' \ -Q '-f' -Q "$HOME/keyfiles/test_key_phrase" -Z \ -v -y '*ASKME*' /dev/cdrom view .../*ASKME* Compressed : ... afio -i -P 'sencrypt' -Q '-d' -Q '-g' \ ... For testing decryptability of an archive which is encrypted as a whole, it is sufficient to test whether afio resp. star can produce a table of content. Uncompressed afio-archive : sencrypt -d -s 322 -f "$HOME/keyfiles/test_key_phrase" = 0.1.2 (If providing option -as mkisofs) 2nd genisoimage (This surely provides modern mkisofs features) 3rd mkisofs (There have been some not-so-good versions long ago) The default choice may also be set by variables before the run of ./CONFIGURE_CD resp. ./CONFIGURE_DVD. Select xorriso export SCDBACKUP_USE_GENISOIMAGE="xorriso" export SDVDBACKUP_USE_GENISOIMAGE="xorriso" Select mkisofs export SCDBACKUP_USE_GENISOIMAGE="mkisofs" export SDVDBACKUP_USE_GENISOIMAGE="mkisofs" Select genisoimage export SCDBACKUP_USE_GENISOIMAGE="genisoimage" export SDVDBACKUP_USE_GENISOIMAGE="genisoimage" Temporary Selection The use of the formatter program is controlled by variable SCDBACKUP_MKISOFS for CD: export SCDBACKUP_MKISOFS="xorriso -return_with MISHAP 32 -abort_on FATAL -as mkisofs -graft-points" export SCDBACKUP_MKISOFS="genisomage -l -allow-leading-dots -R -D -graft-points" export SCDBACKUP_MKISOFS="mkisofs -l -allow-leading-dots -R -D -graft-points" and with the same contents by SDVDBACKUP_MKISOFS for DVD. ----------------------------------------------------------------------- Appendix CD : The script ./CONFIGURE_CD tries to find and evaluate suitable programs for burning CD media. There are several program names builtin for this check: cdrecord The CD subsystem traditionally depends on Joerg Schilling's cdrecord which comes in package cdrtools together with mkisofs. cdrecord or a near clone of it is supposed to be part of any older Linux distribution. Source of current cdrtools is best fetched from: ftp://ftp.berlios.de/pub/cdrecord/alpha The usable addresses have undergone some evolution following the changes in hardware customs. Real SCSI CD recorders have become rare. But USB and SATA appear as pseudo-SCSI devices in Linux. On older Linux kernels the module ide-scsi makes them appear like SCSI devices. Newer kernels offer them as pseudo hard disks /dev/hdX. The following table tries to give you an overview over Linux kernels and possible addresses for CD burners : Kernel SCSI,USB,SATA IDE + ide-scsi IDE (Joerg) IDE (Linus) 2.2 0,0,0 0,0,0 -unknown- -not usable- 2.4 0,0,0 0,0,0 ATA:1,0,0 -not usable- 2.6 0,0,0 -deprecated- ATA:1,0,0 /dev/hdc The Joerg-style addresses know other prefixes like "REMOTE:". Run cdrecord dev=HELP to get a list. Read the prefix from lines "Transp. layer ind.:". Use prefix without ":" to learn about the according devices. Like : cdrecord dev=ATA -scanbus Note: ATA does not work while the recorder is guarded by ide-scsi. wodim wodim is the cdrecord fork within package cdrkit. It stems from cdrecord 2.01.01 and therefore is fully suitable for scdbackup. Package cdrkit can be obtained from: http://debburn.alioth.debian.org/ A valuable feature of wodim is that it needs no superuser authority but only rw-permission for /dev/sgN resp. /dev/hdX. It might nevertheless be a bit of guesswork to find out the /dev/sgN addresses which to chmod +rw. cdrskin For scdbackup on Linux kernel 2.4 and above, cdrskin is a complete alternative to cdrecord. It is based on the library libburn from libburnia-project.org. Its greatest appeal is that it is written by me. :)) Make sure to have version cdrskin-0.2.6 or newer in order to get the features needed by scdbackup. With Linux kernel 2.4 the burner device must be under control of ide-scsi emulation. With more modern kernels it shouldn't. cdrskin is provided as source code and as Linux-x86-binaries at http://scdbackup.webframe.org/cdrskin_eng.html http://scdbackup.sourceforge.net/cdrskin_eng.html To configure scdbackup for the use of cdrskin, the superuser should first ensure that the CD devices have proper permissions set. Normal users will only see devices which offer rw-permissions. As superuser: cdrskin --devices The reply will be something like 0 dev='/dev/sr0' rwrwr- : 'TEAC' 'CD-ROM CD-532S' 1 dev='/dev/sr1' rwrw-- : 'LITE-ON' 'LTR-48125S' from which you may derive a command which allows full access for everybody (you may be more restrictive, if you want): chmod a+rw /dev/sr0 /dev/sr1 xorriso xorriso is able to perform single session burn runs in the way as needed by scdbackup. Its writing capabilities are based on libburn from libburnia-project.org. See also above at the end of Appendix ISO. You need at least xorriso-0.1.2 for the "-as cdrecord" emulation Although it is sufficient, its appearance is not very similar to cdrecord, wodim or cdrskin. It does not understand the Bus,Lun,Target addresses of cdrecord tradition but only /dev/... . A run of xorriso -devices will show a list of available drive addresses. This deviation of traditions earns xorriso the last place in the automatic selection list. In general it is less flexible than cdrskin which uses the same foundation software. Selection during Configuration ./CONFIGURE_CD tries to find the above programs and evaluates which might be best suited. The list of found programs becomes part of the user prompt: What CD burn program to use ? ( cdrskin cdrecord xorriso wodim ) If no CD burn program was set before, then the first list member will be offered as default and thus can be chosen by an empty input line. The normal ranking is 1st cdrskin >= 0.2.6 (In case of trouble, i can give support for this) 2nd cdrecord (Not easy to tell what gets found under that name) 3rd xorriso >= 0.1.2 (In case of trouble, i can give support for this) 4th wodim (In summer 2008 there seems to be no support for wodim) As long as no CD burn program is set, it matters whether a program actually can detect a CD drive to which it would be able to write. So a properly installed cdrecord might outrank a cdrskin which suffers from missing rw-permissions. The default choice may also be set by variables before the run of ./CONFIGURE_CD resp. ./CONFIGURE_DVD. Select cdrecord export SCDBACKUP_USE_CDRSKIN="cdrecord" Select wodim export SCDBACKUP_USE_CDRSKIN="wodim" Select cdrskin export SCDBACKUP_USE_CDRSKIN="cdrskin" Select xorriso export SCDBACKUP_USE_CDRSKIN="xorriso" Then run ./CONFIGURE_CD To trigger the full initial examination of programs: export SCDBACKUP_USE_CDRSKIN="default" ./CONFIGURE_CD Temporary Selection For switching temporarily between modern versions of the programs it should be enough to set one out of this list: export SCDBACKUP_CDRECORD="cdrecord -v" export SCDBACKUP_CDRECORD="wodim -v" export SCDBACKUP_CDRECORD="cdrskin -v" # Important: xorriso cannot handle cdrecord style addresses like "0,0,0". export SCDBACKUP_CDRECORD="xorriso -as cdrecord -v" to set a suitable drive address export SCDBACKUP_SCSI_ADR="/dev/sr0" and then to run the backup command scdbackup ... ----------------------------------------------------------------------- Appendix DVD : The script ./CONFIGURE_DVD tries to find and evaluate suitable programs for burning DVD media. There are several program names builtin for this check: growisofs The DVD subsystem traditionally depends on Andy Polyakov's dvd+rw-tools and especially on his program growisofs. The source of dvd+rw-tools is best fetched from http://fy.chalmers.se/~appro/linux/DVD+RW/tools/ See also below: "How to get and install growisofs". For BD (Blu-ray) media, you need at least growisofs-7.1. cdrskin For scdbackup on Linux kernel 2.4 and above and except some exotic media, cdrskin is a complete alternative to growisofs, dvd+rw-mediainfo and dvd+rw-format. It is based on the library libburn from libburnia.pykix.org. Make sure to have version cdrskin-0.3.4 or newer in order to get all DVD features needed by sdvdbackup. Older versions will not get selected automatically. With Linux kernel 2.4 the burner device must be under control of ide-scsi emulation. With more modern kernels it shouldn't. cdrskin is provided as source code and as Linux-x86-binaries at http://scdbackup.webframe.org/cdrskin_eng.html http://scdbackup.sourceforge.net/cdrskin_eng.html See above in Appendix CD, paragraph "cdrskin" for necessary drive preparation. For BD (Blu-ray) media, you need at least cdrskin-0.6.0. xorriso xorriso is able to perform single session burn runs on the same media as cdrskin with which it shares the fundamental burn software. libburnia-project.org. See also above Appendix ISO. Both programs are about equivalent from the view of sdvdbackup. Because cdrskin accepts a larger set of drive addresses, it is slightely preferrable over xorriso. For BD (Blu-ray) media, you need at least xorriso-0.3.2. wodim wodim is the cdrecord fork within package cdrkit. It stems from cdrecord 2.01.01 and has own DVD capabilities. See "Restrictions with cdrecord and wodim". Package cdrkit can be obtained from: http://debburn.alioth.debian.org/ cdrecord As of cdrtools-2.01.01a09, DVD burning is included in the source code version. In the past there were time limited binaries "cdrecord-ProDVD" which are obsoleted by the current cdrtools source releases. You may obtain 2.01.01aXY releases from: ftp://ftp.berlios.de/pub/cdrecord/alpha Restrictions with cdrecord and wodim Regrettably both, cdrecord(-ProDVD) and wodim, only offer DAO mode for DVD which implies that without a buffer file on disk every DVD has to be padded up to the maximum size. Another disadvantage of both is that they do not offer convenient mode "Restricted Overwrite" with DVD-RW media. It has not been tested whether they work properly with BD (Blu-ray) media. Selection during Configuration ./CONFIGURE_DVD tries to find the above programs and evaluates which might be best suited. The list of found programs becomes part of the user prompt: What DVD burn program to use ? ( growisofs cdrskin cdrecord xorriso wodim ) If no CD burn program was set before, then the first list member will be offered as default and thus can be chosen by an empty input line. The normal ranking is 1st xorriso >= 0.1.8 (In case of trouble, i can give support for this) 2nd cdrskin >= 0.4.6 (In case of trouble, i can give support for this) 3rd growisofs (Well tested with sdvdbackup) 4th cdrskin >= 0.3.4 (Does not know stream_recording= for BD) 5th xorriso >= 0.1.2 (Does not know stream_recording= for BD) 6th wodim (In summer 2008 there seems to be no support for wodim) 7th cdrecord(-ProDVD)(Must issue "This copy of cdrecord is licensed for:" or "Cdrecord-ProDVD") As long as no DVD burn program is set,it matters whether a program actually can detect a CD drive to which it would be able to write. So a properly installed cdrecord might outrank a cdrskin which suffers from missing rw-permissions. The default choice may also be set by variables before the run of Select growisofs export SDVDBACKUP_USE_PRODVD="growisofs" Select cdrskin export SDVDBACKUP_USE_PRODVD="cdrskin" Select xorriso export SDVDBACKUP_USE_PRODVD="xorriso" Select cdrecord export SDVDBACKUP_USE_PRODVD="cdrecord" Select wodim export SDVDBACKUP_USE_PRODVD="wodim" Then run ./CONFIGURE_DVD To trigger the full initial examination of programs: export SDVDBACKUP_USE_PRODVD="default" ./CONFIGURE_DVD Temporary Selection Switching temporarily between those is a bit more complicated, because not only the address of the burn program has to be changed, but also several companion variables: ------------------------------------------------------------------------------- Select growisofs temporarily ------------------------------------------------------------------------------- export SDVDBACKUP_USE_PRODVD="0" export SDVDBACKUP_CDRECORD="$(sdvdbackup -where scripts)/growisofs_wrapper" export SDVDBACKUP_CDRECORD_OPT="-final_eject fs=16m" export SDVDBACKUP_DEV_ADR="/dev/sr0" ------------------------------------------------------------------------------- Select cdrskin temporarily ------------------------------------------------------------------------------- export SDVDBACKUP_USE_PRODVD="cdrskin" export SDVDBACKUP_CDRECORD="cdrskin -v" export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject -data -tao assert_write_lba=0 fifo_start_at=0 driveropts=burnfree" export SDVDBACKUP_SCSI_ADR="0,0,0" or export SDVDBACKUP_SCSI_ADR="/dev/sr0" (But not SDVDBACKUP_DEV_ADR) ------------------------------------------------------------------------------- Select xorriso temporarily ------------------------------------------------------------------------------- export SDVDBACKUP_USE_PRODVD="xorriso" export SDVDBACKUP_CDRECORD="xorriso -as cdrecord -v" export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject" export SDVDBACKUP_SCSI_ADR="/dev/sr0" (But not SDVDBACKUP_DEV_ADR) ------------------------------------------------------------------------------- Select cdrecord temporarily ------------------------------------------------------------------------------- export SDVDBACKUP_USE_PRODVD="1" export SDVDBACKUP_SCSI_ADR="0,0,0" export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject driveropts=burnfree -data -sao" Without buffer file (i.e. on-the-fly) export SDVDBACKUP_BUFFER="-" export SDVDBACKUP_MEDIA_CAP=4480m export SDVDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_cdrecord_sao cdrecord" With buffer file export SDVDBACKUP_BUFFER="/dvdbuffer/isofs_dvd" export SDVDBACKUP_CDRECORD="cdrecord -v" ------------------------------------------------------------------------------- Select wodim temporarily ------------------------------------------------------------------------------- export SDVDBACKUP_USE_PRODVD="wodim" export SDVDBACKUP_SCSI_ADR="0,0,0" export SDVDBACKUP_CDRECORD_OPT="padsize=300k fs=16m -eject driveropts=burnfree -data -sao" Without buffer file (i.e. on-the-fly) export SDVDBACKUP_BUFFER="-" export SDVDBACKUP_MEDIA_CAP=4480m export SDVDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_cdrecord_sao wodim" With buffer file export SDVDBACKUP_BUFFER="/dvdbuffer/isofs_dvd" export SDVDBACKUP_CDRECORD="wodim -v" ------------------------------------------------------------------------------- How to get and install growisofs Get the source package from : http://fy.chalmers.se/~appro/linux/DVD+RW/tools/ and install according to the instructions given. Meanwhile binary versions of dvd+rw-tools are part of Linux distributions. It is a good idea to read the tutorial of Andy Polyakov's DVD software : http://fy.chalmers.se/~appro/linux/DVD+RW/ because growisofs is useful independendly of scdbackup. Error codes (like "SK=5h/ASC=21h/ACQ=00h") can be looked up at http://fy.chalmers.se/~appro/linux/DVD+RW/keys.txt In case you find no other source for dvd+rw-tools there is provided a (outdated) copy under http://scdbackup.webframe.org/dvd+rw-tools-5.19-1.4.9.7.tar.gz Untar, enter the directory dvd+rw-tools-5.19.4.9.7, and execute make . Then copy the programs to a directory where they can be executed by all interested users. E.g. : cp growisofs dvd+rw-format dvd+rw-mediainfo /usr/bin Find out the SCSI device address of your DVD recorder. In my case it was /dev/sr0 . You may count the entries of "Type: CD-ROM" in the output of cat /proc/scsi/scsi until you reach the entry with your recorder. The first entry will be /dev/sr0 , the next one /dev/sr1 and so on. With Linux kernel version below 2.5 the EIDE DVD recorder has to be emulated as an SCSI device as described by Andy's tutorial (or by README appendix IDE where an ATAPI CD-RW gets attached as an SCSI device). With newer kernels and recent versions of growisofs it is possible to use IDE device addresses directly without SCSI emulation. Make sure that the device offers read+write permissions to all interested users. If it is not readable you will get slightely confusing messages from growisofs. ----------------------------------------------------------------------- Appendix DVD-TYPES (as of june 2008) : There are many types of DVD media and BD (Blu-ray) media. In general, media and writer firmware have to match properly. It is not easy to determine this match in advance. Generally it is a good idea to stick with the media list of the writer's manufacturer. Nevertheless, there is no guarantee that those media will work or that others won't. All my DVD burners achieve good results with 4x DVD+RW media. My BD burner works fine with several media brands except the BD-RE disc which was provided by its manufacturer. That one is totally unusable. The reusal peculiarities of DVD+RW and DVD-RW are dealt automatically by sdvdbackup if you answer "y" to ./CONFIGURE_DVD's question "Automatically erase DVD before writing ? (y/n)". This automat recognizes DVD+R, DVD-R, BD-R for which it omits erasing. DVD+R, DVD+R DL and DVD-R Each of these media can be used only once. sdvdbackup writes one backup on it and then it cannot write to that disc again. With DVD+R DL you need quite recent versions of the DVD burn programs. Use sdvdbackup Option -dvd_dl to announce its unusual media size. DVD+RW DVD+RW seem to be the least problematic media. They are writeable and re-writeable without any further precautions. The writer programs growisofs, cdrskin, cdrecord and wodim can deal with them and re-use them without further precautions. I tested Fuji Film 4xDVD+RW, TDK DVD+RW47C, Tevion/Platinum 4x, Octron 4x. They all are reported by dvd+rw-mediainfo as Media ID: "RICOHJPN/W11". Meanwhile there appeared "RITEK/004" which work well, too. Under Linux kernel 2.6 it is possible to use DVD+RW like DVD-RAM. The restricted number of re-writes makes it not suitable for an rw-mounted filesystem, though. The "tape device" method described with DVD-RAM works fine. DVD-RW A DVD-RW is in one of several media states which automatically select a certain write behavior. Additionally to growisofs one needs program dvd+rw-format from the same package dvd+rw-tools. cdrskin and xorriso are able to do the necessary formatting and blanking by themselves. cdrecord and wodim can blank by themselves. - Restricted Overwrite media may be overwritten by sdvdbackup and will stay in state Restricted Overwrite. Media initially get into that state by : dvd+rw-format -force /dev/... cdrskin -v dev=... blank=format_overwrite xorriso -outdev ... -format as_needed Or via sdvdbackup_blank The burn programs cdrecord and wodim can neither format DVD-RW nor write to formatted DVD-RW. Less appealing with sdvdbackup are the sequential states. - Sequential Blank .... media may be overwritten by sdvdbackup and will then be in state Sequential Nonblank. Blank is the state of previously unused media. Written sequential DVD-RW can be in one of the two following states: - Sequential Closed media is not suitable for sdvdbackup. It may be erased and converted into state Sequential Blank by : dvd+rw-format -blank=full /dev/... cdrskin -v dev=... blank=all xorriso -outdev ... -blank all cdrecord -v dev=... blank=all wodim -v dev=... blank=all Or via: sdvdbackup_blank all - Sequential Appendable media is not suitable for sdvdbackup. growisofs and cdrskin would be willing to add more data, but sdvdbackup is not prepared to do so. The same blank commands apply as with Closed media. A special state can cause trouble and therefor it is better to avoid it: - Sequential Fast Blank media can only be written in DAO mode. This demands a buffer file or script pipe_to_cdrecord_sao as well as modification of eventual cdrskin options. In fact, this state is only suitable for burn programs cdrecord and wodim, which have DAO as only write mode. A DVD-RW gets into this state by: cdrecord -v dev=... blank=fast wodim -v dev=... blank=fast For use with sdvdbackup via growisofs, cdrskin, or xorriso i advise state "Restricted Overwrite" despite its misleading name. This state is stable in itself and makes a DVD-RW behave much like a DVD+RW. It may be necessary or desirable, though, to use the Sequential write mode. If so, you will have to bring the media into state Sequential Blank before each reusal. This lasts as long as a full 4.7 GB write run, if you do not want to get "Fast Blank" media which is unsuitable for sdvdbackup together with the better DVD burn programs. With cdrecord and wodim there is only one possible write mode (-sao) and the media can be in the sequential states. DVD-RAM growisofs, xorriso and cdrskin can use DVD-RAM like DVD+RW. See above. DVD-RAM is supported directly at least by Linux kernels 2.4 or higher. There is no media specific formatting needed (unlike DVD-RW). You may install a filesystem on DVD-RAM like on a hard disk partition and mount it for read-write filesystem operations. If so, sdvdbackup can make use of it in the same way it could make use of a hard disk file system. If the DVD-RAM is *not mounted* sdvdbackup can use it like a tape device with quite similar behavior as a DVD+RW. This is achieved via script pipe_to_raedchen which may replace growisofs_wrapper. With reporting in megabytes steps : export SDVDBACKUP_CDRECORD=\ "$(scdbackup -where scripts)/pipe_to_raedchen /dev/sr0 1m 0 -flush" Without intermediate messages : export SDVDBACKUP_CDRECORD=\ "$(scdbackup -where scripts)/pipe_to_raedchen /dev/sr0 0 0 -flush" The fact that the slow DVD-RAM media share i/o buffer space with the system hard disk is not very helpful for system performance during DVD-RAM writing. Therefore the somewhat clumsy -flush option of raedchen has to be used. The write and read performance of DVD-RAM was much lower than those of DVD-RW or DVD+RW. There have been failed verifications, too. Be aware that the usable capacity of a DVD-RAM is less than 4.7e9 bytes. My tests with Panasonic LM-AF120U media yielded raw 4368m and with a vanilla ext2 file system 4076m. sdvdbackup option -dvd_ram adapts to the size of raw DVD-RAM media. By default DVD-RAM is checkreading while writing. This reduces speed to half or even less. The drive reaction in experiments with bad media has not been convincing. The drives just need longer to abort a bad burn. Programs xorriso and cdrskin can avoid this checkreading by option stream_recording=. Be aware that DVD-RAM might not be readable in older DVD-ROM drives. BD-RE BD-RE are much like very large DVD-RAM. They are sold unformatted and thus have to get formatted first. This is done automatically by the BD burn programs or may be done explicitely: dvd+rw-format /dev/... cdrskin -v dev=... blank=as_needed xorriso -outdev ... -format as_needed You need quite recent burn program versions for handling BD-RE. The usable size depends on the individual formatting. Usually it is 22.5g for media with nominal "25 GB". sdvdbackup* commands can deal with the media. Use their option -bd to announce "25 GB" media and option -bd_dl for "50 GB". BD-RE by default perform Defect Management which shall detect small groups of bad media blocks and replace them by reserve blocks. This usually slows down writing speed by a factor of 2 or 3. The burn programs cdrskin and xorriso are able to circumvent this mechanism and to achieve full nominal writing speed of the media. This is controlled by file sdvdbackup_stream_recording_value resp. variable SDVDBACKUP_STREAM_REC, and may be configured together with the input of speed. BD-R BD-R can be used only once, like CD-R, DVD+R. They are sold unformatted and may be used in that state. Unformatted BD-R do not perform Defect Management and can be written with full nominal speed. growisofs automatically formats them before first use. The burn programs cdrskin and xorriso use unformatted BD-R if given. BD-R may also get formatted explicitely: dvd+rw-format /dev/... cdrskin -v dev=... blank=format_if_needed xorriso -outdev ... -format as_needed sdvdbackup option -bd announces "25 GB" media. Option -bd_dl is for "50 GB". To exploit the full capacity of unformatted BD-R you may specify a slightly larger size by -max_size 22.2g resp. -max_size 44.4g . Credits: Some of the test media have been donated by Mike Evans, MFaya, Thomas Weber. ----------------------------------------------------------------------- Appendix IDE (as of june 2008) : This text applies to CD or DVD writers which are connected to ATA controllers under Linux kernel 2.4. It does not apply to drives which appear as SCSI device files /dev/sgN and /dev/srM unconditionally. I.e. those connected via genuine SCSI or USB. With Linux kernel 2.6 the ide-scsi emulation has fallen into disgrace. Modern versions of cdrecord can access ATAPI devices directly via addresses like ATA:0,1,0 or ATAPI:0,1,0 . Use cdrecord dev=ATA -scanbus wodim dev=ATA -scanbus cdrskin dev=ATA -scanbus to get an overview of available devices and their addresses. Text ftp://ftp.berlios.de/pub/cdrecord/README.ATAPI tells more. (ATA seems to be preferrable over ATAPI.) Although cdrecord issues complaints, addresses like "/dev/hdc" work with kernel 2.6 for all three programs. Such files may be listed by: wodim --devices cdrskin --devices xorriso -devices growisofs (for DVD) on kernel 2.6 accepts /dev/hdX files as of at least dvd+rw-tools-5.17.4.8.6 . On kernel 2.4 it is /dev/srN, which might occur on 2.6 systems too. During installation of a modern consumer distribution with Linux kernel 2.4 an IDE writer should be recognized automatically and ide-scsi emulation should get set up properly. If there are no real SCSI devices present, the addresses should be 0,0,0 for CD and /dev/sr0 for DVD. Then you are lucky and already done. If not (or if you are no consumer), you will have to do some sysadmin work. With a Linux kernel 2.4 or below, following text applies : How my IDE CD writer came to SCSI address 1,0,0 My Yamaha SCSI writer deceased in 2003 and i was forced to purchase an IDE writer (with ridiculous high speed for an incredibly cheap price). Now i needed to present it to cdrecord as another SCSI device. I did this with SuSE 7.2 using LILO as bootloader. With other systems this might be completely different. Refer to the manuals of your Linux distribution. The IDE writer (actually an ATAPI writer) is master at secondary IDE. Therefore in Linux : hdc LILO : File /etc/lilo.conf I added the kernel option "hdc=ide-scsi" to the append line. Like: append="...existing.options... hdc=ide-scsi" There was no old append line. So it looks now as this : append="hdc=ide-scsi" Then i ran the command lilo as superuser. SuSE 7.2 : File /etc/modules.conf There was a line : alias scsi_hostadapter off which i changed according to the manuals alias scsi_hostadapter ide-scsi My SCSI CD-ROM still works :)) SuSE 7.2 : File /etc/init.d/boot.local Since the module did not load automatically, i added to the boot script: /sbin/modprobe ide-scsi After a reboot, cdrecord -scanbus listed a second SCSI bus with a single device at address 1,0,0 . The new CD reading device became /dev/sr1 like with the old SCSI writer. That possibly depends on the SCSI addresses of old CD-ROM and writer. So you may have to correct /etc/fstab and softlinks in /dev (like /dev/cdrom and /dev/cdrecorder) if sr1 and sr0 get swapped by the new SCSI-Adress 1,0,0. Not a good idea was to attach the IDE writer as slave hdb of the hda disk. Transactions like blanking or fixating blocked the disk for minutes. Also disk performance was awful during data writing. I therefore transplanted my disk hdc to hdb and attached the writer to an IDE controller of its own. That new writer was a LITE-ON LTR-48125S (40x12x40) which shows convincing performance as well as a fine protection against buffer underrun. No special options needed with cdrecord_prog (an old cdrecord-1.6). So it should also work with other CD recording software based on cdrecord. With newer versions of cdrecord you may need to enable -driveropts=burnfree during the run of ./CONFIGURE (which will notify you about this feature). Meanwhile i am using a LG GSA-4082B DVD writer with SuSE 9.0 which during installation got nicely recognized and SCSI-emulated with address 0,0,0 resp. /dev/sr0 . It can be used as /dev/hdc resp. ATA:0,0,0 on a kernel 2.6 based RIP-Linux system. ----------------------------------------------------------------------- Appendix NET : scdbackup mainly is designed only to backup data on the computer with the CD burner. What to do with a network of computers ? If you want to backup data accessible via NFS or other file systems, you may run scdbackup on the computer with the CD burner and give it those NFS addresses. But there still is the issue of backups where you need to be local superuser and there also may be the need to backup the file systems directly without the additional filter of a network file system. A more general approach which uses SSH is presented here. It consists of normal scdbackup installations on each of the hosts for each user who wants to do backups. All formatting is done locally on these hosts and only the data stream is transported over the network to the remote burner. On the host with the burner drive there is a user identity with a scdbackup installation properly configured for the burner. This user identy has to offer SSH login to the scdbackup user on the burnerless machine in the network. The burner configurations of the login user will be in effect. I.e. locking, blanking, buffering and checksum handling will mainly be controlled by the burner host installation. Nevertheless the configured activities around burning will be performed on the client host according to its configuration, too. So a remote backup will get two checksum tags appended. The first one verifies the remote data transfer and the second one verifies the burner success. Verification option -auto is advised. If the burner is vulnerable to misburns caused by low data transfer, it is advised to configure a buffer file for that user. Network traffic may become slow due to external influences. After these preconditions are established and tested, the scdbackup user needs to do an own installation on the burnerless host. As address for the burner one may enter "-" (rather than "0,0,0" or "/dev/sr0"). The archiver program afio and an ISO formatter are needed to do the backup formatting. The role of the burn program will be fulfilled by scripts/pipe_to_ssh . On the burnerless machine, enter as "burn program" a string of the form username@hostname:passwordflag username and hostname are the usual ssh parameters. password_flag is either 0 or 1, depending on whether the SSH server asks for a password on login. If a password request is expected, there will be a sleep period of 15 seconds and an input request at /dev/tty (not stdin). So in that case you will be forced to attend the backup and press Enter once per media. If you have concerns about a SSH shell account without password then consider a specialized account without shell interpreter but rather a script which starts scdbackup -pipe_to_media resp. sdvdbackup -pipe_to_media . That will require different users for CD and DVD, of course. The appropriate remote_command and remote_option for pipe_to_ssh would be "-" each. Like : .../pipe_to_ssh ts4 burndvd 0 - - Other scdbackup tasks like verifying or blanking should be performed on the machine which got the appropriate drive. Usually the human operator will sit at the machine with the burner and use a SSH login to start the scdbackup runs on the burnerless hosts (which then use SSH to forward their data stream). There is no way to temporarily override the remode scdbackup settings. You will have to configure several login users to switch between usage models. Examples of configuration on the burnerless machine: To use the scdbackup installation of user thomas on host ts2 where the SSH server is expected not to ask for a password, check in advance whether this will work with SSH : ssh ts2 -l thomas scdbackup -version Run ./CONFIGURE_CD and answer the question "What CD burn program to use ? ( ... )" by input thomas@ts2:0 Additionally disable local blanking to avoid (harmless) error messages. "Automatically erase ... before writing ? (yj1/n0)" by input n Server ts4 asks thomas for a password on login. So use passwordflag value 1: thomas@ts4:1 Again, disable blanking. One may temporarily redirect backups to host ts2, user thomas and use its configured CD burner ("scdbackup" "-pipe_to_media") : export SCDBACKUP_CDRECORD="$(scdbackup -where scripts)/pipe_to_ssh ts2 thomas 0 scdbackup -pipe_to_media" \ export SCDBACKUP_BLANKEN=0 ----------------------------------------------------------------------- Appendix TUNING : Optimized Compilation versus Debuggable Compilation The binary programs of scdbackup are generated with optimization for low CPU load and no support for debugger programs. In case that the compiler option -O2 is suspicicous or debugging is desired, one may do (while the burners are not busy): cd $(scdbackup -where inst) export SCDBACKUP_COMPILE_MODE="-debuggable" ./CONFIGURE_CD and answer the usual questions by empty input lines. It has to be stressed that usually the runtime of scdbackup isn't determined by its cpu load but by disk performance and/or burner speed. Optimizations after Compiling Depending on size and structure of the backup area, scdbackup's planning phase can last a while. With nicely shaped file trees, the speed is mostly restricted by filesystem performance. Besides the speed there is the memory consumption of the planning run which can cause problems. Although scdbackup tries to economize on memory it eventually has to model large trees and directories which inherently need their space. In some special situations, nevertheless, scdbackup's implementation is to blame for long planning runs or for exaggerated use of memory. This is due to the fact that the balance between run time and memory consumption is tuned for average trees on normal Linux systems. If both, run time and memory are a problem then you should consider to divide the backup area into several ones and to backup each of them with a frequency which is appropriate to its importance and agility. As long as there are still reserves of either run time or memory, it might help to change the balance. Composition Memory Some backup areas contain very large trees which have to be split at deep subdirectory levels in order to obtain digestible pieces. In this situation it is repeatedly necessary to determine the sizes of subtrees by rescanning the appropriate part of the filesystem. This can be a big waste of time because it has to be redone for every directory level before the split depth is reached. Option -composition_memory allows to record subtrees up to a particular depth and with. So the sizes of these subtrees can be reused and there is no need to access the filesystem again. Regrettably this can use up substantial memory, so this buffering is quite restricted by default. If the processing after "beginning to compose backup volumes" lasts very long after reporting "... splitting /some/directory", then try a run with -composition_memory 5 1000 64m \ -pacifier -timestamp -pacifier -composition_memory The number 5 sets the depth of the provisionary buffer (5 directory levels). The number 1000 sets the width, i.e. the maximum size of a directory's file list so that it is allowed to be buffered. The number 64m allows to use at most 64 MB of memory for that buffering. Watch consumption of memory during the first test run. Wether the limit is obeyed in fact, depends much on the system's dynamic memory management. Watch for pacifier messages with the same directories again after the announcement "... splitting /some/directory". Sufficient buffering will reduce these repetions resp. will prevent them at all if the parameters are generous enough (and if you can affort that generosity). The pacifier line will have the format seconds directory_count bufferstate emerging_tree_size address bufferstate is composed of 3 numbers current_depth:number_of_buffered_directories:used_buffer_bytes If recording is disabled due to the limits, current_depth is displaid as "X". directoy_count is usually a single number. With large directories there may be a file counter appended after a "/". address is at or above the current crawler position and may be truncated in order not to exceed the total message lenght of 79 character. The pacifier messages emerge on stderr. You may direct them to a log file by scdbackup ... 2>&1 | tee -i /tmp/scdbackup_run.log and later view them (in a more readable layout) by $(scdbackup -where bin)/cd_backup_planer -debug_stderr_log \ /tmp/scdbackup_snapstamp ... the commands for creating the snapshot ... ... resume programs and services which have been halted ... scdbackup -level_timestamp /tmp/scdbackup_snapstamp -level 1 or ... timestamp=$(date '+%m%d%H%M%Y.%S') ... scdbackup -level_timestamp "$timestamp" -level 1 or just date it back one hour : ... scdbackup -level_timestamp -1h -level ... ----------------------------------------------------------------------- Appendix WRITER : The content of parameter *_cdrecord_value has to point to a program which in some aspects has to be compatible with cdrecord. This program will be started by scdbackup for performing several tasks and will get described these tasks by arguments suitable for cdrecord. So the most general demand is that it must tolerate any cdrecord option without aborting and without doing inappropriate things. In addition it may expect own arguments, which preceed the cdrecord arguments and set specific configuration parameters of the writer program. It has to be able to recognize where its arguments end and where those for cdrecord begin. This is achieved in most cases by prescribing a strictly fixed number of own arguments. The own arguments are set together with the program's address in *_cdrecord_value . In scdbackup's view they fixely belong to that program. A very simple writer will only evaluate its own few parameter arguments and skip all others up to the last one. If this last argument is "-", then the writer will take its input from stdin. If the last argument is the address of an existing file, then it will read that file up to EOF. In any other case it will do nothing and return exit value 0. Examples for this kind of writer are pipe_data_to and pipe_to_ssh . A writer may offer further capabilities by the following options, like it is done by growisofs_wrapper : -atip retrieve media type data and print DVD type as "book type:" CD types as " Is erasable" resp. " Is not erasable". blank=type blank a DVD-RW disk (type= "fast" or "all") -eject eject after work To be recognized by scdbackup, these must be announced in the output of option -help print help text and exit in a way that combined stdout and stderr match these grep expressions : .*-atip.*retrieve .*blank=type .*-eject.*after.*work Such a writer has to be able to fulfill all blanking operation requests in a senseful manner. growisofs_wrapper employs dvd+rw-mediainfo and dvd+rw-format. Other writers may well just discard such requests silently. It depends much on the needs of their target media. If the particular capabilities are not offered in the writer program's -help text then scdbackup will not request them but try to work around that lack and eventually even abort if this lack is fatal (e.g. automatic blanking with no fixed media type while writer offers blank=type but not -atip). The most complete writers, of course, are modern cdrecord and cdrskin or scripts which finally hand over all unknown arguments to said programs. An example is pipe_to_cdrecord_sao , which manipulates some arguments as well as the data input of a cdrecord run. Nevertheless, the growisofs_wrapper class above does cover all current needs of scdbackup. It is more compatible than old cdrecord_prog (no -atip). A writer may reply a version text by -version if only argument : print a version text which should match grep expressiom [Cc]drecord only if -atip, blank and -eject are supported properly.