AUTHOR: Mike McCarty Mike dot McCarty at sbcglobal dot net DATE: 2010-03-04 LICENSE: GNU Free Documentation License as of 2010-03-04. SYNOPSIS: How to and what to back up in an LFS system. PRIMARY URI: N/A DESCRIPTION: This hint is intended for those who need help planning a backup system for LFS. ATTACHMENTS: N/A PREREQUISITES: Have a bootable LFS system and external removable medium for storage of the backup. HINT: Ok, you've built your shiny new LFS system, and it boots. The book suggests that you back up your system, and then proceed to build whatever portions of BLFS appeal to you. Unfortunately, the book doesn't tell you what you need to backup, and how to do it. The first thing to take note of is that there are more than one thing called a "backup", and that they differ significantly from each other, and in what they are good for. One type of backup, sometimes called "bare metal" or "disaster recovery", is one which one can simply put in a CD-ROM, boot, answer a few prompts, and after some amount of time and CD-ROM swapping have the system back in the state it was in. This hint does not address that type of backup. Another type of backup, sometimes called "full" stores all the information which is needed in order to start from a freshly installed system, and then restore the system back to the state it was previously in. For an LFS system, in which "install" is synonymous with "build", this type of backup does not make much sense. Another type of backup, sometimes called "incremental", "differential", or similar terms, allows one, in conjunction with a "full" backup, to bring a freshly installed system back into a previous state, by first restoring from a "full" backup, and then applying changes from the incremental or differential backup. Again, since "fresh install" is synonymous with "build", this type of backup is not quite applicable to LFS, and is also not covered in this hint. Ok, so what is covered? This hint describes backups which allow you to recover from accidental deletion of files, and also do disaster recovery from bare metal, but do not themselves contain any means to boot. Both a "full" and "incremental" or "differential" style of backup can be created, but a separate boot medium must be supplied by the LFS user. The hint covers what needs to be backed up, and how to create a true "snapshot" which represents the system state at a single point in time. It does not cover the style of backup medium nor how to secure the backups themselves, though suggestions are made. The first consideration in any backup, is that it actually represents a single moment in time. Making the backup must be an atomic action, that is, one which appears to have taken place in an instant. The only way absolutely to guarantee this, is for all file systems to be in a consistent state, and for them not to be modified for the duration of the backup. More than one technique for accomplishing that have been suggested, but the simplest to achieve with an LFS system is simply to reboot to single user mode. How this is accomplished depends upon what you chose as your boot method, so the details of that can't be covered in full here. However, if you use GRUB, as suggested in the LFS book, then you can create an entry for single user mode by adding an entry to /boot/grub/menu.lst similar to this: # LFS 6.5 Single User Mode for Rescue and Backups title LFS 6.5 Single User Mode root (hd0,3) kernel /boot/lfskernel-2.6.30.2 root=/dev/hda4 single It is not adequate to boot your system and then use # init 0 since that leaves you still logged in as yourself, and you won't be able to unmount your file systems. You also need to make sure that root can log in, and has a good password. Reboot, and select the single user mode entry, and when prompted for the root password, enter it. You should then be given a single user mode maintenance prompt. Now that you are rebooted to single user mode, you need to unmount all file systems, except the root file system. So, for each file system you have mounted, simply do an unmount. Obviously, you can't unmount your root file system /, so it must be remounted read only. To find out what you have mounted, simply use the mount command. I'll give the example here of what happens when you have /dev/hda1 for /boot, /dev/hda2 for /, and /dev/hda5 for /home mounted. You'll need to adapt your procedure to your setup. # mount /dev/hda2 on / type ext3 (rw) /dev/hda1 on /boot type ext3 (rw) /dev/hda5 on /home type ext3 (rw) I've omitted the entries for /proc, and others, since these are not true file systems. You want to make note of each file system which corresponds to one of your hard disc partitions. Now, for each file system you intend to back up, except for /, which you have mounted, issue the umount command. # umount /boot # umount /home Finally, remount the root file system read only # umount -n -o remount,ro / You need the -n so umount doesn't attempt to modify /etc/mtab after the file system is marked read only. At this point, we have ensured that the file systems won't be modified by any background stuff going on, but we don't yet know that the file systems are in a consistent state. For that, we need to run fsck on each of the file systems. So, going through the list you got from mount earlier, run fsck on each file system. Still using the example above # fsck -c /dev/hda1 # fsck -c /dev/hda2 # fsck -c /dev/hda5 If you have other file systems which you sometimes mount, but which are not mounted automatically at boot, then use fsck on them as well. Be prepared for the fscks to take some time. If all the checks pass, then you are ready to begin your backup. You need to mount all your file systems read only, except for the one to receive the actual backup. It's common to put the backups in /var/backups. In that case, you need to mark the file system containing it as read write, and all others as read only. It's common, however, for / not to be large enough to contain the actual backups, so /var or /var/backups to be on a separate partition. In that case, you can leave / mounted read only. This example shows # mount -n -o remount,rw / # mount -o ro /dev/hda1 /boot # mount -o ro /dev/hda5 /home If you are not marking / as rw, and putting the backup in /home/var/backups, for example, you would use # mount -n -o ro /dev/hda1 /boot # mount -n /dev/hda5 /home In this case, you need the -n to prevent attempting to modify /etc/mtab when it is on a read only file system. Now mount any other file systems which are not automatically mounted at boot, and which you intend to back up, but be sure to mount them read only. At this point, you are ready to begin the backup procedure. Well, the obvious question is, what needs to be in the backup? The answer, unfortunately, is "It depends." Partly, it depends on how you set your system up, and it also depends on what you want on each backup set you make. However, for guidance to those who are relatively new to UNIX system's administration, here is a list of what you'll typically see on a freshly built LFS system, what is in each piece, and suggestions to help you determine whether you want to back it up. /bin this is the main program directory, and should be included in all backups which intend to back up the system itself, but not necessarily on data only backups. /boot this is usually a collection of kernels, boot configuration files, and the like, and should be included in all backups which intend to back up the system itself, but not necessarily on data only backups. /dev this is the devices directory, usually managed by udev on LFS systems, and should not be included in backups. /etc this is necessary configuration files and scripts and should be included in all backups which intend to back up the system itself, but not necessarily on data only backups. /home this is user information and data, and is not needed to back up the system itself, but would be included in all data only backups. /lib this is various library files, and should be included in all all backups which intend to back up the system itself, but not necessarily on data only backups. /lost+found this is a file system maintenance directory, and should not be included in any backups. /media this is used to hold mount points for various removable medium drives, like USB disc drives, floppy discs, CD-ROM drives and the like. Unless you are creating a backup which is intended to contain a file system which is mounted on, say, /media/my-usb-disc, then it should not be included in any backups. /mnt this is used as a temporary mount point by root, and normally should not be included in any backups. /opt this contains add on packages, and should normally be included in backups which are intended to back up the system add ons from, for example, BLFS, unless you put your add ons into /usr/local, or the like. /proc this is not a real file system, but rather a view port into the kernel's view of the system, and should not be included in any backups. /root this is normally root's home directory, and should be included in all backups which intend to back up the system itself, but not necessarily on data only backups. This directory should be pretty spare. /sbin this is the main maintenance program directory, and should be included in all backups which intend to back up the system itself, but not necessarily on data only backups. /sys this is not a real file system, but rather a view port into the kernel's view of the system, and should not be included in any backups. /tmp this is a temporary directory, and should not be included in any backups. /tools this is the main LFS build tools directory, and is not needed to back up the system itself, but would be included in all data only backups. /usr this is user information and data, and is not needed to back up the system itself, but would be included in all data only backups. In fact, you might not have anything in it at all, if you put all your add ons into /opt. You may have only man pages in it. /var this is user information and data, and is not needed to back up the system itself, but would be included in all data only backups. In fact, you might not have anything in it at all, if you put all your add ons into /opt. Note that, if you put you backup into /var/backups, then you need to be careful not to include /var/backups in the backup set. Ok, so now you've rebooted, and you've decided what to put into your backup set. How do you create your backup set? Well, that depends on what you like. I like to use tar, and use it to create backups. One tool which I've found very useful is yakup, which has very nice abilities to split the archive into CD-ROM or DVD-ROM sized pieces, and manage inserting and swapping media etc. That can be done after the backup is made, and with the system on line. Others may prefer to use cpio style backups. It's up to you how you chose to build your backup set. I highly recommend yakup, but that's just my personal preference. At this point, you have your backup set made, and are ready to reboot to normal user mode, after which you write your backup to removable media, unless it already is on one, like a USB disc. If you have your root file system mounted read only, then you need to remount it, so it can be marked "clean" when you go down. Otherwise, skip the first command below. # mount -o remount,rw / # reboot What you do now depends heavily on what other software you have installed. You need some way to get the backup set off your machine, and save it away, preferably at a remote location. I like to use DVD-ROMs for my backups, and so I have growisofs and cdrecord installed. How to do that is beyond the scope of this hint. One other thing you need to do is make a backup of the scripts or programs you use to restore files. If you simply use a tarball which fits on one DVD-ROM or CD-ROM, then this may not be necessary, as you'll need a boot disc, and it should contain tar. If you use a script like yakup, then one of your backup discs should have that script as one of the files on it, not in the tarball. Ok, disaster has struck! How do you get your information back on your disc? If all that has happened is you accidentally deleted a file, then simply pull that one file back from your backup set, using whatever means is appropriate for the format. This may be as simple as putting a DVD-ROM into the drive, issuing a mount command, and running tar. I recommend testing every backup set you make by restoring at least one file to another location, and ensuring that it contains the same data as the file it supposedly backs up. How about if, as root, you fat fingered # rm -rf /fred and typed instead # rm -rf / fred and your root directory is gone? You need to get a bootable CD-ROM with some version of Linux on it. I like to use KNOPPIX. Others like sysrescuecd. Almost anything will work. Boot your rescue medium, insert your backup medium, like DVD, into the drive, and mount it. This may require using two DVD drives, and is something you should think about before disaster strikes. Another possibility is to use something like Feather Linux, or Puppy Linux, which can load entirely into RAM, and free up the drive. Depending upon how serious the disaster is, you may need to repartition your disc, and create file systems. How to do that is highly dependent upon how you set your system up, and some of the details are in the LFS book. When you have your disc set up where it is ready to accept the restored files, then copy the restore scripts you intend to use, and use them to pull all files back from the backup. Recreate any directories you have on your system which are not on the backup proper, but are needed for boot. This is usually just /dev, which you don't back up, but udev needs in order to get set up. You then need to reinstall and set up your boot block, like GRUB, per its needs. At this point, you should have a bootable system. I highly recommend doing at least one simulated disaster recovery, using the rescue boot medium you hope never to have to use later, and verify that you can recover at least one file from your backup. A real test would require putting a blank hard drive in and recreating your system. Doing that is very instructive, but also requires some outlay in money. I highly recommend it. Don't be learning how to do disaster recovery while trying to recover from a disaster. ACKNOWLEDGMENTS: N/A CHANGELOG: Include timestamped changes that have occurred in the hint. Add latest entries at the end. Entries in this section should be as follows: * 2010-03-08 * * slightly obfuscate e-mail address * * fix error in mount of /dev/hda1 to indicate read only * * add information about recreating some necessary directories * which may not be present on the backup, but necessary for boot