Freeze and crash recovery strategies
29 January 2005

For severe crash recovery, consider using Knoppix; more information under boot disk.

Debugging

# Start the program under gdb
gdb avidemux
# give the command that causes the problem
gdb> r --quit
# after the crash, get the backtrace:
gdb> bt

Send the backtrace to the developer.

Freezes

Magic SysRq button -- see /usr/src/linux/Documentation/sysrq.txt.

Brief instructions:

• hold alt, and then hold down SysRq as well
  
• If you don't have a SysRq key, it's the prt screen button
  
• After that, while still holding those two keys, push "s"
  
• Then wait a moment [disk activity here is considered a good thing]
  
• Still holding alt and SysRq, push "u"
  
• Wait a moment [disk activity here is also good]
  
• Then push "b"
  
• Read the file named above to find out what all this is about

Superblock failure

Great guide on how to recover from corrupt file system: http://www.oz.net/~trav/linux/lost_partition.html

The ext2 file system sometimes becomes corrupted and you get an error message about a corrupt superblock. If this happens first thing on a new kernel, take it as a hint that the kernel is non-functional, not that the file system is really corrupted (this happened to me on 29 May 2002).

If you realy have lost your primary superblock, you can do this to find and use one of the other ones:

1. Boot a rescue disk, so that you get a root file system other than your hard drive. The rescue disk situation is as follows. You seem to need two disks: a rescue disk and a root disk. The rescue disk provides initrc, and boot disk provides the file system. I found that Debian's disks work, but SuSE's don't. First insert rescue.bin, then root.bin (you have to make them with rawrite2 if you make them under Windows, which is simple enough). In root.bin you'll get a menu for installing Debian that you don't need, but you'll have the option of going to shell. There, you can check the partitions, as follows.

2. To see the extent of the damage, issue

e2fsck -n /dev/hdc7

3. To find a backup superblock address, issue in succession until you find a valid superblock (multiples of 8192 +1):

e2fsck -b 8193 /dev/hdc7
e2fsck -b 16385 /dev/hdc7
e2fsck -b 24577 /dev/hdc7
e2fsck -b 32769 /dev/hdc7
e2fsck -b 40961 /dev/hdc7
e2fsck -b 49153 /dev/hdc7

4. Keep a copy of your fstab file in Panel.

Recovery from corrupt HDD

How do I get my data back?

1) Update to something more recent that e2fsprogs 1.24 --- there have been a lot of bugfixes since then, especially in the arena of dealing with various "special" cases of corruption.

2) Assuming that you are using filesystems with 4k blocksizes, the backup superblock is at 32768. More modern e2fsck's will correctly fall back to the backup sb at 32768, but ancient versions (like the one you're using) don't; that's one of the bugs that since been fixed. :-)

You could *try* using e2fsck -b 32768 with your current version of e2fsck, and it *probably* will be able to fix things without doing any more damage, but without knowing more about how your filesystem got screwed up, using a more recent version of e2fsck would be the safer course of action.

Theodore Ts'o (who is the e2fsprogs maintainer)

I went and got the latest 1.29 version and installed it on gubbio, using ./configure --prefix=/     
That worked; the files went to /sbin, which is where SuSE had them.

Hard drive operations

You may need to check your harddrive (e2fsck) when it's faulty; it needs to be unmounted when you do that. That means you need another way of booting Linux and get a shell -- the simplest way may be to maintain a minimal partition for such emergency purposes.

Unable to resize your master partition?

In order to resize a partition in Linux, you need to run Linux on some other file system and have the partition to be resized unmounted. So what do you do when you run out of space on your master partition and need to enlarge it?

Well, what you should do is boot from a floppy or two, but I haven't yet figured out how to do that in Linux. What I actually ended up doing is creating a new partition in the free space and copying the whole thing over!

I followed the instructions in Hard Disk Upgrade Mini How-To, tailoring them to my situation.

I changed fstab and lilo.conf and that seemed to do it; I'm now on /dev/hdc8. Afterwards, I succeeded in resizing /dev/hdc7! I haven't tested the resized partition, but I assume I could now switch back to it if I wanted. In fact, that became a life saver (see below), even though in the meantime I had decided to start living on the new partition.

Note that this problem may be solved by LVM, which may be what you need to get more flexible management of partitions: Heinz Mauelshagen's LVM: Linux Volume Manager. LVM allows the growth and shrinkage of partitions, as one would expect, but also allows the creation of snapshot partitions, enabling consistent online backups or recovery in filesystems and database systems. http://linux.msede.com/lvm/

Unable to log into your computer?

In early June 2002, I erased pam and was unable to log into gubbio. After several deadends I realized I could just boot into /hdc7, the old master partition. I did so with no difficulty -- it was a choice in Lilo -- and mounted the current master partition, hdc8, as /root/mnt/test. I downloaded the pam rpms and moved them onto hdc8. Then I did this ingenious little thing:

chroot /root/mnt/test

This simply worked. It redefined /hdc8 as the root partition, and the rpms installed as if I had booted into that drive. RPM files go where they need to go just based on root I guess. I was even able to unmount hdc7 -- the partition I had booted into -- after the chroot. This suggests a simple way to resize a partition (see above): mount another partition, make it root with a chroot, unmount the first, and then resize it. The catch is presumably that you actually have to have a functional version of Linux on the partition you chroot to -- I wonder what would happen if you didn't.

I was subsequently able to boot as usual into hdc8, and a whole file system was saved -- one that I've put a lot of work into, and that performs beautifully.