Linux’owy dramat w 3 aktach
August 26, 2008 10:47 pm wszystkoAkt I
Scena I
# dmesg| tail
EXT3-fs error (device xvda): ext3_readdir: bad entry in directory #376470: directory entry across blocks - offset=0, inode=3085352006, rec_len=62128, name_len=235
Aborting journal on device xvda.
ext3_abort called.
EXT3-fs error (device xvda): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
W tym momencie przestają działać wszystkie serwisy www stojące na serwerze.
Scena II
# fsck / -> wszysto ładnie i clean, tylko że dalej / jest zamontowany tylko do odczytu
Scena II
# reboot
(w czase rebootu zaczynam sie zastanawiać gdzie/czy jest backup i jak często był robiony.
Akt II
Scena I
>ping 1.2.3.4
Pinging 1.2.3.4 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Scena II (tragiczna)
(recovery console)
INIT: version 2.86 booting
mount: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
touch: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
mount: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
mount: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
* Filesystem type ‘devpts’ is not supported. Skipping mount.
mount: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
umount: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
Activating swap…failed.
mount: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
Setting the system clock..
date: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
System Clock set. Local time: .
/etc/rcS.d/S12mtab.sh: line 144: /proc/mounts: No such file or directory
Cleaning up ifupdown…rm: cannot remove `/etc/network/run/ifstate’: Read-only file system
Loading device-mapper support.
Checking file systems…fsck: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
fsck died with exit status 127
failed (code 127).
* File system check failed.# fsck
fsck: relocation error: /lib/libpthread.so.0: symbol _h_errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference
Scena III
(Recovery Finnix Image)
# fsck /dev/costam
fsck: costam było źle, ale naprawiłem i jest ładnie
Scena IV
(patrz scena II)
Scena V;
srv://lost+found# ls |wc -l
1053
po przyjrzeniu się zawartości lost+fount okazało się, że jest tam śmiertelna mieszanka /lib /var/www i troszke z /usr. Bohater umarł.
Akt III
Błąd w Xen, coś z rozmiarem bloku ext3. Cały dzień zmarnowany.
