...making Linux just a little more fun!
Ben Okopnik [ben at linuxgazette.net]
Earlier today, I had a problem with my almost-brand-new Ubuntu 9.10 install: my netbook suddenly stopped booting. Going into "rescue" mode showed that it was failing to mount the root device - to be precise, it couldn't mount /dev/disk/by-uuid/cd4efbe9-9731-40a5-9878-06deff19af06 (normally, a link to "/dev/sda1") on '/'. When the system finally timed out and dropped me into the "initramfs" environment/prompt, I did an "ls" of /dev/disk/by-uuid - which showed that the link didn't exist. Yoiks... had my hard drive failed???
I was quickly comforted by the existence of another device in there, one that pointed to /dev/sda5, a swap partition on the same drive, but that could still have meant damage to the partition table. I tried a few things, none of which worked (i.e., rebooting the system would always bring me back to the same point)... until I decided to create the appropriate symlink in the above directory - i.e.,
cd /dev/disk/by-uuid ln -s ../../sda1 cd4efbe9-9731-40a5-9878-06deff19af06
and exit "initramfs". The system locked up when it tried to boot further, but when I rebooted, it gave me a login console; I remounted '/' as 'read/write', then ran "dpkg-reconfigure -plow ubuntu-desktop", the "one-size-fits-all" solution for Ubuntu "GUI fails to start" problems, and the problem was over.
NOW, Cometh The REALLY Big Problem.
1) What would cause a device in /dev/disk/by-uuid to disappear? Frankly, the fact that it did scares the bejeebers out of me. That shouldn't happen randomly - and the only system-related stuff that I did this morning was installing a few packages (several flavors of window manager so I could experiment with the Ubuntu WM switcher.) I had also rebooted the system a number of times this morning (shutting it down so I could take it off the boat and to my favorite coffee shop, etc.) so I knew that it was fine up until then.
2) Why in the hell did changing anything in the "initramfs" environment - i.e., creating that symlink - actually affect anything? Isn't "initramfs", as indicated by the 'ram' part, purely temporary?
3) What would be an actual solution to this kind of thing? My approach was based on knowing about the boot process, etc., but it was part guesswork, part magical hand-waving, and a huge amount of luck. I still don't know what really happened, or what actually fixed the problem.
I'd be grateful for any insights offered.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Mon, 16 Nov 2009, Ben Okopnik wrote:
> 1) What would cause a device in /dev/disk/by-uuid to disappear? Frankly, > the fact that it did scares the bejeebers out of me.
AIUI this link is created by "udev" on the Ubuntu system so your experience is truly difficult to understand. The initrd does not even contain a "/dev" directory --- the initrd scripts creates it and the relevant nodes when they run.
> 2) Why in the hell did changing anything in the "initramfs" environment > - i.e., creating that symlink - actually affect anything? Isn't > "initramfs", as indicated by the 'ram' part, purely temporary?
Under "udev" the /dev is a ram file system. In particular, the init script initrd:/scripts/init-bottom/udev moves the /dev from the initramfs to the real/final file system!
I can't see much of a solution. Here are some magical incantations that may help you diagnose/fix the problem. 1. This should backup your initramfs and and create a new one. You can then compare the two.
update-initramfs -u
2. To examine what there is in your initrd you can do:
mkdir /tmp/initrd
zcat /boot/initrd*img | cpio -idumv
cd /tmp/initrd
3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
Also check this with the entry that "grub" has for the kernel
command line. This may provide some hints of whether the block
device id's are somehow incorrectly specified.
Regards,
Kapil. --
Ben Okopnik [ben at linuxgazette.net]
On Tue, Nov 17, 2009 at 06:50:31AM +0530, Kapil Hari Paranjape wrote:
> Hello, > > On Mon, 16 Nov 2009, Ben Okopnik wrote: > > 1) What would cause a device in /dev/disk/by-uuid to disappear? Frankly, > > the fact that it did scares the bejeebers out of me. > > AIUI this link is created by "udev" on the Ubuntu system so your > experience is truly difficult to understand. The initrd does not even > contain a "/dev" directory --- the initrd scripts creates it and the > relevant nodes when they run.
That's exactly why I was so annoyed by this whole thing. Why wouldn't the scripts create that UUID link? Particularly since they did create /dev/sda1, etc.?
It bugs me because it a) violates my understanding of how the "initrd" process works, which I thought was pretty solid, and b) because the one thing I rely on to tell me what happened, the system logs, don't exist in an "initrd" environment. I know, I know - I'm asking for too damn much - but without logs, there's no way to do any forensics... and so no way to know what actually happened.
> > 2) Why in the hell did changing anything in the "initramfs" environment > > - i.e., creating that symlink - actually affect anything? Isn't > > "initramfs", as indicated by the 'ram' part, purely temporary? > > Under "udev" the /dev is a ram file system. In particular, the init > script initrd:/scripts/init-bottom/udev moves the /dev from the > initramfs to the real/final file system!
[Nod]
> I can't see much of a solution. Here are some magical incantations > that may help you diagnose/fix the problem.
Well, given that now the system works just fine, there won't be anything to diagnose. I just hate "magic" solutions like this: there's no learning going on there, nothing you can use later.
> 1. This should backup your initramfs and and create a new one. > You can then compare the two. > update-initramfs -u
OK, that makes sense. I suppose I can always rename the backup back to the original filename if the update breaks anything.
> 2. To examine what there is in your initrd you can do: > mkdir /tmp/initrd > zcat /boot/initrd*img | cpio -idumv > cd /tmp/initrd
Oh, nice! Thank you, Kapil - I hadn't thought of that. Although as I see it, the process of creating the UUID links is something that happens when the "initrd" scripts run, and is not a "permanent" part of the structure.
> 3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
[blink] That's... interesting. It seems that my /etc/blkid.tab is a broken link to /dev/.blkid.tab - and the latter doesn't exist.
Hm. This may be a clue, although I'm not quite sure of what the mechanism would be.
> Also check this with the entry that "grub" has for the kernel > command line. This may provide some hints of whether the block > device id's are somehow incorrectly specified.
No, the device IDs are actually correct - I checked that first thing. The onlty problem was that the /dev/disk/by-uuid/<UUID> for /dev/sda1 wasn't being created by the scripts, although the entry for /dev/sda5 was. When I plugged in a bootable USB thumbdrive and managed to boot to a console, /dev/disk/by-uuid contained an entry for the USB media and /dev/sda5 - and still no UUID link for /dev/sda1.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Suramya Tomar [security at suramya.com]
Hi Ben,
>> 3. Check the output of "blkid" and compare it with "/etc/blkid.tab". > > [blink] That's... interesting. It seems that my /etc/blkid.tab is a > broken link to /dev/.blkid.tab - and the latter doesn't exist. > > Hm. This may be a clue, although I'm not quite sure of what the > mechanism would be.
I just checked on my Ubuntu 9.10 setup and the /etc/blkid.tab file on my system is also a broken link to /dev/.blkid.tab.
So I think that this is probably a separate issue 'cause I am not facing the same problem as you. yet.
- Suramya
-- ------------------------------------------------- Name : Suramya Tomar Homepage URL: http://www.suramya.com -------------------------------------------------************************************************************
Ben Okopnik [ben at linuxgazette.net]
On Wed, Nov 18, 2009 at 01:25:39AM +0530, Suramya Tomar wrote:
> Hi Ben, > > >>3. Check the output of "blkid" and compare it with "/etc/blkid.tab". > > > >[blink] That's... interesting. It seems that my /etc/blkid.tab is a > >broken link to /dev/.blkid.tab - and the latter doesn't exist. > > > >Hm. This may be a clue, although I'm not quite sure of what the > >mechanism would be. > > I just checked on my Ubuntu 9.10 setup and the /etc/blkid.tab file > on my system is also a broken link to /dev/.blkid.tab.
OK, that's good data - thanks.
> So I think that this is probably a separate issue 'cause I am not > facing the same problem as you. yet.
I'm just noodling here, but... what if it went something like this?
a) Something somewhere somehow [1] hoses the data in /etc/blkid.tab - specifically, the data for the boot device. Since "blkid" (and more to the point, libblkid(3)) uses the cached block device data in /etc/blkid.tab, we now a problem - but it's not going to be visible until the next boot (SURPRISE!!!)
b) The "initrd" scripts would normally "pivot_root" over to the boot device, which would then run its own part of the boot sequence including setting up the UUIDs for the devices and mounting everything via "mount -a" - but that's not going to happen, since at least one boot-related program depends on the info from 'libblkid' to figure out the devices. In fact (and I had a bet with myself on this, which I won), here's the list:
ben@Jotunheim:~$ apt-cache rdepends libblkid1 libblkid1 Reverse Depends: samba4-testsuite samba4 pysdm pmount partitionmanager loop-aes-utils kvpm dump util-linux ocfs2console nfs-kernel-server mount libblkid-dev hal e2fsprogs
(Look at that: "mount". This logic chain is holding up so far.)
3) We now have a Catch-22 situation: the system "thinks" that we don't need to detect the block devices, since we have an "/etc/blkid.tab", etc.
So, the remaining question is, does "initrd" copy /etc/blkid.tab to the /dev of the "full" environment? Did my creating the device and doing a "mount -a" rewrite that file? Not having one (or having one that's a broken symlink, I'd guess) causes 'libblkid' to actually re-read the devices, according to the docs - so this works. Having a borken or non-empty cache file may well be the "poison pill" that causes the blowup.
If it ever happens again, I'll check that out. Perhaps, just for experiment's sake, I should go ahead and create a "mostly-valid" cache file and try rebooting...
[1] Does the exquisite precision of definition simply take your breath away, or what?
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Tue, 17 Nov 2009, Ben Okopnik wrote:
> > 3. Check the output of "blkid" and compare it with "/etc/blkid.tab". > > [blink] That's... interesting. It seems that my /etc/blkid.tab is a > broken link to /dev/.blkid.tab - and the latter doesn't exist.
That is certainly mysterious. While /etc/blkid.tab is indeed a link to /dev/.blkid.tab, the latter file should exist. (See /etc/blkid.conf for where this is configured)
Actually, I was wrong in my previous mail. The data in /etc/blkid.tab is the cached data used while running '/sbin/blkid' so the output of these is likely to be the same unless one uses '/sbin/blkid' to regenerate the cache. So perhaps you should run
/sbin/blkid -w /tmp/blkid.tabto see what file it will write and then write /etc/blkid.tab. It looks like this for me:
<device DEVNO="0x0802" TIME="nnnn" UUID="uuid1" TYPE="swap">/dev/sda2</device> <device DEVNO="0x0803" TIME="mmmm" LABEL="label1" UUID="uuid2" SEC_TYPE="ext2" TYPE="ext3">/dev/sda3</device> <device DEVNO="0x0804" TIME="llll" UUID="uuid3" TYPE="lvm">/dev/sda4</device>
If 'blkid' is managing to produce the correct output then you will have to look deeper ... ;-)
You may want to look through /lib/udev/rules.d/60-persistent-storage.rules which is the set of rules that creates the by-uuid links.
There are a number of cases in which the by-uuid rules seem to be skipped (look for GOTO="persistent_storage_end"). Perhaps this will help you to figure out some reason why the uuid link was not created.
Regards,
Kapil. --
Ben Okopnik [ben at linuxgazette.net]
On Wed, Nov 18, 2009 at 07:21:48AM +0530, Kapil Hari Paranjape wrote:
> Hello, > > On Tue, 17 Nov 2009, Ben Okopnik wrote: > > > 3. Check the output of "blkid" and compare it with "/etc/blkid.tab". > > > > [blink] That's... interesting. It seems that my /etc/blkid.tab is a > > broken link to /dev/.blkid.tab - and the latter doesn't exist. > > That is certainly mysterious. While /etc/blkid.tab is indeed a link to > /dev/.blkid.tab, the latter file should exist. > (See /etc/blkid.conf for where this is configured)
This, to me, is looking like a bug in the whole "blkid" caching concept.
> Actually, I was wrong in my previous mail. The data in /etc/blkid.tab > is the cached data used while running '/sbin/blkid' so the output > of these is likely to be the same unless one uses '/sbin/blkid' to > regenerate the cache. So perhaps you should run > /sbin/blkid -w /tmp/blkid.tab > to see what file it will write and then write /etc/blkid.tab. It > looks like this for me: > <device DEVNO="0x0802" TIME="nnnn" UUID="uuid1" TYPE="swap">/dev/sda2</device> > <device DEVNO="0x0803" TIME="mmmm" LABEL="label1" UUID="uuid2" SEC_TYPE="ext2" TYPE="ext3">/dev/sda3</device> > <device DEVNO="0x0804" TIME="llll" UUID="uuid3" TYPE="lvm">/dev/sda4</device> > > If 'blkid' is managing to produce the correct output then you will > have to look deeper ... ;-)
"blkid" definitely does the wrong thing here:
ben@Jotunheim:~$ /sbin/blkid -w /tmp/blkid.tab /dev/sda1: UUID="cd4efbe9-9731-40a5-9878-06deff19af06" TYPE="ext3" /dev/sda5: UUID="e9d48ae1-d67e-45f1-ab49-b36396287966" TYPE="swap" ben@Jotunheim:~$ cat /tmp/blkid.tab cat: /tmp/blkid.tab: No such file or directory
In other words, it prints the output to the screen instead of writing it to the specified file. This is contrary to what its man page says. I'll be filing a bug in a minute.
> You may want to look through /lib/udev/rules.d/60-persistent-storage.rules > which is the set of rules that creates the by-uuid links. > > There are a number of cases in which the by-uuid rules seem to be > skipped (look for GOTO="persistent_storage_end"). Perhaps this will > help you to figure out some reason why the uuid link was not created.
Actually, I've just managed to reproduce the initial problem I wrote about - and, if I recall correctly, in the same way that I originally created it. It still is a reportable bug against "blkid", and here's what happened.
Shortly before the first crash, I was trying to create a bootable USB stick for a friend (this is a surprisingly difficult task under 9.10. No, "usb-creator" doesn't work. No, "unetbootin" doesn't work. No, copying the CD contents plus running "syslinux" doesn't work. It requires ugly "fdisk" hacking, plus copying, plus "syslinux -s", and even then it only works sometimes.) Of course, in order to test this thing, I had to reboot again and again... and at one point, I forgot to set the BIOS to boot from the flash drive, at which point it started to boot from my hard drive.
(Here's the denouement; drum roll, please.)
Being the impatient sort, I said "no, dammit!", and held down the power button, aborting the boot half-way through. Now, I've done this before, countless times, and didn't think anything of it... but the next time I booted, the system couldn't find my /dev/sda1, or at least its UUID link. Given the whole "blkid" caching scheme, it's easy to see why.
As to reproducing it - last night, right before going to bed, I was
trying to create a bootable thumbdrive for myself (I guess I just like
pain) - and repeated the exact sequence from above (forgot to set the
BIOS, interrupted the boot.) Again, I got the same crash half-way
through - but this time, because of the discussion we've had, I looked
at "/etc/blkid.tab". Bingo: it only contained the entry for "/dev/sda5"
(my swap partition), and none for /dev/sda1. I did a "cat /dev/null >
/etc/blkid.tab", rebooted, and got the console login prompt - not the
perfect outcome, but one easily solvable with "sudo dpkg-reconfigure
-plow ubuntu-desktop". One more reboot, and I'm now back into the GUI
environment and typing this email. 
I've just checked "/etc/blkid.tab", by the way, and now it looks fine:
<device DEVNO="0x0801" TIME="1258548088" UUID="cd4efbe9-9731-40a5-9878-06deff19af06" TYPE="ext3">/dev/sda1</device> <device DEVNO="0x0805" TIME="1258548088" UUID="e9d48ae1-d67e-45f1-ab49-b36396287966" TYPE="swap">/dev/sda5</device>
So, again, the solution here seems to be
1) Delete the contents of /etc/blkid.tab
2) Reboot, and run "dpkg-reconfigure -plow ubuntu-desktop" if necessary.
Again, this whole dance shouldn't be necessary. Relying on cached device data _even when the system fails to boot_ is a bad, bad idea.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Wed, 18 Nov 2009, Ben Okopnik wrote:
> "blkid" definitely does the wrong thing here: > > `` > ben@Jotunheim:~$ /sbin/blkid -w /tmp/blkid.tab > /dev/sda1: UUID="cd4efbe9-9731-40a5-9878-06deff19af06" TYPE="ext3" > /dev/sda5: UUID="e9d48ae1-d67e-45f1-ab49-b36396287966" TYPE="swap" > ben@Jotunheim:~$ cat /tmp/blkid.tab > cat: /tmp/blkid.tab: No such file or directory > ''
This seems to be a bug in the handling of the '-w' option. The following alternate approach to generate the blkid.tab in an different location works.
/sbin/blkid -c /tmp/blkid.tab
This creates the /tmp/blkid.tab
> at which point it started to boot from my hard drive.
> (Here's the denouement; drum roll, please.)
> Being the impatient sort, I said "no, dammit!", and held down the power > button, aborting the boot half-way through. Now, I've done this before, > countless times, and didn't think anything of it...
Ctrl-Alt-Del or Alt-SysRq-S,U,O is probably safer.
> Given the whole "blkid" caching scheme, it's easy to see why.
> So, again, the solution here seems to be > > 1) Delete the contents of /etc/blkid.tab > 2) Reboot, and run "dpkg-reconfigure -plow ubuntu-desktop" if necessary.
You can also use:
/sbin/blkid -c/etc/blkid.tab(though even running /sbin/blkid should be enough!)
> Again, this whole dance shouldn't be necessary. Relying on cached > device data _even when the system fails to boot_ is a bad, bad idea.
I'm still mystified how this data or the lack of it is making it to the initrd. Some investigation is called for!
Regards,
Kapil. --
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Wed, 18 Nov 2009, Kapil Hari Paranjape wrote:
> > Again, this whole dance shouldn't be necessary. Relying on cached > > device data _even when the system fails to boot_ is a bad, bad idea. > > I'm still mystified how this data or the lack of it is making it to > the initrd. Some investigation is called for!
There is one more magic mantra that can be used to debug the Debian (and hence Ubuntu) boot process. That is to give 'break=<value>' option on the kernel command-line. (Don't forget to disable the 'splash' kernel option as well!).
Here <value> can be top, modules, premount, mount and init.
This will introduce a 'break-point' in the boot process at the designated place.
Using this, I was able to interrupt the boot proces on my system at the top to check whether there was any sign of the /etc/blkid.tab file. There was not. I then put a break at premount to check whether the /dev/disk/by-uuid links had been created. They had indeed!
AFAICS the initrd portion of the boot process does not use cached data from /etc/blkid.tab.
Moreover, the '/dev' directory is a tmpfs which is 'move'd from the initrd, so the links created during the initrd process should have persisted on the live system.
Finally, creating a link in this filesystem from the initramfs should have nothing to do with the link existing at the time of the next boot!
All in all the mystery groweth ...
Regards,
Kapil. --
Ben Okopnik [ben at linuxgazette.net]
On Wed, Nov 18, 2009 at 09:32:04PM +0530, Kapil Hari Paranjape wrote:
> Hello, > > On Wed, 18 Nov 2009, Kapil Hari Paranjape wrote: > > > Again, this whole dance shouldn't be necessary. Relying on cached > > > device data _even when the system fails to boot_ is a bad, bad idea. > > > > I'm still mystified how this data or the lack of it is making it to > > the initrd. Some investigation is called for! > > There is one more magic mantra that can be used to debug the Debian > (and hence Ubuntu) boot process. That is to give 'break=<value>' > option on the kernel command-line. (Don't forget to disable the > 'splash' kernel option as well!). > > Here <value> can be top, modules, premount, mount and init. > > This will introduce a 'break-point' in the boot process at the > designated place.Kapil, you're now our official guru for the boot process.
> Using this, I was able to interrupt the boot proces on my system at > the top to check whether there was any sign of the /etc/blkid.tab > file. There was not. I then put a break at premount to check whether > the /dev/disk/by-uuid links had been created. They had indeed!
So, somewhere in between those two, my system fails to create the UUID link for /dev/sda1 when I hard-reboot it - although it clearly does detect the device itself, and create the /dev/sda{1,5} entries for it. Moreover, it creates the UUID link for /dev/sda5.
I don't know the mechanism of this part well enough to understand how or why it would fail, though.
> AFAICS the initrd portion of the boot process does not use cached data > from /etc/blkid.tab. > > Moreover, the '/dev' directory is a tmpfs which is 'move'd from the > initrd, so the links created during the initrd process should have > persisted on the live system.
Well, once booted, everything looks fine - but that's a sort of a moot point, since it wouldn't have booted without it all being fine. The problem seems to be that there's no graceful fallback when there's no UUID entry. The closest thing to it is when (or if) you get a shell - but then, you have to know where things failed, etc.
> Finally, creating a link in this filesystem from the initramfs should > have nothing to do with the link existing at the time of the next > boot!
I agree. That's why I was so annoyed; I hate "magic" fixes.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Ben Okopnik [ben at linuxgazette.net]
On Wed, Nov 18, 2009 at 08:35:51PM +0530, Kapil Hari Paranjape wrote:
> Hello, > > On Wed, 18 Nov 2009, Ben Okopnik wrote: > > > Being the impatient sort, I said "no, dammit!", and held down the power > > button, aborting the boot half-way through. Now, I've done this before, > > countless times, and didn't think anything of it... > > Ctrl-Alt-Del or Alt-SysRq-S,U,O is probably safer.
The former didn't work - and I suspect the latter wouldn't have either,
since I was only about three seconds into the boot process. If it
happens again (and it just might, since I haven't actually managed to
create that bootable USB key for myself yet), I'll try the 'Alt-SysRq'
bit. Much like 'Ctrl-\' for those times when 'Ctrl-C' doesn't work,
'Alt-SysRq' is one of those key combos that most people know about but
never remember at the right time.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Wed, 18 Nov 2009, Ben Okopnik wrote:
> On Wed, Nov 18, 2009 at 09:32:04PM +0530, Kapil Hari Paranjape wrote: > > There is one more magic mantra that can be used to debug the Debian > > (and hence Ubuntu) boot process. That is to give 'break=<value>' > > option on the kernel command-line. (Don't forget to disable the > > 'splash' kernel option as well!). > > > > Here <value> can be top, modules, premount, mount and init. > > > > This will introduce a 'break-point' in the boot process at the > > designated place. > > Kapil, you're now our official guru for the boot process.
This is what comes out of staring too hard at the init script in the initrd, I suppose. However, I could have saved myself the trouble since there is documentation at 'man initramfs-tools' as I just discovered!
Regards,
Kapil. --