Thursday, January 27, 2011

The mystery of the disappearing root.disk

In the past month or so I've noticed a small but noticeable number of users losing their root.disk files (or the entire \ubuntu\disks\ directory). The reason for this is unclear and, so far, the recovery rate is 0%.

In one case, booting from a live CD and running "ls /host/ubuntu/disks gave the following:
# cd /media/win/ubuntu/disks
# ls -l
# d??????? ? ? ? ? ? disks

It's hard to know from the other side of the internet what attempts have been made to recover. But suggestions to run chkdsk don't seem to fix the problem. It's also impossible to know whether this is a new Wubi problem or whether this is a regular issue that has always affected Wubi. Time will tell.

What can you do in this case? The best insurance is a regular backup of your root.disk file, or the important contents contained within.

On that note, I also saw a thread where a user tried to reinstall Wubi to fix a problem. Reinstalling Wubi will always remove the current install, delete the root.disk, and destroy any existing Ubuntu data/settings.

"Just uninstall"
Unfortunately I also see many community members advising Wubi users to uninstall because they assume the problem is Wubi-related. Often this is not the case, but rather it is a normal Ubuntu problem that can be solved (and would still have to be solved after replacing Wubi with a normal install). It is important to let these well-meaning members know that this advice could destroy the user's important data. It's not hard to add a cautionary warning, and perhaps suggest to the user how to recover data from a root.disk from Windows (e.g. http://ext2read.blogspot.com/) before uninstalling.

A final comment... in my opinion, reinstalling an OS is not a solution to a problem - it's the last resort when there is no solution. All too often I see users quickly reinstalling at the first sign of a problem e.g. when their master boot record MBR is overwritten by Grub2. This is the simplest thing to fix and doesn't damage installed OS's or their data, so if they could just resist the urge to give up and reinstall it would save them a lot of trouble.

And remember - you can rid yourself of Wubi, but not lose any of the look and feel, settings or data on your current Wubi install, simply by migrating. There is a new migration script coming soon that supports grub-legacy (Wubi installed prior to release 9.10 and later upgraded) as well as migrations from a live CD/USB simply by pointing at a root.disk file.

Sunday, January 16, 2011

WUBILDR, WUBILDR.MBR and GRLDR

It's fairly common to see messages such as Try (hd0,0) : FAT16 when booting Wubi. Usually it's innocuous, but sometimes booting fails. You might see it stuck on Try (hd0,0) : Ext2  


Rarely you'll see something like:
Try (hd0,0) : FAT16: No WUBILDR
Try (hd0,1) : NTSF5: No wubildr
Try (hd0,2) : FAT32: No WUBILDR
Try (hd0,3) : invalid or null
Try (fd,0) : invalid or null
ERROR: Cannot find GRLDR in all devices: Press ctrl+alt+delete to restart



So what does this all mean, and what is GRLDR? 


Wubi uses GRUB4DOS combined with Grub2 to boot Ubuntu. GRUB4DOS has two parts: grldr.mbr and grldr. Wubi has renamed these to wubildr.mbr and wubildr (you are free to rename them as long as you insert the new name of grldr in the .mbr file). 


The wubildr.mbr file corresponds to a grub legacy bootloader in the Master Boot Record. It is limited to scanning all partitions to find the rest of itself - the part that doesn't fit in the tiny MBR. These are the messages you see as it scans all partitions - and since it is based on grub legacy it outputs partitions with a zero index (unlike Grub2 which still uses zero based indexing on the drives, but not on partitions) i.e. hd0,0 in wubildr.mbr corresponds to hd0,1 in Grub2.


No matter where wubildr.mbr is, when it is called, it starts scanning the root on ALL partitions in the order dictated by BIOS to find the wubildr file. If it doesn't find it, you get an error that it can't find GRLDR, not WUBILDR. The reason it says GRLDR is likely a hardcoded message - for Wubi it's always WUBILDR that it can't find.


So what are the reasons for failure to find WUBILDR? 
1. The rather obvious, but unlikely - there is no WUBILDR 
2. Excessive fragmentation apparently
3. The wubildr file is > 137GB from the start of the partition
4. You have an ext4 partition that is read prior to the one containing WUBILDR.
WUBILDR.MBR cannot read ext4 partitions. The GRUB4DOS code it uses is from 2007 and it simply hangs when it encounters one of these. If you have an ext4 partition higher up in the BIOS order than your Windows partition, you're out of luck.


Some interesting points: when you install Wubi it installs WUBILDR.MBR and WUBILDR to the windows drive C:. But it also installs it to other partitions, like recovery partitions. If these happen to be physically before the windows partition, WUBILDR.MBR will find this WUBILDR and never use the one on your C: partition (it uses the first one it finds). And this is interesting as Grub2 updates the WUBILDR on your /host partition - not always the one used. Maybe this is a clue as to why Grub2 updates no longer boot on some installs.

Wednesday, January 12, 2011

Natty 11.04 Alpha 1 Upgrade with Wubi - Latest

I ran the Natty upgrade again today from my backed up maverick root.disk using the daily Alternate CD (January 12*). There are some interesting changes. For one, lots more errors during the upgrade including one saying the whole upgrade failed. But it didn't.

And a nice change is that as I logged in - forgetting to switch from Unity to the Classic desktop - there was a popup informing me that my hardware did not support unity and, would I like to switch to classic? Great! It didn't work, but a good idea all the same - and I assume it will work soon.

So after re-creating the 'logout launcher' I escaped Unity and switched manually to Classic Desktop and then it logged in fine (apart from having to reload all the crashing panel objects.)

Here are some pics from the upgrade process. The following popup appeared at least 3 times (and might have something to do with the fact that Natty has my keyboard set up as USA/Afghanistan now) ...


This one looked serious:

This one sounded like it was toast:


Despite the serious message - the upgrade worked, Natty booted first time... and nothing too exciting has happened since.

* I ran the upgrade from the daily image of the Alternate CD on January 10 and January 12 successfully. The images shown are from Jan 12; there were no material differences that I've noticed between the two. When I attempted to do it on Jan 11 it wouldn't allow the upgrade to begin due to some package with an incorrect version. The message is, if you find that the upgrade fails one day, just wait and try the following day (this is normal for alpha testing since the repositories are updated frequently and sometimes packages get out of synch). Also, be careful if you are offered a Partial Upgrade when running the Update Manager in Natty. Sometimes this is required but often it's an out of synch set of updates so it's best to be sure - use apt-get update and dist-upgrade to figure out what will be removed and decide whether it's an error or required.

Saturday, January 8, 2011

Wubi and hibernation

Wubi doesn't support hibernation... out of the box. But as Agostino Russo (all hail the creator) points out this is only because Wubi uses swap files, not partitions (if you could partition, why use Wubi?)

So yes it's true you can hibernate a Wubi install. And if you're just using it to test, you may well already have a swap partition lying around, so why not use it? Especially with 10.04 and 10.10 that seem to be a bit heavier on memory use than 9.10.

To set up hibernation on Wubi is the same as on a regular install. Just take the UUID of the swap partition and make sure it's in /etc/fstab and /etc/initramfs-tools/conf.d/resume. Then run "update-initramfs -u" to update the initial ram disk to point to it and look for it to resume from hibernation.
For example, assuming your swap is /dev/sda7:
$ sudo blkid /dev/sda7
/dev/sda7: UUID="xxxxxx" TYPE="swap"

$ cat /etc/fstab | grep swap
UUID=xxxxxx none swap sw 0 0

$ cat /etc/initramfs-tools/conf.d/resume
RESUME=UUID=xxxxxx

# if that's all good
$ sudo update-initramfs -u

Hint: if you are already using the Swap partition elsewhere don't run mkswap again or it will change the UUID.

Note: if you don't have a swap partition lying around and want to create one, make sure that it's big enough - to be safe I'd say the size of your RAM plus 1GB for overflow. Then run "sudo mkswap /dev/sda#" substituting # for your new swap partition and continue with the instructions above.
But if you're going to start creating partitions, I'd just take it one step further and migrate the Wubi install.

Thursday, January 6, 2011

Warning on modifying Wubi Boot menus

When you install Ubuntu with Wubi, there really should be a release note popup - and it should contain some warnings:
1. Make sure you backup
2. Don't mess with the boot menus
3. Don't update packages grub-pc and grub-common

Well what can you say about point 1: Obvious. Point 3: covered ad nauseum already. But let's look at Point 2.

For some reason, many people are irritated about there being 2 boot menus. And they decide to remove one of them. And as I'll explain, that's not a good idea.

The first thing they try is to remove Grub. But grub is a little too smart for that - it prevents itself from being hidden if it detects multiple OS's. So you're out of luck unless you happened to find Ulrich Meierfrankenfeld's nice workaround.

So the next thing they try is to mess with the Windows Boot Manager. They have already noticed that the Grub menu already has an entry for Windows, so - they think - why not just set Ubuntu as the default in the Windows Boot Manager, and then set the Timeout to zero.

What they fail to realize is that the Windows option in Grub, does indeed boot Windows, however, Windows boots using it's Boot Manager - and they've just set that to boot Ubuntu automatically without a delay or prompt. Bye bye Windows.

So what can you do?
Apparently holding down the F8 key doesn't work as it's supposed to...

So if you're on XP, you can manually edit the boot.ini file from within Ubuntu to change the timeout - just be VERY careful - if you add the wrong line endings (i.e. carriage return/newline) then you'll be able to boot nothing - just get a HAL.DLL error.

If you're on Vista/7 I think you're out of luck. You will need a Windows repair disk and use the command line bcdedit to modify the timeout.

Final note, don't install burg on Wubi. It tends to install itself in the drive Master Boot Record and this stops your computer from booting.

Monday, January 3, 2011

More detailed Wubi grub analysis

Problem
A fresh Wubi 10.04.1 install will fail to boot after updating package grub-pc. When you select Ubuntu, the computer reboots, hangs or drops you to a grub prompt. This is nothing new - many people have been affected by this bug.

Analysis
A fresh Wubi install has two files in /boot/grub: grubenv and grub.cfg. That is all. When you update grub-pc, the script grub-install runs and this copies all the grub modules etc. into /boot/grub. This results in a modified grub.cfg as some of the scripts directly reference objects in /boot/grub, and the result is a Wubi that cannot boot.

For a fresh 10.04.1 install, the differences in grub.cfg before and after updating grub-pc are:
31a32,54
> insmod ntfs
> set root='(hd0,2)'
> search --no-floppy --fs-uuid --set 80e4e195e4e18daa
> loopback loop0 /ubuntu/disks/root.disk
> set root=(loop0)
> if loadfont /usr/share/grub/unicode.pf2 ; then
> set gfxmode=640x480
> insmod gfxterm
> insmod vbe
> if terminal_output gfxterm ; then true ; else
> # For backward compatibility with versions of terminal.mod that don't
> # understand terminal_output
> terminal gfxterm
> fi
> fi
> insmod ntfs
> set root='(hd0,2)'
> search --no-floppy --fs-uuid --set 80e4e195e4e18daa
> loopback loop0 /ubuntu/disks/root.disk
> set root=(loop0)
> set locale_dir=($root)/boot/grub/locale
> set lang=en
> insmod gettext


By resetting /boot/grub back to normal, the grub.cfg that is generated boots fine:
sudo mv /boot/grub /boot/grubold # backup /boot/grub
sudo mkdir /boot/grub # make a new /boot/grub
sudo cp /boot/grubold/grubenv /boot/grub # copy only required file
sudo update-grub # regenerate grub.cfg

That's 10.04.1...

With 10.10, there are some script changes in Grub. These result in a new function in grub.cfg: 
function load_video {
}

This is empty by default on Wubi installs. Actually it's empty on my normal Maverick install too. But the same fix on 10.10 for Wubi fixes boot problems on Maverick - regardless of the empty function.
PS you don't have to fix 10.10 unless you upgraded to 10.10 from 10.04.1 or - if you want to break it - just reinstall the same version of grub-pc.  But almost guaranteed the next update to grub-pc on Maverick will break it unless a fix for Wubi is provided beforehand.

11.04  Natty Alpha
On Natty, the grub.cfg is syntax checked by 'update-grub' (actually grub-mkconfig) before the update is completed. It finds and rejects this empty load_video function, and results in the grub.cfg NOT being generated at all. All you get is grub.cfg.new.
This affects a brand new Natty Wubi install, so there is no grub.cfg at all. Applying the same fix as for 10.04.1 does not work, as update-grub never completes. You can rename grub.cfg.new to grub.cfg and it boots, but this is a manual step, so not a fix. In any case, it seems that Natty Wubi will boot with the full complement of Grub modules in /boot/grub. It spits out a bunch of errors, but completes. When these errors cause reboots and when they are harmless is unclear but I guess it has to do with the WUBILDR file.

In any case, Natty is early Alpha, so we should expect weirdness. But if you find yourself confused, you're not alone.

Summary
On supported releases 10.04.1 and 10.10, Wubi installs fail after updating package grub-pc. The current community-provided workaround fix does not work on Natty - this is only significant because fixes are usually applied to development releases before supported releases, and to get the fix into 10.04.1 we probably need a fix that will work on all.

Sunday, January 2, 2011

Upgrade to Natty 11.04 Alpha 1 with Wubi

An alternative to installing Wubi 11.04 Natty Alpha 1 from scratch - which is a convoluted process at the moment... is to upgrade from a current Maverick install. This bypasses the bad wubildr and Ubiquity installer issues. It does take a little bit longer to get setup, but it's quicker to repeat the upgrade.

Warning: with Alpha testing, something that works one day, may break the next. Please don't attempt this unless you have a dedicated test machine. The following example installs Wubi to the same partition as Windows (C:) - there is a difference, see note at bottom.

Step 1 - Install Wubi 10.10 Maverick
I picked 6GB for my install size.
Run all available updates.
Boot back into Windows and backup the root.disk and c:\wubildr (for repeat testing)

Step 2 - Download the daily 11.04 Natty Alternate CD
Since you'll probably want to repeat the upgrade at some point, it's a good idea to use zsync to keep your .iso up to date. If this is a once off, you could skip this step and just run the update online, but it takes much longer, in my experience than just downloading the alternate CD.

To download the 32-bit alternate CD:
zsync http://cdimage.ubuntu.com/daily/current/natty-alternate-i386.iso.zsync
See this for more info on zsync: https://help.ubuntu.com/community/ZsyncCdImage

Step 3 - Upgrade to Natty
Boot into the Wubi maverick install and run the upgrade from the alternate CD:
sudo mount -o loop natty-alternate-i386.iso /mnt
sudo /mnt/cdromupgrade

When you start the upgrade it asks you whether to check the internet for the latest packages - it's not really necessary since you have the latest daily image - and there's less chance of catching a buggy update - plus it's much quicker running just off the alternate image.

During the upgrade you get prompted a few times, but nothing exciting happened. I noticed by watching the grub.cfg file get regenerated (at least) 3 times and seeing the exception output, that the /boot/grub folder remains empty until the final time, when grub-install runs. This updates the wubildr and also places all the grub modules in /boot/grub. At the end the grub.cfg is regenerated in the same way that breaks wubi installs on 10.04.1 and upgrades to 10.10.

Step 4 - Reboot into Natty
The Wubi install didn't shut down when I requested a restart - it needed a little push (ALT+SysRq R-S-U-B). On restart, as expected due to the grub2/Wubi problems already covered in other posts, Wubi doesn't boot (but I had to try). I ended up at a grub prompt (not a rescue prompt), which is easy to boot from. I then chose the Classic desktop to login to, and had to click on multiple "Reload" buttons to fix the broken panel objects.

Welcome to Natty!!!

Step 5 - A new Grub2 wrinkle
So the accepted fix with Wubi installs is to clean out the /boot/grub directory and regenerate grub.cfg. This does spit out one error (on 10.10) but it's not an issue - about a missing /boot/grub/video.lst. However, with Natty, now Grub checks the syntax of the grub.cfg and rejects it if it finds errors. So the empty load_video function causes a syntax error.

You can get around this by leaving video.lst in /boot/grub, but that just spits out errors and reboots your machine when trying to boot your install. So... instead you have to manually rename grub.cfg.new to grub.cfg. Alternatively you can patch /etc/grub.d/00_header. What a pain! There must be an easier way to do this!!!

The instructions to patch grub, and the new command to manually rename grub.cfg.new follow:
sudo mv /boot/grub /boot/grubold
sudo mkdir /boot/grub
sudo cp /boot/grub/grubenv /boot/grub
sudo update-grub
sudo mv /boot/grub/grub.cfg.new /boot/grub/grub.cfg

That's about it. Natty boots fine after that, apart from having to click RELOAD on all the panel objects each time. Don't forget to select the "Classic" desktop unless you have a decent test machine that can handle Unity.

PS another nice thing is the Firefox beta. It looks very similar to Chrome - which is good for Firefox.

NOTE: if you install Wubi to a partition other than Windows, there is no way to update the wubildr file that is required to boot the Wubi install. Likely the upgrade will fail as the grub.cfg file has changed radically (I haven't tested it yet, but it pays to be cautious) and my previous test showed that the Maverick wubildr did not work. There are ways to generate a new wubildr file if you need to, but that's not covered in this article.