OS Hacking and more.

9.12.2008

OpenSolaris Kernel Modules: Load'em later

One of the biggest problems with the current OpenSolaris ZFS live boot is the size of the "minimal" microroot image used to boot the system to a useable state(to mount the root filesystem). It can be brought down to 170MBs decompressed, which is not too bad, except:
Loading the ramdisk image with grub is incredible slow over USB
And
When we chroot to the ZFS root filesystem, we relinquish the ramdisk, which hogs that much RAM and does no good.

A close examination of the microroot reveals that it is primarily filled by the contents of /kernel/* and /platform/*, especially those files in */drv/. These are definitely kernel modules and device drivers, which are not all necessary for finding the rootfs. Question is, which of these are actually necessary for the primitive boot process? and further, How can we convince kernel to fetch the rest of the modules from the ZFS root.
This would allow us to store only boot-essential modules in the ramdisk and keep it small.

And I have answers:
OpenSolaris' kernel loads modules and drivers "on demand", as they are called for by system services or applications. That means that it will take only the modules it needs, leaving others untouched until late in the boot. Moreover, OpenSolaris loads kernel modules from three distinct sources, /platform/kernel/, /kernel/, and /usr/kernel/. It is important to note that the modules are loaded from the filesystem, which means kernel also follows such filesystem links as loopback mounts(lofs). Finally, there is distinct set of modules that kernel will load before reaching /sbin/init, and these are for the most part boot essential.

Which means,
We CAN separate "non-essential" kernel modules from the ramdisk image loaded in a live usb boot, and allow kernel to load them from the ZFS filesystem later in the boot, when it is ready and the modules are actually needed. Telling kernel to load modules from the ZFS filesystem should be easy, except for the fact that kernel still thinks the root filesystem is the ramdisk when we are chrooted. In other words, userland thinks the ZFS filesystem is /, and kernelland thinks the ramdisk is /. This means kernel will search in [ramdisk]/kernel for modules rather than our /kernel when it wants them. However, by experimentation I discovered that kernel will follow loopback mounts to load modules, so in order to fool kernel into loading modules from [zfs]/kernel, all we need to do is loopback mount /mnt2/kernel on /kernel BEFORE we have switch root.

End Result:
We save ramdisk image space by storing more kernel modules on the zfs root and not loading them until we mount it.

I'll make the necessary procedure edits soon.

2 comments:

Derek Smith said...

Hi Anand,

Could you please provide the necessary information/scripts for creating a live usb version of openSolaris release 0811, preferably in one tar file?

Thanks.

Anand Gupta said...

Yes indeed. I'm posting a complete package in my latest post. Have a look there.