Sep 13, 2012

Gentoo Boot Optimization


There is something Zen about boot time optimization. Let's face it, most of us don't reboot our Linux machines all that often. Yet shaving off a second or two from boot process gives me certain type of satisfaction.

Recent post on Google+ by Lukáš Zapletal made me try out bootchart for the first time. Original post was about e4rat - a tool for defragmenting ext4 partitions to optimize them for boot speed when using traditional rotational media. I decided to have a look at my bootcharts and see what can be done on my SSD based Gentoo system.

Bootchart Installation

I am not going to go into details. Just install from your distribution repositories. Gentoo contains app-benchmarks/bootchart2 in base portage tree, and Gentoo wiki has all the instructions you'll need.

First run

After I installed bootchart, my initial result was 11 seconds from init to X server running and showing password prompt. Let's analyze the Bootchart (200kB png) a little bit. It looks like slim was waiting for to finish. Not visible in the image, but that is actually net.eth0 script. In other words: network configuration, dhcp. Since I have stable IP address in my local network, I decided to stop using DHCP. For a simple Gentoo system this can be achieved by editing /etc/conf.d/net (configuring IPv4 and IPv6 statically):
    $ cat /etc/conf.d/net
    config_eth0="A.B.C.D/24 2001:470:413b:0:2e2:d4ff:ff8d:ccd1/64"
    routes_eth0="default via A.B.C.1"
So how are we faring after making our IP static? 5 seconds! At this point I'd be willing to say mission accomplished, but something told me there's more to do...

Let the fun begin

Looking at the second bootchart tells us one thing: xdm/slim is waiting for my ntfs partition to get mounted. We should probably avoid that!

What I decided to try was installing autofs and just mounting my /shared-data partition when it's actually accessed. To my dismay, the resulting bootchart showed that the boot got even slower! (5.6 seconds). Time for the big guns baby!


The problem with XDM seems to be that it is waiting for something. Let's have a look at /etc/init.d/xdm snippet:
depend() {
        need localmount xdm-setup

        # this should start as early as possible
        # we can't do 'before *' as that breaks it
        # (#139824) Start after ypbind and autofs for network authentication
        # (#145219 #180163) Could use lirc mouse as input device
        # (#70689 comment #92) Start after consolefont to avoid display corruption
        # (#291269) Start after quota, since some dm need readable home
        # (#390609) gdm-3 will fail when dbus is not running
        # (#366753) starting keymaps after X causes problems
        after bootmisc consolefont modules netmount
        after readahead-list ypbind autofs openvpn gpm lircmd
        after quota keymaps
        before alsasound

        # Start before X
        use consolekit dbus xfs
XDM seems to have a lot of dependencies. It is understandable because distributions will always prefer correctness over speed (hopefully). We are running a simple desktop. No network authentication, no heavyweight display manager or desktop environment like KDE or GNOME. So what happends if we remove netmount and autofs from requirements of xdm? After all we don't need them to start slim. Final bootchart is much more interesting. Roughly 3.5-4 seconds from init to X!

Quo Vadis

I finished with that 3.5-4 second boot. But as the final bootchart shows there's still room for improvement. List of things that could probably be looked into:
  • blkid takes too long. Perhaps we could avoid it completely?
  • e1000 (network card) and i915 (graphics card) take quite a while to initialize. Perhaps having e1000e as module and loading it later during boot would be faster
  • Not using LVM would speed things up, but I like its advantages
  • Avoiding udev could be useful as well for static system where no USB devices are to be attached dynamically
  • While we are at it, disabling USB completely would save around 250ms as well

Feb 21, 2012
Recently I wanted to make use of my 16GB usb drive in a sensible way, and I didn't really need another classic pendrive for moving data. In the end I decided to install BackTrack on it. BackTrack is a general forensic analysis/penetration testing distribution based on Debian. And it's fairly nice as far as a rescue distribution too.
I could have installed with with UNetbootin, which has direct support for BackTrack, but I wanted something a little more fancy: full disc encryption and persistence of data.
There is a very nice how-to linked from main BackTrack website for doing exactly this sort of thing. But I didn't want to burn the image first or even reboot. We have virtualization for that today! Right? Right! Or not...
So I downloaded BackTrack KDE/64bit variant iso, checked the md5sum to be correct, and started installation. Silly me thoght that running a KVM VM like this would make it possible to install BackTrack on the usb drive:
$ virt-install -n test -r 1024 --cdrom BT5R1-KDE-64.iso \
             --boot cdrom --nonetworks --graphics spice \
             --disk path=/dev/sdg
Where BT5R1-KDE-64.iso would be my BackTrack iso image and /dev/sdg would be my USB drive. Sadly this failed with ugly error message after BackTrack started booting:
# (initramfs) mount: mounting dev/loop0 on //filesystem.squashfs failed
After some investigation I found out that BackTrack booted fine if it was the only drive in the system, but failed with the above messages when I tried to attach my USB drive. Never found the reson, but the solution was to make the USB drive use virtio bus like this:
$ virt-install -n test -r 1024 --cdrom BT5R1-KDE-64.iso \
             --boot cdrom --nonetworks --graphics spice \
             --disk path=/dev/sdg,bus=virtio
After that I just continued according to the how-to with a few differences (such as USB key being seen as /dev/vda). Welcome our encypted overlords.

We've had Developer Conference (DevConf) in Brno last weekend and there have been numerous interesting talks and hackfests. You can see the full programme on Fedora wiki.
This year we've had a pleasure to welcome a lot of our colleagues from other Red Hat offices around globe. And they in turn had some of the most interesting talks. I spend most of my time at talks dealing with filesystems, storage and other core components, but there were a few not-so-technical talks that sparked my interest.
Bryn Reeves had two talks, one titled "Supporting the Open Source enterprise" and the other "How to lose data and implicate people". Sadly I had my own lab around fedora-review at the time of the second presentation, but if the first one was any indication the second one must have been great. The talk I've seen was dealing mostly with processes and tools our support uses to help customers deal with problems. And examples. Lots of interesting, fun examples of ingenuity of our engineers when dealing with bugs. Yeah, try replicating customer's setup of few thousand machine grid where the problem occurs. Apparently "git, git, git, git, git, git" is the tool that is saving their lives every day (not surprising).
Other talk that sparked my interest was Lukáš Czerner's "Btrfs - Design, Implementation and the Current Status". While Lukáš is not a Btrfs developer, he is a kernel developer familiar with its internals and the talk contained a lof of technical information I haven't known about before. "The root of the root of the roots" tree must probably be the motto of Btrfs. It looks to me that Btrfs has very powerful abstraction where everything is either a tree or node in a tree, but I guess only time will tell if this abstraction is going to be strong enough for the years to come. I am definitely looking forward to trying Btrfs in a controlled environment (for now).
Another feature that I was drooling over a bit was thin provisioning in LVM that was discussed in a talk given by Edward "Joe" Thornber & Zdeněk Kabeláč. It is a fairly recent feature (first upstream release with support for this was done in January) that allows one to thinly provide LVM volumes. What this means you ask? Well it means you can create 20 GiB volume "pool" that can contain 3 10 GiB thinly provisioned volumes. I.e. these volumes will start small and grow as needed. They will eventually also shrink when space is freed by the underlying filesystem. This of course requires the filesystem to support discard/TRIM commands, but this is not a problem for modern linux filesystems. As I see it thin provisioning teamed with snapshotting will change the way I manage my virtual machines for sure and I can't wait to try it out.
Many people believe Btrfs will take over role of LVM in following years, but the way I see it Btrfs will simplify use cases that LVM is too complex for, while LVM will keep on improving support for more demanding scenarios. Because let's be honest, LVM on desktop is just too darn complicated for an ordinary user/administrator.
I've been to FOSDEM few weeks back, and I have to say that DevConf was smaller, but no less interesting. There are projects or new features that I haven't heard about, but caused some "WOW" monents for me. Definitely looking forward to next year! (and you should try to come too)

Jan 26, 2012
I often browse through various Fedora packages and miss having git tags in package repositories corresponding with builds done in koji. Therefore I created following simple bash script that will create git tags in current git repo (if it's a Fedora package).

giturl=`fedpkg giturl`
if [ $? -ne 0 ];then
    echo "This doesn't look like fedora package directory"
    exit 1

pkgname=`echo "${giturl}" |\
         sed -e 's|git://\(.*\)?.*|\1|'`
# make sure we are up-to-date
git fetch

# go through last 3 releases (incl. rawhide)
for dist in f15 f16 f17;do
    builds=`koji list-tagged "${dist}" "${pkgname}" | \
        grep "${pkgname}" | awk '{print $1}'`
    for build in $builds;do
        # task urls sometimes have ".git" suffix
        git_sha=`koji buildinfo "${build}" | grep '^Task:' | \
            sed -e "s|.*${pkgname}\(\.git\)*:\(.*\))|\2|"`
        version=`echo $build | sed -e "s:${pkgname}-::"`
        echo BUILD: $pkgname\($version\) = $git_sha
        git tag "${version}" "${git_sha}"