Deploying User Mode Linux for

Linux Kernel Debugging

Preparing the Host System

Your Host Kernel Version

The host kernel is the Linux kernel that you are running on the machine you are working on right now. It is NOT the UML kernel, it is the Linux kernel that will be actually running the UML executable.

It is important for you to be using a clean, vanilla source code base when applying the patch (SKAS) in next section. If you are using RedHat (or some other distribution) and you are using their stock kernel, you should go to www.kernel.org and download a full vanilla version of the Linux source code. A vanilla version is a standard release that comes from www.kernel.org and not a kernel that RedHat gives you which was constructed from modified source code.

Your kernel source code should be located in /usr/src/linux-2.6.8.1 where the 2.6.8.1 portion is replaced by whatever version of the kernel you are running. You also need to make a symlink from this folder to /usr/src/linux.

Symlinking your code to /usr/src/linux:

ln -sf /usr/src/linux-2.6.8.1 /usr/src/linux

After you have downloaded, compiled, correctly installed, and successfully booted your system using your new vanilla kernel, continue with the next section.

SKAS Patch

Before you even download the UML source code, you should download and install the SKAS (Separate Kernel Address Space) mode patch. I've found that debugging the kernel does not work without it!

The SKAS patch should be applied to your host kernel. This is the kernel that your are running right now, NOT the UML kernel you will be creating. The SKAS patch makes changes to the host kernels operating environment to help solve some UML addressing problems.

Obtain the latest SKAS patch for your kernel version here:

http://user-mode-linux.sourceforge.net/dl-sf.html

You can get more information on SKAS mode here:

http://user-mode-linux.sourceforge.net/skas.html

In case you can't find an exact match for your kernel, try the closest match that does not exceed your kernel version. If you are using a 2.4.27 Linux kernel, download and install the 2.4.25 SKAS patch. It should work just fine.

Place the SKAS patch file in your /usr/src/linux folder and using the following command to patch your host kernel.

Applying the SKAS patch:

cat host-skas3-24.25.patch | patch -p1

Do not recompile your kernel yet, just ensure that the SKAS patch applied correctly. The patch program will tell you if there has been an error. In the event of an error, its probably best to erase your kernel source code and start with a fresh copy before retrying the patch process.

Kernel Options Needed

At this point you will need to ensure that your host kernel has been compiled with support for certain things that you absolutely need to get UML up and running. Please ensure the following options are enabled for your host kernel:

If you are running a 2.4 series host kernel:

Block Devices --> Loopback device support

Networking Options --> Network packet filtering

Networking Options --> IP: netfilter Configuration --> Connection tracking

Networking Options --> IP: netfilter Configuration --> IP tables support

Networking Options --> IP: netfilter Configuration -->Packet filtering

Networking Options --> IP: netfilter Configuration -->Full NAT

Networking Options --> IP: netfilter Configuration -->MASQUERADE target

support

Network Device Support --> Universal TUN/TAP device driver support

File Systems --> EXT3 journaling file system support

If you are tunning a 2.6 series host kernel:

Device Drivers --> Block devices --> Loopback device support

Device Drivers --> Networking support --> Universal TUN/TAP device driver

File Systems --> Ext3 journaling file system support

under this section

Device Drivers --> Networking support --> Networking options -->

You will need the following:

Network packet filtering

Network packet filtering --> IP: netfilter Configuration --> Connection tracking

Network packet filtering --> IP: netfilter Configuration --> IP tables support

Network packet filtering --> IP: netfilter Configuration --> Packet filtering

Network packet filtering --> IP: netfilter Configuration --> Full NAT

Network packet filtering --> IP: netfilter Configuration --> MASQUERADE target

support

Recompiling Your Kernel (Host kernel)

Now that you've enabled support for all this stuff, you should recompile your kernel and reboot so that are using the updated kernel.

UML Utilities

There is a set of user land utilities needed to use UML. Without them, you will have no way of interacting (via console) with the UML system once you get it booted. UML Utilities also provides other cool things that might come in handy. If you are using Gentoo Linux then you can simply emerge usermode-utilities. If you are using any other Linux distribution you should either check to see if this package exists for your distribution, or get the source code from http://user-mode-linux.sourceforge.net/dl-sf.html.

Debugging Software Needed

Before you can debug you will need a debugger. The debugger we will use is gdb. If you also wish to debug graphically, you'll need ddd which is a GUI for gdb. If you are running Gentoo Linux just emerge gdb and ddd. If you are running any other Linux distribution then either get your distributions packages for gdb and ddd, or get the source code from www.gnu.org/software/gdb/gdb.html and http://www.gnu.org/software/ddd/.

Downloading Patching and Compiling UML

Downloading UML

The UML code is already in the Linux kernel source tree. But before you can create a UML kernel, you'll need to download a brand new fresh copy of the source code from www.kernel.org or make a copy of your current source code. You cannot build UML with the same source tree that you use to build your host kernel. Instead, you should install a separate copy of the kernel source code in your home directory (~/uml-2.6.8.1) or in /usr/src/uml-2.6.8.1 where 2.6.8.1 is replaced by whatever kernel version you're using.

Note: The 2.4 series kernel code does not actually contain UML. But this is not a problem because we'll be downloading and applying a patch to fix that in the next section.

Patching UML

Even if you use the 2.6 kernel code, you will still need to download and install the latest UML patch which will undoubtedly fix bugs that might prevent you from being able to properly debug your UML kernel. You can get the latest UML patch from http://user-mode-linux.sourceforge.net/dl-sf.html. Again, just like with the SKAS patch, you'll want to grab the closest patch that does not exceed your kernel version, and if the patch fails, grab an entirely new copy of the source tree before trying to patch it again. The file

uml-patch-2.4.26-3.bz2 is what you want if you have a 2.4.27 kernel.

Download the file to your /usr/src/linux directory and use the following command to apply the patch.

Applying the latest UML patch:

bzcat uml-patch-2.4.26-3.bz2 | patch -p1

Cleaning UML

Now that you have the latest UML patch correctly applied, you're almost ready to begin configuring your UML kernel. Before we do that, we need to ensure that we are working with an absolutely clean source tree. You can ensure that this is the case by cd'ing into your UML source directory and executing the following command.

Cleaning UML source tree:

make clean mrproper

Configuring Your UML Kernel

To configure your UML kernel, run the following command:

make menuconfig ARCH=um

For the most part, you can configure your UML kernel how ever you want. Just make sure to enable the following options. Also, to keep things simple, do not use modules!

UML-specific options --> Tracing thread support

UML-specific options --> Separate Kernel Address Space support

UML-specific options --> Networking support

UML-specific options --> Kernel support for ELF binaries

UML-specific options --> Kernel support for MISC binaries

UML-specific options --> Support for host based filesystems.

UML Network Devices --> Virtual network device

UML Network Devices --> TUN/TAP transport

File systems --> Psuedo filesystems - /proc file system support

File systems --> Psuedo filesystems - /dev file system support

Kernel hacking --> Enable kernel debugging symbols

Kernel hacking --> Enable ptrace proxy

Here is a small (very small) list of things that you should make sure you don't enable. These are known to break the build (and/or debugging) process.

Kernel hacking --> Enable gprof support

The build process is very fragile and can be broken easily. If you find that UML will not compile for you, you should do a “make mrpoper clean” (this will erase your .config file which holds your kernel build configuration!) and try the configuration again, but remove things that you do not need.

Fixing UML Makefile

At this point, if you were to compile the UML kernel and try to debug it you would find that gdb would jump all other the place when you try and step through the code. This is because the kernel code is being optimized which causes the debugger to incorrectly step through the code at run time.

To prevent the optimizations from happening edit the file “Makefile” in the top level of your kernel source tree. The following sections should be erased or commented out.

#ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE

#CFLAGS += -Os

#else

#CFLAGS += -O2

#endif

#ifndef CONFIG_FRAME_POINTER

#CFLAGS += -fomit-frame-pointer

#endif

Remove the “-O2” component of both of the following lines.

HOSTCFLAGS = -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer

HOSTCXXFLAGS = -O2

Compiling UML

At this point you are ready to compile the UML kernel with the following command issued from the top level of the source tree.

make linux ARCH=um

After this make process finishes, you will have an executable file name 'linux' in the top level of the source tree. This file is your linux kernel. You can try executing it to see if it works. It should panic because we have not made a root file system for it yet.

Creating an Up To Date Gentoo File system

We will be installing Gentoo Linux into a file instead of a disk partition in order to provide a root file system for our UML system. We will be doing a stage1 install because we wish for the resulting file system to be as efficient and as up to date as possible. You may save time by doing a stage3 install.

Obtaining Installation Media

Download the latest stage3 tarball file (i.e. stage1-x86-2004.2.tar.bz2) from http://distro.ibiblio.org/pub/Linux/distributions/gentoo/ or any other Gentoo mirror. For a list of mirrors visit www.gentoo.org.

Preparing The UML File System

Create a file system of sufficient size to hold the Gentoo Linux install and whatever else you want to install. We will be doing a minimal installation so we'll settle on a 2GB file system which will leave us with about 1024MB left over. Be careful if you decide to shrink this, some packages require lots of temporary disk space to build properly. You can change the size of the resulting file by varying the '2048' in the dd command below. The units are megabytes.

Create the file to hold the file system:

dd if=/dev/zero of=~/gentoo-fs.img bs=1M count=2048

Create a file to act as a swap partition to the UML system:

dd if=/dev/zero of=~/gentoo-swap.img bs=1M count=512

Create an actual file system:

/sbin/mke2fs -jF ~/gentoo-fs.img

Create swap space in the swap file:

/sbin/mkswap -f ~/gentoo-swap.img

Creating and Entering the Gentoo Install Environment

mounting our Gentoo file system (you'll need to be root for this):

mkdir /mnt/genoofs

mount -o loop ~/gentoo-fs.img /mnt/gentoofs

Creating the base system from stage1 tarball:

cd /mnt/gentoofs

tar -jxvpf ~/stage1-x86-2004.2.tar.bz2

mount -t proc none /mnt/gentoofs/proc

cp /etc/resolv.conf /mnt/gentoofs/etc/

Entering the chroot environment:

chroot /mnt/gentoofs /bin/bash

env-update

source /etc/profile

Gentoo Installation

Obtain up to date copy of portage tree:

emerge sync

Bootstrap Gentoo system (takes hours):

cd /usr/portage

./scripts/bootstrap.sh

Obtain basic functional Gentoo system (takes hours):

emerge system

Final configuration Details

We just have to do a little configuration and then we'll be finished with the setup of the root file system,

The first task is to edit the /etc/fstab file. We need to make /dev/ubd/0 be our root file system. Also, /dev/ubd/1 should be the swap device. Here is an example fstab that should work for you:

# <fs> <mountpoint> <type> <opts> <dump/pass>

/dev/ubd/0 / ext3

/dev/ubd/1 none swap

none /proc proc defaults 0 0

none /dev/shm tmpfs defaults 0 0

Notice we use ubd/0 for root and ubd/1 for swap. This is correct, when we execute the UML kernel, we'll pass some command line arguments that set those devices to correspond to the correct files.

Installing a kernel message logger:

emerge sysklogd

rc-update add sysklogd default

Setting root password:

passwd root

Next we need to configure the UML system's network settings. Edit

/etc/conf.d/net and make the file look exactly like this:

# The IP of your UML system will be 192.168.0.1

# The IP of your host system (on the virtual private network)

# will be 192.168.0.254

iface_eth0="192.168.0.1 broadcast 192.168.0.255 netmask 255.255.255.0"

gateway="eth0/192.168.0.254"

Allow network device to initialize at boot:

rc-update add net.eth0 default

At this point you should have a fully functional basic Gentoo file system that you can use to boot your UML kernel.

Enabling Network Support for UML

In order to supply networking support from the host system to the UML system, we will set up the host system as a router and put the UML system on a private class B network. The UML system will set the host system as its default gateway and the host system will use NAT and IP Masquerading to perform routing services for the UML system.

Host System Configuration

You need to set your host system to act as a router for UML system. This consists of telling iptables to do routing and telling the kernel's ipv4 layer to do ip forwarding.

Setting host system as router:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

echo 1 > /proc/sys/net/ipv4/ip_forward

Note, these should be executed upon host system startup.

UML System Considerations

All you need to do to create a virtual eth0 device in your UML guest is pass the UML kernel the command line option “eth0=tuntap,,,192.168.0.254”.

Booting UML

Now that you've gotten your disk image, swap space, and network configuration all set up you can go ahead and boot the system from the UML source directory and ensure that everything is working ok.

Booting UML:

./linux ubd0=~/gentoo-fs.img ubd1=~/gentoo-swap.img

eth0=tuntap,,,192.168.0.254

Note: The above text represents a single command, not two separate commands.

Booting UML For Debugging

Starting A Debugging Session For UML Kernel

Start gdb ( or ddd):

gdb ./linux

ddd ./linux

Ignoring Signals

UML uses certain signals for special things and gdb needs to be told to ignore those signals or else you will constantly find you UML kernel being interrupted.

Tell gdb to Ignore certain signals:

handle SIGUSR1 pass nostop noprint

handle SIGSEGV pass nostop noprint

Setting a Break Point

At this point, you may want to set a break point so once you start the UML kernel running, you'll be able to stop it again by triggering the break point. You can also stop the kernel at any time by sending it an interrupt signal. See the section “Interrupting The UML kernel” for more on this.

Setting a break point:

break net/ipv4/tcp_input.c:2784

The above line will cause the UML kernel to break on the tcp_ack() function for the 2.6.8.1 kernel. You should set a break point at a place of your interest.

Starting the UML Kernel

Now that you've prepared the debugging environment you're ready to start the kernel running.

Start the UML kernel in gdb:

run ubd0=~/gentoo-fs.img ubd1=~/gentoo-swap.img

eth0=tuntap,,,192.168.0.254

Note: The above text represents a single command, not two separate commands.

Another Note: If the kernel stops executing right away because of some mysterious break point at proc_fs.h:177, this is OK. This is a feature left over from before SKAS mode and was put there to ensure that you could put a break point in before the kernel booted.

Interrupting The UML kernel

If you ever find yourself needing to interrupt the kernel and get back to gdb, you can do so by sending an INT signal (that's signal #2) to the UML kernel process from the host system.

First, you need to identify the UML main kernel process. This is easy. Just run 'ps aux' and look for first process under the gdb process that is named “linux”. In the example below, the process I want to interrupt is 1942.

jason 1935 0.0 0.4 2144 1280 pts/1 Ss 04:56 0:00 -bash

jason 1938 10.1 15.1 40860 38792 pts/1 S+ 04:57 0:01 gdb ./linux

jason 1939 0.0 0.4 2144 1252 pts/2 Ss 04:57 0:00 -bash

jason 1942 0.0 0.0 4064 12 pts/1 T 04:57 0:00 /home/jason/uml-2.6.8.1/linux

jason 1944 0.0 0.3 2324 808 pts/2 R+ 04:57 0:00 ps aux

To send the process (in the example, the PID is 1942) an interrupt signal use the kill command to send the PID a signal #2.

Interrupting a process with kill:

kill -2 1942

Stepping Through The Code

At this point, you should be able to step through the code and use any other gdb command.