Debugging with User Mode Linux

Comments on this tutorial may be directed to Michael McCabe <mccabemt@clarkson.edu>

Comments on this tutorial may be directed to Demetrios Dimatos <dimatosd@clarkson.edu>

This document, Debugging with User Mode Linux is copyright (c) 2006 by the Clarkson Open Source Institute. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is available at http://www.gnu.org/copyleft/fdl.html.

No liability for the contents of this document can be accepted. Use the concepts, examples and information at your own risk. There may be errors and inaccuracies, that could be damaging to your system. Proceed with caution, and although this is highly unlikely, the author(s) do not take any responsibility.

All copyrights are held by their by their respective owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Naming of particular products or brands should not be seen as endorsements.

You must have completed the previous tutorial Installing User Mode Linux

Without frame pointers GDB will not have all of the information that it needs and some debugging functions will not work properly.

I noticed that others who followed the tutorial had for some reason Show command line arguments on the host in TT mode enabled which in gdb resulted in this error:

	Program received signal SIGTRAP, TraceProgram received signal SIGTRAP, Trace/breakpoint trap.
	0xa001500a in _start () at mem.h:19
	19      {/breakpoint trap
		

I was able to fix this problem by disableing Show command line arguments on the host in TT mode

The second two things that involve signal handling are required because GDB will pause the program whenever one of these signals is received. Without this you will end up pausing several hundred times during the course of the boot process. This will make it next to impossible to successfully debug and boot the kernel

To set the signal handling for gdb we must begin gdb and the kernel.

	#gdb linux
	

	#handle SIGSEGV pass nostop noprint
	#handle SIGUSR1 pass nostop noprint
	

Now we can start the kernel by referencing the file system created from the previous turtorial.

	#r ubd0=/opt/uml/debian-root
	

At this point you are now ready to setup break points at various points throughout the Linux kernel. The functions that I set the breakpoints in are do_exit, do_fork, and copy_process. If you step through these functions you can see various important information about the process. Some of these things can include, the number of bytes read and written, and the numbers of different system calls. You can also access information about the size of the timeslice and the amount of cpu time that has been used.

Here is a short reference of some gdb functions.

Some things we tried:

Here I will briefly step through a couple of functions using gdb. This is by no means a detailed example, time is limited for me and this would take quite a bit of documentation. It is best you use my example as an introduction and proof of concept then being setting your own break points and exploring. If you want to view the source from other than a text window, try using an html linked tour of the code at Linux Kernel Source

I set two break points do_fork and alloc_pidmap in my exmaple.

	#gdb linux
	#handle SIGSEGV pass nostop noprint
	#handle SIGUSR1 pass nostop noprint
	#break do_fork
	#break alloc_pidmap
	

There is no particular reason why I chose do_fork other than its pretty it's pretty interesting and naturally I chose alloc_pidmap because it's a function call within do_fork and returns the pid. If you are unfamiliar with fork I suggest just doing a google on this, it is a basic operating system concept you should become familiar with.

	#r ubd0=/opt/uml/debian-root
	

At this point you should see the first break point catch.

	 Breakpoint 2, do_fork (clone_flags=8391424, stack_start=0, 
     regs=0xa04acf2c, stack_size=0, parent_tidptr=0x0, 
     child_tidptr=0x0) at kernel/fork.c:1237
     1237    {
	

Try doing a list to see what the source looks like in this break point.

	 (gdb)list
     1232                  unsigned long stack_start,
     1233                  struct pt_regs *regs,
     1234                  unsigned long stack_size,
     1235                  int __user *parent_tidptr,
     1236                  int __user *child_tidptr)
     1237    {
     1238            struct task_struct *p;
     1239            int trace = 0;
     1240            long pid = alloc_pidmap();
     1241
     1242            if (pid < 0)
     1243                    return -EAGAIN;
     1244            if (unlikely(current->ptrace)) {
     1245                    trace = fork_traceflag (clone_flags);
     1246                    if (trace)
     1247                            clone_flags |= CLONE_PTRACE;
     1248            }
     1249    
     1250            p = copy_process(clone_flags, stack_start, regs, stack_size, parent_tidptr, child_tidptr, pid);
     1251            /*
     1252             * Do this prior waking up the new thread - the thread pointer
     1253             * might get invalid after that point, if the thread exits quickly.
     1254             */
     1255            if (!IS_ERR(p)) {
     1256                    struct completion vfork;
	

Few things to notice, first note that the pid is set by the called funtion alloc_pidmap() which we have set a break point for and will be the next break afte we "next" through the code. Might want to check out out unlikely is defined also not the vfork (virtual fork ) in line 1256. Do you know what the difference is between fork and vfork? Little OS trivia!

	(gdb) n
    1240            long pid = alloc_pidmap();
    (gdb) n
    
    Breakpoint 2, alloc_pidmap () at kernel/pid.c:76
    76              int i, offset, max_scan, pid, last = last_pid;
    (gdb) print pid
    Variable "pid" is not available.
    (gdb) n
    80              if (pid >= pid_max)
    (gdb) n
    76              int i, offset, max_scan, pid, last = last_pid;
	

Notice after uisng next several times we came to where pid is set by alloc_pidmap(). At this piont the second break we set will be called and we will look closer at this using the same techine, "next" and "print". Print will display the value of the varialbe, so "print x" will reference "x".

    Breakpoint 2, alloc_pidmap () at kernel/pid.c:76
    76              int i, offset, max_scan, pid, last = last_pid;
    (gdb) n
    80              if (pid >= pid_max)
    (gdb) n
    76              int i, offset, max_scan, pid, last = last_pid;
	
	(gdb) print pid
    $5 = 300
    (gdb) n
    79              pid = last + 1;
    (gdb) print pid
    $6 = 300
	

Seeing as how the pid is being set by the funciton call alloc_pidmap(), we can do a list to see the code in the current state.

	(gdb) list
    74      int alloc_pidmap(void)
    75      {
    76              int i, offset, max_scan, pid, last = last_pid;
    77              pidmap_t *map;
    78      
    79              pid = last + 1;
    80              if (pid >= pid_max)
    81                      pid = RESERVED_PIDS;
    82              offset = pid &; BITS_PER_PAGE_MASK;
    83              map = &amppidmap_array[pid/BITS_PER_PAGE];
	

This ends our intro to using gdb to debug the kernel. The above example of a stepping through the code is a short intro but to really get a hanlde on this you should look over the source and find funtions that interest you and how they are used in the kernel and invastigate them using break points and some of the techniques I did above.