Read the original at my company’s blog.
Blue screens are not a rare commodity when working with virtualization. Most of the times, full crash dumps do the trick, but sometimes live kernel debugging is required. Hard disk related crashes that prevent memory dumping is a good example where it is required, but there are times where it’s just easier to follow the entire crash flow instead of just witnessing the final state.
Type 2 (hosted) virtualization usually comes with an easy solution. But type 1 (bare metal) virtualization, like Xen, complicates matters. Debugging must be offloaded to a remote Windows machine. The common solution, it seems, is to tunnel the hosted machine’s serial connection over TCP to another Windows machine where WinDBG is running, waiting anxiously for a bug check. There are many websites describing this setup in various component combinations. I have gathered here all the tricks I could find plus some more of my own to streamline the process and get rid of commercial software.
Lets dive into the nitty gritty little details, shall we?
Kernel debugging requires some boot parameters. Windows XP includes a utility called bootcfg.exe that makes this easy.
bootcfg /copy /id 1 /d "kernel debug" bootcfg /raw "/DEBUG /DEBUGPORT=COM1" /id 2 /a bootcfg /raw "/BAUDRATE=115200" /id 2 /a bootcfg /copy /id 2 /d "kernel debug w/ break" bootcfg /raw "/BREAK" /id 3 /a
This assumes you have only one operation system configured in Windows boot loader. If the boot loader menu shows up when Windows boots, you might need to add the flags on your own to C:\boot.ini.
Windows will now try to access the serial port in search of a debugger. Xen’s domain configuration file can be used to forward the serial port over TCP. Locate your domain configuration file and add the following line. The configuration files are usually located under /etc/xen.
The server side is set and it’s time to move on to the client. As previously mentioned, WinDBG doesn’t care for TCP. Instead of the usual TCP to RS-232 solution, named pipes are used here. I wrote a little application called tcp2pipe (download available on the bottom) which simply pumps data between a TCP socket and a named pipe. It takes three parameters – IP, port and named pipe path. The IP address is the address of the Xen host and the port is 4444. For named pipe path, use \\.\pipe\XYZ, where XYZ can be anything.
tcp2pipe.exe 192.168.0.5 4444 \\.\pipe\XYZ
All that is left now is to fire up WinDBG and connect it to \\.\pipe\XYZ. This can be done from the menus, or from command line.
windbg -k com:pipe,port=\\.\pipe\XYZ
To make this even simpler, you can use kdbg.bat and pass it just the IP. It assumes WinDBG.exe is installed in c:\progra~1\debugg~1. If that’s not the case, you’ll have to modify it and point it to the right path.
Source code is included in zip file under public domain.
Download tcp2pipe.zip (mirror).
2 thoughts on “Debug Xen Hosted Windows Kernel Over Network”
Thank you for the great piece of read and the software. Wouldn’t be better to have a pair program, like pipe2tcp, so that it would work without coupling with the hypervisor. Something functions like KDNET but not the same protocol behind. I do not request it by the way, I just like to think and discuss about it.
Thanks! I agree these days KDNET is probably an all around simpler solution.
For pipe2tcp, do you mean something running on the target? Wouldn’t that require a driver like KDNET?