Compatible Django Middleware

Django 1.10 added a new style of middleware with a different interface and a new setting called MIDDLWARE instead of MIDDLEWARE_CLASSES. Creating a class that supports both is easy enough with MiddlewareMixin, but that only works with Django 1.10 and above. What if you want to create middleware that can work with all versions of Django so it can be easily shared?

Writing a compatible middleware is not too hard. The trick is having a fallback for when the import fails on any earlier versions of Django. I couldn’t find a full example anywhere and it took me a few attempts to get it just right, so I thought I’d share my results to save you some time.

import os

from django.core.exceptions import MiddlewareNotUsed
from django.shortcuts import redirect

try:
    from django.utils.deprecation import MiddlewareMixin
except ImportError:
    MiddlewareMixin = object

class CompatibleMiddleware(MiddlewareMixin):
    def __init__(self, *args, **kwargs):
        if os.getenv('DISABLE_MIDDLEWARE'):
            raise MiddlewareNotUsed('DISABLE_MIDDLEWARE is set')

        super(CompatibleMiddleware, self).__init__(*args, **kwargs)

    def process_request(self, request):
        if request.path == '/':
            return redirect('/hello')

    def process_response(self, request, response):
        return response

CompatibleMiddleware can now be used in both MIDDLWARE and MIDDLEWARE_CLASSES. It should also work with any version of Django so it’s easier to share.

Stale MapReduce Staging Directories

I had a problem where HDFS would fill up really fast on my small test cluster. Using hdfs dfs -du I was able to track it down to the MapReduce staging directory under /user/root/.staging. For some reason, it wasn’t always deleting some old job directories. I wasn’t sure why this kept happening on multiple clusters, but I had to come up with a quick workaround. I created a small Python script that lists all staging directories and removes any of them not belonging to a currently running job. The script runs from cron and I can now use my cluster without worrying it’s going to run out of space.

This script is pretty slow and it’s probably possible to make it way faster with Snakebite or even some Java code. That being said, for daily or even hourly clean-up, this script is good enough.

import os
import re
import subprocess

all_jobs_raw = subprocess.check_output(
  'mapred job -list all'.split())
running_jobs = re.findall(
  r'^(job_\S+)\s+(?:1|4)\s+\d+\s+\w+.*$',
  all_jobs_raw, re.M)

staging_raw = subprocess.check_output(
  'hdfs dfs -ls /user/root/.staging'.split())
staging_dirs = re.findall(
  r'^.*/user/root/.staging/(\w+)\s*$',
  staging_raw, re.M)

stale_staging_dirs = set(staging_dirs) - set(running_jobs)

for stale_dir in stale_staging_dirs:
  os.system(
    'hdfs dfs -rm -r -f -skipTrash ' +
    '/user/root/.staging/%s' % stale_dir)

The script requires at least Python 2.7 and was tested with Hadoop 2.0.0-cdh4.5.0.

Download PDB by GUID

Sometimes you get stuck with a broken or no dump at all. You know what you’re looking for but WinDBG just keeps refusing to load symbols as you continue to beg for mercy from the all knowing deities of Debugging Tools for Windows. You know what PDB you’re looking for but it just wouldn’t load. The only thing you do know is that you don’t want to go digging for that specific version of your product in the bug report and build a whole setup for it just so you can get the PDB. For those special times, some WinDBG coercion goes a long way.

To download the PDB create a comma separated manifest file with 3 columns for each row. The columns are the requested PDB name, its GUID plus age for a total of 33 characters and the number 1. Finally call symchk and pass the path to the manifest file with the /im command line switch. Use the /v command line switch to get the download path of the PDB.

To demonstrate I’ll use everyone’s favorite debugging sample process.

C:\>echo calc.pdb,E95BB5E08CE640A09C3DBF3DFA3ABCB42,1 > manifest

C:\>symchk /v /im manifest
[...]
SYMSRV: Get File Path: /download/symbols/calc.pdb/E95BB5E08CE640A09C3DBF3DFA3ABCB42/calc.pdb
[...]
DBGHELP: C:\ProgramData\dbg\sym\calc.pdb\E95BB5E08CE640A09C3DBF3DFA3ABCB42\calc.pdb - OK

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

To force load the PDB you need to update the PDB path, turn SYMOPT_LOAD_ANYTHING on, and use the .reload command with /f to force and /i to ignore any so called mismatches.

kd> .sympath C:\ProgramData\dbg\sym\calc.pdb\E95BB5E08CE640A09C3DBF3DFA3ABCB42
kd> .symopt+0x40
kd> .reload /f /i calc.exe=0x00400000

You should now have access to all the data in the PDB file and stack traces should start making sense.

Android LXR

An open source OS makes debugging applications so much easier. Instead of firing up IDA and going through opcodes, you can simply read the code and sometimes even find comments. However, searching through millions of lines of code can be a daunting task. Operation systems usually have a huge codebase and even the simple task of looking for one function can take a few good minutes. After reading that function, you usually want to search for functions it calls or functions that call it to better understand the flow. Those extra searches take time too. A good IDE would solve this issue but it requires downloading and indexing the massive source code first.

LXR was created for this exact reason. It allows hosting a fully indexed copy of the source code. It even makes it easy to publish an index of multiple versions of the source code. Want to compare a certain function between two versions of the Linux kernel? No problem. Want to know which functions use a certain function? Easy. LXR is awesome and fast.

Setting up LXR on your own, however, does take some time and effort. That is why I was happy to find AndroidXref.com while trying to hunt down a bug in one of my Android applications. It indexes both Android and patched Linux kernel sources for all major versions of Android. It is an invaluable resource every Android developer should know.

I originally had a question about this topic open on StackOverflow with AndroidXref as the accepted answer. It was recently deleted, probably because it didn’t have anything to do with C operator precedence. This is my AndroidXref.SEO++.

Old GDB find

Newer versions of GDB come with the nifty find command. The old version of GDB I have to use does not. It is also incapable of generating a proper stack trace for the platform it supposedly serves. But that’s a whole other story…

Anyway, I found a piece of code that almost does the same. I tweaked it a bit, fixed the stray bug ($x -> %p) and would like to never do it again. So here it is for my future reference and your indulgent.

define find
  set $start = (uint64 *) $arg0
  set $end = $start + $arg1
  set $pattern = (uint64) $arg2
  set $p = $start
  while $p < $end
    if (*(uint64 *) $p) == $pattern
      printf "pattern %p found at %pn", $pattern, $p
    end
    set $p++
  end
end

Hello Android

Humanoid, search engine and one of the most addictive FPS games ever created walk into a bar. A few refreshing cups of coffee later, a joke is born and its name is MW2 Guide.

I’ve created a pretty simple application for Android that helps Call of Duty: Modern Warfare 2 addicts, such as myself, to make some sense of the bombardment of dialog boxes popping after a match. It’s basically a list of all available callsign titles and their description. What sets it apart from a few dozen similar apps is the quick search ability and auto completion voodoo, in accordance with Android’s search centric vision.

Search for MW2 Guide on the market, or use the QR code below.

MW2 Guide QR code

SCSIPORT debugging

Microsoft provides useful extensions for debugging SCSIPORT drivers in WinDbg. But with some versions of scsiport.sys, the symbol files don’t contain type information. This produces fun errors like the following.

kd> !scsikd.scsiext 8a392a38
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Your debugger is not using the correct symbols                 ***
***                                                                   ***
***    In order for this command to work properly, your symbol path   ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: scsiport!_DEVICE_OBJECT                       ***
***                                                                   ***
*************************************************************************
scsikd error (3): ...\storage\kdext\scsikd\scsikd.c @ line 188

This makes the common task of getting your device extension object very daunting. After some digging, I came up with this code to at least get my device extension object from SCSIPORT’s device extension object.

!drvobj mydriver
* get relevant DevObj
!devobj <devobj>
* get DevExt
dt mydriver!MY_DEVICE_EXTENSION poi(<DevExt> + b4)

I’ve only tried it on Windows XP SP3. The offset may be different with other configurations. Anyone knows a better way around this? Preferable method would naturally be making scsikd work.

Debug Xen Hosted Windows Kernel Over Network

Read the original at my company’s blog.

Blue screens are not a rare commodity when working with virtualization. Most of the times, full crash dumps do the trick, but sometimes live kernel debugging is required. Hard disk related crashes that prevent memory dumping is a good example where it is required, but there are times where it’s just easier to follow the entire crash flow instead of just witnessing the final state.

Type 2 (hosted) virtualization usually comes with an easy solution. But type 1 (bare metal) virtualization, like Xen, complicates matters. Debugging must be offloaded to a remote Windows machine. The common solution, it seems, is to tunnel the hosted machine’s serial connection over TCP to another Windows machine where WinDBG is running, waiting anxiously for a bug check. There are many websites describing this setup in various component combinations. I have gathered here all the tricks I could find plus some more of my own to streamline the process and get rid of commercial software.

Lets dive into the nitty gritty little details, shall we?

Hosted Windows

Kernel debugging requires some boot parameters. Windows XP includes a utility called bootcfg.exe that makes this easy.

bootcfg /copy /id 1 /d "kernel debug"
bootcfg /raw "/DEBUG /DEBUGPORT=COM1" /id 2 /a
bootcfg /raw "/BAUDRATE=115200" /id 2 /a
bootcfg /copy /id 2 /d "kernel debug w/ break"
bootcfg /raw "/BREAK" /id 3 /a

This assumes you have only one operation system configured in Windows boot loader. If the boot loader menu shows up when Windows boots, you might need to add the flags on your own to C:\boot.ini.

Xen Host

Windows will now try to access the serial port in search of a debugger. Xen’s domain configuration file can be used to forward the serial port over TCP. Locate your domain configuration file and add the following line. The configuration files are usually located under /etc/xen.

serial='tcp::4444,server,nowait'

Debugger Machine

The server side is set and it’s time to move on to the client. As previously mentioned, WinDBG doesn’t care for TCP. Instead of the usual TCP to RS-232 solution, named pipes are used here. I wrote a little application called tcp2pipe (download available on the bottom) which simply pumps data between a TCP socket and a named pipe. It takes three parameters – IP, port and named pipe path. The IP address is the address of the Xen host and the port is 4444. For named pipe path, use \\.\pipe\XYZ, where XYZ can be anything.

tcp2pipe.exe 192.168.0.5 4444 \\.\pipe\XYZ

All that is left now is to fire up WinDBG and connect it to \\.\pipe\XYZ. This can be done from the menus, or from command line.

windbg -k com:pipe,port=\\.\pipe\XYZ

To make this even simpler, you can use kdbg.bat and pass it just the IP. It assumes WinDBG.exe is installed in c:\progra~1\debugg~1. If that’s not the case, you’ll have to modify it and point it to the right path.

tcp2pipe

Source code is included in zip file under public domain.

Download tcp2pipe.zip (mirror).

Happy debugging!

SourceForge tracker and SVN integration

TortoiseSVN and some other Subversion clients have a cool feature for integration with issue trackers. By setting a few properties for a project, you can have automatically generated links to tracked issues directly from SVN commit messages. TortoiseSVN also supports a special input method for issue numbers in commit boxes for easier integration.

The basic property that must be set is bugtraq:url which lets SVN know how to form a URL from an issue number. This is a bit difficult with SourceForge, where there are in fact at least three trackers and each with its own URL scheme including the tracker identifier in the URL. Toying with the URL and removing both group and tracker identifiers results in an error. After playing around some more, I found the following URL that works like a charm for every group and any tracker so there isn’t even a need to figure out the group identifier.

http://sourceforge.net/support/tracker.php?aid=%BUGID%

It took me a while to figure out and I couldn’t find any mention of it online, so I thought I’d post it here to help others.

For more information on all of the related properties see TortoiseSVN’s page.