Apache Modules, SEOversite and Debugging C

At Axiom Software we’re working on the next release of SEOversite. It’s been exciting. This is basically a tutorial on debugging the Apache HTTPd side of things (keep in mind that this is as much an informational thing for you as it is a reminder for me. I assume that I’m going to have to re-read this at some point).

Background

In production, we’ve been using Apache on Ubuntu Server. This has worked out well seeing as aptitude rocks pretty hard. But, we run into some issues doing debugging and profiling. It’s easier to have the code all within one place, so for development purposes we’re pulling the source so that we can specify the configuration and compiler options for Apache.

Debugging

What other tool do you use to debug C code than the GDB? With any language you need a way to be able to properly debug your code and usually print statements just don’t get it done. So, why stop at the ability to print out from the code when you can actually step into your functions, print out variables, and see back traces. With normal C applications you can simply just load the binaries into GDB:
# gdb <binary>

This doesn’t quite work in our setup though. We are in the situation where we need to debug modules, not just Apache. And, to really be able to step into the code we need to compile it appropriately. When we’re dealing with setting up our module for debugging we need to make sure we pass in the -g flag. The -g flag just makes it so that your code can be debugged with gdb.

So, we have our module which we’ll call seoversite.c. We’re looking to debug so our build looks something like this:
~/seoversite/# /opt/apache/bin/apxs -ci -Wc,-g seoversite.c

This turns on debugging within the binary which means that we are letting GDB process the source of our file. With apxs, the -Wc, flag allows you to pass compiler flags along for when your code is actually built.

We then have a really simple startup for GDB.
/opt/apache/# gdb bin/httpd

As soon as this fires we get a prompt that lets us specify what we want to do. Today I was debugging a simple method that processes a URL so I’ll go through some of that. When we start up gdb we’re presented with a prompt. This is where we need to enter some input.
(gdb) b get_fullurl
Function "get_fullurl" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (get_fullurl) pending.

Because our module is shared, gdb doesn’t find the method signatures right away. That’s ok. We’ll wait till they load on the request. Basically, the above puts a breakpoint on the method call get_fullurl. Thus, any code paths that use get_fullurl will halt activity until processed in gdb. So, we start the server.
(gdb) run -X
Starting program: /opt/apache/bin/httpd -X
[Thread debugging using libthread_db enabled]
[New Thread 0x7fce2930d760 (LWP 14174)]
warning: Temporarily disabling breakpoints for unloaded shared library "/opt/apache/modules/seoversite.so"
[Switching to Thread 0x7fce2930d760 (LWP 14174)]

The command we typed was “run -X” meaning, start the binary with these command line options. -X is the debugging flag for Apache. So, we’re telling it to start Apache HTTPd in debug mode. This is what we want. It tells you that it’d loading a shared library and is going to disable the breakpoints while it loads. Afterward it enables the breakpoints you’ve defined. At this point, because our code is dependent on a request, we should make one. I hit a virtual host url that is setup to use our module and immediately see that our method is hit.
Breakpoint 1, get_fullurl (ctx=0x19d4908, use_base=0, args=1) at seoversite.c:330
330 if (use_base) {

This isn’t entirely useful information. I know where my method starts. However, we aren’t limited to just this information. Let’s say that I want to see the back trace of method calls that got me to this method.
(gdb) bt
#0 get_fullurl (ctx=0x19d4908, use_base=0, args=1) at seoversite.c:330
#1 0x00007fce2402299a in filtering (f=0x19cba78, bb=0x19d0298) at seoversite.c:1739
#2 0x000000000044f446 in ap_pass_brigade ()
#3 0x000000000047b2ed in ap_proxy_http_process_response ()
#4 0x000000000047bad0 in proxy_http_handler ()
#5 0x000000000046af37 in proxy_run_scheme_handler ()
#6 0x0000000000467524 in proxy_handler ()
#7 0x0000000000441450 in ap_run_handler ()
#8 0x0000000000441ce9 in ap_invoke_handler ()
#9 0x000000000048823c in ap_process_request ()
#10 0x00000000004851f0 in ap_process_http_connection ()
#11 0x000000000044ae72 in ap_run_process_connection ()
#12 0x000000000044b2f9 in ap_process_connection ()
#13 0x00000000004ac48b in child_main ()
#14 0x00000000004ac56e in make_child ()
#15 0x00000000004acb03 in ap_mpm_run ()
#16 0x000000000042857c in main ()

This is now getting useful. Not only is this nice for debugging purposes, it’s really nice for inspecting call traces of modules you’ve never used before. Inspecting the actual behavior will provide you a much better clarity of the source code. In our case, this confirmed my assumption of the calls that lead up to getting to get_fullurl.

Now I know how we got to my method, but I need to see where it’s failing. Ideally I have the source up in Emacs and can look at line numbers while debugging. What I don’t get with this however is the ability to go one line at a time and inspect the values and types of variables. GDB provides this capability.
(gdb) n
334 if (ctx->f->r->parsed_uri.query != NULL && args) {

As with most debugging software, this allows me to step through code, line by line, to see what the execution does. Obviously, this is a useful thing. But, to enhance it we get the ability to inspect variables.
(gdb) n
334 if (ctx->f->r->parsed_uri.query != NULL && args) {
(gdb) p args
$2 = 1
(gdb) p ctx->f->r->parsed_uri.query
$3 = 0x0

And, to make sure you have the appropriate context for what you are dealing with, you can print out a handful of lines to accompany your inspection.
(gdb) l
327 char* baseuri;
328 char* fullurl;
329
330 if (use_base) {
331 baseuri = get_baseuri(ctx->f->r);
332 } else {
333 if (ctx->f->r->parsed_uri.query != NULL && args) {

Ultimately this has made things easier. And, there is no longer the “ok, add a new print line in here, recompile, run, check logs, rinse and repeat” type of process. It really simplifies my life when I have to hack on C code and I know it helped one of our other guys track down some memory based issues. The only downside here is that there is no longer any sword fighting.