Jan 12, 2011

Die bug!

Dead fly

Every once in a while I have to fix a bug that really drives me crazy. When this happens I usually recall one morning in the Computer Lab, back in college, one of my professors saying: “It doesn’t matter what you do, it doesn’t matter if you get upset or mad. You won’t beat the computer”. That morning I didn’t understand what he was trying to say. Actually I thought he was making fun of us. Foolish Mario, he wasn’t trying to do that. He was trying to advice us: to always stay calm.

Software Development could be stressful sometimes, if you write software you already know that. Even if you are an experienced programmer knowing the tools and loving writing software you will eventually face a situation, usually fixing a bug, that will drive you crazy and this gets worse when time is constrained, which usually is. At that time is when you have to breath deeply and evaluate every single piece of the puzzle. That’s when you begin enjoying the situation.

Software programmers like to fix, create and understand stuff, specially people with technical background, like me. We are also kind of masochists and we never give up. In my case I like saving those tips because I’m certain that I will need that information in the future.

The following is a list of tips that I wanted to share (and save!) and might be useful when debugging and/or writing software on Linux. Almost all tips are for debugging shared libraries however most of them will work for anything else besides shared libraries. There might be similar alternatives for the other OSs, I’m not sure though.


If you write shared libraries or if have built any package from source you might be familiar with LD_LIBRARY_PATH, an environment variable meant to be used for specifying directories where to search libraries first, recommended for development and testing, nothing else. Well there’s another environment variable called LD_DEBUG, really useful for debugging. It basically outputs all dl* function calls. Use LD_DEBUG=help command for help.

nm and objdump

GNU Binutils is fascinating and has several utilities, however most of the time you will use only those two commands. Reading the manual is the best way to understand them. Basically they help you by listing the symbols used and exported in object files. I’ve used this two commands to make sure the final file, after linking, doesn’t reference to any other library I don’t need, and to make sure my shared libraries export the functions that were meant to export.


One of the hidden gems of gdb. Setting stop-on-solib-events to 1 will allow you to pause the execution of your program when a shared library is loaded. Really useful to know why is my shared library loaded twice? or who is loading what library?. See the other commands to specify files.

ldd and pmap

These are quite similar, both somehow tell you dependencies. The main difference is that ldd shows what libraries are required and pmap shows you process memory mapping, this means memory used and what libraries are actually used for the program.


Traces all calls and signals used by the program. Using -e to define expressions is required unless you want to see lines and lines of calls. It could be useful if you want to profile you program.


Required for profiling. In my previous job I used a lot since all applications were running 24x7 and memory was one of the most important things. I usually leave the application running for days using the following arguments: –leak-check=full –leak-resolution=high –show-reachable=yes then I comeback ctrl+c and I see the results.

Something else?

I’m sure I’m missing more commands, some of them language/platform-specific however I’m also sure this list will help me recalling what to check first when debugging shared libraries.