BugSlasher

The more complex your information system is, the more you will have bugs, and the more you will have symbol files to manage. The cool thing is that second ones can help you resolve the first ones, read my first post if you are not sure about that. The sad thing is that you could end up with tons of symbol files, and finding the right one for every version of you binaries is just too complicated.

Introducing the symbol server

A symbol server is a big word for just a simple share (even if you can also setup an http server, as Microsoft does, but I will talk about it in a next post), that you may protect for writing except for one specific user, and setup the rest of the users for read-only. Here is a classic Symbol Server folder hierarchy:

As you can see, there is a folder for each symbol file, in which you have a sub-folder for each version, and finally the real pdb file inside.

The set of tools provided by Microsoft that allows you to seamless read and write  from that share does not support (unfortunately !) concurrent write, so you have to make sure that only one process write to it at a time. A simple server scrapping your generated symbol files on top of your software factory should do the trick.

How to write to a Symbol Server?

No matter which tool you use to browse symbols and retrieve details inside, you always end with the file DbgHelp.dll (and a little bit of SymSrv.dll). You can find them in the package “Debugging Tools For Windows” available here. Of course Microsoft provide a lot of tools using this dlls, among which you can find “SymStore.exe”. It is a simple command-line utility that can copy a pdb file to a symbol share with the right folder structure. A standard call may look like this:

C:\Tools>symstore.exe add /f <pdb file> /s <share> /t <product name>

You got plenty other options (recursive add, compression, etc.), just browse the help to know which ones suit your needs.

Now that I have a new fancy Symbol Server, how can I use it?

You have two options: setup the symbol server for all applications, or setup for each application one by one. Although the first option may look easier, it could be a penalty hit for each debug session if you add a lot of symbol servers AND try to get a symbol file that nobody have (unfortunately it happens often with third-parties vendor ; at the moment, at work, Sybase and their drivers makes me cringe).

Now you’ve been warned, here is the magic trick to setup all applications: define a system environment variable named ‘_NT_SYMBOL_PATH‘ and put the value ‘srv*C:\LocalCache*\\symbolserver\symbols‘ (of course you have to replace C:\LocalCache and \\symbolserver\symbols by your own values…and please, the single quote must not be typed).

To be more generic, a symbol path must look like ‘expression1;expression2;expression3;…’ with expressionx being a simple directory (for browsing symbol files without the magic folder structure that we saw before) of something like ‘srv*<local store>*<symbol server 1>*<symbol server 2>*…’. The Local store is where you will put pdb files once downloaded, and symbol server x is where you’re searching for them. Let’s see an example:

C:\Windows\System32;srv*C:\SymbolCache*\\server\appsymbols*\\server\mssymbols

This symbol path will make debuggers search for symbol files in C:\Windows\System32 (there is no good reason to put symbols here, I just picked up a folder for the sample). If not found, it will search in C:\SymbolCache ; if not found, it will search in \\server\appsymbols and download it to C:\SymbolCache ; if not found, it will search in \\server\mssymbols and download it to C:\SymbolCache.

I forgot to highlight something: every debuggers will always have a look to a symbol file next to your binary, whether you define a symbol path of not. So don’t add all the folders where you binaries are loaded, it is redundant.

Ok, and if I want to setup each application separately?

Well, you have to read the documentation of your application :) For the moment I’m just going to say that this symbol path syntax can be used out-of-the-box in WinDbg and ProcessExplorer (ProcessExplorer can display the callstack of threads, so it needs a symbol path to do so).

Visual Studio use a fancy graphical interface, I will do a quick post soon about it.

As a conclusion

Now you’re ready to manage tons of symbol files without getting crazy when you’re searching for them, and you can focus on hunting bugs. Happy slashing.