NJAMD - Not Just Another Malloc Debugger
SYNOPSIS
#include <stdlib.h>
void *calloc(size_t nmemb, size_t size);
void *malloc(size_t size);
void free(void *ptr);
void *realloc(void *ptr, size_t size);
new, new[], delete, delete[]
#include <string.h>
char *strdup(const char *s);
And so much more...
export LD_PRELOAD=libnjamd.so
export NJAMD_PROT=val
export NJAMD_CHK_FREE=val
export NJAMD_ALIGN=num
export NJAMD_DUMP_LEAKS_ON_EXIT=num
export NJAMD_DUMP_STATS_ON_EXIT=1
export NJAMD_DUMP_CORE=soft,hard
export NJAMD_PERSISTENT_HEAP=1
export NJAMD_TRACE_LIBS=1
export NJAMD_NO_TRACE=1
export NJAMD_ALLOW_READ=1
export NJAMD_ALLOW_FREE_0=1
export NJAMD_ALLOW_MALLOC_0=1
kill -USR1 <pid>
DESCRIPTION
This manpage describes the library subsystem of NJAMD,
which can be used standalone, with the front end (console
only, and needs work), from within gdb(1) or from within
any other debugger. It also comes with a very useful util
ity njamdpm(1) that allows you to do postmortem heap anal
ysis.
NJAMD is a full featured malloc debugger. That is, it pro
tects against all common dynamic memory bugs, including
overflow, underflow, writes to freed memory, and memory
leaks, all without recompiling or even relinking your exe
cutable. In addition, it is able to trace memory leaks
even through arbitrary library functions that wrap mal
loc(3), such as strdup(3), GUI widget allocators, and even
C++ new and delete.
Normally, when a program does something illegal with its
dynamic memory (such as writing past the end of a buffer
returned by malloc(3), ie: an overflow), its execution may
pected results can occur later on during program execu
tion. This is due to the fact that malloc implementations
store book keeping information before and after allocated
segments. So overwriting these regions won't cause your
program to crash right away, but will cause chaos during
subsequent malloc requests, or even during usuage of mem
ory returned from otherwise seemingly valid malloc()
NJAMD changes all this. It provides immediate notification
(through segmentation fault) if you do anything illegal
with your memory. Using your favorite debugger, you can
pinpoint the source of error to the line, and even to the
assembly instruction.
USAGE
Overview
With one exception, the behavior of the debugger is con
trolled entirely through the following environment vari
ables. In fact, the debugger needn't even be linked to
your program on systems that suport the LD_PRELOAD envi
ronment variable. Unless otherwise stated, these environ
ment variables are tested for existance only. That is,
doing something like export NJAMD_ALLOW_READ=0 will still
allow read access past the ends of buffers.
Environment Variables
LD_PRELOAD=library
This environment variable instructs the dynamic
linker to override system supplied functions with
those in a specified library, namely ours. To use
LIBNJAMD to debug your programs, enter the equiva
lent to
export LD_PRELOAD=libnjamd.so
into your shell. All subsequent programs run from
that shell will then use LIBNJAMD's allocator rou
tines instead of those in the standard libc.
Alternatively, to debug only one command, enter
LD_PRELOAD=libnjamd.so <program>
NOTE on Irix systems, the variable is _RLD_LIST and
the syntax is _RLD_LIST=libnjamd.so:DEFAULT. Oth
erwise the behavior is the exact same.
NJAMD_PROT=val
To use LIBNJAMD to protect against overflows
(accessing memory past the end of an allocated
buffer), enter the equivalent to
export NJAMD_PROT=overflow
tion.
There are two ways to protect against underflows
(accessing memory before an allocated buffer),
strict and weak. Weak is considerably faster than
strict, and uses half as much memory. However, weak
will only catch underflows greater than 4 or 8
bytes, depending on your archetecture.
To protect against ALL underflows, enter
export NJAMD_PROT=strict
To protect against most larger underflows, enter
export NJAMD_PROT=underflow
For memory leak checking only, enter
export NJAMD_PROT=none
This option uses standard libc malloc, and is thus
is much faster and lighter than the other options,
for people who just want memory leak accounting.
Note that it is unavailable on platforms that don't
support dlopen(2). Also, this option will misre
port leaked memory by one malloc on some platforms
(GNU/Linux w/ glibc 2.1), becuase malloc calls
itself to set up some data structures.
Do note that each version of the library performs
consistancy checks so that you know if the opposite
error occured when you try to free that block. For
example, when you free a buffer, the overflow ver
sion checks to make sure that the data before your
buffer hasn't changed, and the underflow versions
checks to make sure that the data after your buffer
hasn't changed. So at the worst, you always know of
a memory error by the time you free the memory.
This even applies to the "none" option.
NJAMD_CHK_FREE=val
There are various methods of handling accesses to
freed memory, and each has its advantages and dis
advantages. If you select NJAMD_PROT=none this set
ting has no effect (NJAMD operates as if you
selected no free protection).
export NJAMD_CHK_FREE=segv
The default method is to protect freed memory. A
double free will yield a segmentation fault and no
error message, and any access to a memory region
fault.
export NJAMD_CHK_FREE=error
This option both protects freed memory and provides
you with some sort of notification when you try to
free a chunk twice. While this causes no physical
memory loss, it does pollute the address space a
bit, and can bog down the operating system kernel
with excessive mappings to keep track of. The BSD's
especially are hurt by this option, and Linux has a
limit of 65536 mappings, which can be used up
pretty quickly.
export NJAMD_CHK_FREE=none
This method provides no protection of freed memory.
Writes to freed memory may produce the same bizzare
and unpredicatble results as when using a normal
malloc implementation. Use of this option is recom
mended when allocation intensive progams run out of
memory under either of the preceding options. Do
note that by default, some OS's do not allow you to
map the entire address space. You must use
sysctl(2) to allow this. In Linux, for example, you
must issue
sysctl -w vm.overcommit_memory=1
to use the entire address space. Try doing this
instead of turning off the checking of freed mem
ory. Remember, the memory isn't actually being
used, only mapped, so system performace won't suf
fer.
export NJAMD_CHK_FREE=nofree
Setting this option will cause NJAMD to neglect to
free ANY memory. I can't think of any circum
stances where this would be useful, but it was very
easy to implement :)
NJAMD_ALIGN=num
In order for programs to work correctly under cer
tain archetectures (ie, sparc and most other RISC
CPUs), malloc must return memory aligned to a cer
tain number of bytes if you want that section of
memory to contain pointers and floating point val
ues. The alignment of your archetecture is detected
automatically when you install NJAMD. However,
note that aligning memory to n bytes will cause the
overflow detection to miss overflows of up to n
bytes. If you are on a RISC CPU, but know that
it only deals with strings), then
export NJAMD_ALIGN=1 to set alignment to 1 byte.
NJAMD_DUMP_LEAKS_ON_EXIT=num
Setting this option instructs LIBNJAMD to dump mem
ory leak diagnostics to the front end (or standard
error when running standalone) upon program termi
nation. num levels of stack trace are provided for
each malloc and free (the default max is 3, and can
be set at NJAMD compile time in
./include/lib/njamd.h, via the TRACE_DEPTH define)
Do note that it is common practice for short-lived
programs such as ls(1) to simply exit without free
ing memory.
export NJAMD_DUMP_LEAKS_ON_EXIT=3
NJAMD_DUMP_STATS_ON_EXIT=1
Setting this environment variable instructs NJAMD
to dump a short summary of memory usage versus
address space usage. This option will help you fig
ure out how much overhead is being used by NJAMD,
and how much address space in total was needed to
debug your application. In other words, it will
either give you an excuse to buy a 64 bit CPU, or a
few more RAM chips ;)
NJAMD_DUMP_CORE=hard,soft
By default, NJAMD will catch all deadly signals in
order to perform cleanup, provide statistics, and
give its own backtrace of when the fault occurred.
However, when you are using a debugger, this behav
ior is not always desirable.
If you would like NJAMD to perform cleanup and
statistics information, but would also like a core
file, then
export NJAMD_DUMP_CORE=soft
Using soft core dumping will cause the return
address information INSIDE the core file to make no
sense, but it will allow NJAMD to provide a call
stack dump upon exit. Using softcore also limits
the coresize to 4megs (defined through
NJ_LIMIT_SOFTCORE in the source).
If you would like the core file to be perfectly
valid and complete at the expense of statistics and
post-mortem heap integrity, then
NJAMD_PERSISTENT_HEAP=1
Setting this option will cause the program heap to
remain after execution in a file forrmatted
./njamd-<pid>-heap. Unlike other malloc debuggers,
saving the heap in NJAMD imposes no extra perfor
mance overhead on the system. The heap is always
mapped to a file in tmp, but this file only per
sists if this option is set. To utilize this file,
use the njamd post-mortem utility njamdpm(1). Be
advised that the heap can take up as much as 8 megs
on 32 bit systems. ls may report it as 8 megs due
to the lseek. The actual size is much less. Use
du(1)
export NJAMD_PERSISTENT_HEAP=1
NJAMD_TRACE_LIBS=1
NJAMD's default action is to ignore shared library
return addresses and only give you return addresses
in the statically linked portion of the program.
This can be a problem if your program consists of a
large amount of supporting libraries, and the
static section is simply a main loop. So instead,
to provide return addresses in the libraries, set
export NJAMD_TRACE_LIBS=1
NJAMD_NO_TRACE=1
When debugging programs that use libraries compiled
with optimization greater than -O2 or with -fomit-
frame-pointer, you must disable tracing, or NJAMD
will segfault. This isn't really NJAMD's fault. Gcc
(see 'info gcc') claims that if
__builtin_return_address is unavailable it will
just return NULL. Instead it segmentation faults.
I've notified the gcc team, but received no
response.
export NJAMD_NO_TRACE=1
NJAMD_ALLOW_READ=1
Glibc versions up to and including 2.1.2 had a bug
relating to the sscanf(3) code that was tripped by
this library. As much as we would like to, unfortu
nately, we cannot take credit for being the first
to discover this bug. The glibc folk found and
fixed the bug in version 2.1.3. At any rate, this
option is provided as a workaround. It allows reads
past the end of a buffer to not segfault your pro
gram. The default is to forbid read, write, and
execute attempts.
NJAMD_ALLOW_MALLOC_0=1, NJAMD_ALLOW_FREE_0=1
ANSI C specifies that malloc's of 0 are legal, and
that free's of NULL are illegal. However, NJAMD's
default action is to warn when either takes place,
as either usually indicates a bug (or at the very
least, an unneeded call). To turn this behavior off
for either free or malloc, issue
export NJAMD_ALLOW_FREE_0=1, or
export NJAMD_ALLOW_MALLOC_0=1 respectively.
Dumping memory information
Issuing a kill -USR1 to a program running under LIBNJAMD
will cause memory usage diagnostics to be dumped to stan
dard error. These are human readable lines of the form
njamd: Memory leak of XX bytes allocated at address
0xXXXXXXX.
The address given is the actual address of the correspond
ing memory allocation in your code. In future versions of
NJAMD, scripts or runtime functionality will be provided
to translate these addresses on the fly to functions and
line numbers. Until then, read on to find out how to use
gdb(1) to translate them for you.
Debugging from within gdb
Using LIBNJAMD from within gdb is simple. The command to
set environment variables in gdb is set env VARI
ABLE=value. So, to instruct gdb to use LIBNJAMD, issue
set env LD_PRELOAD=libnjamd.so
from INSIDE gdb. Issuing an LD_PRELOAD command to your
shell before starting gdb causes gdb to use that library
as well, which means gdb would be using NJAMD, and unless
you're on the gdb development team, you're probably not
interested in debugging gdb :)
All other options can be set in the same mannor from with
gdb, or in the shell's environment outside gdb.
You can obtain memory leak information at any point by
setting a break point, and then issuing signal SIGUSR1 at
that breakpoint. This will provide a memory leak dump as
described above. In addition, so will
call __nj_dump_leaks(__nj_display_depth)
njamd by issuing call __nj_ptr_info(address)
This will dump out a call stack of the malloc or free that
contains that address. This is very useful for gaining
information about a segmentation fault. Ie if the segfault
occurs on a line that does buf[i] = 2, issue
call __nj_ptr_info(&buf[i]) to gdb.
To get gdb to translate these return addresses into some
thing meaningful, issue
info line *0xaddress to obtain the line number of the
allocation request, or
list *0xaddress to see the adjacent code as well.
Another neat trick you may find handy for tracking down
things like free(NULL) and malloc(0) and other behavior
that produces warnings is to set a breakpoint at
__nj_eprintf so that you can determine the location of the
offending instruction. __nj_eprintf is NJAMD's general
purpose error function. It is called to print out any
NJAMD warning or error message you see. Note that if you
are using LD_PRELOAD, you may have to set a breakpoint in
main and start the program before setting this breakpoint
for gdb to know that __nj_eprintf is a valid symbol.
BUGS
If anything goes wrong, please read the NOTES file in the
NJAMD source directory, ESPECIALLY before reporting any
bugs. Many platforms and compiler flag combinations cause
problems. Please read that file for more info. Also, when
ever sending me a bug report, please send a self-contained
code snippet to reproduce the bug.
PERFORMANCE
If you notice large performance problems, especially on
the xBSD's,
export NJAMD_CHK_FREE=segv or none
This is because the BSD's do a linear search of mappings
(O(F)), where as Linux maintains an AVL tree after a cer
tain number of mappings (O(lg F)). F is the number of
calls free's so far when export NJAMD_CHK_FREE=error.
Also, there is a define at the top of
./include/lib/njamd.h in the NJAMD source tree,
TRACE_DEPTH, that allows you to set how large of a stack
trace is recorded. Lowering this to 1 (default is 3) may
speed things up a bit, as well as cut the size of the heap
NOTES
I've discovered that the limit of 65536 mappings in Linux
includes permission variations. In other words, you will
not be able to debug large apps due to this limit. To get
around this, apply either fix_map-<kernel_version>.patch
or proc_map-<kernel_version>.patch to your kernel, and
recompile. See the ./kernel_mod/README file for more info.
Core dump sizes are limited to 4 megs (and are probably
next to useless) because of the huge amount of mapping
that takes place. Some OS's (again, those evil BSD's ;)
actually zero-fill mapped but unfaulted memory as it is
dumped to disk, causing a core dump to take a horrendous
amout of time and disk space.
For information on how the system works, read the program
mers documentation in the source tree, and check out my
(admittantly incomplete) Shared Memory HOWTO:
http://fscked.org/writings/SHM/shm.html
AUTHORS
Mike Perry <mikepery@fscked.org> - libnjamd
Steve Engelhardt <sengelha@yahoo.com> - Front End
SEE ALSO
http://freshmeat.net/appindex/development/debugging.html
njamdpm(1), efence(3), malloc(3), mmap(2), mprotect(2)
Man(1) output converted with
man2html