NJAMD - Not Just Another Malloc Debugger


SYNOPSIS

       #include <stdlib.h>

       void *calloc(size_t nmemb, size_t size);
       void *malloc(size_t size);
       void free(void *ptr);
       void *realloc(void *ptr, size_t size);

       new, new[], delete, delete[]

       #include <string.h>

       char *strdup(const char *s);

       And so much more...

       export LD_PRELOAD=libnjamd.so
       export NJAMD_PROT=val
       export NJAMD_CHK_FREE=val
       export NJAMD_ALIGN=num
       export NJAMD_DUMP_LEAKS_ON_EXIT=num
       export NJAMD_DUMP_STATS_ON_EXIT=1
       export NJAMD_DUMP_CORE=soft,hard
       export NJAMD_PERSISTENT_HEAP=1
       export NJAMD_TRACE_LIBS=1
       export NJAMD_NO_TRACE=1
       export NJAMD_ALLOW_READ=1
       export NJAMD_ALLOW_FREE_0=1
       export NJAMD_ALLOW_MALLOC_0=1
       kill -USR1 <pid>


DESCRIPTION

       This  manpage  describes  the  library subsystem of NJAMD,
       which can be used standalone, with the front end  (console
       only,  and  needs work), from within gdb(1) or from within
       any other debugger. It also comes with a very useful util­
       ity njamdpm(1) that allows you to do postmortem heap anal­
       ysis.

       NJAMD is a full featured malloc debugger. That is, it pro­
       tects  against  all  common dynamic memory bugs, including
       overflow, underflow, writes to freed  memory,  and  memory
       leaks, all without recompiling or even relinking your exe­
       cutable.  In addition, it is able to  trace  memory  leaks
       even  through  arbitrary  library functions that wrap mal­
       loc(3), such as strdup(3), GUI widget allocators, and even
       C++ new and delete.

       Normally,  when  a program does something illegal with its
       dynamic memory (such as writing past the end of  a  buffer
       returned by malloc(3), ie: an overflow), its execution may

       pected  results  can  occur later on during program execu­
       tion. This is due to the fact that malloc  implementations
       store  book keeping information before and after allocated
       segments. So overwriting these regions  won't  cause  your
       program  to  crash right away, but will cause chaos during
       subsequent malloc requests, or even during usuage of  mem­
       ory returned from otherwise seemingly valid malloc()

       NJAMD changes all this. It provides immediate notification
       (through segmentation fault) if you  do  anything  illegal
       with  your  memory.  Using your favorite debugger, you can
       pinpoint the source of error to the line, and even to  the
       assembly instruction.


USAGE

   Overview
       With  one  exception, the behavior of the debugger is con­
       trolled entirely through the following  environment  vari­
       ables.  In  fact,  the  debugger needn't even be linked to
       your program on systems that suport the  LD_PRELOAD  envi­
       ronment  variable. Unless otherwise stated, these environ­
       ment variables are tested for existance  only.   That  is,
       doing  something like export NJAMD_ALLOW_READ=0 will still
       allow read access past the ends of buffers.

   Environment Variables
       LD_PRELOAD=library
              This environment  variable  instructs  the  dynamic
              linker  to  override system supplied functions with
              those in a specified library, namely ours.  To  use
              LIBNJAMD  to debug your programs, enter the equiva­
              lent to

              export LD_PRELOAD=libnjamd.so

              into your shell. All subsequent programs  run  from
              that  shell will then use LIBNJAMD's allocator rou­
              tines instead of those in the standard libc.

              Alternatively, to debug only one command, enter

              LD_PRELOAD=libnjamd.so <program>

              NOTE on Irix systems, the variable is _RLD_LIST and
              the  syntax is _RLD_LIST=libnjamd.so:DEFAULT.  Oth­
              erwise the behavior is the exact same.

       NJAMD_PROT=val
              To  use  LIBNJAMD  to  protect  against   overflows
              (accessing  memory  past  the  end  of an allocated
              buffer), enter the equivalent to

              export NJAMD_PROT=overflow

              tion.

              There  are  two  ways to protect against underflows
              (accessing  memory  before  an  allocated  buffer),
              strict  and  weak. Weak is considerably faster than
              strict, and uses half as much memory. However, weak
              will  only  catch  underflows  greater  than 4 or 8
              bytes, depending on your archetecture.

              To protect against ALL underflows, enter

              export NJAMD_PROT=strict

              To protect against most larger underflows, enter

              export NJAMD_PROT=underflow

              For memory leak checking only, enter

              export NJAMD_PROT=none

              This option uses standard libc malloc, and is  thus
              is  much faster and lighter than the other options,
              for people who just want  memory  leak  accounting.
              Note that it is unavailable on platforms that don't
              support dlopen(2).  Also, this option  will  misre­
              port  leaked memory by one malloc on some platforms
              (GNU/Linux w/  glibc  2.1),  becuase  malloc  calls
              itself to set up some data structures.

              Do  note  that each version of the library performs
              consistancy checks so that you know if the opposite
              error  occured when you try to free that block. For
              example, when you free a buffer, the overflow  ver­
              sion  checks to make sure that the data before your
              buffer hasn't changed, and the  underflow  versions
              checks to make sure that the data after your buffer
              hasn't changed. So at the worst, you always know of
              a  memory  error  by  the time you free the memory.
              This even applies to the "none" option.

       NJAMD_CHK_FREE=val
              There are various methods of handling  accesses  to
              freed  memory, and each has its advantages and dis­
              advantages. If you select NJAMD_PROT=none this set­
              ting  has  no  effect  (NJAMD  operates  as  if you
              selected no free protection).

              export NJAMD_CHK_FREE=segv

              The default method is to protect  freed  memory.  A
              double  free will yield a segmentation fault and no
              error message, and any access to  a  memory  region

              fault.

              export NJAMD_CHK_FREE=error

              This option both protects freed memory and provides
              you  with some sort of notification when you try to
              free a chunk twice. While this causes  no  physical
              memory  loss,  it  does pollute the address space a
              bit, and can bog down the operating  system  kernel
              with excessive mappings to keep track of. The BSD's
              especially are hurt by this option, and Linux has a
              limit  of  65536  mappings,  which  can  be used up
              pretty quickly.

              export NJAMD_CHK_FREE=none

              This method provides no protection of freed memory.
              Writes to freed memory may produce the same bizzare
              and unpredicatble results as when  using  a  normal
              malloc implementation. Use of this option is recom­
              mended when allocation intensive progams run out of
              memory  under  either  of the preceding options. Do
              note that by default, some OS's do not allow you to
              map   the   entire  address  space.  You  must  use
              sysctl(2) to allow this. In Linux, for example, you
              must issue

              sysctl -w vm.overcommit_memory=1

              to  use  the  entire  address space. Try doing this
              instead of turning off the checking of  freed  mem­
              ory.  Remember,  the  memory  isn't  actually being
              used, only mapped, so system performace won't  suf­
              fer.

              export NJAMD_CHK_FREE=nofree

              Setting  this option will cause NJAMD to neglect to
              free ANY memory.  I  can't  think  of  any  circum­
              stances where this would be useful, but it was very
              easy to implement :)

       NJAMD_ALIGN=num
              In order for programs to work correctly under  cer­
              tain  archetectures  (ie, sparc and most other RISC
              CPUs), malloc must return memory aligned to a  cer­
              tain  number  of  bytes if you want that section of
              memory to contain pointers and floating point  val­
              ues. The alignment of your archetecture is detected
              automatically when  you  install  NJAMD.   However,
              note that aligning memory to n bytes will cause the
              overflow detection to miss overflows  of  up  to  n
              bytes.   If  you  are  on a RISC CPU, but know that

              it only deals with strings), then

              export NJAMD_ALIGN=1 to set alignment to 1 byte.

       NJAMD_DUMP_LEAKS_ON_EXIT=num
              Setting this option instructs LIBNJAMD to dump mem­
              ory leak diagnostics to the front end (or  standard
              error  when running standalone) upon program termi­
              nation.  num levels of stack trace are provided for
              each malloc and free (the default max is 3, and can
              be    set    at    NJAMD    compile     time     in
              ./include/lib/njamd.h, via the TRACE_DEPTH define)

              Do  note that it is common practice for short-lived
              programs such as ls(1) to simply exit without free­
              ing memory.

              export NJAMD_DUMP_LEAKS_ON_EXIT=3

       NJAMD_DUMP_STATS_ON_EXIT=1
              Setting  this  environment variable instructs NJAMD
              to dump a short  summary  of  memory  usage  versus
              address space usage. This option will help you fig­
              ure out how much overhead is being used  by  NJAMD,
              and  how  much address space in total was needed to
              debug your application. In  other  words,  it  will
              either give you an excuse to buy a 64 bit CPU, or a
              few more RAM chips ;)

       NJAMD_DUMP_CORE=hard,soft
              By default, NJAMD will catch all deadly signals  in
              order  to  perform cleanup, provide statistics, and
              give its own backtrace of when the fault  occurred.
              However, when you are using a debugger, this behav­
              ior is not always desirable.

              If you would like  NJAMD  to  perform  cleanup  and
              statistics  information, but would also like a core
              file, then

              export NJAMD_DUMP_CORE=soft

              Using soft  core  dumping  will  cause  the  return
              address information INSIDE the core file to make no
              sense, but it will allow NJAMD to  provide  a  call
              stack  dump  upon  exit. Using softcore also limits
              the   coresize   to    4megs    (defined    through
              NJ_LIMIT_SOFTCORE in the source).

              If  you  would  like  the core file to be perfectly
              valid and complete at the expense of statistics and
              post-mortem heap integrity, then


       NJAMD_PERSISTENT_HEAP=1
              Setting  this option will cause the program heap to
              remain  after  execution  in  a   file   forrmatted
              ./njamd-<pid>-heap.  Unlike other malloc debuggers,
              saving the heap in NJAMD imposes no  extra  perfor­
              mance  overhead  on  the system. The heap is always
              mapped to a file in tmp, but this  file  only  per­
              sists  if this option is set. To utilize this file,
              use the njamd post-mortem utility  njamdpm(1).   Be
              advised that the heap can take up as much as 8 megs
              on 32 bit systems.  ls may report it as 8 megs  due
              to  the  lseek.  The  actual size is much less. Use
              du(1)

              export NJAMD_PERSISTENT_HEAP=1

       NJAMD_TRACE_LIBS=1
              NJAMD's default action is to ignore shared  library
              return addresses and only give you return addresses
              in the statically linked portion  of  the  program.
              This can be a problem if your program consists of a
              large  amount  of  supporting  libraries,  and  the
              static  section  is simply a main loop. So instead,
              to provide return addresses in the libraries, set

              export NJAMD_TRACE_LIBS=1

       NJAMD_NO_TRACE=1
              When debugging programs that use libraries compiled
              with  optimization greater than -O2 or with -fomit-
              frame-pointer, you must disable tracing,  or  NJAMD
              will segfault. This isn't really NJAMD's fault. Gcc
              (see     'info     gcc')     claims     that     if
              __builtin_return_address  is  unavailable  it  will
              just return NULL. Instead it  segmentation  faults.
              I've   notified  the  gcc  team,  but  received  no
              response.

              export NJAMD_NO_TRACE=1

       NJAMD_ALLOW_READ=1
              Glibc versions up to and including 2.1.2 had a  bug
              relating  to the sscanf(3) code that was tripped by
              this library. As much as we would like to, unfortu­
              nately,  we  cannot take credit for being the first
              to discover this bug.  The  glibc  folk  found  and
              fixed  the  bug in version 2.1.3. At any rate, this
              option is provided as a workaround. It allows reads
              past  the end of a buffer to not segfault your pro­
              gram. The default is to  forbid  read,  write,  and
              execute attempts.


       NJAMD_ALLOW_MALLOC_0=1, NJAMD_ALLOW_FREE_0=1
              ANSI  C specifies that malloc's of 0 are legal, and
              that free's of NULL are illegal.  However,  NJAMD's
              default  action is to warn when either takes place,
              as either usually indicates a bug (or at  the  very
              least, an unneeded call). To turn this behavior off
              for either free or malloc, issue

              export NJAMD_ALLOW_FREE_0=1, or
              export NJAMD_ALLOW_MALLOC_0=1 respectively.

   Dumping memory information
       Issuing a kill -USR1 to a program running  under  LIBNJAMD
       will  cause memory usage diagnostics to be dumped to stan­
       dard error. These are human readable lines of the form

       njamd: Memory  leak  of  XX  bytes  allocated  at  address
       0xXXXXXXX.

       The address given is the actual address of the correspond­
       ing memory allocation in your code.  In future versions of
       NJAMD,  scripts  or runtime functionality will be provided
       to translate these addresses on the fly to  functions  and
       line  numbers.  Until then, read on to find out how to use
       gdb(1) to translate them for you.

   Debugging from within gdb
       Using LIBNJAMD from within gdb is simple. The  command  to
       set   environment  variables  in  gdb  is  set  env  VARI­
       ABLE=value.  So, to instruct gdb to use LIBNJAMD, issue

       set env LD_PRELOAD=libnjamd.so

       from INSIDE gdb. Issuing an  LD_PRELOAD  command  to  your
       shell  before  starting gdb causes gdb to use that library
       as well, which means gdb would be using NJAMD, and  unless
       you're  on  the  gdb development team, you're probably not
       interested in debugging gdb :)

       All other options can be set in the same mannor from  with
       gdb, or in the shell's environment outside gdb.

       You  can  obtain  memory  leak information at any point by
       setting a break point, and then issuing signal SIGUSR1  at
       that  breakpoint.  This will provide a memory leak dump as
       described above.  In addition, so will
       call __nj_dump_leaks(__nj_display_depth)

       njamd by issuing call __nj_ptr_info(address)

       This will dump out a call stack of the malloc or free that
       contains that address. This is  very  useful  for  gaining
       information about a segmentation fault. Ie if the segfault
       occurs on a line that does buf[i] = 2, issue
       call __nj_ptr_info(&buf[i]) to gdb.

       To get gdb to translate these return addresses into  some­
       thing meaningful, issue

       info  line  *0xaddress  to  obtain  the line number of the
       allocation request, or

       list *0xaddress to see the adjacent code as well.

       Another neat trick you may find handy  for  tracking  down
       things  like  free(NULL)  and malloc(0) and other behavior
       that  produces  warnings  is  to  set  a   breakpoint   at
       __nj_eprintf so that you can determine the location of the
       offending instruction.  __nj_eprintf  is  NJAMD's  general
       purpose  error  function.  It  is  called to print out any
       NJAMD warning or error message you see. Note that  if  you
       are  using LD_PRELOAD, you may have to set a breakpoint in
       main and start the program before setting this  breakpoint
       for gdb to know that __nj_eprintf is a valid symbol.


BUGS

       If  anything goes wrong, please read the NOTES file in the
       NJAMD source directory, ESPECIALLY  before  reporting  any
       bugs.  Many platforms and compiler flag combinations cause
       problems. Please read that file for more info. Also, when­
       ever sending me a bug report, please send a self-contained
       code snippet to reproduce the bug.


PERFORMANCE

       If you notice large performance  problems,  especially  on
       the xBSD's,

       export NJAMD_CHK_FREE=segv or none

       This  is  because the BSD's do a linear search of mappings
       (O(F)), where as Linux maintains an AVL tree after a  cer­
       tain  number  of  mappings  (O(lg  F)). F is the number of
       calls free's so far when export NJAMD_CHK_FREE=error.

       Also,   there   is   a    define    at    the    top    of
       ./include/lib/njamd.h    in   the   NJAMD   source   tree,
       TRACE_DEPTH, that allows you to set how large of  a  stack
       trace  is  recorded. Lowering this to 1 (default is 3) may
       speed things up a bit, as well as cut the size of the heap



NOTES

       I've  discovered that the limit of 65536 mappings in Linux
       includes permission variations. In other words,  you  will
       not  be able to debug large apps due to this limit. To get
       around this, apply  either  fix_map-<kernel_version>.patch
       or  proc_map-<kernel_version>.patch  to  your  kernel, and
       recompile. See the ./kernel_mod/README file for more info.

       Core  dump  sizes  are limited to 4 megs (and are probably
       next to useless) because of the  huge  amount  of  mapping
       that  takes  place.  Some OS's (again, those evil BSD's ;)
       actually zero-fill mapped but unfaulted memory  as  it  is
       dumped  to  disk, causing a core dump to take a horrendous
       amout of time and disk space.

       For information on how the system works, read the program­
       mers  documentation  in  the source tree, and check out my
       (admittantly    incomplete)    Shared    Memory     HOWTO:
       http://fscked.org/writings/SHM/shm.html


AUTHORS

       Mike Perry <mikepery@fscked.org> - libnjamd
       Steve Engelhardt <sengelha@yahoo.com> - Front End


SEE ALSO

       http://freshmeat.net/appindex/development/debugging.html

       njamdpm(1), efence(3), malloc(3), mmap(2), mprotect(2)


Man(1) output converted with man2html