klogd - Kernel Log Daemon
SYNOPSIS
klogd [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [
-p ] [ -s ] [ -k fname ] [ -v ] [ -x ] [ -2 ]
DESCRIPTION
klogd is a system daemon which intercepts and logs Linux
kernel messages.
OPTIONS
-c n Sets the default log level of console messages to
n.
-d Enable debugging mode. This will generate LOTS of
output to stderr.
-f file
Log messages to the specified filename rather than
to the syslog facility.
-i -I Signal the currently executing klogd daemon. Both
of these switches control the loading/reloading of
symbol information. The -i switch signals the dae
mon to reload the kernel module symbols. The -I
switch signals for a reload of both the static ker
nel symbols and the kernel module symbols.
-n Avoid auto-backgrounding. This is needed espe
cially if the klogd is started and controlled by
init(8).
-o Execute in 'one-shot' mode. This causes klogd to
read and log all the messages that are found in the
kernel message buffers. After a single read and
log cycle the daemon exits.
-p Enable paranoia. This option controls when klogd
loads kernel module symbol information. Setting
this switch causes klogd to load the kernel module
symbol information whenever an Oops string is
detected in the kernel message stream.
-s Force klogd to use the system call interface to the
kernel message buffers.
-k file
Use the specified file as the source of kernel sym
bol information.
-x Omits EIP translation and therefore doesn't read
the System.map file.
-2 When symbols are expanded, print the line twice.
Once with addresses converted to symbols, once with
the raw text. This allows external programs such
as ksymoops do their own processing on the original
data.
OVERVIEW
The functionality of klogd has been typically incorporated
into other versions of syslogd but this seems to be a poor
place for it. In the modern Linux kernel a number of ker
nel messaging issues such as sourcing, prioritization and
resolution of kernel addresses must be addressed. Incor
porating kernel logging into a separate process offers a
cleaner separation of services.
In Linux there are two potential sources of kernel log
information: the /proc file system and the syscall
(sys_syslog) interface, although ultimately they are one
and the same. Klogd is designed to choose whichever
source of information is the most appropriate. It does
this by first checking for the presence of a mounted /proc
file system. If this is found the /proc/kmsg file is used
as the source of kernel log information. If the proc file
system is not mounted klogd uses a system call to obtain
kernel messages. The command line switch (-s) can be used
to force klogd to use the system call interface as its
messaging source.
If kernel messages are directed through the syslogd daemon
the klogd daemon, as of version 1.1, has the ability to
properly prioritize kernel messages. Prioritization of
the kernel messages was added to it at approximately ver
sion 0.99pl13 of the kernel. The raw kernel messages are
of the form:
<[0-7]>Something said by the kernel.
The priority of the kernel message is encoded as a single
numeric digit enclosed inside the <> pair. The defini
tions of these values is given in the kernel include file
kernel.h. When a message is received from the kernel the
klogd daemon reads this priority level and assigns the
appropriate priority level to the syslog message. If file
output (-f) is used the prioritization sequence is left
pre-pended to the kernel message.
The klogd daemon also allows the ability to alter the pre
sentation of kernel messages to the system console.
the inclusion of default messaging levels for the kernel.
In a stock kernel the the default console log level is set
to 7. Any messages with a priority level numerically
lower than 7 (higher priority) appear on the console.
Messages of priority level 7 are considered to be 'debug'
messages and will thus not appear on the console. Many
administrators, particularly in a multi-user environment,
prefer that all kernel messages be handled by klogd and
either directed to a file or to the syslogd daemon. This
prevents 'nuisance' messages such as line printer out of
paper or disk change detected from cluttering the console.
When -c is given on the commandline the klogd daemon will
execute a system call to inhibit all kernel messages from
being displayed on the console. Former versions always
issued this system call and defaulted to all kernel mes
sages except for panics. This is handled differently
nowardays so klogd doesn't need to set this value anymore.
The argument given to the -c switch specifies the priority
level of messages which will be directed to the console.
Note that messages of a priority value LOWER than the
indicated number will be directed to the console.
For example, to have the kernel display all mes
sages with a priority level of 3 (KERN_ERR) or more
severe the following command would be executed:
klogd -c 4
The definitions of the numeric values for kernel messages
are given in the file kernel.h which can be found in the
/usr/include/linux directory if the kernel sources are
installed. These values parallel the syslog priority val
ues which are defined in the file syslog.h found in the
/usr/include/sys sub-directory.
The klogd daemon can also be used in a 'one-shot' mode for
reading the kernel message buffers. One shot mode is
selected by specifying the -o switch on the command line.
Output will be directed to either the syslogd daemon or to
an alternate file specified by the -f switch.
For example, to read all the kernel messages after
a system boot and record them in a file called
krnl.msg the following command would be given.
klogd -o -f ./krnl.msg
KERNEL ADDRESS RESOLUTION
If the kernel detects an internal error condition a gen
eral protection fault will be triggered. As part of the
report indicating the state of the processor at the time
of the fault. Included in this display are the contents
of the microprocessor's registers, the contents of the
kernel stack and a tracing of what functions were being
executed at the time of the fault.
This information is EXTREMELY IMPORTANT in determining
what caused the internal error condition. The difficulty
comes when a kernel developer attempts to analyze this
information. The raw numeric information present in the
protection fault printout is of very little use to the
developers. This is due to the fact that kernels are not
identical and the addresses of variable locations or func
tions will not be the same in all kernels. In order to
correctly diagnose the cause of failure a kernel developer
needs to know what specific kernel functions or variable
locations were involved in the error.
As part of the kernel compilation process a listing is
created which specified the address locations of important
variables and function in the kernel being compiled. This
listing is saved in a file called System.map in the top of
the kernel directory source tree. Using this listing a
kernel developer can determine exactly what the kernel was
doing when the error condition occurred.
The process of resolving the numeric addresses from the
protection fault printout can be done manually or by using
the ksymoops program which is included in the kernel
sources.
As a convenience klogd will attempt to resolve kernel
numeric addresses to their symbolic forms if a kernel sym
bol table is available at execution time. If you require
the original address of the symbol, use the -2 switch to
preserve the numeric address. A symbol table may be spec
ified by using the -k switch on the command line. If a
symbol file is not explicitly specified the following
filenames will be tried:
/boot/System.map
/System.map
/usr/src/linux/System.map
Version information is supplied in the system maps as of
kernel 1.3.43. This version information is used to direct
an intelligent search of the list of symbol tables. This
feature is useful since it provides support for both pro
duction and experimental kernels.
For example a production kernel may have its map file
stored in /boot/System.map. If an experimental or test
kernel is compiled with the sources in the 'standard'
/usr/src/linux/System.map. When klogd starts under the
experimental kernel the map in /boot/System.map will be
bypassed in favor of the map in /usr/src/linux/System.map.
Modern kernels as of 1.3.43 properly format important ker
nel addresses so that they will be recognized and trans
lated by klogd. Earlier kernels require a source code
patch be applied to the kernel sources. This patch is
supplied with the sysklogd sources.
The process of analyzing kernel protections faults works
very well with a static kernel. Additional difficulties
are encountered when attempting to diagnose errors which
occur in loadable kernel modules. Loadable kernel modules
are used to implement kernel functionality in a form which
can be loaded or unloaded at will. The use of loadable
modules is useful from a debugging standpoint and can also
be useful in decreasing the amount of memory required by a
kernel.
The difficulty with diagnosing errors in loadable modules
is due to the dynamic nature of the kernel modules. When
a module is loaded the kernel will allocate memory to hold
the module, when the module is unloaded this memory will
be returned back to the kernel. This dynamic memory allo
cation makes it impossible to produce a map file which
details the addresses of the variable and functions in a
kernel loadable module. Without this location map it is
not possible for a kernel developer to determine what went
wrong if a protection fault involves a kernel module.
klogd has support for dealing with the problem of diagnos
ing protection faults in kernel loadable modules. At pro
gram start time or in response to a signal the daemon will
interrogate the kernel for a listing of all modules loaded
and the addresses in memory they are loaded at. Individ
ual modules can also register the locations of important
functions when the module is loaded. The addresses of
these exported symbols are also determined during this
interrogation process.
When a protection fault occurs an attempt will be made to
resolve kernel addresses from the static symbol table. If
this fails the symbols from the currently loaded modules
are examined in an attempt to resolve the addresses. At
the very minimum this allows klogd to indicate which load
able module was responsible for generating the protection
fault. Additional information may be available if the
module developer chose to export symbol information from
the module.
Proper and accurate resolution of addresses in kernel mod
ules requires that klogd be informed whenever the kernel
to signal the currently executing daemon that symbol
information be reloaded. Of most importance to proper
resolution of module symbols is the -i switch. Each time
a kernel module is loaded or removed from the kernel the
following command should be executed:
klogd -i
The -p switch can also be used to insure that module sym
bol information is up to date. This switch instructs
klogd to reload the module symbol information whenever a
protection fault is detected. Caution should be used
before invoking the program in ´paranoid´ mode. The sta
bility of the kernel and the operating environment is
always under question when a protection fault occurs.
Since the klogd daemon must execute system calls in order
to read the module symbol information there is the possi
bility that the system may be too unstable to capture use
ful information. A much better policy is to insure that
klogd is updated whenever a module is loaded or unloaded.
Having uptodate symbol information loaded increases the
probability of properly resolving a protection fault if it
should occur.
Included in the sysklogd source distribution is a patch to
the modules-2.0.0 package which allows the insmod, rmmod
and modprobe utilities to automatically signal klogd when
ever a module is inserted or removed from the kernel.
Using this patch will insure that the symbol information
maintained in klogd is always consistent with the current
kernel state.
SIGNAL HANDLING
The klogd will respond to eight signals: SIGHUP, SIGINT,
SIGKILL, SIGTERM, SIGTSTP, SIGUSR1, SIGUSR2 and SIGCONT.
The SIGINT, SIGKILL, SIGTERM and SIGHUP signals will cause
the daemon to close its kernel log sources and terminate
gracefully.
The SIGTSTP and SIGCONT signals are used to start and stop
kernel logging. Upon receipt of a SIGTSTP signal the dae
mon will close its log sources and spin in an idle loop.
Subsequent receipt of a SIGCONT signal will cause the dae
mon to go through its initialization sequence and re-
choose an input source. Using SIGSTOP and SIGCONT in com
bination the kernel log input can be re-chosen without
stopping and restarting the daemon. For example if the
/proc file system is to be un-mounted the following com
mand sequence should be used:
# kill -TSTP pid
# kill -CONT pid
Notations will be made in the system logs with LOG_INFO
priority documenting the start/stop of logging.
The SIGUSR1 and SIGUSR2 signals are used to initiate load
ing/reloading of kernel symbol information. Receipt of
the SIGUSR1 signal will cause the kernel module symbols to
be reloaded. Signaling the daemon with SIGUSR2 will cause
both the static kernel symbols and the kernel module sym
bols to be reloaded.
Provided that the System.map file is placed in an appro
priate location the signal of generally greatest useful
ness is the SIGUSR1 signal. This signal is designed to be
used to signal the daemon when kernel modules are
loaded/unloaded. Sending this signal to the daemon after
a kernel module state change will insure that proper reso
lution of symbols will occur if a protection fault occurs
in the address space occupied by a kernel module.
FILES
/proc/kmsg
One Source for kernel messages klogd
/var/run/klogd.pid
The file containing the process id of klogd
/boot/System.map, /System.map, /usr/src/linux/System.map
Default locations for kernel system maps.
BUGS
Probably numerous. Well formed context diffs appreciated.
AUTHOR
The klogd was originally written by Steve Lord
(lord@cray.com), Greg Wettstein made major improvements.
Dr. Greg Wettstein (greg@wind.enjellic.com)
Enjellic Systems Development
Oncology Research Divsion Computing Facility
Roger Maris Cancer Center
Fargo, ND 58122
Man(1) output converted with
man2html