|
Using gdb
|
If you use the gcc or egcs compilers, it is likely that the best
debugger for your system is gdb. This is only a brief summary of how
to run gdb on Apache -- you should look at the info and man files for
gdb to get more information on gdb commands and common debugging techniques.
Before running gdb, be sure that the server is compiled with the
-g option in EXTRA_CFLAGS to include the
symbol information in the object files.
The only tricky part of running gdb on Apache is forcing the server
into a single-process mode so that the parent process being debugged
does the request-handling work instead of forking child processes.
We have provided the -X option for that purpose, which will
work fine for most cases. However, some modules don't like starting
up with -X, but are happy if you force only one child to run
(using "MaxClients 1"); you can then use gdb's attach command
to debug the child server.
The following example, with user input in green,
shows the output of gdb run on a server executable (httpd) in the current
working directory and using the server root of
/usr/local/apache:
% gdb httpd
GDB is free software and you are welcome to distribute copies of it
under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.16.gnat.1.13 (sparc-sun-solaris2.5),
Copyright 1996 Free Software Foundation, Inc...
(gdb) b ap_process_request
Breakpoint 1 at 0x49fb4: file http_request.c, line 1164.
(gdb) run -X -d /usr/local/apache
Starting program: /usr/local/apache/src/httpd -X -d /usr/local/apache
[at this point I make a request from another window]
Breakpoint 1, ap_process_request (r=0x95250) at http_request.c:1164
1164 if (ap_extended_status)
(gdb) s
1165 ap_time_process_request(r->connection->child_num, START_PREQUEST);
(gdb) n
1167 process_request_internal(r);
(gdb) s
process_request_internal (r=0x95250) at http_request.c:1028
1028 if (!r->proxyreq && r->parsed_uri.path) {
(gdb) s
1029 access_status = ap_unescape_url(r->parsed_uri.path);
(gdb) n
1030 if (access_status) {
(gdb) s
1036 ap_getparents(r->uri); /* OK --- shrinking transformations... */
(gdb) n
1038 if ((access_status = location_walk(r))) {
(gdb) n
1043 if ((access_status = ap_translate_name(r))) {
(gdb) n
1048 if (!r->proxyreq) {
(gdb) n
1053 if (r->method_number == M_TRACE) {
(gdb) n
1062 if (r->proto_num > HTTP_VERSION(1,0) && ap_table_get(r->subprocess_env, "downgrade-1.0")) {
(gdb) n
1071 if ((access_status = directory_walk(r))) {
(gdb) s
directory_walk (r=0x95250) at http_request.c:288
288 core_server_config *sconf = ap_get_module_config(r->server->module_config,
(gdb) b ap_send_error_response
Breakpoint 2 at 0x47dcc: file http_protocol.c, line 2090.
(gdb) c
Continuing.
Breakpoint 2, ap_send_error_response (r=0x95250, recursive_error=0)
at http_protocol.c:2090
2090 BUFF *fd = r->connection->client;
(gdb) where
#0 ap_send_error_response (r=0x95250, recursive_error=0)
at http_protocol.c:2090
#1 0x49b10 in ap_die (type=403, r=0x95250) at http_request.c:989
#2 0x49b60 in decl_die (status=403, phase=0x62db8 "check access", r=0x95250)
at http_request.c:1000
#3 0x49f68 in process_request_internal (r=0x95250) at http_request.c:1141
#4 0x49fe0 in ap_process_request (r=0x95250) at http_request.c:1167
#5 0x439d8 in child_main (child_num_arg=550608) at http_main.c:3826
#6 0x43b5c in make_child (s=0x7c3e8, slot=0, now=907958743)
at http_main.c:3898
#7 0x43ca8 in startup_children (number_to_start=6) at http_main.c:3972
#8 0x44260 in standalone_main (argc=392552, argv=0x75800) at http_main.c:4250
#9 0x449fc in main (argc=4, argv=0xefffee8c) at http_main.c:4534
(gdb) s
2091 int status = r->status;
(gdb) p status
$1 = 403
(gdb)
There are a few things to note about the above example:
- the "
gdb httpd" command does not include any command-line
options for httpd: those are provided when the "run" command
is done within gdb;
- I set a breakpoint before starting the run so that execution would stop
at the top of ap_process_request();
- the "
s" command steps through the code and into called
procedures, whereas the "n" (next) command steps through
the code but not into called procedures.
- additional breakpoints can be set with the "
b" command,
and the run continued with the "c" command.
- use the "
where" command (a.k.a. "bt") to see
a stack backtrace that shows the order of called procedures and their
parameter values.
- use the "
p" command to print the value of a variable.
A file in the src/ directory, .gdbinit,
provides a useful macro for printing out the contents of a table
structure, called dump_table.
If you are debugging a repeatable crash, simply run gdb as above
and make the request -- gdb should capture the crash and provide a
prompt where it occurs.
If you are debugging an apparent infinite loop, simply run gdb as above
and type a Control-C -- gdb will interrupt the process and provide a
prompt where it was stopped.
If you are debugging a system crash and you have a core file from
the crash, then do the following:
% gdb httpd -c core
(gdb) where
and it will (hopefully) print a stack backtrace of where the core dump
occurred during processing.
|
|
Getting a live backtrace
|
A backtrace will let you know the hierarchy of procedures that
were called to get to a particular point in the process. On some platforms
you can get a live backtrace of any process.
For SVR4-based variants of Unix, the pstack command for proc can
be used to display a a live backtrace. For example, on Solaris it looks
like
% /usr/proc/bin/pstack 10623
10623: httpd -d /usr/local/apache
ef5b68d8 poll (efffcd08, 0, 3e8)
ef5d21e0 select (0, ef612c28, 0, 0, 3e8, efffcd08) + 288
00042574 wait_or_timeout (0, 75000, 75000, 7c3e8, 60f40, 52c00) + 78
00044310 standalone_main (5fd68, 75800, 75c00, 75000, 2, 64) + 240
000449f4 main (3, efffeee4, efffeef4, 75fe4, 1, 0) + 374
000162fc _start (0, 0, 0, 0, 0, 0) + 5c
Another technique is to use gdb to attach to the running process
and then using "where" to print the backtrace, as in
% gdb httpd 10623
GDB is free software and you are welcome to distribute copies of it
under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.16.gnat.1.13 (sparc-sun-solaris2.5),
Copyright 1996 Free Software Foundation, Inc...
/usr/local/apache/src/10623: No such file or directory.
Attaching to program `/usr/local/apache/src/httpd', process 10623
Reading symbols from /usr/lib/libsocket.so.1...done.
Reading symbols from /usr/lib/libnsl.so.1...done.
Reading symbols from /usr/lib/libc.so.1...done.
Reading symbols from /usr/lib/libdl.so.1...done.
Reading symbols from /usr/lib/libintl.so.1...done.
Reading symbols from /usr/lib/libmp.so.1...done.
Reading symbols from /usr/lib/libw.so.1...done.
Reading symbols from /usr/platform/SUNW,Ultra-1/lib/libc_psr.so.1...done.
0xef5b68d8 in ()
(gdb) where
#0 0xef5b68d8 in ()
#1 0xef5d21e8 in select ()
#2 0x4257c in wait_or_timeout (status=0x0) at http_main.c:2357
#3 0x44318 in standalone_main (argc=392552, argv=0x75800) at http_main.c:4273
#4 0x449fc in main (argc=3, argv=0xefffeee4) at http_main.c:4534
(gdb)
|
|
Using 'truss/trace/strace' to trace system calls and
signals
|
Most Unix-based systems have at least one command for displaying
a trace of system calls and signals as they are accessed by a running
process. This command is called truss on most SVR4-based
systems and either trace or strace on many
other systems.
A useful tip for using the truss command on Solaris is
the -f option; it tells truss to follow and continue tracing
any child processes forked by the main process. The easiest way to get
a full trace of a server is to do something like:
% truss -f httpd -d /usr/local/apache >& outfile
% egrep '^10698:' outfile
to view just the trace of the process id 10698.
|
|
Getting the server to dump core
|
Strangely enough, sometimes you actually want to force the server
to crash so that you can get a look at some nutty behavior. Normally
this can be done simply by using the gcore command.
However, for security reasons, most Unix systems do not allow a setuid
process to dump core, since the file contents might reveal something
that is supposed to be protected in memory.
Here is one way to get a core file from a setuid Apache httpd process
on Solaris, without knowing which httpd child might be the one to die
[note: it is probably easier to use the MaxClients trick in the first
section above].
# for pid in `ps -eaf | fgrep httpd | cut -d' ' -f4`
do
truss -f -l -t\!all -S SIGSEGV -p $pid 2>&1 | egrep SIGSEGV &
done
The
undocumented '-S' flag to truss will halt the process in place
upon receipt of a given signal (SIGSEGV in this case).
At this point you can use:
# gcore PID
and then look at the backtrace as discussed above for
gdb.
|
|
Solaris 2.7 and coredumps
|
On Solaris 2.7 use coreadm to make setuid()
processes actually dump core. By default an setuid() process does not
dump core.
Jens-Uwe Mager wrote:
For example I am using:
# coreadm
global core file pattern: /var/core/core.%f.%p.u%u
init core file pattern: core
global core dumps: enabled
per-process core dumps: enabled
global setid core dumps: enabled
per-process setid core dumps: enabled
global core dump logging: disabled
|
|
