20 Debugging Multiprocess Applications

With Ladebug, you can debug more than one program or process. This chapter explains how to:

Bring a process under debugger control
Set the current process context
Display a list of processes
Load images and core files
Remove process information from the debugger
Debug programs that fork a child process and/or exec a program

In addition, it presents a sample multiprocess debugging session.

For information about multiprocess debugging from the window interface, see Chapter 5.

20.1 Bringing a Process Under Debugger Control

You can bring a process into debugger control in the following ways:

From the command line, using various Ladebug command line options and arguments (for example, specifying a core file for core file debugging or the -k flag for local kernel debugging).
From the Ladebug prompt, you can load a program using the load command or attach to a running process using the attach command.
When the Ladebug variable $catchforks is set to 1, Ladebug keeps track of newly forked processes.

For more information on the various ways to invoke Ladebug, see Section 1.4.

20.2 Displaying a List of Processes

Use the show process command to display the currently executing process or all processes in your application. The syntax is as follows:

show process
show process *
show process all

If you specify the show process command without a qualifier, Ladebug displays information for the current process only. Using the asterisk (*) or the all option displays information for all processes. For each process listed, Ladebug shows the process ID, image file, the number of threads, and state.

20.3 Setting the Current Process

After bringing a process under debugger control, you can switch between processes using the process command. This sets the current process. The syntax is as follows:

process
process [process_id | image_file | debugger_variable]

Specify a specific process using the process ID number or the name of the image. Ladebug sets the current process context to the process ID or the process that runs the binary image. If there are more than one processes running the same binary image, Ladebug warns you and leaves the process context unchanged.

The debugger variables $childprocess and $parentprocess can also be specified in place of the process ID. (Ladebug automatically sets these variables when an application forks a child process.)

The process command without any argument displays information for the current process only.

20.4 Loading Image and Core Files

The load command reads the symbol table information of an image file and a core file.

To debug a core file, specify the name of the core file with the image file. After loading a program, specify the run command to start execution. The syntax is as follows:

load [image_file [core_file]]

20.5 Removing Process Information from the Debugger

Use the unload command to remove the process-related information. The syntax of the unload command is as follows:

unload [process_id | image_file]

The unload command removes the symbol table information if the debuggee process isn't running or stopped. You can specify the detach to release control of the running process or the kill command to terminate the debugger process if the process is created by Ladebug.

If you do not specify a process ID or image file, Ladebug unloads the current process.

20.6 Sample Multiprocess Debugging Session

Example 20-1 demonstrates the use of Ladebug commands to debug a multiprocess application.

In the first part of the program, the process command shows the current process. The load lets you load an image or core file. Specifying show process all displays a list of processes, running or stopped.

Example 20-1 Debugging a Multiprocess Application - Loading an Image File and Showing Processes

$ ladebug
Welcome to the Ladebug Debugger Version 4.0-9
(ladebug) process
There is no current process.
  You may start one by using the `load' or `attach' commands.
(ladebug) load a.out
Reading symbolic information ...done
(ladebug) process1
Current Process: localhost:18340 (a.out).
(ladebug) show process all2
>localhost:18340 (a.out) Unstarted.
(ladebug) load file-func
Reading symbolic information ...done
(ladebug) process
Current Process: localhost:18551 (file-func).3
(ladebug) show process all4
 localhost:18340 (a.out) Unstarted.
>localhost:18551 (file-func) Unstarted.
(ladebug) process 183405
(ladebug) process
Current Process: localhost:18340 (a.out).
(ladebug) list 16
      1
      2 int main(int argc, char* argv[])
      3 {
      4  int a = sizeof(**argv);
      5  int b = sizeof(+(**argv));
      6  int c = sizeof(-(**argv));
      7  return a+b+c;
      8 }

Ladebug sets the current process.
Ladebug shows information for all processes. In this case, there is only one process.
After loading file-func , Ladebug displays a new current process (process 18551).
Ladebug now shows two processes. The arrow (>) indicates the current process.
When you specify the process ID, Ladebug switches to that process. In this case, it switches to process 18340.
Process 18340 is now the current process. Ladebug lists the process' source code.

Switching between processes sets the current process context, as shown in Example 20-2.

Example 20-2 Debugging a Multiprocess Application - Switching Between Processes

(ladebug) process 185511
(ladebug) process
Current Process: localhost:18551 (file-func).
(ladebug) list 12
      1
      2
      3 #include <stdio.h>
      4
      5 int gfi = 100;
      6
      7 int g1 = 10;
      8
      9 int funcA(i) {
     10
     11     int a, x;
     12     x = 1;
     13     a = i + g1 + x;
     14     return a;
     15 }
     16
     17 int funcB(i) {
     18
     19     int a;
     20     a = i;
     21     return a;

Ladebug toggles back to process 18551 and makes it the current process.
Ladebug shows the source code for process 18551.

20.7 Debugging Programs That Fork and/or Exec

This section describes Ladebug's support for debugging programs that fork a child process and/or execs a program.

20.7.1 Predefined Debugger Variables for Fork/Exec Debugging

Ladebug contains the following predefined variables that you set for debugging a program that fork and/or execs. By default, the settings are turned off. When activated, the settings apply to all processes you debug.

$catchexecs - When set to 1, this variable instructs the debugger to notify the user and stop the program when a program execs. The default is 0.
$catchforks - When set to 1, this variable instructs the debugger to notify the user when a program forks a child process. The child process stops and is brought under debugger control. By default, the parent process is not stopped. The default is 0.
$stopparentonfork -When set to 1, this variable instructs the debugger to stop the parent process when a program forks a child process. The default is 0.

When a fork occurs, Ladebug automatically sets the debugger variables $childprocess and $parentprocess to the new child or parent process ID. All examples in this section assume these variables are set.

20.7.2 Debugging Programs That Fork Child Processes

Set $catchforks to 1 to instruct Ladebug to to keep track of newly forked processes when a child process is forked. Ladebug stops the child process immediately after the fork occurs. The child process inherits all breakpoints from the parent process. It also inherits the signals list for the catch/ignore commands from the parent process.

The child process runs the same image file as the parent process and the process context is unchanged. You can switch to the child process using the process command.

When the child process finishes, you cannot rerun the child process. By setting the process context to the parent process, you can rerun the program .

20.7.2.1 Setting the Predefined Variables

Before you debug you application, you can check the setting of the predefined variables using the set command, as shown in Example 20-3. The default settings for $catchforks, $catchexecs, and $stoponparentfork are all 0. In Example 20-3, these variables appear highlighted for illustration.

Example 20-3 Default Settings for Predefined Variables

$ ladebug mp-fork
Welcome to the Ladebug Debugger Version 4.0-10
------------------
object file name: mp-fork
Reading symbolic information ...done
(ladebug) set
$ascii = 1
$beep = 1
$catchexecs = 0
$catchforks = 0
$curevent = 0
$curfile = "mp-fork.c"
.
.
.
$stopparentonfork = 0
$threadlevel = "decthreads"
$verbose = 0

If your programs frequently fork or exec, you may want to set these variables in your .dbxinit initialization file.

20.7.2.2 Scenario for Debugging a Forked Process with the Parent Process Running

To instruct Ladebug to report when the program forks a child process, set the $catchforks predefined variable to 1, as follows:

(ladebug) set $catchforks=1

In Example 20-4, when you run the program, Ladebug notifies you that the child process has stopped. The parent process continues to run.

Example 20-4 Debugging a Forked Process - Showing the Child Process

(ladebug) run
Process 200 forked.  The child process is 201.
Process 201 stopped on fork. 1
stopped at [void main(void):14 0x120001248] 2
     14   if ((pid=fork()) == 0)
Process has exited with status 18 3
(ladebug) show process all 4
>localhost:200 (mp-fork) dead.
    \_localhost:201 (mp-fork) stopped.
(ladebug)

Indicates that the child process has stopped.
Tells where the child process stopped.
Indicates that the parent process, which was not stopped, has completed execution.
Shows that the child process (process 201) has stopped and the parent process has completed execution. The parent process (process 200) remains in the current context, as indicated by the arrow (>).

In Example 20-5, the process context is changed. Listing the source code shows the source for the child process.

Example 20-5 Debugging a Forked Process - Changing the Process Context

(ladebug) process 201 1
(ladebug) show process all
 localhost:200 (mp-fork) dead.
>  \_localhost:201 (mp-fork) stopped.  2
(ladebug) process
Current Process: localhost:201 (mp-fork).
(ladebug) list 3
     15     {
     16       printf("about to exec\n");
     17       execlp("./c_whatis", "./c_whatis", NULL);
     18       perror(" execve failed.");
     19     }
     20
     21   else if (pid != -1)
     22     {
     23       printf("in parent process\n");
     24     }
     25
     26   else
     27     {
     28       printf("Error in fork!");
     29       exit(0);
     30     }
     31 }

The current process context is changed to the child process (process 201).
The arrow now indicates process 201 is the current process.
Ladebug lists the source code for the current process. Notice that it began the listing from the line where the parent process forked.

You can continue to debug the current process (the child process). When the child process finishes, you cannot rerun the child process. By setting the process context to the parent process, you can rerun the program, as shown in Example 20-6.

Example 20-6 Debugging a Forked Process - Rerunning the Program

(ladebug) next
stopped at [void main(void):16 0x12000125c]
     16       printf("about to exec\n");
(ladebug)  next
about to exec
stopped at [void main(void):17 0x120001274]
     17       execlp("./c_whatis", "./c_whatis", NULL);
(ladebug)  next
result of foo = 5040
result of foo = 5040
result of baz = 720
factorial(1) = 1
factorial(2) = 2
factorial(3) = 6
factorial(4) = 24
factorial(5) = 120
factorial(6) = 720
factorial(7) = 5040
factorial(8) = 40320
abcdefghij
abcdefghij
Process has exited with status 0 1
(ladebug) show process all
 localhost:200 (mp-fork) dead.
>  \_localhost:201 (mp-fork) dead.
(ladebug) rerun 2
Error: cannot restart existing process.

(ladebug) process 200 3

(ladebug) rerun 4
Process 200 forked.  The child process is 201.
Process 201 stopped on fork.
stopped at [void main(void):14 0x120001248]
     14   if ((pid=fork()) == 0)
in parent process
Process has exited with status 18
(ladebug)

Indicates that the child process has finished executing.
You cannot rerun the child process.
Setting back to the parent process (process 200), you can now specify rerun .
The program reruns; a new child process is created.

20.7.2.3 Scenario for Debugging a Forked Process with the Parent Process Stopped

In this scenario, you set the predefined variable $catchforks and $stopparentonfork to 1. Setting $catchforks to 1 tells Ladebug to notify the user when the program forks and stop the child process. By setting $stopparentonfork to 1, the parent process also stops when the program forks a child process. The variable $stopparentonfork has no effect when $catchforks is set to 0.

To instruct Ladebug to report when the program forks a child process and stop the parent and child processes, set the variables $catchforks and $stopparentonfork to 1, as follows:

(ladebug) set $catchforks=1
(ladebug) set $stopparentonfork=1

In Example 20-7, Ladebug stops the parent process when it forks the child process. The current context is the parent process. You can change the process context to the child process using the process command.

Example 20-7 Debugging a Forked Process with Parent and Child Processes Stopped

(ladebug) run
Process 200 forked.  The child process is 201.
Process 201 stopped on fork.
stopped at [void main(void):14 0x120001248]
     14   if ((pid=fork()) == 0)
Process 200 stopped on fork.
stopped at [void main(void):14 0x120001248] 1
     14   if ((pid=fork()) == 0)
(ladebug) show process all
>localhost:200 (mp-fork) stopped.
   \_localhost:201 (mp-fork) stopped.
(ladebug) process 201 2
(ladebug) show process all
 localhost:200 (mp-fork) stopped.
>  \_localhost:201 (mp-fork) stopped.
(ladebug) list 3
     15     {
     16       printf("about to exec\n");
     17       execlp("./c_whatis", "./c_whatis", NULL);
     18       perror(" execve failed.");
     19     }
     20
     21   else if (pid != -1)
     22     {
     23       printf("in parent process\n");
     24     }
     25
     26   else
     27     {
     28       printf("Error in fork!");
     29       exit(0);
     30     }
     31 }

Shows that the parent process has stopped at line 14.
Changes the current process context to the child process (process 201).
Lists the source code for the current process (the child process).

In Example 20-8, you continue to debug the current process (the child process). When the child process finishes, you can switch to the parent process and continue debugging.

Example 20-8 Debugging a Forked Process - Switching to the Parent Process

(ladebug) next 1
stopped at [void main(void):16 0x12000125c]
   LDBGD$:[SLOVENKAI.LADEBUG.MANUAL]LADEBUG_CH_MULTIPROC.SDML;19
(ladebug) next
about to exec
sstopped at [void main(void):17 0x120001274]
     17       execlp("./c_whatis", "./c_whatis", NULL);
(ladebug) next
result of foo = 5040
result of foo = 5040
result of baz = 720
factorial(1) = 1
factorial(2) = 2
factorial(3) = 6
factorial(4) = 24
factorial(5) = 120
factorial(6) = 720
factorial(7) = 5040
factorial(8) = 40320
abcdefghij
abcdefghij
Process has exited with status 0 2
(ladebug) show process all
 localhost:200 (mp-fork) stopped.
>  \_localhost:201 (mp-fork) dead.
(ladebug) process 200
(ladebug) list 3
     15     {
     16       printf("about to exec\n");
     17       execlp("./c_whatis", "./c_whatis", NULL);
     18       perror(" execve failed.");
     19     }
     20
     21   else if (pid != -1)
     22     {
     23       printf("in parent process\n");
     24     }
     25
     26   else
     27     {
     28       printf("Error in fork!");
     29       exit(0);
     30     }
     31 }
(ladebug) next
stopped at [void main(void):21 0x1200012b4]
     21   else if (pid != -1)
(ladebug) next
stopped at [void main(void):23 0x1200012c0]
     23       printf("in parent process\n");
(ladebug) cont
in parent process
Process has exited with status 18 4

Continues execution in the child process.
The child process has finished execution.
After switching to the parent process, you can now list its source code.
The parent process terminates.

20.7.3 Debugging a Process That Execs

Set $catchexecs to 1 to instruct Ladebug to notify the user when the program execs. The program stops before executing any user program code or static initializations that are passed to the exec system call. You can debug the newly exec'd program using basic debugging techniques. Ladebug keeps a history of the progression of the exec'd binary files.

In the following scenario, you set the predefined variable $catchforks and $catchexecs to 1. Ladebug will notify you when an exec occurs. Because $catchforks was set, you will be notified of any execs in the child process.

To instruct Ladebug to report when a program execs on the current process context, set the variables $catchexecs and $catchforks to 1, as follows:

(ladebug) set $catchexecs=1
(ladebug) set $catchforks=1

When you run the program in Example 20-9, Ladebug notifies you that an exec occurred on the current context and that the child process has stopped on the runtime-loader entry point.

Example 20-9 Debugging a Process That Execs

(ladebug) run
Process 200 forked.  The child process is 201.
Process 201 stopped on fork.
stopped at [void main(void):14 0x120001248]
     14   if ((pid=fork()) == 0)
in parent process
Process has exited with status 18
(ladebug) show process all
>localhost:200 (mp-fork) dead.
   \_localhost:201 (mp-fork) stopped.

(ladebug) process 201 1
(ladebug) list
     15     {
     16       printf("about to exec\n");
     17       execlp("./c_whatis", "./c_whatis", NULL); 2
     18       perror(" execve failed.");
     19     }
     20
     21   else if (pid != -1)
     22     {
     23       printf("in parent process\n");
     24     }
     25
     26   else
     27     {
     28       printf("Error in fork!");
     29       exit(0);
     30     }
     31 }
(ladebug) next
stopped at [void main(void):16 0x12000125c]
     16       printf("about to exec\n");
(ladebug) next
about to exec
stopped at [void main(void):17 0x120001274]
     17       execlp("./c_whatis", "./c_whatis", NULL);
(ladebug) next
The process 201 has execed the image `./c_whatis'. 3
stopped at [???:62 0x3ff8001c3b8] 4
     62     i = 1;

Ladebug sets the current process context to the child process.
Listing the source code, you can see the process is about to exec on line 17.
Ladebug notifies you when the exec executes.
The child process is stopped on the runtime-loader entry point. The source display shows the code in the main routine.

In Example 20-10, you can set breakpoints in the current process (child process). Ladebug shows the current process and the current executing program.

Example 20-10 Debugging a Process That Execs - Setting Breakpoints

(ladebug) stop in main 1
[#1: stop in int main(void) ]
(ladebug) c
[1] stopped at [int main(void):62 0x1200013a4]
     62     i = 1;
(ladebug) list
     63     foo();
     64     baz(x,3,x+1);
     65
     66     i = 1;
     67     printf("factorial(%d) = %d\n", i, factorial(i));
     68     i = 2; printf("factorial(%d) = %d\n", i, factorial(i));
     69     i = 3; printf("factorial(%d) = %d\n", i,
     70                   factorial(
     71                       i));
     72
     73     i
     74       =
     75         4;
     76     printf(
     77         "factorial(%d) = %d\n",
     78         i,
     79         factorial(i));
     80
     81     if (i < 5)
     82        i = 5;
     83     else
(ladebug) process 2
Current Process: localhost:201 (./c_whatis).
(ladebug) show process all
 localhost:200 (mp-fork) dead.
>  \_localhost:201 (mp-fork->./c_whatis) stopped. 3
(ladebug) next
stopped at [int main(void):63 0x1200013ac]
     63     foo();
(ladebug) next
result of foo = 5040
stopped at [int main(void):64 0x1200013bc]
     64     baz(x,3,x+1);
(ladebug) next
result of foo = 5040
result of baz = 720
stopped at [int main(void):66 0x1200013dc]
     66     i = 1;
(ladebug) next
stopped at [int main(void):67 0x1200013e4]
     67     printf("factorial(%d) = %d\n", i, factorial(i));
(ladebug) next
factorial(1) = 1
stopped at [int main(void):68 0x12000141c]
     68     i = 2; printf("factorial(%d) = %d\n", i, factorial(i));
(ladebug) next
factorial(2) = 2
stopped at [int main(void):69 0x12000145c]
     69     i = 3; printf("factorial(%d) = %d\n", i,
(ladebug) step
stopped at [int factorial(int):8 0x1200011e8]
      8     switch (n) {
(ladebug) where
>0  0x1200011e8 in factorial(n=3) c_whatis.c:8
#1  0x120001470 in main() c_whatis.c:69
(ladebug) cont
factorial(3) = 6
factorial(4) = 24
factorial(5) = 120
factorial(6) = 720
factorial(7) = 5040
factorial(8) = 40320
abcdefghij
abcdefghij
Process has exited with status 0
(ladebug) show process all
 localhost:200 (mp-fork) dead.
>  \_localhost:201 (mp-fork) dead.
(ladebug)

You can set a breakpoint on the current process.
Shows the current process and the current executing program.
Shows the image file history as a progression of images.