ssh_pbs.pl and gdbwhere.pl can be combined to yield a useful
debugging combination that scales. Those perl scripts are at the end of this page and versions for LSF and SGE schedulers are also shown. ssh_pbs.pl will run a command on all machines [nodes] in your job, and for quick checks it can be configured to
skip over nodes in your job [running with every Mth node], or you can use it to
run a command on the first N nodes. Here's the help information
for ssh_pbs.pl:
$ ssh_pbs.pl -help
usage: /usr/local/bin/ssh_pbs.pl [OPTIONS] pbs_jobid "/some/command1; /another/command2"
Run a command on nodes in your job via ssh to each node.
--help
this help screen
-n <N>
ssh to only the first N nodes of the job
-skip <M>
ssh to every Mth node [including the first] in the job
This will ssh to every Mth node of the first N nodes when combined
with the -n option.
The gdbwhere.pl command will run a gdb backtrace [(gdb) where ] for the
running processes on a machine [state R from the ps command]. You
may also specifiy the name of a process as an argument to gdbwhere.pl. When combined with ssh_pbs.pl, the debug technique can
scale to jobs using arbitrary numbers of nodes. The general form of the command line is-
ssh_pbs.pl pbs_jobid "~consult/debug/gdbwhere.pl my_a.out.exe"
|
In the example below, the output from gdbwhere.pl is saved to a file
so that it can be searched or sorted [or both]. The grep command
is used to scan the output for occurences of hello since the
source filename was known to start with that word. Note the output
where the line numbers differ. This is typical of an mpi
application. Not all mpi ranks will be executing the same line number
in your programs.
|
[arnoldg@honest1 ~/debug]$ ssh_pbs.pl 28780 "~consult/debug/gdbwhere.pl" > mygdb.out
132 src/cq.c: No such file or directory.
470 src/cq.c: No such file or directory.
132 src/cq.c: No such file or directory.
484 src/cq.c: No such file or directory.
132 src/cq.c: No such file or directory.
[arnoldg@honest1 ~/debug]$ grep hello mygdb.out
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#7 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#6 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#6 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#7 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#3 0x0000000000401cf2 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:34
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#6 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#3 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401cf2 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:34
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#6 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
|
The first example below will skip over nodes, checking only every 2nd
node for uptime. These examples demonstrate how to check
load and filesystem availability on some of the nodes in the job.
Using -n 2 syntax checks just the first 2 nodes in the job list.
|
[arnoldg@honest1 ~/debug]$ ssh_pbs.pl 28780 -skip 2 uptime
abe1099: 14:38:32 up 31 days, 22:29, 0 users, load average: 7.41, 3.15, 1.67
abe0307: 14:38:33 up 31 days, 23:40, 0 users, load average: 7.19, 2.92, 1.18
abe0305: 14:38:33 up 4 days, 3:16, 0 users, load average: 7.14, 2.85, 1.17
abe0303: 14:38:34 up 4 days, 3:17, 0 users, load average: 7.30, 3.00, 1.23
[arnoldg@honest1 ~/debug]$ ssh_pbs.pl 28780 -n 2 "df -h $HOME"
abe1099: Filesystem Size Used Avail Use% Mounted on
abe-mds1@o2ib0,abe-mds2@o2ib0:/home/client
11T 4.0T 5.8T 42% /u
abe0308: Filesystem Size Used Avail Use% Mounted on
abe-mds1@o2ib0,abe-mds2@o2ib0:/home/client
11T 4.0T 5.8T 42% /u
|
Detail output from gdbwhere.pl for each process looks like
this.
|
[arnoldg@honest1 ~/debug]$ more mygdb.out
abe1099: PROCESS ID: 6482
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6482)]
[New Thread 1084229984 (LWP 6492)]
0x0000002a956d62a8 in MPIDI_CH3I_SMP_pull_header ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d62a8 in MPIDI_CH3I_SMP_pull_header ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d5f90 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a956d314a in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6483
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6483)]
[New Thread 1084229984 (LWP 6490)]
0x0000002a956d5d79 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d5d79 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d314a in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6484
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6484)]
[New Thread 1084229984 (LWP 6496)]
0x0000002a956d7a25 in MPIDI_CH3I_SMP_write_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d7a25 in MPIDI_CH3I_SMP_write_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d3117 in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6485
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6485)]
[New Thread 1084229984 (LWP 6495)]
0x0000002a956d7c0b in MPIDI_CH3I_SMP_write_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d7c0b in MPIDI_CH3I_SMP_write_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d3117 in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6486
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6486)]
[New Thread 1084229984 (LWP 6491)]
0x0000002a956d5d8c in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d5d8c in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d314a in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6487
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6487)]
[New Thread 1084229984 (LWP 6497)]
0x0000002a95715534 in MPIDI_CH3I_MRAILI_Get_next_vbuf ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a95715534 in MPIDI_CH3I_MRAILI_Get_next_vbuf ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d3e70 in MPIDI_CH3I_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a956d3187 in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6488
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6488)]
[New Thread 1084229984 (LWP 6493)]
0x0000002a957161bd in MPIDI_CH3I_MRAILI_Cq_poll ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a957161bd in MPIDI_CH3I_MRAILI_Cq_poll ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d3e80 in MPIDI_CH3I_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a956d3187 in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#5 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
PROCESS ID: 6489
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 6489)]
[New Thread 1084229984 (LWP 6494)]
0x0000002a956d6040 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d6040 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d314a in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
abe0308: PROCESS ID: 31142
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 182916111744 (LWP 31142)]
[New Thread 1084229984 (LWP 31150)]
0x0000002a956d5d52 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#0 0x0000002a956d5d52 in MPIDI_CH3I_SMP_read_progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#1 0x0000002a956d314a in MPIDI_CH3I_Progress ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#2 0x0000002a9572a13c in MPID_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#3 0x0000002a95701ca8 in PMPI_Finalize ()
from /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so
#4 0x0000000000401d25 in main (argc=1, argv=0x7fbfffe1b8) at hello_hang.c:37
|
This example demonstrates use of ssh_pbs.pl and gdbwhere.pl
with a running application. Note that many of the processes are
executing the same area of source code, while some are not. This
may be normal or an indication of a problem. For this test
application it's the expected behavior.
|
[arnoldg@honest1 ~/debug]$ ssh_pbs.pl 28780 "~consult/debug/gdbwhere.pl allall" | grep allall
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e21 in main (argc=4, argv=0x7fbfffe188) at allall.c:85
#6 0x0000000000407e21 in main (argc=4, argv=0x7fbfffe188) at allall.c:85
#5 0x0000000000407e21 in main (argc=4, argv=0x7fbfffe188) at allall.c:85
#5 0x0000000000407e21 in main (argc=4, argv=0x7fbfffe188) at allall.c:85
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe188) at allall.c:134
476 src/cq.c: No such file or directory.
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#8 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#5 0x0000000000407e21 in main (argc=4, argv=0x7fbfffe188) at allall.c:85
#5 0x0000000000407e21 in main (argc=4, argv=0x7fbfffe188) at allall.c:85
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#4 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
132 src/cq.c: No such file or directory.
#5 0x0000000000407f70 in main (argc=4, argv=0x7fbfffe198) at allall.c:101
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#8 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
132 src/cq.c: No such file or directory.
#5 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#8 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#5 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#11 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
470 src/cq.c: No such file or directory.
461 src/cq.c: No such file or directory.
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#8 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#8 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe198) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
[arnoldg@honest1 ~/debug]$ sleep 60
[arnoldg@honest1 ~/debug]$ ssh_pbs.pl 28780 "~consult/debug/gdbwhere.pl allall" | grep allall
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x000000000040818b in main (argc=4, argv=0x7fbfffe188) at allall.c:134
#9 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe188) at allall.c:134
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#6 0x000000000040813e in main (argc=4, argv=0x7fbfffe188) at allall.c:131
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
470 src/cq.c: No such file or directory.
#9 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe188) at allall.c:134
#9 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#8 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#6 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#5 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
307 src/cq.c: No such file or directory.
#9 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#8 0x000000000040818b in main (argc=4, argv=0x7fbfffe188) at allall.c:134
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#9 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe188) at allall.c:134
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x000000000040818b in main (argc=4, argv=0x7fbfffe198) at allall.c:134
#5 0x000000000040813e in main (argc=4, argv=0x7fbfffe188) at allall.c:131
#5 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#7 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
#6 0x0000000000407e6e in main (argc=4, argv=0x7fbfffe188) at allall.c:88
[arnoldg@honest1 ~/debug]$
Scripts
