Re: make -j lots building and testing gcc

[Bill Northcott <w dot northcott at unsw dot edu dot au>]

On 09/02/2005, at 6:32 PM, Zack Weinberg wrote:
> > When running the test suite with 'make -k -j 20 check'  there is never
> > more than one instance of cc1 or cc1plus running although there may be
> > up to 4 instances of expect.
>
> DejaGNU does not parallelize, and neither does the custom script used
> for the Ada test suite.  Thus, the maximum number of concurrent
> processes under 'make -k -jN check' is the number of distinct
> testsuites, which I believe is approximately 10 with all front ends
> enabled.
>
> You can in theory get up to that number of concurrent compiler
> processes, not just expect processes.  I suspect what you saw is a
> sampling artifact - most of the test cases are quite small, and so the
> compiler will complete before top refreshes.
>
> As Andrew pointed out upthread, 20001226-1.c after macro expansion is
> a very large piece of code.  It is relatively normal for it to cause
> the compiler to run out of memory.  I suspect that most of your
> failures are down to something else.  (And yes, moving the build
> directory breaks stuff.)

FYI below is a top display from the machine running a bootstrap with 
make -j 20 (all the cc1 s) and 'make -k -j 20 check'  As you can see 
there is only one instance of cc1plus being run by the test suite.

There is also a huge amount of free memory.  When I was running the 
earlier stuff there was a sas job which probable working on a 500GB 
file, which might explain the lack of memory.

Bill

load averages: 22.95, 20.58, 16.12                                      
    19:27:33
194 processes: 23 running, 63 sleeping, 106 idle, 2 zombie
CPU states: 95.0% user,  0.0% nice,  4.9% system,  0.0% idle
Memory: Real: 809M/8027M act/tot  Virtual: 227M/8183M use/tot  Free: 
6509M

  PID    UID   PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
324311    116    52    0   46M   39M run     0:17 46.50% cc1
324536    116    55    0   27M   22M run     0:05 46.20% cc1
324499    116    55    0   28M   22M run     0:05 46.00% cc1
324300    116    53    0   47M   39M run     0:18 45.10% cc1
324284    116    53    0   44M   38M run     0:21 45.00% cc1
324204    116    53    0   46M   41M run     0:22 44.30% cc1
324537    116    53    0   24M   19M run     0:03 35.20% cc1
324496    116    51    0   28M   22M run     0:05 34.30% cc1
324416    116    51    0   38M   31M run     0:11 34.00% cc1
322289    116    53    0   65M   56M run     2:51 33.90% cc1plus
324329    116    51    0   43M   36M run     0:16 33.70% cc1
324271    116    52    0   48M   40M run     0:19 32.90% cc1
324501    116    52    0   27M   21M run     0:05 32.60% cc1
324507    116    52    0   29M   23M run     0:05 32.50% cc1
324545    116    52    0   20M   16M run     0:02 32.20% cc1
324532    116    52    0   24M   19M run     0:04 31.50% cc1
324559    116    52    0   21M   16M run     0:02 31.40% cc1
324430    116    51    0   33M   27M run     0:08 30.30% cc1
324538    116    52    0   18M   13M run     0:01 29.70% cc1
324553    116    51    0   18M   13M run     0:01 28.60% cc1
284992    100    45    0 9048K 6078K sleep   5:42  6.30% top
324007    116    46    0 9048K 6045K run     0:04  3.60% top
 1072      0    44    0 6160K  663K sleep   4:15  1.20% envmond
323532    116    45    0 4952K 3039K sleep   0:09  1.10% make
39386    116    44    0   20M   13M sleep  16:46  0.30% Xvnc
39666    116    44    0   32M   20M run     3:14  0.10% xemacs-21.1.11
324548    116    44    0 2176K  262K sleep   0:00  0.10% gcc-3.4
324540    116    44    0 2176K  262K sleep   0:00  0.10% gcc-3.4
 1283    213    42    0   16M 2662K run     0:41  0.00% mysqld
30237    296    44    0   47M   17M sleep 409:54  0.00% sas
74371    296    44    0   48M   16M sleep  55:52  0.00% sas
 1209    266    44    0  406M   21M sleep  22:43  0.00% oracle
   58      0    44    0 1936K  114K sleep   6:21  0.00% update
 1297    197    44    0   33M 6840K sleep   5:25  0.00% dataserver
 1199    266    44    0  413M 5005K sleep   4:32  0.00% oracle
39491    116    42    0   47M 1228K sleep   3:32  0.00% real-netscape
 1201    266    42    0  409M 5775K sleep   2:42  0.00% oracle
 1207    266    44    0  405M 4243K sleep   2:14  0.00% oracle
117988    116    44    0   11M 1556K sleep   2:13  0.00% dtterm
 4027    161    44    0   13M  147K sleep   2:12  0.00% dtfile