Sun compares their compiler tools to GNU tools

[mal at bewoner dot dma dot be]

Colin Douglas Howell writes:
 > Sun has published a marketing comparison of their compiler suite,
 > Visual WorkShop C++ 3.0, with the GNU compiler tools.  See the URL:
 > 
 > 	http://www.sun.com/workshop/visual/
 > 
 > The comparison includes performance test results on UltraSPARC.  Of
 > course, this is designed to sell Sun's compiler tools, and is thus
 > rather biased.  Also, their performance tests were done against old
 > versions of gcc; they compare version 4.2 of their C, C++, and FORTRAN
 > compilers with egcs 1.0.1 (and in one case with gcc 2.7.2.2).  So the
 > recent egcs optimizations and SPARC backend improvements are not
 > tested.  (On the other hand, Sun is currently beta-testing version 5.0
 > of their compilers.)
 > 
 > I figured people might want to know what the competition is up to.  Is
 > anyone in a position to comment on the specific tests or do
 > comparisons with the current egcs versions?
 > 

Around here we have the SunSoft Visual Workshop and egcs. I'll add a
few comments on their text.

In the introduction one of their points is "Specific code, such as C++ 
exception handling, ran up to 10 times faster." If EH is on your
critical path, you have problems no compiler can solve.

WRT the chapter on performance, 34% difference on specint is not all
that bad. Specfp that needs fortran is a known problem. g77 is still
not up to commercial standards wrt FORTRAN. On the other hand their
hint that the difference carries over to C because they share the same 
backend is tenuous and doesn't show in their own data.

Another distortion is their claim that they used the "enhanced version 
of Cygnus' gcc, which is usually a version ahead of the freeware
version in terms of performance and functionality" while they were
using egcs which is a "freeware version".

Their case study with Vividata gave numbers between 7 and 30%
difference on runs where the numbers represented the *best times* for
each compiler. This is methodologically unsound since you have to take 
into account scheduling issues etc. They should take the means or
publish the whole data. Further for this case study they used gcc
2.7.2.2 instead of egcs.

Their most touted advantages are not compiler related. They hype their 
Workshop integrated tools (of which an incredibly buggy version of
Xemacs 20.0 is the core ;-) ). There, the comparison is decidedly
unfair since they compare their debug session with tooltalk integrated 
dbx and emacs to a command line gdb session. I personally find gud in
Xemacs more usable than the Sun stuff. And for some really complex
debugging sessions ddd is even better. 

Other tools include an enhanced performance profiler tcov and
automatic profile feedback-based optimisations. This is an area in
which egcs is lacking.

They also have a class browser. Since OOBR is no longer distributed
with XEmacs there is also a lacune there. Although of our 8 developers 
here nobody uses the thing. YMMV

Their Runtime Checking can be done with a special malloc library and
things like efence are more advanced. It's something of a poor man's
Purify. 

Their appendix B claims to publish the full SPEC test results but
their is some error in it since they don't seem to publish the gcc
results. I also couldn't find which flags they were using since they
only publish the flags used for Sun cc.

It would also be interesting to test the difference something like GNU 
rope would make.

Arguing about preferences between tools ends pretty fast in a
religious war but I find their comparison very flawed.

 > Also, this comparison is currently being discussed at slashdot.org,
 > just in case anyone wants to add a voice of reason into that
 > discussion.
 > 

Life is too short to take up the task of inserting reason in
/. discussions. 

 > -- 
 > Colin Douglas Howell                    Systems Administrator
 > e-mail:  howell@cs.stanford.edu         Computer Facilities Group
 > office:  (650) 723-2491                 Computer Science Department
 >                                         Stanford University