Bug 43601 - Enormous increase in DLL object files size in 4.5
Summary: Enormous increase in DLL object files size in 4.5
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: c++ (show other bugs)
Version: 4.6.0
: P3 normal
Target Milestone: 4.6.0
Assignee: Dave Korn
URL: http://gcc.gnu.org/ml/gcc-patches/201...
Keywords: patch
Depends on: 47218
Blocks:
  Show dependency treegraph
 
Reported: 2010-03-31 12:30 UTC by Vadim Zeitlin
Modified: 2011-06-08 14:26 UTC (History)
15 users (show)

See Also:
Host: i686-pc-mingw32
Target:
Build:
Known to work: 3.4.5
Known to fail: 4.3.4, 4.5.0, 4.6.0
Last reconfirmed: 2011-01-09 17:25:25


Attachments
enable auto-import in ld (338 bytes, patch)
2010-05-15 09:29 UTC, boqun li
Details | Diff
don't emit dllexport'd inline functions (858 bytes, patch)
2010-05-15 09:31 UTC, boqun li
Details | Diff
appbase.cpp file from wxWidgets compiled with g++ 4.4 (36.35 KB, application/octet-stream)
2010-10-14 15:42 UTC, Vadim Zeitlin
Details
appbase.cpp file from wxWidgets compiled with g++ 4.5 (186.33 KB, application/octet-stream)
2010-10-14 15:44 UTC, Vadim Zeitlin
Details
appbase.cpp file from wxWidgets compiled with g++ 4.4 (33.59 KB, application/octet-stream)
2010-10-14 15:45 UTC, Vadim Zeitlin
Details
appbase.cpp file from wxWidgets compiled with g++ 3.4 (31.48 KB, application/octet-stream)
2010-10-14 16:01 UTC, Vadim Zeitlin
Details
appbase.cpp file from wxWidgets compiled with MSVC 9 (a.k.a. 2008) (76.27 KB, application/octet-stream)
2010-10-14 16:01 UTC, Vadim Zeitlin
Details
appbase.cpp file from wxWidgets compiled with MSVC 9 (a.k.a. 2008) (119.60 KB, application/octet-stream)
2010-10-14 16:05 UTC, Vadim Zeitlin
Details
trial patch (817 bytes, patch)
2011-01-09 17:30 UTC, Dave Korn
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Vadim Zeitlin 2010-03-31 12:30:46 UTC
Hello,

 This is not necessarily a bug but I don't know about any other place to
discuss this so I'm reporting it here -- please let me know if I should post
somewhere else.

 Anyhow, there seems to be a big problem with building shared C++ libraries
using the typical

	class __attribute(dllexport) SomePublicClass
	{
	public:
		... a lot of inline functions without dllexport ...

		... and also a lot of non-inline functions ...
	};

pattern, wxWidgets being one particular example (but AFAIK there is really
nothing special about wxWidgets here). I've described the problem in details
in http://thread.gmane.org/gmane.comp.gnu.mingw.user/32435 but, basically,
the size of object files produced by mingw32 4.5 has been multiplied by 10
compared to mingw32 4.4 making linking them extremely time consuming and
plainly impossible on machines with modest amounts of RAM.

 This seems to be due to the following change in gcc itself (and not
mingw32): http://gcc.gnu.org/viewcvs?view=revision&revision=147799 and I'd
like to know if there is any way to undo its effect. For example, the reply
from Cesar Strauss in the thread above suggested using -fhide-inline-functions
option for this. IMHO ideal would be to revert to the old behaviour at
least until some better approach can be found but any solution would be
nicer than nothing.

 Thanks in advance,
VZ
Comment 1 Andrew Pinski 2010-03-31 21:47:51 UTC
This is by design and the patch even mentions that  Visual Studio 2005 follows the same rule. 
Comment 2 Vadim Zeitlin 2010-03-31 22:04:51 UTC
I'm sorry but is this really all you have to say about this? Granted, VS does
follow the same rule but the size of object files produced by it was twice
less than that of object files produced by gcc _before_ this change and it
would seem to me that keeping the size of object files reasonable should have
higher precedence than implementing the same behaviour that the Microsoft
compiler implements. I don't know how does VS manage to avoid this exponential
growth of object files but it demonstrably does while gcc does not.

Once again, it's a serious problem for many developers that the size of object
files is now 500MB instead of 50MB. Linking a module from 500MB of object
files requires a lot of RAM and takes a long time. And, to top it all, the
change resulting in this regression (because from the point of view of any gcc
user this is how it will look) doesn't bring any tangible benefits --
compatibility with VS notwithstanding. While generating inline methods in
DLL might be desirable (although the patch also only speaks about ARM ABI
so it doesn't seem like this is really _required_ under x86/x64), it seems
strange to completely disregard practical consequences of this lofty idea.

Please reconsider your decision, IMO at least an option restoring the old
behaviour (with a prominent mention in release notes/changelog) is badly
needed.

P.S. Personally, I'd also love to understand why exactly was this change
     considered so desirable at all. But this is just out of my personal
     curiosity. What really matters is that plenty of people will be simply
     unable to compile projects which they had no trouble compiling with
     gcc 4.4. This will result in massive amounts of pain all around.
Comment 3 Andrew Pinski 2010-03-31 22:11:27 UTC
Also this change was done to so we would be compliant to the ARM EABI.

http://gcc.gnu.org/ml/gcc-patches/2009-05/msg01481.html

And if the linker is taking a lot of memory, then it is a bug in the linker.  The linker should not take much more memory for functions which are linked once.
Comment 4 Andrew Pinski 2010-03-31 22:12:22 UTC
And really it make sense that the functions are exported are marked as dllexport even if they are vague linkage.
Comment 5 Vadim Zeitlin 2010-03-31 22:25:23 UTC
(In reply to comment #3)
> And if the linker is taking a lot of memory, then it is a bug in the linker. 
> The linker should not take much more memory for functions which are linked
> once.

Let's admit this. How does it help us in practice? I'm not aware of any plans to optimize the linker to deal with this situation better so the fact remains that the users who could build wxWidgets before won't be able to build it any more. How should we, as its developers, deal with it? Advise them to contribute to binutils development? Somehow I am not sure if this is going to work really well.

And I won't even mention that we still have .5GB of object files for each build (and you may need to have several and this is not negligible even nowadays if you use a 80GB SSD).

So while the ideal solution might be to avoid the generation of duplicate copies of the functions in all object files or making the linker smarter the fact is that in practice we have a big uncalled for and unwanted regression. With the only justification (for non-ARM platforms) that "it really makes sense". IMNSHO it doesn't make sense to break working builds of many thousands of projects. If you think it does, please explain how should we deal with the inevitable complaints about it when people start switching to gcc 4.5. I frankly have absolutely no idea.

I won't insist any more but for one last time: Please, please, please try to view the situation from a gcc user and not gcc developer point of view. Just imagine that you try a new minor release of the compiler and suddenly the build takes much longer or maybe doesn't complete at all because linker runs out of memory. What are you going to do and how are you going to deal with it? And this is not a whimsical scenario but will likely apply to the *majority* of non-trivial C++ projects as many of them use some class library with a lot of inline function inside dllexported classes, be it wxWidgets, Qt or whatever else.
Comment 6 Andrew Pinski 2010-03-31 22:32:20 UTC
My view this is a bug in how wxWidgets uses (abuses) dllexport and wanting not to export inline functions also.
Comment 7 Vadim Zeitlin 2010-03-31 22:36:15 UTC
(In reply to comment #6)
> My view this is a bug in how wxWidgets uses (abuses) dllexport and wanting not
> to export inline functions also.

Andrew, could you please provide a reasonable alternative to what we do?

Also, once again, I'm only aware of this problem because of a bug report of a wxWidgets user but I really don't think there is anything specific to wx here. To the best of my knowledge absolutely nobody is masochistic enough to export every non-inline function in a class separately, all C++ projects supporting Win32 DLL I ever saw used dllexport declaration with the class itself.

So, once again, if this is our bug, would you be so kind as to propose a solution? I'm sure many other C++ library developers would be interested in hearing it as well.

TIA!
Comment 8 Wolfgang Bangerth 2010-04-02 11:37:08 UTC
I think this is a bug the MingW maintainers should handle.

While I understand Andrew's position, it seems to me that this is nevertheless
a definite regression from the user's perspective.

W.
Comment 9 Vadim Zeitlin 2010-04-03 17:15:50 UTC
Just to bring some more hard numbers into this discussion, I've installed both
4.4 and 4.5 (in addition to 3.4.5 which I'll use as a kind of baseline) on my
own machine (4/8 physical/logical CPUs, 8GB of RAM, Windows 7 64 bits). The
results of building the (main) part of wxWidgets with default configure
options excluding some third party libraries (jpeg/tiff/regex):

- With 3.4.5 (mingw-vista special r3):

% time make -sj8 wxcore
make -sj8 wxcore  30.29s user 51.71s system 46% cpu 2:55.61 total
% du -sh lib
24M lib
% du -sSh .
29M .

- With 4.4.0 (GCC):

% time make -sj8 wxcore
make -sj8 wxcore  24.50s user 43.66s system 46% cpu 2:27.97 total
% du -sh lib
29M lib
% du -sSh .
90M .

- With 4.5.0 20100311 (experimental) (GCC):

% time make -sj8 wxcore
make -sj8 wxcore  33.25s user 57.26s system 6% cpu 23:26.19 total
% du -sh lib
80M lib
% du -sSh .
554M .

- Summary of the sizes of the DLL:

% du -h */lib/wxmsw291u_core_gcc_custom.dll
9.2M    wx-mswudll-gcc3.4/lib/wxmsw291u_core_gcc_custom.dll
12M     wx-mswudll-gcc4.4/lib/wxmsw291u_core_gcc_custom.dll
47M     wx-mswudll-gcc4.5/lib/wxmsw291u_core_gcc_custom.dll


The numbers are quite clear and this is without debug information! Notice the
drastic reduction in the CPU usage for 4.5: most of the time is clearly spent
writing the files (and it's an Intel G2 SSD, things would probably be even
worse with a slower disk) and not compiling at all. And while the increase in
the size of the object files (they are mostly what makes for "du -sSh" output)
is less than what was reported before, the size of the DLLs themselves
increased as well (the previous tester couldn't report this as he didn't even
manage to link them, of course).

And while the compilation time change alone (10 times slower!) makes 4.5
unusable IMO, the change in library sizes is pretty horrible too (almost 4
times larger).

I don't know who should handle it but I don't understand how can anybody not
see that it is a real problem.

Comment 10 Andrew Pinski 2010-04-03 17:25:20 UTC
>And while the compilation time change alone

How did you configure 4.5?  Did you use --enable-checking=release ?  If not then the compile time numbers are not comparable at all.
Comment 11 Vadim Zeitlin 2010-04-03 17:46:37 UTC
(In reply to comment #10)
> >And while the compilation time change alone
> 
> How did you configure 4.5?  Did you use --enable-checking=release ?  If not
> then the compile time numbers are not comparable at all.

Ok, maybe this is the reason then? Because I have, in the version downloaded from http://sourceforge.net/projects/mingw/files/MinGW%20Proposed/gcc-4.5.0_20100311-2/gcc-core-4.5.0_20100311-2-mingw32-bin.tar.lzma/download, the following:

% /dev/mingw/4.5.0/bin/gcc -v
Using built-in specs.
COLLECT_GCC=c:\dev\mingw\4.5.0\bin\gcc.exe
COLLECT_LTO_WRAPPER=c:/dev/mingw/4.5.0/bin/../libexec/gcc/mingw32/4.5.0/lto-wrapper.exe
Target: mingw32
Configured with: ../gcc-4.5.0_20100311/configure --enable-languages=c,c++,ada,fortran,objc,obj-c++ --disable-sjlj-exceptions --with-dwarf2 --enable-shared --enable-libgomp --disable-win32-registry --enable-libstdcxx-debug --enable-version-specific-runtime-libs --disable-werror --build=mingw32 --prefix=/mingw
Thread model: win32
gcc version 4.5.0 20100311 (experimental) (GCC)

i.e. --enable-checking=release is not there. But can this affect the object file and DLL sizes too?
Comment 12 Vadim Zeitlin 2010-04-03 18:17:42 UTC
Actually I don't think --enable-checking=release changes anything. I've just tried Cesar Strauss's suggestion to not use __attribute__((dllexport)) in the code at all but use --enable-auto-import linker option. And miraculously this solves all the problems:

% ..configure with LDFLAGS=-Wl,--enable-auto-import ...
% time make -sj8 wxcore
make -sj8 wxcore  18.02s user 34.75s system 34% cpu 2:34.81 total
% du -sh lib
26M     lib
% du -sSh .
24M .
% du -h lib/wxmsw291u_core_gcc_custom.dll
8.7M    lib/wxmsw291u_core_gcc_custom.dll

To summarize: the compilation time is the same as with 4.4 (7s difference is roughly the same as measurement precision anyhow) and the library size is even smaller than with 3.4.5.

So it seems that dllexport attribute should simply never be used with 4.5. I was opposed to using --enable-auto-import initially because it exports too much (there are plenty of symbols which shouldn't be exported in the DLL export table now) but its advantages are such that we're clearly going to do it with 4.5.

I still wish somebody could explain me why can auto import work so efficiently but dllexport had to be made unusable like this.
Comment 13 Danny Smith 2010-04-04 08:20:44 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > >And while the compilation time change alone
> > 
> > How did you configure 4.5?  Did you use --enable-checking=release ?  If not
> > then the compile time numbers are not comparable at all.
>
<snip> 
> gcc version 4.5.0 20100311 (experimental) (GCC)
> 

This will cause defau;t checks for experimental DEV-PHASE to be enabled.
Danny 
Comment 14 John E. / TDM 2010-04-23 00:05:17 UTC
This affects a *ton* of code in the wild. Can we at least get a command line flag like "-fno-emit-inline-dllexports"?
Comment 15 Yuchen Deng 2010-05-03 11:11:46 UTC
The problem is too serious! I have 4G memory, but not able to compile wxWidgets 2.8.10.
Comment 16 boqun li 2010-05-15 06:01:16 UTC
(In reply to comment #8)
> I think this is a bug the MingW maintainers should handle.
> 
> While I understand Andrew's position, it seems to me that this is nevertheless
> a definite regression from the user's perspective.
> 
> W.
> 

Yes, I think this is mostly a mingw bug, but I think developers of cygwin, a very similar project had already resolved this issue, they had enabled auto-import by default in ld, please follow this post
[PATCH] Silence ld auto-import warnings for pe-i386.
http://sourceware.org/ml/binutils/2009-02/msg00341.html
for details.
   With auto-import enabled by default in ld, there is no need to emit exported inline functions in gcc code.
   See my workaround here:
FYI: GCC 4.5.0 on Mingw.org
http://forums.codeblocks.org/index.php/topic,12183.msg85066.html#msg85066
Comment 17 boqun li 2010-05-15 09:29:11 UTC
Created attachment 20662 [details]
enable auto-import in ld

This patch comes from a post by Dave Korn and is slightly modified by me
[PATCH] Silence ld auto-import warnings for pe-i386.
http://sourceware.org/ml/binutils/2009-02/msg00341.html
Comment 18 boqun li 2010-05-15 09:31:19 UTC
Created attachment 20663 [details]
don't emit dllexport'd inline functions

This patch is just a removal of nathan's code.
See
r147799 | nathan | 2009-05-22 22:57:15 +0800 (Fri, 22 May 2009) | 16 lines
in gcc svn log
Comment 19 Tim Stahlhut 2010-05-15 11:53:32 UTC
Should not the options -fvisibility-inlines-hidden or -fvisibility=hidden fix this problem? Option found on http://gcc.gnu.org/wiki/Visibility.
Note: The MinGW GCC 4.5.0-1 does not make smaller DLLs using above options.


 
Comment 20 boqun li 2010-05-15 13:24:56 UTC
(In reply to comment #19)
> Should not the options -fvisibility-inlines-hidden or -fvisibility=hidden fix
> this problem? Option found on http://gcc.gnu.org/wiki/Visibility.
> Note: The MinGW GCC 4.5.0-1 does not make smaller DLLs using above options.
> 
> 
> 
> 

I don't think these option can undo the change by nathan at 2009-05-22
------------------------------------------------------------------------
r147799 | nathan | 2009-05-22 22:57:15 +0800 (Fri, 22 May 2009) | 16 lines

   gcc/
   * tree.c (handle_dll_attribute): Mark dllexport'd inlines as
   non-external.

   gcc/cp
   * decl2.c (decl_needed_p): Consider dllexport'd functions needed.
   * semantics.c (expand_or_defer_fn): Similarly.

   gcc/testsuite/
   * gcc.dg/dll-6.c: New test.
   * gcc.dg/dll-6a.c: Likewise.
   * gcc.dg/dll-7.c: Likewise.
   * gcc.dg/dll-7a.c: Likewise.
   * g++.dg/ext/dllexport2.C: Likewise.
   * g++.dg/ext/dllexport2a.cc: Likewise.

------------------------------------------------------------------------
   
   BTW, say SORRY to nathan, your change is indeed useful for building libstdc++ dll, but I have found an alternative approach. 

Indeed, the explanation page
http://gcc.gnu.org/wiki/Visibility
says "How to use the new C++ visibility support
In your header files, wherever you want an interface or API made public outside the current DSO, place __attribute__((visibility("default"))) in struct, class and function declarations you wish to make public (it's easier if you define a macro as this). You don't need to specify it in the definition. Then, alter your make system to pass -fvisibility=hidden to each call of GCC compiling a source file.", 
this means to use these options, you should alter your header files first, but wxwidgets source code apparently don't contain anything like "__attribute__((visibility("default")))".
    Correct me if I'm wrong.
Comment 21 Ruben Van Boxem 2010-09-22 10:06:53 UTC
What is the status of this problem? Having every project depending on MinGW(.org/-w64) toolchains modify their code is not an option. I see that the main problem is dllexported *inline* functions. Can Nathan's change be modified to only emit dllexported *non-inline* functions? I see no reason why that wouldn't work (and together with auto-import enabled in ld).

IMO this would 
a) fix the large object file size
b) consequently fix ld out of memory issues
c) work fine for old sources which expect old behavior because ld has auto-import enabled.

This has been a long-standing issue, which needs a fix soon.

PS: this bug should be marked as NEW and the affected host and target should be *-*-mingw*. Could someone who can change that, do that please?
Comment 22 Dave Korn 2010-09-23 03:56:19 UTC
(In reply to comment #20)

> Indeed, the explanation page
> http://gcc.gnu.org/wiki/Visibility
 [ ... ]
> this means to use these options, you should alter your header files first, but
> wxwidgets source code apparently don't contain anything like
> "__attribute__((visibility("default")))".

Visibility is only supported on platforms that use the ELF file format; dllexport/dllimport already is the equivalent feature for windows.

Comment 23 Dave Korn 2010-09-23 04:08:18 UTC
(In reply to comment #21)
> I see that the main problem is dllexported *inline* functions.

  That is my understanding of it too.

> Can Nathan's change be modified
> to only emit dllexported *non-inline* functions? I see no reason why that
> wouldn't work (and together with auto-import enabled in ld).

  Hmm.  I can think of three approaches, any or all of which might be useful:

- implement "-fno-emit-inline-dllexports" as suggested in comment 14.
- add "nodllexport" and "nodllimport" attributes that can be applied to individual class members to prevent them inheriting the overall attribute applied to the class
- modify Nathan's patch to not emit dllexported inline functions when they have attribute always_inline.

  Of course two of these require annotating the code, so I think that we'd probably still want #1 as well for convenience.

  I see the thread at http://thread.gmane.org/gmane.comp.gnu.mingw.user/32435 (mentioned in the initial bug report) petered out without any analysis of exactly where all the file size bloat is coming from.  It would be really instructive if one of the mingw guys could look at the .o files built by the two different compiler versions (maybe using "objdump -h" or similar) and try to find out where all the space is going, particularly making sure it's not mostly debug or EH tables or something like that.

  Also, it would be good to know if GCC and MSVC really are emitting the same set of function definitions, as it would be surprising if the figures were that far apart from each other if they were doing so.  Unless it's some kind of unexpected contribution from debug/EH data, as suggested earlier, but maybe there's something malfunctioning in function cloning or something, without a bit of analysis we can't be sure.

Comment 24 Cesar Strauss 2010-09-24 04:19:19 UTC
The wxWidgets build enables precompiled headers on MSVC by default, while the GCC build does not use them. As it turns out, this setting affects the object size on MSVC. When I disabled precompiled headers on MSVC, the object sizes increased, although the final DLL size did not change much. Both cases were built with -O2 and without debug information.

Object size in bytes for wxWidgets file src/common/any.cpp:
MSVC, precompiled headers enabled:   314515
MSVC, precompiled headers disabled: 2889844
GCC,  precompiled headers disabled: 1080971

MSVC version: 2008 Express
GCC  version: 4.5.0

At the very least, this indicates one should disable precompiled header use when comparing the output of both compilers, as Dave suggested on comment #23.
Comment 25 Dave Korn 2010-09-24 05:00:16 UTC
  I'm going to be working on a patch in this area in the next few weeks (I want to make the dllexport/dllimport attributes work on C++ namespaces), and I can probably do something about this issue while I'm in there anyway.

  Before I put a bunch of time into that, however, I'd like to see actual diagnosis of what's in the different .o files so that I know that I won't be wasting my time, e.g. what if the extra emitted functions is actually only contributing 10% of this growth in file sizes and the other 90% comes from debug info or EH tables?  If something like that did turn out to be the case, then the effort would be wasted as the new option wouldn't actually solve your link-time problems anyway.

  So I would like to see some proper detailed analysis on object files establishing exactly what constitutes all this bloat and where it comes from before I commit to what might be the wrong path of action.


[ Vadim, it's no use sending me off-list mail if your domain doesn't accept replies from gmail.com!  Email addresses @gcc.gnu.org are just redirectors to maintainers' primary accounts. ]
Comment 26 Yu Simin 2010-09-24 20:21:08 UTC
[English is not my native language ...]
In GCC 4.5 or later, both the compilation time and the sizes of the generated object files is far longer or far larger than MSVC when PCH is enabled in both of the compilers.
MSVC requires a DUMMY source file eg:dummy.cpp to be used when generating the PCH file and will generate an object file eg:dummy.obj. But the dummy object file is not dummy at all. It contains all the common codes generated from the sources/headers of the PCH. 
In GCC, PCH is just preprocess + compiling, no common code (is this case dllexports) generating. so every compilation unit will have to generate the codes in the PCH file again and again while MSVC will not generate them once more.

We may add a flag for example -fgenerate-pch-codes/-fno-generate-pch-codes (or some pragmas) to make GCC work similar to MSVC.
For example, if we make -fno-generate-pch-codes the default
gcc -O2 -o foo.h.gch foo.h
gcc -O2 -fgenerate-pch-codes -o dummy.o dymmy.cpp  # add this to generate the codes of foo.h to dummy.o
# assuming that other1.cpp/other2.cpp use foo.h.gch
gcc -O2 -o other1.o other1.cpp # no code in foo.h will be generated on other1.o
gcc -O2 -o other2.o other2.cpp # no code in foo.h will be generated on other2.o
...


Here is my patch (picked from my local copy so maybe someting is missing, a bit different from above and only handle the dllexport case and may have bug for trunk). with this patch I successfully built wxWidget 2.8.10 & trunk (slightly modified to make PCH enabled for GCC and added a dummy file with my pragma below), the monolithic DLL seems ok (~9M in size, at least the wxWidget demos & examples work well). 


I only care about the compilation time and the size of objects because it make ld failed to link the wxWidget DLL on my windows box (I do not want to revert the dllexports change. before this patch is out, I had to link it on 64bit linux in VirtualBox by a cross toolchain and the generated DLL is ~25MB! This size explosion seems to be caused by the inline strategy, but I did not look into it)

I won't have any time to maintain the patch, just hope it will be useful to help solving the problem finally.

In this patch I've introduced one flag
-fpch-ignore-inline-dllexports (-fno-pch-ignore-inline-dllexports)
and one pragma
#pragma pch_ignore_inline_dllexports on/off


Index: c-family/c-pragma.c
===================================================================
--- c-family/c-pragma.c (版本 164605)
+++ c-family/c-pragma.c (工作副本)
@@ -1314,7 +1314,9 @@
   if (!flag_preprocess_only)
     cpp_register_deferred_pragma (parse_in, "GCC", "pch_preprocess",
                                  PRAGMA_GCC_PCH_PREPROCESS, false, false);
-
+  cpp_register_deferred_pragma (parse_in, "GCC", "pch_ignore_inline_dllexports",
+                               PRAGMA_GCC_PCH_IGNORE_INLINE_DLLEXPORTS, false, false);
+
 #ifdef HANDLE_PRAGMA_PACK
 #ifdef HANDLE_PRAGMA_PACK_WITH_EXPANSION
   c_register_pragma_with_expansion (0, "pack", handle_pragma_pack);
Index: c-family/c-pragma.h
===================================================================
--- c-family/c-pragma.h (版本 164605)
+++ c-family/c-pragma.h (工作副本)
@@ -47,6 +47,8 @@

   PRAGMA_GCC_PCH_PREPROCESS,

+  PRAGMA_GCC_PCH_IGNORE_INLINE_DLLEXPORTS,
+
   PRAGMA_FIRST_EXTERNAL
 } pragma_kind;

Index: cgraph.h
===================================================================
--- cgraph.h    (版本 164605)
+++ cgraph.h    (工作副本)
@@ -125,6 +125,10 @@
   /* True if the function is going to be emitted in some other translation
      unit, referenced from vtable.  */
   unsigned vtable_method : 1;
 };

 /* Information about the function that needs to be computed globally
Index: tree.h
===================================================================
--- tree.h      (版本 164605)
+++ tree.h      (工作副本)
@@ -2706,6 +2706,13 @@
 #define DECL_GIMPLE_REG_P(DECL) \
   DECL_COMMON_CHECK (DECL)->decl_common.gimple_reg_flag

+/* To support fkeep-dllexport-inline-functions-in-pch,
+   indicates that the dllexport inline function is defined in PCH */
+#define DECL_DLLEXPORT_INLINE_IN_PCH_P(DECL) \
+  DECL_COMMON_CHECK (DECL)->decl_common.dllexport_inline_in_pch
+
+
 struct GTY(()) tree_decl_common {
   struct tree_decl_minimal common;
   tree size;
@@ -2759,6 +2765,10 @@
   unsigned int off_align : 8;

   /* 24-bits unused.  */
+
+  /* To support fkeep-dllexport-inline-functions-in-pch */
+  unsigned int dllexport_inline_in_pch : 1;

   /* DECL_ALIGN.  It should have the same size as TYPE_ALIGN.  */
   unsigned int align;
Index: common.opt
===================================================================
--- common.opt  (版本 164605)
+++ common.opt  (工作副本)
@@ -1023,6 +1023,10 @@
 Common Report Var(flag_keep_inline_functions)
 Generate code for functions even if they are fully inlined

+fpch-ignore-inline-dllexports
+Common Report Var(flag_pch_ignore_inline_dllexports) Init(0)
+Do not generate code for inline functions defined in the PCH file
+
 fkeep-static-consts
 Common Report Var(flag_keep_static_consts) Init(1)
 Emit static const variables even if they are not used
Index: cp/semantics.c
===================================================================
--- cp/semantics.c      (版本 164605)
+++ cp/semantics.c      (工作副本)
@@ -3436,9 +3436,19 @@
         be emitted; there may be callers in other DLLs.  */
       if ((flag_keep_inline_functions
           && DECL_DECLARED_INLINE_P (fn)
-          && !DECL_REALLY_EXTERN (fn))
-         || lookup_attribute ("dllexport", DECL_ATTRIBUTES (fn)))
+          && !DECL_REALLY_EXTERN (fn)))
        mark_needed (fn);
+      else if(lookup_attribute ("dllexport", DECL_ATTRIBUTES (fn)))
+        {
+         if (pch_file)
+           {
+             DECL_DLLEXPORT_INLINE_IN_PCH_P (fn) = 1;
+           }
+         mark_needed (fn);
+        }
+
     }

   /* There's no reason to do any of the work here if we're only doing
Index: cp/decl2.c
===================================================================
--- cp/decl2.c  (版本 164605)
+++ cp/decl2.c  (工作副本)
@@ -1784,6 +1784,13 @@
      COMDAT until that point.  */
   gcc_assert (at_eof);

+  if (DECL_DLLEXPORT_INLINE_IN_PCH_P (decl)
+      && flag_pch_ignore_inline_dllexports
+      && !pch_file)
+    return false;
+
   /* All entities with external linkage that are not COMDAT should be
      emitted; they may be referred to from other object files.  */
   if (TREE_PUBLIC (decl) && !DECL_COMDAT (decl))
Index: cp/parser.c
===================================================================
--- cp/parser.c (版本 164605)
+++ cp/parser.c (工作副本)
@@ -23828,6 +23828,30 @@
   cp_lexer_get_preprocessor_token (NULL, first_token);
 }

+static void
+cp_parser_pch_ignore_inline_dllexports (cp_parser *parser)
+{
+  cp_token* tok;
+  /* Handle pch_ignore_inline_exports */
+  tok = cp_lexer_peek_token (parser->lexer);
+  if (tok->type == CPP_NAME)
+    {
+      const char *p = IDENTIFIER_POINTER (tok->u.value);
+
+      if (!strcmp ("on", p))
+       flag_pch_ignore_inline_dllexports = 1;
+      else if(!strcmp ("off", p))
+       flag_pch_ignore_inline_dllexports = 0;
+      else
+       error_at (tok->location,
+                 "expected %<on%> or %<off%>");
+      cp_lexer_consume_token (parser->lexer);
+    }
+  else
+    flag_pch_ignore_inline_dllexports = 1;
+
+}
+
 /* Normal parsing of a pragma token.  Here we can (and must) use the
    regular lexer.  */

@@ -23849,6 +23873,10 @@
                "%<#pragma GCC pch_preprocess%> must be first");
       break;

+    case PRAGMA_GCC_PCH_IGNORE_INLINE_DLLEXPORTS:
+      cp_parser_pch_ignore_inline_dllexports (parser);
+      break;
+
     case PRAGMA_OMP_BARRIER:
       switch (context)
        {
Index: c-parser.c
===================================================================
--- c-parser.c  (版本 164605)
+++ c-parser.c  (工作副本)
@@ -7130,6 +7130,29 @@
 }

+static void
+c_parser_pragma_pch_ignore_inline_dllexports (c_parser *parser)
+{
+  c_parser_consume_pragma (parser);
+
+  if (c_parser_next_token_is (parser, CPP_NAME))
+    {
+      const char *p = IDENTIFIER_POINTER (c_parser_peek_token (parser)->value);
+      if (!strcmp ("on", p))
+       flag_pch_ignore_inline_dllexports = 1;
+      else if(!strcmp ("off", p))
+       flag_pch_ignore_inline_dllexports = 0;
+      else
+       c_parser_error (parser, "expected %<on%> or %<off%>");
+      c_parser_consume_token (parser);
+    }
+  else
+    flag_pch_ignore_inline_dllexports = 1;
+
+  parser->error = true;
+  c_parser_skip_to_pragma_eol (parser);
+}
+
 /* Handle pragmas.  Some OpenMP pragmas are associated with, and therefore
    should be considered, statements.  ALLOW_STMT is true if we're within
    the context of a function and such pragmas are to be allowed.  Returns
@@ -7194,6 +7217,10 @@
       c_parser_skip_until_found (parser, CPP_PRAGMA_EOL, NULL);
       return false;

+    case PRAGMA_GCC_PCH_IGNORE_INLINE_DLLEXPORTS:
+      c_parser_pragma_pch_ignore_inline_dllexports (parser);
+      return false;
+
     default:
       if (id < PRAGMA_FIRST_EXTERNAL)
        {
Comment 27 Cesar Strauss 2010-09-25 03:07:42 UTC
(In reply to comment #25)
>   So I would like to see some proper detailed analysis on object files
> establishing exactly what constitutes all this bloat and where it comes from
> before I commit to what might be the wrong path of action.

Understood. I'll proceed by performing such a detailed analysis.

Also, if you come up with something you want to test, I'll be glad to try any patches you send my way.

Regards,
Cesar
Comment 28 Cesar Strauss 2010-09-26 01:11:57 UTC
(In reply to comment #25)
 
>   So I would like to see some proper detailed analysis on object files
> establishing exactly what constitutes all this bloat and where it comes from
> before I commit to what might be the wrong path of action.

Here is a comparison of the output of objdump -h for two compiler versions.
The source is src/common/any.cpp from wxWidgets 2.9.1.
The GCC compilers were downloaded from mingw.org.

GCC 4.5.0 (4.4.0)

Total object file size in bytes: 1,080,971 (110,037)

Total number of sections: 3,062 (318)
Number of:
LINK_ONCE_DISCARD .text$xxx sections: 2,887 (145)
LINK_ONCE_SAME_SIZE typeinfo .rdata$xxx sections: 128 (128)
LINK_ONCE_SAME_SIZE vtable .rdata$xxx sections: 36 (35)
.data$xxx sections: 3 (2)

Size of:
.text: 5,692 (5,308)
.data: 0 (0)
.bss: 72 (88)
.rdata: 2,432 (448)
.gcc_except_table: 5,232 (572)
.ctors: 4 (4)
.eh_frame: 41,996 (2,340)
.drectve: 126,440 (5,416)

Total size of:
LINK_ONCE_DISCARD .text$xxx sections: 132,344 (6,004)
LINK_ONCE_SAME_SIZE typeinfo .rdata$xxx sections: 2,552 (2,552)
LINK_ONCE_SAME_SIZE vtable .rdata$xxx sections: 1,184 (1,160)
.data$xxx sections: 76 (12)

Let me know if you need further info.

I hope this is helpful.

Thanks,
Cesar
Comment 29 Vadim Zeitlin 2010-09-26 22:09:16 UTC
Thanks Cesar for your analysis, I was doing the same thing but you beat me to it. Anyhow, I can confirm your results, i.e. that the increase in size is first and foremost due to the inflation of the total number of sections (which is multiplied roughly by 10) as well as the changes to the .eh_frame and .drectve sections sizes.

I compared the object files for 4.4, 4.5 and 4.5 when using auto-export, i.e. not using "__attribute__((dllexport))" on exported classes declarations. The typical results (the exact numbers are, of course, different for each object file but the trend is the same) are:

             4.4      4.5      4.5-autoexp
------------------------------------------
file size    151K     1.2M       87K
# sections    371     3145        85
.text size  16888    17264     24856
.eh_frame    4004    64424        16
.drectve     8620   113232         0

The difference in number of sections seems to correspond to the fact that 4.5 now generates one section per method of any exported class used by the object file instead of just one section per class as in 4.4.

Please let me know if I can provide any other information.
Comment 30 Cesar Strauss 2010-09-27 02:00:39 UTC
(In reply to comment #29)

Dear Vadim

> The difference in number of sections seems to correspond to the fact that 4.5
> now generates one section per method of any exported class used by the object
> file instead of just one section per class as in 4.4.

Sorry, but I do not completely agree with this assessment. If you run

objdump -h <object> | c++filt

you will see that 4.4 still generates one section per method, not per class (the name of the method is reveled by c++filt).

In my view, the difference is only that 4.5 emits a section for every inline method of every dllexported class in sight. 4.4 only emitted those which the code actually needed.

Why there is one section for each emitted inline method? As I understand it, since the same inline method can be exported by multiple object files, there would be a danger of having multiple definitions at link time. To avoid this, they reside in separate sections marked as LINK_ONCE_DISCARD, so the linker can discard these multiple copies and keep just the first one encountered. 

To avoid all these copies in the first place, the idea from Yu Simin in comment #26 seems to be a good one.

Regards,
Cesar
Comment 31 Vadim Zeitlin 2010-09-27 22:42:55 UTC
(In reply to comment #30)
> Sorry, but I do not completely agree with this assessment. If you run
> 
> objdump -h <object> | c++filt
> 
> you will see that 4.4 still generates one section per method, not per class
> (the name of the method is reveled by c++filt).

This is somewhat off topic but unfortunately (MinGW) c++filt doesn't work for me, e.g.:

% echo '.text$_ZNK17wxMBConvUTF16Base11GetMBNulLenEv'|/mingw/bin/c++filt.exe
.text$_ZNK17wxMBConvUTF16Base11GetMBNulLenEv

I don't know what could be wrong with it but, anyhow, you're right, of course, and I was wrong (I managed to look at the classes whose methods were not used by the object file I was checking...), sorry for misinformation and thanks for correcting it!

> In my view, the difference is only that 4.5 emits a section for every inline
> method of every dllexported class in sight. 4.4 only emitted those which the
> code actually needed.

Yes, exactly.
 
> Why there is one section for each emitted inline method?

Excellent question which I'm unfortunately totally unable to answer.
Comment 32 Cesar Strauss 2010-09-28 00:16:48 UTC
(In reply to comment #31)
> This is somewhat off topic but unfortunately (MinGW) c++filt doesn't work for
> me, e.g.:
> 
> % echo '.text$_ZNK17wxMBConvUTF16Base11GetMBNulLenEv'|/mingw/bin/c++filt.exe
> .text$_ZNK17wxMBConvUTF16Base11GetMBNulLenEv

Try echo '__ZNK17wxMBConvUTF16Base11GetMBNulLenEv'|/mingw/bin/c++filt.exe
Anyway, look at the end of each line of objdump -h <object> | c++filt, where it says 'COMDAT <name of method>'. 

> > Why there is one section for each emitted inline method?
> 
> Excellent question which I'm unfortunately totally unable to answer.

When two or more source files include the same header, the exported inline functions end up duplicated in both object files. When linking these objects together in the final DLL, the linker would give an error because of the multiple definitions. To avoid this, each exported inline function is placed in a separate section, identified by the COMDAT name. When the linker sees two or more sections with the same COMDAT name, it can safely keep one of them and discard the rest.
Comment 33 Yuchen Deng 2010-10-14 12:18:58 UTC
Because of this issue, I have been using GCC4.4.x, but do not want to upgrade to 4.5.x.
Why this issue can not been confirmed?
Comment 34 Andrew Pinski 2010-10-14 15:09:52 UTC
(In reply to comment #33)
> Because of this issue, I have been using GCC4.4.x, but do not want to upgrade
> to 4.5.x.
> Why this issue can not been confirmed?

The way I read it, this is not a bug as Microsoft Visual Studio acts the same way (when their precompiled headers are not included).
Comment 35 Vadim Zeitlin 2010-10-14 15:24:57 UTC
(In reply to comment #34)
> (In reply to comment #33)
> > Because of this issue, I have been using GCC4.4.x, but do not want to upgrade
> > to 4.5.x.
> > Why this issue can not been confirmed?
> 
> The way I read it, this is not a bug as Microsoft Visual Studio acts the same
> way (when their precompiled headers are not included).

This reply either demonstrates complete misunderstanding of the issue or is deliberate misinformation. To set the record straight, Microsoft Visual Studio absolutely doesn't act the same way and doesn't generate neither object files nor DLLs of this ridiculous size, whether precompiled headers are used or not. Besides, the issue does arise when precompiled headers _are_ used with gcc.

Continuing to justify what is clearly and uncontroversially a bad regression in gcc by "compatibility with MSVC" which never suffered from this problem and refusing to even acknowledge the issue, let alone revert the breakage, does no good whatsoever neither to the gcc project nor to its image.
Comment 36 Dave Korn 2010-10-14 15:37:34 UTC
Hi everyone, sorry I've been busy working on LTO stuff for a bit but I haven't forgotten this.

Before this discussion gets too heated, could Vadim and/or Cesar please add some of the object files we've been discussing as attachments to this bug report, so that we can all take a close look at why gcc's files are so much bigger?  Thanks.

I think depending what we uncover this might be an enhancement request rather than a bug, but we'll see.
Comment 37 Vadim Zeitlin 2010-10-14 15:42:59 UTC
Created attachment 22037 [details]
appbase.cpp file from wxWidgets compiled with g++ 4.4
Comment 38 Vadim Zeitlin 2010-10-14 15:44:23 UTC
Created attachment 22038 [details]
appbase.cpp file from wxWidgets compiled with g++ 4.5
Comment 39 Vadim Zeitlin 2010-10-14 15:45:41 UTC
Created attachment 22039 [details]
appbase.cpp file from wxWidgets compiled with g++ 4.4

Replaces previous file which was uncompressed, sorry for the double post.
Comment 40 Vadim Zeitlin 2010-10-14 15:47:36 UTC
(In reply to comment #36)
> could Vadim and/or Cesar please add
> some of the object files we've been discussing as attachments to this bug
> report, so that we can all take a close look at why gcc's files are so much
> bigger?

I attached 4.4 and 4.5 versions and can also add 3.4 one as well as MSVC9 build of the same file, please let me know if this could be useful.

Thanks!
Comment 41 Dave Korn 2010-10-14 15:50:53 UTC
(In reply to comment #40)
> (In reply to comment #36)
> > could Vadim and/or Cesar please add
> > some of the object files we've been discussing as attachments to this bug
> > report, so that we can all take a close look at why gcc's files are so much
> > bigger?
> 
> I attached 4.4 and 4.5 versions and can also add 3.4 one as well as MSVC9 build
> of the same file, please let me know if this could be useful.
> 
> Thanks!


  Yes please, I'm particularly interested in comparing the gcc files with the MSVC ones.  (I'm going to be AFK for a few hours now and will look through them this evening.)
Comment 42 Vadim Zeitlin 2010-10-14 16:01:20 UTC
Created attachment 22040 [details]
appbase.cpp file from wxWidgets compiled with g++ 3.4
Comment 43 Vadim Zeitlin 2010-10-14 16:01:55 UTC
Created attachment 22041 [details]
appbase.cpp file from wxWidgets compiled with MSVC 9 (a.k.a. 2008)
Comment 44 Vadim Zeitlin 2010-10-14 16:05:02 UTC
Created attachment 22042 [details]
appbase.cpp file from wxWidgets compiled with MSVC 9 (a.k.a. 2008)

Oops, sorry for another double post, I accidentally took a non-DLL version of the file the first time, this one is the right one.
Comment 45 Vadim Zeitlin 2010-10-14 16:12:00 UTC
Here are the files.

Notice that about half of the size of the MSVC object file is taken by debug information ("/Zi" option was used when compiling it) while all gcc versions contain no debug information at all. More details about the file contents can be obtained using dumpbin Microsoft utility which I'd be glad to run for you if you don't have it. Here is just an overview for now:

Microsoft (R) COFF/PE Dumper Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file build/msw/vc_mswudll/base/appbase.obj

File Type: COFF OBJECT

  Summary

           4 .CRT$XCU
          29 .bss
         3E9 .data
       345D0 .debug$S
          50 .debug$T
        68D2 .drectve
         E67 .rdata
         8BC .rdata$r
         178 .sxdata
        8A34 .text
        13D3 .text$x
          18 .text$yc
          14 .text$yd
        14B4 .xdata$x
Comment 46 Vadim Zeitlin 2010-10-14 17:09:05 UTC
Another data point after having a closer look at .drectve section in all of the files: as previously noticed, 4.4 generates "-export" directives for 180 symbols while 4.5 generates them for 2724 symbols. MSVC however records "/EXPORT" directives for 526 symbols when using PCH and 3764 symbols when not using PCH. In the latter case the size of the object file also inflates to 3.1MB (with all sections growing, not just the ".drectve" one). So while it's difficult to compare the sets of symbols because of the different name mangling schemes used, it does seem that MSVC generates these directives for all of the class members used, as 4.5 does and unlike 4.4. However many classes included in 4.5 don't appear at all in the MSVC file when using PCH resulting in much more reasonable object file sizes.

It still remains that the size of the DLL produced by MSVC is very small compared to gcc. E.g. the wxbase DLL is 1.9MB with MSVC and 6.5MB with gcc 4.4 while the entire wxcore DLL is just 3.9MB with MSVC and 11.6MB with gcc 4.4, without even mentioning 4.5 numbers. This is without using any "fancy" optimizations (i.e. "/LG" and/or "/LTCG" MSVC options). And the size of the DLL does _not_ depend on whether PCH was used or not (although the compilation speed and disk consumption of the object files definitely do, by a factor of more than 10 for both of them).

So maybe the new 4.5 behaviour is indeed more MSVC-like. But MSVC seems to avoid generating ~80% of the stuff gcc creates when using PCH and its linker still creates reasonably sized files in reasonable time even when the object files contain a lot of redundant sections. But using MSVC strategy for gcc currently is simply disastrous and I still don't understand why shouldn't the change of r147799 be reverted now and be reapplied later when PCH handling is improved and/or the linker is capable of dealing with it. If this is not a regression I honestly don't know what counts as one.
Comment 47 Andrew Pinski 2010-10-14 17:13:01 UTC
One should note that GCC's implementation of PCH is way different from MSVC's.  So comparing with PCH is not the correct thing to do really.  PCH in GCC is really preprocessed/sematicized headers rather than header that is an object file  like in MSVC's case.
Comment 48 Vadim Zeitlin 2010-10-14 17:29:46 UTC
(In reply to comment #47)
> One should note that GCC's implementation of PCH is way different from MSVC's. 
> So comparing with PCH is not the correct thing to do really.

I understand this in theory but in practice nobody uses MSVC without PCH (and I guess by now most people use PCH with gcc as well but this doesn't change anything here, of course, as after this change gcc is unusable, with or without PCH). Anyhow, notice that MSVC still manages to produce correctly-sized DLLs even without using PCH.

And in any case while comparing with MSVC may be interesting, surely the most urgent goal is to fix gcc to actually work and not to make it more similar to MSVC at the price of breaking it completely? I just can't wrap my head around your arguments, everything you say may be correct (except comment 6) but how do you deduce from it that the current situation is fine and don't even want to acknowledge this change for a bug and regression that it is is beyond me.
Comment 49 Dave Korn 2011-01-09 17:30:31 UTC
Created attachment 22935 [details]
trial patch

brings the earlier change that nathan made to always keep dllexported inlines under control of a command-line flag.  Before:

----------------------------------snip----------------------------------
make[1]: Leaving directory `/tmp/wx/obj-x-ming-clean/utils/wxrc'
Creating library file: /tmp/wx/obj-x-ming-clean/lib/libwx_mswd_core-2.8.dll.a/op
t/mingw-pr43601-clean/lib/gcc/i686-pc-mingw32/4.6.0/../../../../i686-pc-mingw32/
bin/ld: final link failed: Memory exhausted
collect2: ld returned 1 exit status

make: *** [/tmp/wx/obj-x-ming-clean/lib/wxmsw28d_core_gcc_custom.dll] Error 1

real    79m46.081s
user    50m1.166s
sys     5m45.723s

$ du -cxsh lib/*.dll
59M     lib/wxbase28d_gcc_custom.dll
1.4M    lib/wxbase28d_net_gcc_custom.dll
512K    lib/wxbase28d_xml_gcc_custom.dll
61M     total
----------------------------------snip----------------------------------

... and after ... 

----------------------------------snip----------------------------------
Creating library file: /tmp/wx/obj-x-ming-new/lib/libwx_mswd_aui-2.8.dll.a
Creating library file: /tmp/wx/obj-x-ming-new/lib/libwx_mswd_richtext-2.8.dll.a
Creating library file: /tmp/wx/obj-x-ming-new/lib/libwx_mswd_xrc-2.8.dll.a

real	39m32.531s
user	21m52.641s
sys	4m53.111s


$ du -cxsh lib/*.dll
33M     lib/wxbase28d_gcc_custom.dll
1.3M    lib/wxbase28d_net_gcc_custom.dll
512K    lib/wxbase28d_xml_gcc_custom.dll
5.2M    lib/wxmsw28d_adv_gcc_custom.dll
2.2M    lib/wxmsw28d_aui_gcc_custom.dll
117M    lib/wxmsw28d_core_gcc_custom.dll
4.7M    lib/wxmsw28d_html_gcc_custom.dll
576K    lib/wxmsw28d_qa_gcc_custom.dll
4.2M    lib/wxmsw28d_richtext_gcc_custom.dll
8.2M    lib/wxmsw28d_xrc_gcc_custom.dll
176M    total
----------------------------------snip----------------------------------

Needs testcase + doco before I can submit it.
Comment 50 Yuchen Deng 2011-01-09 23:48:09 UTC
Good news!
Comment 51 Dave Korn 2011-01-21 04:17:24 UTC
Patch submitted.
Comment 52 Dave Korn 2011-01-26 01:50:14 UTC
Committed rev.169268.

http://gcc.gnu.org/viewcvs?view=revision&revision=169268

(Then added the missing PR refs to the changelogss in the next rev:
http://gcc.gnu.org/viewcvs?view=revision&revision=169269)
Comment 53 xunxun 2011-03-16 04:42:20 UTC
If I don't use LTO Optimization, Vadim Zeitlin's patch works well.
But if I use LTO Optimization, the compiling speed becomes vey slow, and the linker stage fails. I will get the information: 

lto1.exe: out of memory allocating 1900552 bytes
lto-wrapper: g++ returned 1 exit status
ld.exe: lto-wrapper failed
collect2: ld returned 1 exit status

mingw32-make: *** [wxmsw28u_gcc.dll] Error 1
Comment 54 xunxun 2011-03-16 06:53:47 UTC
(In reply to comment #53)
> If I don't use LTO Optimization, Vadim Zeitlin's patch works well.
> But if I use LTO Optimization, the compiling speed becomes vey slow, and the
> linker stage fails. I will get the information: 
> 
> lto1.exe: out of memory allocating 1900552 bytes
> lto-wrapper: g++ returned 1 exit status
> ld.exe: lto-wrapper failed
> collect2: ld returned 1 exit status
> 
> mingw32-make: *** [wxmsw28u_gcc.dll] Error 1

BTW, I use gcc 4.6.
Comment 55 Dongsheng Song 2011-04-06 06:35:11 UTC
(In reply to comment #54)
> (In reply to comment #53)
> > If I don't use LTO Optimization, Vadim Zeitlin's patch works well.
> > But if I use LTO Optimization, the compiling speed becomes vey slow, and the
> > linker stage fails. I will get the information: 
> > 
> > lto1.exe: out of memory allocating 1900552 bytes
> > lto-wrapper: g++ returned 1 exit status
> > ld.exe: lto-wrapper failed
> > collect2: ld returned 1 exit status
> > 
> > mingw32-make: *** [wxmsw28u_gcc.dll] Error 1
> 
> BTW, I use gcc 4.6.

me too.

If I use 32bit compiler on windows/Linux, target 32bit windows,
single huge monolithic library build with or without LTO will out of memory, 
multiple libraries build failed with LTO, OK for without LTO.
Comment 56 Dave Korn 2011-04-07 15:15:14 UTC
What works for me on Cygwin, and so may well also work for anyone using MSYS, is setting the heap_chunk_in_mb registry parameter to some value in the range 1024 - 1536.  I use 1024 myself and that gives me sufficient headroom to link libgcj dll, which is huge; if it works for that, it's likely to help with wx dll as well.

http://cygwin.com/cygwin-ug-net/setup-maxmem.html
Comment 57 Dongsheng Song 2011-04-07 15:53:38 UTC
(In reply to comment #56)
> What works for me on Cygwin, and so may well also work for anyone using MSYS,
> is setting the heap_chunk_in_mb registry parameter to some value in the range
> 1024 - 1536.  I use 1024 myself and that gives me sufficient headroom to link
> libgcj dll, which is huge; if it works for that, it's likely to help with wx
> dll as well.
> 
> http://cygwin.com/cygwin-ug-net/setup-maxmem.html

I don't think so. Because I observed ld.exe use memory over 1.7GB, so link wx monolithic library require use more memory than 32 bit OS limit. For cross compile under Linux, link wx can use near 3G memory, it still failed.

Then link wx require 4G or more memory, maybe someone can try use 64bit linker to build single huge monolithic library, tell us the max memory ld used.
Comment 58 Dongsheng Song 2011-04-10 04:32:23 UTC
(In reply to comment #57)
> (In reply to comment #56)
> > What works for me on Cygwin, and so may well also work for anyone using MSYS,
> > is setting the heap_chunk_in_mb registry parameter to some value in the range
> > 1024 - 1536.  I use 1024 myself and that gives me sufficient headroom to link
> > libgcj dll, which is huge; if it works for that, it's likely to help with wx
> > dll as well.
> > 
> > http://cygwin.com/cygwin-ug-net/setup-maxmem.html
> 
> I don't think so. Because I observed ld.exe use memory over 1.7GB, so link wx
> monolithic library require use more memory than 32 bit OS limit. For cross
> compile under Linux, link wx can use near 3G memory, it still failed.
> 
> Then link wx require 4G or more memory, maybe someone can try use 64bit linker
> to build single huge monolithic library, tell us the max memory ld used.

I test on 64bit Linux VM, after ld running 6 CPU hours, use 4728MB memory, still not finished, so I think there maybe have potential infinite loop and memory leaks when link many object files.

I don't know this is GCC bug, or ld bug, or both ?

http://sourceware.org/bugzilla/show_bug.cgi?id=12658

ld running 6 hours cpu time, use 4.6GB memory:

VmPeak:  4776608 kB
VmSize:  4734656 kB
VmLck:         0 kB
VmHWM:   3020260 kB
VmRSS:   2982304 kB
VmData:  4728000 kB
VmStk:       100 kB
VmExe:       912 kB
VmLib:      1504 kB
VmPTE:      9232 kB
Threads:        1

Cpu(s):  7.9%us, 11.5%sy,  0.0%ni, 44.4%id, 33.3%wa,  0.2%hi,  2.7%si,  0.0%st
Mem:   3095296k total,  3073516k used,    21780k free,       16k buffers
Swap:  7811064k total,  3905532k used,  3905532k free,     9484k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1110 dongshen  20   0 4623m 2.8g  196 D   49 96.1 358:15.13 ld
Comment 59 xunxun 2011-04-17 11:06:13 UTC
Yea, I review the patch(In reply to comment #57)
> (In reply to comment #56)
> > What works for me on Cygwin, and so may well also work for anyone using MSYS,
> > is setting the heap_chunk_in_mb registry parameter to some value in the range
> > 1024 - 1536.  I use 1024 myself and that gives me sufficient headroom to link
> > libgcj dll, which is huge; if it works for that, it's likely to help with wx
> > dll as well.
> > 
> > http://cygwin.com/cygwin-ug-net/setup-maxmem.html
> 
> I don't think so. Because I observed ld.exe use memory over 1.7GB, so link wx
> monolithic library require use more memory than 32 bit OS limit. For cross
> compile under Linux, link wx can use near 3G memory, it still failed.
> 
> Then link wx require 4G or more memory, maybe someone can try use 64bit linker
> to build single huge monolithic library, tell us the max memory ld used.

I review the patch, and found that we can add "-fno-keep-inline-dllexport" to the compiler option, and then, the compiler and linker stage works well. But the wxWidgets release mono dll's size is so large.(about 17M)
Comment 60 Dongsheng Song 2011-04-18 03:48:19 UTC
With Kai's great work on binutils, after ld running 172 minutes
(usr + sys), and the memory usage growing to:

VmPeak:  5995156 kB
VmSize:  5995156 kB
VmHWM:   1900732 kB
VmRSS:   1219200 kB
VmData:  5986232 kB

The link finished:
-rwxr-xr-x 1 dongsheng dba 10499584 Apr 18 11:27 wxmsw28u_gcc.dll

I think the memory usage (6GB) and CPU time consume (172 minutes)
have a great room for improvement.
Comment 61 Dave Korn 2011-04-21 00:40:17 UTC
(In reply to comment #59)

> I review the patch, and found that we can add "-fno-keep-inline-dllexport" to
> the compiler option, and then, the compiler and linker stage works well. But
> the wxWidgets release mono dll's size is so large.(about 17M)

In newer versions of GCC there is also a lot more debug info and Dwarf-2 exception table data that didn't used to be there.  Stripping the dll and/or building a compiler with SJLJ rather than dwarf exceptions might help, although SJLJ trades off executable size for slower runtime.  (Badly; dwarf exceptions only take CPU time when one is actually thrown, whereas SJLJ exceptions take CPU time every time you enter or exit a try block.  Since exceptions are meant to be exceptional conditions that only happen occasionally, this tradeoff is probably worthwhile on desktop platforms where memory is not in short supply, but SJLJ might still be better on embedded platforms where memory is so critical that slower runtime performance might be the preferable option.)
Comment 62 xunxun 2011-04-21 01:28:02 UTC
(In reply to comment #61)
> (In reply to comment #59)
> 
> > I review the patch, and found that we can add "-fno-keep-inline-dllexport" to
> > the compiler option, and then, the compiler and linker stage works well. But
> > the wxWidgets release mono dll's size is so large.(about 17M)
> 
> In newer versions of GCC there is also a lot more debug info and Dwarf-2
> exception table data that didn't used to be there.  Stripping the dll and/or
> building a compiler with SJLJ rather than dwarf exceptions might help, although
> SJLJ trades off executable size for slower runtime.  (Badly; dwarf exceptions
> only take CPU time when one is actually thrown, whereas SJLJ exceptions take
> CPU time every time you enter or exit a try block.  Since exceptions are meant
> to be exceptional conditions that only happen occasionally, this tradeoff is
> probably worthwhile on desktop platforms where memory is not in short supply,
> but SJLJ might still be better on embedded platforms where memory is so
> critical that slower runtime performance might be the preferable option.)

I know that, but I have stripped the dll...
With dw2 gcc, mingw gcc4.5.2 can produce the 6M wx release dll (after strip).
Comment 63 Vadim Zeitlin 2011-04-21 14:04:37 UTC
(In reply to comment #61)
> (In reply to comment #59)
> 
> > I review the patch, and found that we can add "-fno-keep-inline-dllexport" to
> > the compiler option, and then, the compiler and linker stage works well. But
> > the wxWidgets release mono dll's size is so large.(about 17M)
> 
> In newer versions of GCC there is also a lot more debug info and Dwarf-2
> exception table data that didn't used to be there.

FWIW wxWidgets hardly uses exceptions (it has only two try/catch blocks that basically rethrow an exception thrown by user code but the library doesn't throw any exceptions itself). It does/can be configured to use the standard containers which do use exceptions however.
Comment 64 xunxun 2011-06-08 14:26:05 UTC
I found that the shared or static gcc edition made a great difference of the wx unicode release mono dll size.
-----------------------------
At first, I build the gcc 4.6 with "--enable-static --enable-shared", and then I use the gcc edition to compile wx2.8.12 unicode release mono dll, that is, 

mingw32-make -f makefile.gcc CXXFLAGS="-Os -fno-keep-inline-dllexport" LDFLAGS="-static -Wl,-s"

this gcc edition can produce the wx dll size about 6-7M, and the dll depends no any gcc dll.
----------------------------
Another day, I build the gcc4.6 with "--enable-static --disable-shared", and then do the similar thing, that is,

mingw32-make -f makefile.gcc CXXFLAGS="-Os -fno-keep-inline-dllexport" LDFLAGS="-Wl,-s"

this gcc edition can produce the wx dll size very large, about 17M, and the dll depends no any gcc dll, too.
----------------------------

That's why?

Thanks.