This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: new plugin events
Richard Guenther wrote:
On Fri, Nov 6, 2009 at 10:41 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
Richard Guenther wrote:
(stop reading now, what follows is pure cynicism - I have to do it, because
the following can't be let un-responded, rather than un-answered)
Actually, I tend to believe that the plugin infrastructure should ideally
permit to replace almost every single file in GCC by an equivalent file in
the plugin. Take for example the pass manager (file gcc/passes.c) I
understood that ICI have their own pass manager (By the way, I Basile don't
care about the pass manager). So ICI guys would probably be happy to bypass
all the functions in the gcc-4.5/gcc/passes.c and provide their own variant
of these. My intuition is that this should be doable.
I meant that some developers (not me) might dream of replace one single file X.c with their XX.c.
For ICI, I understood that X.c is gcc/passes.c (and I don't know what is their XX.c replacing it)
I don't understand the cynicism adjective (which I feel too aggressive).
In other words, I believe that the plugin functionality should permit experimentation on most features of GCC. Of
course I do not mean all of them at once, I just mean making an experiment where one single feature or functionality X
[to simplify, an important GCC function or source file) of GCC is replaced by something provided by the plugin.
It should be possible already if you build GCC with -fPIC. You then
can LD_PRELOAD a plugin replacing all functions you want to replace.
Thanks to the wonders of ELF.
I probably am happy enough with the current set of hooks. I might wish in a
year some hooks for CPP, but I understood Diego need them, so I believe they
will appear.
But I am not sure that every GCC experiment is pluginable today (I really
don't know). Take for instance all the papers in the recent GCC summits
(let's say after 2007). All of them describe code related to GCC. I am not
sure that every such code could fit as a plugin. I believe it should.
Ah, you mean like doing the tuples conversion as plugin? Or to
build the cgraph infrastructure and IPA optimization infrastructure
as plugin? I guess what you say is - "stop developing gcc! develop plugins!"?
To be more precise, I believe that some few plugins (not the ones I Basile will write) might perhaps replace the branch
role as experimental ground for GCC. And I don't see any issue here, on the contrary. Plugins are for their users
perhaps easier to try than a whole GCC branch. What's the difference between writing a plugin and writing a branch
nobody will use? So perhaps people would start plugins instead of starting their branch. Why should that be bad? I feel
it is very positive!
Perhaps all of GCC code is bypassable by a plugin which would handle
PLUGIN_START_UNIT, copy or call all of the existing code of GCC inside it,
and exit without returning... So probably, plugin could do anything.
What's the point? We have plugins for the kernel for this - it's called
programs, the plugin architecture is processes.
No plugins for the kernel are called modules. The command to list the set of loaded kernel modules is lsmod (not ps).
The debian kernel I am using now have 93 modules loaded.
But I believe we should provide a lot more hooks for them.
Then add a GCC option you build GCC with, -finstrument-functions.
Wait - that already exists! Or do you maybe mean a more structured
way to expose GCC internals?
Plugins are important for a very social, non-technical reason: nobody today
is compiling GCC -except the GCC community & the linux distribution makers.
The art of compiling monster software like GCC is almost lost. People don't
compile free software anymore. (this has changed since the 1980s. At the
time of SunOS3.5, people did compile gcc & emacs). And I recently discovered
that compiling GCC is too difficult a task for most GCC users (at least
outside academia). GCC plugin will help experimental extensions towards this
population of GCC users (not able or not wanting to compile GCC).
Another important motivation for plugins is the licence (more exactly the
legal requirement of FSF ownership). Coding GPLv3 code is accepted by many
managers & organizations (because the GPLv3 is a mainstream licence), but
transfering copyright to FSF is painful to most big organizations - because
it is an unusual procedure -.
I insist that the copyright to FSF transfer is a big brake to enter GCC development. And this brake is not effective
inside plugins. Plugins can be simply GPLv3, but not FSF copyrighted. I am sure it makes an important difference for
putative developpers (more precisely for their bosses).
Bullshit. Replacing "monster software like GCC" with "moster plugins
like melt" isn't going to help. It will just fragment the developer
communities.
You might tell everything about MELT, but not that is is monstrous. It is a small one-man work. To get the figures
right, MELT code (the one written by a human, me [I am excluding the generated code]) is about 38KLOC and the GCC trunk
is more than 3900KLOC - this is more than a hundred times more. [measured by sloccount, by copying all MELT specific
files somewhere and renaming *.melt as *.lisp]. Of course MELT generated C code is big. That is true of most code
generators!
And the GCC community is large enough to be fragmented today. People working on C++ or Ada front-ends are not very close
of people working on AMD processor back-ends, or on register allocators. A lose community of hundreds of developers is
necessarily fragmented. I would even wish it to be more fragmented: I would prefer a set of several gcc-middle-end@
gcc-front-ends@ gcc-back-ends@ mailing lists instead of the single gcc@ one.
This is why defining a large enough set of hooks is important *today*
[especially for hooks or events easy to implement today]. I am pretty sure
that gcc-4.5 will still be a lot used in 2013 or 2017, when the GCC
I am pretty sure it will not. I am pretty sure no single plugin developer
will keep its plugin compilable for gcc 4.5 in 2013.
We don't know. Maybe you are right. But I would believe that a plugin written in 2010 for 4.5.0 could be used in 2013
for the same 4.5.0, or perhaps if it was written for 4.5.0 could be recompiled for a gcc-4.5.4
community will be working on GCC-5.5 or even 6.5 (I know about some small
SMEs in the embedded software economy which are using today a 2.95 version
of GCC, and not really able to compile a 4.3 version of GCC, even if
produces better code for their embedded ARM processor). So the set of hooks
today is important for many years!
They can patch in hooks as they like. Their plugins will only work
against a patched GCC anyway.
I don't understand that. My claim was the opposite. If we add the several hooks needed for ICI or another plugins, these
plugins should work for an unpatched GCC.
Notice also another thing. there have been a lot of plugin experimentation
(at least in GCC summit papers). And every experiment was about code which
the developer thought would never be accepted into the trunk (and that
intuition is right: nothing of all the plugin experiments landed into the
trunk [for example, TreeHydra did not land inside GCC, etc.], except the
idea of plugins and the small plugin API).
Ah, I responded to that already.
This means that plugins would probably appear mostly for things that have no
chance to land into GCC core. Plugin will more proably do everything but the
kitchen sink. I would expect AMD & Intel to continue contributing to the GCC
core to target their newest processors - I cannot imagine why AMD or Intel
would target their 2011 processor with only a plugin.
[there is a simple reason for that: plugin are a bit more expensive to use
that straight GCC - you need extra options & extra installations-, and human
nature is lazy; the interest of constructors is to sell chips, not GCC
plugins!]. Toplevel processor makers will continue to work inside GCC.
So you believe the only changes in GCC that will happen is newer
processor support? That's sad.
No I did not tell that. I have no idea of where GCC will go in the future (I am not even sure GCC will remain dominant
in 2015 - nobody knows!). I just wrote that the interest of AMD & Intel is to get code supporting their new processors
inside the GCC trunk. They have no reason to put that code in a plugin, and they have strong and obvious reasons to put
that code inside the GCC trunk. And there are large enough corporations to afford a few extra developers to push code
into the trunk (instead of leaving it as a plugin).
What could happen if some plugin events are not added quickly? People
interested in them will simply lose interest in GCC (they could for instance
go towards LLVM if they feel it is easier for them). People waited much too
long for plugins to appear... they won't wait till 4.6 for an event
essential for them which is basically a dozen-line patch; they will just
stop looking at (or inside) GCC
Good. People not wanting or being able to interact with the GCC
developer community neither do have a chance to successfully
develop a plugin nor are they of interest to GCC.
If the GCC community cared about experimental languages front-end (stuff
like Modula3, or Mercury, ...), we should also have hooks for other
front-ends [I personally have no real need for that]. As a concrete example,
GNU CLISP could use GCC instead of GNU lightning (to translate its custom
bytecode into binary). No, I am not volunteering -and when needing a Common
Lisp, I'll use SBCL.
The frontend interface is very well separated nowadays. Pluginifizing
it should be possible - but please - this needs at least _some_ work
inside GCC. Plugins never should be an excuse to not improve or
fix GCC, instead they should offer an opportunity to do so.
I begin to understand a bit more Richard's position. Let me try to rephrase it.
"There are some current weaknesses in today's GCC architecture, and and it is more important to fix them first than to
permit plugins - in particular plugins experimenting some workaround or improvements of these weaknesses."
I see these two issues as completely independent. There are indeed weaknesses in the GCC architecture. I perceive that
as a normal state for an old -more than twenty years old- software like GCC (and given its age, GCC is in good shape).
In addition, it is related to the way the GCC community works - lack of a benevolent dictator (like my understanding of
the role of Linus Torvalds in the Linux kernel - but I admit I don't understand the details of Linus' work today). Also,
I am not sure that any corporation has strong enough interest in improving the GCC architecture to pay a dozen of world
class developers for that sole purpose of improving the GCC architecture. At least, I never saw that...
(the only possible exception is perhaps Google, but I am not naive enough to believe that Google would tell really why
they invest in GCC [we can only guess that]. But I am still perceiving that even Google has not enough interest in
improving the GCC architecture - it is only a good side effect of their work; Google does not seem to pay a dozen of
talented individuals with the improvement of GCC architecture as their *only* goal!).
So I agree that GCC architecture has shortcomings. I am not quite sure that we (not only Richard Guenther & me Basile,
but the whole community) agree on the exact list of GCC architectural weaknesses (as usual, the evil is in the details).
And correcting GCC weaknesses is always much more complex than expected (look at the various efforts on register
allocators - I know nothing of them, except that there have been many register allocator tentatives in the past, and
that these efforts have been painful).
But why should the GCC architecture weakness prohibit experimental plugins? Why should we refuse the few events needed
to ICI or others? Some ICI people might perhaps think that their alternative pass manager is better than the current
one.. [I have no idea if this is true or not, and most importantly, there are a lot of different but valid points of
view.]. Why could we let them experiment inside a plugin if that required only a few more PLUGIN_* events? Very
probably, nobody looked precisely inside their branch!
So in contrast to Richard Guenther, I don't percieve any danger today in
adding several new plugin events (even if they later happen to be nearly
useless). Precisely because the plugin API is newborn.
We don't add GCC internal APIs like that, so why should plugin APIs
any different here? On the contrary - the hurdle to add external
interfaces should be much higher than adding internal ones.
Well, the internal headers of GCC [files gcc/*.h] are still significantly evolving [more than 20 *.h files appear in
gcc/ChangeLog just for the current week of November 2009] I perceive them as the GCC internal API and it did evolve this
week.
So adding a PLUGIN_ event in 4.5 and removing another one in 4.7 is not so important. I guess that a few additional
PLUGIN_* events are enough to permit experimenting their pass manager instead of the current one. Sorry, I cannot list
these additional hooks (because I don't know the details of ICI, and I don't master entirely the current pass manager. I
was dreaming of hooks inside execute_ipa_pass_list of passes.c or elsewhere, perhaps toplevel.c). I leave the details
and effort of submitting such patches to other people (including ICI people).
The point is that push a few PLUGIN_* events into the trunk is a few small patches (a hundred lines at most), while
pushing a whole new pass manager is a lot of big patches (several thousand lines), and pushing such big patches is
extremely difficult and take years, and probably can only be done by old GCC top level contributors (like Diego Novillo
or Richard Ghenther, not like me or ICI people).
I do agree that there are shortcomings in the GCC architecture. But I won't even dare proposing a patch on them [because
I am 100% sure it won't even be reviewed, so that I am sure to lose my time.] This is a task for respected GCC
developers (like Diego & Richard), not for a single Write-After-Approval random hacker like me.
My hypothesis is that trying a plugin is cheaper (from the GCC user's point of view) than trying a whole GCC branch.
Compiling a whole branch is so hard in practice that nobody cares about branches (because almost nobody takes the time
to compile GCC). As a case in point, I don't know of any GCC branch packaged in Debian (even Debian/Unstable!) or
Mandriva. And I believe perhaps naively that in 2011 some Debian or Mandriva package will provide some plugin to gcc-4.5
(or gcc-4.6?) there.
I definitely agree with Richard that the current pass management is not entirely satisfactory (however, the pass
organizations of GCC is a strength of GCC!). But providing a few hooks inside it is much easier than replace the pass
manager entirely. Adding a few hooks inside is quite simple, replacing it entirely by something better is very
difficult, and very probably impossible for most GCC contributors. Even if someone -outside of the core GCC team, I mean
outside of the set of reviewers or eminent GCC people like Richard Guenther, Diego Novillo and a small dozen of others-
proposed a big patch about pass management, nobody would review or even read it. And the point is that some
architectural changes in GCC cannot be split in small patches!
I believe that GCC architectural deficiencies will stay for a long time, because the set of people able to improve them
is small (it is a select club), and because most of these people are not paid for improving GCC architecture as a whole,
but for pushing some subset of features inside GCC. I can imagine that LTO & WHOPR are more important to Diego (and to
his boss) than improving GCC architecture. And the set of GCC reviewers is also small.
So improving GCC architecture will continue to happen in small steps, each of them being quite big for the elite people
involved. For instance, in the past, I can imagine that the tree -> gimple transition took several person*years of
effort (on a small set of people).
We should not be afraid of new plugin events. We could be afraid of people
wanting to start coding plugins going elsewhere (to some other compiler
technology).
I'm not afraid of the second.
I am not afraid of ICI or TreeHydra guys adding useless plugin events. I am
afraid of ICI or TreeHydra guys losing interest in GCC (and perhaps going to
other compilers). If that happens, it will be a loss for the GCC community.
I do not see them as part of the GCC community (sorry, no offense intended,
exaggerating again here).
We strongly disagree on that too. I really feel that ICI or TreeHydra people posting on gcc@ or gcc-patches@ lists are
members of the GCC community as [un-]important as me.
I also understand that the GCC community is like Orwell's animal farm (this is very probably true of other big free
software communities). Everyone is equal but some are more equal than others. That fact is normal in every human
society. It is really normal that people care more about what Diego Novillo or Richard Guenther propose than about
random other patches -including those I Basile submit.
Also, I don't view any precise PLUGIN_* event as a long term commitment to stay. Only time will tell which events are
important, and which events are unneeded. This is exactly the same with passes inside GCC. Some of them are important,
other are less used (we cannot tell in november 2009 what is the essential set of PLUGIN_* events. For example, I don't
understand precisely the role of PLUGIN_START_UNIT, but I do believe someone else is needing that and that does not
bothers me. In contrast, the PLUGIN_GGC_MARKING event is absolutely essential to MELT, but I don't know of any other
plugin needing it, and I believe almost no other plugin will need it, even if it can use it. Any I am sure that
PLUGIN_GGC_MARKING or PLUGIN_START_UNIT don't bother much GCC.). So some PLUGIN_* events could perhaps disappear later.
What is the problem? What is the difference between dropping a particular builtin or pragma or target architecture?
I really believe it is acceptable to admit (before 4.5 release) a half-dozen or a dozen more events in enum plugin_event
[but I Basile don't need any now]. This won't break any code today (because plugins are newborns). This will help the
people proposing them for the plugins they *want* to develop - I don't claim that people develop plugins for their
private branch of GCC. Why should we reject a priori any such additions?
And it is not related to the deficiencies of the GCC architecture. On the contrary, perhaps additional plugin hooks
could permit experimentation - on a larger scale than GCC branches nobody cares about - of new features...
I really think that some plugins will appear and be widely distributed (in particular packaged in Debian or Redhat or
SuSE or Mandriva or other Linux distributions.). This is not the case for GCC experimental branches. From the random GCC
user point of view, branches are totally inexistent, while plugins are merely superfluous (like is -O3 today, or even
more the optimisations not even triggered by -O3; almost nobody uses them; the average GCC user is using -g when
developping and -O2 at most when deploying; he is not using -O3 and won't use plugins; both -O3 and GCC plugins have few
users..).
Is there any experimental branch of GCC packaged in any Linux distribution? Of course, I am not thinking of vendor GCC
branches (like suse ubuntu redhat...).
Suppose that the Mozilla people make the effort to make TreeHydra a real gcc-4.5 plugin when GCC 4.5 is released. Then I
would imagine that this TreeHydra plugin could be packaged in Debian, much like there is a xulrunner-dev package in
Debian today (I have no idea if the treeehydra.so GCC-4.5 plugin binary would go in xulrunner-dev or in
gcc-4.5-treehydra Debian package; this is the responsability of Debian). But I just checked with "apt-cache search
hydra" - there is no treehydra-gcc package in Debian/Unstable!
So I still hope ICI people and others will submit short patches for some few plugin events (for GCC 4.5). This is more
likely to be accepted than a big thousand-line patch (like e.g. a patch pushing the ICI pass manager as a whole; nobody
amongst the GCC reviewers cares...).
And I do agree with Richard that there are some GCC architectural deficiencies. In an ideal world, I would like to have
them improved [I don't have the authority, wisdom, or time to improve them myself; I am sadly sure that even if I or
some other random GCC hacker worked a whole year on them, nobody would care : any big patch won't even be reviewed,
unless it has been proposed by an important person in the GCC community..; and architectural improvements are disruptive
and require big patches; they cannot be proposed as a sequence of small patches.]. In the real world, I prefer that a
few more plugin hooks would appear to permit experimenting new ways of improving these deficiencies inside plugins. I
understood that Richard don't want that. [I guess Richard want such improvements to go straight into GCC code; I Basile
think that won't happen, unless it is originated by the elite GCC developers.].
Personally, I Basile is now happy enough with the current set of plugin events, assuming that Diego will push a patch
for extending the preprocessor (that is for offering from a plugin new preprocessor tricks like the current __COUNTER__
which is a GCC specific extension, not an ISO C standard if I understood it well). And probably, even without
preprocessor plugins (pushed by Diego), I Basile can work for two years inside GCC (that is inside MELT).
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***