This is the mail archive of the gcc@gcc.gnu.org mailing list for the GCC project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Defining a common plugin machinery

Hi Brendon,

Thanks for reading all this :-) Comments in line.

Brendon Costa wrote: 
I have notes inline below, following is my summary of libplugin from
what i understand of your posts:
* It exists as a fraemwork that works with GCC now
* It uses xml files to define plugins (Allows making new plugins as
combinations of others without making a new shared library, i.e. just
create an xml file that describes the plugin)
* It handles issues with inter-dependencies between plugins
* It uses a "push" framework, where function pointers are
replaced/chained in the original application rather than explicit calls
to plugins (Provides more extensibility in a application that makes
heavy use of function pointers, but produces a less explicit set of
entry points or hooks for plugins)
* Currently it provides automatic loading of plugins without specific
user request
* It already has a framework for allowing plugins to interact with the
pass manager

If you can think of any other points to summarize the features it might
be helpful as you are closer to it.

The issues i see with this framework:
* it seems to provide a lot of features that we may not necessarily
need (That should be up for discussion)
Yes, it's entirely possible that it has more features than most people will need. Some have been taken from Eclipse, since they've some years fleshing out the problems with a major plugin system, some come from AOP. If we only ever see there being <10 entry points (i.e. not very fine grained) and not having cooperating plugins, then libplugin is definitely overkill.
* plugin entry points are not well defined but can be "any function
pointer call"
Achh, hoist by my own petard! Part of my design goals was to make the changes almost invisible in GCC. I thought it might make it more acceptable to people. The difference between plugin-GCC and normal GCC would be 10 lines in toplev.c.

Two solutions come to mind. One would be to provide an empty define for documentation purposes. Something like #define EXTENSION? The other is to say, well, entry points are pretty well defined in the plugin XMLs. Again, this is the approach taken by Eclipse and seems to work well.
Actually, I have another which I describe later when I talk about enforcing application plugins.

It is also possible to have entry points that aren't events, join-points or lists, those are just the ones that the system makes easy for you.
Some questions:
* How does the framework interact with the compile command line arguments?
I wrote a bit about that in a previous email http://gcc.gnu.org/ml/gcc/2008-10/msg00011.html.
I didn't put all the details in. For example, at the moment, the plugin system gets to see the command line before any other part of GCC. This allows plugins to alter the command line themselves.
Secondly, though I was thinking of making this optional, the plugin system removes all plugin command line options from the command line before GCC gets to see it.
* Does this work on platforms that dont support -rdynamic or can it be
modified to do so in the future?
It uses ltdl. I think that can be made to statically load dlls on those systems. You'd then have all the capabilities your base libraries provide + anything you can describe in XML only. I have no idea how practical this would be.

Personally, I think plugins on those systems might be more trouble than it's worth. I've never used a compiler on one though (only compiled for them on my PC), so I don't really know much about it.

I've only been developing on Linux. I don't know how much work it would be to port to other machines.

Oh, I also use libffi for events and join-points. The system would certainly work without events and join-points, it just wouldn't be as fun. If you leave it in then it limits libplugin to systems with libffi.




Hugh Leather wrote:
*Separating Plugin system from appliction*
Libplugin ships as a library. Apart from a few lines of code in
toplev.c, the only other changes to GCC will be refactorings and
maybe calling a few functions through pointers.
As i understand the difference between the pull vs push, a plugin will
load, and then modify existing function pointers in GCC to insert its
own code and chain the existing code to be called after it. Is this correct?
Yes, although really the plugin system does it on the plugin's behalf.

We also have a 'non-enforced' distinction between plugins. There are those which use symbols in the application, making application events and join-points and those which provide their own shared libraries (some do both).
So, some plugins make bits of the application extensible, some provide additional services, or both.

It would be relatively easy to enforce that distinction. We could insist that no plugin is able to make bits of the application extensible unless they come with GCC. A bit like sealed packages in Java, people could break it if you really wanted to but they'd be unlikely to persuade anyone to use their cheat.

It would only take another 10 or so lines of code in libplugin to do this. At the moment, when people want to describe symbols in the XML, they say which libraries the symbols could come from (with optional names):
<library path="somewhere"/>
<library path=""/>
The latter one is special and means the symbols come from the app. It would be easy to check in that case that the plugin's XML file is in GCCs install dir, failing if not.
Doing this will be able to make use of existing function pointers as
plugin hook locations, but some hooks we may want are not already called
by function pointers and so would need to be changed. This means that
plugin hook locations are not explicitly defined, but rather any place
where a function pointer is used can be modified. Personally i prefer
explicit specification of plugin hook locations.
Yes, but you could explicitly mark them. Together with enforced app plugins from above, would that fix it?

Some refactoring is going to be required. Take loop unrolling for example. I wanted the heuristic to be replaceable. There had to be some refactoring because it wasn't built to be in one nicely replaceable chunk, but was spread a bit through the code. This is, I think, going to be unavoidable for any plugin system. My hope is that libplugin requires the minimum amount of code from you which isn't just refactoring. I.e. fewer lines of plugin code.
I think it's important to separate the plugin system from the
application. Doing plugins well, IMO, requires a lot of code. It
shouldn't be spread through the app. It also cleanly separates
plugin mechanism from the actual extensions the app wants.
Finally, plugins have to be extensible too. They should really be on
a nearly equal footing with the app. Otherwise plugin developers
who want the plugins to be extensible will need to reimplement there
own extensibility system.
Without the use of plugin meta-data in XML files and auto-loading and
many of the things discussed, i am not so sure that plugins will be such
a large body of code. It is really a matter of deciding if such features
that libplugin provides are desirable for GCC. If so, then there is a
lot of code required for plugins and libplugin becomes a good idea IMO.
If not, then libplugin may just be more than we need. It really depends
on what "doing plugins well" means for the specific application.
I totally buy that :-)

*Scalable and Granularity*
The system is very scalable. Really this is due to the push
architecture.
The granularity as i understand it is only as fine/coarse as the number
of function pointers in the system that can be overwritten. This is no
different from the pull method (i.e. The granularity depends on where
you put the hook locations) except that function pointers "may already
exist". Though i may have mis-understood something...

I.e. For the "pull" method you can:

Add a "pull" for firePluginEvent() or add a "pull" inside each existing
event handler. Where as the push method requires that the existing event
handlers are called via function pointers and the "push" chains itself
to that.
That's true. But, now you have to write all that code each time (well, maybe. More likely a fat macro) maintain all the listener lists yourself, etc. Not a huge deal, but maybe not as easy as having a plugin system do it all for you. 
I have used a similar method for the "push" plugin in python. The
advantage here is that basically "anything" can be pushed in python so
the system becomes very flexible to extend via "plugins". In C/C++ the
areas that can be extended need to be defined and turned into function
pointers for the push method to work.

Again, assuming i have understood how it works.
I think you have it :-) 
   *Mutliple cooperating plugins
   *I think some of the proposals don't allow multiple plugins or
   plugins aren't able to be extended in the same way that the
   application is.  In libplugin you can have lots of plugins all
   depending on each other.  Plugins can provide extension points as
   well as the application - this means it isn't just a matter of the
   application deciding what's important and everyone else having to
   make do.

In some senses, this is the difference between a plugin system and
loading a few shared libraries. A plugin system provides an
holistic framework for building and using plugins.
One thing this has brought up, is plugin inter-dependencies. I am not
certain that we need to be able to have plugins for plugins, however
some plugins may require others to also be present to operate. My first
thought is that such dependencies would best be avoided but is that an
acceptable restriction?

I.e. If pluginfoo requires pluginblah to have been loaded previously to
work. Do we automatically load pluginblah first or do we expect the user
to specify: -fplugin=blah -fplugin=foo?

The inter-dependency issue between plugins is something we need to
decide on. I.e. Is it necessary, and if so what mechanism to use?
I'm not sure why there's any reason to disallow it. Certainly I can foresee uses for it. Even my existing plugins make use of each other.
Imagine you had a continuous optimisation framework, where each time you compile you program it gets info from a big DB and every time you run a program it stores performance data in the DB. To me this cries out for separate, cooperating plugins. You could do it other ways, but I think that small plugins doing small bits well that do a lot together is better.

If you do ever expect the user to load a few plugins, they shouldn't have to specify them all on the command line. They really shouldn't have to know the dependencies themselves, those are implementation details.

*
Separation of plugins from shared libraries*
If a plugin is only a shared library, then you have to load that
library to find out anything about it. 
Not exactly. I would propose to embed certain symbols or data into the
shared library to provide the necessary meta-data. You can use libbfd to
extract such data from a shared library without having to load it. This
could be as simple as does the plugin export a symbol:
gcc_version_4_3_2. As far as i remember the 3D engine "Crystal Space"
used a combination of the two methods. I.e. It had an XML file that
contained meta-data about the plugin, on platforms that supported it
this data was embedded into the binaries. On others it was a separate
file. What we really need to do is determine what sorts of meta-data we
need embedded into the plugin and after identifying that is an XML file
good for this or is it overkill. For example:

plugin-name
plugin-version
application the plugin is for
application version it works with
That's pretty cool :-) Would be a bit more difficult for structured data or for arrays etc. Using symbol names also clutters the program's namespace.
But if we require lots of other plugin meta-data, it may not be suitable
to use a simple mechanism such as requiring certain exported symbols and
an XML file might be a good idea.
Yup. Libplugin's dependency stuff - which takes the view that nothing should be loaded that isn't needed and nothing should be created that isn't used - needs info that would be difficult to express with only symbol names. (But, if no plugin dependencies are wanted then that's no problem)

The other problem with plugins being only shared libraries is that
you always have to write C code. What if you just want something
really simple? Do you always have to go to the effort of writing a
shared library, compiling it, putting it in the right place, and so
on?
That again is another feature we need to decide if it is desirable?


Libplugin, though, makes every plugin an XML file, not a shared
library. The plugin might use a shared library but it might not.
It can also use more than one.
That is an interesting concept.


*Passes*
Here's a quick description about what happens with passes. I've split
the discussion in two. The first, short part describes what will go
into the next release, in mid-October(ish). The next part is working
but I'm not happy with it yet, so it will wait.
Interesting. So far, we have not looked in detail at how to work with
the pass manager. I only briefly read this but it sounds nice.

*Licensing*

I don't know anything about licensing, but we could do something
similar to the approach that Joern suggested. We could only load
plugins that included the GPL or other approved OSS lisence at the
top of the file. The plugin would then declare that it and
everything it used was good. I don't think people could avoid that
declaration. Maybe I'm wrong.
In previous discussions (See wiki page on GCC plugins) the main issue
was not that the plugins themselves were GPL or not. As i understand it
they will have to also be GPL'ed under the existing GPL (They will need
to link with a GCC internals code to work). The issue was rather GPL
plugins that exported the AST in some proprietary format to a file and
then writing a proprietary program that loads that data and does
something with it. I.e. GCC doing the hard work of generating the tree
and then the proprietary application doing something with that data.

With this in mind, technical measures such as requiring a plugin to
return a specific licence text string etc may not be necessary. If they
are, then we can worry about them when we get more information.

Anyhow, this is not a list for discussing legal stuff so i will be quiet
after this. Everything that people are saying again in other threads has
previously been said on this list and will not produce anything
productive until we get more information. Just check the wiki page or
archives for the histroy on this debate if you are more interested.








Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]