Differences between revisions 38 and 39
Revision 38 as of 2013-01-27 14:43:39
Size: 26488
Comment: fix typo in link
Revision 39 as of 2013-01-28 19:27:55
Size: 26768
Comment: more qualifiers about the content of the page.
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
'''This is a summary of the early discussion from 2008 about plugins for GCC. This is not documentation. This does not describe current, official GCC policy. '''The following is a summary created in 2008 of discussions from that time period about plugin support for GCC. The web page contains various personal points of view. It is not documentation for GCC Plugins and does not describe the licensing policy for GCC Plugins. This page does not describe current, official GCC policy.
Line 4: Line 4:
[[http://gcc.gnu.org/onlinedocs/gccint/Plugins.html|GCC Internals Manual: Plugins]]''' [[http://gcc.gnu.org/onlinedocs/gccint/Plugins.html|GCC Internals Manual: Plugins]]

GCC Plugins may use any GPL-compatible license, as described in the documentation about the GCC Plugins API.
'''

The following is a summary created in 2008 of discussions from that time period about plugin support for GCC. The web page contains various personal points of view. It is not documentation for GCC Plugins and does not describe the licensing policy for GCC Plugins. This page does not describe current, official GCC policy.

For official documentation, please see the GCC Internals Manual: Plugins

GCC Plugins may use any GPL-compatible license, as described in the documentation about the GCC Plugins API.

This page, being inspired and based on the "Progress on GCC plugins ?" thread on the gcc devel mailing list, is among others dedicated to providing a factually-formalized overview and summary of key issues and -concerns about possibly providing a plugin architecture in gcc, as discussed and identified in the aforementioned thread. A more recent thread is "Defining a common plugin machinery", which is currently (september 2008) ongoing and not yet summarized here http://gcc.gnu.org/ml/gcc/2008-09/msg00456.html.

Please feel free to populate this page with relevant contents.

"So, how do we permit plugins while prohibiting proprietary plugins, and how do we do it while staying within the bounds of copyright law which is the basis of the GPL?"

* most people participating in the related discussions on the gcc mailing list, suggested already that an unstable plugin API would bring all major advantages of plugins in gcc, while complicating the scenario of proprietary plugins. Indeed, it would probably even make sense to consider having a default policy of the plugin API to be modified for each major release, this could be achieved using automated scripts-which would also increase the motivation for plugin authors to keep their plugins in the main repository.

* it was previously suggested (see below for details) to consider the possible advantages of using a scripting interpreter to provide "plugin" functionality in a platform-agnostic fashion (i.e. no DSO support required), depending on whether such an effort would be based on an existing mainstream scripting language (Python, Perl etc) or possibly a custom one, specifically designed and crafted for gcc-use, it would be possibe to pretty tightly couple the corresponding vm/interpreter and its hooks to gcc, so that include files providing the required API bindings to scripts, would be covered by the GPL. By literally "including" such files from scripts, all sources would automatically have to be GPL'ed as well.

* In addition, all such "plugins" would inevitably be in "human-readable" form, so that all plugins can be easily inspected and modified by users and developers. This, would additionally decrease the motivation to spend money on efforts to develop proprietary plugins, given that all source code of the final product would inevitably need to be shared among potential "clients". Even in scenarios where a proprietary plugin might be successful, it should be rather straight forward to provide an enhanced, GPL'ed variant of it due to all source code being readable.

* A gcc-based scripting interpreter could by default check for a mandatory license statement (header) in every source file, so that "stock gcc" could deny running plugins (scripts) with a non-matching license header/marker telling the user that stock gcc only runs plugin scripts covered by the GPL. This in addition would help clarify the licensing situation and would also require potential providers of proprietary, non-GPL'ed plugins to specifically provide a modified/patched gcc version that runs non-GPL modules, thereby removing the major advantage of plugins in the first place: not having to recompile gcc to add new functionality.

* Other open source projects providing support for scripted plugins, only provide a stable API to those plugins that are maintained in the project's own HEAD repository, which might be another incentive for plugin authors, instead of having to keep up with CVS/HEAD manually, i.e. only those plugins that are being officially maintained in the main repository are considered "officially supported", everything else may stop working anytime.

* GCC could come by default with a GPL'ed set of SDK/API files that are required to create gcc plugins, anybody aiming to produce proprietary, non-GPL'ed plugins would have to come up with their own set of SDK files because they cannot use the GPL'ed "gcc SDK".

* In addition, SDK users could -by design- be required to explicitly set up each individual plugin to be covered by the GPL, e.g. by making the proper "init" calls at startup, along the lines of setLicense(GPLv2);

* In order to make it harder for non-GPL'ed plugin SDKs to be used with gcc, stock gcc could by default also require each plugin to provide a certain set of hooks that provide licensing information to the host, so that stock gcc may refuse to run such binaries that do not provide the required information.

* Furthermore, it would be possible to require all plugins to be statically linked with the corresponding plugin SDK (making the plugin itself GPL'ed thereby), thus whenever the checksum/hashes of linked in files doesn't match (the ones of the SDK), gcc could refuse to run any such plugins. Thus, requiring users to also modify the gcc source code to disable the corresponding checks if they want to run plugins created with an inofficial gcc plugin SDK.

* [http://gcc.gnu.org/ml/gcc/2008-09/msg00495.html |"Couldn't the plugin interface require a copyrighted passphrase to be sent by the plugin when it registeres, and the passphrase can then be licensed under the GPL. Thus, plugins that want to interface with an official FSF gcc will have to be GPLed in order to talk to the plugin interface, and we don't need to tinker with the runtime licensing."]

* A copyrighted passphrase, or requiring static linking, sounds a lot like what Lexmark tried to prevent people from cloning its ink cartridges (see Lexmark Int'l v. Static Control Components). Lexmark embedded a program into each cartridge, then had the printer query the cartridge for its program; if it didn't match, the printer wouldn't work. They then sued a cloner for copyright violation, but lost, on the grounds that the copyrighted program was being used strictly as a passphrase.

Introduction

Scope of potential plugin architecture within GCC as is being discussed currently

Advantages of module/plugin architectures in general

  • dynamically loaded code can be run in applications that thus don't require recompilation to integrate new functionality

Disadvantages of module/plugin architectures in general

Potential advantages of supporting a plugin architecture in GCC

  • "Making it easier to provide additional GCC features"
  • "easily getting code to run within a current version of gcc, that would normally never make it into the mainline for various reasons, without having to patch/bootstrap gcc"

*"But, as you say, the whole idea of the GPL is that people can go build what they want -- including plugin frameworks to GCC that might or might not allow them to interface with proprietary code. If it's really worth it to them, then they will. The FSF's best defense is to make it not worth it by providing enough valuable free software."

Potential disadvantages of supporting a plugin architecture in GCC

Points in support of a plugin architecture in gcc

* "aren't we already in this situation? There are at least 2 compilers out there that re-use parts of GCC by serializing trees and then reading them into a different back end." * "...proprietary projects can already (without plugins) make exporters for GCC which are GPL and then create the majority of their code in other closed source apps that use that data." * "I don't see plugins as changing this except to make it easier. Which is really a major reason for proving plugins isn't it? Making it easier to provide additional GCC features?"

* "Of course, GCCfss also shows that if people want to do this, they will. Adding a plugin framework doesn't even make it notably easier. Once you've written code to translate GIMPLE into some other IR, dropping it into gcc is a matter of changing a few lines. Providing a plugin interface is not the issue."

* " More deeply, I think his concern is misplaced. I think that gcc has already demonstrated that the only widely used compilers are free software. Proprietary compilers don't keep up over time, outside of niche markets. Hooking proprietary code into gcc, one way or another, is just going to create a dead end for the people who do it. Certainly it's not a good thing, and certainly it would be preferable to prevent it if possible. But it is not the worst possible thing that could happen; it is merely a cost."

* "I won't enumerate the benefits of plugins here. But it is clear to me that the benefits outweigh the costs."

* " Proprietary extensions using a standard plug-in interface will eventually get replaced by free ones if they're of value. In the long term, this type of direct assault on GPL would backfire in a big way."

* "I don't believe this is a strong argument. My contention is, and has always been, that GCC is _already_ trivial to integrate into a proprietary compiler. There is at least one compiler I know that does this. In fact, much/most of the effort we have been spending in making GCC easier to maintain, necessarily translates into a system that is easier to integrate into other code bases."

* "IMO, the benefits we gain in making GCC a more attractive code base, far outweigh the fears of someone co-opting it for their own proprietary uses."

* "We gain nothing by holding infrastructure advances in GCC. While GCC still has the advantage of being widely used, its internal infrastructure is still relatively arcane and hard to deal with. We have already kicked it into the mid 90s, but we still have a lot of ground to cover. An antiquated and arcane infrastructure will only help turn new developers away."

* We can make it as technically hard as possible, but it's way too late to make it technically hard. In fact, it's easy. You have to write some code to translate from tree to your proprietary IR, and then you have to plug that code into passes.c.

* If gcc supports plugins, then all we've eliminated is the need to plug that code into passes.c. But that is the easiest part of the job. Adding plugins is not going to require us to support a stable tree interface or anything along those lines; if it did, I would oppose that.

* Adding plugins does not make it noticeably easier to integrate gcc's frontend with a proprietary compiler. And adding plugins would not change the issue of whether such a combination violated the GPL.

* "I think it's quite important for gcc's long-term health to permit and even encourage academic researchers and students to use it. And I see plugins as directly supporting that goal. Note that I don't see any problem with requiring (or attempting to require) that any plugin be covered by the GPL."

* "So from my perspective the downside of plugins is very small, and the upside is very large."

* "Typically, researchers are interested in rapid-prototyping to keep the paper mill going. Plug-ins offers a simple method for avoiding the latencies of repeated bootstrap cycles. Several projects will survive the initial prototyping stages and become techniques we can apply in industrial settings. We want to attract that. Plus we want to attract the grad students that did the research and graduate with a favourable attitude towards using GCC in their future career."

*"All you need is an existing GCC binary installed somewhere in your path that accepts -fplugin=mypass.so. All you compile is your pass, you don't need to even build GCC."

* "Plug-ins also facilitate other kinds of work that are not related to optimization passes. Static analysis, refactoring tools, visualization, code navigation, etc. Sean described a variety of different applications they had implemented with their plug-in framework at the last summit. It was all pretty impressive."

* Nope, you don't. If you are doing static analysis, for instance, you don't care nor need to bootstrap GCC. You just need to load your module every time a file is compiled.

* "If you are doing a pass that you are testing and/or prototyping, you don't want to waste time rebuilding the whole compiler. If/when your pass reaches certain maturity, you think it's ready for production, and people think it's a good idea to have it in the compiler, then you convert it into a static pass and you go through the traditional bootstrap process."

* "Similarly, for visualization plug-ins, you don't want/need to bootstrap the compiler. That's the core idea with plug-ins, they allow more flexibility for experimentation. They are not a means for implementing permanent features in the compiler. We would not guarantee ABI nor API stability, there is just no point to doing that."

* but users who are not gcc developers have trouble doing a bootstrap. Our build process is complicated, and it is not getting simpler.

* "And the sorts of researchers that dnovillo is talking about don't need to do a bootstrap routinely. They are experimenting with new optimizations or with static analysis. They aren't generating actual code. A bootstrap is beside the point."

* " As a research student who spent 6 months working on an improvement to GCC, I agree with all of Diego's remarks. Out of the 6 months, 4 were spent learning the GCC internals and fighting the GCC build process, 1 was spent writing up leaving 1 month of actual productive research. While not all of this would be solved by a plugin system (a lot was down to documentation) it would have significantly increased the amount of time I had to make useful contributions."

* "I can not deny the possibility of somebody writing an (effectively) proprietary optimization pass. However, I consider that to be both unlikely and irrelevant. People can always make private changes to gcc. We care about proprietary changes if they start distributingthem. But that is an unlikely scenario. There is no money to be made in compiler optimization."

* "If we make it clear that we consider any plugin to be covered by the GPL, then any attempt to sell a proprietary optimization will face community opposition, thus making even less business sense."

* "And in the end, what will they have? A marginal improvement over the standard compiler. Since they can not hide the results, we can figure out what they are doing and recreate it. In fact, it will be easier for us to do so than it would be for a private port."

Points in support of not implementing a plugin architecture in gcc

* RMS regularly points to the examples of C++ and Objective-C as an argument for trying to force all extensions to be GPL (Mike Tiemann's employer tried to figure out a way of making g++ proprietary; NeXT tried a "user does the link" hack to get around the GPL for their original Objective-C compiler)."

* "(wrt shim layers)I believe that any such compiler would violate the GPL. But I also believe it's not in the best interest of the FSF to litigate that matter if the linkage between the compiler is anything other than linked in a single executable. Therefore, I think it's important for us to make it as technically hard as possible for people to do such a linkage by readin and writing tree or communicating as different libraries or DLLs. I'm very much against any sort of "plug in" precisely for this reason."

Potential Use Cases of supporting modules/plugins in GCC

* I'm coming from the verification community and I'm currently very interested in '(automatic) code verification'. What our community is really seeking for by now is a tool aiming at providing an abstract (i.e. simplified) model of program sources. And actually, GCC do it internally. Indeed, you can provide a gimplified CFG of C, C++, Java, ... source files and many other informations, which overpass tools such as CIL (C Intermediate Language, http://manju.cs.berkeley.edu/cil/) because of the numerous front-end of GCC."

* I am the maintainer of an open source project called EDoc++. I use GCC to gather data about exceptions and the callgraph of a C/C++ program. This provides the information necessary to perform analysis of C++ exception propagation. I have identified the main issues that have arisen by using a patched GCC in this project that would be solved by using a GCC plugin framework:

  • Distribution : Packages that distribute a patched GCC are large, may miss platform specific patches that are applied to the base GCC distribution, and can cause conflicts or difficulties in determining where/how best to install the patched GCC.

  • Usability : Having multiple versions of GCC installed means that users have to perform some "environment configuration" to determine which GCC to use for which tasks, it can confuse users and may also cause conflicts when the two compilers are accidentally mixed.

  • Maintenance : GCC is actively maintained and new minor versions are released regulary. For a project such as EDoc++ taking advantage of this regular GCC maintenance that is done by others can free me up to develop better "additional" open source projects, rather than taking on the same tasks maintaining the patched GCC in my project that others are already doing for the mainstream GCC. Much of the work i have to do for my project could be eliminated by using a GCC plugin infrastructure.

For example. EDoc++ currently uses GCC 4.0.1 (Yes it is very old) which fails to build on FreeBSD 7. To update my project to use GCC 4.1.0 (for example) I need to modify the build system to check CRC's for the new tarballs and change the patches a little and a few other tasks. What is the real back-breaker is when you start working on the platform specific package maintenance such as creating: dep, rpm, cygwin, pkgsrc packages. If using a GCC plugin framework instead of maintaining my own patched GCC then a lot of this work could be avoided as it is already done by other people. Also my experience is that people are confused about having two versions of GCC installed on their PC. Also my debian package was rejected for inclusion in debian because of the fact that it is "yet another" patched GCC which takes up huge disk space resources. *MANY* of these problems would just disappear with the availability of a GCC plugin framework in the GCC mainline.

* "Another case would be somebody who wanted to keep an optimizer proprietary by making it a plug-in. My view is that because of the linkage with the GCC IR, it can't be proprietary in that case, but that's the harder argument to make legally."

* "To provide an example: Mozilla has been using elsa/oink to do static analysis and rewriting of the codebase. We would love to use a more mature and correct frontend such as GCC... and we would like to eventually have this static analysis be a standard part of the build process when using the GCC compiler."

Functionality wishlist WRT module/plugin support in GCC

Other opensource projects/efforts directly benefiting from potential module/plugin support in gcc

  • gcc-xml could be distributed as a plugin for an unhacked GCC

  • GCC Plugins are currently being deployed at Mozilla to assist with refactoring, having those in mainline would make setting up a development environment much easier.

  • EDoc++ (Static C++ exception analysis) Could be distributed as a plugin providing easier integration into existing platforms (Debian wont accept another modified GCC package for example).

Technical concerns about supporting binary modules/plugins in GCC

  • cross-platform DSOs

Possible ways to address technical issues

  • consider embedding a scripting interpreter to provide support for modules/plugins

* "how about we define plugins in an interpreted or bytecoded language and don't allow any IO whatsoever?"

Legal concerns about supporting binary modules/plugins in GCC

*Moreover, if we did this in conjunction with a carefully-defined API, the defendant could argue that the purpose of that API was precisely to *not* have it be viewed as a single program, so the result of such a case would be far less certain."

  • enforcing licensing (GPL) restrictions

Possible ways to address legal issue

* "[if] no stability in the API [is] offered by GCC to plugins [...] any proprietary plugin would be a nightmare to maintain" * " if we were really clever we could maybe define an interface that's entirely non-enumerable: it calls out to hooks, providing them with just enough information and interfaces to do the job they have to do, but we don't make it possible to derive information about the overall AST because we don't provide any way to know 'what's out there'."

*"I wonder if we could adapt some kind of privsep model, so that once the compiler has finished init, it gives away its rights and can't even open files or create pipes or sockets any more. The gcc.c driver could remain at full user prives and take care of opening everything (a bit like I guess it already does when you're using -pipe?) and cc1 and pals could all be hobbled."

*"if they are using plug-ins, they are forced to include GPL'd header files and link against GPL'd code. The rest is up to the FSF legal folks."

Opensource/-philosophical issues

  • possibly not encouraging people to get involved and contribute code back

* "As was pointed out here numerous times, 90+% of the difficulty in working with is the difficulty in understanding the inherently-complex algorithmic structure of the compiler and its data structures (no real-world compiler will ever be able to use algorithms unmodified from a textbook) coupled with limited documentation. Doing something to make the remaining few percent easier will not, in my opinion, increase the participation in GCC enough to justify the legal risk to Free Software."

Possible ways to address opensource/philosophical issues
* don't guarantee a stable plugin/module API

Notable real life cases of Open Source projects supporting the concept of modules or plugins

  • Apache
  • Mozilla, Firefox
  • KDevelop

GPL'ed projects supporting the notion of binary (GPL'ed) modules/plugins

  • The Linux Kernel

The MELT branch (MiddleEndLispTranslator) provides a sort of plugin machinery by generating C code (and then .so plugins) at compile time from some Lisp dialect.

The Dehydra-GCC project (at Mozilla) is also plugin based.

MILEPOST GCC - uses Interactive Compilation Interface based on plugins to tune compiler optimization heuristics

This page over at the boost wiki seems to be heavily related to this discussion.

"legal" or license issues

It seems (from GCC summit 2008 steering committee Q&A session) that licensing & legal issues related to plugins are been worked upon.

None: GCC_Plugins (last edited 2013-01-28 19:27:55 by DavidEdelsohn)