RE: Using of parse tree externally

["Dupont, Michael" <michael dot dupont at mciworldcom dot de>]

Lee,

You are right about a using a proper DTD and I think one of the tasks of
this group here is to work one out together.

Here is my contribution and how I got there : 

My first goal was be able to Dump the AST tree-nodes into a XML-like HTML.
Because when I started on this work, I did not understand the tools involved
in XML processing,
I naively thought that I could display an entire tree with DIV's and
Cascading Style Sheets in Netscape!
This would work for 100 nodes, but Netscape hangs after anything bigger than
that.

I have been using the DTD that like the structure of the tree itself.
Every node becomes an entity, every attribute becomes an sub-entity.
Every pointer becomes an href, and the nodes are pushed on the stack as they
are marked to be outputted.

The function is a recursive print routine that is called starting from
toplev.c line : 2583
right after 
	DECL_DEFER_OUTPUT (decl) = 0;
I inserted a function : print_Tree_Node(decl,"toplev.c globals"); 
This is a c interface to the c++ output routines that I have been working
on,
All found in CHandles.cpp.

You can see all this in my first release :
http://www.geocities.com/mdupont777/release1.tgz		

The nodes are then locked and dumped into a format that looks like this : 

	<DIV 	class="field_decl">
		<DIV class="pTreeNode" datatype="ptree"
pointer="0x93332"></DIV>		
		<DIV class="DECLSIZE" datatype="ptree" pointer="0x932345"
></DIV>		
	</DIV>

This is from a overloaded function html_td that dumps one attribute or node.
in html.cpp you find that function 	void html_basic::print_begin(ostream
& os) 
they would be easy to change to produce something like :

	<field_decl>
		<pTreeNode datatype="ptree"  pointer="0x93332"></>

		<decl_size datatype="ptree"  pointer="0x123453"/>

	</field_decl>

If you look in DerivedNodes.cpp
at the function ostream &  field_decl_TreeRef_D::print (ostream & os)
you will see the function the prints one node.
		
From the first pass of all that dumping, I derived from the  perl
HTML::Parser  
I then scanned over all the fields and recordtypes that I have dumped to
produce a file called "output.cpp"
you can see in the file output.cpp the function to dump a tree_common object
:

In the function ParseBig.pl there is a function called
"print_Struct_Process" that produces the following definition : 

void print_node(ostream & os,tree_common & robj)
{
  html_basic tr(os,"DIV","btree_common");

	html_td(os,"chain", robj.chain);
	html_td(os,"type", robj.type);
	html_td(os,"code", robj.code);
	html_td(os,"side_effects_flag", robj.side_effects_flag);
	html_td(os,"constant_flag", robj.constant_flag);
	html_td(os,"permanent_flag", robj.permanent_flag);
	html_td(os,"addressable_flag", robj.addressable_flag);
	html_td(os,"volatile_flag", robj.volatile_flag);
	html_td(os,"readonly_flag", robj.readonly_flag);
	html_td(os,"unsigned_flag", robj.unsigned_flag);
	html_td(os,"asm_written_flag", robj.asm_written_flag);
	html_td(os,"used_flag", robj.used_flag);
	html_td(os,"raises_flag", robj.raises_flag);
	html_td(os,"static_flag", robj.static_flag);
	html_td(os,"public_flag", robj.public_flag);
	html_td(os,"private_flag", robj.private_flag);
	html_td(os,"protected_flag", robj.protected_flag);
	html_td(os,"lang_flag_0", robj.lang_flag_0);
	html_td(os,"lang_flag_1", robj.lang_flag_1);
	html_td(os,"lang_flag_2", robj.lang_flag_2);
	html_td(os,"lang_flag_3", robj.lang_flag_3);
	html_td(os,"lang_flag_4", robj.lang_flag_4);
	html_td(os,"lang_flag_5", robj.lang_flag_5);
	html_td(os,"lang_flag_6", robj.lang_flag_6);
	html_td(os,"pOONode", robj.pOONode);
};

The html_td function is overloaded much like the ostream << operator to
produce correct output for each different type.
I have been moving over to use the second generation output routines, which
creates a Bootstrap-like Feedback loop effect that I have been trying to
create : 
	
	1. Write a program
	2. Read that program in an transform that program into a new
functions for that program with more structure and information.
	3. Include those new functions in the old program and extend it.
	3. Repeat.

So as you see, it is possible to derive a DTD or something like it from the
structure of the header files of a program,
given a well structure data model to begin with, which the gcc has.

The work will have to be made to see if this is an appropiate model, and if
the struture of the tree object *IS* what we 
want to output. 

There are standards emerging for the representation of models like the OMG's
MOF, and we could even make a SOAP server to query and return snippets of
code as needed. 
But real question is, 
	How can we transform from the tree_node structure like DTD to some
future standard DTD, and is this 
	something that can be done with a style sheet transformation
language like XSLT or DSSSL?

Sorry for the long spiel... What do you think?

Mike

-----Original Message-----
From: Lee Iverson [mailto:leei@Canada.AI.SRI.COM]
Sent: 19 October 2000 19:48
To: Mark Mitchell
Cc: Dupont, Michael; gdr@codesourcery.com; espie@quatramaran.ens.fr;
gcc@gcc.gnu.org; unrev-ii@egroups.com
Subject: Re: Using of parse tree externally 



[I've cc'ed this to the unrev-ii@egroups.com mailing list.  For those
 of you seeing this there, this is a response to a discussion of a
 suggestion to provide XML output of the parse tree created by the GCC
 compiler suite. -- leei]

In message <20001019082036K.mitchell@codesourcery.com>, Mark Mitchell
writes:
>>>>>> "Dupont," == Dupont, Michael <michael.dupont@mciworldcom.de> writes:
>
>    Dupont,> Hi Marc, Gaby.
>
>    Dupont,> I am still thinking about how I am going to publish my
>    Dupont,> work properly.
>
>    Dupont,> The work that I have done so far is :
>
>The mode that would be most interesting would be dumping the tree
>nodes as XML.  That facilitates a lot of things.

It certainly does.  It is of course the easiest way to enable almost
all of the *dangerous* possibilities being discussed.

Another tactic would be to only dump cross-referencing info in XML.

One of the greatest difficulties we have in discussing and documenting
software is the difficulty in linking those documents to the source
code in a meaningful and transparent manner.  Consider the usefulness
in having a means of publishing sources in a form whereby it is
trivially possible to navigate from source to documentation to
discussion and back.  We'd potentially be able to combine the
advantages of literate programming, cvsweb and structured external
documentation to hopefully make all of our jobs easier.

Why does there need to be compiler support for this?  Well, for C++ at
least and inhertance-based object-oriented languages in general it can
be extremely difficult to accumulate good cross-referencing
information external to the compiler.  You just have to duplicate too
much of the compiler's work.

Where am I going with this?  Mark knows.  If we can come up with a
decent, flexible, language-independent XML DTD for source-code
cross-referencing then we can dump this information from the compiler
and build other tools to allow use to fully XMLize the sources with
this cross-referencing info.  Given that, we'd have a way to
incorporate actual sources directly into discussions and documentation
via hyperlinking.

There are open sourcce projects that are moving in this direction.
The SDS (http://sds.sourceforge.net) is definitely moving in this
direction, and they'd benefit greatly from such an effort.

Of much more potential long-term benefit is the Open Hyperdocument
System (OHS) project being led by Doug Engelbart (see
http://bootstrap.org, the unrev-ii list on http://egroups.com and
http://ohs.sourceforge.net) which is about collaborative document
management and community building. The targeted bootstrap strategy for
the OHS group places the augmentation of open source software
development at the core of its mission.

Anyone else interested?  I know Mark Mitchell and I are.

----------------------------------------------------------------------------
---
Lee Iverson     		SRI International
leei@ai.sri.com			333 Ravenswood Ave., Menlo Park CA 94025
http://www.ai.sri.com/~leei/	(650) 859-3307

First Release.url