Re: ObjC/ObjC++: bug fixes for @catch

[Mike Stump <mikestump at comcast dot net>]

On Nov 29, 2010, at 1:14 PM, IainS wrote:
> OK. That's reasonable, and I can re-jig things to replace the additional drivers with a facility to drive a list of sub-dirs from the dg.exp, lto.exp and dg-torture.exp (that doesn't seem like a major headache).

The correct algorithm is, any directory with a .exp file, should be use that exp file.  Any directory without an .exp file, should use the .exp file above it.

Net result, no explicit list of sub dirs, no `exceptions' list for recursive finds...  The existing code, last I looked, sucked in this regard.

> well, I'd argue for  eh/
> -- it has an interesting property that test-cases need not be named "eh-*"-- if there is a more logical name (e.g. PRxxxx) but it will still be clear that they are tests related to eh.

What is old is new again, have you ever noticed:

 find . -name \*.C | xargs grep '// GROUPS'

in the testsuite?  Ever wonder what these lines were for?  Turns out that is the past, back before dejagnu (yes, there was such a time), the driver we used for testing had that feature.  The groups a testcase were in was independent of the filename and you could have 0 or more groups.  The groups persisted though file renaming and movement.

Been there, done that.  History has shown that a mere one or two groups is sufficient for most people most of the time.  In the modern world, that would be the .exp file name, and the prefix the file begins with.

Actually, I was going for the four technical needs for which exceptions.exp was the answer or which answered the question, what problems are solved by not having everything together.

> 1. A Plan that's clear, reasonably robust to growth and explicable to new and existing developers in moderate documentation.
>   (since the ObjC test-suite is currently undocumented - and it does take time to get to grips with it)

I can attempt to explain anything that's unclear...  though, I don't know precisely what is.

Roughly, developers create and use prefixes for the testsuite that serve their needs.  Generally, we respect the desires of each developer to so choose and create the prefix.  Generally, we expect developers to reuse existing prefixes rather than create random new ones.  Also, I'd encourage their use.  pr, in my book, is a poor prefix.  It conveys very, very little information.  For any specific prefix, you'd have to ask that developer.  Most aren't meant to be other than completely transparent.  For example, eh for exception handling for C++.  vbase for virtual bases and so on.  I don't think they are generally collected and documented.  Generally, the testsuite is a write once, read never sort of thing.  Though, sometimes people use it as compiler documentation and feature documentation.

For objc specifically, I don't expect I could explain the existing groups better than I suspect you already know.  If you want to ask about one, just ask.

> 2. an organization that makes it clear what is grouped together

ls -R is meant to make it somewhat clear what the tests are and how they group together.  More than this, READMEs can be added to help people along the way.  See README.gcc for an example of such a document.  After reading it, you can see just how pointless it is.  You could contribute enhancements to it that explained it all...  Generally, no one else has deemed it worth their time.

>  -- so that new and existing developers can find what's been tested and not duplicate tests or effort.

That's easy to answer, if when you fix it, no test cases then passes that failed before, then the reduced testcase isn't duplicative of any other in the testsuite.  If there are testcases, then, you'd have to examine those by hand to see if they actually test the issue you had in mind.  This is in essence the definition of a regression test suite.  If it already works, the general hope is that there is a testcase (or a group of testcases) that tested it, but this in general isn't a given.

The testsuite isn't really a comprehensive feature/functionality testsuite.  That said, it does at times take on that appearance.

Generally, the gcc list or the wiki can be used to try and avoid duplicative work. For objc work, I think you could just send an email to Nicola.

> 3. if possible, the ability to run groups of related tests (preferably without excessive keystrokes) such that when working on one aspect of the FE one can reasonably quickly get feedback on problems.

When developing, I use ^R.exp^M.  Fairly short, runs just the testsuite I'm interested in.  At times, these can be lines that wrap 5 times.  That's my recommendation on what to use minute by minute.  For preping the first one, we'll, there is an unlimited number of answers I could put here, and they are all dependent upon what one wants to do.  The open ended command would be just make check.  If you have specific needs in mind, you'd have to ask a specific question for the specific answer.  There is no document that describes all the possible answers.  For example, running with -fnext-time and -fgnu-runtime is a wonderful test suite run, and makes for an interesting command line...

> 4. I'd prefer to avoid dg.exp (or any other dir) growing until it is 2000 files and the job is Too Big to fix it .. (but that might be a restatement of 1..3  ;-) )

It is _never_ too big to fix.  In fact, the time to fix it, is just after it _has_ grown too big.  The solution is equally trivial.  Run:

	ls | sed 's/-.*//' | sort | uniq -c | sort -nr | sed 3q 

For, for example, in a gcc.dg directory, I get:

110 c99 80 uninit 61 builtins 

110 c99
 80 uninit
 61 builtins

So, trivially, the command to fix this would be something like:

	svn mkdir c99 && svn mv c99-*.c c99

We can discuss what the right n is for splitting.  I might say 50, others might say 200.

>   -- we are already heading for 250 tests in dg.exp (and it would be nearer 450 without the ones we've already split out).

I'm told that modern machines can easily handle more than 500 files in a single directory.  :-)

> Mechanically, I think this can be done with dg.exp=<group-dir>/*.m .. as well as group.exp=*.m ...
> ... so I'm happy to retire the three or so 'un-necessary' .exp files and figure out a change to dg/lto/dg-torture.exp to handle the sub-dirs.

Yes, retiring unnecessary .exp files is good.

> I guess the pre-requisite is the bare-bones of (1) - or effort in re-organizing things is wasted (and time is a short-supply resource).

I don't favor changing test case names once created.  Splitting out larger swaths to reduce 4k files in a directory is generally considered good.

>> Wait, consensus, I thought we practiced benevolent dictator?  :-)
> 
> dang, and I thought this was floating anarchy :-)

Only to the untrained eye.  :-)  Sorry, couldn't resist.

> It creeps up on you...

:-)  If you ever notice it.  I last noticed it when I wanted to split testsuite jobs up for a -j8 testsuite run, and things like builtins.exp and dg.exp (or was it compile.exp or execute.exp) were a bit larger than I wanted.  I merely used =[a-m]*.c and [n-z]*.c.... to avoid the issue.

> ... you might be surprised to count up and see that we've almost doubled the ObjC test-case count (and more than doubled the ObjC++ count) since 4.5 forked.

Wake me when we get to 2k entries or 50 in a single group.  Not there yet.