source mgt. requirements solicitation

[Walter Landry]

Hi,

> I'm now going to list the requirements which I would place on a
> replacement for CVS, in rough decreasing order of importance.  I
> haven't done any research to back them up -- this is just off the top
> of my head (but having thought about the issue quite a bit).
> 
> 0. Must be at least as reliable

To my knowledge, arch doesn't have any real reliability problems.  Tom
didn't make a fast implementation, but it is reliable.  There is a bug
database [1] with all of the bugs that I can think of, so you can
decide for yourself whether it is reliable.

> and at least as portable as CVS.

Arch currently doesn't work on 64 bit machines.  The problem is in the
non-shell parts.  It is not an insurmountable problem, it is just that
no one has taken the time to try to fix the bugs.

Someone got it running under cygwin once, but the patches have
disappeared.  It wasn't usable there.  Too slow.

Otherwise, it seems to work on posix machines.

>    GCC is a very large development effort.  We can't afford to lose
>    contributors because their preferred platform is shut out, nor
>    can we afford to lose work due to bugs, and we *especially*
>    cannot risk a system which has not been audited for security
>    exposures.  It would be relatively easy to give much stronger
>    data integrity guarantees than CVS currently manages:

arch doesn't interact at all with root.  The remote repositories are
all done with sftp, ftp, and http, which is as secure as those servers
are.

> 0a. All data stored in the repository is under an end-to-end
>     checksum.  All data transmitted over the network is independently
>     checksummed (yes, redundant with TCP-layer checksums).  CVS does
>     no checksumming at all.

Sort of.  Patches are gzipped, which have their own checksum, but
there is isn't any way to make sure that what you get is the same
thing as what you put in.  That is, there is some individual
checksums, but no end-to-end checksum.

> 0b. Anonymous repository access is done under a user ID that has only
>     OS-level read privileges on the repository's files.  This cannot
>     be done with (unpatched) CVS.

Is http access good enough?

> 0c. Remote write operations on the repository intrinsically require
>     the use of a protocol which makes strong cryptographic integrity
>     and authority guarantees.  CVS can be set up like this, but it's
>     not built into the design.

Currently, we allow writeable ftp servers and sftp servers.  If we
disallowed writeable ftp servers, would that be good enough?  (Don't
tempt me.  I've considered it in the past.)

> 0d. The data stored in the repository cannot be modified by
>     unprivileged local users except by going through the version
>     control system.  Presently I could take 'vi' to one of the ,v
>     files in /cvs/gcc and break it thoroughly, or sneak something into
>     the file content, and leave no trace.

There is no interaction with root, so if you own the archive, you can
always do what you want.  To get anything approaching this, you have
to deal with PGP signatures, SHA hashes, and the like.  OpenCM is
probably the only group (including BitKeeper) that even comes close to
doing this right.

> 1. Must be at least as fast as CVS for all operations, and should be
>    substantially faster for all operations where CVS uses a braindead
>    algorithm.  I would venture to guess that everyone's #1 complaint
>    about CVS is the amount of time we waste waiting for it to complete
>    this or that request.  To be more specific:

Arch is slow, slow, slow.  Don't let Tom beguile you into thinking
that it is even reasonably fast right now.  It isn't.  It is a subject
of great interest to the developers, but we're not there yet.  Part of
this is the shell implementation.  Once certain parts are rewritten in
a compiled language, it should get _much_ better.

> 1a. Efficient network protocol.  Specifically, a network protocol that,
>     for *all* operations, transmits a volume of data proportional --
>     with a small constant! -- to the size of the diff involved, *not*
>     the total size of all the files touched by the diff involved, as
>     CVS does.

Arch has this, although some of the implementations could do with a
little improvement (e.g. the mirroring script seems to take forever).

> 1b. Efficient tags and branches.  It should be possible to create
>     either by creating *one* metadata record, rather than touching
>     every single file in the repository.

Don't know.  I haven't looked at the actual implementation.  There
isn't a fundamental reason why not, though.

> 1c. Efficient delta storage algorithm, such that checking in a change
>     on the tip of a branch is not orders of magnitude slower than
>     checking in a change on the tip of the trunk.  There are several
>     sane ways to do this.

Arch has this

> 1d. Efficient method for extracting a logical change after the fact,
>     no matter how many files it touched.  (Currently the easiest way
>     to do this is: hunt through the gcc-cvs archive until you find the
>     message describing the checkin you care about, then use wget on
>     all of the per-file diff URLs in the list and glue them all
>     together.  Slow, painful, doesn't always work.)

Arch has this

> 2. Should support this laundry list of features, none of which is
>    known to CVS.  Most of them would be useful independent of the
>    others, though there's not much point to 2b without 2a, nor 2e
>    without 2d.
> 
> 2a. Atomic application of a logical change that touches many files,
>     possibly not all in the same directory. (This is commonly known as
>     a "change set".)  One checkin log per change set is adequate.

Arch has this.  It's why I started using it.

> 2b. Ability to back out an entire change set just as atomically as it
>     went in.

In theory, easy to do (just a few rm's and an mv).  There are larger
policy questions, though (Do we want to allow that?).  Some day, I may
just hack something together that does that.

> 2c. Ability to rename a file, including the ability for a file to have
>     different names on different branches.

Arch has this

> 2d. Automatically remember that a merge occurred from branch A to
>     branch B; later, when a second merge occurs from A to B, don't
>     apply those changes again.

Arch has this

> 2e. Understand the notion of a single-delta merge, either applying
>     just one change from branch A to branch B, or removing just one
>     change formerly on branch A ("subtractive merge").

Single delta forward merges are no problem.  Reverse merges are more
difficult.  This is one of those "lurking design issues" that I
mentioned earlier.

> 2f. Perform conflict resolution by automatic formation of
>     microbranches.

I'm not quite sure what you mean here.

> 3. Should allow a user without commit privileges to generate a change
>    set, making arbitrary changes to the repository (none of this "you
>    can edit files and generate diffs but you can't add or delete
>    files" nonsense), which can be applied by a user who does have
>    commit privileges, and when the original author does an update
>    he/she doesn't get spurious conflicts.

Are you thinking of sending patches by email?  Arch doesn't have that.

> 4. The repository's on-disk data should be stored in a highly compact
>    format, to the maximum extent possible and consonant with being
>    fast.  Being fast is much more important; however, GCC's CVS
>    repository is ~800MB in size and compresses down to ~100MB.  You
>    can do interesting things (like keep a copy of the entire
>    repository on every developer's personal hard disk, as Bitkeeper
>    does) with a 100MB repository that are not so practical when it's
>    closer to a gigabyte.

Arch stores the repository as tar.gz of the initial revision, plus
tar.gz of the patches.  This will be about as compact as anything.

The problem comes when you want to get older revisions.  If you're at
patch-51, getting patch-48 means starting from patch-0 and applying
all 48 patches.  This can be sped up by saving entire trees along the
way, but that kills the "highly compact format".

> 5. Should have the ability to generate ChangeLog files automagically
>    from the checkin comments.  (When merging to basic-improvements I
>    normally spend more time fixing up the ChangeLogs than anything
>    else.  Except maybe waiting for 'cvs tag' and 'cvs update -j...'.)

That apparently works, although I've never used it.

By the way, I thought that your comments were quite illuminating, so I
put them up on the arch web site [2].

I also think that Tom should stop telling everyone to work on arch.
At this point, it just causes more trouble than any help I'll get.

Regards,
Walter Landry
wlandry@ucsd.edu

[1] http://bugs.fifthvision.net:8080/
[2] http://www.fifthvision.net/open/bin/view/Arch/GccHackers