coalesced repies

[Tom Lord]

       "Unique hash" is, of course, a contradiction in terms.

Right.

	There are a bunch of different hash approaches that are
	possible and sensible, but it depends on what you want to do.

Zack seems to think that hw failures of repositories are common enough
to worry about -- and that makes sense to me.

Arch repositories in the reference implementation have a useful
property: aside from transient locks and cache files, files are only
ever _added_ to repositories.  They are never modified or removed.
Even cache files are only added and removed -- never modified.  The
repository format is very, very simple. 

It is easy for clients, when handing a file to a repository, to also
hand, say, an MD5 checksum.  It is simple to run a cron job on servers
to verify these periodically -- and would add very little new network
traffic or client complexity to verify checksums when retrieving
files.  arch is already layered in such a way that these new checks
can be added in just one place: a few hundred lines of code.

    If you want to have a sufficiently high probability that no two
    files or file versions have the same hash, then none of the above
    are good enough.  You'd need a longer hash; MD5 is a good example.

In this case, MD5 is handy because its available, strong, and exists
as a separate command on most systems (which is convenient for people
looking at repositories without using an arch implementation).  In
addition to serving as a check of disks (and net connections), since
it is strong, it gives us a nice hook for guarding against some kinds
of naughtiness (one can use a `find' command to to generate a list of
hashes to store ex-repository, and double-check those from time to
time).

So, for Zack's concern about data integrity in the face of disks with
measurable failure rates: we can do this in `arch' with just a few
hundred lines of new code.  If one wanted, one might be able to get
help from the people actively developing arch to add such a feature
quickly.  Personally, I'd like to add it to the larger project of
formally specing the archive format for 1.0.

Part of why I replied to this in so much detail is because I hope to
communicate what I think is an important point about arch.  It's
really quite simple, both in implementation and repository format.  
I think that helps to make it trustworthy, and also helps to make
it a good foundation for ongoing efforts to improve process automation
in your and others' projects.

-t