Re: coalesced repies

[Paul Koning <pkoning at equallogic dot com>]

>>>>> "Tom" == Tom Lord <lord@emf.net> writes:

 Tom> I've compiled a list of all asked and implied questions, and
 Tom> provided a reply to each.
 Tom> ...
 Tom> * BK uses extensve checksumming to protect against HW failures.
 Tom> What about arch and svn?  (Joseph S. Myers)

 Tom> Arch is, at present, very weak in this area.  It is very
 Tom> straightforward (O(500) lines of code) to add these protections
 Tom> to the reference implementation.  Because this is a very
 Tom> important issue (IMO), I would want those 500 lines of code to
 Tom> be backed up by carefully crafted design documents and testing.
 Tom> I've been intrigued by some of the lessons offered by the OpenCM
 Tom> design in this area (file revisions in opencm are _identified_
 Tom> by their unique hashes).  This is one of the items on the agenda
 Tom> for my 6-engineer/1-year/$1.2M plan for arch 1.0.

"Unique hash" is, of course, a contradiction in terms.

There are a bunch of different hash approaches that are possible and
sensible, but it depends on what you want to do.

Very simple check codes like the TCP checksum provide basic error
control, but they aren't particularly strong.  For example, the TCP
checksum will not detect data corruption that moves an even-length
block by an even-distance amount within the file.

Somewhat more complex check codes like CRC are quite a lot better.  If
you're worried about random errors (hardware messed up) then CRC-32
may suffice; a suitable 64-bit CRC may be better yet.

If you want to have a sufficiently high probability that no two files
or file versions have the same hash, then none of the above are good
enough.  You'd need a longer hash; MD5 is a good example.

Secure hash functions like MD5 also offer tamper-detection, but only
if you have off-line trusted storage of the "known good" hash values.
If you want tamper detect codes stored with the sources, then you need
digital signatures (e.g., a PGP signature).

In picking a hash code length, remember the birthday paradox.

   paul