hi all,
here's a quick rundown of checksumming in GFAL and the various storage systems used throughout biggrid:
dCache ====== - source code not available - support (HEP-preferred?) ADLER32 checksums only - checksum is calculated when the file is uploaded and stored in a database; this means that it is not possible to verify whether the file on disk/tape actually matches the checksum. A dCache operator can trigger a command to re-calculate the checksum
DPM ==== - supports ADLER32, MD5 and CRC32 checksums - checksum is calculated if not present in the DPM namespace and then stored there; this means that the first time you request a checksum it can take a while (and can put quite some stress on the DPM worker nodes) - uses a plain gridftp server with a DPM plugin to calculate the ADLER32 and CRC32 checksums. - I browsed through the sources to check/find out some of this
StoRM ===== - does not seem to support checksums in a consistent manner at all - uses a plain gridftp server which only supports MD5 checksums yet lcg-cr .... --checksum --checksum-type md5 always fails - it is possible to calculate the checksum of a file on the storm gridftp server but it is not stored anywhere for fast access: this makes it ideal for a Denial-of-Server-Storage attack :) - is actually only SRMv2.1 compliant - also does not support ACLs/permissions - no new version has been released since 2007 - I browsed through the sources to check/find out some of this
Hopefully I've missed a few things on the storm front, otherwise things don't look too good ...
share and enjoy,
JJK