Hello Jan Just,

Luckily (for us) there are a few mistakes in your lines about StoRM. The checksum behaviour is nevertheless exactly what I also notice. I will have to report this to the developers. Here are some corrections for the other parts:

- First a link to the StoRM homepage with up-to-date information: http://storm.forge.cnaf.infn.it/home - SRMv2.2 compliant - Latest release 1.4.0 (15/05/2009) - With respect to ACLs I'm not completely sure yet what the precise capabilities are. If missing it should be fairly straightforward to implement these because StoRM already makes use of extended ACLs on the file system, and these could be made more fine grained. - The plain GridFTP is the VDT 1.6.1 version also used in gLite 3.1. - Checksumming support is currently being improved. Adler32 seems to be optional according to the release notes, but I will have to look into this.

Best wishes,

Fokke

Jan Just Keijser wrote:

hi all,

here's a quick rundown of checksumming in GFAL and the various storage systems used throughout biggrid:

dCache

• source code not available
• support (HEP-preferred?) ADLER32 checksums only
• checksum is calculated when the file is uploaded and stored in a

database; this means that it is not possible to verify whether the file on disk/tape actually matches the checksum. A dCache operator can trigger a command to re-calculate the checksum

DPM

• supports ADLER32, MD5 and CRC32 checksums
• checksum is calculated if not present in the DPM namespace and then

stored there; this means that the first time you request a checksum it can take a while (and can put quite some stress on the DPM worker nodes)

• uses a plain gridftp server with a DPM plugin to calculate the

ADLER32 and CRC32 checksums.

• I browsed through the sources to check/find out some of this

StoRM

• does not seem to support checksums in a consistent manner at all
• uses a plain gridftp server which only supports MD5 checksums yet

lcg-cr .... --checksum --checksum-type md5 always fails

• it is possible to calculate the checksum of a file on the storm

gridftp server but it is not stored anywhere for fast access: this makes it ideal for a Denial-of-Server-Storage attack :)

• is actually only SRMv2.1 compliant
• also does not support ACLs/permissions
• no new version has been released since 2007
• I browsed through the sources to check/find out some of this

Hopefully I've missed a few things on the storm front, otherwise things don't look too good ...

share and enjoy,

JJK _______________________________________________ Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid

-----Original Message----- From: Ron Trompert Sent: woensdag 19 augustus 2009 17:55 To: 'Jan Just Keijser'; ct-grid@nikhef.nl Subject: RE: [Ct-grid] action point: Look at checksumming in GFAL

Hi Jan-Just,

Here is a comment regarding dCache

...

dCache

• source code not available

You can get it if you ask Patrick Fuhrmann nicely.

...

• support (HEP-preferred?) ADLER32 checksums only

It also supports md5

...

• checksum is calculated when the file is uploaded and stored in a

database; this means that it is not possible to verify whether the file on disk/tape actually matches the checksum. A dCache operator can trigger a command to re-calculate the checksum

A checksum is computed when a file is received (on the fs), if there is a mismatch the transfer will fail. Hence you are sure, that when the file transfer is successful, the file is a correct state on disk at that time. If it stays that way is another matter but dCache can be configured so that when a file is transfers to a client or from pool to pool or restored from tape a checksum is computed and compared with the one in the database. If there is a mismatch a CRCException is thrown. AFAIK a file transfer action will fail in that case which means that you can configure dCache so that if you get the file, the file is OK. Otherwise you don't get it at all. I am not 100% sure about the last statement but I have seen pool to pool transfers fail because of this so I presume it is the same for read and restore actions.

Cheers,

Ron