ct-grid October 2008

ct-grid@nikhef.nl

14 participants
32 discussions

[Ct-grid] egee/wlcg intervention procedures
by Jeff Templon 23 Oct '08

23 Oct '08

Hi *, Ik had beloofd om dit rond te sturen: https://edms.cern.ch/cedar/plsql/doc.info?cookie=7918831&document_id=829986… Jullie hadden gelijk: in het EGEE geval, voor minder dan 4 uur down, is een broadcast, 1 dag van te voren, voldoende. JT

1 0

[Ct-grid] Fwd: Re: next grid meeting
by Oscar Koeroo 23 Oct '08

23 Oct '08

Subject: Re: [Ct-grid] next grid meeting From: Tristan Glatard <tristan.glatard(a)wanadoo.fr> Date: Mon, 20 Oct 2008 10:57:15 +0200 To: "ct-grid(a)nikhef.nl" <ct-grid(a)nikhef.nl> CC: "Silvia D. Olabarriaga" <silvia(a)science.uva.nl> Hoi, vlemed has nothing special to report to the meeting, except a good 100% success ratio: we've submitted 1292 jobs last week and they were all successful. Cheers, Tristan Ronald Starink wrote: > The next grid meeting will be on Monday October 20 at 11:00 in room > H3.31. The video-conf setup has been tested and worked well, so we'll > give it a try. > > Agenda: > - Gridview & Infrastructure > - Middleware > - Applications > - Any Other Business > > The minutes of previous meetings are available at: > http://www.nikhef.nl/pub/departments/ct/grid/ > > Minutes will be taken by: Sven > > Enjoy the weekend, > Ronald > > > > > _______________________________________________ > Ct-grid mailing list > Ct-grid(a)nikhef.nl > https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid > > -- ------------------------------------------------------------- Tristan Glatard Institute of Informatics / Amsterdam Medical Center University of Amsterdam http://staff.science.uva.nl/~glatard

1 0

[Ct-grid] Fwd: [Sp-leaders] registratie VL-e workshop. Herinnering!!!
by Jeff Templon 22 Oct '08

22 Oct '08

Hi wie gaat er naar de VL-e workshop? registreren, mijn koekje mag je niet opeten. Begin forwarded message: > From: Jacqueline van der Velde <J.H.vanderVelde(a)uva.nl> > Date: October 22, 2008 13:08:06 GMT+02:00 > To: sp-leaders(a)lists.vl-e.nl, Ammar Benabdelkader <ammar(a)science.uva.nl > >, "Guido van 't Noordende" <noordend(a)science.uva.nl>, "M. Scott > Marshall" <marshall(a)science.uva.nl> > Subject: [Sp-leaders] registratie VL-e workshop. Herinnering!!! > > LS, > Tot nu hebben zich de volgende mensen officieel bij mijn aangemeld. > Kunt u aub de mensen uit uw groep er aan herinneren zich te > registreren. A.s. vrijdag moet ik nl. de catering regelen. > Mvgr > Jacqueline > > > > Matthijs Koot - goup Cees > Jeff Templon > Henk Sips - TUDelft > Keith Cover - VUMC > Ronald Schut - PCC > Maik Nijhuis - VU > Ana-Maria Oprescu - VU > Kees Verstoep - VU > Ceriel Jacobs - VU > Nick Palmer - VU > Lodewijk Bogaards - DANS > Roelof Kemp - VU > Dick Epema - TUDelft > Victor Guevara - UvA > Dirk Out - Unilever > Arie Kaizer - Philips > Ronald van Driel - Philips > Willem de Bruijn - VU > Thilo Kielmann - VU > Alexandru Iosup - TUDelft > Arthur van Schijndel - HP (via Ammar) > Dick Kamphuis - HP (via Ammar) > Kees van Reeuwijk - VU > Bart Grundeken - TU Delft > _______________________________________________ > Sp-leaders mailing list > Sp-leaders(a)lists.vl-e.nl > https://lists.vl-e.nl/mailman/listinfo/sp-leaders

2 2

[Ct-grid] Bugje in kernel kan je netwerkkaart verklooien
by Oscar Koeroo 20 Oct '08

20 Oct '08

Getroffen NIC type e1000e. Vooral in Ubuntu, OpenSUSE 11 en andere distro's met 2.6.27 inclusief rc7. http://tweakers.net/nieuws/55854/grote-linux-distros-getroffen-door-netwerk… Persoonlijk ken ik het e1000 labeltje vooral van die Dell 1950's of zit ik er naast? Deze kernel zou ik overslaan. groet, Oscar

3 3

[Ct-grid] new poll for dates of the grid meeting
by Dennis van Dok 20 Oct '08

20 Oct '08

Hi, some of you have already voiced objections against Tuesday mornings... There is now a Doodle poll, where everyone can enter his/her preferences. I've listed Monday through Friday, and for each day there is a 'before lunch' and 'after lunch' option. Please indicate the when you are 'mostly' available to attend this meeting. http://doodle.ch/participation.html?pollId=ii6z7kyxnizgym95 By end of this week it should be clear which day/time to pick as our new default. Thanks, Dennis -- D.H. van Dok :: Software Engineer :: www.nikhef.nl :: www.vl-e.nl Phone +31 20 592 50 12 :: http://www.nikhef.nl/~dennisvd/

1 0

Re: [Ct-grid] FW: [Condor-team] Hadoop and Condor
by Oscar Koeroo 20 Oct '08

20 Oct '08

Hi Brian, The performance and scalability of HDFS is very interesting. A few questions: - How do you mount HDFS on a system? Is FUSE the only option for a regular mount? As I understand the information on the Hadoop website it would require application to use the Java Virtual FS bindings to Hadoop FS or redirect trough FUSE. - If FUSE is the option, what would the performance penalty be? - For Grid-enabling HDFS do you refer to the Java version of the GridFTPd or the C implementation? thanks, Oscar Brian Bockelman wrote: > Hey Jeff, > > I'd be happy to come and talk about Hadoop and quantify my comments. It > really is an amazing, well-engineered file system. We currently have a > 30TB install (completely from scratch space in worker node disks). Some > of the highlights: > - Resource consumption: If I push it as hard as I can (several hundred > CMSSW processes reading the same file via POSIX I/O), it maybe uses a > 1/3 of a CPU. Can't make its memory consumption show up in graphs. > - Performance: Doing the same test with *one file* and 2 replicas, I can > hit 80Gbps. Yahoo claims that their biggest clusters get up to > 2.2Tbps. I haven't been able to make the namenode break a sweat > - Manageability: Straightforward. The namenode is journaled, and can > checkpoint. There's only two types of nodes (namenode and datanode); > both are straightforward to manage. Out of the box, there are no knobs > to tweak, no optimizations to perform to get decent performance. Just > works. If you lose a data node, it starts re-replicating files within > about 10 seconds. [As a side note, data recovery is actually pretty > straightforward. We lost the disk which holds the namenode's info this > weekend, and restoring from an automatically-created checkpoint was easy.] > > If you want to grid-enable it, you reuse the Globus GridFTP server (plus > bindings) and the BeStMan SRM server. I've clocked it at 8.2Gbps for > WAN traffic. > > For us, the best potential upside is the manageability, although the > fact that performance-wise, it does things dCache can't do. If it can > prove to require far less admin time, we are hoping to expand use beyond > the current test bed. > > Brian > > > On Oct 18, 2008, at 1:50 AM, Jeff Templon wrote: > >> Yo, >> >> I'm talking to Brian Bockleman about other things, maybe if I can >> convince him to come, he can tell us about his experiences with HDFS. >> >> Brian, now you have another reason to come ... give us a short >> technical seminar about HDFS? It could just be you with a piece of >> chalk and a blackboard, as informal as you like? >> >> JT >> >> Oscar Koeroo wrote: >>> Hi Sander, >>> I'm very interested in the details, opportunities and experiences with >>> Hadoop, but I don't have a clue about any of these things. >>> Hadoop: http://hadoop.apache.org/core/ >>> Hadoop FAQ >>> 1. What is Hadoop? >>> [WWW] Hadoop is a distributed computing platform written in Java. It >>> incorporates features similar to those of the [WWW] Google File System >>> and of [WWW] MapReduce. For some details, see HadoopMapReduce. >>> Oscar >>> Sander Klous wrote: >>>> Hi >>>> What is Hadoop exactly? Reading through the specs it doesn't look >>>> like a >>>> competitor for dCache. So, can anybody comment on the remark below: >>>> "HDFS is >>>> much better than dCache"? >>>> Thanks, >>>> Sander >>>> >>>>> Plan: >>>>> >>>>> 1) Gain experience w/ the Hadoop File System via a large >>>>> installation on the cluster in B240. >>>>> >>>>> 2) Assuming it really works according to our expectations, >>>>> investigate how to best "pseudo-daemoncore-ify" the HDFS service >>>>> so that runs under the condor_master, responds to Condor >>>>> administrative commands such as on, off, restart, reconfig, etc, >>>>> and hopefully has similar behavior for a debug log and config >>>>> settings. Note: some challenges here include HDFS service is >>>>> implemented in Java, perhaps relies on ssh for network >>>>> authentication, etc. >>>>> >>>>> 3) Enable the Condor file transfer service to stage files in/out >>>>> of HDFS, in addition to just the shadow. >>>> That's really interesting. >>>> >>>> As one datapoint: I was talking to Brian Bockelman about HDFS. He >>>> has been deployed HDFS at his OSG site in Nebraska, and thinks that >>>> it is great, much better than dCache. >>>> >>>> An interesting question, once we get far enough along with this >>>> work: can the interface to HDFS be made pluggable? That is, can >>>> Condor call out to a file transfer service? I don't know if it >>>> makes sense, but we might be able to plug in other services one day. >>>> >>>> >>>> _______________________________________________ >>>> Ct-grid mailing list >>>> Ct-grid(a)nikhef.nl >>>> https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid >>> ------------------------------------------------------------------------ >>> _______________________________________________ >>> Ct-grid mailing list >>> Ct-grid(a)nikhef.nl >>> https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid

2 1

[Ct-grid] proposed dates for the remaining grid meetings 2008
by Dennis van Dok 20 Oct '08

20 Oct '08

As discussed in today's meeting, we're going to change the time, frequency and agenda of the grid meeting. The plan is as follows: 1. To have a bi-weekly meeting, starting next week, 2. on tuesdays at 11.00 a.m., 3. at Nikhef, 4. with a focus on applications. 'Operations' will be summarily discussed, as most operational issues will move to a separate meeting. The proposed dates for the remaining meetings of 2008 are: 28 October, 11 November, 25 November, 9 December. If there are no serious objections, I'll finalize these dates by end of this week. Cheers, Dennis -- D.H. van Dok :: Software Engineer :: www.nikhef.nl :: www.vl-e.nl Phone +31 20 592 50 12 :: http://www.nikhef.nl/~dennisvd/

1 0

[Ct-grid] Foto's open dag
by David Groep 19 Oct '08

19 Oct '08

Hi all, Foto's van de open dag staan nu op: http://www.nikhef.nl/grid/photos/?dir=od2008 Grtn, DavidG. -- David Groep ** National Institute for Nuclear and High Energy Physics, PDP projectgroup ** ** Room: H1.57 Phone: +31 20 592 2179, PObox 41882, NL-1009 DB Amsterdam NL **

1 0

[Ct-grid] FW: [Condor-team] Hadoop and Condor
by Sander Klous 18 Oct '08

18 Oct '08

Hi What is Hadoop exactly? Reading through the specs it doesn't look like a competitor for dCache. So, can anybody comment on the remark below: "HDFS is much better than dCache"? Thanks, Sander > Plan: > > 1) Gain experience w/ the Hadoop File System via a large > installation on the cluster in B240. > > 2) Assuming it really works according to our expectations, > investigate how to best "pseudo-daemoncore-ify" the HDFS service so > that runs under the condor_master, responds to Condor administrative > commands such as on, off, restart, reconfig, etc, and hopefully has > similar behavior for a debug log and config settings. Note: some > challenges here include HDFS service is implemented in Java, perhaps > relies on ssh for network authentication, etc. > > 3) Enable the Condor file transfer service to stage files in/out of > HDFS, in addition to just the shadow. That's really interesting. As one datapoint: I was talking to Brian Bockelman about HDFS. He has been deployed HDFS at his OSG site in Nebraska, and thinks that it is great, much better than dCache. An interesting question, once we get far enough along with this work: can the interface to HDFS be made pluggable? That is, can Condor call out to a file transfer service? I don't know if it makes sense, but we might be able to plug in other services one day.

3 2

[Ct-grid] Fwd: glite WN Client Distribution (revisited)
by Jeff Templon 17 Oct '08

17 Oct '08

FYI, Begin forwarded message: > From: "Gordon, JC (John)" <john.gordon(a)stfc.ac.uk> > Date: 17 October 2008 16:54:21 GMT+02:00 > To: "project-lcg-gdb (LCG - Grid Deployment Board)" <project-lcg-gdb(a)cern.ch > >, "project-egee-roc-managers (EGEE ROC managers and alternates)" <project-egee-roc-managers(a)cern.ch > > > Subject: glite WN Client Distribution (revisited) > > You will remember that we discussed a proposal at the September GDB > about remote installation of WN client middleware. The opinion of > the meeting was that multiple releases was an issue for the VO so we > sent Oliver Keeble homewards to think again. With our (and other) > feedback from the first 'client distribution' proposal the concept > has been rethought. > > https://twiki.cern.ch/twiki/bin/view/EGEE/ParallelMWClients > > The idea of a central agency installing updates has been removed, > and the emphasis is now on making it as easy as possible for sites > themselves to maintain parallel distributions of the client > middleware. > > We will discuss this at the next GDB on November 12th. Email > discussion until then is encouraged. As is discussion in the wider > community > > John Gordon > > > > -- > Scanned by iCritical for STFC. > >

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

ct-grid October 2008