Ltop

ltop@nikhef.nl

16 discussions

Start a nNew thread

ESC14
by Gerhard Raven 12 Aug '14

12 Aug '14

Hi *, I just received some posters for https://web.infn.it/esc14/. Looks like a relevant school… — Gerhard

1 0

gcc (and clang) vector extensions...
by Gerhard Raven 05 Aug '14

05 Aug '14

Can’t resist sharing this one ;-) With a fairly small code change (see https://svnweb.cern.ch/trac/lhcb/changeset/175997) which does some explicit vectorisation using the gcc vector extension, I’ve been able to get the overall HLT to run 3% faster (according to valgrind). Not bad for 2 days of work ;-) (now, if only I could get 3% of the investment money in the event filter farm payed out to me, I could go on vacation for the remainder of the year) — Gerhard

1 0

Building latest brunel
by Wilco Baan Hofman 30 Apr '14

30 Apr '14

Hi all, Sorry I've been silent for awhile, it's been quite busy at the Computer Technology side of things and I've been ill for the last few weeks. So today, I'm back and have ported my build environment to my development machine. I notice that the latest brunel does not run with the default cmt options.. That probably means that the gcc 4.8 update went through, I can get things to compile one by one using cmt with CMTCONFIG=x86_64-slc6-gcc48-opt, but somehow it can't find the dependencies automatically as I've had before. Rob, would it be possible to go over the build environment somewhere this week? -- Wilco

2 1

Fwd: Structure of next computing workshop
by Gerhard Raven 10 Apr '14

10 Apr '14

FYI Begin forwarded message: > From: Marco Cattaneo <Marco.Cattaneo(a)cern.ch> > Subject: Structure of next computing workshop > Date: 9 Apr 2014 14:15:10 GMT+2 > To: "lhcb-comp-mgmt (LHCb Computing Management)" <lhcb-comp-mgmt(a)cern.ch>, Marco Clemencic <Marco.Clemencic(a)cern.ch>, Ben Couturier <Ben.Couturier(a)cern.ch>, Gerhard Raven <gerhard.raven(a)nikhef.nl>, "Niko Neufeld" <Niko.Neufeld(a)cern.ch>, Federico Stagni <Federico.Stagni(a)cern.ch>, Silvia Amerio <Silvia.Amerio(a)cern.ch>, Christopher Rob Jones <jonesc(a)hep.phy.cam.ac.uk>, Ulrik Egede <u.egede(a)imperial.ac.uk>, "Gloria Corti" <Gloria.Corti(a)cern.ch> > > Dear all > > Please find below a suggested structure for the next computing workshop, May 19th to 23rd. Please let me have your comments and suggestions for changes by Monday next week, I would like to announce the preliminary agenda to lhcb-general during next week. The scope of the sessions is the same as the corresponding chapter of the computing tasks note that most of you will have seen > > Monday 19th, pm: Core Computing Infrastructure > > Tuesday 20th, am: Distributed Computing I (focus on LHCbDirac) > Tuesday 20th, pm: Core Software Development and Support (including multicore/GPU R&D reports) > > Wednesday 21st, am: Data Preservation, Analysis tools > Wednesday 21st, pm: Computing operations and Data Processing (including preparations for Stripping21 and for 2015 data taking) > Wednesday 21st, evening: Workshop Dinner > > Thursday 22nd, am: Distributed Analysis, Distributed Computing II (focus on databases) > Thursday 22nd, pm: Distributed Computing III (focus on input to Dirac workshop the following week) > > Friday 23rd, am: Project management (including preparations for CB discussion on resources, reports on external initiatives). > > I have deliberately left plenty of time for Distributed computing, since in the past these are the sessions that have over-run. But there is room to dedicate a few hours to other themes if necessary. Gerhard/Chris: should we dedicate more time to 2015 preparations? > > Please let me know if this looks reasonable. If you'd like to reshuffle some of the sessions, please reply-all so that everyone can comment on proposed changes. > > Thanks > > Marco > > ======================================================== > marco.cattaneo(a)cern.ch - J09110, CERN, CH-1211 Geneve 23 > Tel: +41 22 767 4046 Fax: +41 22 766 9939 > ======================================================== >

3 2

That sandbox-making script I was talking about
by Rob Lambert 14 Mar '14

14 Mar '14

Hi All, A little while ago I was mentioning that a script to excise a particular algorithm from gaudi, and then run just that algorithm on some custom-made sandbox file should be very possible. I've implemented that now, and I think you should be able to find out how to use it through: https://savannah.cern.ch/task/index.php?49868 Cheers, Rob ------------------------------------------ Robert Lambert FOM-VU-NIKHEF-Bfys LHCb Email: rob.lambert(a)cern.ch ------------------------------------------ Nikhef N251 Tel: +31 20 592 2131 Fax: +31 20 592 5155 ------------------------------------------ CERN, 13-1-018 Tel: +41 22 767 4024 Fax: +41 22 766 8109 ------------------------------------------

1 0

LHCb performance monitoring infrastructure contact
by Gerhard Raven 13 Feb '14

13 Feb '14

As just discussed, the LHCb core software contact for ‘performance monitoring infrastructure’ is Ben Couturier <ben.couturier(a)cern.ch> — and the question to ask whether there is a Gaudi Auditor implementation to access the various performance counters on a fine-grained level, i.e. pre- and post- an algorithm is invoked for a single event). I know that Ben at some point E-mailed me a link to http://icl.cs.utk.edu/papi/software/index.html so maybe there indeed is already something integrated with Gaudi… Cheers, — Gerhard

1 0

HEP software collaboration
by Jeff Templon 13 Feb '14

13 Feb '14

Yo, https://indico.cern.ch/event/272618/ see Ian Bird's talk. Gerhard / Rob, were either of you planning to go to this meeting on the 4th of April? I just got the invitation after I complained about not having seen it yet. JT

2 1

Performance impact of lambda functions
by Wilco Baan Hofman 11 Feb '14

11 Feb '14

Hi Gerhard, I've been playing around with your toy example, and as far as lambda functions go, it will only perform better if the alternative is an actual (not inline) function call. Consider the two following functions (not inlined so I can find the assembly code): static __attribute__ ((noinline)) double do_things_lambda(const Hits &c) { // define a predicate auto layer2 = [](const Hit& h) { return h.layer() == 2; }; // define an operation.. double x = 0; auto sumx = [&x](const Hit& h) { x += h.x(); }; // and... action! . for ( const auto& hit : c ) { if (!layer2(*hit)) continue; sumx( *hit ); } return x; } static __attribute__ ((noinline)) double do_things_normal(const Hits &c) { double x = 0; for ( const auto& hit : c ) { if (hit->layer() != 2) continue; x += hit->x(); } return x; } Both functions result in the same assembly code: .LFB1852: .cfi_startproc movq (%rdi), %rax movq 8(%rdi), %rcx xorpd %xmm0, %xmm0 cmpq %rcx, %rax jne .L4 jmp .L8 .p2align 4,,10 .p2align 3 .L3: addq $8, %rax cmpq %rax, %rcx je .L7 .L4: movq (%rax), %rdx cmpl $2, 12(%rdx) jne .L3 movss (%rdx), %xmm1 addq $8, %rax cmpq %rax, %rcx cvtps2pd %xmm1, %xmm1 addsd %xmm1, %xmm0 jne .L4 .L7: rep ret .L8: ret .cfi_endproc As for me, I'd rather see the immediately obvious code as in do_things_normal(), rather than trying to figure out what all lambda-one does. To be continued... :) -- Wilco

2 2

possible optimization of PatFowardTool::fillStereoList???
by Gerhard Raven 11 Feb '14

11 Feb '14

Note: this is a complete tangent to what we discussed today… Looking at the code for PatForwardTool::fillStereoList, I see that the ‘temp' list of hits is sorted at some point: std::sort( temp.begin(), temp.end(), Tf::increasingByProjection<PatForwardHit>() ); and a bitter further down in the code, there are then various loop like this (where itP is an iterator into temp, with itE > itP): //== Add all hits inside the maximum spread. If not enough planes, restart while ( itE != temp.end() && spread > (*itE)->projection() - (*itP)->projection() ) ++itE; which can (I’m pretty sure) be rewritten as: itE = std::find_if( itE, temp.end(), [&](const PatFwdHit* h) { return h->projection() > (*itP)->projection() + spread ; } ); but then I started to think that the sorted-by-projection list is also sorted by the criteria in the above find_if, and hence the std::find_if can be replaced by std::binary_sort… and I suspect that in a few places here actually a std::equal_range is what is really going on. (then again, might be that the # of items skipped over in find_if may be less than log(N)) Anyway, this code really would benefit from someone transforming it into code using C++11 and generic STL algorithms, and by making it more readable so one can recognize what’s really going on… -- Gerhard

1 1

Moore (Brunel) Sandbox
by Rob Lambert 10 Feb '14

10 Feb '14

> 3) "Sandbox" Moore version running HLT2-like tracking. Turns out the nice little Moore trick made by Sebastian to simplify the running of Moore will only be 100% available from Moore v22r0 (a future version to be released in around 2 weeks). We could mimic it's behaviour with sufficient amount of various local checked-out packages, but might be best to stick to Brunel for this first set of tests, to setup the tools you want to use. If you're running on lxplus you can run a Brunel test quite easily, but I think you want to be flexible enough not to have to run there, so I've copied a test file to our local system for you: /data/bfys/rlambert/2012_raw_nu6_125745_0000000007.raw To use it: SetupProject Brunel v44r9 --build-env getpack PRConfig head make SetupProject Brunel v44r9 gaudirun.py --option="from Configurables import Brunel; from PRConfig import TestFileDB; from GaudiConf import IOExtension; TestFileDB.test_file_db['2012_raw_nu6'].run(configurable=Brunel()); IOExtension().inputFiles(['/data/bfys/rlambert/2012_raw_nu6_125745_0000000007.raw'],clear=True); Brunel().SkipEvents=27; Brunel().EvtMax=10;" Which I think is the simplest way to do it... Notice that the input file, the number of events, and the name of the output file are steered here. You will end up with some file matching WilcoTestBrunel*.dst in the directory where you run. This input file has a lot of very large events in it, so in general I expect the events to take a long time to process and several events to be aborted due to too-high occupancy. However I think the events 27->37 are a reasonable example set with no aborts in them. Now, this Brunel will probably run more than you really want it to run, it will spend a lot of time in particle identification and less than 50% of the time in tracking, but it will give the most realistic memory layout, that's for sure. To eliminate the algorithms that you don't really need to be running yourself, you could also add the options: Brunel().Detectors=['Velo','PuVeto','TT','IT','OT','Muon','Magnet','Tr']; from Gaudi.Configuration import *; GaudiSequencer('RecoCALOSeq').Enable=False; GaudiSequencer('RecoRICHSeq').Enable=False; GaudiSequencer('LumiSeq').Enable=False; GaudiSequencer('RecoPROTOSeq').Enable=False;" And then the tracking will take the majority of the time (at the moment, PatVeloTT takes the lions share... I don't really know why that is, probably exploding combinatorics from high TT-occupancy). Cheers, Rob ------------------------------------------ Robert Lambert FOM-VU-NIKHEF-Bfys LHCb Email: rob.lambert(a)cern.ch ------------------------------------------ Nikhef N251 Tel: +31 20 592 2131 Fax: +31 20 592 5155 ------------------------------------------ CERN, 13-1-018 Tel: +41 22 767 4024 Fax: +41 22 766 8109 ------------------------------------------ ________________________________________ From: ltop-bounces(a)nikhef.nl [ltop-bounces(a)nikhef.nl] on behalf of Rob Lambert [Rob.Lambert(a)cern.ch] Sent: 10 February 2014 15:47 To: Jeff Templon Cc: ltop(a)nikhef.nl Subject: Re: [Ltop] Recent news A summary of the stand-up meeting. This week we will aim to define our baseline, answering the following questions. A) What performance gains can we simulate in a realistic toy, what are the tools and numbers we can extract which demonstrate the improvement most effectively? B) What is the current situation in tracking in HLT2 applying those same tools to the real tracking? For this we need three ingredients. 1) Performance monitoring tools [insert favourite performance checking tools here] - also see: https://twiki.cern.ch/twiki/bin/view/LHCb/CodeAnalysisTools 2) Gerhard's "toy" to validate those tools: - This toy allocates some memory and then tests how fast we access it through current techniques versus accessing the exact same data with redesigned data layout redesigned with data-orientated techniques. - needs extra playing with to sparsify the accessed memory by an order of magnitude in order to make it more realistic - needs attaching to the tools from part (1) 3) "Sandbox" Moore version running HLT2-like tracking. - Rob will provide this (might need to talk to Sebastian Neubert to finalize) - needs attaching to the tools from part (1) Cheers, Rob ------------------------------------------ Robert Lambert FOM-VU-NIKHEF-Bfys LHCb Email: rob.lambert(a)cern.ch ------------------------------------------ Nikhef N251 Tel: +31 20 592 2131 Fax: +31 20 592 5155 ------------------------------------------ CERN, 13-1-018 Tel: +41 22 767 4024 Fax: +41 22 766 8109 ------------------------------------------ ________________________________________ From: Jeff Templon [templon(a)nikhef.nl] Sent: 10 February 2014 13:22 To: Rob Lambert Cc: ltop(a)nikhef.nl Subject: Re: [Ltop] Recent news sory meeting with director but go ahead without me. JT On Feb 10, 2014, at 13:10 , Rob Lambert <Rob.Lambert(a)cern.ch> wrote: > Wilco responded that 15:15 would be good for him. Jeff? > > Rob > > ------------------------------------------ > Robert Lambert > FOM-VU-NIKHEF-Bfys LHCb > Email: rob.lambert(a)cern.ch > ------------------------------------------ > Nikhef N251 > Tel: +31 20 592 2131 > Fax: +31 20 592 5155 > ------------------------------------------ > CERN, 13-1-018 > Tel: +41 22 767 4024 > Fax: +41 22 766 8109 > ------------------------------------------ > > ________________________________________ > From: ltop-bounces(a)nikhef.nl [ltop-bounces(a)nikhef.nl] on behalf of Rob Lambert [Rob.Lambert(a)cern.ch] > Sent: 10 February 2014 10:05 > To: Jeff Templon > Cc: ltop(a)nikhef.nl > Subject: Re: [Ltop] Recent news > > Hi All, > > Today at 11, or 15:15? It would be a good start to the week Gerhard+I think. > > Cheers, > > Rob > > ------------------------------------------ > Robert Lambert > FOM-VU-NIKHEF-Bfys LHCb > Email: rob.lambert(a)cern.ch > ------------------------------------------ > Nikhef N251 > Tel: +31 20 592 2131 > Fax: +31 20 592 5155 > ------------------------------------------ > CERN, 13-1-018 > Tel: +41 22 767 4024 > Fax: +41 22 766 8109 > ------------------------------------------ > > ________________________________________ > From: Jeff Templon [templon(a)nikhef.nl] > Sent: 06 February 2014 16:27 > To: Rob Lambert > Cc: Wilco Baan Hofman; ltop(a)nikhef.nl > Subject: Re: [Ltop] Recent news > > yo > > next week :-) > > J " just getting back to my email" T > > On Feb 6, 2014, at 15:23 , Rob Lambert <Rob.Lambert(a)cern.ch> wrote: > >> Hi Jeff, >> >> Sorry, I didn't see any confirmation of "let's do this!" I was very much available. >> >> I could pass by right now if you would like. >> >> Thanks, >> >> Rob >> >> ------------------------------------------ >> Robert Lambert >> FOM-VU-NIKHEF-Bfys LHCb >> Email: rob.lambert(a)cern.ch >> ------------------------------------------ >> Nikhef N251 >> Tel: +31 20 592 2131 >> Fax: +31 20 592 5155 >> ------------------------------------------ >> CERN, 13-1-018 >> Tel: +41 22 767 4024 >> Fax: +41 22 766 8109 >> ------------------------------------------ >> >> ________________________________________ >> From: Jeff Templon [templon(a)nikhef.nl] >> Sent: 06 February 2014 15:20 >> To: Wilco Baan Hofman >> Cc: Rob Lambert; ltop(a)nikhef.nl >> Subject: Re: [Ltop] Recent news >> >> We were there. >> i guess it was a stand up meeting because we got stood up ;-) >> >> JT >> >> On Feb 6, 2014, at 12:52 , Wilco Baan Hofman <wilcobh(a)nikhef.nl> wrote: >> >>> On 05/02/14 17:47, Rob Lambert wrote: >>>> Hi Wilco, etc. >>>> >>>> Some useful progress from my side: >>>> >>>> a) We now have a configuration of the trigger (Moore) which runs only the tracking sequence, which should be a nice compliment to any tests we want to do in Brunel (the much simpler reconstruction application). >>>> >>>> b) We also have gotten together a set of "high-occupancy test files" which should be very useful for this proof-of-principle, and eventual profiling, examining memory footprints/patterns, since this is when it should be the most stressed out with lots of hist and tracks per event. >>>> >>>> c) We're thinking of providing a script where you can slice out an algorithm from a complicated Gaudi set of options for stand-alone running, which should further simplify the making of a sandbox (but since you're probably changing the data structures themselves it is unclear how useful this will be for you). >>>> >>>> Perhaps we should have a 15-minute coffee meeting in the next few days to discuss our first targets? Thursday or Monday would be good for me. >>>> >>> Sure. I can make it today at 15:00h. Not much to report from my side, >>> still haven't found a full day to work on this, though next week is >>> looking pretty good (I've reserved the entire week for this project). >>> >>> -- Wilco >>> >>> >>> >>> _______________________________________________ >>> Ltop mailing list >>> Ltop(a)nikhef.nl >>> https://mailman.nikhef.nl/mailman/listinfo/ltop >> > > _______________________________________________ > Ltop mailing list > Ltop(a)nikhef.nl > https://mailman.nikhef.nl/mailman/listinfo/ltop _______________________________________________ Ltop mailing list Ltop(a)nikhef.nl https://mailman.nikhef.nl/mailman/listinfo/ltop

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

Ltop