April 2024 - Igwn-dhtc-ops - Nikhef Listerserver

Re: [#76253] LIGO-WA missing IGWN jobs (and missing glidein logs)
by Paschalis Paschos 30 Apr '24

30 Apr '24

Hi James, Jeff, Re: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… Jeff: Admins are on the ticket. @ligo If the pilots are failing and factory doesn't get logs back, we will need site admins to investigate a job for the error encountered. On Tue, 30 Apr at 11:05 AM , Jeffrey Dost <support(a)opensciencegrid.org> wrote: Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… Indeed for this site we don't seem to be getting pilot logs back.. although they do seem to be running, so I'm suspecting something is going wrong at startup / validation time, none are registering since april 11 [1] When I check on the batch side at ldas-osg-ce.ligo-wa.caltech.edu, I see plenty of idle glideins from the osgpilot user, and only 1 running currently. Without getting the logs back there isn't much else we can determine from our side. Are any admins on the ticket? Can anyone find any stderr / stdout from recent osgpilot user jobs from the workers? Thanks, Jeff [1] http://gfactory-2.opensciencegrid.org/factory/monitor/factoryStatus.html?en… On Thu, 18 Apr at 9:58 PM , Andrijauskas, Fabio <support(a)opensciencegrid.org> wrote: Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… We are checking it. Bests, On Thu, 18 Apr at 3:34 PM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender It looks like the last user payload at LIGO-WA ended about a week ago with no activity and plenty of demand since. The frontend shows glideins cut out on the 4C entrypoint about the same time [1] and the 8C entrypoint earlier than that [2]. The weird (to me) thing is the factory seems to think there are 9 glideins running right now [3]. Can anyone remind me how to interpret that? Finally, I don't actually see *any* glidein logs from the new(er) LIGO-WA CE entries at the expected location @ CIT (and the most recent LIGO-CIT glidein log is from October 13 last year). Is the rsync from the production factory working ok? [1] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [2] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [3] http://gfactory-2.opensciencegrid.org/factory/monitor/factoryEntryStatusNow… -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

2 1

Re: [#76253] LIGO-WA missing IGWN jobs (and missing glidein logs)
by Jeffrey Dost 30 Apr '24

30 Apr '24

Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… Indeed for this site we don't seem to be getting pilot logs back.. although they do seem to be running, so I'm suspecting something is going wrong at startup / validation time, none are registering since april 11 [1] When I check on the batch side at ldas-osg-ce.ligo-wa.caltech.edu, I see plenty of idle glideins from the osgpilot user, and only 1 running currently. Without getting the logs back there isn't much else we can determine from our side. Are any admins on the ticket? Can anyone find any stderr / stdout from recent osgpilot user jobs from the workers? Thanks, Jeff [1] http://gfactory-2.opensciencegrid.org/factory/monitor/factoryStatus.html?en… On Thu, 18 Apr at 9:58 PM , Andrijauskas, Fabio <support(a)opensciencegrid.org> wrote: Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… We are checking it. Bests, On Thu, 18 Apr at 3:34 PM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender It looks like the last user payload at LIGO-WA ended about a week ago with no activity and plenty of demand since. The frontend shows glideins cut out on the 4C entrypoint about the same time [1] and the 8C entrypoint earlier than that [2]. The weird (to me) thing is the factory seems to think there are 9 glideins running right now [3]. Can anyone remind me how to interpret that? Finally, I don't actually see *any* glidein logs from the new(er) LIGO-WA CE entries at the expected location @ CIT (and the most recent LIGO-CIT glidein log is from October 13 last year). Is the rsync from the production factory working ok? [1] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [2] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [3] http://gfactory-2.opensciencegrid.org/factory/monitor/factoryEntryStatusNow… -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

1 0

Re: [#76259] IGWN pool glidein logs
by Jeffrey Dost 30 Apr '24

30 Apr '24

Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… I see your new ticket to investigate LIGO-WA (https://support.opensciencegrid.org/a/tickets/76253) Let's continue the discussion there. I'll close this one. Thanks, Jeff On Fri, 26 Apr at 1:08 PM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender No, we generally do not have quotas at CIT. I just had a chance to spot-check and we *do* have glidein logs for the LIGO-CIT entries as recently as 30 minutes ago. So I am reasonably convinced the rsync itself is indeed working and that I must have been looking at old / defunct entries for LIGO-WA. We are not actually getting any logs from LIGO-WA yet but I think that will be a separate matter and this particular ticket can be closed. On 4/26/24 13:30, OSG Support wrote: > There is a new comment in the ticket submitted by James Clark to OSG > User Documentation > > > Ticket URL: > https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… <https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70…> > > Comment added by : Paschalis Paschos > > Comment Content: > [EXTERNAL] – This message is from an external sender > do you have a quota on that /home/osg.factory/ directory? > > On Thu, Apr 25, 2024 at 1:11 PM Clark, James A. <jaclark(a)caltech.edu > <mailto:jaclark@caltech.edu>> wrote: > > Hi Pascal, > > Right, I moved the old logs out the way and let it start over. > I'm still on my way back home from Madison but it did look like > it has repopulated when I looked yesterday. > > I will check in a little later today/tomorrow. > > Get Outlook for Android > <https://urldefense.com/v3/__https://aka.ms/AAb9ysg__;!!BpyFHLRN4TMTrA!5cCZm…> > ------------------------------------------------------------------------ > *From:* Paschalis Paschos <support(a)opensciencegrid.org > <mailto:support@opensciencegrid.org>> > *Sent:* Thursday, April 25, 2024 11:26:24 AM > *To:* james.clark(a)ligo.org <mailto:james.clark@ligo.org> > <james.clark(a)ligo.org <mailto:james.clark@ligo.org>> > *Cc:* paschos(a)uchicago.edu <mailto:paschos@uchicago.edu> > <paschos(a)uchicago.edu <mailto:paschos@uchicago.edu>>; > agraves10(a)unl.edu <mailto:agraves10@unl.edu> <agraves10(a)unl.edu > <mailto:agraves10@unl.edu>>; jdost(a)ucsd.edu > <mailto:jdost@ucsd.edu> <jdost(a)ucsd.edu > <mailto:jdost@ucsd.edu>>; igwn-dhtc-ops(a)nikhef.nl > <mailto:igwn-dhtc-ops@nikhef.nl> <igwn-dhtc-ops(a)nikhef.nl > <mailto:igwn-dhtc-ops@nikhef.nl>>; Clark, James A. > <jaclark(a)caltech.edu <mailto:jaclark@caltech.edu>> > *Subject:* Re: [#76259] IGWN pool glidein logs > Hi Jeff, > > *Re:* > https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… <https://urldefense.com/v3/__https://support.opensciencegrid.org/support/tic…> > > there is got to be a process (cron) that rsyncs factory pilots > logs for LIGO back to CIT. > > James, I see logs in /home/osg.factory/gfaclogs/sdsc/igwn from > various sites > > - P. > > > > > On Wed, 24 Apr at 1:20 PM , Clark, James A. > <jaclark(a)caltech.edu <mailto:jaclark@caltech.edu>> wrote: > [EXTERNAL] – This message is from an external sender > Hi Jeff, > > At the time, I was having trouble finding recent glidein > logs for any entry and couldn't find correctly named > directories for the LIGO-WA entries (capital C suffix, > rather than lower case). > > I noticed some more recent logs yesterday, however, so let > me dig around a bit more. > > > ------------------------------------------------------------------------ > ** > > From: Jeffrey Dost <support(a)opensciencegrid.org > <mailto:support@opensciencegrid.org>> > *Sent:* Tuesday, April 23, 2024 12:14 PM > *To:* james.clark(a)ligo.org <mailto:james.clark@ligo.org> > <james.clark(a)ligo.org <mailto:james.clark@ligo.org>> > *Cc:* paschos(a)uchicago.edu <mailto:paschos@uchicago.edu> > <paschos(a)uchicago.edu <mailto:paschos@uchicago.edu>>; > agraves10(a)unl.edu <mailto:agraves10@unl.edu> > <agraves10(a)unl.edu <mailto:agraves10@unl.edu>>; > jdost(a)ucsd.edu <mailto:jdost@ucsd.edu> <jdost(a)ucsd.edu > <mailto:jdost@ucsd.edu>>; igwn-dhtc-ops(a)nikhef.nl > <mailto:igwn-dhtc-ops@nikhef.nl> > <igwn-dhtc-ops(a)nikhef.nl <mailto:igwn-dhtc-ops@nikhef.nl>> > *Subject:* Re: [#76259] IGWN pool glidein logs > Hi James, > > *Re:* > https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… <https://urldefense.com/v3/__https://support.opensciencegrid.org/support/tic…> > > Can you please confirm which CIT entries you want me to > investigate? looks like we have several but I see this > one serving a lot of glideins: > http://gfactory-2.opensciencegrid.org/factory/monitor/factoryStatus.html?en… <https://urldefense.com/v3/__http://gfactory-2.opensciencegrid.org/factory/m…> > > Thanks, > Jeff > > > On Tue, 23 Apr at 11:07 AM , Paschalis Paschos > <support(a)opensciencegrid.org > <mailto:support@opensciencegrid.org>> wrote: > Hi James, > > *Re:* > https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… <https://urldefense.com/v3/__https://support.opensciencegrid.org/support/tic…> > > I am not sure how do I check on that. I don't see > any logs on CIT. Is it a pull request from CIT via a > cron job or the factory actively sending those to you? > > > On Fri, 19 Apr at 11:00 AM , James Clark > <james.clark(a)ligo.org > <mailto:james.clark@ligo.org>> wrote: > [EXTERNAL] – This message is from an external sender > > As noted in #76253 I'm having a hard time > finding any recent factory > glidein logs in the rsync'd directory at CIT. > > Can we confirm access is working ok? I know > there was a mysterious > issues for the ITB logs which seemed to resolve > itself. > > > -- > James Alexander Clark > LIGO Laboratory > California Institute of Technology > email: james.clark(a)ligo.org > <mailto:james.clark@ligo.org> > Tel. (cell): 413-230-1412 > > > > -- > /Pascal Paschos, Ph.D./ > /OSG/PATh Collaboration Support / > /Enrico Fermi Institute - //University of Chicago/ > /ph: 773-702-4679/ > > > 76259:196291 > > _______________________________________________ > Igwn-dhtc-ops mailing list -- igwn-dhtc-ops(a)nikhef.nl > To unsubscribe send an email to igwn-dhtc-ops-leave(a)nikhef.nl -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412 _______________________________________________ Igwn-dhtc-ops mailing list -- igwn-dhtc-ops(a)nikhef.nl To unsubscribe send an email to igwn-dhtc-ops-leave(a)nikhef.nl

1 0

New comment - [#76259] IGWN pool glidein logs
by OSG Support 26 Apr '24

26 Apr '24

There is a new comment in the ticket submitted by James Clark to OSG User Documentation Ticket URL: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… Comment added by : Paschalis Paschos Comment Content: <div>[EXTERNAL] – This message is from an external sender</div> <div></div> <div> <div dir="ltr">do you have a quota on that <span style="font-family:verdana,arial,sans-serif;font-size:13px">/home/osg.factory/ </span>directory? </div> <br> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr"></div> </div> </div> <div class="freshdesk_quote"><blockquote class="freshdesk_quote"> <div>On Thu, Apr 25, 2024 at 1:11 PM Clark, James A. <<a href="mailto:jaclark@caltech.edu" rel="noreferrer">jaclark(a)caltech.edu</a>> wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div> <div dir="auto">Hi Pascal,</div> <div dir="auto"> <br> </div> <div dir="auto">Right, I moved the old logs out the way and let it start over. I'm still on my way back home from Madison but it did look like it has repopulated when I looked yesterday. </div> <div dir="auto"> <br> </div> <div dir="auto">I will check in a little later today/tomorrow. <span></span> </div> <div> <br> </div> <div id="m_1621215644628360949ms-outlook-mobile-signature" dir="auto">Get <a href="https://urldefense.com/v3/__https://aka.ms/AAb9ysg__;!!BpyFHLRN4TMTrA!5cCZm…" target="_blank" rel="noreferrer"> Outlook for Android</a> </div> <hr style="display:inline-block;width:98%"> <div id="m_1621215644628360949divRplyFwdMsg" dir="ltr"> <font style="font-size:11pt" color="#000000"><b>From:</b> Paschalis Paschos <<a href="mailto:support@opensciencegrid.org" target="_blank" rel="noreferrer">support(a)opensciencegrid.org</a>><br> <b>Sent:</b> Thursday, April 25, 2024 11:26:24 AM<br> <b>To:</b> <a href="mailto:james.clark@ligo.org" target="_blank" rel="noreferrer">james.clark(a)ligo.org</a> <<a href="mailto:james.clark@ligo.org" target="_blank" rel="noreferrer">james.clark(a)ligo.org</a>><br> <b>Cc:</b> <a href="mailto:paschos@uchicago.edu" target="_blank" rel="noreferrer">paschos(a)uchicago.edu</a> <<a href="mailto:paschos@uchicago.edu" target="_blank" rel="noreferrer">paschos(a)uchicago.edu</a>>; <a href="mailto:agraves10@unl.edu" target="_blank" rel="noreferrer">agraves10(a)unl.edu</a> <<a href="mailto:agraves10@unl.edu" target="_blank" rel="noreferrer">agraves10(a)unl.edu</a>>; <a href="mailto:jdost@ucsd.edu" target="_blank" rel="noreferrer">jdost(a)ucsd.edu</a> <<a href="mailto:jdost@ucsd.edu" target="_blank" rel="noreferrer">jdost(a)ucsd.edu</a>>; <a href="mailto:igwn-dhtc-ops@nikhef.nl" target="_blank" rel="noreferrer">igwn-dhtc-ops(a)nikhef.nl</a> <<a href="mailto:igwn-dhtc-ops@nikhef.nl" target="_blank" rel="noreferrer">igwn-dhtc-ops(a)nikhef.nl</a>>; Clark, James A. <<a href="mailto:jaclark@caltech.edu" target="_blank" rel="noreferrer">jaclark(a)caltech.edu</a>><br> <b>Subject:</b> Re: [#76259] IGWN pool glidein logs</font> <div> </div> </div> <div> <div style="font-family:verdana,arial,sans-serif;font-size:13px"> <div style="font-family:verdana,arial,sans-serif;font-size:13px"> <div dir="ltr"> <div dir="ltr">Hi Jeff,</div> <div> <br> </div> <div> <strong>Re:</strong> <a href="https://urldefense.com/v3/__https://support.opensciencegrid.org/support/tic…" rel="noreferrer" target="_blank"> https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70…</a> </div> <div> <br> </div> <div dir="ltr">there is got to be a process (cron) that rsyncs factory pilots logs for LIGO back to CIT. </div> <div dir="ltr"> <br> </div> <div dir="ltr">James, I see logs in /home/osg.factory/gfaclogs/sdsc/igwn from various sites</div> <div dir="ltr"> <br> </div> <div dir="ltr">- P.</div> <div dir="ltr"> <br> </div> <div> <br> </div> <div> <br> </div> </div> <div dir="ltr"> <div> <br> </div> </div> <div> <blockquote>On Wed, 24 Apr at 1:20 PM <span>, Clark, James A. <<a href="mailto:jaclark@caltech.edu" target="_blank" rel="noreferrer">jaclark(a)caltech.edu</a>> wrote: <div>[EXTERNAL] – This message is from an external sender</div> <div></div> <div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Hi Jeff,</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> At the time, I was having trouble finding recent glidein logs for any entry and couldn't find correctly named directories for the LIGO-WA entries (capital C suffix, rather than lower case).</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> I noticed some more recent logs yesterday, however, so let me dig around a bit more.</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> <br> </div> <div id="m_1621215644628360949x_appendonsend"></div> <hr style="display:inline-block;width:98%"> <div id="m_1621215644628360949x_divRplyFwdMsg" dir="ltr"><font color="#000000" style="font-size:11pt"><b></b></font></div> </div> <div> <blockquote> <div>From: Jeffrey Dost <<a href="mailto:support@opensciencegrid.org" target="_blank" rel="noreferrer">support(a)opensciencegrid.org</a>><br> <b>Sent:</b> Tuesday, April 23, 2024 12:14 PM<br> <b>To:</b> <a href="mailto:james.clark@ligo.org" target="_blank" rel="noreferrer">james.clark(a)ligo.org</a> <<a href="mailto:james.clark@ligo.org" target="_blank" rel="noreferrer">james.clark(a)ligo.org</a>><br> <b>Cc:</b> <a href="mailto:paschos@uchicago.edu" target="_blank" rel="noreferrer">paschos(a)uchicago.edu</a> <<a href="mailto:paschos@uchicago.edu" target="_blank" rel="noreferrer">paschos(a)uchicago.edu</a>>; <a href="mailto:agraves10@unl.edu" target="_blank" rel="noreferrer">agraves10(a)unl.edu</a> <<a href="mailto:agraves10@unl.edu" target="_blank" rel="noreferrer">agraves10(a)unl.edu</a>>; <a href="mailto:jdost@ucsd.edu" target="_blank" rel="noreferrer">jdost(a)ucsd.edu</a> <<a href="mailto:jdost@ucsd.edu" target="_blank" rel="noreferrer">jdost(a)ucsd.edu</a>>; <a href="mailto:igwn-dhtc-ops@nikhef.nl" target="_blank" rel="noreferrer">igwn-dhtc-ops(a)nikhef.nl</a> <<a href="mailto:igwn-dhtc-ops@nikhef.nl" target="_blank" rel="noreferrer">igwn-dhtc-ops(a)nikhef.nl</a>><br> <b>Subject:</b> Re: [#76259] IGWN pool glidein logs </div> <div> </div> <div> <div style="font-family:verdana,arial,sans-serif;font-size:13px"> <div style="font-family:verdana,arial,sans-serif;font-size:13px"> <div dir="ltr"> <div>Hi James,</div> <div> <br> </div> <div> <strong>Re:</strong> <a href="https://urldefense.com/v3/__https://support.opensciencegrid.org/support/tic…" rel="noreferrer" target="_blank"> https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70…</a> </div> <div> <br> </div> <div dir="ltr">Can you please confirm which CIT entries you want me to investigate? looks like we have several but I see this one serving a lot of glideins:</div> <div><a href="https://urldefense.com/v3/__http://gfactory-2.opensciencegrid.org/factory/m…" rel="noreferrer" target="_blank">http://gfactory-2.opensciencegrid.org/factory/monitor/factoryStatus.html?en…</a></div> <div> <br> </div> <div dir="ltr">Thanks,</div> <div dir="ltr">Jeff<br> </div> <div> <br> </div> </div> <div dir="ltr"> <div> <br> </div> </div> <div> <blockquote>On Tue, 23 Apr at 11:07 AM <span>, Paschalis Paschos <<a href="mailto:support@opensciencegrid.org" target="_blank" rel="noreferrer">support(a)opensciencegrid.org</a>> wrote: <div style="font-family:verdana,arial,sans-serif;font-size:13px"> <div dir="ltr"> <div>Hi James,</div> <div> <br> </div> <div> <strong>Re:</strong> <a href="https://urldefense.com/v3/__https://support.opensciencegrid.org/support/tic…" rel="noreferrer" target="_blank"> https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70…</a> </div> <div> <br> </div> <div dir="ltr">I am not sure how do I check on that. I don't see any logs on CIT. Is it a pull request from CIT via a cron job or the factory actively sending those to you? <br> </div> <div> <br> </div> </div> <div dir="ltr"> <div> <br> </div> </div> <div> <blockquote>On Fri, 19 Apr at 11:00 AM <span>, James Clark <<a href="mailto:james.clark@ligo.org" target="_blank" rel="noreferrer">james.clark(a)ligo.org</a>> wrote: <div>[EXTERNAL] – This message is from an external sender<br> <br> As noted in #76253 I'm having a hard time finding any recent factory<br> glidein logs in the rsync'd directory at CIT.<br> <br> Can we confirm access is working ok? I know there was a mysterious<br> issues for the ITB logs which seemed to resolve itself.<br> <br> <br> --<br> James Alexander Clark<br> LIGO Laboratory<br> California Institute of Technology<br> email: <a href="mailto:james.clark@ligo.org" target="_blank" rel="noreferrer">james.clark(a)ligo.org</a><br> Tel. (cell): 413-230-1412<br> </div> </span> </blockquote> </div> </div> </span> </blockquote> </div> </div> </div> </div> </blockquote> </div> </span> </blockquote> </div> </div> </div> </div> </div> </blockquote> <br> <div> <br> </div> <span class="gmail_signature_prefix">-- </span><br> <div dir="ltr" class="gmail_signature"> <div dir="ltr"> <font><i>Pascal Paschos, Ph.D.</i></font> <div> <i style='font-family:"times new roman",serif'>OSG/PATh Collaboration Support </i><br> </div> <div> <font><i>Enrico Fermi Institute - </i></font><i style='font-family:"times new roman",serif'>University of Chicago</i> </div> <div><font><i>ph: 773-702-4679</i></font></div> </div> </div> </blockquote></div>

2 1

Re: [#76259] IGWN pool glidein logs
by Paschalis Paschos 26 Apr '24

26 Apr '24

Hi Jeff, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… there is got to be a process (cron) that rsyncs factory pilots logs for LIGO back to CIT. James, I see logs in /home/osg.factory/gfaclogs/sdsc/igwn from various sites - P. On Wed, 24 Apr at 1:20 PM , Clark, James A. <jaclark(a)caltech.edu> wrote: [EXTERNAL] – This message is from an external sender Hi Jeff, At the time, I was having trouble finding recent glidein logs for any entry and couldn't find correctly named directories for the LIGO-WA entries (capital C suffix, rather than lower case). I noticed some more recent logs yesterday, however, so let me dig around a bit more. From: Jeffrey Dost <support(a)opensciencegrid.org> Sent: Tuesday, April 23, 2024 12:14 PM To: james.clark(a)ligo.org <james.clark(a)ligo.org> Cc: paschos(a)uchicago.edu <paschos(a)uchicago.edu>; agraves10(a)unl.edu <agraves10(a)unl.edu>; jdost(a)ucsd.edu <jdost(a)ucsd.edu>; igwn-dhtc-ops(a)nikhef.nl <igwn-dhtc-ops(a)nikhef.nl> Subject: Re: [#76259] IGWN pool glidein logs Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… Can you please confirm which CIT entries you want me to investigate? looks like we have several but I see this one serving a lot of glideins: http://gfactory-2.opensciencegrid.org/factory/monitor/factoryStatus.html?en… Thanks, Jeff On Tue, 23 Apr at 11:07 AM , Paschalis Paschos <support(a)opensciencegrid.org> wrote: Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… I am not sure how do I check on that. I don't see any logs on CIT. Is it a pull request from CIT via a cron job or the factory actively sending those to you? On Fri, 19 Apr at 11:00 AM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender As noted in #76253 I'm having a hard time finding any recent factory glidein logs in the rsync'd directory at CIT. Can we confirm access is working ok? I know there was a mysterious issues for the ITB logs which seemed to resolve itself. -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

3 2

Re: [#76259] IGWN pool glidein logs
by Jeffrey Dost 24 Apr '24

24 Apr '24

Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… Can you please confirm which CIT entries you want me to investigate? looks like we have several but I see this one serving a lot of glideins: http://gfactory-2.opensciencegrid.org/factory/monitor/factoryStatus.html?en… Thanks, Jeff On Tue, 23 Apr at 11:07 AM , Paschalis Paschos <support(a)opensciencegrid.org> wrote: Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… I am not sure how do I check on that. I don't see any logs on CIT. Is it a pull request from CIT via a cron job or the factory actively sending those to you? On Fri, 19 Apr at 11:00 AM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender As noted in #76253 I'm having a hard time finding any recent factory glidein logs in the rsync'd directory at CIT. Can we confirm access is working ok? I know there was a mysterious issues for the ITB logs which seemed to resolve itself. -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

2 1

Re: [#76259] IGWN pool glidein logs
by Paschalis Paschos 23 Apr '24

23 Apr '24

Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… I am not sure how do I check on that. I don't see any logs on CIT. Is it a pull request from CIT via a cron job or the factory actively sending those to you? On Fri, 19 Apr at 11:00 AM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender As noted in #76253 I'm having a hard time finding any recent factory glidein logs in the rsync'd directory at CIT. Can we confirm access is working ok? I know there was a mysterious issues for the ITB logs which seemed to resolve itself. -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

1 0

Added as CC - [#76259] IGWN pool glidein logs
by OSG Support 19 Apr '24

19 Apr '24

James Clark submitted a new ticket to OSG User Documentation and requested that we copy you Ticket URL: https://support.opensciencegrid.org/support/tickets/public/4d0b40167271fc70… Ticket Description: [EXTERNAL] – This message is from an external sender As noted in #76253 I'm having a hard time finding any recent factory glidein logs in the rsync'd directory at CIT. Can we confirm access is working ok? I know there was a mysterious issues for the ITB logs which seemed to resolve itself. -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

1 0

Re: [#76253] LIGO-WA missing IGWN jobs (and missing glidein logs)
by Andrijauskas Fabio 19 Apr '24

19 Apr '24

Hi James, Re: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… We are checking it. Bests, On Thu, 18 Apr at 3:34 PM , James Clark <james.clark(a)ligo.org> wrote: [EXTERNAL] – This message is from an external sender It looks like the last user payload at LIGO-WA ended about a week ago with no activity and plenty of demand since. The frontend shows glideins cut out on the 4C entrypoint about the same time [1] and the 8C entrypoint earlier than that [2]. The weird (to me) thing is the factory seems to think there are 9 glideins running right now [3]. Can anyone remind me how to interpret that? Finally, I don't actually see *any* glidein logs from the new(er) LIGO-WA CE entries at the expected location @ CIT (and the most recent LIGO-CIT glidein log is from October 13 last year). Is the rsync from the production factory working ok? [1] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [2] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [3] http://gfactory-2.opensciencegrid.org/factory/monitor/factoryEntryStatusNow… -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

1 0

Added as CC - [#76253] LIGO-WA missing IGWN jobs (and missing glidein logs)
by OSG Support 18 Apr '24

18 Apr '24

James Clark submitted a new ticket to OSG User Documentation and requested that we copy you Ticket URL: https://support.opensciencegrid.org/support/tickets/public/f43c02bc6b243d50… Ticket Description: [EXTERNAL] – This message is from an external sender It looks like the last user payload at LIGO-WA ended about a week ago with no activity and plenty of demand since. The frontend shows glideins cut out on the 4C entrypoint about the same time [1] and the 8C entrypoint earlier than that [2]. The weird (to me) thing is the factory seems to think there are 9 glideins running right now [3]. Can anyone remind me how to interpret that? Finally, I don't actually see *any* glidein logs from the new(er) LIGO-WA CE entries at the expected location @ CIT (and the most recent LIGO-CIT glidein log is from October 13 last year). Is the rsync from the production factory working ok? [1] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [2] https://vo-frontend-igwn.igwn-prod.chtc.io/vofrontend/monitor/frontendStatu… [3] http://gfactory-2.opensciencegrid.org/factory/monitor/factoryEntryStatusNow… -- James Alexander Clark LIGO Laboratory California Institute of Technology email: james.clark(a)ligo.org Tel. (cell): 413-230-1412

1 0

2024

Igwn-dhtc-ops April 2024 ----- 2024 ----- May 2024 April 2024 March 2024 February 2024 January 2024

Igwn-dhtc-ops April 2024