Hello,
just recently, users on ribble get from time to time the error message like: "fork: Resource temporarily unavailable". I also get this error when ssh-ing to ribble.
This is quite abnormal. Could you please have a look? Thanks!
Cheers, Hurng
Hi Hurng, all,
Ribble is intended as a login/log-through system, and not intended for running analysis. Over the past months, ribble was frequently loaded by analysis jobs and completely unresponsive to anything else, so Ton and the Helpdesk put in some helpful limits to discourage such activity. The intention is to log through to your own workstation or a group server and do your analysis there. So ulimit -a now has
max user processes (-u) 16
and no more than 4 simultaneous logins on this host. At least now ribble is usable again as a login server ;-) This was in /etc/motd for the last few weeks, but you just missed it for a very good reason :-) The max-user processes of course triggers the fork(2) error.
The help desk should have more info.
Cheers, DavidG.
PS: you still have process running that is eating 100% CPU. I have seen this happen a lot of times, as apparently the Atlas CMT version has a bug that causes this process: sh -c /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/cmt_gcc_version.sh | /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/cmt_filter3_version.sh to hand and eat CPU cycles indefinitely. The helpdesk already killed like processes from many Atlas users :-) Any idea who CMT exhibits that behaviour?
Hurng-Chun LEE wrote:
Hello,
just recently, users on ribble get from time to time the error message like: "fork: Resource temporarily unavailable". I also get this error when ssh-ing to ribble.
This is quite abnormal. Could you please have a look? Thanks!
Cheers, Hurng _______________________________________________ Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid
Thanks David for the explanation.
I also just discovered this funny CMT problem ... I am just trying to see how to reproduce this problem in a simply way.
On the other hand, would it be also used as the front-end of the stoomboot? as the pbs client commands are only available on ribble so I suppose it's the only place for users to submit analysis jobs to stoomboot.
In order to run analysis on stoomboot through Ganga, some preparation work needs to be done within Ganga before job submission (e.g. the cmt setup stuff). What funny is that I can do a cmt setup in a shell; but an equivalent command executed through Ganga gets the fork error. Apparently Ganga does some intelligence in making a system call, but i don't believe that Ganga needs to fork so many processes to just make a system call at the end (so that it runs out of the allowed processes). I still have no clue if this comes from the Python forking or the Ganga itself.
I will do more investigation on this.
Cheers, Hurng
On Nov 6, 2008, at 5:30 PM, David Groep wrote:
Hi Hurng, all,
Ribble is intended as a login/log-through system, and not intended for running analysis. Over the past months, ribble was frequently loaded by analysis jobs and completely unresponsive to anything else, so Ton and the Helpdesk put in some helpful limits to discourage such activity. The intention is to log through to your own workstation or a group server and do your analysis there. So ulimit -a now has
max user processes (-u) 16
and no more than 4 simultaneous logins on this host. At least now ribble is usable again as a login server ;-) This was in /etc/motd for the last few weeks, but you just missed it for a very good reason :-) The max-user processes of course triggers the fork(2) error.
The help desk should have more info.
Cheers, DavidG.
PS: you still have process running that is eating 100% CPU. I have seen this happen a lot of times, as apparently the Atlas CMT version has a bug that causes this process: sh -c /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/ cmt_gcc_version.sh | /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/ cmt_filter3_version.sh to hand and eat CPU cycles indefinitely. The helpdesk already killed like processes from many Atlas users :-) Any idea who CMT exhibits that behaviour?
Hurng-Chun LEE wrote:
Hello,
just recently, users on ribble get from time to time the error message like: "fork: Resource temporarily unavailable". I also get this error when ssh-ing to ribble.
This is quite abnormal. Could you please have a look? Thanks!
Cheers, Hurng _______________________________________________ Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid
-- David Groep
** Nikhef, Dutch National Institute for Sub-atomic Physics,PDP/Grid group ** ** Room: H1.56 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
Hi Hurng,
There is no reason for the pbs commands to only work on ribble. Maybe Ton can install them on all desktops, so that users can more easily submit to stoomboot from their own systems. That installation should be trivial.
Cheers, DavidG.
Hurng-Chun LEE wrote:
Thanks David for the explanation.
I also just discovered this funny CMT problem ... I am just trying to see how to reproduce this problem in a simply way.
On the other hand, would it be also used as the front-end of the stoomboot? as the pbs client commands are only available on ribble so I suppose it's the only place for users to submit analysis jobs to stoomboot.
In order to run analysis on stoomboot through Ganga, some preparation work needs to be done within Ganga before job submission (e.g. the cmt setup stuff). What funny is that I can do a cmt setup in a shell; but an equivalent command executed through Ganga gets the fork error. Apparently Ganga does some intelligence in making a system call, but i don't believe that Ganga needs to fork so many processes to just make a system call at the end (so that it runs out of the allowed processes). I still have no clue if this comes from the Python forking or the Ganga itself.
I will do more investigation on this.
Cheers, Hurng
On Nov 6, 2008, at 5:30 PM, David Groep wrote:
Hi Hurng, all,
Ribble is intended as a login/log-through system, and not intended for running analysis. Over the past months, ribble was frequently loaded by analysis jobs and completely unresponsive to anything else, so Ton and the Helpdesk put in some helpful limits to discourage such activity. The intention is to log through to your own workstation or a group server and do your analysis there. So ulimit -a now has
max user processes (-u) 16
and no more than 4 simultaneous logins on this host. At least now ribble is usable again as a login server ;-) This was in /etc/motd for the last few weeks, but you just missed it for a very good reason :-) The max-user processes of course triggers the fork(2) error.
The help desk should have more info.
Cheers, DavidG.PS: you still have process running that is eating 100% CPU. I have seen this happen a lot of times, as apparently the Atlas CMT version has a bug that causes this process: sh -c /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/cmt_gcc_version.sh |
/data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/cmt_filter3_version.sh to hand and eat CPU cycles indefinitely. The helpdesk already killed like processes from many Atlas users :-) Any idea who CMT exhibits that behaviour?
Hurng-Chun LEE wrote:
Hello,
just recently, users on ribble get from time to time the error message like: "fork: Resource temporarily unavailable". I also get this error when ssh-ing to ribble.
This is quite abnormal. Could you please have a look? Thanks!
Cheers, Hurng _______________________________________________ Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid
-- David Groep
** Nikhef, Dutch National Institute for Sub-atomic Physics,PDP/Grid group ** ** Room: H1.56 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
Hi,
that would be also very helpful ... indeed, some users are using elelxx where the qsub command is not available.
Cheers, Hurng
On Nov 6, 2008, at 6:00 PM, David Groep wrote:
Hi Hurng,
There is no reason for the pbs commands to only work on ribble. Maybe Ton can install them on all desktops, so that users can more easily submit to stoomboot from their own systems. That installation should be trivial.
Cheers, DavidG.
Hurng-Chun LEE wrote:
Thanks David for the explanation.
I also just discovered this funny CMT problem ... I am just trying to see how to reproduce this problem in a simply way.
On the other hand, would it be also used as the front-end of the stoomboot? as the pbs client commands are only available on ribble so I suppose it's the only place for users to submit analysis jobs to stoomboot.
In order to run analysis on stoomboot through Ganga, some preparation work needs to be done within Ganga before job submission (e.g. the cmt setup stuff). What funny is that I can do a cmt setup in a shell; but an equivalent command executed through Ganga gets the fork error. Apparently Ganga does some intelligence in making a system call, but i don't believe that Ganga needs to fork so many processes to just make a system call at the end (so that it runs out of the allowed processes). I still have no clue if this comes from the Python forking or the Ganga itself.
I will do more investigation on this.
Cheers, Hurng
On Nov 6, 2008, at 5:30 PM, David Groep wrote:
Hi Hurng, all,
Ribble is intended as a login/log-through system, and not intended for running analysis. Over the past months, ribble was frequently loaded by analysis jobs and completely unresponsive to anything else, so Ton and the Helpdesk put in some helpful limits to discourage such activity. The intention is to log through to your own workstation or a group server and do your analysis there. So ulimit -a now has
max user processes (-u) 16
and no more than 4 simultaneous logins on this host. At least now ribble is usable again as a login server ;-) This was in /etc/motd for the last few weeks, but you just missed it for a very good reason :-) The max-user processes of course triggers the fork(2) error.
The help desk should have more info.
Cheers, DavidG.
PS: you still have process running that is eating 100% CPU. I have seen this happen a lot of times, as apparently the Atlas CMT version has a bug that causes this process: sh -c /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/ cmt_gcc_version.sh |
/data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/ cmt_filter3_version.sh to hand and eat CPU cycles indefinitely. The helpdesk already killed like processes from many Atlas users :-) Any idea who CMT exhibits that behaviour?
Hurng-Chun LEE wrote:
Hello,
just recently, users on ribble get from time to time the error message like: "fork: Resource temporarily unavailable". I also get this error when ssh-ing to ribble.
This is quite abnormal. Could you please have a look? Thanks!
Cheers, Hurng _______________________________________________ Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid
-- David Groep
** Nikhef, Dutch National Institute for Sub-atomic Physics,PDP/Grid group ** ** Room: H1.56 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
-- David Groep
** Nikhef, Dutch National Institute for Sub-atomic Physics,PDP/Grid group ** ** Room: H1.56 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
Hi,
On Nov 6, 2008, at 6:04 PM, Hurng-Chun LEE wrote:
Hi,
that would be also very helpful ... indeed, some users are using elelxx where the qsub command is not available.
just to clarify ... elel1 doesn't have qsub but elel22 has.
Hurng
Cheers, Hurng
On Nov 6, 2008, at 6:00 PM, David Groep wrote:
Hi Hurng,
There is no reason for the pbs commands to only work on ribble. Maybe Ton can install them on all desktops, so that users can more easily submit to stoomboot from their own systems. That installation should be trivial.
Cheers, DavidG.
Hurng-Chun LEE wrote:
Thanks David for the explanation.
I also just discovered this funny CMT problem ... I am just trying to see how to reproduce this problem in a simply way.
On the other hand, would it be also used as the front-end of the stoomboot? as the pbs client commands are only available on ribble so I suppose it's the only place for users to submit analysis jobs to stoomboot.
In order to run analysis on stoomboot through Ganga, some preparation work needs to be done within Ganga before job submission (e.g. the cmt setup stuff). What funny is that I can do a cmt setup in a shell; but an equivalent command executed through Ganga gets the fork error. Apparently Ganga does some intelligence in making a system call, but i don't believe that Ganga needs to fork so many processes to just make a system call at the end (so that it runs out of the allowed processes). I still have no clue if this comes from the Python forking or the Ganga itself.
I will do more investigation on this.
Cheers, Hurng
On Nov 6, 2008, at 5:30 PM, David Groep wrote:
Hi Hurng, all,
Ribble is intended as a login/log-through system, and not intended for running analysis. Over the past months, ribble was frequently loaded by analysis jobs and completely unresponsive to anything else, so Ton and the Helpdesk put in some helpful limits to discourage such activity. The intention is to log through to your own workstation or a group server and do your analysis there. So ulimit -a now has
max user processes (-u) 16
and no more than 4 simultaneous logins on this host. At least now ribble is usable again as a login server ;-) This was in /etc/motd for the last few weeks, but you just missed it for a very good reason :-) The max-user processes of course triggers the fork(2) error.
The help desk should have more info.
Cheers, DavidG.
PS: you still have process running that is eating 100% CPU. I have seen this happen a lot of times, as apparently the Atlas CMT version has a bug that causes this process: sh -c /data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/ cmt_gcc_version.sh |
/data/atlas/offline/14.2.20/CMT/v1r20p20080222/mgr/ cmt_filter3_version.sh to hand and eat CPU cycles indefinitely. The helpdesk already killed like processes from many Atlas users :-) Any idea who CMT exhibits that behaviour?
Hurng-Chun LEE wrote:
Hello,
just recently, users on ribble get from time to time the error message like: "fork: Resource temporarily unavailable". I also get this error when ssh-ing to ribble.
This is quite abnormal. Could you please have a look? Thanks!
Cheers, Hurng _______________________________________________ Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid
-- David Groep
** Nikhef, Dutch National Institute for Sub-atomic Physics,PDP/Grid group ** ** Room: H1.56 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
-- David Groep
** Nikhef, Dutch National Institute for Sub-atomic Physics,PDP/Grid group ** ** Room: H1.56 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
Ct-grid mailing list Ct-grid@nikhef.nl https://mailman.nikhef.nl/cgi-bin/listinfo/ct-grid
-
David Groep -
Hurng-Chun LEE