Jenkins runners are broken

can someone please kick the runners on https://jenkins.plone.org back to life?

I temporarily disabled the broken node 2. It should be restarted and re-activated.

Node 2 seems to have run out of disk space.
Could someone cleanup or disable node 2, thanks!

Pinging @pbauer and/or @gforcada

they are both back to live

Seems like node2 has run out of disk (again): https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.7/18/console

Do we have some kind of monitoring on it? Can folks on the admin team help?

Look like 40GB are not enough anymore for a Jenkins node. Whats recommended? I can resize our disk-image.

[edit] see also Node 2 problems: Missing Python 3.8, polluted caches · Issue #261 · plone/jenkins.plone.org · GitHub

Can we double it?

I can. But if it fills up after some time b/c of some missing cleanup it does now help much.

The jenkins 5.2 on py3.8 pull request builder is currently segfaulting on every build. Is that a new thing or could it have to do with the issues mentioned here?

1 Like

I also got the Segfault: https://jenkins.plone.org/job/pull-request-5.2-3.8/28/

@gforcada @jensens who else can manage these? Let me know so we have more people who can respond?

I will try to have a look at night. More manpower is always welcome, nobody offered so far

@gforcada I did above but it got lost

I will DM you

I disabled mr.roboto to warn about python 3.8 pull request jobs so far until the segfault gets resolved (I bumped from 3.8.0beta1 to beta2, that might have been the problem), but unfortunately starts aligned so well, that in a few hours I will be on vacations for 10 days, so hopefully Plone can survive for the next 10 days without knowing if a pull request breaks on python 3.8 :cold_sweat:

4 Likes

Thanks, @gforcada!