Jenkins runners are broken

Development Testing and Automation
1

can someone please kick the runners on https://jenkins.plone.org back to life?

2

I temporarily disabled the broken node 2. It should be restarted and re-activated.

3

Node 2 seems to have run out of disk space.
Could someone cleanup or disable node 2, thanks!

Pinging @pbauer and/or @gforcada

4

they are both back to live

5

Seems like node2 has run out of disk (again): https://jenkins.plone.org/view/PLIPs/job/plip-dx-siteroot-3.7/18/console

6

Do we have some kind of monitoring on it? Can folks on the admin team help?

7

Look like 40GB are not enough anymore for a Jenkins node. Whats recommended? I can resize our disk-image.

[edit] see also Node 2 problems: Missing Python 3.8, polluted caches · Issue #261 · plone/jenkins.plone.org · GitHub

8

Can we double it?

9

I can. But if it fills up after some time b/c of some missing cleanup it does now help much.

10

The jenkins 5.2 on py3.8 pull request builder is currently segfaulting on every build. Is that a new thing or could it have to do with the issues mentioned here?

1 Like
11

I also got the Segfault: https://jenkins.plone.org/job/pull-request-5.2-3.8/28/

12

@gforcada @jensens who else can manage these? Let me know so we have more people who can respond?

13

I will try to have a look at night. More manpower is always welcome, nobody offered so far

14

@gforcada I did above but it got lost

I will DM you

15

I disabled mr.roboto to warn about python 3.8 pull request jobs so far until the segfault gets resolved (I bumped from 3.8.0beta1 to beta2, that might have been the problem), but unfortunately starts aligned so well, that in a few hours I will be on vacations for 10 days, so hopefully Plone can survive for the next 10 days without knowing if a pull request breaks on python 3.8 :cold_sweat:

4 Likes
16

Thanks, @gforcada!