test-equinix_mnx-ubuntu1804-x64-1 is offline in Jenkins

[richardlau]

Refs: https://ci.nodejs.org/manage/computer/test-equinix_mnx-ubuntu1804-x64-1/

ubuntu@test-equinix--mnx-ubuntu1804-x64-1:~$ sudo systemctl status jenkins.service
● jenkins.service - Jenkins Slave
   Loaded: loaded (/lib/systemd/system/jenkins.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Mar 31 22:05:04 test-equinix--mnx-ubuntu1804-x64-1 systemd[1]: Dependency failed for Jenkins Slave.
Mar 31 22:05:04 test-equinix--mnx-ubuntu1804-x64-1 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
ubuntu@test-equinix--mnx-ubuntu1804-x64-1:~$

sudo systemctl restart jenkins.service fails

ubuntu@test-equinix--mnx-ubuntu1804-x64-1:~$ sudo systemctl restart jenkins.service
A dependency job for jenkins.service failed. See 'journalctl -xe' for details.
ubuntu@test-equinix--mnx-ubuntu1804-x64-1:~$

[richardlau]

sudo journalctl -xe has this error:

Apr 04 15:51:20 test-equinix--mnx-ubuntu1804-x64-1 systemd[1]: Starting Initial cloud-init job (pre-networking)...
-- Subject: Unit cloud-init-local.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit cloud-init-local.service has begun starting up.
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Cloud-init v. 23.1.1-0ubuntu0~18.04.1 running 'init-local' at Tue, 04 Apr 2023 15:51:21 +0000. Up 323258.26 seconds.
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: 2023-04-04 15:51:22,070 - util.py[WARNING]: failed stage init-local
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: failed run of stage init-local
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: ------------------------------------------------------------
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Traceback (most recent call last):
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 767, in status_wrapper
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     ret = functor(name, args)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 433, in main_init
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     init.apply_network_config(bring_up=bring_up_interfaces)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 939, in apply_network_config
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     return self.distro.apply_network_config(
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 278, in apply_network_config
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     self._write_network_state(network_state, renderer)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 142, in _write_network_state
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     return super()._write_network_state(*args, **kwargs)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 167, in _write_network_state
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     renderer.render_network_state(network_state)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/net/netplan.py", line 298, in render_network_state
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     self._netplan_generate(run=self._postcmds)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/net/netplan.py", line 305, in _netplan_generate
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     subp.subp(self.NETPLAN_GENERATE, capture=True)
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:   File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in subp
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:     raise ProcessExecutionError(
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Command: ['netplan', 'generate']
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Exit code: 1
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Reason: -
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Stdout:
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: Stderr: Traceback (most recent call last):
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:           File "/usr/sbin/netplan", line 20, in <module>
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:             from netplan import Netplan
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:           File "/usr/share/netplan/netplan/__init__.py", line 18, in <module>
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:             from netplan.cli.core import Netplan
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:           File "/usr/share/netplan/netplan/cli/core.py", line 24, in <module>
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:             import netplan.cli.utils as utils
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:           File "/usr/share/netplan/netplan/cli/utils.py", line 25, in <module>
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:             import netifaces
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]:         ModuleNotFoundError: No module named 'netifaces'
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 cloud-init[11396]: ------------------------------------------------------------
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 systemd[1]: cloud-init-local.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 systemd[1]: cloud-init-local.service: Failed with result 'exit-code'.
Apr 04 15:51:22 test-equinix--mnx-ubuntu1804-x64-1 systemd[1]: Failed to start Initial cloud-init job (pre-networking).

I'm assuming that this is somehow a dependency of jenkins.service (if I attempt to restart the jenkins.service I get a fresh instance of the cloud-init failure).

[richardlau]

Confused as pip3 shows that netifaces is installed. 😞

[richardlau]

So this is something to do with the Python 3.8 update from #3272 (cc @MoLow ).
I changed the symlink for /usr/bin/python3 to point to /usr/bin/python3.6 and jenkins.service/cloud-init now start up.

[richardlau]

I imagine changing the python3 symlink back to 3.6 has probably broken tap2junit on this machine, but that rerunning the playbook would put us back in the Jenkns-won't-start state we were in that this issue was raised for. Maybe we'll need to figure out a way to keep the python3 symlink pointing at 3.6 but to still be able to use 3.8 for tap2junit.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.