macos11.0 release hosts are both offline

[targos]

release-nearform-macos11.0-arm64-1 is offline and its IP address is unreachable
release-macstadium-macos11.0-arm64-1 is offline and refuses ssh connections

[richardlau]

I've sent an email to our NearForm contacts regarding the NearForm machine. (FYI @efrisby).

[mhdawson]

@UlisesGascon is

release-macstadium-macos11.0-arm64-1 is offline and #3084

Something you can help with?

[UlisesGascon]

I can help and see if I can restore the machine, but I will need some support to re-ansible as we did for release-orka-10.15-x64-1 in the past

[targos]

Before re-ansibling, maybe there is a secondary way to access the machine from Macstadium's infra/network?

[UlisesGascon]

I will check the machine and share the status before I do a destructive or non-reversible action 👍

[UlisesGascon]

Here is an update on what I have tried. Thanks to @targos for the great support:

• I reviewed the internal documents about MacStadium but did not find a new way to access the machines.
• The MacStadium web UI does not offer any option to access the machine, but the stats confirmed that they are connected and running.
• I hard rebooted all the bare metal machines (including the test ones, macstadium macos11 immediate closed connection  #3084), but even after the machines were back online, none of them had improved connectivity.
• I tried to VNC connect with them following an official tutorial (from 10 years ago) but had no luck, possibly because the username and password were not the same as in the internal documentation or the VNC Server is disabled (as expected).
• I also reviewed the Bare Metal official documentation but had no luck.
• I attempted to communicate to the machines from the Orka VPN but was unsuccessful.

I see two options: we can raise a ticket and ask for support, pointing to this issue, or we can add one or several additional release VMs in the Orka platform using the backups I made. We would only need to register the new machines in the inventory and re-Ansible them.

[richardlau]

I see two options: we can raise a ticket and ask for support

Am I misremembering or did @AshCripps try that before (or perhaps it was a different VM)?

[UlisesGascon]

In the meantime I will add a new macos11 in Orka for releases

[AshCripps]

I see two options: we can raise a ticket and ask for support

Am I misremembering or did @AshCripps try that before (or perhaps it was a different VM)?

Yes it was a very similar issue and took some effort to get them to acknowledge it - should still be in the system somewhere of what I asked for.

[richardlau]

In the meantime I will add a new macos11 in Orka for releases

Ah, forgot that these are arm64 machines/VMs and therefore not in Orka.

[UlisesGascon]

🎉 Good News! I created a new bare metal instance, and the machine is up and running.

So... here is the plan:

• [ ] I will create a PR to replace the IP for release-macstadium-macos11.0-arm64-1 in the inventory, and I will ask you to reansible the machine for releases and to include it in the secret repo. Current status: Chore: IP replacement for macstadium release arm machine #3187
• [ ] I will create 2 new machines, reansible them to replace the test-macstadium--arm64-. Current status: IP replacement for test-macstadium-macos11.0-arm64-3 #3189 and IP replacement for test-macstadium-macos11.0-arm64-4 #3188
• [ ] Once we are happy with the new machines I will remove the old ones from macstadium

So... here is the NEW plan

• I will ask support to restore the credentials for the arm machines (ticket)
• Check that SSH is working in all the machines with the new credentials
• Add build SSH key to test-macstadium-macos11.0-arm64-3
• Re-ansible test-macstadium-macos11.0-arm64-3
• Re-ansible test-macstadium-macos11.0-arm64-4
• I will ask release team to reansible the release one (release-macstadium-macos11.0-arm64-1)

[richardlau]

FWIW the Nearform machine is now back online. (Thank you @efrisby.)

[richardlau]

I've reansibled the release and two test macos11 arm64 macstadium machines.
Currently running a release rebuild on today's nightly (osx11-tar happened to fail) with just osx11-tar and osx11-pkg: https://ci-release.nodejs.org/job/iojs+release/9138/

[richardlau]

Currently running a release rebuild on today's nightly (osx11-tar happened to fail) with just osx11-tar and osx11-pkg: https://ci-release.nodejs.org/job/iojs+release/9138/

Build passed. Release machine looks good.

[UlisesGascon]

Thanks for the conformation @richardlau. I will close the issue then 😉