NearForm hosted macOS 10.15 mac minis offline

[richardlau]

The three test and one release macOS 10.15 VMs on two mac minis we have hosted at NearForm are currently offline and cannot be ssh'ed into (No route to host). I've emailed Eamonn at NearForm. We still have the MacStadium hosted VMs so the CI should still be okay (albeit at reduced capacity).

[AshCripps]

These should be able to be recovered by logging on to the host machine itself and restarting the VMs inside VMWare Fusion.

[richardlau]

The release VM and two of the test VMs are now back.
The VM for test-nearform-macos10.15-x64-3 fails to start (shows Apple logo with a progress bar that moves but the VM stops before the bar fills).

[richardlau]

Going to admit defeat here. I have no idea how to work out what's up with that VM. Maybe it needs to be rebuilt. We have the other two NearForm-hosted macOS 10.15 test VM's and one orka-hosted for now (not sure what's up with the other orka macOS 10.15 VM) to get by.

[mhdawson]

Is it possible to just clone one of the others and then change the ip or something like that?

[UlisesGascon]

Quick update from my side, as I was digging into #2917 and I tried to connect all the nearform machines to collect some disk usage report. test-nearform-macos10.15-x64-1 and test-nearform-macos11.0-arm64-1 are fine, but test-nearform-macos10.15-x64-2 and test-nearform-macos10.15-x64-3 are appear offline (Network is unreachable )

SSH output

➜  ~ ssh test-nearform-macos10.15-x64-2
ssh: connect to host 83.147.191.71 port 22: Network is unreachable
➜  ~ ssh test-nearform-macos10.15-x64-3
ssh: connect to host 83.147.191.72 port 22: Network is unreachable

Ping output
This is the one for test-nearform-macos10.15-x64-1 all fine

➜  ~ ping test-nearform-macos10.15-x64-1
PING 83.147.191.70 (83.147.191.70): 56 data bytes
64 bytes from 83.147.191.70: icmp_seq=0 ttl=51 time=58.061 ms
64 bytes from 83.147.191.70: icmp_seq=1 ttl=51 time=54.390 ms
64 bytes from 83.147.191.70: icmp_seq=2 ttl=51 time=55.127 ms
^C
--- 83.147.191.70 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 54.390/55.859/58.061/1.586 ms

In the case of test-nearform-macos10.15-x64-2 seems Unreachable

➜  ~ ping test-nearform-macos10.15-x64-2
PING 83.147.191.71 (83.147.191.71): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
92 bytes from 78.137.157.66: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 054c   0 0000  32  01 aea7 192.168.1.51  83.147.191.71

92 bytes from 78.137.157.66: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 7134   0 0000  32  01 42bf 192.168.1.51  83.147.191.71

92 bytes from 78.137.157.66: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 30e5   0 0000  32  01 830e 192.168.1.51  83.147.191.71

Request timeout for icmp_seq 3
^C
--- 83.147.191.71 ping statistics ---

[richardlau]

Quick update from my side, as I was digging into #2917 and I tried to connect all the nearform machines to collect some disk usage report. test-nearform-macos10.15-x64-1 and test-nearform-macos11.0-arm64-1 are fine, but test-nearform-macos10.15-x64-2 and test-nearform-macos10.15-x64-3 are appear offline (Network is unreachable )

test-nearform-macos10.15-x64-2 and test-nearform-macos10.15-x64-3 are VMs on the same mac mini -- are you able to get into the host (details in build/infra-mac/nearform.md in the secrets repo)? test-nearform-macos10.15-x64-3 is broken but test-nearform-macos10.15-x64-2 may be recoverable (the VM may need restarting).

[targos]

I connected to NodeMacMini2 using VNC and started test-nearform-macos10.15-x64-2 (it was just stopped).

[targos]

Today I spent some time one the NearForm-hosted macs minis.

• NodeMacMini2
  • Updated to Catalina 10.15
  • Updated macOS on test-nearform-macos10.15-x64-2 (patch-level)
  • Recreated test-nearform-macos10.15-x64-3. See Recreate test-nearform-macos10.15-x64-3 #3124
• NodeMacMini
  • Updated to Catalina 10.15
  • Updated macOS on test-nearform-macos10.15-x64-1 and release-nearform-macos10.15-x64-1 (patch-level)

I think this issue can be closed now.