Follow up: test-macos10.15-x64-2 offline #3218

richardlau · 2023-03-13T13:02:21Z

All of our macOS 10.14 VMs (all in Orka) are offline.

Initially it was because the agents needed to be updated after #3176 -- I've done that (via the upgrade-jar playbook), but now the agents are refusing to start as they need newer Java. It looks like these are still running Java 8.

richardlau · 2023-03-13T13:08:08Z

It also looks like test-orka-macos10.14-x64-2 is unreachable

test-orka-macos10.14-x64-1 : ok=3    changed=2    unreachable=0    failed=0    skipped=5    rescued=0    ignored=0
test-orka-macos10.14-x64-2 : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
test-orka-macos10.14-x64-3 : ok=3    changed=2    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

richardlau · 2023-03-13T13:18:45Z

Looks like the orka 10.15 VMs are also offline. I logged into test-orka-macos10.15-x64-1 and that appears to be running Java 8 as well.

% java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.242-b08, mixed mode)

richardlau · 2023-03-13T13:22:08Z

These were updated to Java 17 previously #3030 (comment) / #3085.

richardlau · 2023-03-13T13:24:48Z

These were updated to Java 17 previously #3030 (comment) / #3085.

We rebuilt these VMs in January #3112.

sxa · 2023-03-13T14:14:09Z

Nearform macs are running with Eclipse Temurin 17 with /usr/bin/java pointing there.

administrator@test-nearform-macos10 ~iojs % java -version
openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+10 (build 17.0.6+10)
OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)
administrator@test-nearform-macos10 ~iojs % ls -l /usr/bin/java
lrwxr-xr-x  1 root  wheel  74 18 Dec 13:41 /usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java
administrator@test-nearform-macos10 ~iojs %

sxa · 2023-03-13T14:17:22Z

So steps required to fix the orka systems:

Install a Temurin 17 (The location being used by the nearform makes still contains JDK8 on the orka systems)
Ensure that /usr/bin/java points there (or set the path accordingly in the start.sh script
Update the agent.jar (was 4.2, needs 4.7)
Fix the start.sh so that SECRET is replaced accordingly for the machine (and update PATH, and name of agent.jar if required)

We should check the correct upgrade path should be for java and whether it makes sense to be able to ensure the playbooks can do that.

The above steps have allowed test-orka-macos10.15-x64-1 to come back online, although we will need to ensure that this does not conflict with anything in the playbooks. Note that I've left the old slave.jar on the machine, but it is now using the new agent.jar downloaded from the server.

richardlau · 2023-03-13T14:56:35Z

@sxa Does the playbook not work?

sxa · 2023-03-13T15:09:40Z

I haven't tried running - was just looking into the cause :-)
From your comment earlier should the two online 10.14 machines be all "playbooked up" now? -1 and -3 don't seem to have come back yet.

richardlau · 2023-03-13T15:12:49Z

@sxa I only ran the playbook to update the agent, not the full playbook for Jenkins workers.

richardlau · 2023-03-17T15:09:21Z

@UlisesGascon is this something you could take a look at?

UlisesGascon · 2023-03-18T14:17:14Z

Yes, let me see 👍

UlisesGascon · 2023-03-19T10:21:51Z

Current situation

I managed to manually recover:

test-orka-macos10.14-x64-1 (current Java version: 17.0.6)
test-orka-macos10.14-x64-3 (current Java version: 19.0.2)

The recovering process I followed was very similar to #3218 (comment)

Before

After

Next Steps

Solve connectivity issues with test-orka-macos11-x64-2, test-orka-macos11-x64-1, test-orka-macos10.14-x64-2
~~- [ ] Solve configuration issues with test-orka-macos10.15-x64-2 as the Java version and Jenkins settings are fine but the machine is not connecting to Jekins~~
~~- [ ] Free Disk space for test-orka-macos11-x64-1~~

UlisesGascon · 2023-03-19T13:13:15Z

Current status

I managed to manually recover:

test-orka-macos11-x64-2 (current Java version: 19.0.2)

Solved connectivity issues for:

test-orka-macos11-x64-2 (current Java version: 19.0.2)
test-orka-macos11-x64-1 (current Java version: 19.0.2)
test-orka-macos10.14-x64-2 (current Java version: 17.0.5)
test-orka-macos10.15-x64-2 (current Java version: 19.0.2)

Before

After

Current challenges

1. Release machine(s)

The machine test-orka-macos11-x64-1 connected to Jenkins is not the actual test-orka-macos11-x64-1, probably this machine is release-macos1015-x64-1. In case that release-macos1015-x64-1 is not the machine that is kidnapping the agent, then worth to check manually release-macos11-x64-1 from #3185 as it is not in the inventory.

I can't access to these machines via SSH as they are release ones, but we will need to check the connectivity and Java version too. Maybe @targos can help here.

Important

The machine test-orka-macos11-x64-1 should be fine and full configured including Jenkins agent, but I disabled that node in Jenkins until is clear that other machine is not kidnapping the agent

1. Jenkins configuration

The machines test-orka-macos10.15-x64-2 and test-orka-macos10.14-x64-2 are up and running but not connecting to Jenkins as Nodes

test-orka-macos10.15-x64-2 Errors:

Waiting 10 seconds before retry
Failing to obtain https://ci.nodejs.org/computer/test-macstadium-macos10.15-x64-1/slave-agent.jnlp?encrypt=true
java.io.IOException: Failed to load https://ci.nodejs.org/computer/test-macstadium-macos10.15-x64-1/slave-agent.jnlp?encrypt=true: 404 Not Found
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:521)
	at hudson.remoting.Launcher.run(Launcher.java:347)
	at hudson.remoting.Launcher.main(Launcher.java:298)
Waiting 10 seconds before retry
Failing to obtain https://ci.nodejs.org/computer/test-macstadium-macos10.15-x64-1/slave-agent.jnlp?encrypt=true
java.io.IOException: Failed to load https://ci.nodejs.org/computer/test-macstadium-macos10.15-x64-1/slave-agent.jnlp?encrypt=true: 404 Not Found
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:521)
	at huds^C

Not sure why is pointing to Jnlp *-macstadium-* when the start.sh is setup to the right address.

...
PATH="/usr/local/opt/ccache/libexec:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" java -Xmx128m \
    -jar /Users/iojs/slave.jar -secret REDACTED \
    -jnlpUrl https://ci.nodejs.org/computer/test-orka-macos10.15-x64-2/slave-agent.jnlp

I tried to use the new agent and the old one with different settings and I got no luck, I tried to mimic the settings from other machines and no luck.

test-orka-macos10.14-x64-2 Errors:

INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: test-orka-macos10.14-x64-1 is already connected to this controller. Rejecting this connection.
	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
	at hudson.remoting.Engine.innerRun(Engine.java:740)
	at hudson.remoting.Engine.run(Engine.java:518)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: test-orka-macos10.14-x64-1 is already connected to this controller. Rejecting this connection.
	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.nio.channels.ClosedChannelException
		... 7 more

I checked the configuration and all seems correct including the secrets:

...

export PATH="/usr/local/opt/python3/Frameworks/Python.framework/Versions/Current/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
export PATH="$(brew --prefix)/opt/ccache/libexec:$PATH"
java -Xmx128m \
    -jar /Users/iojs/agent.jar -secret REDACTED \
    -jnlpUrl https://ci.nodejs.org/manage/computer/test-orka-macos10.14-x64-2/jenkins-agent.jnlp

Next steps

I am happy to provide support for the next steps but I will need help with Jenkins settings and Release access (cc: @nodejs/build ).

Check the Jenkins settings and Java Version for the Release machines in the Release Jenkins CI
Patch the Jenkins agent for test-orka-macos10.14-x64-2
Patch the Jenkins agent for test-orka-macos10.15-x64-2

targos · 2023-03-20T17:29:26Z

I'm unable to SSH into release-macos1015-x64-1 (it doesn't accept the release ssh key and ends up asking for a password). Same for -macos11-x64-1

UlisesGascon · 2023-03-20T17:38:15Z

@targos if you want I can try to redeploy the release-macos1015-x64-1 vm, maybe that solves the ssh key issue. But the release-macos11-x64-1 hasn't yet been configured with ansible or ssh key access (is a fresh vm) if remember well from the pending actions in #3185.

richardlau · 2023-03-23T19:08:14Z

Just a thought, but didn't we snapshot the VMs back in January #3112 (comment)?

UlisesGascon · 2023-03-23T19:37:06Z

Yes, Should I try to restore the release ones from the snapshots? Maybe this restore the machines. Great point @richardlau

richardlau · 2023-03-30T12:37:29Z

If we have snapshots I think it would be worth trying.

FWIW test-orka-macos10.15-x64-1 (set up in #3218 (comment)) is failing v14.x builds, e.g. https://ci.nodejs.org/job/node-test-commit-osx/51258/nodes=osx1015/console

12:14:16 No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.
12:14:16 
12:14:16 No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.
12:14:16 
12:14:16 No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.
12:14:16 
12:14:16 gyp: No Xcode or CLT version detected!

which is https://github.com/nodejs/build/blob/main/ansible/MANUAL_STEPS.md#install-command-line-tools-for-xcode
Hopefully the snapshots contain a properly configured command line tools for xcode.

UlisesGascon · 2023-05-07T15:40:20Z

@richardlau I will try to restore test-orka-macos10.15-x64-2 from snapshot and then close this issue.

As today, failing v14.x builds and macos 10.14 related issues are not relevant 🤔

UlisesGascon · 2023-05-08T19:53:08Z

I tried to use the images macos1015-x64-2_11012023 and nodejs-test-1015.img to restore the VM and It didn't work at all, the macos vm didn't allow any SSH or VNC Connection.

I will try to see if by generating a new image from vm macos1015-x64-1 can solve the problem and then I will re-ansible the vm.

UlisesGascon · 2023-07-07T10:00:17Z

I created a new VM using nodejs-test-1015.img as the base image and the connectivity is down even using VNC inside the VPN. I will open a ticket to the support team

UlisesGascon · 2023-07-07T10:55:32Z

Ticket opened with support: SERVICE-164961. I will keep tracking the communications

UlisesGascon · 2023-07-07T17:59:06Z

The response:

Hi,

This is due to an issue regarding Docker images being removed from some of our source control code bases. We can fix this by upgrading your environment. We have moved our internal routes to a different repository.

Would you like me to conduct an upgrade today?

--

Anahit

I think this is a clear yes. Maybe this solve #3415 as well 🤔. Last upgrade required us to re-deploy the VMs in the correct spots, and re-ansible also if I remember well

What do you think @nodejs/build ?

UlisesGascon · 2023-07-11T07:59:27Z

The VM after the upgrade didn't connect, so I asked for the additional support.

UlisesGascon · 2023-07-13T17:54:12Z

**test-macos10.15-x64-2 is back 🥳 **

I purged the previous VM and regenerated the new VM following the recommendation from the support team. I re-ansibled the machine.

I see that this VM is named with 1015 in its name, is it safe to assume this is running MacOS 10.15? I see that this VM is launched with Net-Boost & IO-Boost enabled. While IO-Boost is supported on MacOS 10.14.5 or later, Net-Boost requires MacOS 11.0 or later.

If Net-Boost is enabled on a VM running an older version of MacOS, networking will not function properly. When testing within the cluster’s network, I was unable to connect either. Which leads me to believe this is the root cause.

Try recreating this VM’s config with Net Boost disabled, that should resolve the issue for this VM. I’ve attached a snippet showing your launched VMs and the configurations. I attempted to SSH into the first in the list which seems to also be on 10.15, and was able to get a response.

Previous

Current

richardlau added platform:osx incident labels Mar 13, 2023

richardlau changed the title ~~orka macOS 10.14 VMs offline~~ orka macOS 10.14 and 10.15 VMs offline Mar 13, 2023

UlisesGascon self-assigned this Mar 18, 2023

UlisesGascon removed their assignment Mar 19, 2023

richardlau closed this as completed Mar 30, 2023

richardlau reopened this Mar 30, 2023

UlisesGascon self-assigned this May 7, 2023

UlisesGascon mentioned this issue May 10, 2023

Orka allocated storage Alert #3257

Closed

UlisesGascon changed the title ~~orka macOS 10.14 and 10.15 VMs offline~~ Follow up: test-macos10.15-x64-2 offline Jul 11, 2023

UlisesGascon mentioned this issue Jul 11, 2023

Orka node (macpro-6) replacement #3415

Closed

UlisesGascon closed this as completed Jul 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Follow up: test-macos10.15-x64-2 offline #3218

Follow up: test-macos10.15-x64-2 offline #3218

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

sxa commented Mar 13, 2023

sxa commented Mar 13, 2023 •

edited

Loading

richardlau commented Mar 13, 2023

sxa commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 17, 2023

UlisesGascon commented Mar 18, 2023

UlisesGascon commented Mar 19, 2023 •

edited

Loading

UlisesGascon commented Mar 19, 2023

targos commented Mar 20, 2023

UlisesGascon commented Mar 20, 2023

richardlau commented Mar 23, 2023

UlisesGascon commented Mar 23, 2023

richardlau commented Mar 30, 2023

UlisesGascon commented May 7, 2023

UlisesGascon commented May 8, 2023

UlisesGascon commented Jul 7, 2023

UlisesGascon commented Jul 7, 2023

UlisesGascon commented Jul 7, 2023 •

edited

Loading

UlisesGascon commented Jul 11, 2023

UlisesGascon commented Jul 13, 2023

Follow up: test-macos10.15-x64-2 offline #3218

Follow up: test-macos10.15-x64-2 offline #3218

Comments

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 13, 2023

sxa commented Mar 13, 2023

sxa commented Mar 13, 2023 • edited Loading

richardlau commented Mar 13, 2023

sxa commented Mar 13, 2023

richardlau commented Mar 13, 2023

richardlau commented Mar 17, 2023

UlisesGascon commented Mar 18, 2023

UlisesGascon commented Mar 19, 2023 • edited Loading

Current situation

Next Steps

UlisesGascon commented Mar 19, 2023

Current status

Current challenges

1. Release machine(s)

1. Jenkins configuration

Next steps

targos commented Mar 20, 2023

UlisesGascon commented Mar 20, 2023

richardlau commented Mar 23, 2023

UlisesGascon commented Mar 23, 2023

richardlau commented Mar 30, 2023

UlisesGascon commented May 7, 2023

UlisesGascon commented May 8, 2023

UlisesGascon commented Jul 7, 2023

UlisesGascon commented Jul 7, 2023

UlisesGascon commented Jul 7, 2023 • edited Loading

UlisesGascon commented Jul 11, 2023

UlisesGascon commented Jul 13, 2023

sxa commented Mar 13, 2023 •

edited

Loading

UlisesGascon commented Mar 19, 2023 •

edited

Loading

UlisesGascon commented Jul 7, 2023 •

edited

Loading