Replacing machines from Joyent data center #2552

mhdawson · 2021-02-28T00:18:07Z

We found out late Friday that the machines in the Joyent data center needed to be moved by next week. These have been moved:

ip	hostname	changes	halted old?
147.75.88.51	test-joyent-ubuntu1804-x64-1	updated firewall on ci	yes
147.75.88.54	test-joyent-smartos18-x64-3	updated firewall on ci	yes
147.75.88.55	test-joyent-smartos18-x64-4	updated firewall on ci	yes
147.75.88.53	release-joyent-smartos18-x64-2	updated firewall on ci-release	yes
147.75.88.56	release-joyent-ubuntu1804_docker-x64-1	updated firewall on ci-release	yes
147.75.88.58	test-joyent-ubuntu1604_arm_cross-x64-1	updated firewall on ci	yes
147.75.88.52	release-joyent-ubuntu1604_arm_cross-x64-1	updated firewall on ci-release	yes
147.75.88.59	test-joyent-smartos17-x64-3	updated firewall on ci	yes
147.75.88.60	release-joyent-smartos17-x64-2	updated firewall on release-ci	yes
147.75.88.61	test-joyent-smartos17-x64-4	updated firewall on ci	yes
147.75.88.57	infra-joyent-ubuntu1604-x64-1-new/unencrypted	updated cloudflare	yes
147.75.88.62	infra-joyent-debian10-x64-1/grafana	updated cloudflare, updated telegraf configs on many machines	yes
139.178.83.227	infra-joyent-smartos15-x64-1-new	sniff test that looks ok	yes

The smartos machines below 17 will not be moved over.

We'll need to:

Update the ansible inventory file with the new IPs
remove smartos16 from node-test-commit-smartos in test ci
remove smartos 16 from node-test-addon-api-new
remove smartos 15, 16 from the libuv tests - FYI to libuv team asking that they do that - Smartos 15 and 16 machines will no longer be available libuv/libuv#3121
remove the smartos 15,16 machines from the CIs and inventory
figure out how we can reboot/manage the new machines (ie is there a management system we'll have access to)
See if there is a good way to test the new release machines in before the next 10, 12 releases (which are the only
ones that we current release on for smartos)

jbergstroem · 2021-02-28T01:09:50Z

Just chipping in, me and Michael have been doing most of the effort up to now with lots of help from a person at Joyent. I will tomorrow review the backup and www mirror seeing how these are very important to our operations. The final switch for these VM's are likely Monday.

mhdawson · 2021-03-01T16:29:27Z

@richardlau can you take care of the libuv side - making sure team is aware and removing smartos 15,16 from the CI before Wednesday.

richardlau · 2021-03-01T17:28:27Z

@richardlau can you take care of the libuv side - making sure team is aware and removing smartos 15,16 from the CI before Wednesday.

I've done https://ci.nodejs.org/view/libuv/job/libuv-test-commit-smartos/ (see libuv/libuv#3121). As noted in libuv/libuv#3121 (comment) the only other libuv job runs https://ci.nodejs.org/job/node-test-commit/ so ends up running https://ci.nodejs.org/job/node-test-commit-smartos/ which will need to be updated but that's a Node.js, not libuv, job.

FWIW these are the hits for smartos15/smartos16 in our CI configs after the libuv job was updated:

https://github.com/nodejs/jenkins-config-release

$ grep 'smartos\(15\|16\)' jobs/*.xml
jobs/iojs+release-mdawson-major.xml:        <string>smartos15-release</string>
jobs/iojs+release.xml:        <string>smartos15-release</string>
$

https://github.com/nodejs/jenkins-config-test

$ grep 'smartos\(15\|16\)' jobs/*.xml
jobs/llnode-continuous-integration.xml:alpine-latest-x64 debian8-64 debian9-64 fedora-latest-x64 centos7-ppcle ubuntu1404-64 ubuntu1604-64 rhel72-s390x smartos16-64 smartos15-64 win10 win2012r2</description>
jobs/llnode-pipeline.xml:      debian8-64 fedora22 fedora23 centos7-ppcle ubuntu1404-64 ubuntu1604-64 rhel72-s390x aix71-ppc64 smartos16-64 smartos15-64 win10 win2012r2</description>
jobs/node-inspect.xml:          <description>Space-separated list of nodes, e.g. debian8-64 centos7-ppcle ubuntu1404-64 ubuntu1604-64 rhel72-s390x aix71-ppc64 smartos16-64 smartos15-64 win10 win2012r2</description>
jobs/node-inspect.xml:      description: 'Space-separated list of nodes, e.g. debian8-64 centos7-ppcle ubuntu1404-64 ubuntu1604-64 rhel72-s390x aix71-ppc64
smartos16-64 smartos15-64 win10 win2012r2',
jobs/node-stress-single-test.xml:        <string>smartos15-64</string>
jobs/node-stress-single-test.xml:        <string>smartos16-64</string>
jobs/node-test-commit-smartos.xml:        <string>smartos15-64</string>
jobs/node-test-node-addon-api-new.xml:ubuntu1804-64 smartos16-64 smartos15-64 win10 win2012r2 aix71-ppc64 centos7-ppcle</description>
jobs/node-test-node-addon-api.xml:smartos16-64 smartos15-64 win10 win2012r2 aix71-ppc64</description>
jobs/node-test-node-addon-api.xml:        <string>smartos16-64</string>
jobs/node-test-node-addon-api.xml:        <string>smartos15-64</string>
jobs/node-test-node-addon-api.xml:  <combinationFilter>(MACHINES=="all" || MACHINES.contains(MACHINE)) &amp;&amp; !((NODE_VERSION=="v4" || NODE_VERSION=="v5") &amp;&amp; (MACHINE.contains("s390") || MACHINE.contains("aix"))) &amp;&amp; !((NODE_VERSION=="v6" || NODE_VERSION=="v7") &amp;&amp; (MACHINE.contains("smartos15") || MACHINE.contains("smartos16")))</combinationFilter>
jobs/readable-stream-continuous-integration.xml:smartos16-64 smartos15-64 win10 win2012r2</description>
jobs/readable-stream-pipeline.xml:smartos16-64 smartos15-64 win10 win2012r2</description>
jobs/string_decoder-continuous-integration.xml:smartos16-64 smartos15-64 win10 win2012r2</description>
jobs/string_decoder-pipeline.xml:smartos16-64 smartos15-64 win10 win2012r2</description>
$

Anything in a description is benign (although we can tidy up for appearances) but any <string>...</string> instances need to be updated. Instances in <combinationFilter>...</combinationFilter> could also be tidied up but won't cause problems (they'd just be redundant filtering).

@mhdawson I think node-test-node-addon-api has been superseded... https://ci.nodejs.org/job/node-test-node-addon-api/ is currently disabled. Perhaps it can be removed entirely?

Let me know if you'd like/need me to update the Node.js release and test jobs.

richardlau · 2021-03-01T17:36:31Z

Also FWIW in terms of release machines we build Node.js 10 releases on smartos17 and Node.js 12 releases on smartos18 (i.e. we don't use smartos15/smartos16 in the release CI). The reference to smartos15 in iojs+release is something we could have cleaned up previously -- probably when Node.js 8 went End-of-Life according to

build/jenkins/scripts/VersionSelectorScript.groovy

Lines 83 to 91 in 3d9672e

    
           // SmartOS ----------------------------------------------- 
        
           [ /^smartos15/,                     anyType,     lt(8)   ], 
        
           [ /^smartos15/,                     anyType,     gte(10) ], 
        
           [ /^smartos16/,                     anyType,     lt(8)   ], 
        
           [ /^smartos16/,                     anyType,     gte(12) ], 
        
           [ /^smartos17/,                     anyType,     lt(10)  ], 
        
           [ /^smartos17/,                     anyType,     gte(12) ], 
        
           [ /^smartos18/,                     anyType,     lt(12)  ], 
        
           [ /^smartos18/,                     releaseType, gte(14) ],

mhdawson · 2021-03-01T20:46:27Z

Yes, I figure out we don't use 15 and 16 which is why I asked that we bring over smartos17 but not those 2. (did not look like there were enough IPs to bring them all over)

mhdawson · 2021-03-01T20:46:42Z

@mhdawson I think node-test-node-addon-api has been superseded... https://ci.nodejs.org/job/node-test-node-addon-api/ is currently disabled. Perhaps it can be removed entirely?

Deleted.

mhdawson · 2021-03-01T20:47:47Z

@richardlau can you take care of removing the references from the rest of the places you mentioned.

mhdawson · 2021-03-01T20:49:41Z

I delete the job related to jobs/iojs+release-mdawson-major.xml: <string>smartos15-release</string>

richardlau · 2021-03-02T10:48:34Z

Removed smartos15-release from https://ci-release.nodejs.org/job/iojs+release.
https://ci-release.nodejs.org/job/iojs+release/jobConfigHistory/showDiffFiles?timestamp1=2021-02-11_04-47-10&timestamp2=2021-03-02_04-43-51
https://github.com/nodejs/jenkins-config-release/commit/e61d490750fd36d12471eade63f2697836e61556

Removed smartos15-64 from https://ci.nodejs.org/job/node-test-commit-smartos.
https://ci.nodejs.org/job/node-test-commit-smartos/jobConfigHistory/showDiffFiles?timestamp1=2021-02-27_19-13-55&timestamp2=2021-03-02_05-54-21
https://github.com/nodejs/jenkins-config-test/commit/cc7831a22dea2f8c548214d69f36011a4d807817

Removed smartos15-64 and smartos16-64 from https://ci.nodejs.org/job/node-stress-single-test/ (also added smartos18-64).
https://github.com/nodejs/jenkins-config-test/commit/f8bb3f89fb2eccf8117a596663b01c303ea279aa

Updated remaining descriptions: https://github.com/nodejs/jenkins-config-test/compare/f8bb3f89fb2eccf8117a596663b01c303ea279aa...fc5947369b28d2d9f492f0a51863a1dfe7f867fd

jbergstroem · 2021-03-02T11:42:06Z

Small update based on last nights work: we only have one machine left to migrate which is going to be done today.

richardlau · 2021-03-02T11:56:56Z

I've removed the release and test smartos 15 and 16 machines from secrets and opened a PR to remove from the ansible inventory: #2554

richardlau · 2021-03-02T15:56:41Z

I've also removed the release and test smartos 15 and 16 machines from the CI.

jbergstroem · 2021-03-03T11:41:06Z

Update: the backup machine completed the jobs successfully. I've had to remove the bits where we try to backup our mysql for benchmarks due to that machine not being available any longer. This should conclude the migration. Next steps is opening PR(s) to update our repo.

mhdawson · 2021-03-03T20:36:47Z

We've turned off all of the old machines and the Joyent data-center is being de-commissioned today. We'll know soon if we updated/migrated everything that was needed :)

richardlau · 2021-03-04T11:57:54Z

Kicked off test builds on smartos only to validate the replacement release machines:
10.x: nodejs/node@ab8d3c5a12: https://ci-release.nodejs.org/job/iojs+release/6715/
12.x, nodejs/node@ce800870b4: https://ci-release.nodejs.org/job/iojs+release/6716/

richardlau · 2021-03-04T14:51:14Z

Did some basic tests (node -p process.versions, 2+2 in repl) with

No issues to report, so I'm happy the new release machines look okay.

Refs: nodejs#2552

richardlau · 2021-03-12T15:06:08Z

Based on the table in the description I've opened #2568 to update the ansible inventory.

Refs: nodejs#2552

Refs: #2552

richardlau · 2021-11-15T16:47:44Z

@mhdawson @jbergstroem the one remaining task here is

figure out how we can reboot/manage the new machines (ie is there a management system we'll have access to)

I'm guessing at this point that the current "process" if we need to do any of that for the Joyent machines is to contact @bahamat? I can update the outdated notes in the secrets repo and close this off, otherwise I'd suggest we open another issue if it's something we want to keep open.

mhdawson · 2021-11-16T21:37:02Z

@richardlau +1. Lets just document and close this out.

richardlau · 2021-11-24T16:20:06Z

I've updated the notes for the Joyent machines in the secrets repo.

richardlau added ansible ci-change PSA of configuration changes platform:smartos labels Feb 28, 2021

richardlau mentioned this issue Mar 2, 2021

ansible: remove smartos 15 and 16 from inventory #2554

Merged

richardlau mentioned this issue Mar 5, 2021

Errors running Ansible across current cluster #2531

Closed

55 tasks

richardlau added a commit to richardlau/build that referenced this issue Mar 12, 2021

ansible: update Joyent hosted IP addresses

b371ca5

Refs: nodejs#2552

richardlau mentioned this issue Mar 12, 2021

ansible: update Joyent hosted IP addresses #2568

Merged

This was referenced Mar 15, 2021

Remove test-joyent-ubuntu1804_docker-x64-1 containers from Jenkins #2572

Closed

Do we need to replace test-joyent-ubuntu1804_docker-x64-1? #2573

Closed

richardlau added a commit to richardlau/build that referenced this issue Mar 16, 2021

ansible: update Joyent hosted IP addresses

afdca56

Refs: nodejs#2552

richardlau added a commit that referenced this issue Mar 22, 2021

ansible: update Joyent hosted IP addresses (#2568)

7b81d46

Refs: #2552

richardlau mentioned this issue Sep 15, 2021

gcc 8.3 on SmartOS #2757

Closed

richardlau closed this as completed Nov 24, 2021

sxa mentioned this issue Dec 5, 2022

All Joyent machines are offline #3104

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replacing machines from Joyent data center #2552

Replacing machines from Joyent data center #2552

mhdawson commented Feb 28, 2021 •

edited

Loading

jbergstroem commented Feb 28, 2021

mhdawson commented Mar 1, 2021

richardlau commented Mar 1, 2021

richardlau commented Mar 1, 2021 •

edited

Loading

mhdawson commented Mar 1, 2021

mhdawson commented Mar 1, 2021

mhdawson commented Mar 1, 2021

mhdawson commented Mar 1, 2021

richardlau commented Mar 2, 2021 •

edited

Loading

jbergstroem commented Mar 2, 2021

richardlau commented Mar 2, 2021

richardlau commented Mar 2, 2021

jbergstroem commented Mar 3, 2021

mhdawson commented Mar 3, 2021

richardlau commented Mar 4, 2021

richardlau commented Mar 4, 2021

richardlau commented Mar 12, 2021

richardlau commented Nov 15, 2021

mhdawson commented Nov 16, 2021

richardlau commented Nov 24, 2021

Replacing machines from Joyent data center #2552

Replacing machines from Joyent data center #2552

Comments

mhdawson commented Feb 28, 2021 • edited Loading

jbergstroem commented Feb 28, 2021

mhdawson commented Mar 1, 2021

richardlau commented Mar 1, 2021

richardlau commented Mar 1, 2021 • edited Loading

mhdawson commented Mar 1, 2021

mhdawson commented Mar 1, 2021

mhdawson commented Mar 1, 2021

mhdawson commented Mar 1, 2021

richardlau commented Mar 2, 2021 • edited Loading

jbergstroem commented Mar 2, 2021

richardlau commented Mar 2, 2021

richardlau commented Mar 2, 2021

jbergstroem commented Mar 3, 2021

mhdawson commented Mar 3, 2021

richardlau commented Mar 4, 2021

richardlau commented Mar 4, 2021

richardlau commented Mar 12, 2021

richardlau commented Nov 15, 2021

mhdawson commented Nov 16, 2021

richardlau commented Nov 24, 2021

mhdawson commented Feb 28, 2021 •

edited

Loading

richardlau commented Mar 1, 2021 •

edited

Loading

richardlau commented Mar 2, 2021 •

edited

Loading