tools: add download testing

[gdams]

This script will take an environment variable (NODE_VERSION) and
download all of the files from rsync://unencrypted.nodejs.org/nodejs/release/v<NODE_VERSION>/
It will then loop through the SHASUMS256.txt file to check if the
SHA256SUMS match up. Initial issue here: #513

[gdams]

updated PTAL

[mhdawson]

Started building the CI job, noted a few issues in earlier comments.

What I think would also be useful is if it could take an option like "latest" which would figure out the latest for a give release line. That we won't have to update the job as new releases come out. Generally I'm thinking we'll only need to be testing the latest.

[mhdawson]

still getting this

cat: nodejs.org/dist/v8.1.4/SHASUMS256.txt: No such file or directory
./build/tools/download-test.sh: line 49: ${TAP}: ambiguous redirect

[gdams]

@mhdawson should be fixed now if you want to re-run?

[gdams]

CI job here for reference: https://ci.nodejs.org/view/All/job/validate-downloads/

[gireeshpunathil]

I think the robot.txt is not allowing recursive download. We may need to parse the index.html to get links individually and then download them.

bash-4.1$ wget -r -d  http://nodejs.org/dist/v8.1.4/
...
Deciding whether to enqueue "http://nodejs.org/dist/v8.1.4/node-v8.1.4-x86.msi".
Rejecting path dist/v8.1.4/node-v8.1.4-x86.msi because of rule “dist/”.
Not following http://nodejs.org/dist/v8.1.4/node-v8.1.4-x86.msi because robots.txt forbids it.
Decided NOT to load it.

bash-4.1$ cat nodejs.org/robots.txt 
User-Agent: *
Disallow: /dist/
Disallow: /docs/
Allow: /dist/latest/
Allow: /dist/latest/docs/api/
Allow: /api/
bash-4.1$ 

[gireeshpunathil]

ok, the latest script line
wget -r --no-parent --execute="robots = off" http://nodejs.org/dist/v${NODE_VERSION}/
seem to have addressed the robot block.

[mhdawson]

Ok, so seems to run but I am seeing this

./build/tools/download-test.sh: line 36: [: !=: unary operator expected

@gdams I'd also like to be able to be able to specify something like 8.latest and have it figure out the latest version.

[rvagg]

Worth noting that unencrypted.nodejs.org is currently on a 15 minute cron to update its binaries from the main server, that may need to be taken into account if this tool is for automating something close to release. The 15 minute number is arbitrary and could be changed if needed fwiw, it's even possible that we could use a flag mechanism to trigger a sync when a release is promoted so it's near instant, although that's a bit more complicated obviously.

[gibfahn]

@gdams when I said tools in #513 (comment) I linked to the tools/ directory in nodejs/node. I don't think it makes sense to start keeping scripts in this repo. so could you move this PR over there?

[gdams]

@gibfahn I was asked by @mhdawson to move to over to nodejs/build nodejs/node#14363

[gibfahn]

@gdams ahh, don't know how I missed that PR. I'll comment over there.

[mhdawson]

Now that I'm back I'd like to move forward on this. @gdams have you had time to look at my suggestions for it being able to automatically run using the latest release ?

[gdams]

@mhdawson @gibfahn updated PTAL. You can now either pass in a full node version (v6.11.2) or just (v6) and it will fetch the latest 6 build. You can also export DOWNLOAD_LOCATION=nightly and fetch the latest nightly builds

[mhdawson]

@rvagg @jbergstroem any input on whether you think this best fits in the build or core node repo ? My feeling is the build repo but I could be convinced as @gibfahn suggests that it go into the core node repo.

[rvagg]

Seems like a reasonable case could be made either way. If it's in nodejs/node then others could run it independently to confirm.

@mhdawson can you explain how the CI job runs? I can't quite work it out. It runs on a schedule but does that mean it's grabbing the latest version every hour to confirm? Also, you may want to make it run on jenkins-workspace because those machines are perfect for this kind of work.

[rvagg]

just saw the description in #513 about how the script works, fine by me.

I've changed restrict nodes from ubuntu1404-64 || benchmark to jenkins-workspace which is serviced by two packet.net boxes with big 'ol disk.

[mhdawson]

@rvagg thanks for the change to where it runs, I was wondering what the best machines would be.

[gibfahn]

Seems like a reasonable case could be made either way. If it's in nodejs/node then others could run it independently to confirm.

Yeah it's a tricky one. Would be good to get opinions from more people in @nodejs/build.

[joaocgreis]

My opinion on this is that nodejs/node is not the place for this. When we checkout a commit from nodejs/node, everything there should relate to that exact version, and be depended on by something there.

The way I would have done this would be directly in a Jenkins job script. We have backups of the server, so it would be safe. However, it would not be public and easily reviewable, but that again links to the discussion about reviewing the jobs source. Since this is already done as a script, I don't see why not land it here.

[mhdawson]

@gibfahn I thinking from your +1 on @joaocgreis' comment that you are now ok with landing this this repo ? If so I assume we just need @gdams to address your latest review comments and we'll be good to land ?

[gibfahn]

@joaocgreis

My opinion on this is that nodejs/node is not the place for this. When we checkout a commit from nodejs/node, everything there should relate to that exact version, and be depended on by something there.

I don't think this applies to a lot of things in nodejs/node:tools/.

The way I would have done this would be directly in a Jenkins job script. We have backups of the server, so it would be safe. However, it would not be public and easily reviewable, but that again links to the discussion about reviewing the jobs source.

Yeah I think putting things in a Jenkins script is the worst possible option (we can't review, there may be backups but in my experience they aren't as easy to version control as git, it's hidden by default).

@mhdawson

I'm -0 on it, yes. If we're going to do this I'd like us to decide where we're going to put these kinds of scripts, and maybe move more out of Jenkins configs and into this repo. IDK if tools/ is the best name for this folder in that case, but no strong opinions.

Given that we have a meeting on Tuesday, we could probably talk about this there (in particular the general question of where files like this should go). I'll add it to the agenda.

[joaocgreis]

I don't think this applies to a lot of things in nodejs/node:tools/.

In my opinion, it should. But I have no strong feelings about this.

Let's discuss at the meeting.

[gibfahn]

@gdams consensus in the meeting (minutes: #845) was to create a nodejs/build:/jenkins/ folder to put this in, as we might want to move more scripts in here going forwards.

If you could fix up my suggestions and change the folder, this should be good to go.

[gdams]

@gibfahn updated PTAL

[gdams]

@gibfahn updated PTAL

[gibfahn]

@nodejs/build any other issues? If not I'll land this later today.

[gdams]

remember that the download testing job needs to be updated with the new path

[mhdawson]

About to land and then fix up the testing job as its already broken since it uses the branch :)

[mhdawson]

Landed as 365c785

Jobs fixed up. Still need to discuss email notifications but we can do that in the original issue as opposed to this PR so closing.