Ideas on smoketesting

[phillipj]

Now that we've got greenkeeper keeping an eye on our dependencies and opening PRs (ex #221) whenever something updates, it would have been great to smoketest the content generation. ATM we have a very limited set of tests, mainly the release blog post script, which doesnt verify that the main purpose of the website still works. With those tests missing, greenkeeper isnt really that valueable for us - until we fix it :)

Any ideas on how we should (smoke)test the content generation, ensuring we dont merge & deploy something that's broken?

[mikeal]

I wrote this a while back https://github.com/mikeal/webtouch it parses through a page, finds all links and validates those links and their links and all resources they reference, including resources in the CSS.

[bnb]

Could you define smoketesting? Never heard that term before. Also, are you talking about using this with the overall collection of Node projects, or just those here in evangelism?

[phillipj]

A smoketest might start a process and verifies that it started and exited cleanly or still running after a short periode of time. I really like to smoketesting my webapps before deploy by starting it and doing a GET request against the frontpage or some health check URL. These trivial tests has saved me from deploying totally broken apps more than once.

One concrete example from this project, was one PR that had a syntax error in one of the .styl files, which made the static file generation fail horribly. ATM that is something we have to check manually before merging any PR.

The simplest smoketest I can think of is running the file generating and checking the exit code. Though using something like @mikeal's webtouch is alot more thorough and gives more confidence to collabs before merging anything to master. They could even be combined?

Also, are you talking about using this with the overall collection of Node projects, or just those here in evangelism?

This website project only.

[bnb]

Okay, that sounds good. Would implementing this be a good first contribution, or is it a higher level of skill?

[phillipj]

Absoutely a good first contribution!

On Saturday, 10 October 2015, &! (bitandbang) notifications@github.com
wrote:

Okay, that sounds good. Would implementing this be a good first
contribution, or is it a higher level of skill?

—
Reply to this email directly or view it on GitHub
#225 (comment)
.

[sotojuan]

Hi, I'm interested in helping out with this. I read up on webtouch and it seems simple and useful enough, bu when @phillipj said

The simplest smoketest I can think of is running the file generating and checking the exit code.

For this repo/site, what would that be? Alternatively, perhaps an example of someone' repo with a smoke test could help out too. I'll keep looking into it as well. Thanks!

[jasnell]

We have an ongoing effort... http://github.com/nodejs/citgm

[thefourtheye]

@jasnell citgm is for core Node.js, right? I think this thread is about the website; to check if the PRs which this receives don't break the site in any way.

[jasnell]

Ooohhhh lol... Missed that small fact lol. Thanks!
On Nov 14, 2015 8:45 PM, "thefourtheye" notifications@github.com wrote:

@jasnell https://github.com/jasnell citgm is for core Node.js, right? I
think this thread is about the website; to check if the PRs which this
receives don't break the site in any way.

—
Reply to this email directly or view it on GitHub
#225 (comment)
.

[phillipj]

For this repo/site, what would that be?

@sotojuan lets imagine someone opens a PR with a JavaScript syntax error, or using a stylus variable which doesn't exist. Those two scenarios will make the process exit with status code 1 if one tries to run the build script $ node build.js.

# remove require('Metalsmith') from build.js
→ node build.js
[metalsmith] build/static finished: 53ms
/nodejs/new.nodejs.org/build.js:59
  const metalsmith = Metalsmith(__dirname)
                     ^
ReferenceError: Metalsmith is not defined
...
→ echo $?
1

A good smoketest would be having a small script that actually runs build.js as above, and verifies the exit code equals 0. Having that kind of test run automatically on Travis would be very valuable for us!