Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the benchmarks in a nightly build only #65

Closed
ldionne opened this issue Mar 24, 2016 · 2 comments
Closed

Update the benchmarks in a nightly build only #65

ldionne opened this issue Mar 24, 2016 · 2 comments
Labels

Comments

@ldionne
Copy link
Owner

ldionne commented Mar 24, 2016

Right now, the full benchmarks get generated each time we push to master. This is inconvenient, because the CI takes a long time to complete. It is also useless, because we already have testing that should catch errors.

What we'd really like is to only run the benchmark jobs (and update gh-pages with the data) once per day. In theory, we could use a service like nighli.es to achieve this, but I don't think it is possible to know from the CI script that we're being launched by the service. Hence, we wouldn't know when to actually run the benchmarks.

@ldionne ldionne changed the title Do not run the benchmarks all the time Update the benchmarks in a nightly build only Mar 28, 2016
@ldionne
Copy link
Owner Author

ldionne commented Mar 29, 2016

I tried using Nightli.es, but there is no way to know that we're being run as a nightly build. So I've set up a basic Heroku app which performs the following command every night:

curl -s -X POST 
  -H "Content-Type: application/json" 
  -H "Accept: application/json" 
  -H "Travis-API-Version: 3" 
  -H "Authorization: token ${TRAVIS_TOKEN}" 
  -d '{"request": {
    "message": "Automated nightly update of the benchmarks.", 
    "branch":"master",                                                      
    "config": { "before_install": "export METABENCH_NIGHTLY_BUILD=true" }   
  }}'                                                                         
  https://api.travis-ci.org/repo/ldionne%2Fmetabench/requests

This uses the Travis CI API, as documented here, to start a build. I tried several different ways to make the script aware that it's being run in a nightly build, but the only thing that worked was to set METABENCH_NIGHTLY_BUILD=true explicitly. It's hacky, but it works and I documented the hack properly so it does not bite us in the future. If nightli.es gets the ability to set an environment variable, we'll be able to easily change to it and get rid of the Heroku app.

So the benchmarks should start getting updated tonight. The build will fail for Clang 3.8 because of travis-ci/apt-source-safelist#255, but everything should work just fine otherwise.

@brunocodutra
Copy link
Collaborator

We come back to this once coderanger/nightlies#12 is addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants