[proposal] run whitelist of popular nimble packages as a part of non-blocking CI suite, to catch regressions early

[timotheecour]

proposal

• add a tool to install, build, run testsuite of nimble packages (from https://github.com/nim-lang/packages/blob/master/packages.json)
• add a line in CI (travis, appveyor) that calls this tool on a subset all nimble packages, using a whitelist
• whitelist favors packages that are commonly used (using download statistics, which should be irrespective of direct vs indirect nimble install calls), that are fast to build and test (to avoid adding too much stress on travis/appveyor), and that are not broken on devel
• this whitelist can specify for each package an optional tag (but default of #head should be preferred to make sure we're current; Note: that revision can itself depend on non-head dependencies, that's fine)
• failures should be considered as "Allowed Failures" but can still be used for figuring out whether a PR breaks some packages
• a html page containing summary of passed/failed/skipped results is generated
• all this should be written in Nim, except a 1 line shell call in travis/appveyor

Note

looks like support is already there; just not being run in CI:
Nim/tests/testament/categories.nim:

  of "nimble-core":
    testNimblePackages(r, cat, pfCoreOnly)
  of "nimble-extra":
    testNimblePackages(r, cat, pfExtraOnly)
  of "nimble-all":
    testNimblePackages(r, cat, pfAll)

(also, on OSX, ./koch tests cat nimble-core ran into dom96/choosenim#80)

links

• in D, the code to do that is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L457 (just as FYI, probably not much to copy over to Nim)
• FedericoCeratto/nim-ci: Continuous Integration for the Nim compiler and repositories
• add CI testing (eg via travis) sdl2#111 (comment) run nim check <mainfile.nim> for whiltelisted repos that don't have a nimble test section

motivation

• would help with regressions, eg recent regressions suhc as --nilseqs:on should show warning when nil is used; or add --nilseqs:warning option #8668 (comment) (these are ongoing, caused by recent nil string change)

[dom96]

This could (and probably should) be done in a separate repo/CI.

[timotheecour]

possibly, but then how would one run CI automatically for each PR ?
another option is to run a cron-job regularly (eg every x hours or every day); that would probably be more economical than after each PR update.
what would u recommend for running this kind of kron-job?

[dom96]

another option is to run a cron-job regularly (eg every x hours or every day); that would probably be more economical than after each PR update.

Indeed. Travis can do this.

[FedericoCeratto]

@timotheecour The package regression testing was already being performed by https://github.com/FedericoCeratto/nim-ci but at the time it attracted little interest. The tool was monitoring the Nim repos and triggering rebuilds.

[andreaferretti]

I have been proposing this since forever, really hope this takes off!

@FedericoCeratto was nim-ci actually building anything? Last time I remembered it was just doing nimble install, which in most cases just does nothing. It really should have been calling nimble test or (even better) letting users register packages and specify how to test them

[genotrance]

I'll be happy to help with this. I am already doing a small subset with nimgen and nimterop which test over 30 wrapper packages daily with devel, stable and stable - 1 and have found and reported issues in the past.

Please let me know how I can contribute.

[Araq]

I would patch "nightlies" to do the stress testing. One question then is how to integrate this with CIs.

[timotheecour]

@genotrance might be worth starting from (a fork of) https://github.com/FedericoCeratto/nim-ci instead of from scratch /cc @FedericoCeratto

[Araq]

This has been implemented. Add your packages to important_packages.nim please.