Nix: no drun by default, and build with Github Actions #1224
Conversation
This sets up a simple GitHub Actions workflow that builds all targets on both Linux and Darwin. The workflow only builds the targets specified in the [DFX Interface](./design/DFX-Interface).
if they are not, warn, but do not try to run the tests, and do not allow accepting tests. This means that people who do not have access to `dfinity` can run some tests (including a public CI like Github Actions). It also means that `make accept` will not commit `could not find ic-stub-run` as the expected output, as has happened to us a few times before.
the reiterates on #1220. The motivation for the refactoring was that the generated files were not using `gitSource`, so I would get cache misses and dirty files in the repo. Fixing this was easier if all generated nix files are in one place, so I did the following changes: * All generated nix files are in `nix/generated`. This de-pollutes the normal working area with generated files, and keeps them out from sources (more precise derivations). * The local generated files now use `gitSource`. * The generator is now `nix/generate.nix`. * I moved complex logic from `nix/default.nix` to `nix/generate.nix`, this means everything is in one place, instead of spread over several nix files. * There is now only a single check to check if the generated files are upto date, and a single command to update the files. * This check is no longer marked `allowSubstitutes = false`. The check isn’t cheap (it requires `cabal2nix`) and we want the check to happen on builders, not hydra itself, so that the cache has `cabal2nix`. * The check is now included in `all-systems-go`.
|
I guess without a (public) cache running this on GitHub Actions (or external users) is not feasible, as the amount of building happening will likely exceed all limits there? |
|
https://github.com/dfinity-lab/motoko/pull/1224/checks?check_run_id=456633738 reports: 22m to build |
in #1189 we switched to a fork of nixpkgs to get two patches not in nixpkgs `master` at that time. Looks like they have made it to `nixpkgs/master`, so we can track that again (and make the Motoko repo more self-contained).
better public cache hits
this removes some redundant logic; cherry-picked from #1224
this removes some redundant logic; cherry-picked from #1224
|
Thanks to #2540 this now only disables If we can get |
nomeata
changed the title
| - uses: cachix/cachix-action@v10 | ||
| with: | ||
| name: ic-hs-test | ||
| authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' |
There was a problem hiding this comment.
I'm assuming you're the one holding this secret at the moment. Who else should have a copy?
There was a problem hiding this comment.
This is currently a cache on my personal cachix account. I suggest we run with that until we are open and things work, then we can migrate to a cache owned by some organizational account (but I fear the confusion if we try to solve that problem at the same time).
Using a different cache is rather simple and not very disruptive.
There was a problem hiding this comment.
Understood. Let's just make sure we don't forget about this. "Nichts hält so lange wie ein Provisorium"
There was a problem hiding this comment.
I think the idiom goes “wie ein gutes Provisorium” :-)
| curl -L https://nixos.org/nix/install | sh | ||
| ``` | ||
|
|
||
| You should also enable a nix cache to get all dependencies pre-built. |
There was a problem hiding this comment.
Do we (the internal users) also need to run these steps now?
There was a problem hiding this comment.
No. Or rather, not yet. It doesn’t hurt though (you might get faster downloads than over VPN?)
Co-authored-by: Christoph Hegemann <6189397+kritzcreek@users.noreply.github.com>
| external contributors cannot run it. | ||
|
|
||
| The build system is setup by default to _not_ load the `drun` tool. If you have | ||
| access to this tool, create an empty file called `enable-internals` in this repository |
There was a problem hiding this comment.
I guess this file should not be added to git right?
There was a problem hiding this comment.
Right, that’s why it’s in .gitignore, so you’d have to try hard to add it. And it would break CI, so we’d notice.
|
Shall we merge this? |
|
@nomeata since this doesn't add an |
Why would external contributors need drun to build Motoko? It's used for testing, not building, right? |
Developers with internal access should touch the file locally. The internal CI sets |
Yes, but external contributors should also be able to run the test suite, at least as far as possible |
This helps eventual external contributors to build Motoko. The only internal dependency left is
drun.By default,
drunis not used. If you have access to the internal repositories, runand things are as before.
This also sets up a Github Action build that builds all stuff, on Linux and Darwin, and uses the
ic-hs-testcachix cached (shared withdfinity/ic-hs) to cache build artifacts.