Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release download script #35

Closed

Conversation

cameel
Copy link
Member

@cameel cameel commented Jul 9, 2020

Part of ethereum/solidity#9258.

The script downloads latest release binaries for Linux, Mac OS and wasm from CircleCI. Files are placed in the following directory structure:

  • emscripten-asmjs/solc-emscripten-asmjs-<version>.js
    • Files symlinked to bin/
  • emscripten-wasm32/solc-emscripten-wasm32-<version>.js
    • Files symlinked to wasm/
  • linux-amd64/solc-linux-amd64-<version>
  • macosx-amd64/solc-macosx-amd64-<version>
  • windows-amd64/solc-windows-amd64-<version>.zip

TODO:

  • Files do not have the executable bit set. Done in Script for downloading release binaries from github #41
  • Make sure b_ubu is the right source for the Linux build.
    • We don't have static linux builds on CircleCI
  • Add latest symlinks for Linux and Mac OS and make the update script update them. Done in Emscripten releases #40
  • Query github API only once rather than once per downloaded binary.
  • Make the script actually stop and display a sensible error message when it runs into Github API rate limiting (currently it prints curl: (3) bad range specification in URL position 2: and continues).
  • Don't overwrite binaries if they already exist (and maybe add a flag to force overwrite).
    • Added information that they will be overwritten instead. They're supposed to be versioned anyway.

@cameel cameel requested review from ekpyron and chriseth July 9, 2020 18:35
@cameel cameel self-assigned this Jul 9, 2020
@cameel cameel force-pushed the release-download-script branch 3 times, most recently from 8af43b2 to 7da11e3 Compare July 10, 2020 20:20
@cameel
Copy link
Member Author

cameel commented Jul 10, 2020

Script updated:

  • Now it can also download releases from github.
  • Added --help.
  • List of tags is now fetched from github only once.
  • Script now properly exits on errors. It wasn't working because local is a shell command by itself and its return value shadows the one from $(...). Unfortunately fixing this requires separate declaration and assignment (local a; a=$(...)) or using only global variables.
  • Regex in the update script is now correct.

@cameel cameel force-pushed the release-download-script branch from 7da11e3 to cf778c7 Compare July 10, 2020 20:24
update Outdated
@@ -11,7 +11,7 @@ const swarmhash = require('swarmhash')
// This script updates the index files list.js and list.txt in the bin directory,
// as well as the soljson-latest.js files.

var dirs = ['/bin', '/wasm']
var dirs = ['/macos', '/linux', '/wasm', '/bin']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not have windows because it does not have a circelci config, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly. I wanted to take care of windows builds in a separate PR, when I can actually get them.

@chriseth
Copy link
Contributor

Should we include an architecture for linux, macos and windows?

@cameel
Copy link
Member Author

cameel commented Jul 13, 2020

Good point, I think we should. We may want to support Mac OS on both arm and x86 at some point.

I'd put that both in the file name and in the directory name. I don't have a strong opinion on how granular the directories should be - other than that they should not contain too many entries - so for now I'm going with the directories representing the things that are forced on you (system, platform, etc.) and slight variations that are up to your preference (i.e. static vs dynamic, gcc vs clang) would be kept in the same directory and just have different file names.

@cameel
Copy link
Member Author

cameel commented Jul 13, 2020

To summarize the discusion from Gitter, this is the naming convention I'm going to use:

emscripten-asmjs/solc-emscripten-asmjs-v0.5.0+commit.1d4f565a.js
emscripten-wasm32/solc-emscripten-wasm32-v0.5.0+commit.1d4f565a
linux-amd64/solc-linux-amd64-v0.5.0+commit.1d4f565a
macosx-amd64/solc-macosx-amd64-v0.5.0+commit.1d4f565a
windows-amd64/solc-windows-amd64-v0.5.0+commit.1d4f565a.zip

@chriseth
Copy link
Contributor

As said on gitter, I think we should start with the os instead of the architecture. Are we really using wasm32 instead of 64?

Otherwise this looks good!

@cameel
Copy link
Member Author

cameel commented Jul 13, 2020

Right. I updated the comment.

Are we really using wasm32 instead of 64?

I'm not 100% sure but I think that's the case. @ekpyron confirmed it when I asked. I did a quick search earlier and I got the impression that wasm64 is still just on the roadmap. But maybe that was just about some specific implementation? Anyway, I'll check out build options to make sure.

@ekpyron
Copy link
Member

ekpyron commented Jul 13, 2020

It's wasm32 - I think that's the only option that really exists today and is actually usable in browsers.

@cameel
Copy link
Member Author

cameel commented Jul 14, 2020

@chriseth I'm trying to figure out if b_ubu is really the job I should use for the linux build and it looks like the size is wildly different from the solc-linux-static binary posted on the release page:

  • b_ubu: 36.7 MB
  • solc-linux-static: 9.0 MB

None of the other jobs matches either:

  • b_ubu_release: 36.7 MB (and byte-for-byte identical to b_ubu)
  • b_ubu_clang: 36.9 MB
  • b_ubu18: 36.2 MB
  • b_archlinux: 11.7 MB
  • b_ubu_cxx20: no artifacts

See CircleCI job for v0.6.11

soljson.js from b_ems job is also different, but only slightly: 22 629 520 bytes (github) vs 22 630 292 (circleci). Wasn't it supposed to be deterministic?

Surprisingly, the Mac OS build is identical byte-for-byte.

@ekpyron
Copy link
Member

ekpyron commented Jul 14, 2020

@cameel 0.6.11 is the only exception there - the scripts used different emscripten versions for that one, but usually that won't be the case.
EDIT: Also as of now I don't think there are any static linux binaries built on CircleCI that can be used here...

@cameel
Copy link
Member Author

cameel commented Jul 14, 2020

If CircleCI linux build is not static then why is it larger than the static one from Travis?

@ekpyron
Copy link
Member

ekpyron commented Jul 14, 2020

While the CircleCI version is dynamically linked, it contains a static version of Z3, which is quite large - that's not in the "travis" build (although arguably it should be in the static release binary).

@cameel
Copy link
Member Author

cameel commented Jul 14, 2020

ok. In that case it seems that everything is in order. I'll use b_ubu.

@ekpyron
Copy link
Member

ekpyron commented Jul 14, 2020

You can't use that - that's dynamically linked :-).

@cameel
Copy link
Member Author

cameel commented Jul 14, 2020

Oh, right, sorry. In that case: do we add a static build to CircleCI? An alternative would be to use one of @ekpyron's GH actions and fetch the binary from there (this would also ease the transition to fully automating it later).

@ekpyron
Copy link
Member

ekpyron commented Jul 14, 2020

I think we should postpone this whole thing altogether for the time being - @chriseth also said on the gitter channel, that we don't really need this script for now, but rather need a directory structure, index files and hosting for the binaries (and we can just keep fetching the actual binaries manually from circleci, travis and appveyor, respectively from the release page the latter two will keep pushing to for now)

@cameel cameel force-pushed the release-download-script branch 3 times, most recently from 0f43e24 to 80c24c3 Compare July 17, 2020 00:52
@cameel cameel changed the base branch from gh-pages to release-download-script-github July 17, 2020 01:23
@cameel
Copy link
Member Author

cameel commented Jul 17, 2020

I'm not sure if this will still be useful for anything so I'm leaving it as draft for now. It does work for MacOS and emscripten builds but obviously can't get static linux and windows ones because we don't have any.

I salvaged the github part and put it into #41. I added some modifications there: downloading windows release, not running the update script, asm.js/wasm detection, etc.

@chriseth
Copy link
Contributor

Maybe we can just close this.

@cameel cameel force-pushed the release-download-script-github branch from 29c2692 to e8565ae Compare July 17, 2020 21:33
@cameel cameel force-pushed the release-download-script branch from 80c24c3 to 5340440 Compare July 17, 2020 21:33
@cameel
Copy link
Member Author

cameel commented Oct 22, 2021

The script here is now a bit out of date (our artifacts changed, CircleCI has released API v2.0, etc.).

I'm going to close this in favor of ethereum/solidity#12181, which reuses parts of this script.

@cameel cameel closed this Oct 22, 2021
@axic axic deleted the release-download-script branch November 4, 2021 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants