Snapcraft build has been broken since 0.5.16

[lightyear15]

Abstract

https://snapcraft.io/build
Auto-build and publish new solc version on the snapcraft app store.

Motivation

solc on snapcraft is at version 0.5.16. Last time it was built was January 2020.

Specification

As explained in the link attached in the Abstract section, a snapcraft app can be connected to a github repo, so new builds are triggered every new commit.

Backwards Compatibility

[cameel]

Looks like the snapcraft build is already automated. It's just broken. According to the discussion in #2475 it's tied to https://launchpad.net/solidity. In ~elopio/+snap/solc I see that builds were running (and failing) until a year ago. Then they stopped. The build from 2020-09-28 has some build errors related to Z3:

[  9%] Building CXX object libsmtutil/CMakeFiles/smtutil.dir/Z3CHCInterface.cpp.o
�[01m�[K/build/solc/parts/solidity/src/libsmtutil/Z3Interface.cpp:�[m�[K In member function ‘�[01m�[Kz3::expr solidity::smtutil::Z3Interface::toZ3Expr(const solidity::smtutil::Expression&)�[m�[K’:
�[01m�[K/build/solc/parts/solidity/src/libsmtutil/Z3Interface.cpp:201:15:�[m�[K �[01;31m�[Kerror: �[m�[K‘�[01m�[Kint2bv�[m�[K’ is not a member of ‘�[01m�[Kz3�[m�[K’
    return z3::�[01;31m�[Kint2bv�[m�[K(size, arguments[0]);
               �[01;31m�[K^~~~~~�[m�[K
�[01m�[K/build/solc/parts/solidity/src/libsmtutil/Z3Interface.cpp:207:15:�[m�[K �[01;31m�[Kerror: �[m�[K‘�[01m�[Kbv2int�[m�[K’ is not a member of ‘�[01m�[Kz3�[m�[K’
    return z3::�[01;31m�[Kbv2int�[m�[K(arguments[0], intSort->isSigned);
               �[01;31m�[K^~~~~~�[m�[K
�[01m�[K/build/solc/parts/solidity/src/libsmtutil/Z3Interface.cpp:207:15:�[m�[K �[01;36m�[Knote: �[m�[Ksuggested alternative: ‘�[01m�[Kis_int�[m�[K’
    return z3::�[01;36m�[Kbv2int�[m�[K(arguments[0], intSort->isSigned);
               �[01;36m�[K^~~~~~�[m�[K
               �[32m�[Kis_int�[m�[K
[  9%] Building CXX object liblangutil/CMakeFiles/langutil.dir/SemVerHandler.cpp.o
libsmtutil/CMakeFiles/smtutil.dir/build.make:158: recipe for target 'libsmtutil/CMakeFiles/smtutil.dir/Z3Interface.cpp.o' failed
make[2]: *** [libsmtutil/CMakeFiles/smtutil.dir/Z3Interface.cpp.o] Error 1

Our snapcraft.yaml is using Z3 4.8.4, which is still supported so it's not that. @leonardoalt could you take a look at these errors?

But overall, the questions is how to prevent such breakage in the future. We do advertise the snap version under Linux Packages so we should fix it but I think no one is checking that regularly. As far as I can tell from past PRs, this was a contribution from @ElOpio and set up in a way that should not require too much maintenance. Looks like we still do need to monitor it in some way though. Perhaps we need a step in the release checklist to make sure the snap package was built corectly?

[chriseth]

@ElOpio are you still available for maintaining the snap build?

[come-maiz]

Yes. If you think this is still useful I can give it some love.

[chriseth]

That would be very nice of you! Some people seem to be using it.

[come-maiz]

Alright! We have a successful release on the edge branch for amd64.

I would appreciate people testing it with:

sudo snap install solc --edge

Building on the other architectures is being harder. On cvc they suggested to upgrade to 1.8, so I'm trying options.

[cameel]

So you're trying to build also arm64, armhf and i386? Any compilation issues or just trouble with libraries? In CI we only build for amd64 and wasm so it's interesting that it even works :)

I wanted to install it via snap but looks like snap does not like running in docker too much so we'll need someone with Ubuntu. @lightyear15 can you test it?

[lightyear15]

Thanks @ElOpio ,
I have tested the new edge solc you pushed to snapcraft.
I tried to compile a contract I have and it seems it's working.
The only concern I have is about the size of the vm

channels:
  latest/stable:    v0.5.16 2020-01-29 (6746)  36MB -
  latest/candidate: v0.5.17 2020-03-17 (6854)  36MB -
  latest/beta:      v0.6.8  2020-05-18 (6979)  43MB -
  latest/edge:      develop 2021-10-02 (7006) 294MB -
installed:          develop            (7006) 294MB -

from 43 MB to 294MB. I don't know if this change in size is due to the compiler itself or not.

Regarding releasing the compiler to the different channels ( stable, candidate, ecc...) AFAIK it's something that it can only be done manually in the snapcraft dev admin webpage. You can play with git tags though, so that all the automatically triggered builds will be tagged differently.

[cameel]

The compiler is definitely not that big. The binary itself is around ~10 MB. Statically linked Z3 adds ~25 MB. Not sure about CVC, but I would not expect it to take ~300 MB.

I suspect that some stuff needed only at build time gets left in the image. Which is weird because after #12078 it actually does less manual building and gets more stuff from system package. Does stuff listed under build-packages get uninstalled after the build?

[come-maiz]

There is a snapcraft action that can be used to test that the snap builds correctly. Building the whole thing is slow though, so I'm not sure when would you want to run this.

It can be used to push tags to the candidate channel too.

https://github.com/snapcore/action-build

[come-maiz]

The bulk in size is in the built binaries:

> ls -sh /snap/solc/current/usr/local/bin
total 1.2G
486M solc*  479M solidity-upgrade*  208M yul-phaser*

[cameel]

Are you building them in debug mode? In release mode they're much smaller. In debug they're also pretty big but even then only about half that size:

198Mi solc
196Mi solidity-upgrade
 96Mi yul-phaser

[lightyear15]

not seeing any progress here 😕
now we are stuck to version 0.6.8, releases in the beta channel
stable channel is still stuck in 0.5.16

[come-maiz]

Hello. I've checked the status today. The builds seem ok, only s390x and armhf failed today.

This is what we have in the edge channel.

solc --version
solc, the solidity compiler commandline interface
Version: 0.8.12-develop.2021.12.21+commit.b65e0933.Linux.g++

So what we are missing is to move from the edge channel to beta when there is a tag. This could be done manually, or automated on CI calling snapcraft release. And then to move to the stable channel, I think that should be done manually.
This could be useful: https://forum.snapcraft.io/t/selective-checkout-check-out-the-tagged-release-revision-if-it-isnt-promoted-to-the-stable-channel/10617

Has anybody been testing the edge channel?

[cameel]

Did you figure out why the binary is so big? Is it getting built in debug mode or something?

One thing that might be the cause is that in 0.8.1 we added defaults for build type (#10758) and the default is now Release - unless you're building inside a git repository - then it's RelWithDebInfo, which might explain the size.

[mrx23dot]

Why does it uses so much space? I expect the binary be <5MB
sudo snap install solc --edge
on debian 10

/snap/solc/7114/usr/local/bin
560.2 MiB [##########] solc
545.0 MiB [######### ] solidity-upgrade
238.7 MiB [#### ] yul-phaser

[NunoFilipeSantos]

There is also a snap package <https://snapcraft.io/solc>_, however, it is currently unmaintained.

#12507