Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NixOS support (and other FHS-challenged distros) #545

Open
shazow opened this issue Jan 22, 2022 · 32 comments
Open

NixOS support (and other FHS-challenged distros) #545

shazow opened this issue Jan 22, 2022 · 32 comments
Labels
A-releases Area: releases/packaging T-blocked Type: blocked

Comments

@shazow
Copy link

shazow commented Jan 22, 2022

Problem

Right now the shipped binaries make assumptions about where dynamically-linked libraries live. This is a problem for NixOS and other Linux distros that use less usual FHS layouts (there are dozens of us!).

  1. The binaries that ship in the release are not statically linked, so there's several shared libraries that aren't loaded successfully (I suspect this can be patchelf'd in a pinch)
$ ldd forge
	linux-vdso.so.1 (0x00007ffe0b58f000)
	libssl.so.1.1 => not found
	libcrypto.so.1.1 => not found
[...]
  1. forge build downloads a solc binary which suffers from the same problem.
$ forge build
compiling...
Error:
   0: "/home/shazow/.svm/0.8.2/solc-0.8.2": No such file or directory (os error 2)
[...]
$ ldd /home/shazow/.svm/0.8.2/solc-0.8.2
	linux-vdso.so.1 (0x00007ffec07b9000)
	libdl.so.2 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libdl.so.2 (0x00007fe833da6000)
	libm.so.6 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libm.so.6 (0x00007fe833c65000)
	libc.so.6 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libc.so.6 (0x00007fe833aa0000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib64/ld-linux-x86-64.so.2 (0x00007fe834965000)

I tried using forge build --no-auto-detect but looks like it's refusing to use my local version.

$ which solc
/nix/store/5q9xn2s9pjav5fsh7qg8nj94gnj3jqdg-solc-0.8.2/bin/solc
$ forge build --no-auto-detect
compiling...
Error:
   0: "/home/shazow/.svm/0.8.2/solc-0.8.2": No such file or directory (os error 2)
[...]

Solutions

  1. The low hanging fruit would be to build the shipped binaries as more statically linked than they are now. I do this for some other projects I maintain (mainly Go), it helps reduce all kinds of problems (can make it runnable on Alpine and other minimalist distros!) in exchange for larger binaries.

  2. The more sustainable solution is to rely less on out-of-band binary fetching, and more on distro-friendly pre-packaging.

I understand that the current state of the project is in flux and perhaps the main demographic is people who copypasta curl-bash into their terminals (per the README), but realistically it does not need to be a big change to make this more distro packaging friendly. Even if we're not ready to start tagging releases, small things like changing the default Cargo dependencies to load specific versions rather than git main would go a long way (e.g. ethers-rs). Happy to send some PRs to this effect if it sounds desirable. :)

@onbjerg
Copy link
Member

onbjerg commented Jan 22, 2022

Unfortunately we can't statically link because of libusb. We're going to add packages on different package managers (apt, pacman, yum, brew are on my list), are there others you think we should have on our list?

@shazow
Copy link
Author

shazow commented Jan 22, 2022

I'd love nix support. :)

If you could include a flake.nix derivation, so that any distro that supports the nix package manager (basically everything) will be able to automagically build foundry from source in one command. This would also make it trivial to add to the nixpkgs package set as a full derivation (which would also give binary caches for free).

I started some work on this here, but still needs more work: shazow#1

Somewhat related to #315 but I wanted to clarify that you can have this declaration in a generic way that does not require any further maintenance (unless your build process changes substantially), and you don't need to use it to manage your releases or anything like that.

[Edit] The other option is to make a derivation that just downloads the latest binary releases, has all the correct inputs, and tries to patchelf the broken ldd's. It's not preferred but there are lots of binary-only packages that do this.

[Edit2] Worth noting that the solc problem will still be a thing, though. (And any other on-demand fetched binaries.)

@onbjerg
Copy link
Member

onbjerg commented Jan 22, 2022

I'll note this but do note that it is not top of my priority list currently. Feel free to open up a PR if you feel this is pressing, keeping in mind that we want to support releasechannels where possible (stable, nightly) and fixed versions for multiple package repositories, so maintainability is key here 😄

@shazow
Copy link
Author

shazow commented Jan 22, 2022

I'll try to make it work out of band somehow, but right now none of this works on any of my devices unfortunately. So it is equally as pressing as me (or any other NixOS/etc user) wanting to use foundry. 😅

@onbjerg onbjerg added T-meta Type: meta A-releases Area: releases/packaging labels Jan 22, 2022
@shazow
Copy link
Author

shazow commented Jan 23, 2022

One more note: I compiled forge etc from source, and I'm able to reliably patch the svm-downloaded solc binaries with patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" ~/.svm/0.8.2/solc-0.8.2 for example, but unfortunately that changes the hash of the file which prompts svm to overwrite it before building again and putting me back in square one.

I'd love for there to be a place to disable that svm behaviour, or an alternative approach.

@onbjerg onbjerg added T-feature Type: feature and removed T-meta Type: meta labels Jan 24, 2022
@gakonst
Copy link
Member

gakonst commented Jan 24, 2022

@shazow all good points. You can force local solc version usage (under PATH or SOLC_PATH) with --no-autodetect when building or testing or disable them via the foundry toml.

Am supportive of making svm-rs more configurable, so that the paths issue does not manifest. cc @roynalnaruto

Further removing dynamic libraries seems hard, I've given it a lot of effort so far, but if you have any ideas for it I'd also be supportive, if we can let the user choose from static/dynamically linked to not inflate binary sizes too much or get us in violated license land.

@roynalnaruto
Copy link
Contributor

roynalnaruto commented Jan 24, 2022

@shazow Hi, if I'm not wrong, you are probably getting this error? (You could be sure by enabling tracing with RUST_LOG=ethers=trace and check if the messages are the same as here)
https://github.com/gakonst/ethers-rs/blob/e0ee03328332776f5f18a52b968ea3f62df982cb/ethers-solc/src/resolver.rs#L408-L413

Which would re-fetch the solc binary and your changed dynamic loader is reset?

Could you try the SOLC_PATH with --no-auto-detect flag?

I am not sure how to configure svm-rs in order to support NixOS, but would be happy to take any suggestions from you or if you could PR to the repo?

@gakonst
Copy link
Member

gakonst commented Jan 24, 2022

(We just fixed a bug with --no-auto-detect btw #560)

@shazow
Copy link
Author

shazow commented Jan 24, 2022

Thanks, I'll give it a try soon!

@shazow
Copy link
Author

shazow commented Jan 25, 2022

Confirmed that the --no-auto-detect flag works as expected now.

I'm not sure if something like svm-rs is fixable for NixOS and similar distros. By nature, non-statically-linked binaries don't work. We have decent tooling for patching the dynamically linked paths, and one idea would be to allow for a hook to run on install but that modifies the hash so hash-based version detection goes out the window.

In a full Nix-ified version of this, you would specify the derivation and companion versions you want available, and nix would make sure it's available for you. If foundry has provisions for enabling this scenario (ie. allow for multiple versions of solc to be available for selection but without necessarily relying on a binary fetcher), then I think the rest can be done in the nix packaging out of band.

Would it be possible for it to be an ENV var rather than a flag with every command?

[Edit] Also would be even nicer if the solc version selection still works if present (perhaps via sub-path? or ENV var?) but it doesn't rely on hashes and fetching.

@shazow
Copy link
Author

shazow commented Jan 26, 2022

One more update: I made a stand-alone nix flake which uses the binary releases, it lives here for now (good chance I'll move it eventually): https://github.com/shazow/nixfiles/blob/master/flakes/foundry/flake.nix

This means that anyone who has nix installed can run nix develop git+https://github.com/shazow/nixfiles?dir=flakes/foundry to enter a shell that has forge and cast that has been patchelf'd to the current system.

A few caveats:

  • As expected, it's pinned to specific release binaries/hashes (nix requires this). It would need to be updated with every release (this can be scripted).
  • It doesn't work as a build input for other flakes yet, I think that's because I tossed it into my repo with my other nix configs and flakes don't like that apparently. :)

@shazow
Copy link
Author

shazow commented Jan 26, 2022

Ah figured out how to use my foundry flake as an input for my solidity project's devshell flake, in case this is useful for anyone:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
    foundry.url = "git+https://github.com/shazow/nixfiles?dir=flakes/foundry";
  };

  outputs = { self, nixpkgs, utils, foundry }:
    utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in {
        devShell = with pkgs; mkShell {
          buildInputs = [
            solc
            foundry.defaultPackage.${system}
          ];

          shellHook = ''
            export PS1="\e[1;33m\][dev]\e[1;34m\] \w $ \e[0m\]"
          '';
        };
      });
}

Then can be entered with nix develop

@gakonst
Copy link
Member

gakonst commented Jan 28, 2022

Would it be possible for it to be an ENV var rather than a flag with every command?

You can do that via the foundry.toml file, by specifying a compiler version, see the config package readme!

@shazow
Copy link
Author

shazow commented Feb 4, 2022

Update in case anyone else is using this with nix: I refactored my foundry binaries derivation so it's much cleaner now and can be easily updated with latest nightlies using ./update.sh: https://github.com/shazow/nixfiles/tree/master/flakes/foundry

I'm considering putting this in another repo that is just a nix overlay, and could setup a Github CI to auto-update regularly, but probably won't do it unless other people are relying on this (feel free to ping if that's the case). :) For now I just run the update script when it's convenient for me.

(For non-nixers: this makes the equivalent of foundryup to be nix flake update in your respective project.)

@gakonst
Copy link
Member

gakonst commented Feb 5, 2022

Separating it in an isolated Nix package that people can easily install would be great! Thanks for taking the time. We should then add it to the Readme

@shazow
Copy link
Author

shazow commented Feb 10, 2022

I think I have a working setup here: https://github.com/shazow/foundry.nix

Should auto-update daily. Let me know if anyone has problems!

I'd still like to track this issue as the solc thing is still a problem on NixOS, and also I'd like to get a working from-source build derivation once all of the dependencies have been pinned.

@sambacha
Copy link
Contributor

sambacha commented Nov 2, 2022

Maybe this can be resolved upstream in the forthcoming release for solidity as they are overhauling their build pipeline/automation, ethereum/solidity#13610

@neirenoir
Copy link

Given the Solidity pipeline cleanup does not seem to be going as fast as we would wish, would it make sense to add a compilation flag capable of configuring a hook that invoked a command or shell script after solc binary download? It's a quick and dirty fix, but it would solve this issue.

@hellwolf
Copy link
Contributor

hellwolf commented Jul 1, 2023

  1. I use nix-ld, so this seems not a problem. And I specify solc_version in the foundry.toml so that foundry (setup using foundry.nix project by shazow) wouldn't attempt to download it all the time.

  2. https://github.com/hellwolf/solc.nix this flake may help you to get different version of solc locally available. And with FOUNDRY_SOLC_VERSION environment variable you can choose solc-* binary you need; we use that in our CI pipeline.

@beeb
Copy link
Contributor

beeb commented Jul 23, 2023

I think I have a working setup here: https://github.com/shazow/foundry.nix

@shazow thank you so much for the overlay! Saved me a bunch of time and headaches trying to make one myself <3

@shazow
Copy link
Author

shazow commented Jul 23, 2023

@beeb Yay you're welcome!

Keep an eye on https://github.com/nix-community/ethereum.nix, we might merge into there once foundry has persistent tagged versions. Lots of other goodies there too!

@gakonst
Copy link
Member

gakonst commented Aug 18, 2023

Is this working well enough? Should we close this? Should we highlight this better in the book @shazow ?

@shazow
Copy link
Author

shazow commented Aug 21, 2023

@gakonst Ideally I'd prefer to have versioned tagged releases so that we can do cached builds from source on NixOS. Right now it's fairly hacky (elf patching binaries).

I do agree it's a good idea to document this in the book. Should I add it to https://github.com/foundry-rs/book/blob/master/src/getting-started/installation.md?

@sambacha
Copy link
Contributor

sambacha commented Dec 3, 2023

we will not be forgotten

@sambacha
Copy link
Contributor

https://discourse.nixos.org/t/nix-ld-rs-testers-wanted/42145

"Nix-LD-RS provides a shim layer that allows users to specify the necessary libraries for each executable and improves the user experience by allowing users to easily run binaries from third-party sources and proprietary software"

Testers are wanted, @shazow , see the repo at: https://github.com/Mic92/nix-ld

@Padraic-O-Mhuiris
Copy link

Padraic-O-Mhuiris commented May 6, 2024

I came across this thread after a lot of digging and experimenting with foundry + solc + nix. I've been having a difficult time making foundry reconcile different solc versions and have been following svm-rs path handling.
I was thinking that if we emulate the svm folder structure while symlinking to the static solc.nix versions, it should be sufficient to run forge build --offline. @shazow and @hellwolf, have ye come across a solution for something like this?

{
  pkgs,
  lib,
  fetchFromGitHub,
  ...
}:

pkgs.stdenv.mkDerivation rec {
  pname = "contracts-bedrock";
  version = "1.4.0-rc.2";

  src = fetchFromGitHub {
    owner = "ethereum-optimism";
    repo = "optimism";
    rev = "op-contracts/v${version}";
    hash = "sha256-ryiXWuH3vIjOdMrKSeIeJOWd+7X+hTaDnWx/ugoUI/Q=";
    fetchSubmodules = true;
  };

  nativeBuildInputs = with pkgs; [ foundry-bin ];

  unpackPhase = ''
    cp $src/packages/contracts-bedrock/foundry.toml .
    cp -r $src/packages/contracts-bedrock/src .
    cp -r $src/packages/contracts-bedrock/test .
    cp -r $src/packages/contracts-bedrock/scripts .
    cp -r $src/packages/contracts-bedrock/lib .

    TEMP=$(mktemp -d)

    mkdir -p $TEMP/svm
    touch $TEMP/svm/.global-version

    mkdir -p $TEMP/svm/0.8.15
    ln -s ${lib.getExe pkgs.solc_0_8_15} $TEMP/svm/0.8.15/solc-0.8.15

    mkdir -p $TEMP/svm/0.8.24
    ln -s ${lib.getExe pkgs.solc_0_8_24} $TEMP/svm/0.8.24/solc-0.8.24

    mkdir -p $TEMP/svm/0.8.0
    ln -s ${lib.getExe pkgs.solc_0_8_0} $TEMP/svm/0.8.0/solc-0.8.0

    XDG_DATA_HOME=$TEMP
  '';

  buildPhase = ''
    forge build --offline
  '';

  meta = with lib; {
    description = "Optimism is Ethereum, scaled.";
    homepage = "https://optimism.io/";
    license = with licenses; [ mit ];
    platforms = [ "x86_64-linux" ];
  };
}

Unfortunately, the above is failing with errors:

Encountered invalid solc version in scripts/Artifacts.s.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/ChainAssertions.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Chains.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Config.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Deploy.s.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/DeployConfig.s.sol: No solc version installed that matches the version requirement: =0.8.15
Encountered invalid solc version in scripts/DeployOwnership.s.sol: No solc version installed that matches the version requirement: ^0.8

@Padraic-O-Mhuiris
Copy link

Fixed with missing export and some util functions, could probably be more polished but I think it works as a general solution

{
  pkgs,
  lib,
  fetchFromGitHub,
  ...
}:
let

  mkSolcSvmInstallation =
    solcPkg:
    let
      solcBinPath = lib.getExe solcPkg;
      version = lib.lists.last (
        lib.strings.splitString "-" (lib.lists.last (lib.splitString "/" solcBinPath))
      );
    in
    ''
      mkdir -p $XDG_DATA_HOME/svm/${version}
      ln -s ${solcBinPath} $XDG_DATA_HOME/svm/${version}/solc-${version}
    '';

  mkSolcSvmDataDir =
    solcPkgs:
    let
      solcInstalls = lib.strings.concatLines (lib.lists.forEach solcPkgs mkSolcSvmInstallation);
    in
    ''
      export XDG_DATA_HOME=$(mktemp -d)
      mkdir -p $XDG_DATA_HOME/svm
      touch $XDG_DATA_HOME/svm/.global-version
      ${solcInstalls}
    '';
in
pkgs.stdenv.mkDerivation rec {
  pname = "contracts-bedrock";
  version = "1.4.0-rc.2";

  src = fetchFromGitHub {
    owner = "ethereum-optimism";
    repo = "optimism";
    rev = "op-contracts/v${version}";
    hash = "sha256-ryiXWuH3vIjOdMrKSeIeJOWd+7X+hTaDnWx/ugoUI/Q=";
    fetchSubmodules = true;
  };

  nativeBuildInputs = with pkgs; [ foundry-bin ];

  unpackPhase = ''
    cp $src/packages/contracts-bedrock/foundry.toml .
    cp -r $src/packages/contracts-bedrock/src .
    cp -r $src/packages/contracts-bedrock/test .
    cp -r $src/packages/contracts-bedrock/scripts .
    cp -r $src/packages/contracts-bedrock/lib .

    ${mkSolcSvmDataDir (
      with pkgs;
      [
        solc_0_5_17
        solc_0_8_15
        solc_0_8_19
        solc_0_8_24
        solc_0_8_25
      ]
    )}
  '';

  buildPhase = ''
    forge build --offline
  '';

  installPhase = ''
    mkdir -p $out
    cp foundry.toml $out/
    cp -r src $out/
    cp -r test $out/
    cp -r scripts $out/
    cp -r lib $out/
    cp -r forge-artifacts $out/
    cp -r cache $out/
  '';

  meta = with lib; {
    description = "Optimism is Ethereum, scaled.";
    homepage = "https://optimism.io/";
    license = with licenses; [ mit ];
    platforms = [ "x86_64-linux" ];
  };
}

@shazow
Copy link
Author

shazow commented May 7, 2024

@Padraic-O-Mhuiris Very cool! Any interest in adding that as a helper to https://github.com/shazow/foundry.nix? Or maybe even https://github.com/nix-community/ethereum.nix

@Padraic-O-Mhuiris
Copy link

Padraic-O-Mhuiris commented May 7, 2024

@shazow I'd love to when I have the time, maybe over the weekend, it actually could be a lot more polished where a bit more nix lib code can be used to "grep" for the pragma line in each solidity contract and use that to build the solc package list.

Also to make it even more generalised is utilise the foundry.toml to comprehend what relevant solidity files should be tangled out. There would be complications as some people may use npm/yarn/pnpm to pull in 3rd party contracts.

Additionally, I never understood why smart contract packaging/distribution is not very well standardised, between submodules and npm it just seems messy and maybe something nix can improve upon.

@zerosnacks zerosnacks added this to the v1.0.0 milestone Jul 26, 2024
@zerosnacks zerosnacks added the T-blocked Type: blocked label Jul 31, 2024
@zerosnacks
Copy link
Member

zerosnacks commented Jul 31, 2024

Tagging as effectively blocked by #3895 (which will be included as part of the 1.0 milestone)

@zerosnacks
Copy link
Member

Hi all, would be great to get your input on this PR: #9260

Would this be a valuable addition unblocking you?

@shazow
Copy link
Author

shazow commented Nov 4, 2024

@zerosnacks Thanks for the ping! Looks like a solid improvement for adding support to build foundry from source. I'd say #3895 is still blocking for proper upstream distribution/package management support (not just nixpkgs, but any other distribution too), but this is a worthy improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-releases Area: releases/packaging T-blocked Type: blocked
Projects
Status: Todo
Development

No branches or pull requests

10 participants