-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NixOS support (and other FHS-challenged distros) #545
Comments
Unfortunately we can't statically link because of libusb. We're going to add packages on different package managers (apt, pacman, yum, brew are on my list), are there others you think we should have on our list? |
I'd love nix support. :) If you could include a I started some work on this here, but still needs more work: shazow#1 Somewhat related to #315 but I wanted to clarify that you can have this declaration in a generic way that does not require any further maintenance (unless your build process changes substantially), and you don't need to use it to manage your releases or anything like that. [Edit] The other option is to make a derivation that just downloads the latest binary releases, has all the correct inputs, and tries to patchelf the broken ldd's. It's not preferred but there are lots of binary-only packages that do this. [Edit2] Worth noting that the solc problem will still be a thing, though. (And any other on-demand fetched binaries.) |
I'll note this but do note that it is not top of my priority list currently. Feel free to open up a PR if you feel this is pressing, keeping in mind that we want to support releasechannels where possible (stable, nightly) and fixed versions for multiple package repositories, so maintainability is key here 😄 |
I'll try to make it work out of band somehow, but right now none of this works on any of my devices unfortunately. So it is equally as pressing as me (or any other NixOS/etc user) wanting to use foundry. 😅 |
One more note: I compiled forge etc from source, and I'm able to reliably patch the svm-downloaded I'd love for there to be a place to disable that svm behaviour, or an alternative approach. |
@shazow all good points. You can force local solc version usage (under Am supportive of making svm-rs more configurable, so that the paths issue does not manifest. cc @roynalnaruto Further removing dynamic libraries seems hard, I've given it a lot of effort so far, but if you have any ideas for it I'd also be supportive, if we can let the user choose from static/dynamically linked to not inflate binary sizes too much or get us in violated license land. |
@shazow Hi, if I'm not wrong, you are probably getting this error? (You could be sure by enabling tracing with Which would re-fetch the Could you try the I am not sure how to configure |
(We just fixed a bug with --no-auto-detect btw #560) |
Thanks, I'll give it a try soon! |
Confirmed that the I'm not sure if something like In a full Nix-ified version of this, you would specify the derivation and companion versions you want available, and nix would make sure it's available for you. If foundry has provisions for enabling this scenario (ie. allow for multiple versions of solc to be available for selection but without necessarily relying on a binary fetcher), then I think the rest can be done in the nix packaging out of band. Would it be possible for it to be an ENV var rather than a flag with every command? [Edit] Also would be even nicer if the solc version selection still works if present (perhaps via sub-path? or ENV var?) but it doesn't rely on hashes and fetching. |
One more update: I made a stand-alone nix flake which uses the binary releases, it lives here for now (good chance I'll move it eventually): https://github.com/shazow/nixfiles/blob/master/flakes/foundry/flake.nix This means that anyone who has A few caveats:
|
Ah figured out how to use my foundry flake as an input for my solidity project's devshell flake, in case this is useful for anyone: # flake.nix
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
foundry.url = "git+https://github.com/shazow/nixfiles?dir=flakes/foundry";
};
outputs = { self, nixpkgs, utils, foundry }:
utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs { inherit system; };
in {
devShell = with pkgs; mkShell {
buildInputs = [
solc
foundry.defaultPackage.${system}
];
shellHook = ''
export PS1="\e[1;33m\][dev]\e[1;34m\] \w $ \e[0m\]"
'';
};
});
} Then can be entered with |
You can do that via the |
Update in case anyone else is using this with nix: I refactored my foundry binaries derivation so it's much cleaner now and can be easily updated with latest nightlies using I'm considering putting this in another repo that is just a nix overlay, and could setup a Github CI to auto-update regularly, but probably won't do it unless other people are relying on this (feel free to ping if that's the case). :) For now I just run the update script when it's convenient for me. (For non-nixers: this makes the equivalent of |
Separating it in an isolated Nix package that people can easily install would be great! Thanks for taking the time. We should then add it to the Readme |
I think I have a working setup here: https://github.com/shazow/foundry.nix Should auto-update daily. Let me know if anyone has problems! I'd still like to track this issue as the solc thing is still a problem on NixOS, and also I'd like to get a working from-source build derivation once all of the dependencies have been pinned. |
Maybe this can be resolved upstream in the forthcoming release for solidity as they are overhauling their build pipeline/automation, ethereum/solidity#13610 |
Given the Solidity pipeline cleanup does not seem to be going as fast as we would wish, would it make sense to add a compilation flag capable of configuring a hook that invoked a command or shell script after solc binary download? It's a quick and dirty fix, but it would solve this issue. |
|
@shazow thank you so much for the overlay! Saved me a bunch of time and headaches trying to make one myself <3 |
@beeb Yay you're welcome! Keep an eye on https://github.com/nix-community/ethereum.nix, we might merge into there once foundry has persistent tagged versions. Lots of other goodies there too! |
Is this working well enough? Should we close this? Should we highlight this better in the book @shazow ? |
@gakonst Ideally I'd prefer to have versioned tagged releases so that we can do cached builds from source on NixOS. Right now it's fairly hacky (elf patching binaries). I do agree it's a good idea to document this in the book. Should I add it to https://github.com/foundry-rs/book/blob/master/src/getting-started/installation.md? |
we will not be forgotten |
https://discourse.nixos.org/t/nix-ld-rs-testers-wanted/42145
Testers are wanted, @shazow , see the repo at: https://github.com/Mic92/nix-ld |
I came across this thread after a lot of digging and experimenting with foundry + solc + nix. I've been having a difficult time making foundry reconcile different solc versions and have been following svm-rs path handling. {
pkgs,
lib,
fetchFromGitHub,
...
}:
pkgs.stdenv.mkDerivation rec {
pname = "contracts-bedrock";
version = "1.4.0-rc.2";
src = fetchFromGitHub {
owner = "ethereum-optimism";
repo = "optimism";
rev = "op-contracts/v${version}";
hash = "sha256-ryiXWuH3vIjOdMrKSeIeJOWd+7X+hTaDnWx/ugoUI/Q=";
fetchSubmodules = true;
};
nativeBuildInputs = with pkgs; [ foundry-bin ];
unpackPhase = ''
cp $src/packages/contracts-bedrock/foundry.toml .
cp -r $src/packages/contracts-bedrock/src .
cp -r $src/packages/contracts-bedrock/test .
cp -r $src/packages/contracts-bedrock/scripts .
cp -r $src/packages/contracts-bedrock/lib .
TEMP=$(mktemp -d)
mkdir -p $TEMP/svm
touch $TEMP/svm/.global-version
mkdir -p $TEMP/svm/0.8.15
ln -s ${lib.getExe pkgs.solc_0_8_15} $TEMP/svm/0.8.15/solc-0.8.15
mkdir -p $TEMP/svm/0.8.24
ln -s ${lib.getExe pkgs.solc_0_8_24} $TEMP/svm/0.8.24/solc-0.8.24
mkdir -p $TEMP/svm/0.8.0
ln -s ${lib.getExe pkgs.solc_0_8_0} $TEMP/svm/0.8.0/solc-0.8.0
XDG_DATA_HOME=$TEMP
'';
buildPhase = ''
forge build --offline
'';
meta = with lib; {
description = "Optimism is Ethereum, scaled.";
homepage = "https://optimism.io/";
license = with licenses; [ mit ];
platforms = [ "x86_64-linux" ];
};
} Unfortunately, the above is failing with errors: Encountered invalid solc version in scripts/Artifacts.s.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/ChainAssertions.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Chains.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Config.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Deploy.s.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/DeployConfig.s.sol: No solc version installed that matches the version requirement: =0.8.15
Encountered invalid solc version in scripts/DeployOwnership.s.sol: No solc version installed that matches the version requirement: ^0.8 |
Fixed with missing export and some util functions, could probably be more polished but I think it works as a general solution {
pkgs,
lib,
fetchFromGitHub,
...
}:
let
mkSolcSvmInstallation =
solcPkg:
let
solcBinPath = lib.getExe solcPkg;
version = lib.lists.last (
lib.strings.splitString "-" (lib.lists.last (lib.splitString "/" solcBinPath))
);
in
''
mkdir -p $XDG_DATA_HOME/svm/${version}
ln -s ${solcBinPath} $XDG_DATA_HOME/svm/${version}/solc-${version}
'';
mkSolcSvmDataDir =
solcPkgs:
let
solcInstalls = lib.strings.concatLines (lib.lists.forEach solcPkgs mkSolcSvmInstallation);
in
''
export XDG_DATA_HOME=$(mktemp -d)
mkdir -p $XDG_DATA_HOME/svm
touch $XDG_DATA_HOME/svm/.global-version
${solcInstalls}
'';
in
pkgs.stdenv.mkDerivation rec {
pname = "contracts-bedrock";
version = "1.4.0-rc.2";
src = fetchFromGitHub {
owner = "ethereum-optimism";
repo = "optimism";
rev = "op-contracts/v${version}";
hash = "sha256-ryiXWuH3vIjOdMrKSeIeJOWd+7X+hTaDnWx/ugoUI/Q=";
fetchSubmodules = true;
};
nativeBuildInputs = with pkgs; [ foundry-bin ];
unpackPhase = ''
cp $src/packages/contracts-bedrock/foundry.toml .
cp -r $src/packages/contracts-bedrock/src .
cp -r $src/packages/contracts-bedrock/test .
cp -r $src/packages/contracts-bedrock/scripts .
cp -r $src/packages/contracts-bedrock/lib .
${mkSolcSvmDataDir (
with pkgs;
[
solc_0_5_17
solc_0_8_15
solc_0_8_19
solc_0_8_24
solc_0_8_25
]
)}
'';
buildPhase = ''
forge build --offline
'';
installPhase = ''
mkdir -p $out
cp foundry.toml $out/
cp -r src $out/
cp -r test $out/
cp -r scripts $out/
cp -r lib $out/
cp -r forge-artifacts $out/
cp -r cache $out/
'';
meta = with lib; {
description = "Optimism is Ethereum, scaled.";
homepage = "https://optimism.io/";
license = with licenses; [ mit ];
platforms = [ "x86_64-linux" ];
};
} |
@Padraic-O-Mhuiris Very cool! Any interest in adding that as a helper to https://github.com/shazow/foundry.nix? Or maybe even https://github.com/nix-community/ethereum.nix |
@shazow I'd love to when I have the time, maybe over the weekend, it actually could be a lot more polished where a bit more nix lib code can be used to "grep" for the pragma line in each solidity contract and use that to build the solc package list. Also to make it even more generalised is utilise the foundry.toml to comprehend what relevant solidity files should be tangled out. There would be complications as some people may use Additionally, I never understood why smart contract packaging/distribution is not very well standardised, between submodules and npm it just seems messy and maybe something nix can improve upon. |
Tagging as effectively blocked by #3895 (which will be included as part of the |
Hi all, would be great to get your input on this PR: #9260 Would this be a valuable addition unblocking you? |
@zerosnacks Thanks for the ping! Looks like a solid improvement for adding support to build foundry from source. I'd say #3895 is still blocking for proper upstream distribution/package management support (not just nixpkgs, but any other distribution too), but this is a worthy improvement. |
Problem
Right now the shipped binaries make assumptions about where dynamically-linked libraries live. This is a problem for NixOS and other Linux distros that use less usual FHS layouts (there are dozens of us!).
patchelf
'd in a pinch)forge build
downloads asolc
binary which suffers from the same problem.I tried using
forge build --no-auto-detect
but looks like it's refusing to use my local version.Solutions
The low hanging fruit would be to build the shipped binaries as more statically linked than they are now. I do this for some other projects I maintain (mainly Go), it helps reduce all kinds of problems (can make it runnable on Alpine and other minimalist distros!) in exchange for larger binaries.
The more sustainable solution is to rely less on out-of-band binary fetching, and more on distro-friendly pre-packaging.
I understand that the current state of the project is in flux and perhaps the main demographic is people who copypasta curl-bash into their terminals (per the README), but realistically it does not need to be a big change to make this more distro packaging friendly. Even if we're not ready to start tagging releases, small things like changing the default Cargo dependencies to load specific versions rather than git main would go a long way (e.g. ethers-rs). Happy to send some PRs to this effect if it sounds desirable. :)
The text was updated successfully, but these errors were encountered: