-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add risc-v support for eth-docker #1873
Comments
Current state of building and running ETH execution clients on RISC-V:
|
Current state of building and running ETH consensus clients on RISC-V:
|
For Prysm -> Bazel issue created: bazelbuild/bazel#23018 |
Lodestar: After deciding to not install the binaries for the Electron dependency to see how far I can go, I reached the big blocker, which is classic-level (a LevelDb wrapper) - no support for RISC-V in the assembly code. PR was created to see if it fixes the issue: Level/classic-level#94. One note here, Lodestar depends on a classic-level, which depends on leveldb. Leveldb version used on classic-level seems to be 7 years old (if I am not wrong): https://github.com/google/leveldb/commits/v1.20, see https://github.com/Level/classic-level/commits/main/deps/leveldb. |
Nimbus - DOES NOT WORK: Grandine - DOES NOT WORK: |
Erigon - DOES NOT WORK: github.com/prysmaticlabs/gohashtree@v0.0.3-alpha.0.20230502123415-aafd8b3ca202/hash.go:77:5: undefined: supportedCPU. From how it looks like, the supported amd64 and arm64 arches have the code written in assembly, which is going to be quite a challenge for RISC-V: https://github.com/prysmaticlabs/gohashtree/blob/main/hash_arm64.s |
With @haurog's fix, here is a Dockerfile to build Nimbus beacon/validator: FROM alpine:edge
RUN apk update
RUN apk add nim
RUN nim --version
RUN apk update && apk add --no-cache make gcc musl-dev linux-headers git bash git-lfs
WORKDIR /usr/src
RUN bash -c "git clone --recurse-submodules -j8 https://github.com/lazyprogrammerio/nimbus-eth2 nimbus-eth2"
RUN bash -c "cd nimbus-eth2 && make USE_SYSTEM_NIM=1 -j$(nproc) update"
RUN bash -c "cd nimbus-eth2 && make USE_SYSTEM_NIM=1 -j$(nproc) nimbus_beacon_node nimbus_validator_client" Output:
|
Docker file to build Nimbus for eth-docker
|
I've adjusted Dockerfile.source for Nimbus to build on alpine:3. Test that on your RISC-V machine, if you would. Can you also get me output of |
Went for it by looking for |
You'd need to adjust |
Here is the output from |
Thanks. I could adjust the grep to riscv64 as other riscv architectures won’t work with existing docker images. But, no rush. Let’s see whether it works at all, first |
Checking available products. Milk-V Jupiter seems likely with NVMe and 16 GiB RAM. Ditto Sifive HiFive Unmatched Rev. B Still hard mode compared to Odroid H4 (Ultra), but doable. LicheePi has no NVMe from what I can see, not a good choice. |
That pretty much is our conclusion as well. The boards we have access to at the moment are rather to see if we actually get anything running. Maybe there is a chance to split consensus and execution to two boards, but it still is gonna be difficult. Unfortunately the Jupiter is not widely available at the moment and there are only some preorder units that have been sent out. The development of new CPUs and boards seem to be very fast and in a year or so we might be in a very different position. Ideally we would have clients ready to be used by then. |
Did a pull request for nimbus: status-im/nimbus-eth2#6439 If accepted and merged to the stable branch we will be able to directly build from their repo. The build script automatically checks if it is being built on a risc-v board and we do not have to change any build parameters. |
Nice! Does ethd config detect riscv and offer only nimbus and Geth, in your testing? |
I think @lazyprogrammerio does all the configurations in the .env file directly. As far as I know, nothing has yet been changed in ethd config. |
ethd config already detects riscv and acts accordingly. It offers nimbus and Geth and sets Dockerfile.source. It doesn’t change the source repo as that’s maybe not necessary once your pr has been accepted. that code hasn’t been tested as I don’t have a riscv |
Ah, I see totally forgot that you implemented that already. Thanks for reminding me. @lazyprogrammerio have you tested it? Otherwise I will test it tomorrow. |
@yorickdowne, I just tested it. The config works. dockerfile.source is set, but accidentally you set it for nethermind (NM) instead of Nimbus (NIM). After the config finishes it fails with the following error:
I guess there are no docker entries for riscv64 for most needed images. |
Got it thanks, I’ll fix that! Yes indeed. Arm64 is rare, riscv64 is not a thing. The clients will need to be source compiled locally until / unless some teams start publishing riscv64 images |
I tested some more execution client builds on riscv. I follow the docs from each project to build the client locally. BESU: builds, but when running is missing a library 'ckzg4844jni' might be: https://github.com/ethereum/c-kzg-4844. Needs further investigation. Nethermind: no dotnet available on device. Dotnet has been built for riscv, might need to install manually. Reth: builds, but fails starting:
Might be due to disk space limitations. Needs to be tested again with an actual ssd. To conclude all the build tests: |
While trying to get to the bottom of the lighthouse build failure I found someone from flashbots to try to build clients on risc-v: RustCrypto/utils#1087 I will try to contact them. |
Did a pull request for the failing library in the lighthouse build. Lets hope this will fix the build: sigp/ethereum_hashing#8 |
Lighthouse builds locally with a lot of patching and upgrading dependencies. Will have to see what the best course of action is to get these changes into lighthouse and its dependencies. |
Got a board coming from aliexpress. I think to get besu working will just be a matter of locally building https://github.com/Consensys/jc-kzg-4844/ on a risc-v system, and using that in the besu build. That lib currently only publishes packages for x86_64 and arm64, but I suspect it should build fine on an armbian risc-v system. That would use java-native for things like secp256k1, but that should get the ball rolling (albeit slowly until we get besu-native support for risc-v). |
Linking this document here, if anyone needs it in the future. It compiles all the knowledge gathered in the last few weeks related to execution/consensus usage / support / hacks, board kernels and gotchas/quirks, OS support and more: |
I wrote an issue and a first pull request to lighthouse to make them compatible with RISC-V: sigp/lighthouse#6297 |
Hi Gary, this would apply to Teku as well, right? (Getting " |
Not sure if this is the same issue @lazyprogrammerio mentioned when they tried it: #1873 (comment) |
Do you want to offer that as a PR to the docs? |
Rocksdb compiles on Banana Pi F3 but Teku needs some additional compilation config for rebuilding it with native Rocksdb support;
Asked in Teku Discord but no response so far.
Compiling directy with
|
The new nimbus release (24.8.0) can now be built on RISC-V out of the box. They included my PR. We can now build nimbus directly from stable releases. |
I did a PR in the libp2p library to make it compatible with risc-v: libp2p/rust-libp2p#5590 |
Thanks everybody, this is lovely work. People start to notice and contribute, this is a very powerful idea. I've been learning and playing around: a16z/helios#370 Also, I want to use this jenkins project for nightly tests to make sure we don't regress: |
Update on hardware:
Milk-V is likely the best choice if you can get it, as the CPU is metal-enclosed and easy to cool These do not sync mainnet yet, the CPU is too slow. Hardware coming out in 2025 should bring it to parity or near-parity with ARM64 boards like the RK3588 ones. |
@come-maiz Thanks for the work, I see that some of the PRs are already merged. There was a DevCon presentation on the topic of RISC-V too: https://app.devcon.org/schedule/J3SWYT Currently, only geth offers RISC-V images: https://registry.hub.docker.com/r/ethereum/client-go Thanks alot! |
Please share the video of the presentation once it's out :) |
Hello, the presentation video is here at https://app.devcon.org/schedule/J3SWYT. Thanks! |
As the upstream software RISC-V support has come a long way, there is the real possibility of a dockerized approach to Ethereum staking on RISC-V boards like Sifive, Milk-V, LicheePI.
The text was updated successfully, but these errors were encountered: