Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c34c938
feat: Add nix flake for ffi
maru-ava Sep 26, 2025
cd6171c
fixup: Add crane for dep artifact caching and update static lib action
maru-ava Sep 30, 2025
0a7a0b0
fixup: Document flake usage
maru-ava Sep 30, 2025
93479f9
fixup: Revert ci changes
maru-ava Oct 1, 2025
ed33a9e
fixup: Add ffi jobs for nix
maru-ava Oct 1, 2025
023879c
fixup: Use dontStrip in the nix flake to match regular cargo build
maru-ava Oct 1, 2025
3099e48
fixup: Limit ffi-nix job to ubuntu to minimize runtime
maru-ava Oct 1, 2025
b96b6c7
fixup: Add script to test for build equivalency
maru-ava Oct 1, 2025
2ec10b2
fixup: Update to copy only the maxperf lib
maru-ava Oct 1, 2025
6d54892
fixup: Remove deprecated darwin stubs
maru-ava Oct 1, 2025
ca17113
fixup: Fix macos target to apply to both buildDepsOnly and buildPackage
maru-ava Oct 2, 2025
d2e7488
fixup: Add build equivalency check to ffi-nix job
maru-ava Oct 2, 2025
5bf2243
fixup: Update script to exit non zero if builds differ
maru-ava Oct 3, 2025
66fe251
fixup: Clarify comment justifying test mode
maru-ava Oct 3, 2025
18ae4dc
fixup: Add build-static-ffi cargo alias to minimize the change of div…
maru-ava Oct 9, 2025
eed24bd
fixup: Ensure flake output equivalent to firewood-go-ethhash
maru-ava Oct 9, 2025
81f4794
fixup: Avoid unnecessary rebuilding in build equivalency test
maru-ava Oct 9, 2025
4e66eb4
fixup: Update to use avalanchego's golang flake for 1.24.8
maru-ava Oct 15, 2025
3b2c6ef
fixup: Force sequential jemalloc build for x86_64 reproducibility
maru-ava Oct 15, 2025
cda7583
fixup: Configure dependabot for security updates for github actions
maru-ava Oct 17, 2025
24b54e7
fixup: Minimize duplication in build equivalency script
maru-ava Oct 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[alias]
build-static-ffi = "build --frozen --profile maxperf --package firewood-ffi --features ethhash,logger"
5 changes: 5 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,8 @@ updates:
time: "05:00"
timezone: "America/Los_Angeles"
open-pull-requests-limit: 0 # Disable non-security version updates
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: weekly
open-pull-requests-limit: 0 # Disable non-security version updates
2 changes: 1 addition & 1 deletion .github/workflows/attach-static-libs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:
run: cargo fetch --locked --verbose

- name: Build for ${{ matrix.target }}
run: cargo build --frozen --profile maxperf --features ethhash,logger --target ${{ matrix.target }} -p firewood-ffi
run: cargo build-static-ffi --target ${{ matrix.target }}

- name: Upload binary
uses: actions/upload-artifact@v4
Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,25 @@ jobs:
# cgocheck2 is expensive but provides complete pointer checks
run: GOEXPERIMENT=cgocheck2 TEST_FIREWOOD_HASH_MODE=firewood go test -race ./...

ffi-nix:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 #v20
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 #v13
Comment on lines +251 to +252
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions: How and when are these hardcoded shas updated? I'm sure dependabot won't tell us when they update. Should we add something to the release checklist to see if they should be updated?

What happens if someone finds a security vulnerability in these packages?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dependabot can, in fact, tell you to update. I just checked, though, and it's not yet configured for this repo. This is what needs to be in .github/dependabot.yml:

  - package-ecosystem: github-actions
    directory: "/"
    schedule:
      interval: weekly
    open-pull-requests-limit: 0 # Disable non-security version updates

As per ava-labs/avalanchego#3822, specifying SHAs for 3rd party actions (i.e. not provided by github) is best practice to avoid supply chain attack. A high-profile incident earlier this year galvanized awareness of the problem.

Copy link
Contributor Author

@maru-ava maru-ava Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated dependabot config: dc0f60b

- name: Test build equivalency between Nix and Cargo
run: bash -x ./ffi/test-build-equivalency.sh
- name: Test Go FFI bindings
working-directory: ffi
# - cgocheck2 is expensive but provides complete pointer checks
# - use hash mode ethhash since the flake builds with `--features ethhash,logger`
# - run golang outside a nix shell to validate viability without the env setup performed by a nix shell
run: |
GOLANG="nix run $PWD#go"
cd result/ffi
GOEXPERIMENT=cgocheck2 TEST_FIREWOOD_HASH_MODE=ethhash ${GOLANG} test ./...
shell: bash

firewood-ethhash-differential-fuzz:
needs: build
runs-on: ubuntu-latest
Expand Down
3 changes: 3 additions & 0 deletions ffi/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
dbtest
_obj

# Nix output
result
146 changes: 146 additions & 0 deletions ffi/flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

140 changes: 140 additions & 0 deletions ffi/flake.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
{
# To test with arbitrary firewood versions (alternative to firewood-go-ethhash):
# - Install nix: https://github.com/DeterminateSystems/nix-installer?tab=readme-ov-file#install-nix
# - Clone firewood locally at desired version/commit
# - Build: `cd ffi && nix build`
# - In your Go project: `go mod edit -replace github.com/ava-labs/firewood-go-ethhash/ffi=/path/to/firewood/ffi/result/ffi`

description = "Firewood FFI library and development environment";

inputs = {
nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.2505.*.tar.gz";
rust-overlay.url = "github:oxalica/rust-overlay";
crane.url = "github:ipetkov/crane";
flake-utils.url = "github:numtide/flake-utils";
golang.url = "github:ava-labs/avalanchego?dir=nix/go&ref=f10757d594eedf0f016bc1400739788c542f005f";
};

outputs = { self, nixpkgs, rust-overlay, crane, flake-utils, golang }:
flake-utils.lib.eachDefaultSystem (system:
let
overlays = [ (import rust-overlay) ];
pkgs = import nixpkgs { inherit system overlays; };
inherit (pkgs) lib;

go = golang.packages.${system}.default;

rustToolchain = pkgs.rust-bin.stable.latest.default.override {
extensions = [ "rust-src" "rustfmt" "clippy" ];
};

craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain;

# Extract crate info from Cargo.toml files
ffiCargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);
workspaceCargoToml = builtins.fromTOML (builtins.readFile ../Cargo.toml);

src = lib.cleanSourceWith {
src = craneLib.path ./..;
filter = path: type:
(lib.hasSuffix "\.md" path) ||
(lib.hasSuffix "\.go" path) ||
(lib.hasSuffix "go.mod" path) ||
(lib.hasSuffix "go.sum" path) ||
(lib.hasSuffix "firewood.h" path) ||
(craneLib.filterCargoSources path type);
};

commonArgs = {
inherit src;
strictDeps = true;
dontStrip = true;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nix strips debugging details by default, which was resulting in the .a file being ~6MB instead of ~50MB.


# Build only the firewood-ffi crate
pname = ffiCargoToml.package.name;
version = workspaceCargoToml.workspace.package.version;

nativeBuildInputs = with pkgs; [
pkg-config
];

# Force sequential build of vendored jemalloc to avoid race conditions
# that cause non-deterministic symbol generation on x86_64
# MAKEFLAGS only affects make invocations (jemalloc), not cargo parallelism
# See: https://github.com/NixOS/nixpkgs/issues/380852
MAKEFLAGS = "-j1";
} // lib.optionalAttrs pkgs.stdenv.isDarwin {
# Set macOS deployment target for Darwin builds
MACOSX_DEPLOYMENT_TARGET = "13.0";
};

cargoArtifacts = craneLib.buildDepsOnly (commonArgs // {
# Use cargo alias defined in .cargo/config.toml
cargoBuildCommand = "cargo build-static-ffi";
});

firewood-ffi = craneLib.buildPackage (commonArgs // {
inherit cargoArtifacts;
# Use cargo alias defined in .cargo/config.toml
cargoBuildCommand = "cargo build-static-ffi";

# Disable tests - we only need to build the static library
doCheck = false;

# Install the static library and header
postInstall = ''
# Create a package structure compatible with FIREWOOD_LD_MODE=STATIC_LIBS
mkdir -p $out/ffi
cp -R ./ffi/* $out/ffi/
mkdir -p $out/ffi/libs/${pkgs.stdenv.hostPlatform.config}
cp target/maxperf/libfirewood_ffi.a $out/ffi/libs/${pkgs.stdenv.hostPlatform.config}/

# Run go generate to switch CGO directives to STATIC_LIBS mode
cd $out/ffi
HOME=$TMPDIR GOTOOLCHAIN=local FIREWOOD_LD_MODE=STATIC_LIBS ${go}/bin/go generate
'';

meta = with lib; {
description = "C FFI bindings for Firewood, an embedded key-value store";
homepage = "https://github.com/ava-labs/firewood";
license = {
fullName = "Ava Labs Ecosystem License 1.1";
url = "https://github.com/ava-labs/firewood/blob/main/LICENSE.md";
};
platforms = [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
};
});
in
{
packages = {
inherit firewood-ffi;
default = firewood-ffi;
};

apps.go = {
type = "app";
program = "${go}/bin/go";
};

devShells.default = craneLib.devShell {
inputsFrom = [ firewood-ffi ];

packages = with pkgs; [
firewood-ffi
rustToolchain
go
];

shellHook = ''
# Ensure golang bin is in the path
GOBIN="$(go env GOPATH)/bin"
if [[ ":$PATH:" != *":$GOBIN:"* ]]; then
export PATH="$GOBIN:$PATH"
fi

# Force sequential build of vendored jemalloc for reproducibility
export MAKEFLAGS="-j1"
'';
};
});
}
5 changes: 5 additions & 0 deletions ffi/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@ module github.com/ava-labs/firewood/ffi

go 1.24

// Changes to the toolchain version should be replicated in:
// - ffi/go.mod (here)
// - ffi/flake.nix (update golang.url to a version of avalanchego's nix/go/flake.nix that uses the desired version)
// - ffi/tests/eth/go.mod
// - ffi/tests/firewood/go.mod
toolchain go1.24.9

require (
Expand Down
Loading