Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Constantine bindings for EIP196 #184

Merged
merged 33 commits into from
Aug 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
51496e8
constantine bindings
NickSneo Jun 27, 2024
079cfd3
C bindings and jni used
NickSneo Jul 12, 2024
535504d
C bindings added and build.gradle changed
NickSneo Jul 12, 2024
4d1c4e2
Merge branch 'main' into main
NickSneo Jul 12, 2024
c4c8eff
fixed errors and tests, modified build.gradle
NickSneo Jul 17, 2024
c84d5b5
added test cases from Gnark lib
NickSneo Jul 17, 2024
d6423b9
added nim install in CI
NickSneo Jul 18, 2024
4292e57
choose nim to install
NickSneo Jul 18, 2024
5c1570b
updated install nim command
NickSneo Jul 18, 2024
388eb98
Merge branch 'main' into main
NickSneo Jul 18, 2024
9ecea5e
fixed CI failing due to jfrog
NickSneo Jul 18, 2024
5cea62b
fix test task in CI
NickSneo Jul 19, 2024
6b8952f
fixed typo, corrected nimble
NickSneo Jul 19, 2024
f4877c2
Update constantine/build.gradle
NickSneo Jul 19, 2024
d2bfe1c
Update constantine/build.gradle
NickSneo Jul 19, 2024
17c57b6
Update constantine/build.gradle
NickSneo Jul 19, 2024
cf38680
Update constantine/build.gradle
NickSneo Jul 19, 2024
3dd0160
Update constantine/build.gradle
NickSneo Jul 19, 2024
52281fb
check CI/CD is passing
NickSneo Jul 19, 2024
bb7a57b
which nimble cmd test
NickSneo Jul 19, 2024
dac3084
prev implementation is working
NickSneo Jul 19, 2024
235c91a
just testing
NickSneo Jul 29, 2024
7c9f645
added constantine in build.sh
NickSneo Jul 29, 2024
b96360c
export PATH nimble
NickSneo Jul 29, 2024
18b8050
fix outout dir
NickSneo Jul 29, 2024
620b308
added prepare for linux-arm64
NickSneo Jul 29, 2024
f824860
skip for linux arm64
NickSneo Jul 29, 2024
f35318f
fix linkage and source not found
NickSneo Jul 29, 2024
23a7190
fix lib copy directories
NickSneo Jul 29, 2024
03a6057
fix for lib directory
NickSneo Jul 29, 2024
1423a90
Merge branch 'main' into main
NickSneo Aug 12, 2024
c5b504a
fixed shared lib for linux
NickSneo Aug 13, 2024
98035f1
Merge branch 'main' into main
garyschulte Aug 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ jobs:
# rust dependencies
export CARGO_HOME="$HOME/.cargo"
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain 1.75.0
# nim dependencies
export CHOOSENIM_CHOOSE_VERSION=2.0.4
curl https://nim-lang.org/choosenim/init.sh -sSf | sh -s -- -y
export PATH=$HOME/.nimble/bin:$PATH
Comment on lines +31 to +34
Copy link
Contributor

@garyschulte garyschulte Jul 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we change the image to ubuntu-23.10 , we can install nim from apt. Looks like it is version 1.16.14 though. So maybe downloading from nim-lang is a good idea. the curl-scripted kind of installation is a supply chain attack vector though. Same could be said for the installation of rust right above it 🤔

is 1.16.14 recent enough to use or is there significant benefit to 2.0.4 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is in the final-assembly step, because I checked this step is working fine and installing nim correctly.
I see steps like these in final-assembly

      - name: Download bls12-381
        uses: actions/download-artifact@v3
        with:
          name: bls12-381 native build artifacts
          path: bls12-381/build/

I checked building with nim 2.0.4 which worked fine for me.
In constantine lib - Constantine can be compiled by Nim v1.6.x, v2.0.2 and v2.0.4 but not Nim v2.0.0 (due to a compile-time evaluation crash)
But installing with apt won't change much as even choose-nim is working fine

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, I would expect it will work fine, but giving the contents of a URL the privilege to run anything in the shell is a really sketchy way to install dependencies. We are doing the same thing for rust, and we really shouldn't.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.6.x (latest is 1.6.20) works fine for Constantine and it's tested in CI. No benefits performance-wise for latest Nim versions. Constantine uses low-level Nim and has no library dependencies and even no std-lib dependencies or allocator dependencies to be extremely hard to supply-chain attack (i.e. introducing a compiler bug).
Hence the features that might justify upgrading a compiler are more related to its compile-time capabilities and type system which Constantine uses extensively.

- name: Checkout Repo
uses: actions/checkout@v4
with:
Expand Down Expand Up @@ -81,6 +85,11 @@ jobs:
with:
name: gnark native build artifacts
path: gnark/build/
- uses: actions/upload-artifact@v3.1.0
with:
name: constantine native build artifacts
path: constantine/build/

native-build-linux-arm64:
runs-on: besu-arm64
env:
Expand All @@ -91,7 +100,7 @@ jobs:
with:
submodules: recursive
- name: Build
run:
run: |
docker run -e SKIP_GRADLE=$SKIP_GRADLE -v $(pwd):/home/ubuntu ubuntu:20.04 /home/ubuntu/native-build.sh
- uses: actions/upload-artifact@v3.1.0
with:
Expand Down Expand Up @@ -121,6 +130,7 @@ jobs:
with:
name: gnark native build artifacts
path: gnark/build/

native-build-macos:
runs-on: macos-13
env:
Expand All @@ -139,6 +149,10 @@ jobs:
export PATH="$HOME/.cargo/bin:$PATH"
rustup target add x86_64-apple-darwin
rustup target add aarch64-apple-darwin
# nim dependencies
export CHOOSENIM_CHOOSE_VERSION=2.0.4
curl https://nim-lang.org/choosenim/init.sh -sSf | sh -s -- -y
export PATH=$HOME/.nimble/bin:$PATH
- name: Checkout Repo
uses: actions/checkout@v4
with:
Expand Down Expand Up @@ -173,6 +187,11 @@ jobs:
with:
name: gnark native build artifacts
path: gnark/build/
- uses: actions/upload-artifact@v3.1.0
with:
name: constantine native build artifacts
path: constantine/build/

native-build-m1:
runs-on: macos-13-xlarge
env:
Expand Down Expand Up @@ -201,6 +220,10 @@ jobs:
export PATH="$HOME/.cargo/bin:$PATH"
rustup target add x86_64-apple-darwin
rustup target add aarch64-apple-darwin
# nim dependencies
export CHOOSENIM_CHOOSE_VERSION=2.0.4
curl https://nim-lang.org/choosenim/init.sh -sSf | sh -s -- -y
export PATH=$HOME/.nimble/bin:$PATH
- name: Checkout Repo
uses: actions/checkout@v4
with:
Expand Down Expand Up @@ -248,6 +271,11 @@ jobs:
with:
name: gnark native build artifacts
path: gnark/build/
- uses: actions/upload-artifact@v3.1.0
with:
name: constantine native build artifacts
path: constantine/build/

final-assembly:
runs-on: ubuntu-20.04
needs:
Expand Down Expand Up @@ -293,6 +321,11 @@ jobs:
with:
name: gnark native build artifacts
path: gnark/build/
- name: Download constantine
uses: actions/download-artifact@v3
with:
name: constantine native build artifacts
path: constantine/build/
- name: Set up Java
uses: actions/setup-java@v4
with:
Expand Down
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,16 @@ Cargo.lock

.gradle

*.DS_Store

## build files
*.com
*.class
*.dll
*.exe
*.o
*.so
*.so

*.a
*.dylib
constantine/out
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,6 @@
[submodule "secp256r1/besu-native-ec"]
path = secp256r1/besu-native-ec
url = git@github.com:ConsenSys/besu-native-ec.git
[submodule "constantine/constantine"]
path = constantine/constantine
url = https://github.com/mratsim/constantine.git
40 changes: 40 additions & 0 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -319,13 +319,53 @@ EOF
cp libgnark_eip_196.* "$SCRIPTDIR/gnark/build/${OSARCH}/lib"
}

build_constantine() {
echo "#############################"
echo "####### build constantine ####"
echo "#############################"

# Skip if OSARCH is linux-gnu-aarch64
if [[ "$OSARCH" == "linux-gnu-aarch64" ]]; then
echo "Skipping build for OSARCH: ${OSARCH}"
return
fi
Comment on lines +327 to +331
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a CI-specific workaround? it looks like an arm64 linux host env would not be able to build constantine?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I skipped Linux arm64 build since nim installer was not compatible with it, and I didn't have an Linux arm environment to test it.
Will test and add in the follow on PR


cd "$SCRIPTDIR/constantine/constantine"

# delete old build dir, if exists
rm -rf "$SCRIPTDIR/constantine/build" || true
mkdir -p "$SCRIPTDIR/constantine/build/${OSARCH}/lib"

export PATH=$HOME/.nimble/bin:$PATH

# Build the constantine library
export CTT_LTO=false
nimble make_lib

cd "$SCRIPTDIR/constantine/"

# Compile the native library
if [[ "$OSTYPE" == "darwin"* ]]; then
cp "$SCRIPTDIR/constantine/constantine/lib/libconstantine.dylib" "$SCRIPTDIR/constantine/build/${OSARCH}/lib/"
clang -I"${JAVA_HOME}/include" -I"${JAVA_HOME}/include/darwin" -shared -o "$SCRIPTDIR/constantine/build/${OSARCH}/lib/libconstantineeip196.jnilib" ethereum_evm_precompiles.c -Iconstantine/include -I. -Lconstantine/lib -lconstantine
elif [[ "$OSTYPE" == "linux-gnu"* ]]; then
cp "$SCRIPTDIR/constantine/constantine/lib/libconstantine.so" "$SCRIPTDIR/constantine/build/${OSARCH}/lib/"
clang -I"${JAVA_HOME}/include" -I"${JAVA_HOME}/include/linux" -fPIC -shared -o "$SCRIPTDIR/constantine/build/${OSARCH}/lib/libconstantineeip196.so" ethereum_evm_precompiles.c -Iconstantine/include -I. -Lconstantine/lib -lconstantine
else
echo "Unsupported OS/architecture: ${OSARCH}"
exit 1
fi
}


build_blake2bf
build_secp256k1
build_arithmetic
build_bls12_381
build_ipa_multipoint
build_secp256r1
build_gnark
build_constantine


build_jars
Expand Down
145 changes: 145 additions & 0 deletions constantine/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
plugins {
id 'java-library'
id 'maven-publish'
id 'com.jfrog.artifactory' version '5.2.3'
}

repositories {
mavenCentral()
}

dependencies {
implementation 'net.java.dev.jna:jna:5.12.1'
testImplementation 'com.google.guava:guava:31.1-jre'
testImplementation 'io.tmio:tuweni-bytes:2.4.2'
testImplementation 'junit:junit:4.13.2'
}

java {
toolchain {
languageVersion = JavaLanguageVersion.of(21)
}
}

tasks.withType(JavaCompile) {
options.encoding = 'UTF-8'
}

def osName = System.getProperty('os.name').toLowerCase()
def osArch = System.getProperty('os.arch')
def libDir

if (osName.contains('mac') && osArch.contains('aarch64')) {
libDir = 'darwin-aarch64'
} else if (osName.contains('mac')) {
libDir = 'darwin-x86-64'
} else if (osName.contains('linux') && osArch.contains('aarch64')) {
libDir = 'linux-gnu-aarch64'
} else {
libDir = 'linux-gnu-x86_64'
}

task libCopy(type: Copy) {
from "build/${libDir}/lib/"
into "build/resources/main/lib/${libDir}"
}

processResources.dependsOn libCopy

task compileJavaSource(type: Exec) {
description = 'Compiles the Java source files'
commandLine 'javac', '-d', 'build', 'src/main/java/org/hyperledger/besu/nativelib/constantine/LibConstantineEIP196.java'
dependsOn libCopy
}

NickSneo marked this conversation as resolved.
Show resolved Hide resolved
tasks.named('test', Test) {
description = 'Runs the Java tests'
useJUnit {
include '**/*Test.class'
}
environment 'LD_LIBRARY_PATH', "${System.env.LD_LIBRARY_PATH}:build/resources/main/lib/${libDir}"
jvmArgs = ["-Djava.library.path=build/resources/main/lib/${libDir}"]
dependsOn compileJavaSource
}

jar {
archiveBaseName = 'besu-native-constantine'
includeEmptyDirs = false
manifest {
attributes(
'Specification-Title': archiveBaseName,
'Specification-Version': project.version,
'Implementation-Title': archiveBaseName,
'Implementation-Version': project.version,
'Automatic-Module-Name': 'org.hyperledger.besu.nativelib.constantine'
)
}
}

task sourcesJar(type: Jar, dependsOn: classes) {
archiveBaseName = 'besu-native-constantine'
archiveClassifier = 'sources'
from sourceSets.main.allSource
}

task javadocJar(type: Jar, dependsOn: javadoc) {
archiveBaseName = 'besu-native-constantine'
archiveClassifier = 'javadoc'
from javadoc.destinationDir
}

publishing {
publications {
mavenJava(MavenPublication) {
groupId "org.hyperledger.besu"
artifactId 'constantine'
version "${project.version}"

from components.java
artifact sourcesJar
artifact javadocJar

pom {
name = "Besu Native - ${project.name}"
description = 'Adapter for native constantine library'
url = 'http://github.com/hyperledger/besu-native'
licenses {
license {
name = 'The Apache License, Version 2.0'
url = 'http://www.apache.org/licenses/LICENSE-2.0.txt'
}
}
scm {
connection = 'scm:git:git://github.com/hyperledger/besu-native.git'
developerConnection = 'scm:git:ssh://github.com/hyperledger/besu-native.git'
url = 'https://github.com/hyperledger/besu-native'
}
}
}
}
}

def artifactoryUser = project.hasProperty('artifactoryUser') ? project.property('artifactoryUser') : System.getenv('ARTIFACTORY_USER')
def artifactoryKey = project.hasProperty('artifactoryApiKey') ? project.property('artifactoryApiKey') : System.getenv('ARTIFACTORY_KEY')
def artifactoryRepo = System.getenv('ARTIFACTORY_REPO') ?: 'besu-maven'
def artifactoryOrg = System.getenv('ARTIFACTORY_ORG') ?: 'hyperledger'

artifactory {
contextUrl = "https://hyperledger.jfrog.io/${artifactoryOrg}"
publish {
repository {
repoKey = artifactoryRepo
username = artifactoryUser
password = artifactoryKey
}
defaults {
publications('mavenJava')
publishArtifacts = true
publishPom = true
}
}
}

test {
useJUnit()
}
1 change: 1 addition & 0 deletions constantine/constantine
Submodule constantine added at b13816
51 changes: 51 additions & 0 deletions constantine/ethereum_evm_precompiles.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#include <jni.h>
#include "ethereum_evm_precompiles.h"
#include <constantine.h>
#include <stdio.h>

void printByteArray(const char* label, const byte* array, size_t len) {
printf("%s: [", label);
for (size_t i = 0; i < len; i++) {
printf("%02x", array[i]);
if (i < len - 1) {
printf(", ");
}
}
printf("]\n");
}

JNIEXPORT jint JNICALL Java_org_hyperledger_besu_nativelib_constantine_LibConstantineEIP196_ctt_1eth_1evm_1bn254_1g1add(JNIEnv *env, jobject obj, jbyteArray jr, jint r_len, jbyteArray jinputs, jint inputs_len) {
jbyte *r = (*env)->GetByteArrayElements(env, jr, NULL);
jbyte *inputs = (*env)->GetByteArrayElements(env, jinputs, NULL);

ctt_evm_status status = ctt_eth_evm_bn254_g1add((byte *)r, (ptrdiff_t)r_len, (const byte *)inputs, (ptrdiff_t)inputs_len);

(*env)->ReleaseByteArrayElements(env, jr, r, 0);
(*env)->ReleaseByteArrayElements(env, jinputs, inputs, 0);

return (jint)status;
}

JNIEXPORT jint JNICALL Java_org_hyperledger_besu_nativelib_constantine_LibConstantineEIP196_ctt_1eth_1evm_1bn254_1g1mul(JNIEnv *env, jobject obj, jbyteArray jr, jint r_len, jbyteArray jinputs, jint inputs_len) {
jbyte *r = (*env)->GetByteArrayElements(env, jr, NULL);
jbyte *inputs = (*env)->GetByteArrayElements(env, jinputs, NULL);

ctt_evm_status status = ctt_eth_evm_bn254_g1mul((byte *)r, (ptrdiff_t)r_len, (const byte *)inputs, (ptrdiff_t)inputs_len);

(*env)->ReleaseByteArrayElements(env, jr, r, 0);
(*env)->ReleaseByteArrayElements(env, jinputs, inputs, 0);

return (jint)status;
}

JNIEXPORT jint JNICALL Java_org_hyperledger_besu_nativelib_constantine_LibConstantineEIP196_ctt_1eth_1evm_1bn254_1pairingCheck(JNIEnv *env, jobject obj, jbyteArray jr, jint r_len, jbyteArray jinputs, jint inputs_len) {
jbyte *r = (*env)->GetByteArrayElements(env, jr, NULL);
jbyte *inputs = (*env)->GetByteArrayElements(env, jinputs, NULL);

ctt_evm_status status = ctt_eth_evm_bn254_ecpairingcheck((byte *)r, (ptrdiff_t)r_len, (const byte *)inputs, (ptrdiff_t)inputs_len);

(*env)->ReleaseByteArrayElements(env, jr, r, 0);
(*env)->ReleaseByteArrayElements(env, jinputs, inputs, 0);

return (jint)status;
}
Loading
Loading