go-ethereum private network sgx implementation #18

sho4510 · 2024-01-05T08:21:23Z

I am testing geth's private network using this project as a reference.

The device I'm using is an azure VM:

Linux (Free 20.04)
Size: Standard DC2s v3 (2 VCPU count, 16 GiB memory)　-- sgx2

The command line arguments to geth are:

		./geth \
		    --datadir.ancient=/go-ethereum/geth-network/miner/geth/chaindata/ancient \
		    --networkid=15 \
		    --port=30305 \
			--verbosity=5 \
			--nodiscover \
			--nat=none \
			--http \
			--http.api=eth,net,engine,admin \
			--http.port=8552 \
			--http.corsdomain=* \
			--http.addr=0.0.0.0 \
			--http.api=personal,eth,net,web3,txpool,miner,admin \
			--ws \
			--ws.api=engine,eth,web3,net,debug \
			--authrpc.jwtsecret=/etc/jwt.hex \
			--authrpc.vhosts=* \
			--authrpc.addr=0.0.0.0 \
			--rpc.allow-unprotected-txs \
			--authrpc.port=8553 \
			--allow-insecure-unlock \
			--keystore=/go-ethereum/geth-network/miner/keystore/ \
			--unlock=0x456dfBE5E94ac5915eD811423E97bcdc1C464446 \
			--password=/go-ethereum/geth-network/miner/keystore/pw1.txt \
			--mine \
			--miner.etherbase 0x456dfBE5E94ac5915eD811423E97bcdc1C464446 \
		> $@

The manifest template is:

libos.entrypoint = "{{ entrypoint }}"

loader.log_level = "debug"

loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr/lib:/usr/{{ arch_libdir }}"

loader.argv_src_file = "file:geth.args"

sys.enable_sigterm_injection = true
sys.enable_extra_runtime_domain_names_conf = true
sys.insecure__allow_eventfd = true

sgx.remote_attestation = "none"

fs.mounts = [
{ path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
{ path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
{ path = "/usr", uri = "file:/usr" },
{ path = "/etc", uri = "file:/etc" },
{ type = "tmpfs", path = "/root/.ethereum" },
{ type = "tmpfs", path = "/tmp" },
{ path = "/geth", uri = "file:geth" },
{ path = "/go-ethereum/geth-network/", uri = "file:go-ethereum/geth-network/" },
{ path = "/go-ethereum/geth-network/miner/geth/", uri = "file:go-ethereum/geth-network/miner/geth/" },
{ path = "/go-ethereum/geth-network/miner/", uri = "file:go-ethereum/geth-network/miner/" },
{ path = "/go-ethereum/geth-network/miner/geth/chaindata/", uri = "file:go-ethereum/geth-network/miner/geth/chaindata/" },
{ path = "/go-ethereum/geth-network/miner/geth/chaindata/ancient/", uri = "file:go-ethereum/geth-network/miner/geth/chaindata/ancient/" },
{ path = "/lib/ssl/certs/", uri = "file:/lib/ssl/certs/" },
{ path = "/etc/ssl/certs/", uri = "file:/etc/ssl/certs/" },
]

sgx.nonpie_binary = true
sgx.enclave_size = "{{ enclave_size }}"
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.thread_num = 16
sgx.debug = true

sgx.trusted_files = [
"file:{{ gramine.libos }}",
"file:{{ entrypoint }}",
"file:{{ gramine.runtimedir() }}/",
"file:{{ arch_libdir }}/",
"file:/usr/{{ arch_libdir }}/",
"file:geth",
"file:geth.args",
"file:/etc/ssl/certs/ca-certificates.crt",
"file:/lib/ssl/certs/",
"file:/etc/ssl/certs/",
"file:go-ethereum/geth-network/miner/keystore/",
"file:go-ethereum/geth-network/miner/geth/chaindata/",
"file:go-ethereum/geth-network/miner/geth/",
"file:go-ethereum/geth-network/miner/",
]

sgx.allowed_files = [
"file:/etc/nsswitch.conf",
"file:/etc/localtime",
"file:/etc/hosts",
"file:/etc/passwd",
"file:/etc/jwt.hex",
"file:/data",
"file:data",
]

The execution result is (excerpt of the error part):

DEBUG[01-05|07:57:30.401] Failed to decode keystore key            path=/go-ethereum/geth-network/miner/keystore/pw1.txt err="json: cannot unmarshal number into Go value of type struct { Address string \"json:\\\"address\\\"\" }"

error: Disallowing create/write/append to a trusted file 'go-ethereum/geth-network/miner/geth/chaindata/ancient/chain/bodies.cidx'

Fatal: Failed to register the Ethereum service: open /go-ethereum/geth-network/miner/geth/chaindata/ancient/chain/bodies.cidx: permission denied

The text was updated successfully, but these errors were encountered:

sho4510 · 2024-01-05T08:24:00Z

If I add --dev to the geth command line and run it in developer mode, it will work, but instead it will not be able to process transactions.

What should I do if I am not allowed to write or append files?

fnerdman · 2024-01-05T08:45:53Z

Please make yourself familiar with gramine's documentation on how it handles file access within the SGX enclave. In essence - trusted files are read only, if you want to have read write access, you can either use tmpfs for ephemeral storage or encrypted files for persistent files.

sho4510 · 2024-01-05T09:01:34Z

Please make yourself familiar with gramine's documentation on how it handles file access within the SGX enclave. In essence - trusted files are read only, if you want to have read write access, you can either use tmpfs for ephemeral storage or encrypted files for persistent files.

Thank you for your reply. In other words, is the permission denied error occurring because I am not using tmpfs or encrypted files for chaindata/ancient, which prevents me from creating/writing/appending to those files? When I tried the default builder (sepolia testnet), did it work properly because it utilizes tmpfs?

fnerdman · 2024-01-05T09:04:06Z

Yes, correct.

sho4510 · 2024-01-05T09:31:47Z

Thank you for your advice. The error to the file is gone but instead:

INFO [01-05|09:28:34.937] Loaded most recent local block           number=0 hash=d4e567..cb8fa3 td=17,179,869,184 age=54y9mo2w
WARN [01-05|09:28:34.937] Failed to load snapshot                  err="missing or corrupted snapshot"
INFO [01-05|09:28:34.942] Rebuilding state snapshot
DEBUG[01-05|09:28:34.942] Journalled generator progress            progress=empty
DEBUG[01-05|09:28:34.945] Start snapshot generation                root=d7f897..0f0544
INFO [01-05|09:28:34.945] Resuming state snapshot generation       root=d7f897..0f0544 accounts=0 slots=0 storage=0.00B dangling=0 elapsed=2.878ms
TRACE[01-05|09:28:34.945] Detected outdated state range            kind=account prefix=0x61 last=0x err="wrong root: have 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 want 0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544"
INFO [01-05|09:28:34.946] Regenerated local transaction journal    transactions=0 accounts=0
[P1:T6:geth] debug: Created sigframe for sig: 23 at 0x3484c5390 (handler: 0x477200, restorer: 0x392da2450)
INFO [01-05|09:28:34.950] Initialized transaction indexer          limit=2,350,000
[P1:T3:geth] debug: Created sigframe for sig: 23 at 0x348087390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T3:geth] debug: Created sigframe for sig: 23 at 0x348087390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T3:geth] debug: Created sigframe for sig: 23 at 0x348087390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T3:geth] debug: Created sigframe for sig: 23 at 0x348087390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T3:geth] debug: Created sigframe for sig: 23 at 0x348087390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T3:geth] debug: Created sigframe for sig: 23 at 0x348087390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T6:geth] debug: Created sigframe for sig: 23 at 0x3484c5390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T6:geth] debug: Created sigframe for sig: 23 at 0x3484c5390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T6:geth] debug: Created sigframe for sig: 23 at 0x3484c5390 (handler: 0x477200, restorer: 0x392da2450)
[P1:T6:geth] debug: Created sigframe for sig: 23 at 0x3484c5390 (handler: 0x477200, restorer: 0x392da2450)
DEBUG[01-05|09:28:34.968] Blobpool tip threshold updated           tip=1
INFO [01-05|09:28:34.968] Enabled snap sync                        head=0 hash=d4e567..cb8fa3
INFO [01-05|09:28:34.968] Chain post-merge, sync via beacon client
INFO [01-05|09:28:34.969] Unprotected transactions allowed
INFO [01-05|09:28:34.969] Gasprice oracle is ignoring threshold set threshold=2
WARN [01-05|09:28:34.975] Engine API enabled                       protocol=eth
INFO [01-05|09:28:34.975] Starting peer-to-peer node               instance=Geth/v1.13.9-unstable-c053eb71-20231230/linux-amd64/go1.21.5
[P1:T3:geth] warning: Unsupported system call flock
[P1:T6:geth] debug: Created sigframe for sig: 23 at 0x3484c5390 (handler: 0x477200, restorer: 0x392da2450)
Fatal: Error starting protocol stack: function not implemented
[P1:T3:geth] debug: ---- exit_group (returning 1)
[P1:T1:geth] debug: Creating pipe: pipe.srv:383dcf2ee8035a66726327ed26e1a38c9731ac05b336fd1558c0a28fa902c00f
[P1:T3:geth] debug: clearing POSIX locks for pid 1
[P1:T3:geth] debug: sync client shutdown: closing handles
[P1:T3:geth] debug: sync client shutdown: waiting for confirmation
[P1:T3:geth] debug: sync client shutdown: finished
[P1:T2:geth] debug: Installed async event at 1704446914985806
[P1:T4:geth] debug: Installed async event at 1704446914986691
[P1:libos] debug: IPC worker: exiting worker thread
[P1:T5:geth] debug: Creating pipe: pipe.srv:6cf4395b0ea36371fef6c9adde7346a30ff7bf05484c4e93887b4d5ee4ad2bcb
[P1:T3:geth] debug: process 1 exited with status 1
debug: PalProcessExit: Returning exit code 1

Does enclave_size cause such errors? Currently testing with 16GB.

Fatal: Error starting protocol stack: function not implemented

fnerdman · 2024-01-05T09:39:23Z

What Gramine version are you using? FLOCK syscall was implemented in this PR.

fnerdman · 2024-01-05T09:44:05Z

flock syscall needs to be enabled in the manifest: https://gramine.readthedocs.io/en/stable/manifest-syntax.html?highlight=files#experimental-flock-bsd-style-locks-support

fnerdman · 2024-01-05T09:46:44Z

Please try and see if this branch fixes the issue: https://github.com/flashbots/geth-sgx-gramine/tree/flock

sho4510 · 2024-01-05T10:01:37Z

Gramineのどのバージョンを使用していますか? FLOCK システムコールはこのPRに実装されました。

GRAMINE's version:
ii  gramine-dcap                             1.3.1-1                           amd64        A lightweight guest OS designed to run a single application (using out-of-tree DCAP kernel driver)
ii  gramine-ratls-dcap                       1.3.1-1                           amd64        DCAP-based Remote Attestation TLS (RA-TLS) library for Gramine
ii  gramine-ratls-epid                       1.3.1-1                           amd64        EPID-based Remote Attestation TLS (RA-TLS) library for Gramine

I also ran the install again:

show4510@sgx-test:~/test-geth2$ sudo apt install gramine
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  gramine-dcap
The following NEW packages will be installed:
  gramine
0 upgraded, 1 newly installed, 1 to remove and 3 not upgraded.
Need to get 0 B/2919 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
dpkg: gramine-dcap: dependency problems, but removing anyway as you requested:
 gramine-ratls-epid depends on gramine (= 1.3.1-1) | gramine-dcap (= 1.3.1-1) | gramine-oot (= 1.3.1-1); however:
  Package gramine is not installed.
  Package gramine-dcap is to be removed.
  Package gramine-oot is not installed.
 gramine-ratls-dcap depends on gramine (= 1.3.1-1) | gramine-dcap (= 1.3.1-1) | gramine-oot (= 1.3.1-1); however:
  Package gramine is not installed.
  Package gramine-dcap is to be removed.
  Package gramine-oot is not installed.

sho4510 · 2024-01-05T10:08:04Z

Where sys.experimental__enable_flock = true:

INFO [01-05|10:03:40.172] Starting peer-to-peer node               instance=Geth/v1.13.9-unstable-c053eb71-20231230/linux-amd64/go1.21.5
[P1:T1:geth] warning: Unsupported system call flock
[P1:T1:geth] warning: fallocate only supported with 0 as mode
ERROR[01-05|10:03:40.179] State snapshotter failed to iterate trie err="missing trie node 62bf3cb6efcce02d79120bf333460b24356b33949520ed06bdeb94f03be01a5c (path 0008050f) <nil>"
Fatal: Error starting protocol stack: function not implemented
[P1:T1:geth] debug: ---- exit_group (returning 1)
[P1:T6:geth] debug: Creating pipe: pipe.srv:e3eab034a334e20ce6d86f5cb45f492695b16c0691344845ee7ec5bf6bd1b978
[P1:T1:geth] debug: clearing POSIX locks for pid 1
[P1:T1:geth] debug: sync client shutdown: closing handles
[P1:T1:geth] debug: sync client shutdown: waiting for confirmation
[P1:T1:geth] debug: sync client shutdown: finished
[P1:T6:geth] debug: Installed async event at 1704449020179483
[P1:T2:geth] debug: Installed async event at 1704449020179484
[P1:T4:geth] debug: Installed async event at 1704449020179500
[P1:T5:geth] debug: Installed async event at 1704449020179515
[P1:T7:geth] debug: Installed async event at 1704449020180894
[P1:T3:geth] debug: Installed async event at 1704449020179535
[P1:libos] debug: IPC worker: exiting worker thread
[P1:T1:geth] debug: process 1 exited with status 1
debug: PalProcessExit: Returning exit code 1

ERROR[01-05|10:03:40.179] State snapshotter failed to iterate trie err="missing trie node 62bf3cb6efcce02d79120bf333460b24356b33949520ed06bdeb94f03be01a5c (path 0008050f) "

The error content seems to have changed.

fnerdman · 2024-01-05T10:10:52Z

Please update your gramine version to the recent 1.6 - https://gramine.readthedocs.io/en/stable/installation.html#install-gramine-packages

sho4510 · 2024-01-05T10:17:12Z

Updated gramine version to 1.6:

show4510@sgx-test:~/test-geth2$ dpkg -l | grep gramine
ii  gramine                                  1.6                               amd64        A lightweight usermode guest OS designed to run a single Linux application
ii  gramine-ratls-dcap                       1.6                               amd64        DCAP-based Remote Attestation TLS (RA-TLS) library for Gramine
ii  gramine-ratls-epid                       1.6

sho4510 · 2024-01-05T12:41:23Z

I reached up to another error by adding max_threads = 128,

Fatal: Error starting protocol stack: open /etc/jwt.hex: permission denied

Is etc/jwt.hex not being read?
I also tried it on a 32GB VM, but I also reached the same error with the default builder.

\
Fatal: Error starting protocol stack: open /etc/jwt.hex: permission denied
(libos_exit.c:212:libos_syscall_exit_group) [P1:T5:geth] debug: ---- exit_group (returning 1)
(libos_init.c:561:create_pipe) [P1:T8:geth] debug: Creating pipe: pipe.srv:3233547ee0a1aff6249274ae40e6acc7387ce37ec7853a25ef4437414606ecb0
(libos_fs_lock.c:785:file_lock_clear_pid) [P1:T5:geth] debug: clearing file (POSIX) locks for pid 1
(libos_sync_client.c:331:shutdown_sync_client) [P1:T5:geth] debug: sync client shutdown: closing handles
(libos_sync_client.c:346:shutdown_sync_client) [P1:T5:geth] debug: sync client shutdown: waiting for confirmation
(libos_sync_client.c:359:shutdown_sync_client) [P1:T5:geth] debug: sync client shutdown: finished
(libos_async.c:122:install_async_event) [P1:T8:geth] debug: Installed async event at 1704458122035743
(libos_async.c:122:install_async_event) [P1:T2:geth] debug: Installed async event at 1704458122035749
(libos_async.c:122:install_async_event) [P1:T6:geth] debug: Installed async event at 1704458122035765
(libos_async.c:122:install_async_event) [P1:T3:geth] debug: Installed async event at 1704458122035781
(libos_async.c:122:install_async_event) [P1:T7:geth] debug: Installed async event at 1704458122035902
(libos_async.c:158:libos_async_worker) [P1:libos] debug: Async worker thread started
(libos_ipc_worker.c:285:ipc_worker_main) [P1:libos] debug: IPC worker: exiting worker thread
(libos_init.c:561:create_pipe) [P1:T4:geth] debug: Creating pipe: pipe.srv:f82a1cacd9bbdc2b8d01de71dc6baab0a6f57295a66d8898d7e21705dd9e79ff
(libos_exit.c:58:libos_clean_and_exit) [P1:T5:geth] debug: process 1 exited with status 1
(pal_process.c:248:_PalProcessExit) debug: PalProcessExit: Returning exit code 1
\\

fnerdman · 2024-01-05T12:51:35Z

Does /etc/jwt.hex exist on the host? Does the user running gramine have access to this file?

sho4510 · 2024-01-05T14:03:50Z

※Creating etc/jwt.hex got the default builder up and running. 32GB seems to work fine.

I created etc/jwt.hex and ran it again:

openssl rand -hex 32 | sudo tee /etc/jwt.hex > /dev/null

A new error has occurred

TRACE[01-05|13:57:45.864] Decrease miner recommit interval         from=2s to=2s
INFO [01-05|13:57:45.864] Commit new sealing work                  number=1 sealhash=2fc1b7..951d74 txs=0 gas=0 fees=0 elapsed="422µs"
panic: ethash (pow) sealing not supported any more

goroutine 4249 [running]:
github.com/ethereum/go-ethereum/consensus/ethash.(*Ethash).Seal(0x32082c600?, {0x40c285?, 0x3200009a0?}, 0x320000aa8?, 0x3200009a0?, 0xd79ea16f962c501c?)
        github.com/ethereum/go-ethereum/consensus/ethash/ethash.go:84 +0x25
github.com/ethereum/go-ethereum/consensus/beacon.(*Beacon).Seal(0x3202256a0, {0x1e65ce8, 0x3204ae400}, 0x320138d20, 0x32009d340?, 0x1?)
        github.com/ethereum/go-ethereum/consensus/beacon/consensus.go:399 +0x8d
github.com/ethereum/go-ethereum/miner.(*worker).taskLoop(0x320496d80)
        github.com/ethereum/go-ethereum/miner/worker.go:623 +0x2fd
created by github.com/ethereum/go-ethereum/miner.newWorker in goroutine 1
        github.com/ethereum/go-ethereum/miner/worker.go:294 +0x918
(libos_exit.c:212:libos_syscall_exit_group) [P1:T4:geth] debug: ---- exit_group (returning 2)
(libos_init.c:561:create_pipe) [P1:T6:geth] debug: Creating pipe: pipe.srv:dc741fe782007766f698484e195129887136e3d2e712d34a7521d89c6fe237b0
(libos_fs_lock.c:785:file_lock_clear_pid) [P1:T4:geth] debug: clearing file (POSIX) locks for pid 1
(libos_sync_client.c:331:shutdown_sync_client) [P1:T4:geth] debug: sync client shutdown: closing handles
(libos_sync_client.c:346:shutdown_sync_client) [P1:T4:geth] debug: sync client shutdown: waiting for confirmation
(libos_sync_client.c:359:shutdown_sync_client) [P1:T4:geth] debug: sync client shutdown: finished
(libos_async.c:122:install_async_event) [P1:T6:geth] debug: Installed async event at 1704463065868327
(libos_async.c:122:install_async_event) [P1:T5:geth] debug: Installed async event at 1704463065868363
(libos_async.c:122:install_async_event) [P1:T2:geth] debug: Installed async event at 1704463065868390
(libos_ipc_worker.c:285:ipc_worker_main) [P1:libos] debug: IPC worker: exiting worker thread
(libos_async.c:122:install_async_event) [P1:T3:geth] debug: Installed async event at 1704463065868475
(libos_async.c:122:install_async_event) [P1:T7:geth] debug: Installed async event at 1704463065869026
(libos_exit.c:58:libos_clean_and_exit) [P1:T4:geth] debug: process 1 exited with status 2
(pal_process.c:248:_PalProcessExit) debug: PalProcessExit: Returning exit code 2

panic: ethash (pow) sealing not supported any more

sho4510 · 2024-01-05T15:04:58Z

panic: ethash (pow) sealing not supported any more

It appears that geth update no longer supports POW, so it is no longer possible to include --mine as a command line argument. After removing --mine, it started behaving like the default builder.

makefile/geth.args

--mine
--miner.etherbase 0x456dfBE5E94ac5915eD811423E97bcdc1C464446 \

https://stackoverflow.com/questions/76648917/cant-start-mining-with-geth
You can see the same problem here.
Thank you for your support!
If you know of any mining issues, we would appreciate it if you could let us know.

sho4510 closed this as completed Jan 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

go-ethereum private network sgx implementation #18

go-ethereum private network sgx implementation #18

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024 •

edited

Loading

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024

fnerdman commented Jan 5, 2024

fnerdman commented Jan 5, 2024

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024

sho4510 commented Jan 5, 2024 •

edited

Loading

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024 •

edited

Loading

go-ethereum private network sgx implementation #18

go-ethereum private network sgx implementation #18

Comments

sho4510 commented Jan 5, 2024 • edited Loading

sho4510 commented Jan 5, 2024

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024 • edited Loading

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024

fnerdman commented Jan 5, 2024

fnerdman commented Jan 5, 2024

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024

sho4510 commented Jan 5, 2024 • edited Loading

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024 • edited Loading

sho4510 commented Jan 5, 2024

fnerdman commented Jan 5, 2024

sho4510 commented Jan 5, 2024 • edited Loading

sho4510 commented Jan 5, 2024 • edited Loading

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024 •

edited

Loading

sho4510 commented Jan 5, 2024 •

edited

Loading