code: 38, kind: Unsupported

[tiagorvmartins]

Hello there, I am trying to run another docker container image in GSC and having the following output on the container logs:

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Error: Io(Os { code: 38, kind: Unsupported, message: "Function not implemented" })

The docker image that I am graminizing has a few files that must exist on the final gsc image.
I am wondering, if some file is not being found...
The docker image when running contains the following folder/files structure

/app-bin
/app/ -> multiple files inside
/appconfig/ -> multiple files inside

I tried using gsc build -d such that I would get all the logs, but even using that, log output on top is everything I got.

The base image dockerfile I am trying to graminize is using Rust, and after doing some search here on gramine issues, it can be possible that some semaphores functionality might be causing the issue I am seeing? [link]

Below you can also find my current manifest file:

sys.enable_extra_runtime_domain_names_conf = true
sgx.edmm_enable = false
sgx.nonpie_binary = true
sgx.max_threads = 8

sgx.trusted_files = [
  "file:/app-bin",  
  "file:/app/",
  "file:/app-config/",
]

loader.env.RUST_BACKTRACE = "full"
loader.env.MALLOC_ARENA_MAX = "1"
sys.insecure__allow_eventfd = true

Any clues what can be the problem here? or shed me some light into the issue?
Thank you in advance!

[dimakuv]

I tried using gsc build -d such that I would get all the logs, but even using that, log output on top is everything I got.

This shouldn't happen...

I suggest to remove all Docker images. Maybe you have some dangling old Docker images, and the GSC-built image wasn't rebuilt?

I strongly suggest to build and run with gsc build -d. It definitely must print the Gramine logs.

Btw, you don't need to specify sgx.trusted_files at all in the manifest file -- GSC does it for you.

Error: Io(Os { code: 38, kind: Unsupported, message: "Function not implemented" })

This error means that some system call is not implemented in Gramine. It's sad that your Rust workload doesn't print which syscall exactly, but you'll find it in the Gramine logs.

[dimakuv]

I didn't find the Function not implemented error in this log, instead I see this error:

Error: Service(Client(Backend("IO error: While open a file for appending: /tmp/substrateoCAB5K/chains/dev/db/full/LOG: Permission denied")))

This makes sense because Gramine also complains about this file:

(pal_files.c:77:file_open) warning: Disallowing access to file '/tmp/substrateoCAB5K/chains/dev/db/full/LOG'; file is not trusted or allowed.
(libos_parser.c:1628:buf_write_all) [P1:T1:madara-bin] trace: ---- openat(AT_FDCWD, "/tmp/substrateoCAB5K/chains/dev/db/full/LOG", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0644) = -13

So looks like the root problem is that you didn't enable this file as sgx.allowed_files.

[tiagorvmartins]

Hi @dimakuv just wanted to give feedback,
The problem was related to a folder not allowed in the manifest.

However now, I am having another problem creating a memory pool mapping,

Error in logs:
Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Cannot allocate memory (os error 12)")

Did some search and found this:
#353

Shared untrusted memory via mmap() [ in progress ]

I might be wrong, but seems like mmap is not yet supported?
If that's so, is there any workaround?
Thank you!

[dimakuv]

I might be wrong, but seems like mmap is not yet supported?

The classic mmap() syscall is supported, but mmap(MAP_SHARED) is indeed not supported (and will be supported only for very specific use cases, like secure communication with devices!).

In your case, the problem is not about shared mmap, but about allocating an insanely large memory region -- 0x3080000000 bytes is ~194GB of memory. I'm sure your sgx.enclave_size is much smaller than this number. And even if you will increase the enclave size to e.g. sgx.enclave_size = "256GB", this still sounds wrong.

Why would your workload require 194GB of memory to be allocated? Is it really that big, or does your workload/framework rely on some hacks and tricks with memory? You need to do some research and understand what is this and how to make this requested memory size smaller.

[tiagorvmartins]

Thank you for your answer!

I believe it can be some arguments not being feeded in correctly to the container and because of that changing the behavior of the system being tested,
can you confirm something please?

I am building the graminized container with --insecure-args , which I understand is used to pass in arguments to the original container.

The container has a single entrypoint which points to the binary being launched.

And I have this in the docker compose which uses the gsc container:

image: gsc-container
command:
- "--rpc-external"
- "--rpc-methods=unsafe"
- "--dev"

In logs I can clearly see:

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
+ set -e
++ find /gramine/meson_build_output/lib -type d -path '*/site-packages'
+ export PYTHONPATH=:/gramine/meson_build_output/lib/python3.8/site-packages
+ PYTHONPATH=:/gramine/meson_build_output/lib/python3.8/site-packages
++ find /gramine/meson_build_output/lib -type d -path '*/pkgconfig'
+ export PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
+ PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
+ '[' -z '' ']'
+ gramine-sgx /gramine/app_files/entrypoint --rpc-external --rpc-methods=unsafe --dev

gramine-sgx /gramine/app_files/entrypoint --rpc-external --rpc-methods=unsafe --dev

So it seems like the commands from the docker compose are being sent correctly?
And might not be the case

[dimakuv]

So it seems like the commands from the docker compose are being sent correctly?

Yes, looks like the command-line arguments are correctly propagated into Gramine.

Why do you think it has to do with command-line arguments? I think it's just about your application itself.

[haraldh]

wasmtime is doing some mmap tricks, when compiled with the "pooling-allocator" rust feature
see bytecodealliance/wasmtime#3697

My recommendation is to turn that feature off.

Additionally to avoid memfd():

let mut config = wasmtime::Config::new();
config.memory_init_cow(false);

and for a while without EDMM we had:

let mut config = wasmtime::Config::new();
config.wasm_multi_memory(true);
config.static_memory_maximum_size(0);
config.static_memory_guard_size(0);
config.dynamic_memory_guard_size(0);
config.dynamic_memory_reserved_for_growth(16 * 1024 * 1024);

These have to be changes in the wasmtime binary itsself or the runtime using the wasmtime crate.

[kailun-qin]

If it was caused by what @haraldh hinted, I think changes in the wasmtime binary itsself or the runtime using the wasmtime crate may not be necessary.

Pls kindly try running with e.g., --wasmtime-instantiation-strategy=legacy-instance-reuse, which turned off both pooling and cow in substrate WASM runtime instantiation method (see https://github.com/paritytech/substrate/blob/b9cbb0f4281ba20bd50eab5256230c1586f50c5f/client/executor/wasmtime/src/runtime.rs#L336-L342).

[tiagorvmartins]

Thank you for the help @haraldh and @kailun-qin,
I have been looking into this, initially was trying to build a version of substrate using the suggestion from @haraldh but made a pause on that, and tried running the binary with the suggestion from @kailun-qin.

However no luck yet,
It's still failing with a different amount of memory trying to be allocated,

Error: Service(Client(VersionInvalid("Other error happened while constructing the runtime: failed to instantiate a new WASM module instance: mmap failed to allocate 0x200000000 bytes: Cannot allocate memory (os error 12)")))
However, the error is different from before, now it's not the wasmtime engine but a specific wasm module instance.

Regarding the flags use_pooling and use_cow on the link you provided @kailun-qin I understand that the instantiation method will differ, but it seems like its not using any of the following configurations, describe below, which apparently are being used by @haraldh when not using EDMM - which in fact I am also not using (currently, I have: sgx.edmm_enable = false).

config.static_memory_maximum_size(0);
config.static_memory_guard_size(0);
config.dynamic_memory_guard_size(0);
config.dynamic_memory_reserved_for_growth(16 * 1024 * 1024);

Let me know if you have any other suggestions,
Thank you again!

[kailun-qin]

but it seems like its not using any of the following configurations, describe below, which apparently are being used by @haraldh when not using EDMM

I guess those do impact performance but may not be required, since 0x200000000 bytes is 8GB of memory which is relatively small. May I know what's the enclave size (sgx.enclave_size) currently set in your manifest? You'd probably want to increase it and give it another shot.

Let me know if you have any other suggestions

If it's still not working w/ the above. Unfornately, I don't see/know how to configure tunable static/dynamic wasmtime memory settings directly from substrate (sorry that I'm not an expert). If you do want to try or need to tune them in the future, it looks like the approach would be as @haraldh suggested -- These have to be changes in the wasmtime binary itsself or the runtime using the wasmtime crate.

[tiagorvmartins]

@kailun-qin
This is the enclave_size being used, you can check the manifest below:

sys.enable_extra_runtime_domain_names_conf = true
sgx.edmm_enable = false
sgx.nonpie_binary = true
sgx.max_threads = 32
sgx.enclave_size = "16G"

sgx.trusted_files = [
  "file:/app-bin",  
  "file:/app/",
  "file:/app-specs/",
  "file:/tmp/"
]

sgx.allowed_files = [
  "file:/tmp/",  
]

loader.env.RUST_BACKTRACE = "full"
loader.env.MALLOC_ARENA_MAX = "1"
sys.insecure__allow_eventfd = true

I can't increase the enclave_size more as the host ram is only 32GB,
Also using ./test-sgx
I have the following:

size of EPC section in Processor Reserved Memory, 16384 M

Don't know what else to try,
Do you think trying using EDMM can have different result in this scenario where wasm is trying to allocate large chunks of virtual memory?
Thank you

[dimakuv]

Do you think trying using EDMM can have different result in this scenario where wasm is trying to allocate large chunks of virtual memory?

Yes, that's what EDMM was designed for :)

[tiagorvmartins]

I did an attempt running with EDMM, but seems like the driver doesn't support it.

I ran:
$ cpuid | grep SGX2

And I got the output:

SGX2 supported                         = true
SGX2 supported                         = true
SGX2 supported                         = true
SGX2 supported                         = true

Is it because in the gramine docs it states that EDMM support started on linux kernel 6?

And my kernel according to

$ uname -r

states:

5.15.0-1041-azure

I believe then I have to update the kernel to linux 6 or maybe is there anything I should install to give support to EDMM with the current kernel?

Thank you!

[dimakuv]

I believe then I have to update the kernel to linux 6 or maybe is there anything I should install to give support to EDMM with the current kernel?

Yes, you should update to the Linux v6.0 or above. I think nothing else will be required.

[yingjun-wu]

hello, I also encountered this problem recently. After starting with --wasmtime-instantiation-strategy=recreate-instance-copy-on-write, the block generation time and verification time of Substrate became very long. Is there any way to solve this problem?

[tiagorvmartins]

Hello @dimakuv, as feedback below:

the EDMM feature with increasing memory to match the RAM of the host helped removing the previous error.
(Without EDMM I wasn't able to set the enclave_size to 32G, because it would immediately crash - consuming all host memory).

Having EDMM enabled and enclave_size 32G helped, but now a new problem arised, which I believe it's the real blocker, and won't be able to proceed anymore, but would require confirmation from you or someone if possible.

Here is the error:

Can't listen on /ip6/::/tcp/30333 because: Other(Custom { kind: Other, error: Custom { kind: Other, error: Other(Left(Left(Left(Left(Right(Transport(Os { code: 97, kind: Uncategorized, message: "Address family not supported by protocol" }))))))) } })

Doing a quick search I found this:
#948 (comment)

And I am wondering, if my error is related to the fact that netlink sockets are not supported in Gramine.
Would you care to elaborate more on this error I am facing, in case you know it?

Thank you!

[dimakuv]

Can't listen on /ip6/::/tcp/30333 because: Other(Custom { kind: Other, error: Custom { kind: Other, error: Other(Left(Left(Left(Left(Right(Transport(Os { code: 97, kind: Uncategorized, message: "Address family not supported by protocol" }))))))) } })

This error doesn't say anywhere that it's about NETLINK sockets. Why do you think that these are NETLINK sockets?

I would suggest to enable loader.log_level = "all" and see what are the failing syscalls in Gramine that happen right-before this error message. Maybe indeed it's about trying to socket(AF_NETLINK, ...) -- and then yes, the problem is that Gramine currently doesn't support NETLINK sockets.

[tiagorvmartins]

Will get back to this issue, later in the week with further updates, thank you @dimakuv !