Implement directory mapping and WASI file functions #208

kulcsaradam · 2023-12-01T11:34:29Z

Refactor the importing of WASI functions to be clearer.
Implement path_open, fd_seek, fd_read, environ_sizes_get, environ_get for file acces.
Implement the mapping of real directories to virtual ones so that WASI can use different directories.
Add flag '--mapdirs' 'real' 'virtual' for mapping directories.
Add flag '--env' for sharing host envrionment variables.
Add flag '--help' for printing available walrus options.
Also improve random_get test to not have a result.

src/shell/Shell.cpp

clover2123 · 2023-12-07T08:54:20Z

@kulcsaradam
Would you please elaborate more about the following?

Implement the mapping of real directories to virtual ones so that WASI can use different directories.

What is the main feature of mapping directories and where is this used?

kulcsaradam · 2023-12-08T11:56:20Z

@clover2123
Of course, my sources:
https://github.com/WebAssembly/wasi-filesystem/blob/main/path-resolution.md
https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-overview.md - Section: Capability-Oriented

WASI by design has an idea, where for security reasons runtimes may not allow programs to freely move between host directories. By default they only allow files to manipulate in their own directories and in directories which they have handles to. Also, to obfuscate the system they are on, inside they use virtual directories like '/var' or '/home' which are actualy directories like 'home/user/walrus/test' for example.

Use cases are any file manipulation operations like reading from a file or writing to one.

I hope this is an acceptable answer!

zherczeg

LGTM

kulcsaradam · 2024-01-25T10:05:18Z

The pr was closed by a mistake I caused while tinkering with Visual Studio. Reopened it now.

clover2123 · 2024-02-14T09:10:55Z

src/wasi/Wasi.cpp

+        std::vector<std::string> variables = {
+            // Common Unix environment variables.
+            "PATH",
+            "LIB",
+            "PWD",
+            "SHELL",


I wonder about how did you define these common env variables
Is there any reference?

Yes, I've used these sites:
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08
https://learn.microsoft.com/en-us/windows/deployment/usmt/usmt-recognized-environment-variables
https://pureinfotech.com/list-environment-variables-windows-10/

Please tell me if there are any more that I should add or if I should remove some

clover2123 · 2024-02-14T10:15:03Z

src/shell/Shell.cpp

+                preopen.real_path = reinterpret_cast<char*>(calloc(1, it->length() + 1));
+                it->nonstd::sv_lite::string_view::copy(const_cast<char*>(preopen.real_path), it->length());
+
+                it++;
+
+                preopen.mapped_path = reinterpret_cast<char*>(calloc(1, it->length() + 1));
+                it->nonstd::sv_lite::string_view::copy(const_cast<char*>(preopen.mapped_path), it->length());


Will these strings allocated by calloc be correctly freed at the end of Walrus?

No, I forgot about those. After running valgrind I fixed these and also added initialization to wasi envrionment variables, so valgrind did not return any errors.

clover2123 · 2024-02-15T02:48:01Z

src/wasi/Wasi.cpp

+            "SystemDrive",
+        };
+
+        char** envp = (char**)malloc(sizeof(char**) * variables.size());


What about simply allocating a vector of std::string here?
This kind of manual memory alloc is so complicated to handle and hard for correct freeing

After spending some time thinking how I would change it to be easily used with uvwasi, I could not think of a way. If you have an idea, it would be greatly appriciated.

I see. I'll revise this later

clover2123 · 2024-02-15T02:50:36Z

src/wasi/Wasi.cpp

+    if (init_options.preopenc > 0) {
+        init_options.preopens = reinterpret_cast<uvwasi_preopen_t*>(calloc(init_options.preopenc, sizeof(uvwasi_preopen_s)));
+    }
+    for (unsigned i = 0; i < preopenDirs.size(); i++) {
+        init_options.preopens[i].mapped_path = preopenDirs[i].mapped_path;
+        init_options.preopens[i].real_path = preopenDirs[i].real_path;
+    }


For this case, preopenDirs already contains all the necessary information about directory mapping.
So, it seems that simply assigning the preopenDirs into init_options.preopens is possible like below

init_options.preopens = preopenDirs.data();

What do you think about this?

You are right and it is also easier to understand so I changed it to be this way.

Refactor the importing of WASI functions to be clearer. Implement path_open, fd_seek, fd_read, environ_sizes_get, environ_get for file acces. Implement the mapping of real directories to virtual ones so that WASI can use different directories. Add flag '--mapdirs' 'real' 'virtual' for mapping directories. Add flag '--env' for sharing host envrionment variables. Add flag '--help' for printing available walrus options. Also improve random_get test to not have a result. Signed-off-by: Adam Laszlo Kulcsar <kuladam@inf.u-szeged.hu>

clover2123

LGTM

kulcsaradam requested review from ksh8281 and clover2123 as code owners December 1, 2023 11:34

kulcsaradam force-pushed the wasi_emscripten_more branch 13 times, most recently from ec68b20 to 2c9e383 Compare December 6, 2023 12:00

clover2123 reviewed Dec 7, 2023

View reviewed changes

src/shell/Shell.cpp Outdated Show resolved Hide resolved

kulcsaradam force-pushed the wasi_emscripten_more branch 5 times, most recently from bcd1681 to be1d9fb Compare December 15, 2023 15:14

kulcsaradam force-pushed the wasi_emscripten_more branch 3 times, most recently from e556f25 to 87359e0 Compare January 10, 2024 11:31

kulcsaradam force-pushed the wasi_emscripten_more branch 4 times, most recently from ea81e17 to 478a393 Compare January 19, 2024 12:41

kulcsaradam force-pushed the wasi_emscripten_more branch from 478a393 to 27ad825 Compare January 19, 2024 12:42

zherczeg approved these changes Jan 19, 2024

View reviewed changes

kulcsaradam force-pushed the wasi_emscripten_more branch 2 times, most recently from 0ed0b74 to 88b6c6b Compare January 23, 2024 10:46

kulcsaradam closed this Jan 24, 2024

kulcsaradam force-pushed the wasi_emscripten_more branch from 88b6c6b to 1a02b92 Compare January 24, 2024 14:07

kulcsaradam reopened this Jan 25, 2024

kulcsaradam force-pushed the wasi_emscripten_more branch 2 times, most recently from b5c8f11 to 8e9d83b Compare January 26, 2024 11:24

kulcsaradam force-pushed the wasi_emscripten_more branch 2 times, most recently from 08ed3ce to 9c89d72 Compare February 13, 2024 09:07

clover2123 reviewed Feb 14, 2024

View reviewed changes

kulcsaradam force-pushed the wasi_emscripten_more branch 3 times, most recently from c059163 to fac5807 Compare February 14, 2024 11:33

clover2123 reviewed Feb 15, 2024

View reviewed changes

kulcsaradam force-pushed the wasi_emscripten_more branch from fac5807 to 7bf6c9d Compare February 15, 2024 15:27

clover2123 approved these changes Feb 16, 2024

View reviewed changes

clover2123 merged commit caf34e2 into Samsung:main Feb 16, 2024
12 checks passed

kulcsaradam deleted the wasi_emscripten_more branch February 21, 2024 10:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement directory mapping and WASI file functions #208

Implement directory mapping and WASI file functions #208

kulcsaradam commented Dec 1, 2023 •

edited

Loading

clover2123 commented Dec 7, 2023

kulcsaradam commented Dec 8, 2023 •

edited

Loading

zherczeg left a comment

kulcsaradam commented Jan 25, 2024

clover2123 Feb 14, 2024

kulcsaradam Feb 14, 2024

clover2123 Feb 14, 2024

kulcsaradam Feb 14, 2024

clover2123 Feb 15, 2024

kulcsaradam Feb 15, 2024

clover2123 Feb 16, 2024

clover2123 Feb 15, 2024

kulcsaradam Feb 15, 2024

clover2123 left a comment

Implement directory mapping and WASI file functions #208

Implement directory mapping and WASI file functions #208

Conversation

kulcsaradam commented Dec 1, 2023 • edited Loading

clover2123 commented Dec 7, 2023

kulcsaradam commented Dec 8, 2023 • edited Loading

zherczeg left a comment

Choose a reason for hiding this comment

kulcsaradam commented Jan 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clover2123 left a comment

Choose a reason for hiding this comment

kulcsaradam commented Dec 1, 2023 •

edited

Loading

kulcsaradam commented Dec 8, 2023 •

edited

Loading