Problem reading the FS using rust code

[democ98]

Description of the problem

Description of the problem:
I put the rust program into the docker container upgraded based on the official gramine docker container and run it with sgx. The program loops to detect whether a file under the file path DEFAULT_FILE_SAVE_PATH exists, and DEFAULT_FILE_SAVE_PATH is mapped to the root directory of the host by docker on a certain path A, when an untrusted program on the host computer deletes a file in path A , my rust program running in gramine can still detect the existence of the file, but it cannot open the file ( Error No such file or directory (os error 2))

Steps to reproduce

const DEFAULT_FILE_SAVE_PATH: &str = "/sgx/";
	let mut file_path = DEFAULT_FILE_SAVE_PATH.to_string() + &bs58::encode(&request_body.peer_id).into_string();
	let mut duplicate_file = 0;
	loop {
		let tmp_file_path = file_path.clone() + &format!(".{}", duplicate_file );
		let path = std::path::Path::new(&tmp_file_path);
		if path.try_exists().unwrap() && path.is_file() {
			warn!("File {:?} already exists ,it means rotator still not finished",&tmp_file_path);
			duplicate_file+=1;
			let tmp_file= match File::open(path){
				Ok(f) => {f}
				Err(e) => {
					error!("error when read file {:?},because {:?}",&tmp_file_path,e.to_string());
					continue
				}
			};
			warn!("the file {:?} in gramine system length is {}",&tmp_file_path,tmp_file.metadata().unwrap().len());
			continue
		}
		break
	};

You can try to run this code, after adding a file such as "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0" on the host host delete (by the way "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQ jj3mr.0" file is generated from within sgx), there will be an error,The following is the content of the manifest file.

[fs]
mounts = [
  { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
  { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
  { path = "/usr{{ arch_libdir }}", uri = "file:/usr{{ arch_libdir }}" },
  { path = "/sgx", uri = "file:/sgx" },
  { path = "/kaleido", uri = "file:/kaleido"},
  { path = "/kaleido/encrypted", uri = "file:/kaleido/encrypted", type = "encrypted", key_name = "_sgx_mrsigner" },
  { path = "/etc", uri = "file:/etc", type = "chroot"},
]

[sgx]
debug = true
nonpie_binary = true
enclave_size = "2G"
max_threads = 128
trusted_files = [
  "file:{{ gramine.libos }}",
  "file:{{ self_exe }}",
  "file:{{ gramine.runtimedir() }}/",
  "file:/usr{{ arch_libdir }}/",
  "file:{{ arch_libdir }}/",
]
allowed_files = [
  "file:/kaleido",
  "file:/sgx",
  "file:/etc/hosts",
  "file:/etc/resolv.conf",
]

Expected results

[2023-06-27T02:25:19Z WARN  xxx::api] File "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0" already exists ,it means rotator still not finished
[2023-06-27T02:25:19Z WARN xxx::api] the file "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0" in gramine system length is 8388608

Actual results

[2023-06-27T02:25:19Z WARN  xxx::api] File "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0" already exists ,it means rotator still not finished
[2023-06-27T02:25:19Z ERROR xxx::api] error when read file "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0",because "No such file or directory (os error 2)"

Gramine commit hash

gramineproject/gramine digest:7e5b8b6111a6

[kailun-qin]

@democ98 Thanks for the issue report!

Expected results
[2023-06-27T02:25:19Z WARN xxx::api] File "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0" already exists ,it means rotator still not finished
[2023-06-27T02:25:19Z WARN xxx::api] the file "/sgx/12D3KooWQHuPnuXCqR3c5JhfsM2K6rQUKXwWdoeBuPNV3kQjj3mr.0" in gramine system length is 8388608

I'm a bit confused -- shouldn't you expect the non-existence of the deleted file?

when an untrusted program on the host computer deletes a file in path A , my rust program running in gramine can still detect the existence of the file, but it cannot open the file ( Error No such file or directory (os error 2))

In your case, it looks like you're trying to modify (allowed) files from the host. However, Gramine doesn't share FS state with the host (i.e., the files in mounted directories are only expected to be modified by Gramine processes) so the issue is kind of by-design.

[democ98]

Thank you very much for your answer!could you please explain why it is designed like this? Or is there any documentation to explain why FS is not synchronous? thanks! @kailun-qin

[dimakuv]

Replies to @kailun-qin:

when an untrusted program on the host computer deletes a file in path A , my rust program running in gramine can still detect the existence of the file, but it cannot open the file ( Error No such file or directory (os error 2))

In your case, it looks like you're trying to modify (allowed) files from the host. However, Gramine doesn't share FS state with the host (i.e., the files in mounted directories are only expected to be modified by Gramine processes) so the issue is kind of by-design.

@kailun-qin I wouldn't call your explanation correct. Technically, allowed files can be modified by other processes other than Gramine process. There should be nothing in Gramine that would prevent such scenarios. (But Kailun's note is correct for trusted files and encrypted/protected files -- these are not supposed to be modified by the host.)

Replies to @democ98:

[...] my rust program running in gramine can still detect the existence of the file, but it cannot open the file ( Error No such file or directory (os error 2))

I don't know what syscall(s) are invoked in the Rust's std::fs::try_exists() function that you use, but I would assume something like stat().

The root cause is that Gramine caches file dentries and inodes (the former is like "where in the directory hierarchy the file with this filename is stored" and the latter is like "what are the properties of this file"). This caching plays its role in the stat() syscall, which ends up here:

gramine/libos/src/fs/libos_fs_util.c

Lines 59 to 86 in 0684091

	static int generic_istat(struct libos_inode* inode, struct stat* buf) {
	memset(buf, 0, sizeof(*buf));
	lock(&inode->lock);
	buf->st_mode = inode->type | inode->perm;
	buf->st_size = inode->size;
	buf->st_uid = inode->uid;
	buf->st_gid = inode->gid;
	/* Some programs (e.g. some tests from LTP) require this value. We've picked some random,
	* pretty looking constant - exact value should not affect anything (perhaps except
	* performance). */
	buf->st_blksize = 0x1000;
	/*
	* Pretend `nlink` is 2 for directories (to account for "." and ".."), 1 for other files.
	*
	* Applications are unlikely to depend on exact value of `nlink`, and for us, it's inconvenient
	* to keep track of the exact value (we would have to list the directory, and also take into
	* account synthetic files created by Graphene, such as named pipes and sockets).
	*/
	buf->st_nlink = (inode->type == S_IFDIR ? 2 : 1);
	if (inode->mount->uri)
	buf->st_dev = hash_str(inode->mount->uri);
	unlock(&inode->lock);
	return 0;
	}

If you look closely at this function's implementation, you will see that at no point in time Gramine queries the host FS, to learn if there's anything new about the file's inode. Gramine memorized all the required details about the file properties, and hands them out to the application (your Rust program) from its cache.

Now that I think of it, this indeed has an undesirable side effect of not knowing about the actual file status. In your Rust case, the file was deleted, but Gramine only noticed it when it tried to open the file (which of course requires to query the host FS).

@kailun-qin @mkow Do we want to fix this behavior? It should be actually pretty simple: e.g. perform a dummy open() on the file to make sure it still exists on the host. The negative is that it will be slower than the current implementation, since we will need to leave Gramine process, perform a host open(), and then go back. On the other hand, we currently don't preserve the expected Linux behavior, which we should.

[dimakuv]

@democ98 Could you check the PR #1423?

[kailun-qin]

I wouldn't call your explanation correct. Technically, allowed files can be modified by other processes other than Gramine process. There should be nothing in Gramine that would prevent such scenarios.

Ah yes, I understand. Perhaps I made some confusions, I was NOT talking about can or can't (for different types of FS mounts), but mostly Gramine's current design that doesn't expect such modifications w/o Gramine's notice.

I don't know what syscall(s) are invoked in the Rust's std::fs::try_exists() function that you use, but I would assume something like stat().

It should try statx() first and fallback to newfstatat() if it's not supported, see https://github.com/rust-lang/rust/blob/b5e51db16dfbf5685e32dfe2d9a835a5c695afe4/library/std/src/sys_common/fs.rs#L45-L51 and https://github.com/rust-lang/rust/blob/b5e51db16dfbf5685e32dfe2d9a835a5c695afe4/library/std/src/sys/unix/fs.rs#L838-L856.

Do we want to fix this behavior? It should be actually pretty simple: e.g. perform a dummy open() on the file to make sure it still exists on the host.

Make sense to me.

[kailun-qin]

@democ98 Could you check the PR #1423?

I tested on my setup using some similar Rust code snippets as @democ98 provided, it works fine (showing non-existence of the file, if deleted from host).