std::filesystem support

[chenxiaolong]

Description

The libc++ libraries (both shared and static) seem to be missing symbols needed to use <experimental/filesystem>. For example, trying to compile a simple program that calls std::experimental::filesystem::remove_all(path) will fail at link time with:

/stuff/Applications/Linux/android-ndk/sources/cxx-stl/llvm-libc++/include/experimental/filesystem:1695: error: undefined reference to 'std::experimental::filesystem::v1::__remove_all(std::experimental::filesystem::v1::path const&, std::__ndk1::error_code*)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Sample project: https://github.com/chenxiaolong/ndk_filesystem_test (run ./test.sh to reproduce issue)

Is this expected to happen? I couldn't find any details as to whether the filesystem library should work with the NDK.

Environment Details

• NDK Version: r16b (16.1.4479499)
• Build sytem: any
• Host OS: Fedora 27 x86_64
• Compiler: clang
• ABI: any
• STL: libc++
• NDK API level: 21
• Device API level: n/a

[jmgao]

libc++ compiles its experimental parts into libc++experimental, a library that you need to link in separately. Unfortunately, we don't seem to build it as part of the NDK. Retitling the bug to look into shipping it.

In the mean time, you might be able to just add the required files directly into your project (from $ANDROID_NDK_ROOT/sources/cxx-stl/llvm-libc++/src/experimental/filesystem) to work around this on your end, but I haven't tried it (and this is guaranteed to break in the future, when the files are moved into the standard or abandoned).

[chenxiaolong]

Ah okay, that makes perfect sense. I didn't realize libc++ also had a separate library like gcc's libstdc++fs.

[DanAlbert]

The tests for std::filesystem are not going to be easy to run on Android (they require a python helper). Going to find a way to work around that, but it's not likely to happen during r17.

[DanAlbert]

Unlikely to fit into r18. I want to get this done for the platform before the NDK since it should be a smaller problem to solve, and haven't done that yet.

[andreya108]

So, for now, can we suppose that full support for std::filesystem will appear in r19?

[DanAlbert]

Given how strongly people feel about #463 I will be doing that before this, and I'm skeptical that I'll be able to accomplish that before r19 is complete (there's always a fair amount of overhead in coordinating with external products, and CMake is an unfamiliar code base to all of us here).

Going to move this to the r20 bucket to match the other bug, but odds are there will be some down time while waiting on reviews for #463 so I'll try to make some progress on this then.

[DanAlbert]

So, for now, can we suppose that full support for std::filesystem will appear in r19?

Also, a disclaimer that when this does land, the NDK's support for std::filesystem will be as complete as libc++'s (since that's what we use). The libc++ docs still indicate that this is experimental (and not ABI stable), but it does at least appear to be complete.

[arcanis]

Of note: <filesystem> is now available without going through experimental (cf commit). Would that make it easier to integrate?

[edit] It seems like the symbols are still in a separate library, libc++fs

[DanAlbert]

No. Like I've said, the difficulty here is getting the tests running.

[feliwir]

I'd like to see the filesystem support happening aswell 👍

[diegurrio]

Looking forward to the experimental file system libraries being available.

[KayEss]

In the current NDK the experimental filesystem is still not part of the libc++ library, and now the sources can no longer be installed through the SDK tools so our old work around suggested by @jmgao no longer works.

Any suggestions for what to do now?

[alexcohn]

@KayEss you can probably follow https://android.googlesource.com/platform/external/libcxx/+/master/CMakeLists.txt

[enh]

https://issuetracker.google.com/121291977 was another request for std::filesystem...

[arcanis]

@DanAlbert Is it still planned for r20?

[DanAlbert]

Conveniently, upstream has actually fixed some of the issues that we were going to need to fix about a month ago, so there's a bit less work on our end.

After disabling a bunch of tests (everything relating to hard links, sockets, and fifos, since selinux prevents their use in /data/local/tmp), I do have a passing test run for Q devices.

There's at least one thing preventing me from running the tests on JB devices more or less at all though. I have an idea of how to fix it, but haven't had a chance to try yet. That could easily be where most of the trouble is so it's hard to say for sure whether this is close to done or not, but the few that do run have me optimistic.

[DanAlbert]

Alright, managed to get a full test run on Jelly Bean. Everything looks good within a reasonable margin of error. Tests did uncover two bugs, but they're relatively minor:

1. [BUG] fchmodat with AT_SYMLINK_NOFOLLOW is very inconsistent #1258 std::filesystem::permissions(symlink_path, p perms, std::filesystem::perm_options::nofollow) will not honor nofollow for devices older than API 23.
2. [BUG] realpath doesn't fail on non-existent paths on old devices #1260 std::filesystem::canonical will incorrectly succeed for non-existent paths on devices older than API ~19 (not sure exactly when)

There are a fair number of other things that don't work, most of which depend on API level, but they are all related to Android security policy and std::filesystem signals the error appropriately:

• hardlinks aren't allowed (at least in /data/local/tmp for the shell user)
• no types of special files (mknod, named fifos, unix domain sockets, etc) are allowed (at least in /data/local/tmp for the shell user)
• Ability to set the permissions of a symlink is inconsistent (see [BUG] fchmodat with AT_SYMLINK_NOFOLLOW is very inconsistent #1258)

It's going to take me some time to get the test fixes pushed upstream, but if that takes too long I don't think we need to wait for that before shipping it in r22 beta 1.

As for the two bugs, I don't think those are severe enough that we necessarily need to fix them before shipping. It'd be better to get them fixed if we can, but I don't think we need to hold back the feature for them.

[enh-google]

(see https://android-review.googlesource.com/c/platform/external/libcxx/+/1110835/7/src/filesystem/operations.cpp for my libc++ workaround for those.)

[alexcohn]

Unix domain sockets are supported. So, maybe the test should run in app private sandbox, and not in /data/local/tmp ?

[DanAlbert]

Converting our tests to run in an app is something I've been working on. It's going to take some time, it seems, especially for the libc++ tests. Nothing untenable, just going to take longer than I want to hold this back.

Good to know that it's not wasted effort though :)

[DanAlbert]

The implementation for this is now checked in to master and will be in r22. I've forked the remaining testing cleanup into #1265 so I can close this to make it obvious that the part you folks care about is complete without having to read the whole thing :)

[gulrak]

Yes the current implementation is irritating - either don't include it at all, or include it so it links!

As @Apeopex mentioned further up, there is a workable Android alternative which I am using and haven't had any issues with at all.

https://github.com/gulrak/filesystem

@oviano If I am reading docs right, this is equivalent of using boost::filesystem, right? It doesn't pretend to be std::?

These comments have been a few weeks ago, but I just found this discussion and want to add a small detail to the suggested workaround implementation in case someone might think about it:

It is comparable to boost::filesystem in that it is using its own namespace and so its not the same as std::, but it differs from boost::filesystem in that it pretends to be std::filesystem in questions of api and behavior. There are various differences between boost::fs and std::fs as for example joining absolute paths by operator/ and the above mentioned header-only implementation tries hard to implement std::fs behavior, and when it differs undocumented, I handle it as a bug. The one major documented difference is, it is utf-8 oriented, so if that's your encoding, you get what std::filesystem would do for C++17. Details of behavior differences are in the docs.

Still, to be clear, I'm currently not developing on Android myself, and I have no CI integration for it, so the current level of support is just based on feedback and fixed issues of users. And I am thankful for that! Any feedback or help to get a CI setup on Github going to enhance support would sure be appreciated.

[hcv1027]

The implementation for this is now checked in to master and will be in r22. I've forked the remaining testing cleanup into #1265 so I can close this to make it obvious that the part you folks care about is complete without having to read the whole thing :)

Hi @DanAlbert , can we use std::filesystem::create_directory in r22?

I still have the compile errors:

In function `std::__ndk1::__fs::filesystem::exists(std::__ndk1::__fs::filesystem::path const&)':
/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/filesystem:1615: undefined reference to `std::__ndk1::__fs::filesystem::__status(std::__ndk1::__fs::filesystem::path const&, std::__ndk1::error_code*)'
In function `std::__ndk1::__fs::filesystem::create_directory(std::__ndk1::__fs::filesystem::path const&)':
/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/filesystem:1555: undefined reference to `std::__ndk1::__fs::filesystem::__create_directory(std::__ndk1::__fs::filesystem::path const&, std::__ndk1::error_code*)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

This is my cmake commad:

cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-29 \
    -DANDROID_NATIVE_API_LEVEL=29 \
    -DANDROID_STL=c++_shared \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_BUILD_TYPE=RELEASE \
    -DCMAKE_INSTALL_PREFIX=$DEST \
    ..

Accroding to this link
I also tries to add -lc++experimental or -lc++fs in my CMakeList.txt

target_link_libraries( vmc_io_lite
  c++fs
)

But this will cause below error message:

/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: cannot find -lc++fs
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Can we use these function in r22?
std::filesystem::create_directory, std::filesystem::exists, std::filesystem::remove_all

[rprichard]

This path is for r21d not r22:

/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/filesystem