readdir() fails in WSL with errno=75

[dajhorn]

The clib3 readdir() for Linux fails to access NTFS through the Windows Subsystem for Linux.

Platform:

C:\Users\dajhorn>wsl --version
WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26091.1-240325-1447.ge-release
Windows version: 10.0.26100.1882

$ mount | grep /mnt/c
C:\ on /mnt/c type 9p (rw,noatime,dirsync,aname=drvfs;path=C:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=5,wfd=5)

Reproducer:

$ cat readdir-test.cpp
#include <cerrno>
#include <dirent.h>
#include <iostream>

int main() {
    const char *path = ".";
    DIR *directory = opendir(path);

    if (!directory) {
        std::cerr << "Error opening directory: " << path << std::endl;
        std::cerr << "errno=" << errno << std::endl;
        return errno;
    }

    struct dirent *entry;
    while (entry = readdir(directory)) {
        std::cout << entry->d_name << std::endl << std::flush;
    }
    std::cerr << "errno=" << errno << std::endl << std::flush;

    closedir(directory);
    return 0;
}

$ wcl386 -bt=linux readdir-test.cpp
Open Watcom C/C++ x86 32-bit Compile and Link Utility
Version 2.0 beta Oct  7 2024 02:27:37 (32-bit)
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1988-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
        wpp386 readdir-test.cpp  -bt=linux
Open Watcom C++ x86 32-bit Optimizing Compiler
Version 2.0 beta Oct  7 2024 02:24:09 (32-bit)
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1989-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
readdir-test.cpp: 127 lines, included 1937, no warnings, no errors
Code size: 222
        wlink @__wcl_00.lnk
Open Watcom Linker Version 2.0 beta Oct  7 2024 02:19:13 (32-bit)
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1985-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
loading object files
searching libraries
creating a Linux x86 executable

Expected behavior:

./readdir-test prints a list of file names in the current working directory.

Actual behavior:

./readdir-test fails with readdir() returning NULL and setting errno=75 if the underlying filesystem is NTFS on WSL.

Notes:

• The glitch is peculiar to Open Watcom v2. The same code compiled with g++ works as expected on WSL.
• The readdir-test program behaves properly if the underlying filesystem is vfat or a native filesystem like tmpfs.
• errno=75 corresponds to EOVERFLOW
• I don't know whether this glitch is peculiar to NTFS or large volumes in general.

[jmalak]

What is content of PATH, WATCOM and INCLUDE environment variables?

Anyway OW doesn't support symbolic links, that it must be installed on true path.
We not support WSL, it is not tested.
OW linux directory functions implementation in OW CRTL use standard Linux system calls with standard UNIX file name/path separator, that I don't know how WSL convert filenames and paths for Windows filesystems.
It could be issue with internal name buffer size overflow in CRTL, I will check.

Please how it looks like output from ls -la in your current directory where you test it.

[dajhorn]

What is content of PATH, WATCOM and INCLUDE environment variables?

If I compile using a Windows host, then:

If I compile using a Linux host, then:

Each readdir-test elf is bytewise identical.

[dajhorn]

Please how it looks like output from ls -la in your current directory where you test it.

[jmalak]

All looks OK. Did you tried created executable on "true" Linux?

[dajhorn]

Yes, the build host does not affect behavior. Output binaries are identical regardless of host toolchain.

I checked these things:

• Small filesystems; errno=75 happens on NTFS volumes smaller than 2GB.
• Loopback/direct filesystems. The glitch does not happen with a wsl --mount <disk> --bare style mount or with a mount -t ntfs -o loop style mount.

This means that errno=75 here is peculiar to the default mount -t drvfs style mount in WSL, which uses the 9p network filesystem protocol.

My guess is that errno=75 is coming from the SYS_getdents call, perhaps because watcom defines a dirent that is different than glibc.

Can you suggest some places for me to begin troubleshooting? -- Given that Open Watcom is the only compiler with this failure mode, I'm starting to look at the owc clib sources.

[jmalak]

If you are interested in fixing this issue then problem is probably in readdir C run-time library function implementation.
Code is in bld/clib/posix/readdir.c
It looks like OW uses SYS_getdents with buffer for caching a few dirent structures, that it could be some issue in handling this buffer therefore errno=75 (real buffer overflow). It has nothing to do with glibc, because OW doesn't use it.
Probably it is not correct handling of errno=75 in OW code which can happen but is not error and requires somehow re-call SYS_getdents. Maybe we could check by SYS_getdents64 first and if not available then use SYS_getdents and handle errors.

[jmalak]

Maybe it is bug in WSL for 32-bit application too.

[dajhorn]

Bingo. I can get errno=75 with g++ if I force a 32-bit build, which makes this a platform bug and not a watcom bug. Thanks.

[jmalak]

I check internet and there is some info about WSL2 which should handle 32-bit application properly, but I cannot confirm if it is true.

[dajhorn]

microsoft/WSL#12154

[jmalak]

Thanks for identification source of problem.

[jmalak]

I will try to rewrite readdir to use getdents64 system call and if it data are in 32-bit range then pass it as result back otherwise report error EOVERFLOW