`EncryptedFS.rmdir` does not remove files on Windows

[emmacasolin]

Describe the bug

EncryptedFS.rmdir() does not work as expected on Windows. It removes directories but not files.

---

Update: The issue may in fact be that Linux and Mac are removing files when they shouldn't and Windows is actually behaving correctly. This would mean that the tests are wrong. The tests that fail on Windows due to this all throw an EEXIST error when trying to open a (new) file and create a file descriptor to it. When this happens the error looks something like this:

ErrorEncryptedFSError: EEXIST: file already exists, dir\file1

      2041 |             // Target already exists cannot be created exclusively
      2042 |             if (flags & constants.O_CREAT && flags & constants.O_EXCL) {
    > 2043 |               throw new errors.ErrorEncryptedFSError({
           |                     ^
      2044 |                 errno: errno.EEXIST,
      2045 |                 path: path as string,
      2046 |                 syscall: 'open',

      at constructor_._open (src/EncryptedFS.ts:2043:21)
      at src/EncryptedFS.ts:1885:15
      at Object.maybeCallback (src/utils.ts:405:12)
      at Object.<anonymous> (tests/EncryptedFS.concurrent.test.ts:832:12)

To Reproduce

Run the following script on Windows:

import fs from 'fs';
import os from 'os';
import path from 'path';
import Logger, { LogLevel, StreamHandler } from '@matrixai/logger';
import { DB } from '@matrixai/db';
import EncryptedFS from './src/EncryptedFS';
import * as utils from './src/utils';
import INodeManager from './src/inodes/INodeManager';

async function main() {
  const logger = new Logger(`${EncryptedFS.name} Concurrency`, LogLevel.WARN, [
      new StreamHandler(),
  ]);
  const dbKey: Buffer = utils.generateKeySync(256);
  let dataDir: string;
  let db: DB;
  let iNodeMgr: INodeManager;
  let efs: EncryptedFS;

  dataDir = await fs.promises.mkdtemp(
    path.join(os.tmpdir(), 'encryptedfs-test-'),
  );
  db = await DB.createDB({
    dbPath: dataDir,
    crypto: {
      key: dbKey!,
      ops: {
      encrypt: utils.encrypt,
      decrypt: utils.decrypt,
      },
    },
    // @ts-ignore - version of js-logger is incompatible (remove when js-db updates to 5.* here)
    logger: logger.getChild(DB.name),
  });
  iNodeMgr = await INodeManager.createINodeManager({
    db,
    logger: logger.getChild(INodeManager.name),
  });
  efs = await EncryptedFS.createEncryptedFS({
    db,
    iNodeMgr,
    logger,
  });

  const path1 = path.join('dir', 'file1');
  await efs.mkdir('dir');
  await efs.mkdir('dir/dir2');
  let fd = await efs.open(path1, 'wx+');
  await efs.close(fd);
  console.log('Dir exists before rmdir? ', await efs.exists('dir'));
  console.log('Dir2 exists before rmdir? ', await efs.exists('dir/dir2'));
  console.log('FD exists before rmdir? ', await efs.exists(path1));
  await efs.rmdir('dir', { recursive: true });
  console.log('Dir exists after rmdir? ', await efs.exists('dir'));
  console.log('Dir2 exists after rmdir? ', await efs.exists('dir/dir2'));
  console.log('FD exists after rmdir? ', await efs.exists(path1));

  await efs.stop();
  await fs.promises.rm(dataDir, {
    force: true,
    recursive: true,
  });
}

main()

Output:

Dir exists before rmdir?  true
Dir2 exists before rmdir?  true
FD exists before rmdir?  true
Dir exists after rmdir?  false
Dir2 exists after rmdir?  false
FD exists after rmdir?  true

Expected behavior

The file should not exist after rmdir is called. This is observed when running the same script on Linux:

Dir exists before rmdir?  true
Dir2 exists before rmdir?  true
FD exists before rmdir?  true
Dir exists after rmdir?  false
Dir2 exists after rmdir?  false
FD exists after rmdir?  false

Platform

• OS: Windows

Additional context

May be related to MatrixAI/Polykey-CLI#14

[CMCDragonkai]

What is an FD?

[CMCDragonkai]

Rmdir is not supposed to remove files. See: https://nodejs.org/api/fs.html#fspromisesrmdirpath-options.

[CMCDragonkai]

The rmdir cannot remove file descriptors. It's not supposed to be able to do this. The API on node fs does not allow this.

[CMCDragonkai]

What tests are failing here due to this?

[emmacasolin]

What tests are failing here due to this?

So far it's all of the concurrency tests that have failures, but I haven't gone over all of the failures yet. I can put together a list.

[CMCDragonkai]

Yea please create list. And also identify the offending usage of rmdir on file descriptors. It should never be used on file descriptors.

On Linux and MacOS, it should be throwing ENOTDIR if you are using it with file descriptors.

[tegefaulkes]

Looking at https://nodejs.org/api/fs.html#fspromisesrmdirpath-options I think that this should be throwing an ENOENT or ENOTDIR error if the directories contain a file even with the recursive: true.

Not to be confused with rm witch will remove files when recursive and force are true.

[emmacasolin]

Yea please create list. And also identify the offending usage of rmdir on file descriptors. It should never be used on file descriptors.

On Linux and MacOS, it should be throwing ENOTDIR if you are using it with file descriptors.

It's not being directly supplied a file descriptor, rather rmdir is given a path to a directory and if recursive is set to true then any file descriptors within the directory/sub directories get removed as well.

[CMCDragonkai]

any file descriptors within the directory/sub directories get removed as well.

Directories and subdirectories cannot "contain" file descriptors.

[CMCDragonkai]

Can you point out a part of the EFS where it is doing this? I don't see any file descriptors within directories. There are no such thing. FDs are just numbers.

[emmacasolin]

The example script in the issue description here is essentially what's happening in all of the affected tests. The FD points to a file inside the directory that rmdir gets called on.

[CMCDragonkai]

1. Deleting directories/files should work fine, because it has nothing to do with FDs. Even if an fd is opened on a subdirectory/file. Again deleting a directory and file will work.
2. Any FD opened SHOULD be closed by the user.

The above 2 points are true independently of each other.

[CMCDragonkai]

Deleting directories and files is about deleting their hardlinks. There is no interaction with the FD system.