NSFS | Improve list objects performance on top of NS FS (PR 1/3)

[naveenpaul1]

Explain the changes

1. This is first PR for listing unordered objects.
2. This Pr contains implementation of telldir and seekdir methods in napi layer
3. Global flag is added in config for controlling bucket list flow

Issues: Fixed #xxx / Gap #xxx

1. Improve list objects performance on top of NS FS #6615

Testing Instructions:

1. Not possible with current changes.

• Doc added/updated
• Tests added

[romayalon]

@naveenpaul1 let's add some tests? :)

[guymguym]

@naveenpaul1 This is great!
It would be great to see the performance improvement from the full impl.
See few comments.

[guymguym]

Just two small comments but otherwise looks good.

[guymguym]

First - we need to switch to BigInt - see #7912 (comment)

Second - I see that readdir also returns the same position that telldir returns as d_off -
https://man7.org/linux/man-pages/man3/readdir.3.html

So we should also return this number (as bigint) from our dir.read() call so that we don't have to call telldir again if we just read the last entry where we want to pause. This means we should add it to our struct Entry (in this file).

[guymguym]

I am copying here some findings that I want to keep in mind when implementing list objects using this.

It seems that on BSD the docs for telldir/seekdir say that the position is only valid for the lifetime of the dir handle and once it closes the positions are no longer valid to use:

https://man.freebsd.org/cgi/man.cgi?query=seekdir

       The telldir() function returns a	token representing the	current	 loca-
       tion  associated	 with  the named directory stream.  Values returned by
       telldir() are good only for the lifetime	of the DIR pointer, dirp, from
       which they are derived.	If the directory is closed and then  reopened,
       prior values returned by	telldir() will no longer be valid.  Values re-
       turned by telldir() are also invalidated	by a call to rewinddir().

       The  seekdir()  function	sets the position of the next readdir()	opera-
       tion on the directory stream.  The new position reverts to the one  as-
       sociated	 with  the  directory  stream when the telldir() operation was
       performed.

This article describes a bug that was hidden in BSD implementation of telldir for 25 years until it was found by samba users - https://marcbalmer.ch/when-seekdir-wont-seek-to-the-right-position-a9e2b1986203

In Linux the docs do not specifically say for how long the positions returned by telldir are valid, so it might be that different filesystems will interpret those docs differently.

Overall, I think we should be careful here, and enable the optimization only when we know for sure that the filesystem and the kernel supports stable dir positions even using unrelated dir handles.

[guymguym]

Looks good. Lets get it merged.