proposal: os: fast, comprehensive directory API

[networkimprov]

This is an alternative to #41188, which seems impractical. It offers a replacement for os.Readdir() & os.Readdirnames(). We need a new API to address performance problems and missing features in the current one.

Previous discussion...

• proposal: os: new Readdirentries method to read directory entries and efficiently expose file metadata #40352 - os: new ReaddirEntries method
• ReadDir() argument gives fields to populate
• DirEntry gets native dirent fields and lazy-loads others; .Has() indicates is-loaded

The API below meets all the requirements discussed. It provides explicit lazy-loading, ensures that all accessible fields have data from the same retrieval, and avoids an error check after every field access.

Use cases: When you need...
a) certain fields for every item, request them in ReadDirItems().
b) certain fields for some items, request them in DirItem.Load().
c) any fields of the OS defaults that aren't universal, check for them with DirItem.Has() before access.
d) implementation-specific fields, request them as above, then check for them with DirItem.Has() before access.
e) the latest data for an item, request it with DirItem.Load().

package os

// fields uint64 is a set of |'d fieldnames
// Platforms may define their own fieldnames outside a reserved range
// ReadDirItems() & DirItem.Load() ignore fieldnames not available on this OS
// OS default fields are always accessible in DirItem

type DirItem struct { ... }
func (d *DirItem) Has(fields uint64) bool   // indicates whether fields are loaded
func (d *DirItem) Load(fields uint64) error // (re-)loads fields
func (d *DirItem) Name() string             // always succeeds
func (d *DirItem) IsDir() bool              // this and other getters panic if field not loaded (a code error)
func (d *DirItem) Id() FileId
func (d *DirItem) Size() int64
func (d *DirItem) Mode() FileMode
func (d *DirItem) ModTime() time.Time
func (d *DirItem) CreTime() time.Time       // support OS-specific fields
func (d *DirItem) SameAs(*DirItem) bool     // calls .Id() for both objects; counterpart to os.SameFile

type FileId struct { ... }          // unix inode or winapi fileId; may include device info
func (id *FileId) String() string

func ReadDirItems(name string, opt uint64) ([]DirItem, error) // opt: fields | SortXyz
                                                              // if sorted, fieldnames must include sort key
func GetDirItem(name string, resolve bool) (*DirItem, error)  // does stat() or lstat()

func (f *File) ReadDirItems(n int, fields uint64) ([]DirItem, error)
func (f *File) GetDirItem() (*DirItem, error)

const (
   ItemOS    uint64 = 0 // fieldname for OS default fields
   ItemIsdir uint64 = 1 << iota
   ItemId
   ItemSize
   ItemMode
   ItemModTime
   ItemCreTime
)

const (
   SortDown uint64 = ... // descending switch
   SortName uint64 = ...
   SortId
   SortSize
   SortModTime
   SortCreTime
)

cc @robpike @rsc

[rsc]

Redesigning the os.File interface is explicitly out of scope for the io/fs proposal.
We are only making a simple interface for the current world, not building a new world.

[networkimprov]

I'll retarget this proposal to the os package.

[rsc]

Please stop. We are not redesigning the os package either.

[networkimprov]

Please stop.

Apologies if I've given offense somehow. #41188 looks like a non-starter. Could a better directory API land in the x/ repo?

(I completed the revisions I'd started above, dropping the io/fs references.)

[rsc]

Every person and every project has limited bandwidth for making decisions well, especially given the large amounts of time it can take to evaluate an idea fully. Redesigning the os.File API is not on our priority list right now. I'm sorry if that priority conflicts with yours, but again we have limited time and need to focus on higher-impact things.

It is of course fine to build your own APIs in a third-party package, as always.

[networkimprov]

Then would you tag this "Proposal-Hold" instead of closing? You're gonna need it eventually :-)

[rsc]

A few things.

1. Proposal-Hold is not for "gonna need eventually".
2. It's far from clear that we're ever going to need to completely redo the entire directory API as proposed in this issue.
3. If we did need a new API, the API here is not what we would do. If we were going to redo the API, it would be to simplify it further, not make it significantly more complex. Any new API proposal that increases complexity by double or more is a non-starter, and this looks like about 5-10X.

[rsc]

Based on the discussion above, this seems like a likely decline.

[networkimprov]

We are not redesigning the os package either

Thankfully that didn't turn out to be true :-)

[rsc]

No change in consensus, so declined (and retracted).