os: add ReadDir, ReadFile, WriteFile, CreateTemp, MkdirTemp & deprecate io/ioutil

[rsc]

io/ioutil, like most things with util in the name, has turned out to be a poorly defined and hard to understand collection of things.
As part of #40025, we migrated Discard, NopCloser, and ReadAll to package io.
(They remain in io/ioutil for compatibility, of course, but new code should prefer the ones in io.)
What's left is a few OS file system helpers: ReadDir, ReadFile, TempDir, TempFile, and WriteFile.
I propose we migrate all of these to package os, with a few adjustments.
(Again, they would remain here for compatibility, but new code would prefer the ones in os.)

At that point, io/ioutil would become entirely deprecated.

The signatures would be:

package os

// ReadDir reads the named directory,
// returning all its directory entries sorted by filename.
func ReadDir(name string) ([]DirEntry, error)

// ReadFile reads the named file and returns the contents.
// A successful call returns err == nil, not err == EOF.
// Because ReadFile reads the whole file, it does not treat an EOF from Read
// as an error to be reported.
func ReadFile(name string) ([]byte, error)

// WriteFile writes data to the named file, creating it if necessary.
// If the file does not exist, WriteFile creates it with permissions perm (before umask);
// otherwise WriteFile truncates it before writing, without changing permissions.
func WriteFile(name string, data []byte, perm fs.FileMode) error

// MkdirTemp creates a new temporary directory in the directory dir
// and returns the pathname of the new directory.
// The new directory's name is generated by adding a random string to the end of pattern.
// If pattern includes a "*", the random string replaces the last "*" instead.
// If dir is the empty string, TempFile uses the default directory for temporary files, as returned by TempDir.
// Multiple programs or goroutines calling MkdirTemp simultaneously will not choose the same directory.
// It is the caller's responsibility to remove the directory when it is no longer needed.
func MkdirTemp(dir, pattern string) (string, error)

// CreateTemp creates a new temporary file in the directory dir,
// opens the file for reading and writing, and returns the resulting file.
// The filename is generated by taking pattern and adding a random string to the end.
// If pattern includes a "*", the random string replaces the last "*".
// If dir is the empty string, TempFile uses the default directory for temporary files, as returned by TempDir.
// Multiple programs or goroutines calling CreateTemp simultaneously will not choose the same file.
// The caller can use the file's Name method to find the pathname of the file.
// It is the caller's responsibility to remove the file when it is no longer needed.
func CreateTemp(dir, pattern string) (*File, error)

Notes:

• os.ReadDir returns []DirEntry, in contrast to ioutil.ReadDir's []FileInfo.
  (Providing a helper that returns []DirEntry is one of the primary motivations for this change.)

• os.ReadFile is identical to ioutil.ReadFile.

• os.WriteFile is identical to ioutil.WriteFile.
  I looked into whether the permission bits should be dropped, but Go code in the wild uses a variety of popular settings (for example: 0666, 0644, 0600, 0700, 0777).

• os.MkdirTemp is identical to ioutil.TempDir.
  It cannot be named os.TempDir because that already exists and returns the default temporary directory.
  MkdirTemp seems like it makes clear that the directory is being created and will appear next to Mkdir in docs.
  I looked into whether the directory argument should be dropped, but about 20% of calls in the wild don't use the empty string.

• os.CreateTemp is identical to ioutil.TempFile.
  It could be named os.TempFile, but calling it os.CreateTemp is consistent with os.MkdirTemp, makes clear that the file is being created (using TempFile for this would be a function surprisingly different from TempDir), and will appear next to Create in docs.
  I looked into whether the directory argument should be dropped, but about 25% of calls in the wild don't use the empty string.

[bcmills]

Would it make sense to put ReadDir, ReadFile, and WriteFile in io/fs instead of os?
It sees like they could plausibly work with any filesystem, not just the os one.

I would argue that they fit naturally with the new Walk API, wherever that ends up: ReadDir, ReadFile, WriteFile, and Walk all share the property that they implement a high-level operation in terms of existing lower-level operations.

[bcmills]

(MkdirTemp and CreateTemp seem like they belong in os, though: the location of the temporary directory is OS-specific, and CreateTemp, at least, should return a *os.File rather than an abstract fs.File.)

[bcmills]

Actually, even MkdirTemp and CreateTemp could belong in io/fs, if we define an FS extension interface for a method that returns a default directory for temporary files.

[rsc]

@bcmills, io/fs already has ReadDir and ReadFile. It would be fine to add CreateTemp and MkdirTemp if/when io/fs adds any writing operations.

But it probably wouldn't make sense to put them only in io/fs. Reading from the operating system's file systems is insanely common. For example, even if we were starting over we would certainly keep os.Open(name) rather than tell people to use fs.Open(os.Dir("/"), name) or os.Dir("/").Open(name) (neither of which would be correct on Windows anyway).

[rsc]

Accepting this issue will obsolete #19660, which proposes renaming ioutil.

[gopherbot]

Change https://golang.org/cl/266365 mentions this issue: all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp

[gopherbot]

Change https://golang.org/cl/266364 mentions this issue: os: add ReadFile, WriteFile, CreateTemp (was TempFile), MkdirTemp (was TempDir) from io/ioutil

[gopherbot]

Change https://golang.org/cl/266366 mentions this issue: all: update to use os.ReadDir where appropriate

[rsc]

I've posted the CLs that would implement this change in case anyone wants to see what the diffs look like.
I am particularly happy that in the two CLs converting the tree to use the new locations, 217 "io/ioutil" imports were removed but only 51 "os" imports were added, meaning that in ~75% of cases the code was already importing "os" and is simplified by not having to juggle ioutil as well.

[rsc]

Based on the discussion, this seems like a likely accept.

[earthboundkid]

Getting rid of ioutil is a good conceptual clean up. I know when I first learned Go, I was confused about what was in io vs. ioutil vs. bufio. io itself is a little tricky to understand, but with the cleanup, the explanation is pretty clear. I can imagine writing a blog post for beginners along the lines of:

Package io defines different kinds of "virtual files" either for reading or writing with special features like seeking and closing. io/fs extends the virtual file into a virtual filesystem. In most case, you won't work with io.Readers and io.Writers directly, but either stream them with package bufio or write them to memory with bytes.Buffer or strings.Builder.

The only thing that seems odd given that description is that FileInfo and FileMode are in io/fs instead of io itself. On the one hand, they're part of a "virtual file" so you'd think they'd be in io. OTOH, they only make sense as part of a "virtual filesystem" in which case they should be in io/fs…

[rsc]

Instead of "virtual files" say "virtual data streams" and you're all set!

[rsc]

No change in consensus, so accepted.

[gopherbot]

Change https://go.dev/cl/399854 mentions this issue: io/ioutil: provide an equivalent for the deprecated ReadDir