path/filepath: fix EvalSymLinks documentation

[ericwj]

What version of Go are you using (go version)?

1.14.4

What operating system and processor architecture are you using (go env)?

Windows

What did you do?

Read the docs.

What did you expect to see?

Documentation to the effect that most applications on all operating systems should not use this API, unless they write complex additional logic to resolve just those links that they are interested in (e.g. searching through one volume only, although this API cannot reliably enable this specific scenario) or responsible for (e.g. system administration software).

What did you see instead?

EvalSymLinks intending to resolve all links in the path that it receives as argument, without any hint in the documentation that this might introduce issues due to system reconfigurations, amongst others.

What do you propose to fix this?

I propose documentation changes as the best solution for EvalSymLinks short of deleting this API altogether, which is not possible today since deleting or no-op'ing this API as proposed in #40104 violates go policies.

EvalSymLink violates what Wikipedia calls the Fundamental Theorem of Software Engineering

EvalSymLinks is not suitable for most applications on any operating system because it unconditionally undoes all indirections for a given path. This is a violation of the principle mentioned which states (in my wording more applicable here):

Any problem in software can be solved by adding another layer of indirection.

EvalSymLinks always resolves all indirections, even those that might have been introduced to fix operational issues affecting the particular application using this API, or which might have been introduced to solve administrative problems on a system that applications should generally remain unaware of - such as disk full situations.

I quote from Microsoft documentation:

Mounted Folders - Win32 apps | Microsoft Docs

Neither users nor applications need information about the target volume on which a specific file is located.

This API gives software no options to pick and choose to remove only those indirections that they are aware of, are designed to resolve, or have responsibility for.

Practical reasons for most developers not to use EvalSymLinks

Certainly on Windows, the practical usefulness and applicability of this API for most applications is severely limited by the following four facts:

• Most users do not expect to ever see resolved paths
  On the contrary, especially those users that might in some situations want to see resolved paths may configure links for the specific purpose of abbreviating paths that they frequently use, in which case resolving links and showing them to the user is a severe and annoying usability issue beyond what it in most situations normally already is. Applications should hence always remember the path they were configured or started with and generally only ever show that to the user. I champion that this point stands firmly on any operating system besides Windows.
• Volume access paths and UNC mappings could change at any time
  No reboot is necessary to create, change or delete junctions or links. Such events might hence invalidate results of EvalSymLinks at any time. No notification is given about such changes, unless the application specifically registers for them. But this is an advanced feature which is nontrivial to implement and test, even though go has/if go would have first-class support for this scenario.
• Access paths and mappings could be modified - including deleted - while files are in use that were opened using that access path
  The path through which files or directories on a volume are accessible might hence change at a moments notice, even while the application is using files or directories whose access path(s) changed and the application might continue to open handles to the access path(s) it was configured with on another volume, while the application remains unaware, if it does not use the EvalSymLinks API, or if it takes care not to cache the result of EvalSymLinks in any way.
• Volume access paths and UNC mappings do not have to be created with a drive letter
  And if they do, they do not need to point at the root of the drive with that letter (Z:\). This might be surprising to any developer that is not intimately aware of Windows internals and will produce issues as soon as they do not consistently use API that is itself properly aware of and tested thoroughly to work on Windows.

Similar expectations and issues may arise about paths that require the use of the \\?\ prefix, which complicates all operations with paths further, including long paths and volume GUID paths which EvalSymLinks may also introduce in any application on any machine. It is fair to note that EvalSymLinks is not the only source of such paths and Windows applications should be prepared to deal with them regardless of whether they use EvalSymLinks - such paths might even be introduced by user input directly, or through the operating system working directory at application start, or be intermittently visible while the system is being reconfigured.

EvalSymLinks cannot be used to identify volumes in applicable advanced scenario's

Finally, this API by design and as documented can on Windows not identify the volume on which a file or directory resides. Besides resolving UNC mappings, again only in advanced scenario's, this is - for as far as I have been able to identify - about the only situation where the above arguments against the use of EvalSymLinks may not always apply for all applications.

Paths with a drive letter never identify volumes. Drive letters may in some trivial system configurations happen to correspond to identifiable volumes, but this can not and must never be relied upon. EvalSymLink definition, documentation and behavior on Windows reinforces this wrong assumption.

That EvalSymLinks is of very limited use on Windows is strikingly evidenced by the configuration provided to reproduce issue #40176. This configuration is of particular interest, since EvalSymLinks provides here exactly zero information to the caller. Any path the application requests to be resolved will result in the exact path provided to EvalSymLinks, since the reparse point resolves to the exact same path as that of the directory that points to it. (before making the path relative and cleaning it, perhaps against statements in the documentation if the fact that an absolute link was resolved might have been lost - and a conforming result might confuse anyone)

In fact, until #40095 is part of go, EvalSymLinks is completely unable to identify volumes on 64-bit Windows and other Windows installations that use GPT partitioning at all. With this fix, it is only able to do so if the volume happens to have its volume GUID path as its only access path. However now this is merely accidental, due to a fortunate side effect of Windows system architecture.

On Windows systems that do not use GPT partitioning, or on (external) disks that do not use GPT, identifying volumes is simply not possible with a path alone, neither before nor after calling EvalSymLinks.

How concretely do you propose to fix this?

Change the documentation of EvalSymLinks in src\path\filepath\path.go to read:

// EvalSymlinks returns the path name after the evaluation of any symbolic
// links.
// If path is relative the result will be relative to the current directory,
// unless one of the components is an absolute symbolic link.
// EvalSymlinks calls Clean on the result.
// Use of this function is unsuitable for the vast majority of applications.
// This function always resolves all links on path, but links may have
// been created to solve administrative problems of which 
// most applications should remain unaware.
// Most applications should only resolve specific links that they 
// require to resolve,  use the result immediately, forget the result
// and never show the result to the user.
// On Windows, the result is not stable and must not be cached
// or stored in any way, resolving links may introduce complexities that 
// the application must be  prepared to deal with and the result 
// does not identify the volume that a file or directory resides on.
func EvalSymlinks(path string) (string, error) {
	return evalSymlinks(path)
}

[networkimprov]

The background to this is that Docker misuses EvalSymlinks, so produces confusing logs and/or errors on common Windows configurations.

The Windows bugs are #39786 & #40176. See also kubernetes/utils#172

cc @robpike @ianlancetaylor @bcmills @jstarks

[ericwj]

I propose documentation changes as the best solution (...) short of (...)

go vet improves upon this solution.

It would be beneficial to be able to suggest against using this API on Windows and once there are alternatives which are truly portable and provide solutions to the other issues with this API, against use of this API altogether.

[ericwj]

Here is a list of results for EvalSymLinks that I built running on go 1.14.6 against a series of links that I created.

It shows it may be fixing issues for many more people than introduce breaking changes having this API stop trying to resolve links on Windows and instead return one of

• the input string,
• error 87 The parameter is invalid, 123 The filename, directory name, or volume label syntax is incorrect. or 161 The specified path is invalid for invalid paths
• or a not found error such as in the list below, one of 3, 53 or 67, whichever is applicable.

The list

• evidences grim and potentially severely damaging problems with this API since it interprets links resolved on a remote machine as local paths - and might return them if the resolved path exists on the local machine.
• shows that even links to a local file on an SMB mapping to a local share may have go bypass the SMB mapping it was configured to use, which might be a a security bypass if the SMB share is configured with more restrictive security than the file system.
• shows that the API does something useful (correctly, successfully) in just 2 cases (repeated twice) in a long list of tests - which was shortened quite a lot for purposes of showing it on GitHub in the first place.
• shows that these two cases stop working in non-trivial configurations where volumes aren't rooted at the root of a DOS drive letter or are accessed through the network stack, locally or remotely. This means the usefulness of this API where it actually works is totally dependent on operational configuration of both the system and the application.

Correct useful results are also results with errors 53 and 67 which are the result of correctly establishing that the path does not exist. Other errors are all wrong. 2 for being the wrong error code - a persistent problem accross go's path/filepath - and both 2 and 3 for about never being correct claiming there is nothing.

Errno	Path	Result	LinkType	Target	Error
	C:\go\Peculiar\	C:\go\Peculiar
3	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\Long12\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	The system cannot find the path specified.
2	C:\go\Peculiar\DosMissing\		SymbolicLink	C:\go\Peculiar\DosWasHere	CreateFile C:\go\Peculiar\DosWasHere: The system cannot find the file specified.
	C:\go\Peculiar\FileHardLink.txt	C:\go\Peculiar\FileHardLink.txt	HardLink (File)
	C:\go\Peculiar\FileLink.txt	C:\go\Peculiar\OneFile\file.txt	SymbolicLink	C:\go\Peculiar\OneFile\file.txt
	C:\go\Peculiar\GitLike\RepoA\Submodule\	C:\go\Peculiar\GitLike\RepoB	SymbolicLink	C:\go\Peculiar\GitLike\RepoB
3	C:\go\Peculiar\HardLink\		SymbolicLink	Volume{587c9714-e22e-423a-9421-294a2c043d22}\	readlink C:\go\Peculiar\HardLink: The system cannot find the path specified.
-111	C:\go\Peculiar\Identity\		Junction	Volume{68147e1-d8bf-491d-85d5-98698fb3ee6c}\	EvalSymlinks: too many links
-111	C:\go\Peculiar\Junction\		Junction	Volume{7bbce513-361b-4ff4-aedb-09499749192a}\	EvalSymlinks: too many links
3	C:\go\Peculiar\Long1\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1	The system cannot find the path specified.
3	C:\go\Peculiar\Long2\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	The system cannot find the path specified.
3	C:\go\Peculiar\Long3\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2\3. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .3	The system cannot find the path specified.
53	C:\go\Peculiar\NetMissing\		SymbolicLink	UNC\GONE31915684\ConnectivityLost	The network path was not found.
67	C:\go\Peculiar\NoNetDir\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
67	C:\go\Peculiar\NoNetDirNoParse\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
	C:\go\Peculiar\OneFile\file.txt	C:\go\Peculiar\OneFile\file.txt	HardLink
2	C:\go\Peculiar\UncDir\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	C:\go\Peculiar\UncDirNoParse\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	C:\go\Peculiar\VolMissing\		SymbolicLink	Volume{00000000-0000-0000-0000-000000000000}	readlink C:\go\Peculiar\VolMissing: The system cannot find the file specified.
	B:\	B:\
3	B:\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\Long12\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	The system cannot find the path specified.
2	B:\Peculiar\DosMissing\		SymbolicLink	C:\go\Peculiar\DosWasHere	CreateFile C:\go\Peculiar\DosWasHere: The system cannot find the file specified.
	B:\Peculiar\FileHardLink.txt	B:\Peculiar\FileHardLink.txt	HardLink (File)
	B:\Peculiar\FileLink.txt	C:\go\Peculiar\OneFile\file.txt	SymbolicLink	C:\go\Peculiar\OneFile\file.txt
	B:\Peculiar\GitLike\RepoA\Submodule\	C:\go\Peculiar\GitLike\RepoB	SymbolicLink	C:\go\Peculiar\GitLike\RepoB
3	B:\Peculiar\HardLink\		SymbolicLink	Volume{587c9714-e22e-423a-9421-294a2c043d22}\	readlink B:\Peculiar\HardLink: The system cannot find the path specified.
-111	B:\Peculiar\Identity\		Junction	Volume{68147e1-d8bf-491d-85d5-98698fb3ee6c}\	EvalSymlinks: too many links
-111	B:\Peculiar\Junction\		Junction	Volume{7bbce513-361b-4ff4-aedb-09499749192a}\	EvalSymlinks: too many links
3	B:\Peculiar\Long1\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1	The system cannot find the path specified.
3	B:\Peculiar\Long2\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	The system cannot find the path specified.
3	B:\Peculiar\Long3\		Junction	C:\go\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2\3. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .3	The system cannot find the path specified.
53	B:\Peculiar\NetMissing\		SymbolicLink	UNC\GONE31915684\ConnectivityLost	The network path was not found.
67	B:\Peculiar\NoNetDir\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
67	B:\Peculiar\NoNetDirNoParse\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
	B:\Peculiar\OneFile\file.txt	B:\Peculiar\OneFile\file.txt	HardLink
2	B:\Peculiar\UncDir\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	B:\Peculiar\UncDirNoParse\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	B:\Peculiar\VolMissing\		SymbolicLink	Volume{00000000-0000-0000-0000-000000000000}	readlink B:\Peculiar\VolMissing: The system cannot find the file specified.
	G:\	G:\
3	G:\Long1\		Junction	G:\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1	The system cannot find the path specified.
3	G:\Long2\		Junction	G:\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	The system cannot find the path specified.
3	G:\Long3\		Junction	G:\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2\3. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .3	The system cannot find the path specified.
67	G:\NoNetDir\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
67	G:\NoNetDirNoParse\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
3	G:\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\Long12\		Junction	G:\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	The system cannot find the path specified.
	G:\Peculiar\FileHardLink.txt	G:\Peculiar\FileHardLink.txt	HardLink (File)
	G:\Peculiar\FileLink.txt	G:\Peculiar\OneFile\file.txt	SymbolicLink	G:\Peculiar\OneFile\file.txt
	G:\Peculiar\GitLike\RepoA\Submodule\	G:\Peculiar\GitLike\RepoB	SymbolicLink	G:\Peculiar\GitLike\RepoB
	G:\Peculiar\OneFile\file.txt	G:\Peculiar\OneFile\file.txt	HardLink
2	G:\UncDir\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	G:\UncDirNoParse\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
	\\WS19\GoFilePath\	\\WS19\GoFilePath\
3	\\WS19\GoFilePath\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\Long12\		Junction	C:\notgo\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	CreateFile \\WS19\GoFilePath\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1: The system cannot find the path specified.
2	\\WS19\GoFilePath\Peculiar\DosMissing\		SymbolicLink	C:\notgo\Peculiar\DosWasHere	CreateFile C:\notgo: The system cannot find the file specified.
	\\WS19\GoFilePath\Peculiar\FileHardLink.txt	\\WS19\GoFilePath\Peculiar\FileHardLink.txt	HardLink (File)
2	\\WS19\GoFilePath\Peculiar\FileLink.txt		SymbolicLink	C:\notgo\Peculiar\OneFile\file.txt	CreateFile C:\notgo: The system cannot find the file specified.
2	\\WS19\GoFilePath\Peculiar\GitLike\RepoA\Submodule\		SymbolicLink	C:\notgo\Peculiar\GitLike\RepoB	CreateFile C:\notgo: The system cannot find the file specified.
3	\\WS19\GoFilePath\Peculiar\HardLink\		SymbolicLink	Volume{95f60738-37f1-4f25-9208-1e0e5212683c}\	readlink \\WS19\GoFilePath\Peculiar\HardLink: The system cannot find the path specified.
3	\\WS19\GoFilePath\Peculiar\Identity\		Junction	Volume{776815af-a50d-4157-8f21-629ab53e6f61}\	readlink \\WS19\GoFilePath\Peculiar\Identity: The system cannot find the path specified.
3	\\WS19\GoFilePath\Peculiar\Junction\		Junction	Volume{bb046931-d671-4f07-8748-9e9892f4f4b0}\	readlink \\WS19\GoFilePath\Peculiar\Junction: The system cannot find the path specified.
2	\\WS19\GoFilePath\Peculiar\Long1\		Junction	C:\notgo\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1	CreateFile C:\notgo: The system cannot find the file specified.
2	\\WS19\GoFilePath\Peculiar\Long2\		Junction	C:\notgo\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2	CreateFile C:\notgo: The system cannot find the file specified.
2	\\WS19\GoFilePath\Peculiar\Long3\		Junction	C:\notgo\Peculiar\1. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .1\2. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .2\3. GoFilePath Directory ___(...Path Segment Length MAX_PATH-5...)___ .3	CreateFile C:\notgo: The system cannot find the file specified.
53	\\WS19\GoFilePath\Peculiar\NetMissing\		SymbolicLink	UNC\GONE18cf9be2\ConnectivityLost	The network path was not found.
67	\\WS19\GoFilePath\Peculiar\NoNetDir\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
67	\\WS19\GoFilePath\Peculiar\NoNetDirNoParse\		SymbolicLink	UNC\localhost\NoNetworkGoFilePath	The network name cannot be found.
2	\\WS19\GoFilePath\Peculiar\UncDir\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	\\WS19\GoFilePath\Peculiar\UncDirNoParse\		SymbolicLink	UNC\localhost\GoFilePath	The system cannot find the file specified.
2	\\WS19\GoFilePath\Peculiar\VolMissing\		SymbolicLink	Volume{00000000-0000-0000-0000-000000000000}	readlink \\WS19\GoFilePath\Peculiar\VolMissing: The system cannot find the file specified.

evalsymlinks.results.brief.html

The error code -111 means that there was any go error without an error code.
Situations where go might return a remote path as a local path are evidenced by UNC paths in the Path column and an error code resembling CreateFile C:\notgo: The system cannot find the file specified. as well as a Path on B:\ which resolves to a path on C:\ which is returned.
The Target column lists the link target the way it would appear using dir in PowerShell which means \\?\ is usually stripped and some targets aren't listed at all.

The paths have the following meaning:

• C:\ is a directly attached harddisk.
• B:\ is a drive mapping to an SMB share on the local machine pointing at C:\go
• \\WS19 is a Windows Server 2019 file server hosting the same directory structure under its local directory C:\notgo
• G:\ is the root of a partition configured to use a DOS device name of its own
• *\Junction, *\HardLink and *\Identity are roots of partitions that are accessible through a directory junction, a hard link and by specifying the access path through PowerShell's Add-PartitionAccessPath, respectively.

This list is far from complete, since these tests

• can be repeated e.g. against the same directory structure but with the no parse prefix consistently used
• skipped partitions that did not have their root mapped to a drive letter (although their root paths are in the list and do not resolve)
• there are no tests that run against Storage Spaces disks, although these should look the same as regular virtual disks,
• used paths exclusively formatted with backslashes whereas either Windows itself or .NET, PowerShell and go's own os.Chdir accept mixed forward and backward slashes both for the path and the no parse prefix (e.g. /\?/C:\Dir/).

[ericwj]

I closed #40095 on grounds that there is agreement that EvalSymlinks should not be fixed in any way, but rather replaced and even broken further to fix the issue that it returns local paths from remote machines.

[jingxu97]

is there any progress on this? Is there on going work on replacing EvalSymlinks? Thanks!

[networkimprov]

#42201 failed.

[jingxu97]

I think this is still an important issue which is very hard to solve today. But there are lots of great discussion here and the other thread. So the plan is to document some of the conclusions somewhere?

[networkimprov]

AFAIK, there is no plan. Feel free to submit a change to the docs. It should indicate that ppl who need EvalSymlinks on Windows should use the WinAPI via x/sys/windows. And perhaps reference this issue and the one I mentioned above.

[gillg]

Related to #39786 (comment) as a nieth occurence

[gopherbot]

Change https://go.dev/cl/516695 mentions this issue: filepath: EvalSymlinks to not follow mount points on Windows

[gopherbot]

Change https://go.dev/cl/536655 mentions this issue: os,path/filepath: treat all non-symlink reparse points as irregular files

[gopherbot]

Change https://go.dev/cl/565136 mentions this issue: os: don't treat mount points as symbolic links