-
Notifications
You must be signed in to change notification settings - Fork 823
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Symlinks on the mounted Windows directories are not compatible with native. #353
Comments
Looks like cmd.exe does not (yet?) know WSL symlinks and it treats every unknown reparse point as a junction. But actually this is not a junction. It is a new type of reparse point whose reparse tag is 0xA000001D. Hopefully it will be documented in the future Windows 10 Anniversary SDK. Unlike Windows symlinks/junctions, no filesystem/filter drivers resolve WSL symlinks automatically. I guess this is an intentional decision to prevent Windows apps from bypassing SeCreateSymbolicLinkPrivilege by using WSL symlinks. |
I do owe some documentation on this one. Should get that up in the next few days. @vyv03354 is correct that the primary reason we are not adding symlinks to Windows is because of the privilege required. As of now, symlinks created in WSL are only usable within WSL. This is something we will be looking at in later releases. |
@russalex Can it not be the case that if the user running WSL has the SeCreateSymbolicLinkPrivilege that WSL creates NTFS format symbolic links instead of a custom link type? |
Sven covers this question a little bit in his video on the WSL File system Blog post. There have been some discussions on doing this in the next release, but there are some questions and behavior that we would need to work out first. I know there is a user voice page on this very request. Right now it only has 3 votes. |
Personally, coming from Linux and OSX the entire SeCreateSymbolicLinkPrivilege situation is one that has repeatedly caused me headache and grief. I was hoping against hope that BoW would finally fix the issue but, alas, it seems instead of making symlinks work properly in user space, MS has decided to just make a new standard? I can't follow any of the symlinks created in the linux subsystem even in windows exporer, much less in my IDE or other windows host apps. This makes me sad. :( |
@rcuddy It is very straightforward to award any user group the SeCreateSymbolicLinkPrivilege and then you can create symbolic links to your heart's content! Note that any user with the ability to escalate to administrator still cannot create links, so you'll need to make your user an ordinary user, not administrator capable user for this to work. |
@ned14 Sure, I can add that privilege, but as you've pointed out, I then lose the ability for that user to escalate privileges. That doesn't seem a fair trade. ;) |
Is there any workaround to this issue? I now ran into a problem there when I tried to do a backup of a folder using robocopy. It's a development folder and npm (under WSL) created a few symlinks from |
Repost of the suggestion I just posted on Issue #1475: lxss symlinks also break my win32 backup tools. The tools on both sides of the programming world should be able to view and copy directory trees created on the other side, with all links in the copy usable by both sides as well.
Then the subsystems can convert the link targets to absolute paths formatted as their respective user-mode programs expect. |
Would also like to see this. I've tried it both ways (link and target both on Windows file system):
With approach 1 I observe the behaviour mentioned here. Also the link will simply not appear in my editor on the Windows side (VS Code). In Windows Explorer it will appear but give an error upon trying to open it. However it works with everything on the WSL side. With approach 2 everything works on the Windows side (as you would expect), but the symlink isn't visible and doesn't work on the WSL side. However, subsequently trying approach 1 results in this error: So to sum things up, a symlink created on either side cannot be effectively utilised by the other side, kinda breaking the file system interoperability promise. I would expect DriveFS to correctly handle symlinks created with Thanks for your consideration. |
Yes, this a big mess especially when git gets involved.... |
Bash also doesn't see Windows symlinks(or at least ls refuses to list them). And yea... it gets worse with git. |
I have a mess of utilities installed redundantly on Windows and Ubuntu due to this issue. Hopefully it will get resolved at some point. |
I would also like this. Currently on the Bash side we're using Yarn for dependency management. We're sharing files between Windows and Bash by using /mnt/c/... . Yarn ends up using symlinks and works well if you're entirely in the Bash side. However, these symlinks are not seen on the windows side properly, which breaks development tools over there. Note: if i manually make a junction or symlink on the Windows side (i use the Link Shell extension here: http://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html) then the Bash side sees things fine. So there does appear to be interoperable symlinks, it's just that "ln -s" doesn't make them inside Bash. Fixing this would be a godsend. Thanks! Note: I'm on 16251. |
I was sad to hear that you could not create/edit files in the wsl system from Windows. The workaround was said to be that we should create files (for example, from git/npm/yarn) on the Windows side of things, and then access those files from Windows and WSL. Well when doing what is suggested, you run into this issue with symlinks very quickly. The first thing I tried was log into So accessing WSL files from Windows doesn't work (okay, I was sad but got over it), but then I discovered that in practice, the encouraged workflow (access Windows files from both Windows and WSL) doesn't work in practice either because many of the kinds of files/repos/projects you want to setup on the Windows mount, require symlinks to work correctly. Do the Thank you very much for your investment in WSL - I'm very excited about the long term potential of this project, and I consider this to be the final blocker to actually using WSL instead of Cygwin/MSys for day to day project development. |
@jordwalke The way I work around that is to have separate build directories for each source project, one for WSL, one for Windows. |
@ned14 yeah, I imagine something like that would be possible, but at that point, I would just rather consider rsyncing two directories between the two systems which pretty much treats WSL like a VM but faster. I just think WSL could be so much more than that if Windows would just support Linux created symlinks from the windows drive to the windows drive. (I'm not even asking for the ability for Windows to create/touch Linux files - it should be fine as long as we can create files on the Windows drive using the Linux command line (including symlinks) which Windows will respect). |
@jordwalke - @SvenGroot is working on unifying the symlinks story between WSL and Windows so that WSL will create symlinks in a way that Windows can understand them. |
So glad to hear that. Are you addressing the absymally slow file performance as well :) ? |
There's only so much they can do about that. On native Windows, opening a file is approx 15x slower than Linux. 4Kb i/o is approx 3x slower, and so on. Those figures are for warm cached filesystem. Now, WSL is many times slower again, but ultimately there is only so much they can do. You will have to accept that filesystem heavy Linux programs will always run under WSL at a fraction of Linux. |
File system performance is always a priority for us, and we are working on several ways to improve this. We've already made significant strides in RS3 (the upcoming Fall Creator's Update), and are continuing to work to improve this further. There are limits to what we can do due to requirements of application and file system filter compatibility, but we're very much aware that file system performance is a pain point for WSL (and Windows as a whole) and are working to improve the situation. |
@SvenGroot - Thanks for addressing this issue. If you're looking for a beta tester, I volunteer. |
I'm very much looking forward to this upcoming feature. When does the next Creators update get released? |
Update for anyone following this issue: I installed the Fall Creator's Update, and it looks like symlinks created from the Windows side (e.g. using Links created on the WSL side (e.g. using If you have stuff that relies on symlinks, it may be worth creating them or "initializing" things (e.g. git cloning) on the Windows side if possible (if you want interoperability). |
Lucky me, I've been waiting to really use WSL until today -- thanks @tinysun212 for getting this ball rolling two years ago! |
I'm running version 1803 (build 17134.191), but symlinks to files created on WSL are still being treated as directory junctions, making it inaccessible. |
Thanks for the work you put in the symlinks, @SvenGroot . I do actually have a question about the requirements you posted, in order for WSL to make a Windows symlink:
Today I noticed a strange effect with VirtualBox and its shared folders. I can allow VB to create symlinks as well, for example by running it elevated (or enabling developer mode). Then VB does allow me to create any symlink I want inside the Linux guest, also to non-Windows locations. For example: I think in practice this behaviour (making symlinks even though the requirements are not met) has the same effect as making the symlink windows-incompatible as is done in WSL now: Windows cannot do anything with it, but VB/WSL can. On the other hand, if I were to create this symlink in WSL currently, I cannot access it in VirtualBox, which has actually broken some things for me (the links were incorrectly read by VB). So my question then is: Is VirtualBox doing something tricky here? I would assume the Windows API would also enforce these requirements, but apparently not? Is this behaviour desirable or potentially harmful? Would it be an idea to adopt this behaviour in WSL as well, to provide compatibility between VB and WSL? (or is it better/easier to fix VB to support the reparse tag used by WSL?) Or for that matter, not just VB but I can also think of NFS shares and many other things. |
Can this issue be reopened? |
I know I'm in unsupported territory with the following: Looking to find the culprit, I found this difference between a xfce4 session on /home/USER, and /mnt/h/USER: It looks like the symlink from DrvFs to VolFs is not permitted? I am using Windows 10 Version 1809, and this is with a "fresh" copy of Ubuntu 18.04LTS.... I'm not sure if this relates to the topic in this threat (where the stated problem is that the Windows symlink is not compatible with the WSL symlink)... But it feels to me it is all part of the same underlying problem... Since this issue 353 is closed, must I assume my problem is actually supposed to be resolved? |
Should be, if you've got Windows symlinks enabled. But you've invited OneDrive to the party, that adds a variable. Try the same scenario with a DrvFS mount that is NTFS not OneDrive, and see if it still fails. You'll need WSL metadata enabled, and I take as a given WSL metadata isn't supported by OneDrive (or, say, FAT32). I am not sure whether Windows symlinks are supported by OneDrive or not (never looked). If they are not that would be a OneDrive ask not a WSL ask. But given the base problem ( |
Thank you ! I took OneDrive out of the equation (although I think it just plain NTFS monitored by sync)... This is my /etc/wsl.conf:
This is my /etc/fstab:
The result of WSL 'mount -l':
Some observations:
Then my experiments:
Note that Symbolic links do work on my Windows 10 machine (I have had the Developer Mode setting turned on since the early days of WSL):
Am I missing a setting? I'm worried about 'mount -l' not showing the metadata option... Confirming:
Using Ubuntu 18.04 LTS |
If you mount using a UNC path, DrvFs goes through SMB rather than directly to NTFS. This is both a lot slower and lacks quite a bit of functionality, including metadata, case sensitivity, and indeed symlinks. It is strongly not recommended to mount DrvFs this way for local filesystems. To achieve the effect you want, use bind mounts instead. Put the following in your /etc/fstab:
Note you have to explicitly mount C: before creating the bind mount because we process /etc/fstab before automounting DrvFs volumes. |
Thank you, that does the trick indeed... I have found the blogs (https://devblogs.microsoft.com/commandline/) to be very instructive, but I probably missed the advise you provided above: don't quite know which blogs I should read. Maybe I should just prepare a pot of coffee, and read them all... I think the key for an even more successful WSL would be a comprehensive documentation site that organizes everything there is to know? Anyway, my apologies for not using a bind mount, that is one pearl of great information!!! Oh: Once I switch to a bind mount, there is also no issue with OneDrive: I can put symbolic links in there to my heart's content! My WSL Home Directory is now fully managed in OneDrive, and I can even run XFCE4 from that home directory!!! |
I really wonder why there isn't an easy way out: Instead of the current Windows symlink reparse point, invent a new symlink reparse point which is type agnostic (not file or dir, but "just a symlink"). Create symlinks using this new type on both sides of the fence (Windows/WSL). Support the old symlink reparse point for backward compat only. This new symlink reparse point could be defined to contain a Windows path as well as a POSIX path, so it could simply work for files and dirs both, and it would finally be possible to create working symlinks with no target on the Windows side as well. What is so complicated about this that this hasn't already been done? Corinna |
Corinna, symlinks/reparse points are a feature of the File System. New file systems don't come around that often. And this is not a problem that could force the whole windows ecosystem to switch to a new file system. |
Creating and evaluating a reparse point content is an OS thing, and since Having said that, new reparse points can be easily defined by Microsoft, as you can The idea is that the Windows side refrains from requiring the file type. This can Corinna |
So... Cygwin classic symlinks. Shortcuts. Registry shell objects (aka #2779). WSL reparse point symlinks. Windows native symlinks. Windows NFS symlinks. And a new kind.
Sure they do. The irony is all the WSL devs are doing their kernel development on (MSFT's most unfortunately named) GVFS. It caches metadata (because of course it does) to work-around the underlying problems with MSFT's Plan 9 filesystem implementation was released just last month.
this. |
Removing the "matching file type" restriction on existing symlinks, or introducing a new symlink type that doesn't have that restriction, is definitely something we looked at. Unfortunately, the assumption that the symlink has the same type as what you'll eventually open is baked into the system at a bunch of places, and done in such a way that it would be hard to change without likely breaking something. Backwards compatibility can be a pain sometimes. While it wouldn't be impossible to pull this off, it would definitely not be trivial. |
I see. When Windows introduced symlinks back in Vista, we@Cygwin were very happy at first. Unfortunately the target-type flag broke hope pretty quickly. Which is why Cygwin still doesn't use native symlinks by default and doesn't allow to create dangling native symlinks so as not to break any assumptions outside its own realm. But either way, I'm pretty sure that only symlinks working transparently on both sides will fix all issues with symlinks. Corinna |
Firstly, Corinna I'd like to thank you so much for all the material on the internal workings of the NT kernel API which you have built up over many years on various mailing lists. I found them invaluable during the development of LLFIO, the reference library implementation for https://wg21.link/P1031 Low level file i/o library. Thank you so much for that. I'd like to raise the point that fixing Windows symlinks to work well between WSL and native would help solve a long standing bug when using git on Windows, or git within WSL on a mounted Windows drive, where somebody has committed a symbolic link to the repo. Right now it doesn't work most of the time, and moreover, it doesn't work in a particularly non-obvious and hard to detect way. If Windows symlinks understood both Windows and POSIX semantics, then git-created symlinks - whether created by a git inside WSL or outside - would "just work". This ought to be not hard to fix - git-created symlinks are always internal to the repo, referencing one part of the git repo from another part of the git repo. They should "just work". |
I posted that I tested and it was solved in version 1803. Now, my PC is updated to 1809 (OS Build 17763.379). After testing again, I found that the result of the test which I wrote in the first post is reappeared. The OS is regressed at this issue.
This issue should be reopened. Does anyone know how to re-open this? @therealkenc |
Thanks!
Do you have a pointer to a description what exactly goes wrong?
The fact that the type check is "baked into the system at a bunch of places" as @SvenGroot outlines is really one of the major problems. Oh well. Corinna |
It was some time ago I last wrestled with this issue, so my memory may be faulty. The principle cause are repos which use libtool, and they ship in the repo prebuilt binaries linked by libtool. That said, I've occasionally bumped into repos which are Unix-only, and thus they think it's fine to commit symlinks. There are two separate problems:
Now I appreciate that relative vs absolute symlinks on Windows is a bit fickle. Plenty of absolute symlinks ought to be relative. But it would be a start if Windows symlinks with the Outside WSL, it's much harder. Symlinks have their own reparse type. Me personally, I would have WSL create Windows symlinks all the time if the symlink is a relative path, but with the Obviously absolute path symlinks are much harder to solve. But relative path symlinks I think are very solvable, and without needing to invent new reparse point types, or break anything existing. |
I think this should be the general idea, but it's a bit tricky if you don't bump the
I don't think so. The content of the POSIX path inside this symlink could be absolute Corinna |
As far as I am aware, Therefore using it to store the POSIX symlink ought to work absolutely fine. Nothing breaks. All WSL needs to do is decide if it's a valid POSIX path, and that's trivially easy. |
I only saw this used in cmd's dir output, too, so far. I hope this is a safe bet. Corinna |
|
Nowadays, windows 10 (2004) still can't open symlinks created in WSL2. windows treat them as general files. |
Landing zone for that over at #4868. |
On build 14341,
I created a symbolic link to a file on a mounted Windows directory in BASH.
In the command prompt CMD.exe, the file appeared as a JUNCTION and can not be accessed.
It should be displayed as follows.
I think the 'JUNCTION' can be used when the link references a directory not a file.
The text was updated successfully, but these errors were encountered: