Don't generate symlinks

[DartBot]

Originally opened as dart-lang/sdk#15103

This issue was originally filed by kbec...@gmail.com

---

I am using dropbox to sync all my projects that I am working on. Unfortunately the symbolic packages link causes some severe confusion on the dropbox side because they are pointing outside the dropbox folder. And so it happens that dropbox syncs non-sense and erases pub-cache data.
Then two different computers are running that have a slightly different version set of dart, the sync process can cause the pub cache on both computers to become corrupt. This is highly annoying.
There is an option called selective sync, but when you create a new folder in any dart project that is open in the IDE, a new package symlink is created and the havoc continues.

I would thus recommend the following:
Create the option to make the packages folder in the project root a regular folder and COPY all files there.
Make symlinks from all sub package folders point to this package folder.
A pub get updates the package folder like if it were the pub-cache folder.

But having the option to not use symlinks in favor of code duplication in general would be awesome.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/5479?v=3" align="left" width="48" height="48"hspace="10"> Comment by sethladd

---

Removed Type-Defect label.
Added Type-Enhancement, Area-Pub, Triaged labels.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

We have a broader plan to stop using symlinks entirely in pub, in favor of "pub serve" and "pub build" for web-apps and the as-yet-unimplemented "pub run" command for command-line projects. I don't think we're likely to add a mode that copies files rather than symlinking them, but I'll leave this issue open to track the removal of symlinks, since it looks like we don't have an issue for that yet.

---

Changed the title to: "Don't rely on symlinks in pub".

[DartBot]

This comment was originally written by kbeck...@gmail.com

---

While in general I think getting rid of the symlink altogether is a good idea, relying on pub serve alone is not the best idea. Or at least it would need to get a bit more powerful.
Right now I have the problem that I dynamically generate Dart code that I need to serve and spawn. Unfortunately the origin policy applies to that very stricly. I thus can not spawn from localhost:9998 when my file is served from localhost:1030. The way I worked around that is that serve the static files from localhost:9998. This works because the packages files are there. If you would simply remove the symlink this would break.
So it would be pretty neat to allow pub serve to proxy a certain path to another server. For example /api/* would proxy to localhost:9998/api

Otherwise I am cheering in joy thinking about getting rid of the symlinks :)

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

If you're integrating pub serve with a custom server, we suggest you proxy requests from your server to the pub serve server.

[DartBot]

This comment was originally written by kbec...@gmail.com

---

I would rather prefer it the other way around as pub serve is not meant for production, so having a proxy there does not matter. But integrating proxy code into a server that is deployed into production is something I would rather want to avoid. Although certainly not critical. It is more of a nice to have feature.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Marked this as blocking #415.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Marked this as blocking #492.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/502633?v=3" align="left" width="48" height="48"hspace="10"> Comment by jbdeboer

---

Could you share a design doc on these broader plans? We have processes that are dependant on the current packages/ layout and don't want to be broken unexpectedly.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Removed Priority-Unassigned label.
Added Priority-Medium label.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Issue #843 has been merged into this issue.

---

cc @nex3.

[DartBot]

This comment was originally written by airborn...@gmail.com

---

I would like to see this done. I am using package webstart as a dart webserver and having non symlinked packages would be great for this.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

Issue #880 has been merged into this issue.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

Marked this as being blocked by dart-lang/sdk#19742.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/2130849?v=3" align="left" width="48" height="48"hspace="10"> Comment by kaendfinger

---

If I am correct, npm supports this, or I think it might not symlink at all. Anyways, I think this would be an amazing addition.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/405837?v=3" align="left" width="48" height="48"hspace="10"> Comment by zoechi

---

You could try using dart2dart to get rid of the symlinks
https://code.google.com/p/dart/issues/detail?id=19978#c1

[DartBot]

<img src="https://avatars.githubusercontent.com/u/1203892?v=3" align="left" width="48" height="48"hspace="10"> Comment by PaulAnnekov

---

There is one more problem appears when using symlinks. It's the problem with VirtualBox and its shares. Vagrant uses VirtualBox so it's relevant to developing process.

VB shares does not support symlinks from guest filesystem to shared folders if host OS is Windows. Bug ticket: https://www.virtualbox.org/ticket/10085

This leads to great discomfort when trying to launch console app in VB. I have solved this problem using pub build --all --mode=debug + adding the following transformation:

  transformers:
  - $dart2js:
      commandLineOptions: ['--output-type=dart']

But this solution is not clean. It's workaround. I want release app but I have to set mode=debug :(.

And one more thing. Even if I use pub build or pub run the error message due to symlinks still displayed (Cannot create link, path = '/home/user/packages/args' (OS Error: Protocol error, errno = 71)). But it displayed only if packages is not downloaded yet. I think that it tries to make symlinks.
Do I need to create new issue about this bug? pub run should not create symlinks.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

"pub build" and "pub run" need to run "pub get" if your lockfile is out-of-date, and "pub get" needs to create symlinks (at least until this issue is fixed).

[DartBot]

<img src="https://avatars.githubusercontent.com/u/1203892?v=3" align="left" width="48" height="48"hspace="10"> Comment by PaulAnnekov

---

Yep, that's what I'm talking about. I can't build or run project without errors in VirtualBox.

You should modify pub get so it had opportunity to just download packages without creating links (maybe using some argument). Then pub run can call it with this parameter and error won't happen.
Or pub should stop using symlinks.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

We can't get rid of symlinks entirely until issue dart-lang/sdk#19742 is fixed. Until then, you can pass "--package-symlinks=false" to any pub process and it won't create them. Note that this option is unsupported, though; it may change or disappear in the future without any notice.

[sethladd]

We plan to do this, but I wonder if it makes more sense to break this out into discrete issues. This feels like a meta-bug, which may be better captured by dart-lang/sdk#24112 ?

[jonaskello]

Would be nice if we could specify --package-symlinks=false in the pubspec.yaml. I know this is only temporary but it looks like it will take some time to fully get rid of the symlinks. If we could specify the behaviour of "pub get" in the pubspec.yaml then it would be possible to set it for all project members that work in a git repo. It does not have to be pubspec.yaml, any file that I could commit to a git repo that would make --package-symlinks=false the default would be nice. This way it would also work inside Webstorm (since I have found no way to do pub get from Webstorm GUI with --package-symlinks=false ).

[nex3]

I'd really rather not have a pubspec option that's only meaningful for a single release cycle.

[jonaskello]

Judging from related issues there seems to be several phases to remove symlinks. So I'm guessing it is not a single release cycle? There seems to be quite a few phases left, so the first version that does not create symlinks at all will probably be earliest at 1.15? And 1.13 is yet to come...

Anyway, the --package-symlinks=false does not work in Webstorm. Pubspec was just a suggestion, any way we could stop pub get in Webstorm from making symlinks would be useful. @nex3 what would you suggest instead?

[nex3]

There will be only one release in which --no-package-symlinks is (publicly) available and meaningful in pub. After that release, pub will never generate symlinks, and the flag will be a no-op.

Anyway, the --package-symlinks=false does not work in Webstorm. Pubspec was just a suggestion, any way we could stop pub get in Webstorm from making symlinks would be useful. @nex3 what would you suggest instead?

This seems like an issue to bring up with the folks working on WebStorm integration.

[jonaskello]

Just to clarify, 1.12 has --no-package-symlinks, so what you are saying is that 1.13 will not generate symlinks at all?

[nex3]

--no-package-symlinks is not publicly available in any stable version of pub. It does work for our own internal testing purposes, but it's not included in the pub help output and shouldn't be relied on for anything more than playing around with.

Once it's publicly released, pub will stop generating symlinks at all in the following release.

[jonaskello]

From what I gather, Webstorm used the --no-package-symlinks at one point but turned it off because some packages does not work properly without symlinks. So if pub will not have an option to turn symlinks off as a shared team policy (eg. by config file), and Webstorm will not use the --no-package-symlinks, I guess there is no option left for teams that use Webstorm but to wait for an unnamed future version where it is off by default.

[nex3]

The reason we're providing a period where it's available but off by default is to give users a chance to upgrade their packages to be compatible with a no-symlink world without locking everyone into that world. Like most options, it's supposed to be something that can be turned on and off as situations demand, so it doesn't really make sense for WebStorm to always or never pass the flag. It should pass the flag when the user wants to test without symlinks and not otherwise.

That said, maybe an IDE just isn't the best place for that sort of testing and you should run pub from the command line if you want fine-grained control.

[jonaskello]

Maybe what would make sense, both for now and the future, is that pub supports an environment variable for default parameters to pub get. That way we could set a policy on how a default "pub get" works regardless if it is called from command line or from Webstorm.

[jonaskello]

On second thought, an environment variable would set the policy on machine level which is not really what I want. What I really want is not fine grained control either. Instead I want per package control. I guess that means that pubspec would be the place to control the behaviour. The reason being that it allows you to have the policy set team-wide in git.

For now I found a way to turn off symlink generation on machine level by patching in this way:

On Windows:

• Find the file dart-sdk\bin\pub.bat.
• Patch line 27 to look like this:

"%BIN_DIR%\dart" %VM_OPTIONS% "%BIN_DIR%\snapshots\pub.dart.snapshot" --no-package-symlinks %*

On Mac:

• Find the file dart-sdk\bin\pub (this is a text file).
• Patch line 37 to look like this:

exec "$DART" "${VM_OPTIONS[@]}" "$SNAPSHOT" --no-package-symlinks "$@"

TIP: To find the path to the pub file, you can observe what path Webstorm prints to its terminal windows when you do a pub get from the GUI.

This will turn off symlinks both in Webstorm and command line for your machine. Yes, this is a hack, and would have to be re-done on each SDK update. But I just cannot get rid of the symlinks soon enough :-)

[genggoro]

Please don't get rid of the --packages-dir option entirely as suggested in this announcement (quoted below) at least until dart2js can do its job 10 times faster than now because "package" symlinks are indispensable to test Chrome extensions written in Dart using Dartium quickly (note that Chrome extensions can't be pub served as far as I know).

Once there’s been a release or two with the --no-packages-dir flag and users have a chance to make sure their packages work with only a package config, we’ll disable the flag by default. You’ll need to pass --packages-dir explicitly if you want it generated. A while after that, we’ll hide the flag and stop officially supporting it. We may get rid of it entirely one day, but there’s no hurry—it’s not that hard to support.

[mehaase]

I agree, if you get rid of --packages-dir, it seems like those of us who don't use pub serve will be left in the lurch. The Angular2 tutorial currently says:

We have a few options for running our app. One is to launch a local HTTP server and then view the app in Dartium. We can use any web server, such as WebStorm's server or Python's SimpleHTTPServer.

Is this use case not going to be supported in the future?

[zoechi]

Is this still relevant?

[nex3]

No, this has been fixed.