Add support for --preserve-symlinks

[gaearon]

This is a thorny issue but I believe I'm beginning to understand it.

Basically there's two ways to treat symlinks:

• "Expand" their paths during resolution (aka "not preserving symlinks")
• Don't "expand" their paths during resolution (aka "preserving symlinks")

There are downsides to both (if my understanding is right).

Node by default does not preserve symlinks. They tried to change this behavior in Node 6 (nodejs/node#5950) but that backfired with non-trivial breakages and they reverted it soon and put behind a --preserve-symlinks flag (nodejs/node#6537). There is also an argument that the behavior isn't a bug or needs a different fix (https://github.com/isaacs/node6-module-system-change).

Regardless, it seems like some people prefer one behavior (with its set of tradeoffs) and other people prefer the other behavior (with another set of tradeoffs). Node currently supports both. Webpack does too (https://webpack.js.org/configuration/resolve/#resolve-symlinks).

Jest currently seems to support a mixture of both behaviors (?). I fixed at least some of it to match Node in #4761 (ironically despite "preserve" in the PR name the change aligned with Node's default behavior of not preserving them). I still think that PR was a good idea. However, apparently there are places where Jest still doesn't match node: #5228, #5085.

Assuming those get fixed, and Jest always does not preserve symlinks by default (just like Node), what do you think about allowing the "preserve symlinks" behavior as an opt-in, just like Node and webpack already offer?

[SEAPUNK]

Is there a reason Jest ignores the NODE_PRESERVE_SYMLINKS=1 exported envvar? Does it spawn worker processes ignoring the environment variables of the parent process?

Maybe it is, and I'm doing something wrong.

[lennartjansson]

We use Jest at Dropbox and we are also in need of the ability to preserve symlinks when Jest resolves file paths.

All of our automated test runs and frontend asset build steps are performed using Bazel. Notably, Bazel runs build tools and binaries inside sandboxes, temporary directories inside which every file is actually a read-only symlink that points outside of the sandbox. Bazel expects the test binary to treat the symlinks opaquely — as normal files relative to the sandbox root.

This has a couple consequences for Jest:

• When Jest performs test discovery, we need it to discover tests inside files that are actually symlinks. Unfortunately, the filesystem walk (find) used to discover test files will not return symlinks at all.

• Jest uses fs.realpath (or similar) to resolve symlinks in the default resolver and the script transformer — when these are resolved to paths outside the Bazel sandbox (i.e. outside of Jest's current working directory), code coverage reports will contain duplicated paths: one set of paths from inside the sandbox and another from outside.

We've started an internal fork of Jest at Dropbox, in which we've removed the realpath calls and changed the find call to also pick up symlinks — these changes were sufficient for us to get Jest playing nicely with Bazel. However, I assume symlink-preserving behavior would be useful not only to us or anyone hoping to integrate Jest with Bazel, but also anyone using Jest in any other kind of test or build environment in which source files and test files may actually be symlinks.

We'd be happy to contribute these changes back upstream behind a --preserve-symlinks flag (or perhaps two distinct flags if changes for the find call and the realpath calls are considered orthogonal). Are there any other concerns or details we should be aware of or take into account when implementing a PR for this?

[Globegitter]

@lennartjansson Hi Lenart, we at Ecosia are also using Bazel and just started integrating jest, running into the same issue. A PR would be really useful to us and also others it seems, given that the first issue on this topic has been opened already in 2016. Or if you could even just open-source the fork, then others could work around this issue for the time being.

On a different note, if you already have some Bazel jest rules and would be willing to open source it that would also be great, as this would be the next natural step for us also. Otherwise we would also be happy to collaborate on this.

[rodrigoalmeidaee]

In case it helps anyone, we are also waiting for jest to support symlinks out of the box. Until that happens, we wrote a collection of patches (mainly inspired in @lennartjansson's post and in an abandoned PR - https://github.com/facebook/jest/pull/4387/files). We apply these patches using a postinstall hook:

https://gist.github.com/rodrigoalmeidaee/a12d0c1e72f277d8c125c3c086607449

Of course these may break at any time with jest updates, so they aren't robust at all :-/

(Note: There is one extra patch there that is unrelated, it has to do with webpack-dev-server with we also had to patch to deal well with symlinks)

[Globegitter]

I just opened up #7364 which is functionally almost complete to implement this feature. If anyone wants to help out tu push this over the finish line quicker that would be much appreciated.

[lennartjansson]

@Globegitter I would be interested in helping to pick up work on your PR #7364 if you don't expect to be working more on it soon. I was planning eventually to work on a PR myself implementing what I described here but

@SimenB do you have any overall feedback or suggestions for the proposed behavior of the --preserve-symlinks flag? Namely,

• If --preserve-symlinks is on, symlinks that appear to be test files will be captured during the filesystem walk for test discovery. (without the flag, all symlinks will be skipped during test discovery)
• If --preserve-symlinks is on, source file paths that are actually symlinks will not be resolved to their underlying real file paths -- the symlink paths will be used as is. (without the flag, resolvers will continue resolving sourcemaps with fs.realpath)

(@Globegitter if the above doesn't match your intended implementation please correct me / let me know so we can design a behavior that will work for all of us.)

[Globegitter]

@lennartjansson yep the behaviour you described is the intention of #7364 (so it is a bit of an overload of the flag). I should actually have some time for it today, otherwise it will probably quite some time until I would get around to it again.

[Globegitter]

Ok I did not get around to do any progress after all. But if I should find some time again in the near future I'll make sure to post here first to try and avoid any double work.

[Globegitter]

@lennartjansson is coverage working for you with symlinks? We are on the latest 24.7.1 version with patches to fix symlinks and it has been working well for normal testing, but now with coverage turned on it is not working. If I manually copy the files over coverage is working again. So maybe we missed some part in the code, or istanbul underlying needs to be configured?

[CarlaTeo]

Updating @rodrigoalmeidaee patch to work with jest 24.7.1:
https://gist.github.com/CarlaTeo/710c56f1d481151ce96da823ed9e3cb7

[marcus-sa]

So... is this on the roadmap?

[pauldraper]

Jest now supports Plug'n'Play; it may be worth looking at how that was implemented.

[gcangussu]

Here is a custom resolver that solved problems to us: https://gist.github.com/gcangussu/af52da296aef829eba15ea626453f031
We are using Bazel.

[ACollectionOfAtoms]

@gcangussu I tried using your resolver (thanks for sharing!) doing something like:
jest test/symblinkedfile.test.js"
and no cigar. Is this not the way to use it?

[gcangussu]

@ACollectionOfAtoms I'm not sure if Jest will use the custom resolver to resolve the test suite file. I think it only uses it to resolve the modules you require(). Maybe you will need to use Node's --preserve-symlinks-main.

Edit: you can pass args to Node and run Jest like this:

node --preserve-symlinks --preserve-symlinks-main node_modules/.bin/jest test/symlinkedfile.test.js

[charrondev]

@ACollectionOfAtoms I'm not sure if Jest will use the custom resolver to resolve the test suite file. I think it only uses it to resolve the modules you require(). Maybe you will need to use Node's --preserve-symlinks-main.

Edit: you can pass args to Node and run Jest like this:

node --preserve-symlinks --preserve-symlinks-main node_modules/.bin/jest test/symlinkedfile.test.js

This worked perfectly for me!

[SimenB]

We'll be landing support for crawling symlinks with #9351. Work for preserve-symlinks is done in #9976, but I'm not sure if that's a viable approach or not. It might make sense still to use a custom resolver

[holylander]

@ACollectionOfAtoms I'm not sure if Jest will use the custom resolver to resolve the test suite file. I think it only uses it to resolve the modules you require(). Maybe you will need to use Node's --preserve-symlinks-main.
Edit: you can pass args to Node and run Jest like this:

node --preserve-symlinks --preserve-symlinks-main node_modules/.bin/jest test/symlinkedfile.test.js

This worked perfectly for me!

Thanks mate. This could be indeed the solution I was looking for. Do you know how I could use this line so I can configure my own vscode debug config for running jest in debug mode along the preserve-symlinks? thanks in advance

[shrinktofit]

Any updates?

[mistic]

@SimenB in order to use Bazel and Jest together we really need this to go through. We need an option to preserve the parent process argv containing --preserve-symlinks and --preserve-symlinks-main (or the equivalent set by NODE_OPTIONS) everytime jest spawns a new process.

[missing1984]

any update on this?

[marcus-sa]

any update on this?

If you're looking to resolve an issue regarding using Bazel with jest, then you can preserve symlinks in your jest config like this:

module.exports = {
  // ...
  haste: {
    enableSymlinks: true,
  },
}

[missing1984]

any update on this?

If you're looking to resolve an issue regarding using Bazel with jest, then you can preserve symlinks in your jest config like this:

module.exports = {
  // ...
  haste: {
    enableSymlinks: true,
  },
}

jest yell at me with:

haste.enableSymlinks is incompatible with watchman

Either set haste.enableSymlinks to false or do not use watchman

[missing1984]

There're 2 chunks of work needs to be done for supporting symlinks.

1. as @mistic said, pass --preserve-symlinks and --preserve-symlinks-main (or the equivalent set by NODE_OPTIONS) to every subprocess jest may spawn (this is default behavior in nodejs) . this will make sure native "require" preserve symlinks.

2. if jest uses any libraries that emulate "node require" resolution such as resolve, enhanced-resolve or haste? we need to make sure --preserve-symlinks flags get translated into their configuration correctly.

[centigrade-julian-lang]

Are there any updates? I'm trying to test an angular library which I symlinked into a angular test client app via npm link. It works well when serving the app, but breaks when testing it with jest.

import { MyLibModule } from 'my-lib'; // symlinked via npm link

describe('...', () => {
   // ...
   beforeEach(() => {
        await TestBed.configureTestingModule({
            imports: [MyLibModule],
         }).compileComponents();
   });
});

throws Cannot find module 'my-lib' from 'src/app/app.component.spec.ts'

[original001]

I've found a pretty workaround. I've replaced default jest resolver with my own, which use only resolve from browserify.

resolver.js

const resolve = require("resolve");

module.exports = (path, options) =>
  resolve.sync(path, { ...options, preserveSymlinks: true });

jest.config.js

  ....
  resolver: "<rootDir>/jestSetup/resolver.js",
  ...

Under the hood jest's defaultResolver already use resolver, but also have additional resolving via graceful-fs. But I don't know why. For my project it was enough to use only resolve.

[posti85]

Are there any updates? I'm trying to test an angular library which I symlinked into a angular test client app via npm link. It works well when serving the app, but breaks when testing it with jest.

import { MyLibModule } from 'my-lib'; // symlinked via npm link

describe('...', () => {
   // ...
   beforeEach(() => {
        await TestBed.configureTestingModule({
            imports: [MyLibModule],
         }).compileComponents();
   });
});

throws Cannot find module 'my-lib' from 'src/app/app.component.spec.ts'

I have also a library symlinked to an app (I'm developing both). I have found out my main app Jest tests fail when I run them after I was watching the library for changes. But if I build the library manually with its build command and then run the main app Jest tests, then they work properly 😕.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[cduff]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

I'm still working on projects that would benefit from Jest supporting --preserve-symlinks.

I have a repo structure like:

|- client
    |- node_modules
    |- shared
|- server
    |- node_modules
    |- shared (symlink to client/shared)

shared contains shared code, like shared data model classes, functions, etc.

I can get all other parts of my dev process to respect --preserve-symlinks except Jest.

In order to run Jest for server during dev it requires install of client/node_modules. During CI/CD I work around this by having a step to copy client/shared fully to server/shared before running Jest. Ideally these workarounds wouldn't be required.

[Yehonal]

6 years later...

Jest still lacks basic things like this, full ESM support etc.
Are you guys living the past?

Shall we use the resolver workaround above as a temporary official solution?

[bdemann]

Any update on this? We are running into this with ts-jest and we're having troubles even getting the resolver workaround to work with ES modules. Getting this issue resolved would be really nice

[aagranovExtend]

Any update on this? I'm running into this problem with Jest v6.2.5 on macOS.