Multiple local symlinks wrongly leading to TS2345

[ahom]

Error

While compiling a project that depends on local packages (symlinked from node_modules folder) I'm getting the following error even though both types are exactly the same.

src/app.tsx(21,65): error TS2345: Argument of type 'SqlPipeline' is not assignable to parameter of type 'Pass<string, Node>'.
  Types of property 'transform' are incompatible.
    Type '(input: string) => Node' is not assignable to type '(input: string) => Node'.
      Type 'Node' is not assignable to type 'Node'.
        Property '_children' is protected but type 'Node' is not a class derived from 'Node'.

Symlinks setup

Here I'm trying to compile the debug/sql_to_mongo project

core/

frontend/sql/node_modules/llqm-core -> ../../../core

backend/mongo/node_modules/llqm-core -> ../../../core

debug/sql_to_mongo/node_modules/llqm-core -> ../../../core
debug/sql_to_mongo/node_modules/llqm-frontend-sql-> ../../../frontend/sql
debug/sql_to_mongo/node_modules/llqm-backend-mongo -> ../../../backend/mongo

The problem seems to be happening because when resolving the llqm-core module from debug/sql_to_mongo, frontend/sql and backend/mongo they each point, from Typescript's perspective, to different folders, but they really all resolve to the same one.

Compiling with --listFiles to show the dependencies being resolved

src/app.tsx(21,65): error TS2345: Argument of type 'SqlPipeline' is not assignable to parameter of type 'Pass<string, Node>'.
  Types of property 'transform' are incompatible.
    Type '(input: string) => Node' is not assignable to type '(input: string) => Node'.
      Type 'Node' is not assignable to type 'Node'.
        Property '_children' is protected but type 'Node' is not a class derived from 'Node'.
/home/ahom/projects/llqm/packages/debug/sql_to_mongo/node_modules/typescript/lib/lib.d.ts
node_modules/llqm-core/src/pass.ts
node_modules/llqm-core/src/pipeline.ts
node_modules/llqm-core/src/ir/node.ts
node_modules/llqm-core/src/compiler.ts
node_modules/llqm-core/src/schema.ts
node_modules/llqm-core/src/ir/operation.ts
node_modules/llqm-core/src/ir/query.ts
node_modules/llqm-core/src/ir/index.ts
node_modules/llqm-core/index.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/pass.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/pipeline.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/ir/node.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/compiler.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/schema.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/ir/operation.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/ir/query.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/src/ir/index.ts
node_modules/llqm-frontend-sql/node_modules/llqm-core/index.ts
node_modules/llqm-frontend-sql/src/ast.ts
node_modules/llqm-frontend-sql/src/loc.ts
node_modules/llqm-frontend-sql/src/lexer.ts
node_modules/llqm-frontend-sql/src/lexer_pass.ts
node_modules/llqm-frontend-sql/src/remove_tokens_pass.ts
node_modules/llqm-frontend-sql/src/parser.ts
node_modules/llqm-frontend-sql/src/parser_pass.ts
node_modules/llqm-frontend-sql/src/gencode_pass.ts
node_modules/llqm-frontend-sql/src/pipeline.ts
node_modules/llqm-frontend-sql/src/mode.ts
node_modules/llqm-frontend-sql/index.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/pass.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/pipeline.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/ir/node.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/compiler.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/schema.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/ir/operation.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/ir/query.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/src/ir/index.ts
node_modules/llqm-backend-mongo/node_modules/llqm-core/index.ts
node_modules/llqm-backend-mongo/src/gencode_pass.ts
node_modules/llqm-backend-mongo/index.ts
src/editor.tsx
typings/react/react.d.ts
src/react-object-inspector.d.ts
src/compile_log.tsx
src/utils.ts
src/app.tsx
src/boot.tsx
typings/codemirror/codemirror.d.ts
typings/react-bootstrap/react-bootstrap.d.ts
typings/react/react-dom.d.ts
typings/tsd.d.ts

Hacky fix

By modifying the resolution of external modules to follow symlinks (with fs.realpath) it is now compiling properly.

• Would it be ok to provide a PR for that?
• Would you know a good place where this should be called?

Right now I:

• added a fs.realpath call to ts.sys.resolvePath (for node)
• and called it from ts.resolveModuleNameonly when it is an external import

I'm not too sure if this is the right way to go.

--listFiles after fix

/home/ahom/projects/llqm/packages/debug/sql_to_mongo/node_modules/typescript/lib/lib.d.ts
/home/ahom/projects/llqm/packages/core/src/pass.ts
/home/ahom/projects/llqm/packages/core/src/pipeline.ts
/home/ahom/projects/llqm/packages/core/src/ir/node.ts
/home/ahom/projects/llqm/packages/core/src/compiler.ts
/home/ahom/projects/llqm/packages/core/src/schema.ts
/home/ahom/projects/llqm/packages/core/src/ir/operation.ts
/home/ahom/projects/llqm/packages/core/src/ir/query.ts
/home/ahom/projects/llqm/packages/core/src/ir/index.ts
/home/ahom/projects/llqm/packages/core/index.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/ast.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/loc.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/lexer.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/lexer_pass.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/remove_tokens_pass.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/parser.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/parser_pass.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/gencode_pass.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/pipeline.ts
/home/ahom/projects/llqm/packages/frontend/sql/src/mode.ts
/home/ahom/projects/llqm/packages/frontend/sql/index.ts
/home/ahom/projects/llqm/packages/backend/mongo/src/gencode_pass.ts
/home/ahom/projects/llqm/packages/backend/mongo/index.ts
src/editor.tsx
typings/react/react.d.ts
src/react-object-inspector.d.ts
src/compile_log.tsx
src/utils.ts
src/app.tsx
src/boot.tsx
typings/codemirror/codemirror.d.ts
typings/react-bootstrap/react-bootstrap.d.ts
typings/react/react-dom.d.ts
typings/tsd.d.ts

Thanks!

[greyepoxy]

Nice write up @ahom. I ran into this exact same problem today but was unable to figure out what exactly was going on. +1 for a fix for this please.

[mhegazy]

one thing to note, fs.realPath is significantly slow.. see nodejs/node#2680. until this issue is fixed, i do not think we can use fs.realPath.

[ahom]

@mhegazy I'm not sure I get this right, in the linked ticket they point out that resolving ~1000 symlinks would take about 45ms. I understand it is orders of magnitude slower than the C call but is it really impacting that much the big picture?

I quickly timed the compilation of my project with and without the call to fs.realpath (without the call it fails midway but it was just to see the timings) and 45ms seems very negligible when looking at the ~1.4 sec it takes to compile. (it only needed to resolve 10 symlinks in my case anyway because it is only called at resolution time of a new external module)

without fs.realpath

$ time ./node_modules/.bin/tsc
src/app.tsx(21,65): error TS2345: Argument of type 'SqlPipeline' is not assignable to parameter of type 'Pas
  Types of property 'transform' are incompatible.
    Type '(input: string) => Node' is not assignable to type '(input: string) => Node'.
      Type 'Node' is not assignable to type 'Node'.
        Property '_children' is protected but type 'Node' is not a class derived from 'Node'.
Command exited with non-zero status 2
2.05user 0.04system 0:01.51elapsed 139%CPU (0avgtext+0avgdata 127004maxresident)k
0inputs+768outputs (0major+40701minor)pagefaults 0swaps

with fs.realpath

$ time ./node_modules/.bin/tsc
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/frontend/sql/index.ts
Resolved file: /home/ahom/projects/llqm/packages/backend/mongo/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
Resolved file: /home/ahom/projects/llqm/packages/core/index.ts
1.93user 0.01system 0:01.44elapsed 135%CPU (0avgtext+0avgdata 123764maxresident)k
0inputs+480outputs (0major+35828minor)pagefaults 0swaps

[mhegazy]

We will have to do it on reading any files. i do not see why a module is a special case. the impact of small projects may not be big but would be observable on larger solutions.
@vladima can you give this a try and report perf numbers.
there is still another issue we need to figure out how this can be solved for non-node engines.

[andreialecu]

I'm not sure if this is related, but I'm seeing issues on Windows when trying to extend a class defined in a package that is in a symlinked folder in node_modules.

s:\Projects\snip\snip\node_modules\typescript\lib\typescript.js:44463
                throw new Error("Could not find file: '" + fileName + "'.");
                ^

Error: Could not find file: 's:/Projects/snip/snip/node_modules/snip-shared/lib/Utils.ts'.
    at getValidSourceFile (s:\Projects\snip\snip\node_modules\typescript\lib\typescript.js:44463:23)
    at Object.getSyntacticDiagnostics (s:\Projects\snip\snip\node_modules\typescript\lib\typescript.js:44633:52)
    at s:\Projects\snip\snip\node_modules\ts-loader\index.js:317:42
    at Array.forEach (native)
    at Compiler.<anonymous> (s:\Projects\snip\snip\node_modules\ts-loader\index.js:316:14)
    at Compiler.next (s:\Projects\snip\snip\node_modules\webpack\node_modules\tapable\lib\Tapable.js:69:14)
    at Compiler.<anonymous> (s:\Projects\snip\snip\node_modules\webpack\lib\CachePlugin.js:40:4)
    at Compiler.next (s:\Projects\snip\snip\node_modules\webpack\node_modules\tapable\lib\Tapable.js:69:14)
    at PathRewriter.onAfterCompile (s:\Projects\snip\snip\node_modules\webpack-path-rewriter\index.js:108:9)
    at Compiler.<anonymous> (s:\Projects\snip\snip\node_modules\webpack-path-rewriter\index.js:93:17)

(I replaced my actual project name with snip)

In this example, snip-shared is a library that is npm linked to the node_modules folder of the snip main project.

Utils.ts is a dependency of the base class I'm trying to extend, is only imported by the base class, and is located in snip-shared (inside the symlinked folder).

If I just try to use the base class in snip-shared without trying to extend it from a class in snip, everything works properly. (Edit: apparently I'm wrong, this didn't work either, see update below)

The interesting thing is that the file actually exists at that location even though the error says that it doesn't.

If I copy the entire folder contents to node_modules instead of using a symlink everything works properly.

ts-loader and webpack are also involved here, but I'm not sure if they're at fault. The error seems to come from typescript.

Update: Explicitly importing the Utils.ts file/class in the extending class file fixed the error. I'm not sure if this is expected, but it definitely works without the explicit import when not using a symlink.

[sebastianhoitz]

@andreialecu I have the same issue, also involving webpack and tls-loader. The file exists, but is located in a symlinked folder.

[andreialecu]

@sebastianhoitz I was able to get it to work. The trick is to ensure all other dependencies of the imported class are also explicitly imported in your main project, otherwise you will get that error I pasted above.

Until this is fixed, I can live with this workaround.

Also, make sure your webpack.config.js has

    resolve: {
      fallback: [path.join(__dirname, 'node_modules')]
    },
    resolveLoader: {
      fallback: [path.join(__dirname, 'node_modules')]
    },

Otherwise you may run into other issues with symlinked files.

[ahom]

@andreialecu @sebastianhoitz I also got that issue when using webpack and ts-loader (and decided to drop the usage of ts-loader for now). From what I've seen it is clearly also related to symlink and how they are resolved.

ts-loader provides a resolveModuleName to Typescript and follow the symlinks as shown here: https://github.com/TypeStrong/ts-loader/blob/master/resolver.ts#L53

At some point, it leads to some issues because of a cache (of filename -> content) they are using in which they can't find the "unresolved" symlinked path because it is registered (in the cache) with the resolved path.

I think that this specific issue should be opened on ts-loader side.

[greyepoxy]

@ahom @mhegazy I was able to work around the issue for me without using @ahom 's fix.

Now that I understand what is going on I actually think the solution I landed on will be better in the long term for my project.

The error I was running into (and it looks like @ahom is running into as well) is at its core an issue with Type Compatibility. My solution was to make sure that all of my exported apis take in interfaces instead of classes. This works because interface type compatibility checks are a lot looser then class compatibility checks. For me private and protected members kept causing problems.

When an instance of a class is checked for compatibility, if the instance contains a private member, then the target type must also contain a private member that originated from the same class.
Type Compatability - Private and protected members in classes

@ahom Unless you intend your node modules to always be paired together, in my opinion it is better to change to this style instead of relying on the Typescript compiler to resolve symlinks for node modules. This is because if you ever upload your node modules independently to npm (or another binary repository) then your modules are likely going to have direct dependencies on the versions of what your other modules import. Not to mention your compile will break at that point because the Typescript compiler will run into the same problem as it is currently (resolves class definitions to different files since there would be no symlinking).

Anyway sorry for the unsolicited advice. This is what ended up working for me, let me know if it does not make sense.

[ahom]

@greyepoxy Thanks for sharing your solution! I understand that it can be worked around but I think this is a completely valid usecase that Typescript should be able to handle. When you setup a new project, you want to iterate quickly even if it is spread over multiple libraries and an interface is not always desirable and is just annoying overhead for the most simple cases.

In addition, even when the modules will be uploaded independently, npm will just flattify the dependency tree (of course you should use versions that are compatible) and everything will be resolved correctly. (as it does now without symlinks)

[greyepoxy]

@ahom yeah I agree this should definitely be fixed.

Good point about npm flattening dependencies. For some reason I was under the impression that each module would have all of its dependencies copied in its local node_modules. Your totally right though, the npm install docs actually have a great description of how the flattening works. Would have to make sure that all of the dependent modules are uploaded with the same version numbers but if you control the upload then not really an issue.