[turborepo] `turbo run` exits prematurely with code 2 #5260

be9 · 2023-06-09T11:08:43Z

What version of Turborepo are you using?

1.10.2 (same problem with 1.10.3-canary.0)

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Linux

Describe the Bug

I have a monorepo setup with 2 commands: lint and clean.

// turbo.json
{
  "$schema": "https://turbo.build/schema.json",

  "pipeline": {
    "lint": {},
    "clean": {
      "cache": false
    }
  },

  "globalDependencies": [
    "tsconfig.json",
    "configs/**",
    "lib/eslint-plugin/configs/**"
  ]
}

In a Docker image (Ubuntu 22.04-based), we run:

export TURBO_LOG_LEVEL=TRACE
pnpm turbo run lint --parallel --no-update-notifier --verbosity 3

While it works 100% correctly with turbo@1.10.1, after upgrading to 1.10.2, turbo started to exit prematurely with code 2, not running anything. The log looks like this:

[2023-06-09T10:56:12.100Z] 2023-06-09T10:55:03.129+0000 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.2
[2023-06-09T10:56:12.100Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /usr/src/go.git/node/node_modules/turbo-linux-64/bin/turbo
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /usr/src/go.git/node/node_modules/turbo/node_modules/turbo-linux-64/bin/turbo
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: Local turbo path: /usr/src/go.git/node/node_modules/.pnpm/turbo-linux-64@1.10.2/node_modules/turbo-linux-64/bin/turbo
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: Local turbo version: 1.10.2
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: Repository Root: /usr/src/go.git/node
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: Running local turbo binary in /usr/src/go.git/node/node_modules/.pnpm/turbo-linux-64@1.10.2/node_modules/turbo-linux-64/bin/turbo
[2023-06-09T10:56:12.101Z] 
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.131+0000 [DEBUG] turborepo_lib::shim: supports_skip_infer_and_single_package true
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.139Z [DEBUG] turbo: build tag: rust
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.566Z [INFO]  turbo: skipping turbod since we appear to be in a non-interactive context
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.567Z [DEBUG] turbo: global hash env vars: vars=[]
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.572Z [DEBUG] turbo: global hash: value=fed74f850fb476d9
[2023-06-09T10:56:12.101Z] 2023-06-09T10:55:03.572Z [DEBUG] turbo: local cache folder: path=""
[2023-06-09T10:56:12.101Z] + status=2
[2023-06-09T10:56:12.101Z] + echo 'turbo run exited with 2'
[2023-06-09T10:56:12.101Z] + exit 2

Expected Behavior

What worked in 1.10.1, should continue working in 1.10.2.

To Reproduce

export TURBO_LOG_LEVEL=TRACE
pnpm turbo run lint --parallel --no-update-notifier --verbosity 3

Unfortunately, I cannot provide sources 🤷

Reproduction Repo

No response

The text was updated successfully, but these errors were encountered:

tknickman · 2023-06-09T17:34:15Z

Hey @be9, is this 100% reproducible? Or is this an intermittent issue? (looking for a hint to get a reproduction setup for debugging)

be9 · 2023-06-10T10:46:09Z

It's failing in 100% of cases in Docker on our company's CI instance. Same commands work on macOS.

gsoltis · 2023-06-12T21:53:22Z

@be9 do you happen to know if your docker images have git installed?

be9 · 2023-06-13T03:09:18Z

@gsoltis sure they do. I can also add other stuff there, if needed. If you give me a build of turbo with more debug prints, I can run it, etc.

gsoltis · 2023-06-13T18:52:04Z

@be9 Got it. To be clear, git is not a strict dependency and is occasionally not present in docker containers, I just was curious if I could narrow down the bug to the fallback code.

I will keep digging and see what I can come up with.

gsoltis · 2023-06-15T19:42:39Z

@be9 Can you give it a try with turbo@1.10.4-canary.3 ?

We made some changes that should address it, and if not, should hopefully give us more information.

be9 · 2023-06-19T09:59:19Z

@gsoltis

#60 1.725  ERROR  run failed: error hashing package files: io error: Is a directory (os error 21)
#60 1.725 Turbo error: error hashing package files: io error: Is a directory (os error 21)

Still works on macOS, but fails in Docker.

jaa134 · 2023-06-20T00:39:46Z

+1 Experiencing the same exact problem.

Possible cause?
I was able to narrow down the issue to a change made in 1.10.2-canary.0 by re-running the CI process with various versions of Turbo installed. Switching from 1.10.1 to 1.10.2-canary.0 results in the CI step exiting early. @be9 maybe you can confirm the same? Does version 1.10.2-canary.0 also fail for you?

What exactly am I doing?

I have a monorepo using pnpm v8.6.2
In this monorepo, I have defined a Turbo pipeline that executes at the root level only. This pipeline runs unit tests and coverage collection using Vitest workspaces.
I can run this pipeline locally inside or outside a Docker image and everything works great regardless of the chosen Turbo version
I have a GitLab CI step that executes the same command inside a Docker container, but using any version after 1.10.1 results in the pipeline exiting immediately with error code 2.
- My problem is also 100% reproducible.
- The Docker image that is being used is node:18-alpine3.18
- I can confirm that git is installed using apk package manager.
- No change experienced when upgrading to v1.10.4-canary.3
Using version '1.10.1' or lower and the CI step executes the same as if it was run locally.

What have I found out?
After running the GitLab CI step with debug logging enabled and the Turbo pipeline with verbosity flag enabled (-vvv) I see this exact output.

pnpm turbo run test:coverage -vvv
2023-06-20T01:20:50.919+0000 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.2-canary.0
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /workspace/node_modules/turbo-linux-64/bin/turbo
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /workspace/node_modules/turbo/node_modules/turbo-linux-64/bin/turbo
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: Local turbo path: /workspace/node_modules/.pnpm/registry.npmjs.org+turbo-linux-64@1.10.2-canary.0/node_modules/turbo-linux-64/bin/turbo
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: Local turbo version: 1.10.2-canary.0
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: Repository Root: /workspace
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: Running local turbo binary in /workspace/node_modules/.pnpm/registry.npmjs.org+turbo-linux-64@1.10.2-canary.0/node_modules/turbo-linux-64/bin/turbo
2023-06-20T01:20:50.920+0000 [DEBUG] turborepo_lib::shim: supports_skip_infer_and_single_package true
+ runner_script_trap
+ exit_code=2
+ out_json='{"command_exit_code": 2, "script": "/scripts-23389372-4502013649/step_script"}'
+ echo 
+ echo '{"command_exit_code": 2, "script": "/scripts-23389372-4502013649/step_script"}'

The only difference that I can see between my local output and the CI output (beyond the exit code) is the addition of the runner_script_trap line which looks to be a feature of GitLab.

Upon further investigation...

I do not think GitLab is involved any way. I can make the GitLab CI image on my machine and execute the Turbo pipeline within that Docker environment. I see the same results now on my own machine as I was seeing on the CI.
After upgrading to 1.10.4-canary.3, I can no longer build my test image. I receive this error when using a Turbo pipeline to build my repo.

#21 1.157  ERROR  run failed: error hashing package files: io error: Is a directory (os error 21)
#21 1.158 Turbo error: error hashing package files: io error: Is a directory (os error 21)

After playing around with versions again, it looks like the build broke in 1.10.4-canary.0 maybe as a result of the hashing changes that went out in this release?

gsoltis · 2023-06-20T17:46:21Z

@be9 Odd that there's a platform difference. Are you sure both are using the same version of turbo?

Btw, my theory is that you have a symlink that we are attempting and failing to hash. Any chance you know if that's correct?
A working turbo run with --summarize should produce a json file that includes all of the paths that are hashed, if you want to verify.

gsoltis · 2023-06-20T17:51:18Z

@jaa134 I believe this is the hashing issue. I suspect you have a symlink that we are attempting to hash. The exit code 2 has been replaced with a closer error, io error: Is a directory (os error 21), which is indicative of the symlink situation.

jaa134 · 2023-06-21T00:39:37Z

@gsoltis

Are you sure both are using the same version of turbo?

I ran a CI pipeline for every version mentioned here from 1.10.1 to 1.10.4-canary.3. I ended up trying ~15 different versions of Turbo in total. It took me awhile, but I can assure you that:

every version beyond 1.10.1 is broken
the exit code 2 error message starts with version 1.10.2-canary.0
the io error: Is a directory (os error 21) error message starts with version 1.10.4-canary.0

Also, a different Turbo pipeline is now resulting in the io error: Is a directory (os error 21) error message. The new error occurs while trying to execute a build pipeline. The previous exit code 2 error occurred while trying to execute a unit test pipeline (when the build pipeline already succeeded). So upgrading to version 1.10.4-canary.0 has actually made my CI process fail earlier on than it did in version 1.10.2-canary.0.

Since, 2 different pipelines produced 2 different errors, I assumed this might have been a result of different issues in Turbo. Are you suspecting that's not the case and the 2 errors are actually related?

Later tonight I will report back with results for --sumarize using the new version 1.10.4-canary.4 and first broken version 1.10.2-canary.0.

gsoltis · 2023-06-21T01:14:41Z

We just released a new canary that should address the symlink issue, if that is in fact the underlying cause. The quick version of that is we used to accidentally ignore them, some new code stopped ignoring them, and now we are explicitly ignoring them until we can fix how we handle them. Can you check `turbo@1.10.4-canary.4`?

…

On Tue, Jun 20, 2023, 5:39 PM Jacob Alspaw ***@***.***> wrote: @gsoltis <https://github.com/gsoltis> Are you sure both are using the same version of turbo? I ran a CI pipeline for every version mentioned here <https://github.com/vercel/turbo/releases?page=1> from 1.10.1 to 1.10.4-canary.3. I ended up trying ~15 different versions of Turbo in total. It took me awhile, but I can assure you that: - every version beyond 1.10.1 is broken - the exit code 2 error message starts with version 1.10.2-canary.0 - the io error: Is a directory (os error 21) error message starts with version 1.10.4-canary.0 Also, a different Turbo pipeline is now resulting in the io error: Is a directory (os error 21) error message. The new error occurs while trying to execute a build pipeline. The previous exit code 2 error occurred while trying to execute a unit test pipeline (when the build pipeline already succeeded). So upgrading to version 1.10.4-canary.0 has actually made my CI process fail earlier on than it did in version 1.10.2-canary.0. Since, 2 different pipelines produced 2 different errors, I assumed this might have been a result of different issues in Turbo. Are you suspecting that's not the case and the 2 errors are actually related? Later tonight I will report back with results for --sumarize with version 1.10.4-canary.3 and 1.10.2-canary.0 — Reply to this email directly, view it on GitHub <#5260 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADGD7IE3OSWFGELEPRLTF3XMI7FJANCNFSM6AAAAAAZAP6W6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

be9 · 2023-06-21T01:49:18Z

@gsoltis it was a symlink indeed! 🙏

jaa134 · 2023-06-21T04:52:24Z

@gsoltis Its working for me now too 🙏 . Thank you very much for the support!

Inlustra · 2023-06-25T00:00:31Z

Super weird, I believe I'm seeing something along the same lines after the 1.10.4 canary push?

I'm also unable to run a --summarize due to the same error

Errors:

❯ turbo build --verbosity 3
2023-06-24T23:50:04.689+0000 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.6
2023-06-24T23:50:05.373+0000 [DEBUG] turborepo_lib::shim: Local turbo path: /Internal/rowm/node_modules/turbo-linux-64/bin/turbo
2023-06-24T23:50:05.373+0000 [DEBUG] turborepo_lib::shim: Local turbo version: 1.10.6
2023-06-24T23:50:05.373+0000 [DEBUG] turborepo_lib::shim: Repository Root: /Internal/rowm
2023-06-24T23:50:05.374+0000 [DEBUG] turborepo_lib::shim: Running local turbo binary in /Internal/rowm/node_modules/turbo-linux-64/bin/turbo

2023-06-24T23:50:05.375+0000 [DEBUG] turborepo_lib::shim: supports_skip_infer_and_single_package true
 ERROR  run failed: error hashing package files: invalid glob: failed to parse glob expression
Turbo error: error hashing package files: invalid glob: failed to parse glob expression

1.10.2 - Works
1.10.3 - Works
1.10.4-canary.1 - Broken <---- Assuming something to do with commit #5273
1.10.4-canary.2 - Broken
1.10.4-canary.3 - Broken
1.10.4-canary.4 - Broken
1.10.4 - Broken
1.10.5 - Broken
1.10.6 - Broken

Perhaps it's not the same bug, happy to open a new ticket

Edit, after more investigation, I don't think this is the same issue as it has nothing to do with symlinks, I've opened a new ticket.

be9 added area: turborepo kind: bug Something isn't working needs: triage New issues get this label. Remove it after triage labels Jun 9, 2023

tknickman assigned gsoltis Jun 12, 2023

gsoltis removed the needs: triage New issues get this label. Remove it after triage label Jun 12, 2023

gsoltis mentioned this issue Jun 20, 2023

fix(turborepo): Hashing symlinks is erroring #5340

Merged

gsoltis closed this as completed in #5340 Jun 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[turborepo] `turbo run` exits prematurely with code 2 #5260

[turborepo] `turbo run` exits prematurely with code 2 #5260

be9 commented Jun 9, 2023 •

edited

Loading

tknickman commented Jun 9, 2023 •

edited

Loading

be9 commented Jun 10, 2023 •

edited

Loading

gsoltis commented Jun 12, 2023

be9 commented Jun 13, 2023

gsoltis commented Jun 13, 2023

gsoltis commented Jun 15, 2023

be9 commented Jun 19, 2023

jaa134 commented Jun 20, 2023 •

edited

Loading

gsoltis commented Jun 20, 2023

gsoltis commented Jun 20, 2023

jaa134 commented Jun 21, 2023 •

edited

Loading

gsoltis commented Jun 21, 2023 via email •

edited

Loading

be9 commented Jun 21, 2023

jaa134 commented Jun 21, 2023

Inlustra commented Jun 25, 2023 •

edited

Loading

[turborepo] turbo run exits prematurely with code 2 #5260

[turborepo] turbo run exits prematurely with code 2 #5260

Comments

be9 commented Jun 9, 2023 • edited Loading

What version of Turborepo are you using?

What package manager are you using / does the bug impact?

What operating system are you using?

Describe the Bug

Expected Behavior

To Reproduce

Reproduction Repo

tknickman commented Jun 9, 2023 • edited Loading

be9 commented Jun 10, 2023 • edited Loading

gsoltis commented Jun 12, 2023

be9 commented Jun 13, 2023

gsoltis commented Jun 13, 2023

gsoltis commented Jun 15, 2023

be9 commented Jun 19, 2023

jaa134 commented Jun 20, 2023 • edited Loading

gsoltis commented Jun 20, 2023

gsoltis commented Jun 20, 2023

jaa134 commented Jun 21, 2023 • edited Loading

gsoltis commented Jun 21, 2023 via email • edited Loading

be9 commented Jun 21, 2023

jaa134 commented Jun 21, 2023

Inlustra commented Jun 25, 2023 • edited Loading

[turborepo] `turbo run` exits prematurely with code 2 #5260

[turborepo] `turbo run` exits prematurely with code 2 #5260

be9 commented Jun 9, 2023 •

edited

Loading

tknickman commented Jun 9, 2023 •

edited

Loading

be9 commented Jun 10, 2023 •

edited

Loading

jaa134 commented Jun 20, 2023 •

edited

Loading

jaa134 commented Jun 21, 2023 •

edited

Loading

gsoltis commented Jun 21, 2023 via email •

edited

Loading

Inlustra commented Jun 25, 2023 •

edited

Loading