Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dry run doesn't output tasks that changed for modified root workspace inputs #8762

Open
1 task done
psychobolt opened this issue Jul 16, 2024 · 6 comments · Fixed by #8814
Open
1 task done

Dry run doesn't output tasks that changed for modified root workspace inputs #8762

psychobolt opened this issue Jul 16, 2024 · 6 comments · Fixed by #8814
Labels
area: docs Improvements or additions to documentation

Comments

@psychobolt
Copy link

psychobolt commented Jul 16, 2024

Verify canary release

  • I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://codesandbox.io/p/devbox/test-turbo-inputs-lwndsy

What package manager are you using / does the bug impact?

Yarn v2/v3/v4 (node_modules linker only)

What operating system are you using?

Mac

Which canary version will you have in your reproduction?

2.0.7-canary.1

Describe the Bug

With the new v2, I'm not sure if dry run will output what packages has changed if workspace is filtered e.g. workspaces/*.... It seems to always print the same result.

Expected Behavior

Only show tasks for workspaces that changed

To Reproduce

My use case for this sandbox seems to not work e.g. yarn turbo run build --filter=./packages/\* --filter=\[HEAD\] --dry and yarn turbo run build --filter=./packages/\* --filter=\[HEAD^1\] --dry . I have one workspace with the following config:

{
  "$schema": "https://turbo.build/schema.json",
  "extends": ["//"],
  "tasks": {
    "build": {
      "inputs": ["$TURBO_DEFAULT$", "../../.root/*"],
      "dependsOn": ["^build"]
    }
  }
}

git diff HEAD shows 0 change.

git diff HEAD^1 shows 1 change (root workspace input):

diff --git a/.root/foo b/.root/foo
index 04abf92..29f0220 100644
--- a/.root/foo
+++ b/.root/foo
@@ -1 +1 @@
-bar1245333444442www
\ No newline at end of file
+bar1245333444442ww
\ No newline at end of file

Additional context

You will get similar output for HEAD and HEAD^1:

➜  workspace git:(master) ✗ yarn turbo run build --filter=./packages/\* --filter=\[HEAD\] --dry

Packages in Scope
Name Path 
bar  packages/bar
foo  packages/foo

Global Hash Inputs
  Global Files                          = 0
  External Dependencies Hash            = cab80e071f9bd92a
  Global Cache Key                      = I can’t see ya, but I know you’re here
  Global Env Vars                       = 
  Global Env Vars Values                = 
  Inferred Global Env Vars Values       = 
  Global Passed Through Env Vars        = 
  Global Passed Through Env Vars Values = 
  Engines Values                        = 

Tasks to Run
bar#build
  Task                           = build
  Package                        = bar
  Hash                           = 4196372fa6adbee2
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/bar
  Command                        = echo 'build bar'
  Outputs                        = 
  Log File                       = packages/bar/.turbo/turbo-build.log
  Dependencies                   = 
  Dependents                     = foo#build
  Inputs Files Considered        = 3
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":[],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 
foo#build
  Task                           = build
  Package                        = foo
  Hash                           = e6ca59ab5b70fb8d
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/foo
  Command                        = echo 'build foo'
  Outputs                        = 
  Log File                       = packages/foo/.turbo/turbo-build.log
  Dependencies                   = bar#build
  Dependents                     = 
  Inputs Files Considered        = 3
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":["$TURBO_DEFAULT$","../../.root/*"],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 
➜  workspace git:(master) ✗ yarn turbo run build --filter=./packages/\* --filter=\[HEAD^1\] --dry

Packages in Scope
Name Path 
bar  packages/bar
foo  packages/foo

Global Hash Inputs
  Global Files                          = 0
  External Dependencies Hash            = cab80e071f9bd92a
  Global Cache Key                      = I can’t see ya, but I know you’re here
  Global Env Vars                       = 
  Global Env Vars Values                = 
  Inferred Global Env Vars Values       = 
  Global Passed Through Env Vars        = 
  Global Passed Through Env Vars Values = 
  Engines Values                        = 

Tasks to Run
bar#build
  Task                           = build
  Package                        = bar
  Hash                           = 4196372fa6adbee2
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/bar
  Command                        = echo 'build bar'
  Outputs                        = 
  Log File                       = packages/bar/.turbo/turbo-build.log
  Dependencies                   = 
  Dependents                     = foo#build
  Inputs Files Considered        = 3
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":[],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 
foo#build
  Task                           = build
  Package                        = foo
  Hash                           = e6ca59ab5b70fb8d
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/foo
  Command                        = echo 'build foo'
  Outputs                        = 
  Log File                       = packages/foo/.turbo/turbo-build.log
  Dependencies                   = bar#build
  Dependents                     = 
  Inputs Files Considered        = 3
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":["$TURBO_DEFAULT$","../../.root/*"],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 
@chris-olszewski
Copy link
Member

I realize we have an error in our documentation.

The filter you want is --filter="{./packages/\*}[HEAD^1]" as this will run only tasks in packages that are in the packages/ directory that have changed since HEAD^1.

For all except package negation (--filter=!bar), multiple --filters are treated as a union. So --filter='./packages/* will produces foo and bar and --filter=[HEAD^1] will produce foo since root.txt changed. {foo, bar} U {foo} = {foo, bar}.

@chris-olszewski chris-olszewski added area: docs Improvements or additions to documentation and removed kind: bug Something isn't working needs: triage New issues get this label. Remove it after triage labels Jul 22, 2024
@psychobolt
Copy link
Author

psychobolt commented Jul 23, 2024

While that clarifies the usage, my use case still isn't working even with yarn turbo run build --filter="[HEAD^1]" --dry when a .root file is modified. It will work only if I modify a file from foo or bar packages. Considering two commits HEAD~1: .root/foo and HEAD~2: packages/foo/test . I expect both to return the same result:

➜  workspace git:(master) yarn turbo run build --filter="[HEAD~1]" --dry

Packages in Scope
Name Path 

Global Hash Inputs
  Global Files                          = 0
  External Dependencies Hash            = cab80e071f9bd92a
  Global Cache Key                      = I can’t see ya, but I know you’re here
  Global Env Vars                       = 
  Global Env Vars Values                = 
  Inferred Global Env Vars Values       = 
  Global Passed Through Env Vars        = 
  Global Passed Through Env Vars Values = 
  Engines Values                        = 

Tasks to Run
➜  workspace git:(master) 
➜  workspace git:(master) yarn turbo run build --filter="[HEAD~2]" --dry

Packages in Scope
Name Path 
foo  packages/foo

Global Hash Inputs
  Global Files                          = 0
  External Dependencies Hash            = cab80e071f9bd92a
  Global Cache Key                      = I can’t see ya, but I know you’re here
  Global Env Vars                       = 
  Global Env Vars Values                = 
  Inferred Global Env Vars Values       = 
  Global Passed Through Env Vars        = 
  Global Passed Through Env Vars Values = 
  Engines Values                        = 

Tasks to Run
bar#build
  Task                           = build
  Package                        = bar
  Hash                           = 3565ae58cc7f544e
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/bar
  Command                        = echo 'build bar'
  Outputs                        = 
  Log File                       = packages/bar/.turbo/turbo-build.log
  Dependencies                   = 
  Dependents                     = foo#build
  Inputs Files Considered        = 3
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":[],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 
foo#build
  Task                           = build
  Package                        = foo
  Hash                           = c4f79d52425be1b6
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/foo
  Command                        = echo 'build foo'
  Outputs                        = 
  Log File                       = packages/foo/.turbo/turbo-build.log
  Dependencies                   = bar#build
  Dependents                     = 
  Inputs Files Considered        = 4
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":["$TURBO_DEFAULT$","../../.root/*"],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 

However, they don't show same result. I know .root/* is being accounted from under Inputs Files Considered = 4

Capture

@mehulkar
Copy link
Contributor

Can you try --dry=json and look at the expandedInputs key to see if the root file is there?

chris-olszewski added a commit that referenced this issue Jul 23, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
### Description

Closes #8762 by updating docs to provide the correct filter for the
description.

Multiple filters are combined via union so `--filter=./apps/*
--filter=[HEAD^1]` is really "all packages that are in `apps/` *or* have
changed since `HEAD^1`".

### Testing Instructions

Quick verify that my understanding is correct:
```
# Only web was changed in the last commit
[0 olszewski@chriss-mbp] /tmp/foobar $ turbo build --filter='[HEAD^1]' --dry=json | jq '.tasks | map(.taskId)'
[
  "web#build"
]
# All of these are run if we filter to all packages in apps/
[0 olszewski@chriss-mbp] /tmp/foobar $ turbo build --filter='./apps/*' --dry=json | jq '.tasks | map(.taskId)'
[
  "@repo/eslint-config#build",
  "@repo/typescript-config#build",
  "@repo/ui#build",
  "docs#build",
  "web#build"
]
# Union is taken with multiple filters
[0 olszewski@chriss-mbp] /tmp/foobar $ turbo build --filter='./apps/*' --filter='[HEAD^1]' --dry=json | jq '.tasks | map(.taskId)'
[
  "@repo/eslint-config#build",
  "@repo/typescript-config#build",
  "@repo/ui#build",
  "docs#build",
  "web#build"
]
# Intersection is taken with this syntax
[0 olszewski@chriss-mbp] /tmp/foobar $ turbo build --filter='{./apps/*}[HEAD^1]' --dry=json | jq '.tasks | map(.taskId)'
[
  "web#build"
]
```
@psychobolt
Copy link
Author

psychobolt commented Jul 23, 2024

@mehulkar I ran three runs using --dry=json, I do not see expandedInputs:

➜  workspace git:(master) yarn turbo run build --dry=json --filter="[HEAD]"
{
  "id": "2jeQm18Xk07gBZFkx3eoUVNKTPP",
  "version": "1",
  "turboVersion": "2.0.7-canary.1",
  "monorepo": true,
  "globalCacheInputs": {
    "rootKey": "I can’t see ya, but I know you’re here",
    "files": {},
    "hashOfExternalDependencies": "cab80e071f9bd92a",
    "hashOfInternalDependencies": "",
    "environmentVariables": {
      "specified": {
        "env": [],
        "passThroughEnv": null
      },
      "configured": [],
      "inferred": [],
      "passthrough": null
    },
    "engines": null
  },
  "packages": [],
  "envMode": "strict",
  "frameworkInference": true,
  "tasks": [],
  "user": "",
  "scm": {
    "type": "git",
    "sha": "5c9071ab44e2ec2f1266f81beb80c559dca9fd06",
    "branch": "master"
  }
}

Above output expected, with no changes on HEAD.

➜  workspace git:(master) yarn turbo run build --dry=json --filter="[HEAD~1]"
{
  "id": "2jeQn5fEZ21pJul9ko7X65qUFeE",
  "version": "1",
  "turboVersion": "2.0.7-canary.1",
  "monorepo": true,
  "globalCacheInputs": {
    "rootKey": "I can’t see ya, but I know you’re here",
    "files": {},
    "hashOfExternalDependencies": "cab80e071f9bd92a",
    "hashOfInternalDependencies": "",
    "environmentVariables": {
      "specified": {
        "env": [],
        "passThroughEnv": null
      },
      "configured": [],
      "inferred": [],
      "passthrough": null
    },
    "engines": null
  },
  "packages": [
    "//"
  ],
  "envMode": "strict",
  "frameworkInference": true,
  "tasks": [],
  "user": "",
  "scm": {
    "type": "git",
    "sha": "5c9071ab44e2ec2f1266f81beb80c559dca9fd06",
    "branch": "master"
  }
}

Above output not expected, only outputs root package that has changed. But foo package should detect changes from root.

➜  workspace git:(master) yarn turbo run build --dry=json --filter="[HEAD~2]"
{
  "id": "2jeQndZbeFl8TAxy8tKxkLFLUuO",
  "version": "1",
  "turboVersion": "2.0.7-canary.1",
  "monorepo": true,
  "globalCacheInputs": {
    "rootKey": "I can’t see ya, but I know you’re here",
    "files": {},
    "hashOfExternalDependencies": "cab80e071f9bd92a",
    "hashOfInternalDependencies": "",
    "environmentVariables": {
      "specified": {
        "env": [],
        "passThroughEnv": null
      },
      "configured": [],
      "inferred": [],
      "passthrough": null
    },
    "engines": null
  },
  "packages": [
    "//",
    "foo"
  ],
  "envMode": "strict",
  "frameworkInference": true,
  "tasks": [
    {
      "taskId": "bar#build",
      "task": "build",
      "package": "bar",
      "hash": "3565ae58cc7f544e",
      "inputs": {
        "bar": "6332e25f7949f16f81f6c45edcc296cd78efd9b4",
        "package.json": "cf49a1974a1e6505e4d3edfb74cc0c317386464d",
        "turbo.json": "95960709b10d9e4fa91f251ffafe0819e1a4e681"
      },
      "hashOfExternalDependencies": "7d7cdc66c0570bf8",
      "cache": {
        "local": false,
        "remote": false,
        "status": "MISS",
        "timeSaved": 0
      },
      "command": "echo 'build bar'",
      "cliArguments": [],
      "outputs": null,
      "excludedOutputs": null,
      "logFile": "packages/bar/.turbo/turbo-build.log",
      "directory": "packages/bar",
      "dependencies": [],
      "dependents": [
        "foo#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputLogs": "full",
        "persistent": false,
        "env": [],
        "passThroughEnv": null,
        "interactive": false
      },
      "expandedOutputs": [],
      "framework": "",
      "envMode": "strict",
      "environmentVariables": {
        "specified": {
          "env": [],
          "passThroughEnv": null
        },
        "configured": [],
        "inferred": [],
        "passthrough": null
      }
    },
    {
      "taskId": "foo#build",
      "task": "build",
      "package": "foo",
      "hash": "c4f79d52425be1b6",
      "inputs": {
        "../../.root/foo": "53ed2f08cfcc441eeae915cdebc5225d64c1ad83",
        "package.json": "f9f7da0a9a715a67de735decd64899af44c62f2b",
        "test": "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391",
        "turbo.json": "293cba7f07f40acf360f85dbe4b38c18b5d614d8"
      },
      "hashOfExternalDependencies": "7d7cdc66c0570bf8",
      "cache": {
        "local": false,
        "remote": false,
        "status": "MISS",
        "timeSaved": 0
      },
      "command": "echo 'build foo'",
      "cliArguments": [],
      "outputs": null,
      "excludedOutputs": null,
      "logFile": "packages/foo/.turbo/turbo-build.log",
      "directory": "packages/foo",
      "dependencies": [
        "bar#build"
      ],
      "dependents": [],
      "resolvedTaskDefinition": {
        "outputs": [],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [
          "$TURBO_DEFAULT$",
          "../../.root/*"
        ],
        "outputLogs": "full",
        "persistent": false,
        "env": [],
        "passThroughEnv": null,
        "interactive": false
      },
      "expandedOutputs": [],
      "framework": "",
      "envMode": "strict",
      "environmentVariables": {
        "specified": {
          "env": [],
          "passThroughEnv": null
        },
        "configured": [],
        "inferred": [],
        "passthrough": null
      }
    }
  ],
  "user": "",
  "scm": {
    "type": "git",
    "sha": "5c9071ab44e2ec2f1266f81beb80c559dca9fd06",
    "branch": "master"
  }
}

Above output is expected since I've made a change in foo package's test file. You can verify inputs consist of "../../.root/foo": "53ed2f08cfcc441eeae915cdebc5225d64c1ad83", from root input. I would expect my prior run to have the same output as this, but it's not.

@psychobolt
Copy link
Author

psychobolt commented Jul 23, 2024

Just nothing one work around at the moment is adding .root/foo to global dependencies. However that seems to affect all tasks. The inputs option supposedly is used to affect one task for a particular package, which is my use case. So to circumvent that issue, my CI had a job to copy one or more root files to a package to cache bust. This is particular to setups that rely on root meta files such as .yarn/versions/*.xml to trigger specific CI tests or package releases.

@chris-olszewski
Copy link
Member

@psychobolt Apologies for misunderstanding the issue initially. There are two ways to possibly go about working around this shortcoming:

  • Add .root/foo to global dependencies as you have already done. This will now factor into every task so anytime .root/foo changes, every task will be invalidated.
  • Do not use a source control filter in your setup and instead rely on caching: e.g. --filter='./packages/*'. If .root/foo does not change, then the task that declares on .root/foo will be a cache hit.

The crux of the issue is that our --filter syntax targets packages and not tasks. This leaves us in a weird place when declaring inputs for a task that reside outside of a package: the inputs for foo#build changed, but the contents of foo did not. I realize this isn't a satisfactory answer, but I want to make it clear that offering a proper solution to this will probably require significant changes to either how --filter works or what is considered to be part of a package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: docs Improvements or additions to documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants