failed: Exec failed due to IOException: Input is a directory if declare_directory passed to depset of DefaultInfo #16987

dpoluyanov · 2022-12-11T18:47:23Z

Description of the bug:

When directory, declared via output = ctx.actions.declare_directory passed to DefaultInfo(files=depset([output])) and used as input (srcs) of another rule, cc_library in my case, there is IOException: Input is a directory error.
It isn't happened in previous version of bazel (6.0.0-pre.20221012.2) and started only from 6.0.0rc1 upgrade,
all following rc candidates and last_green (0cc796a) also produces such error.
It only arise when --remote_download_minimal is enabled, so I noted it in reproducible sample

ERROR: 
/private/var/tmp/_bazel_dpoluyanov/11d39f42d934765009feaa26dcc6e6cf/external/kotlin_spec_cpp/BUILD.bazel:3:17: Compiling external/kotlin_spec_cpp/kotlin-spec-cpp_grammar/UnicodeClasses.cpp failed: Exec failed due to IOException: Input is a directory: bazel-out/darwin_arm64-fastbuild/bin/external/kotlin_spec_cpp/kotlin-spec-cpp_grammar

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Repository with reproduce: https://github.com/dpoluyanov/bazel-gh-16987

Which operating system are you running Bazel on?

linux, darwin

What is the output of `bazel info release`?

release 6.0.0rc4

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

(internal repository)

Have you found anything relevant by searching the web?

there is no similar issue

Any other information, logs, or outputs that you want to share?

No response

The text was updated successfully, but these errors were encountered:

fmeum · 2022-12-12T07:45:31Z

This repros even with --disk_cache instead of --remote_cache (easier to set up for testing) and a bisect points to 32b0f5a.

@tjgq

fmeum · 2022-12-12T09:18:56Z

@bazel-io flag

tjgq · 2022-12-12T11:24:06Z

I'm looking into it.

meteorcloudy · 2022-12-12T13:41:44Z

@bazel-io fork 6.0.0

Fixes bazelbuild#16987.

tjgq · 2022-12-13T13:55:46Z

Adding some context here for future spelunking.

cc_library(srcs = [tree artifact]) internally uses an action template (see CppCompileActionTemplate) to generate a separate action for each file inside the tree artifact. Each of these actions contains only that particular TreeFileArtifact in its inputs, but due to #16333, the BwoB prefetcher will attempt to fetch the entire tree artifact, which fails since the metadata for it is missing.

The temporary workaround I'm planning to submit is to fall back to including the full tree artifact in the action inputs in this situation. This might slightly degrade action performance (additional files to fetch) but unblocks the 6.0.0 release and buys us some time to fix #16333 properly and remove the workaround.

…partial tree artifacts. The actions generated by SpawnActionTemplate and CppCompileActionTemplate contain individual TreeFileArtifacts, but not the respective TreeArtifact, in their inputs. This causes the input prefetcher to crash when building without the bytes, as it is only able to fetch entire tree artifacts and expects their input metadata to be available. This PR introduces an ActionInputPrefetcher#supportsPartialTreeArtifactInputs method signaling whether prefetching partial tree artifacts is supported; if not, the entire tree artifact is included in the inputs of any such actions. This is a temporary workaround to unblock the 6.0.0 release. The long-term plan is to fix bazelbuild#16333 to enable prefetching partial tree artifacts, and remove this workaround. Fixes bazelbuild#16987.

…partial tree artifacts. The actions generated by SpawnActionTemplate and CppCompileActionTemplate contain individual TreeFileArtifacts, but not the respective TreeArtifact, in their inputs. This causes the input prefetcher to crash when building without the bytes, as it is only able to fetch entire tree artifacts and expects their input metadata to be available. This PR introduces an ActionInputPrefetcher#supportsPartialTreeArtifactInputs method signaling whether prefetching partial tree artifacts is supported; if not, the entire tree artifact is included in the inputs of any such actions. This is a temporary workaround to unblock the 6.0.0 release. The long-term plan is to fix bazelbuild#16333 to enable prefetching partial tree artifacts, and remove this workaround. Fixes bazelbuild#16987. PiperOrigin-RevId: 495050207 Change-Id: I7d29713085d2cf84ce5302394fc18ff2a96ec4be

…support partial tree artifacts. The actions generated by SpawnActionTemplate and CppCompileActionTemplate contain individual TreeFileArtifacts, but not the respective TreeArtifact, in their inputs. This causes the input prefetcher to crash when building without the bytes, as it is only able to fetch entire tree artifacts and expects their input metadata to be available. This PR introduces an ActionInputPrefetcher#supportsPartialTreeArtifactInputs method signaling whether prefetching partial tree artifacts is supported; if not, the entire tree artifact is included in the inputs of any such actions. This is a temporary workaround to unblock the 6.0.0 release. The long-term plan is to fix bazelbuild#16333 to enable prefetching partial tree artifacts, and remove this workaround. Fixes bazelbuild#16987. PiperOrigin-RevId: 495050207 Change-Id: I7d29713085d2cf84ce5302394fc18ff2a96ec4be

dpoluyanov · 2022-12-13T18:32:57Z

Thanks, @tjgq!
I've successfully rebuilt cc libraries with generated code with bazel version from 25ba76c, so for now workaround is working as expected

…support partial tree artifacts. (#17013) The actions generated by SpawnActionTemplate and CppCompileActionTemplate contain individual TreeFileArtifacts, but not the respective TreeArtifact, in their inputs. This causes the input prefetcher to crash when building without the bytes, as it is only able to fetch entire tree artifacts and expects their input metadata to be available. This PR introduces an ActionInputPrefetcher#supportsPartialTreeArtifactInputs method signaling whether prefetching partial tree artifacts is supported; if not, the entire tree artifact is included in the inputs of any such actions. This is a temporary workaround to unblock the 6.0.0 release. The long-term plan is to fix #16333 to enable prefetching partial tree artifacts, and remove this workaround. Fixes #16987. PiperOrigin-RevId: 495050207 Change-Id: I7d29713085d2cf84ce5302394fc18ff2a96ec4be

…partial tree artifacts. The actions generated by SpawnActionTemplate and CppCompileActionTemplate contain individual TreeFileArtifacts, but not the respective TreeArtifact, in their inputs. This causes the input prefetcher to crash when building without the bytes, as it is only able to fetch entire tree artifacts and expects their input metadata to be available. This PR introduces an ActionInputPrefetcher#supportsPartialTreeArtifactInputs method signaling whether prefetching partial tree artifacts is supported; if not, the entire tree artifact is included in the inputs of any such actions. This is a temporary workaround to unblock the 6.0.0 release. The long-term plan is to fix #16333 to enable prefetching partial tree artifacts, and remove this workaround. Fixes #16987. PiperOrigin-RevId: 495050207 Change-Id: I7d29713085d2cf84ce5302394fc18ff2a96ec4be

sgowroji added type: bug untriaged team-Local-Exec Issues and PRs for the Execution (Local) team labels Dec 12, 2022

bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Dec 12, 2022

bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Dec 12, 2022

bazel-io mentioned this issue Dec 12, 2022

[6.0.0] failed: Exec failed due to IOException: Input is a directory if declare_directory passed to depset of DefaultInfo #16991

Closed

tjgq mentioned this issue Dec 12, 2022

TODO: write description #16998

Closed

tjgq added a commit to tjgq/bazel that referenced this issue Dec 12, 2022

TODO: write description

d9b0c3f

Fixes bazelbuild#16987.

larsrc-google assigned tjgq Dec 13, 2022

larsrc-google added P1 I'll work on this now. (Assignee required) release blocker and removed untriaged labels Dec 13, 2022

tjgq added a commit to tjgq/bazel that referenced this issue Dec 13, 2022

TODO: write description

6969acb

Fixes bazelbuild#16987.

tjgq mentioned this issue Dec 13, 2022

Input prefetcher handles partial tree artifacts and nested artifacts incorrectly #16333

Closed

copybara-service bot closed this as completed in 25ba76c Dec 13, 2022

tjgq mentioned this issue Dec 13, 2022

[6.0.0] Include full tree artifact in inputs when prefetcher doesn't support partial tree artifacts. #17013

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed: Exec failed due to IOException: Input is a directory if declare_directory passed to depset of DefaultInfo #16987

failed: Exec failed due to IOException: Input is a directory if declare_directory passed to depset of DefaultInfo #16987

dpoluyanov commented Dec 11, 2022 •

edited

Loading

fmeum commented Dec 12, 2022

fmeum commented Dec 12, 2022

tjgq commented Dec 12, 2022

meteorcloudy commented Dec 12, 2022

tjgq commented Dec 13, 2022 •

edited

Loading

dpoluyanov commented Dec 13, 2022

failed: Exec failed due to IOException: Input is a directory if declare_directory passed to depset of DefaultInfo #16987

failed: Exec failed due to IOException: Input is a directory if declare_directory passed to depset of DefaultInfo #16987

Comments

dpoluyanov commented Dec 11, 2022 • edited Loading

Description of the bug:

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

fmeum commented Dec 12, 2022

fmeum commented Dec 12, 2022

tjgq commented Dec 12, 2022

meteorcloudy commented Dec 12, 2022

tjgq commented Dec 13, 2022 • edited Loading

dpoluyanov commented Dec 13, 2022

dpoluyanov commented Dec 11, 2022 •

edited

Loading

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

tjgq commented Dec 13, 2022 •

edited

Loading