Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalidation issue: VanillaJavaBuilder keeps stale files. #2941

Closed
damienmg opened this issue May 4, 2017 · 35 comments
Closed

Invalidation issue: VanillaJavaBuilder keeps stale files. #2941

damienmg opened this issue May 4, 2017 · 35 comments
Assignees
Labels
breakage P0 This is an emergency and more important than other current work. (Assignee required)

Comments

@damienmg
Copy link
Contributor

damienmg commented May 4, 2017

@damienmg damienmg added breakage P0 This is an emergency and more important than other current work. (Assignee required) labels May 4, 2017
@damienmg damienmg self-assigned this May 4, 2017
@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

The windows failure seems like a permission error, infra flake?

JDK7 failure is relevant to autovalue apparently:

ERROR: /home/ci/workspace/Bazel/JAVA_VERSION/1.7/PLATFORM_NAME/linux-x86_64/src/main/java/com/google/devtools/build/lib/BUILD:543:1: Java compilation in rule '//src/main/java/com/google/devtools/build/lib:build-base' failed: java failed: error executing command external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac7.jar -XX:+TieredCompilation '-XX:TieredStopAtLevel=1' -jar ... (remaining 2 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
src/main/java/com/google/devtools/build/lib/skyframe/ArtifactSkyKey.java:105: error: interface expected here
src/main/java/com/google/devtools/build/lib/analysis/RuleDefinition.java:83: error: package AutoValue_RuleDefinition_Metadata does not exist
src/main/java/com/google/devtools/build/lib/analysis/actions/ExecutionRequirements.java:77: error: cannot find symbol
  symbol:   class AutoValue_ExecutionRequirements_ParseableRequirement
  location: class com.google.devtools.build.lib.analysis.actions.ExecutionRequirements.ParseableRequirement
src/main/java/com/google/devtools/build/lib/analysis/constraints/EnvironmentCollection.java:44: error: cannot find symbol
  symbol:   class AutoValue_EnvironmentCollection_EnvironmentWithGroup
  location: class com.google.devtools.build.lib.analysis.constraints.EnvironmentCollection.EnvironmentWithGroup
src/main/java/com/google/devtools/build/lib/skyframe/ArtifactSkyKey.java:48: error: incompatible types: com.google.devtools.build.lib.skyframe.ArtifactSkyKey.OwnedArtifact cannot be converted to com.google.devtools.build.skyframe.SkyKey
src/main/java/com/google/devtools/build/lib/skyframe/ArtifactSkyKey.java:136: error: method does not override or implement a method from a supertype
src/main/java/com/google/devtools/build/lib/skyframe/ArtifactSkyKey.java:141: error: method does not override or implement a method from a supertype
src/main/java/com/google/devtools/build/lib/skyframe/TargetCompletionValue.java:68: error: cannot find symbol
  symbol:   class AutoValue_TargetCompletionValue_TargetCompletionKey
  location: class com.google.devtools.build.lib.skyframe.TargetCompletionValue.TargetCompletionKey
src/main/java/com/google/devtools/build/lib/skyframe/TestCompletionValue.java:68: error: cannot find symbol
  symbol:   class AutoValue_TestCompletionValue_TestCompletionKey
  location: class com.google.devtools.build.lib.skyframe.TestCompletionValue.TestCompletionKey
src/main/java/com/google/devtools/build/lib/skyframe/AspectCompletionValue.java:58: error: cannot find symbol
  symbol:   class AutoValue_AspectCompletionValue_AspectCompletionKey
  location: class com.google.devtools.build.lib.skyframe.AspectCompletionValue.AspectCompletionKey
src/main/java/com/google/devtools/build/lib/skyframe/CollectTargetsInPackageValue.java:44: error: cannot find symbol
  symbol:   class AutoValue_CollectTargetsInPackageValue_CollectTargetsInPackageKey
  location: class com.google.devtools.build.lib.skyframe.CollectTargetsInPackageValue.CollectTargetsInPackageKey
src/main/java/com/google/devtools/build/lib/skyframe/TransitiveTraversalValue.java:146: error: incompatible types: com.google.devtools.build.lib.cmdline.Label cannot be converted to com.google.devtools.build.skyframe.SkyKey
src/main/java/com/google/devtools/build/lib/analysis/config/BuildConfiguration.java: note: Some input files use or override a deprecated API.
src/main/java/com/google/devtools/build/lib/analysis/config/BuildConfiguration.java: note: Recompile with -Xlint:deprecation for details.
src/main/java/com/google/devtools/build/lib/analysis/config/BuildConfiguration.java: note: Some input files use unchecked or unsafe operations.
src/main/java/com/google/devtools/build/lib/analysis/config/BuildConfiguration.java: note: Recompile with -Xlint:unchecked for details.

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

JDK7 is due to bfdad90

@damienmg damienmg assigned ulfjack and unassigned ulfjack May 4, 2017
@laszlocsomor
Copy link
Contributor

@janakdr is taking a look

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

Still keeping it hot because I haven't got time to look at the windows failures.

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

Looks like this is a spurious breakage. Running "bazel clean" before the build makes it go away. I think we're consuming stale jars.

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

This is a moral dup of #2936

@janakdr janakdr closed this as completed May 4, 2017
@dslomov
Copy link
Contributor

dslomov commented May 4, 2017 via email

@damienmg damienmg reopened this May 4, 2017
@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

Reopening, we should figure out what is wrong both for windows and the correctness issue (figure out if this is an actual dup of #2936 ). I have a consistent repro.

@dslomov
Copy link
Contributor

dslomov commented May 4, 2017 via email

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017 via email

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

Also the linux bug is probably not a dupe of #2936 because it happens only with jdk7 bazel

@dslomov
Copy link
Contributor

dslomov commented May 4, 2017 via email

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

Simplest repro I have so far:

for br in c5672f8e bfdad902c ; do
git checkout $br
bazel clean
bazel build  --define JAVA_VERSION=1.7 //src/main/java/com/google/devtools/build/skyframe:skyframe
echo "For branch $br:"
jar tf bazel-bin/src/main/java/com/google/devtools/build/skyframe/libskyframe.jar | grep SkyKey
done

Output:
For branch c5672f8
com/google/devtools/build/skyframe/SkyKey$1.class
com/google/devtools/build/skyframe/SkyKey.class
For branch bfdad90

However, if you don't clean in between, you get:
For branch c5672f8
com/google/devtools/build/skyframe/SkyKey$1.class
com/google/devtools/build/skyframe/SkyKey.class
For branch bfdad90
com/google/devtools/build/skyframe/SkyKey$1.class
com/google/devtools/build/skyframe/SkyKey.class

The skyframe jar is getting stale files in it.

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017 via email

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

Looks like bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/skyframe/_javac/skyframe/libskyframe_classes has the stale SkyKey.class file.

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

Is this just as silly as, javac generates .class files into the specified --classdir, and then blindly bundles everything in --classdir into a jar? So if there are any extra files in --classdir, it'll read them too?

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

Yup:

echo "hello" > bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/skyframe/_javac/skyframe/libskyframe_classes/com/google/devtools/build/skyframe/JanakImaginaryClass.class

(cd /private/var/tmp/_bazel_janakr/449f66540f7a827a317d06e34f37f945/execroot/bazel &&   exec env -     LC_CTYPE=en_US.UTF-8   external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac7.jar -XX:+TieredCompilation '-XX:TieredStopAtLevel=1' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/skyframe/libskyframe.jar-2.params)

jar tf bazel-bin/src/main/java/com/google/devtools/build/skyframe/libskyframe.jar | grep Janak
com/google/devtools/build/skyframe/JanakImaginaryClass.class

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

My guess is that https://github.com/bazelbuild/bazel/blob/master/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/VanillaJavaBuilder.java#L257 should not just create the directory, it should delete any existing directory.

@cushon
Copy link
Contributor

cushon commented May 4, 2017

Is the action not sandboxed?

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017 via email

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017 via email

@dslomov
Copy link
Contributor

dslomov commented May 4, 2017 via email

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

@damienmg I think the _classes directory is the root of the issue.

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

I don't see any warning about not sandboxing the action, even when I run explicitly with --spawn_strategy=sandboxed. The bug is still there with --spawn_strategy=standalone, of course.

@cushon
Copy link
Contributor

cushon commented May 4, 2017

Got it. I'm on it, the fix is going to look a lot like

// Necessary for local builds in order to discard previous outputs
.

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

@dslomov \o/

@janakdr ok but that won't explain why the ijar hasn't changed.

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017 via email

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017 via email

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017 via email

@janakdr
Copy link
Contributor

janakdr commented May 4, 2017

@dslomov are you sure that this works at HEAD with JDK7? There appears to be a discrepancy between VanillaJavaBuilder and BazelJavaBuilder, and I don't think it's gone at HEAD.

@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017 via email

@damienmg damienmg changed the title Bootstrap broken at HEAD Invalidation issue: VanillaJavaBuilder keeps stale files. May 4, 2017
@damienmg
Copy link
Contributor Author

damienmg commented May 4, 2017

@cushon IIUC you are working on a fix so assigning to you

@cushon
Copy link
Contributor

cushon commented May 4, 2017

Yep, internal review cl/155081626.

hlopko pushed a commit that referenced this issue May 5, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
bazel-io pushed a commit to bazelbuild/continuous-integration that referenced this issue May 5, 2017
This is a temporary workaround to avoid random failures of JDK7 builds
until 0.5.0 is released.

Change-Id: I1323c128978190274f65018f32fd6b9afe8e681f
@janakdr
Copy link
Contributor

janakdr commented May 5, 2017

Should we make an announcement about this? People should know that they have to run clean before any JDK7 build if they want guaranteed correctness, no? In the case of the commit that triggered this issue, I changed a class to an interface, so the bug resulted in a compile-time error, but I could easily see it not being caught in other cases, and leading to incorrect runtime behavior.

hlopko pushed a commit that referenced this issue May 10, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 10, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 10, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 11, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 17, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 17, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 23, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
damienmg pushed a commit that referenced this issue May 26, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
hlopko pushed a commit that referenced this issue May 31, 2017
to ensure outputs from any previous local builds are discarded.

To cherry-pick for #2692.

Fixes #2941

PiperOrigin-RevId: 155089391
joeleba pushed a commit to joeleba/continuous-integration that referenced this issue Jun 17, 2019
This is a temporary workaround to avoid random failures of JDK7 builds
until 0.5.0 is released.

Change-Id: I1323c128978190274f65018f32fd6b9afe8e681f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breakage P0 This is an emergency and more important than other current work. (Assignee required)
Projects
None yet
Development

No branches or pull requests

6 participants