Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Junit & Specs2 support #163

Merged
merged 40 commits into from
Mar 28, 2017
Merged

Junit & Specs2 support #163

merged 40 commits into from
Mar 28, 2017

Conversation

ittaiz
Copy link
Member

@ittaiz ittaiz commented Mar 9, 2017

Hi,
As discussed long ago ( #72 and bazelbuild/bazel#503) this is junit and specs2 support.
It actually brings with it a powerful test discovery mechanism which imitates the maven behavior. One can define multiple patterns and all classes whose name fits the pattern will be run. In Scala the classes (vs file-name) emphasis is substantial due to many classes in a file and mismatch between file-names and class names.

Features:
JUnit and Specs2 (using specs2 spec) Tests can:

  1. Have jvm_flags (we use this internally to increase xmx for some suites)
  2. Run test classes based on multiple patterns (we use IT*,*IT,*E2E for example)
  3. Use supplied deps and runtime_deps

Notes:

  1. I used bind even though it looks like Bazel want to deprecate it since that's what's used here (and Oscar advocates it)
  2. Specs2 version- I extracted it to a method since I spent a few hours debugging a mismatch between the version of specs2 and specs2_junit :(
  3. Scala build file changes- I've added filegroups since I using "@scala//:lib/scala-reflect.jar" as a label didn't work for me. If you know how I can skip that I'd love to know.

Dilemas I'd love feedback on:

  1. Grep usage- I'm currently using grep to filter the relevant classes. I considered the implications of switching from grep to Scala code (which already does some manipulation on the output). The pro of grep is performance implications since AFAIR Ulf warned me about these traversals being expensive on the JVM when we're talking about big targets. The pro of moving to Scala is to have the logic more cohesive (currently the Scala code assumes stuff from the grep like the fact that everything ends with .class) and to have a more expected regex syntax.

  2. Better handle where a target has no test classes- currently if there are no matches grep returns an exit code of 1 and you'll get a weird message

ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:260:1: error executing shell command: 'unzip -l bazel-out/local-fastbuild/bin/test/JunitWithNoTests.jar | grep  -e Test\.class > bazel-out/local-fastbuild/bin/test/JunitWithNoTests_discovered_classes.txt' failed: bash failed: error executing command /bin/bash -c 'unzip -l bazel-out/local-fastbuild/bin/test/JunitWithNoTests.jar | grep  -e Test\.class > bazel-out/local-fastbuild/bin/test/JunitCustomPattern2_discovered_classes.txt': com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //test:JunitWithNoTests failed to build
Use --verbose_failures to see the command lines of failed build steps.

This might actually be another point towards moving this filtering to the Scala code since we can fail with a clearer message. I don't think this is a crucial point btw and we can live with it.

  1. WDYT about adding default macros for unit tests and "bigger" tests where the first has default suffixes of (Test* and Test) and the latter has the default suffixes of (IT, *IT, *E2E)? We will definitely build this internally since that's what we use. I think this has value in the OSS but didn't want to push too much.

  2. WDYT about removing the "suffix" and "multiple suffixes" tests? They helped me evolve the code but I think that the multiple suffixes test covers them.

  3. Can I merge two structs? Currently I do it manually but didn't kno if there was an easier syntax for this.

@bazel-io
Copy link
Member

bazel-io commented Mar 9, 2017

Can one of the admins verify this patch?

@ittaiz
Copy link
Member Author

ittaiz commented Mar 9, 2017

cc @ulfjack and @damienmg for their thoughts about performance implications of using grep vs jvm filtering (more info in the above message)

@ittaiz
Copy link
Member Author

ittaiz commented Mar 13, 2017

@johnynek wdyt?

@ittaiz ittaiz mentioned this pull request Mar 13, 2017
scala/scala.bzl Outdated
#TODO consider with Damien/Ulf/Oscar the implications of switching from grep to scala code
#Pro-> logic will be cohesive (currently the scala code assumes stuff from the grep)
#Con-> IIRC Ulf warned me about performance implications of these traversals
command="unzip -l {archive} | grep {combined_patterns} > {out}".format(archive=archive.path, combined_patterns=_prep_grep_pattern_from(patterns), out=discovered_classes.path))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use jar tf here and use the bazel_tools jar so we don't depend on unzip?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can, I actually thought about it but decided (for some reason I don't remember) on unzip. I'll change it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved to jar but when I tried to use the one from bazel_tools I got the following error:

____Loading package: test ____Loading package: @local_config_cc// ____Loading package: @local_config_xcode// ____Loading complete. Analyzing... ____Loading package: @io_bazel_rules_scala_org_specs2_specs2_matcher//jar ____Loading package: @io_bazel_rules_scala_org_openjdk_jmh_jmh_generator_asm//jar ____Loading package: @io_bazel_rules_scala_org_openjdk_jmh_jmh_generator_reflection//jar ____Found 89 targets... ____Building... ____[342 / 391] Discovering classes with patterns of ["E2E", "IT"] ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:245:1: error executing shell command: 'external/local_jdk/bin/jar -tf bazel-out/local-fastbuild/bin/test/JunitMultipleSuffixes.jar | grep -e E2E\.class -e IT\.class > bazel-out/local-fastbuild/bin/test/JunitMultipleSuffixes_discovered_...' failed: Process exited with status 1 [sandboxed]. Error: could not find libjava.dylib Error: Could not find Java SE Runtime Environment. Use --strategy=Generating=standalone to disable sandboxing for the failing actions. ____Building complete. ____Elapsed time: 3.011s, Critical Path: 0.66s

My attempt was via adding an internal attribute (like _ijar), add it to the inputs of the action and then use it instead of the vanilla jar command but that doesn't seem like it's enough. It seems more stuff are needed as inputs. Any idea?

@@ -623,6 +689,30 @@ exports_files([
"lib/scala-xml_2.11-1.0.4.jar",
"lib/scalap-2.11.8.jar",
])

filegroup(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have these here:

https://github.com/bazelbuild/rules_scala/blob/master/src/scala/BUILD#L1

Can we consolidate on those targets? Why were these needed?

I'm concerned about supporting 2.12 and 2.11 with all these direct references in different places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can, they were needed since using "@scala//:lib/scala-reflect.jar" as a label didn't work for me. If you know how I can skip that I'd love to know. I know my PR notes were long but this is one of the things I noted there.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:( Out of luck.

____Loading complete. Analyzing... ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: no such target '//src/scala:parser_xml': target 'parser_xml' not declared in package 'src/scala' defined by /Users/ittaiz/workspace/rules_scala/src/scala/BUILD and referenced by '//test:Specs2Tests'. ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: no such target '//src/scala:parser_library': target 'parser_library' not declared in package 'src/scala' defined by /Users/ittaiz/workspace/rules_scala/src/scala/BUILD and referenced by '//test:Specs2Tests'. ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: no such target '//src/scala:parser_reflect': target 'parser_reflect' not declared in package 'src/scala' defined by /Users/ittaiz/workspace/rules_scala/src/scala/BUILD and referenced by '//test:Specs2Tests'. ERROR: Analysis of target '//test:Specs2Tests' failed; build aborted. ____Elapsed time: 0.073s

Any idea why I'm getting this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't the target //src/scala:scala_xml? It seems you have :parser_xml.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any comment about this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, moved to //src/scala.
Gave @scala//:lib/scala-library.jar another try but got the following message so am staying with the filegroup.

ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: in deps attribute of scala_junit_test rule //test:Specs2Tests: file '@scala//:lib/scala-library.jar' is misplaced here (expected no files). Since this rule was created by the macro 'scala_specs2_junit_test', the error might have been caused by the macro implementation in /Users/ittaiz/workspace/rules_scala/scala/scala.bzl:817:11.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@johnynek //src/scala doesn't work when I try to use it in an external repository.
I think it's because it's a macro.
If I prefix it with the name of the rules_scala repo in my workspace then it works but that doesn't sound like the right solution since people might name it otherwise, right?
Any thoughts?

@@ -0,0 +1,51 @@
def specs2_version():
return "3.8.8"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this special function?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

answered in the other comment about it

"//external:io_bazel_rules_scala/dependency/specs2/specs2_matcher",
"//external:io_bazel_rules_scala/dependency/scalaz/scalaz_effect",
"//external:io_bazel_rules_scala/dependency/scalaz/scalaz_core",
"@scala//:scala-xml",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this really need xml and parser combinators? If so, that's a bummer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does. why is that a bummer?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xml literals are deprecated, and the library is being removed from the standard library, I think.

# Aditional dependencies for specs2 junit runner
native.maven_jar(
name = "io_bazel_rules_scala_org_specs2_specs2_junit_2_11",
artifact = "org.specs2:specs2-junit_2.11:" + specs2_version(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see how that function helps (specs2_version) since only one version will have the right sha1 below, no (unless someone generates a sha1 collision for specs2! ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I mentioned in the PR notes the reason I added this was because I (mistakenly) used one version in the specs2.bzl file and a different one in the specs2_junit.bzl file. Finding that bug took me a few hours since the runtime error messages are really unclear. Once I found it out I realized my problem was code duplication so I solved it with a method.

import org.junit.runners.model.RunnerBuilder

@RunWith(classOf[TextFileSuite])
class DiscoveredTestSuite
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment to the approach here? It looks like this is always registered at the test to run, and then by environmental variable all classes are passed in. Is this right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's right. I'll add a comment though I'd rather try and change the name to make it more explicit. If it was called TextFileSuiteEntryPoint would that be clearer that this is just a "vessel"? an entry point? and that the interesting bit is in the TextFileSuite?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think documenting the architecture of how you are doing this is a good idea. To be honest, I'm only probably 80% aware how, and since I won't be using this code, in 6 months I'll probably be at 10% and may have to review patches at that time. I'd really appreciate a paragraph here explaining the approach rather than someone having to go read about junit architecture, then put together the approach from integrating the .bzl and scala code.

private def readDiscoveredClasses(classesRegistry: String): Array[Class[_]] =
entries(classesRegistry)
.map(filterWhitespace)
.map(filterMetadata)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strange spacing here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, sublime editing. Will fix.

zipEntry.split("\\s+")

private def dropFileSuffix(classEntry: String): String =
classEntry.split("\\.").head
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inconsistent indentation in this file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, sublime editing. Will fix.

"run smoothly in bazel" >> {
JUnitCompileTimeDep.hello
success
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add a negative test in the failing tests path? We should show that something that should fail does.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought in the beginning I'll need to add such a test but didn't need to since the tests I added pushed very small parts of the code. I'm really confident about the regression here. In any case I'll add one per your request.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

else
return 1
fi
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this create .xml output? I had to do extra work for scalatest with that. we test it above, note. If this does generate xml output, we should make sure we are testing it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JUnitCore creates an xml output. I actually don't think we should test this since I didn't add any code relating to it. Additionally when we'll (soon) move to java_test and remove the launcher this test will be even more redundant.
WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just mean that bazel expects test xml on a certain path:
https://github.com/bazelbuild/rules_scala/blob/master/scala/support/JUnitXmlReporter.scala#L82

so, I wonder if we can see that we are generating it on the right path.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

@ittaiz
Copy link
Member Author

ittaiz commented Mar 14, 2017

@johnynek thanks for your review! Answered the comments but I'm not sure in the right github way since I started my own review (whatever this means). Hope all is clear.

@ittaiz
Copy link
Member Author

ittaiz commented Mar 14, 2017

@johnynek I've pushed the changes I understand and I was able to do.
Would appreciate your feedback on the outstanding things.

@ittaiz
Copy link
Member Author

ittaiz commented Mar 27, 2017 via email

@johnynek
Copy link
Member

Looks good to me, but the tests are red. As soon as they are green we should merge.

@johnynek
Copy link
Member

test this please

@johnynek
Copy link
Member

johnynek commented Mar 28, 2017 via email

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

this still fails locally but with a weird error:
trying to understand how these changes relate to _build_nosrc_jar for twitter_scrooge:allscrooges

____Loading complete.  Analyzing...
____Found 102 targets...
____[1 / 24] scala //test/src/main/scala/scala/test/twitter_scrooge:allscrooges
ERROR: /Users/ittaiz/workspace/rules_scala/src/scala/scripts/BUILD:19:1: error executing shell command: '
  rm -rf bazel-out/host/bin/src/scala/scripts/generator.jar_tmp
  set -e
  mkdir -p bazel-out/host/bin/src/scala/scripts/generator.jar_tmp
  # copy any resources

  external/local_jdk/bin/java -...' failed: Process exited with status 2 [sandboxed].
Error: could not find libjava.dylib
Error: Could not find Java SE Runtime Environment.
Use --strategy=Generating=standalone to disable sandboxing for the failing actions.
____Elapsed time: 0.737s, Critical Path: 0.61s

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

I wonder if I can trigger as well-
test this please

@johnynek
Copy link
Member

test this please

@johnynek
Copy link
Member

This is weird. I wonder if master is actually broken and somehow we didn't see? But I thought everything was green?

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

test bazel build test/... passes on travis and fails on my machine so I'm not really certain what's going on here :(

@johnynek
Copy link
Member

Travis can't use sandboxing, I don't think, so I think we don't see issues there. I would look most closely at the ci.bazel.io.

I wonder if we can really remove the dependencies on the jdk as was done. cc @sdtwigg

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

@johnynek seems like mac os is broken on CI which makes sense since that's my os. getting kids ready for school. will take a look soon.

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

@johnynek @sdtwigg checkout master and indeed it's broken.

@sdtwigg
Copy link
Contributor

sdtwigg commented Mar 28, 2017

I'm re-reviewing as well.... Is there a way to get the error log from the failing build?
I had tried the CL in a local hermetic build and it was fine with the JDK removed but it IS possible something strange happened. :/

@sdtwigg
Copy link
Contributor

sdtwigg commented Mar 28, 2017

No, we need to add the dependency on the jdk back. The default bazel jdk/java tooling setup, located at https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD, does not properly associate dependencies for the java and javac targets.

'Amusingly' (and I'll try to figure out some way to avert this for future tests), I HAD run this on a hermetic BUILD system; however, the system jdk/java tooling in that system is slightly different. Notably, it was establishing a dependency for java and javac on the jdk.

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017 via email

@sdtwigg
Copy link
Contributor

sdtwigg commented Mar 28, 2017

Since you use _scala_binary_common rather than create your own binary, should be fine with merging this before #172. (I want to wait for that to show green in CI so concerned with delaying your PR further.)

@johnynek
Copy link
Member

okay @ittaiz we should be good to go if you can merge master one more time. Sorry for the pain and thanks for bearing with it in this contribution.

The reason bazel ci is not automatic, I understand, is to limit security risk. A human looks first to make sure it looks sane.

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

@johnynek merged. can you please trigger CI?

@damienmg
Copy link
Contributor

test this please

@ittaiz
Copy link
Member Author

ittaiz commented Mar 28, 2017

@johnynek @damienmg
All tests have passed but strange build failure-

FATAL: Remote call on Channel to /192.168.59.18 failed
java.io.IOException: Remote call on Channel to /192.168.59.18 failed
	at hudson.remoting.Channel.call(Channel.java:830)
	at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection
	at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:575)
	at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:596)
	at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:595)
	at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:517)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:804)
	at ......remote call to Channel to /192.168.59.18(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:822)
	... 6 more

@damienmg
Copy link
Contributor

damienmg commented Mar 28, 2017 via email

@johnynek
Copy link
Member

retest this please

@johnynek
Copy link
Member

It looks like the CI is having an issue after all the tests have passed. Since 2/3 of the platforms on bazel ci are green and travis is green, I'm going to merge.

Thanks for carrying this one over the finish line @ittaiz

@johnynek johnynek merged commit b51e54c into bazelbuild:master Mar 28, 2017
@philwo
Copy link
Member

philwo commented Mar 29, 2017

@johnynek Yes, Bazel CI is currently having trouble, because I replaced the old Ubuntu 15.10 slaves with Ubuntu 16.04. That version is somehow insisting on making a pre-release version of Java 9 the default on the system, which breaks the Jenkins slave service, giving the NoClassDefFoundError exception.

I'll fix this today.

@johnynek
Copy link
Member

johnynek commented Mar 29, 2017 via email

@ittaiz
Copy link
Member Author

ittaiz commented Mar 29, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants