-
Notifications
You must be signed in to change notification settings - Fork 278
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Junit & Specs2 support #163
Conversation
Can one of the admins verify this patch? |
run files currently manually built DiscoveredTestSuite correctly parses the zip entries JunitTest removes the @RunWith since it's not mandatory
@johnynek wdyt? |
scala/scala.bzl
Outdated
#TODO consider with Damien/Ulf/Oscar the implications of switching from grep to scala code | ||
#Pro-> logic will be cohesive (currently the scala code assumes stuff from the grep) | ||
#Con-> IIRC Ulf warned me about performance implications of these traversals | ||
command="unzip -l {archive} | grep {combined_patterns} > {out}".format(archive=archive.path, combined_patterns=_prep_grep_pattern_from(patterns), out=discovered_classes.path)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we use jar tf
here and use the bazel_tools jar so we don't depend on unzip?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can, I actually thought about it but decided (for some reason I don't remember) on unzip. I'll change it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved to jar but when I tried to use the one from bazel_tools I got the following error:
____Loading package: test ____Loading package: @local_config_cc// ____Loading package: @local_config_xcode// ____Loading complete. Analyzing... ____Loading package: @io_bazel_rules_scala_org_specs2_specs2_matcher//jar ____Loading package: @io_bazel_rules_scala_org_openjdk_jmh_jmh_generator_asm//jar ____Loading package: @io_bazel_rules_scala_org_openjdk_jmh_jmh_generator_reflection//jar ____Found 89 targets... ____Building... ____[342 / 391] Discovering classes with patterns of ["E2E", "IT"] ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:245:1: error executing shell command: 'external/local_jdk/bin/jar -tf bazel-out/local-fastbuild/bin/test/JunitMultipleSuffixes.jar | grep -e E2E\.class -e IT\.class > bazel-out/local-fastbuild/bin/test/JunitMultipleSuffixes_discovered_...' failed: Process exited with status 1 [sandboxed]. Error: could not find libjava.dylib Error: Could not find Java SE Runtime Environment. Use --strategy=Generating=standalone to disable sandboxing for the failing actions. ____Building complete. ____Elapsed time: 3.011s, Critical Path: 0.66s
My attempt was via adding an internal attribute (like _ijar), add it to the inputs of the action and then use it instead of the vanilla jar command but that doesn't seem like it's enough. It seems more stuff are needed as inputs. Any idea?
@@ -623,6 +689,30 @@ exports_files([ | |||
"lib/scala-xml_2.11-1.0.4.jar", | |||
"lib/scalap-2.11.8.jar", | |||
]) | |||
|
|||
filegroup( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we already have these here:
https://github.com/bazelbuild/rules_scala/blob/master/src/scala/BUILD#L1
Can we consolidate on those targets? Why were these needed?
I'm concerned about supporting 2.12 and 2.11 with all these direct references in different places.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can, they were needed since using "@scala//:lib/scala-reflect.jar" as a label didn't work for me. If you know how I can skip that I'd love to know. I know my PR notes were long but this is one of the things I noted there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:( Out of luck.
____Loading complete. Analyzing... ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: no such target '//src/scala:parser_xml': target 'parser_xml' not declared in package 'src/scala' defined by /Users/ittaiz/workspace/rules_scala/src/scala/BUILD and referenced by '//test:Specs2Tests'. ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: no such target '//src/scala:parser_library': target 'parser_library' not declared in package 'src/scala' defined by /Users/ittaiz/workspace/rules_scala/src/scala/BUILD and referenced by '//test:Specs2Tests'. ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: no such target '//src/scala:parser_reflect': target 'parser_reflect' not declared in package 'src/scala' defined by /Users/ittaiz/workspace/rules_scala/src/scala/BUILD and referenced by '//test:Specs2Tests'. ERROR: Analysis of target '//test:Specs2Tests' failed; build aborted. ____Elapsed time: 0.073s
Any idea why I'm getting this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
isn't the target //src/scala:scala_xml
? It seems you have :parser_xml
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any comment about this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, moved to //src/scala
.
Gave @scala//:lib/scala-library.jar
another try but got the following message so am staying with the filegroup.
ERROR: /Users/ittaiz/workspace/rules_scala/test/BUILD:258:1: in deps attribute of scala_junit_test rule //test:Specs2Tests: file '@scala//:lib/scala-library.jar' is misplaced here (expected no files). Since this rule was created by the macro 'scala_specs2_junit_test', the error might have been caused by the macro implementation in /Users/ittaiz/workspace/rules_scala/scala/scala.bzl:817:11.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@johnynek //src/scala
doesn't work when I try to use it in an external repository.
I think it's because it's a macro.
If I prefix it with the name of the rules_scala repo in my workspace then it works but that doesn't sound like the right solution since people might name it otherwise, right?
Any thoughts?
@@ -0,0 +1,51 @@ | |||
def specs2_version(): | |||
return "3.8.8" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need this special function?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
answered in the other comment about it
specs2/specs2.bzl
Outdated
"//external:io_bazel_rules_scala/dependency/specs2/specs2_matcher", | ||
"//external:io_bazel_rules_scala/dependency/scalaz/scalaz_effect", | ||
"//external:io_bazel_rules_scala/dependency/scalaz/scalaz_core", | ||
"@scala//:scala-xml", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this really need xml and parser combinators? If so, that's a bummer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it does. why is that a bummer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
xml literals are deprecated, and the library is being removed from the standard library, I think.
# Aditional dependencies for specs2 junit runner | ||
native.maven_jar( | ||
name = "io_bazel_rules_scala_org_specs2_specs2_junit_2_11", | ||
artifact = "org.specs2:specs2-junit_2.11:" + specs2_version(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see how that function helps (specs2_version) since only one version will have the right sha1 below, no (unless someone generates a sha1 collision for specs2! ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I mentioned in the PR notes the reason I added this was because I (mistakenly) used one version in the specs2.bzl file and a different one in the specs2_junit.bzl file. Finding that bug took me a few hours since the runtime error messages are really unclear. Once I found it out I realized my problem was code duplication so I solved it with a method.
import org.junit.runners.model.RunnerBuilder | ||
|
||
@RunWith(classOf[TextFileSuite]) | ||
class DiscoveredTestSuite |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add a comment to the approach here? It looks like this is always registered at the test to run, and then by environmental variable all classes are passed in. Is this right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that's right. I'll add a comment though I'd rather try and change the name to make it more explicit. If it was called TextFileSuiteEntryPoint
would that be clearer that this is just a "vessel"? an entry point? and that the interesting bit is in the TextFileSuite
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think documenting the architecture of how you are doing this is a good idea. To be honest, I'm only probably 80% aware how, and since I won't be using this code, in 6 months I'll probably be at 10% and may have to review patches at that time. I'd really appreciate a paragraph here explaining the approach rather than someone having to go read about junit architecture, then put together the approach from integrating the .bzl and scala code.
private def readDiscoveredClasses(classesRegistry: String): Array[Class[_]] = | ||
entries(classesRegistry) | ||
.map(filterWhitespace) | ||
.map(filterMetadata) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
strange spacing here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
true, sublime editing. Will fix.
zipEntry.split("\\s+") | ||
|
||
private def dropFileSuffix(classEntry: String): String = | ||
classEntry.split("\\.").head |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
inconsistent indentation in this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
true, sublime editing. Will fix.
"run smoothly in bazel" >> { | ||
JUnitCompileTimeDep.hello | ||
success | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we add a negative test in the failing tests path? We should show that something that should fail does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought in the beginning I'll need to add such a test but didn't need to since the tests I added pushed very small parts of the code. I'm really confident about the regression here. In any case I'll add one per your request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added
else | ||
return 1 | ||
fi | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this create .xml output? I had to do extra work for scalatest with that. we test it above, note. If this does generate xml output, we should make sure we are testing it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JUnitCore creates an xml output. I actually don't think we should test this since I didn't add any code relating to it. Additionally when we'll (soon) move to java_test and remove the launcher this test will be even more redundant.
WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just mean that bazel expects test xml on a certain path:
https://github.com/bazelbuild/rules_scala/blob/master/scala/support/JUnitXmlReporter.scala#L82
so, I wonder if we can see that we are generating it on the right path.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added
@johnynek thanks for your review! Answered the comments but I'm not sure in the right github way since I started my own review (whatever this means). Hope all is clear. |
@johnynek I've pushed the changes I understand and I was able to do. |
No need to apologize. I should have run the build locally but GH's resolve
conflicts UI was too compelling.
…On Mon, 27 Mar 2017 at 23:25 Stephen Twigg ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In scala/scala.bzl
<#163 (comment)>
:
> + printFlag = "-Dbazel.discover.classes.print.discovered=%s" % ctx.attr.print_discovered_classes)
+
+def _scala_junit_test_impl(ctx):
+ if (not(ctx.attr.prefixes) and not(ctx.attr.suffixes)):
+ fail("Setting at least one of the attributes ('prefixes','suffixes') is required")
+ deps = ctx.attr.deps + [ctx.attr._suite]
+ jars = _collect_jars(deps)
+ (cjars, rjars) = (jars.compiletime, jars.runtime)
+ junit_deps = [ctx.file._junit,ctx.file._hamcrest]
+ cjars += junit_deps
+ rjars += [ctx.outputs.jar,
+ ctx.file._scalalib
+ ] + junit_deps
+ rjars += _collect_jars(ctx.attr.runtime_deps).runtime
+ test_suite = _gen_test_suite_flags_based_on_prefixes_and_suffixes(ctx, ctx.outputs.jar)
+ launcherJvmFlags = ["-ea", test_suit.archiveFlag, test_suite.prefixesFlag, test_suite.suffixesFlag, test_suite.printFlag]
Yeah, sorry about that. My weak defense is that I was in a coffeeshop and
quickly writing it before I had to run off.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#163 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIF279tE222OYrDhA4T7QPUSovguFIks5rqBtTgaJpZM4MX-WK>
.
|
Looks good to me, but the tests are red. As soon as they are green we should merge. |
test this please |
I think it looks good when the CI is green.
On Mon, Mar 27, 2017 at 5:27 PM Ittai Zeidman <notifications@github.com>
wrote:
… Oscar, ping? Anything left here?
On Mon, 27 Mar 2017 at 23:42 Stephen Twigg ***@***.***>
wrote:
> LGTM
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#163 (comment)
>,
> or mute the thread
> <
https://github.com/notifications/unsubscribe-auth/ABUIF235K402Jaj-H1N9Nlud1o1V8yVpks5rqBUTgaJpZM4MX-WK
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#163 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAEJdmaMWIWstFjN6vOJrhYgLc3yD2voks5rqH4WgaJpZM4MX-WK>
.
|
this still fails locally but with a weird error:
|
I wonder if I can trigger as well- |
test this please |
This is weird. I wonder if master is actually broken and somehow we didn't see? But I thought everything was green? |
|
Travis can't use sandboxing, I don't think, so I think we don't see issues there. I would look most closely at the ci.bazel.io. I wonder if we can really remove the dependencies on the jdk as was done. cc @sdtwigg |
@johnynek seems like mac os is broken on CI which makes sense since that's my os. getting kids ready for school. will take a look soon. |
I'm re-reviewing as well.... Is there a way to get the error log from the failing build? |
No, we need to add the dependency on the jdk back. The default bazel jdk/java tooling setup, located at https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD, does not properly associate dependencies for the java and javac targets. 'Amusingly' (and I'll try to figure out some way to avert this for future tests), I HAD run this on a hermetic BUILD system; however, the system jdk/java tooling in that system is slightly different. Notably, it was establishing a dependency for java and javac on the jdk. |
I think the main way to avert this in the future is to mandate an automatic
ci.bazel.io green light and not settle for a Travis one or a manual run.
Should we merge this and then Stephen will fix on top of it? It was green
before merging master in
…On Tue, 28 Mar 2017 at 7:44 Stephen Twigg ***@***.***> wrote:
No, we need to add the dependency on the jdk back. The default bazel
jdk/java tooling setup, located at
https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD,
does not properly associate dependencies for the java and javac targets.
'Amusingly' (and I'll try to figure out some way to avert this for future
tests), I HAD run this on a hermetic BUILD system; however, the system
jdk/java tooling in that system is slightly different. Notably, it was
establishing a dependency for java and javac on the jdk.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#163 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIF6OT58KbVMgCSO23lyf01OTUjtDZks5rqJAWgaJpZM4MX-WK>
.
|
Since you use _scala_binary_common rather than create your own binary, should be fine with merging this before #172. (I want to wait for that to show green in CI so concerned with delaying your PR further.) |
okay @ittaiz we should be good to go if you can merge master one more time. Sorry for the pain and thanks for bearing with it in this contribution. The reason bazel ci is not automatic, I understand, is to limit security risk. A human looks first to make sure it looks sane. |
@johnynek merged. can you please trigger CI? |
test this please |
@johnynek @damienmg
|
A problem on our CI ? /cc @philwo that have been touching our VM today
And maybe that will do it: retest this please
…On Tue, Mar 28, 2017, 11:25 PM Ittai Zeidman ***@***.***> wrote:
@johnynek <https://github.com/johnynek> @damienmg
<https://github.com/damienmg>
Caused by: java.lang.NoClassDefFoundError: Could not initialize class hudson.util.ProcessTree$UnixReflection
:( build failed and I have no idea why this is happening.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#163 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADjHf_JKY3k3NOjA5cX7vvh7hi7zwMjjks5rqXqogaJpZM4MX-WK>
.
|
retest this please |
It looks like the CI is having an issue after all the tests have passed. Since 2/3 of the platforms on bazel ci are green and travis is green, I'm going to merge. Thanks for carrying this one over the finish line @ittaiz |
@johnynek Yes, Bazel CI is currently having trouble, because I replaced the old Ubuntu 15.10 slaves with Ubuntu 16.04. That version is somehow insisting on making a pre-release version of Java 9 the default on the system, which breaks the Jenkins slave service, giving the NoClassDefFoundError exception. I'll fix this today. |
Thanks for the update.
…On Tue, Mar 28, 2017 at 21:15 Philipp Wollermann ***@***.***> wrote:
@johnynek <https://github.com/johnynek> Yes, Bazel CI is currently having
trouble, because I replaced the old Ubuntu 15.10 slaves with Ubuntu 16.04.
That version is somehow insisting on making a pre-release version of Java 9
the default on the system, which breaks the Jenkins slave service, giving
the NoClassDefFoundError exception.
I'll fix this today.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#163 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAEJdgkkhSY6pUopb27x1oQ1jiKWEC7Vks5rqgUMgaJpZM4MX-WK>
.
|
Sure thing, thank you for your review.
Happy to see this baby growing :)
On Wed, 29 Mar 2017 at 10:17 P. Oscar Boykin <notifications@github.com>
wrote:
… Thanks for the update.
On Tue, Mar 28, 2017 at 21:15 Philipp Wollermann ***@***.***
>
wrote:
> @johnynek <https://github.com/johnynek> Yes, Bazel CI is currently
having
> trouble, because I replaced the old Ubuntu 15.10 slaves with Ubuntu
16.04.
> That version is somehow insisting on making a pre-release version of
Java 9
> the default on the system, which breaks the Jenkins slave service, giving
> the NoClassDefFoundError exception.
>
> I'll fix this today.
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <
#163 (comment)
>,
> or mute the thread
> <
https://github.com/notifications/unsubscribe-auth/AAEJdgkkhSY6pUopb27x1oQ1jiKWEC7Vks5rqgUMgaJpZM4MX-WK
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#163 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIF2renabwr_WSrSDuGGHSG82E-acoks5rqgWhgaJpZM4MX-WK>
.
|
Hi,
As discussed long ago ( #72 and bazelbuild/bazel#503) this is junit and specs2 support.
It actually brings with it a powerful test discovery mechanism which imitates the maven behavior. One can define multiple patterns and all classes whose name fits the pattern will be run. In Scala the classes (vs file-name) emphasis is substantial due to many classes in a file and mismatch between file-names and class names.
Features:
JUnit and Specs2 (using specs2 spec) Tests can:
Notes:
Dilemas I'd love feedback on:
Grep usage- I'm currently using grep to filter the relevant classes. I considered the implications of switching from grep to Scala code (which already does some manipulation on the output). The pro of grep is performance implications since AFAIR Ulf warned me about these traversals being expensive on the JVM when we're talking about big targets. The pro of moving to Scala is to have the logic more cohesive (currently the Scala code assumes stuff from the grep like the fact that everything ends with .class) and to have a more expected regex syntax.
Better handle where a target has no test classes- currently if there are no matches grep returns an exit code of 1 and you'll get a weird message
This might actually be another point towards moving this filtering to the Scala code since we can fail with a clearer message. I don't think this is a crucial point btw and we can live with it.
WDYT about adding default macros for unit tests and "bigger" tests where the first has default suffixes of (Test* and Test) and the latter has the default suffixes of (IT, *IT, *E2E)? We will definitely build this internally since that's what we use. I think this has value in the OSS but didn't want to push too much.
WDYT about removing the "suffix" and "multiple suffixes" tests? They helped me evolve the code but I think that the multiple suffixes test covers them.
Can I merge two structs? Currently I do it manually but didn't kno if there was an easier syntax for this.