Support for selecting scala version (#544)

* WIP compile using worker supplied from provider

* Provider[ScalaWorker] contains (will contain) everything
  needed to run a specific version of the scala compiler
* Manually create ScalaWorker for 2.11/2.12

* Add scalareflect + scalacompiler to [ScalaWorker]

* Run scala_repl using compiler from [ScalaWorker]

* All rules using [ScalaWorker]

* run scala_binary, scala_macro_library,
  scala_test, scala_junit_test using [ScalaWorker]
* Tests now working with 2.12

* Create [ScalaWorker] using new_scala_repository

* Generate build file for scalac_2_12 in repository_rule

* Use filegroup for scalac worker files

* Specify scala version and label in new_scala_repository

* Download scala version specified as parameter
* Set label for ScalaWorker as @{name}//:{name}

* Run formatter

* Run scalatest for both 2.11/2.12

* Add initial [ScalaWorker] for both 2.11/2.12
* import scalatest + scalactic for 2.12
* Add 2.12 versions of scalatest_runner and scalatest_reporter
* Manually specify scalatest_reporter for scala_test rule

* Update scalatest to 3.0.5 + remove defaults from ScalaWorker

- rename scalac -> scalac_2_11

* Symlink scalac_worker sources to repo_rule workspace

* Resolve scalatest_reporter version in rule

- Pass both 2.11/2.12 versions as implicit attrs
- Add "major_version" to [ScalaWorker]

* Use new_scala_repository for default 2.11/2.12

* scala_repositories takes scala versions as parameters

* improve naming

* Use single scala version

* Fix runfiles not being generated

* Support multiple scala versions for tut and scrooge

* Format .bzl and BUILD files

* Tests passing with 2.11

* Make scala_proto work with 2.12

* Support specs2 for both 2.11/2.12

- remove unused attr major_version from ScalacProvider

* Don't pass scalatest dependencies through provider

* Pass _scalac through _implicit_deps

* Remove temp test directory

* Remove debugging println

* Cleanup

* Run formatter

* Use inputs from resolve_command

* Combine scalatest+scalactic to single target

* Add utils

- extract_major_version_underscore()
- default_scala_version()
- run formatter

* Increase scala_mvn_artifact usage, add shas

* Add default_major_scala_version()

* Add scala_jar_shas dict

- Specify only one version of scalatest/scalactic...
  getting the shas from scala_jar_shas

* Fix aspect test

* Use scala_maven_import_external for all imports

specs2, tut, scrooge, protob

* Add util_core to scrooge, fix aspect test

* Supply shas to scala_library,compiler and reflect

- User can add shas for custom scala versions
- run formatter

* namespace scala_default

* Scala default version to 2.11.12

- correct spelling error

* Lookup scala_extra_jar_shas[major_version] once, renaming

* tut server_urls as parameter

* Pass ScalacProvider as paramater

* Replace pass classpaths instead of jars

- Stop using ScalacProvider.scala_library etc.
- Use ScalacProvider.default_compile_classpath etc.

* linting

* Lookup scala_jar_shas[major_version] only once

* Run version specific tests in their own workspace

- Copy version_specific_test_dir to its own folder
- Add WORKSPACE file with scala version to be tested
- Removed tests that don't need to be run for multiple versions

* remove more non version-specific tests

* Remove version from scala labels fix tests

- Add deps required for third_party and tut tests

* Use default_runtime_classpath for macros, fix aspect test

* Run more tests in test_version.sh

* Update naming, default_compile_classpath -> base_compile_classpath

* Use base_classpath instead of base_compile.. + base_runtime...

Using different base classpaths for compile/runtime
requires changes in common.bzl: collect_jars()

Use the same base_classpath for both to limit the scope of
this PR

* Add scala_reflect to default_classpath

* scala_version and shas as on tuple paramter

- Combine two parameters into one tuple to force
  the user the specify shas

* Reduce amount of version specific junit tests

* Run version specific tests

* Run linter

* Remove dynamic creation of scala_default ScalacProvider

- Avoid having to symlink java sources for scalac_worker
- Allows us to only have one scala version at a time

* PR comments, clearer naming

* Use ctx.attr._scala_provider instead of ..attr.scalalib, run linter

* Address comments, clean up WORKSPACE.template

* Remove unused code

* Add section about different scala versions to the readme

README.md

@@ -61,6 +61,25 @@ test --strategy=Scalac=worker
 ```
 to your command line, or to enable by default for building/testing add it to your .bazelrc.
 
+## Selecting Scala version
+
+Rules scala supports all minor versions of Scala 2.11/2.12. By default `Scala 2.11.12` is used and to use another
+version you need to
+specify it when calling `scala_repositories`. `scala_repositories` takes a tuple `(scala_version, scala_version_jar_shas)`
+as a parameter where `scala_version` is the scala version and `scala_version_jar_shas` is a `dict` with
+`sha256` hashes for the maven artifacts `scala_library`, `scala_reflect` and `scala_compiler`:
+```python
+scala_repositories(("2.12.6", {
+ "scala_compiler": "3e892546b72ab547cb77de4d840bcfd05c853e73390fed7370a8f19acb0735a0",
+ "scala_library": "0b3d6fd42958ee98715ba2ec5fe221f4ca1e694d7c981b0ae0cd68e97baf6dce",
+ "scala_reflect": "6ba385b450a6311a15c918cf8688b9af9327c6104f0ecbd35933cfcd3095fe04"
+}))
+```
+If you're using any of the rules `twitter_scrooge`, `tut_repositories`, `scala_proto_repositories`
+or `specs2_junit_repositories` you also need to specify `scala_version` for them. See `./test_version/WORKSPACE.template`
+for an example workspace using another scala version.
+
+
 ## Bazel compatible versions
 
 | bazel | rules_scala gitsha |

WORKSPACE

@@ -24,26 +24,35 @@ load("//specs2:specs2_junit.bzl", "specs2_junit_repositories")
 
 specs2_junit_repositories()
 
-load("//scala:scala_cross_version.bzl", "scala_mvn_artifact")
+load("//scala:scala_cross_version.bzl", "scala_mvn_artifact", "default_scala_major_version")
 
 # test adding a scala jar:
 maven_jar(
  name = "com_twitter__scalding_date",
- artifact = scala_mvn_artifact("com.twitter:scalding-date:0.17.0"),
+ artifact = scala_mvn_artifact(
+ "com.twitter:scalding-date:0.17.0",
+ default_scala_major_version(),
+ ),
  sha1 = "420fb0c4f737a24b851c4316ee0362095710caa5",
 )
 
 # For testing that we don't include sources jars to the classpath
 maven_jar(
  name = "org_typelevel__cats_core",
- artifact = scala_mvn_artifact("org.typelevel:cats-core:0.9.0"),
+ artifact = scala_mvn_artifact(
+ "org.typelevel:cats-core:0.9.0",
+ default_scala_major_version(),
+ ),
  sha1 = "b2f8629c6ec834d8b6321288c9fe77823f1e1314",
 )
 
 # test of a plugin
 maven_jar(
  name = "org_psywerx_hairyfotr__linter",
- artifact = scala_mvn_artifact("org.psywerx.hairyfotr:linter:0.1.13"),
+ artifact = scala_mvn_artifact(
+ "org.psywerx.hairyfotr:linter:0.1.13",
+ default_scala_major_version(),
+ ),
  sha1 = "e5b3e2753d0817b622c32aedcb888bcf39e275b4",
 )
 

scala/BUILD

@@ -1,3 +1,7 @@
+load(
+ "@io_bazel_rules_scala//scala:providers.bzl",
+ _declare_scalac_provider = "declare_scalac_provider",
+)
 load("//scala:scala_toolchain.bzl", "scala_toolchain")
 
 toolchain_type(
@@ -23,3 +27,22 @@ java_import(
  jars = ["@bazel_tools//tools/jdk:TestRunner_deploy.jar"],
  visibility = ["//visibility:public"],
 )
+
+_declare_scalac_provider(
+ name = "scala_default",
+ default_classpath = [
+ "@io_bazel_rules_scala_scala_library",
+ "@io_bazel_rules_scala_scala_reflect",
+ ],
+ default_macro_classpath = [
+ "@io_bazel_rules_scala_scala_library",
+ "@io_bazel_rules_scala_scala_reflect",
+ ],
+ default_repl_classpath = [
+ "@io_bazel_rules_scala_scala_library",
+ "@io_bazel_rules_scala_scala_reflect",
+ "@io_bazel_rules_scala_scala_compiler",
+ ],
+ scalac = "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/scalac",
+ visibility = ["//visibility:public"],
+)

scala/private/rule_impls.bzl

@@ -13,7 +13,10 @@
 # limitations under the License.
 """Rules for supporting the Scala language."""
 load("@io_bazel_rules_scala//scala:scala_toolchain.bzl", "scala_toolchain")
-load("@io_bazel_rules_scala//scala:providers.bzl", "create_scala_provider")
+load(
+ "@io_bazel_rules_scala//scala:providers.bzl",
+ "create_scala_provider",
+ _ScalacProvider = "ScalacProvider")
 load(
  ":common.bzl",
  "add_labels_of_jars_to",
@@ -132,7 +135,7 @@ def compile_scala(ctx, target_label, output, manifest, statsfile, sources,
  cjars, all_srcjars, transitive_compile_jars, plugins,
  resource_strip_prefix, resources, resource_jars, labels,
  in_scalacopts, print_compile_time, expect_java_output,
- scalac_jvm_flags):
+ scalac_jvm_flags, scalac_provider):
  # look for any plugins:
  plugins = _collect_plugin_paths(plugins)
  dependency_analyzer_plugin_jars = []
@@ -223,15 +226,20 @@ StatsfileOutput: {statsfile_output}
  ctx.actions.write(
  output = argfile, content = scalac_args + optional_scalac_args)
 
+ scalac_inputs, _, scalac_input_manifests = ctx.resolve_command(
+ tools = [scalac_provider.scalac])
+
  outs = [output, statsfile]
- ins = (compiler_classpath_jars.to_list() + all_srcjars.to_list() +
- list(sources) + plugins_list + dependency_analyzer_plugin_jars +
- classpath_resources + resources + resource_jars + [manifest, argfile])
+ ins = (
+ compiler_classpath_jars.to_list() + all_srcjars.to_list() + list(sources)
+ + plugins_list + dependency_analyzer_plugin_jars + classpath_resources +
+ resources + resource_jars + [manifest, argfile] + scalac_inputs)
 
  ctx.actions.run(
  inputs = ins,
  outputs = outs,
- executable = ctx.executable._scalac,
+ executable = scalac_provider.scalac.files_to_run.executable,
+ input_manifests = scalac_input_manifests,
  mnemonic = "Scalac",
  progress_message = "scala %s" % target_label,
  execution_requirements = {"supports-workers": "1"},
@@ -263,7 +271,8 @@ def try_to_compile_java_jar(ctx, scala_output, all_srcjars, java_srcs,
  providers_of_dependencies = collect_java_providers_of(ctx.attr.deps)
  providers_of_dependencies += collect_java_providers_of(
  implicit_junit_deps_needed_for_java_compilation)
- providers_of_dependencies += collect_java_providers_of([ctx.attr._scalalib])
+ providers_of_dependencies += collect_java_providers_of(
+ ctx.attr._scala_provider[_ScalacProvider].default_classpath)
  scala_sources_java_provider = _interim_java_provider_for_java_compilation(
  scala_output)
  providers_of_dependencies += [scala_sources_java_provider]
@@ -323,13 +332,13 @@ def _compile_or_empty(ctx, manifest, jars, srcjars, buildijar,
  sources = [
  f for f in ctx.files.srcs if f.basename.endswith(_scala_extension)
  ] + java_srcs
- compile_scala(ctx, ctx.label, ctx.outputs.jar, manifest,
-  ctx.outputs.statsfile, sources, jars, all_srcjars,
-   transitive_compile_jars, ctx.attr.plugins,
-  ctx.attr.resource_strip_prefix, ctx.files.resources,
-  ctx.files.resource_jars, jars2labels, ctx.attr.scalacopts,
-  ctx.attr.print_compile_time, ctx.attr.expect_java_output,
-  ctx.attr.scalac_jvm_flags)
+ compile_scala(
+ ctx, ctx.label, ctx.outputs.jar, manifest, ctx.outputs.statsfile,
+ sources, jars, all_srcjars, transitive_compile_jars, ctx.attr.plugins,
+ ctx.attr.resource_strip_prefix, ctx.files.resources,
+ ctx.files.resource_jars, jars2labels, ctx.attr.scalacopts,
+ ctx.attr.print_compile_time, ctx.attr.expect_java_output,
+ ctx.attr.scalac_jvm_flags, ctx.attr._scala_provider[_ScalacProvider])
 
  # build ijar if needed
  if buildijar:
@@ -474,15 +483,16 @@ def is_dependency_analyzer_off(ctx):
 # Extract very common code out from dependency analysis into single place
 # automatically adds dependency on scala-library and scala-reflect
 # collects jars from deps, runtime jars from runtime_deps, and
-def _collect_jars_from_common_ctx(ctx, extra_deps = [],
+def _collect_jars_from_common_ctx(ctx,
+ base_classpath,
+ extra_deps = [],
  extra_runtime_deps = []):
 
  dependency_analyzer_is_off = is_dependency_analyzer_off(ctx)
 
- # Get jars from deps
- auto_deps = [ctx.attr._scalalib, ctx.attr._scalareflect]
- deps_jars = collect_jars(ctx.attr.deps + auto_deps + extra_deps,
+ deps_jars = collect_jars(ctx.attr.deps + extra_deps + base_classpath,
  dependency_analyzer_is_off)
+
  (cjars, transitive_rjars, jars2labels,
  transitive_compile_jars) = (deps_jars.compile_jars,
  deps_jars.transitive_runtime_jars,
@@ -499,13 +509,14 @@ def _collect_jars_from_common_ctx(ctx, extra_deps = [],
  jars2labels = jars2labels,
  transitive_compile_jars = transitive_compile_jars)
 
-def _lib(ctx, non_macro_lib):
+def _lib(ctx, base_classpath, non_macro_lib):
  # Build up information from dependency-like attributes
 
  # This will be used to pick up srcjars from non-scala library
  # targets (like thrift code generation)
  srcjars = collect_srcjars(ctx.attr.deps)
- jars = _collect_jars_from_common_ctx(ctx)
+
+ jars = _collect_jars_from_common_ctx(ctx, base_classpath)
  (cjars, transitive_rjars) = (jars.compile_jars, jars.transitive_runtime_jars)
 
  write_manifest(ctx)
@@ -551,10 +562,13 @@ def _lib(ctx, non_macro_lib):
  )
 
 def scala_library_impl(ctx):
- return _lib(ctx, True)
+ scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
+ return _lib(ctx, scalac_provider.default_classpath, True)
 
 def scala_macro_library_impl(ctx):
- return _lib(ctx, False) # don't build the ijar for macros
+ scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
+ return _lib(ctx, scalac_provider.default_macro_classpath,
+ False) # don't build the ijar for macros
 
 # Common code shared by all scala binary implementations.
 def _scala_binary_common(ctx,
@@ -600,7 +614,8 @@ def _scala_binary_common(ctx,
  runfiles = runfiles)
 
 def scala_binary_impl(ctx):
- jars = _collect_jars_from_common_ctx(ctx)
+ scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
+ jars = _collect_jars_from_common_ctx(ctx, scalac_provider.default_classpath)
  (cjars, transitive_rjars) = (jars.compile_jars, jars.transitive_runtime_jars)
 
  wrapper = _write_java_wrapper(ctx, "", "")
@@ -616,9 +631,12 @@ def scala_binary_impl(ctx):
  return out
 
 def scala_repl_impl(ctx):
+ scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
  # need scala-compiler for MainGenericRunner below
  jars = _collect_jars_from_common_ctx(
- ctx, extra_runtime_deps = [ctx.attr._scalacompiler])
+ ctx,
+ scalac_provider.default_repl_classpath,
+ )
  (cjars, transitive_rjars) = (jars.compile_jars, jars.transitive_runtime_jars)
 
  args = " ".join(ctx.attr.scalacopts)
@@ -666,8 +684,11 @@ def _scala_test_flags(ctx):
 def scala_test_impl(ctx):
  if len(ctx.attr.suites) != 0:
  print("suites attribute is deprecated. All scalatest test suites are run")
+
+ scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
  jars = _collect_jars_from_common_ctx(
  ctx,
+ scalac_provider.default_classpath,
  extra_runtime_deps = [
  ctx.attr._scalatest_reporter, ctx.attr._scalatest_runner
  ],
@@ -731,8 +752,10 @@ def scala_junit_test_impl(ctx):
  fail(
  "Setting at least one of the attributes ('prefixes','suffixes') is required"
  )
+ scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
  jars = _collect_jars_from_common_ctx(
  ctx,
+ scalac_provider.default_classpath,
  extra_deps = [
  ctx.attr._junit, ctx.attr._hamcrest, ctx.attr.suite_label,
  ctx.attr._bazel_test_runner

scala/providers.bzl

@@ -24,3 +24,35 @@ def create_scala_provider(ijar, class_jar, compile_jars,
  transitive_runtime_jars = transitive_runtime_jars,
  transitive_exports = [] #needed by intellij plugin
  )
+
+ScalacProvider = provider(
+ doc = "ScalaProvider",
+ fields = [
+ "scalac",
+ "default_classpath",
+ "default_macro_classpath",
+ "default_repl_classpath",
+ ])
+
+def _declare_scalac_provider(ctx):
+ return [
+ ScalacProvider(
+ scalac = ctx.attr.scalac,
+ default_classpath = ctx.attr.default_classpath,
+ default_repl_classpath = ctx.attr.default_repl_classpath,
+ default_macro_classpath = ctx.attr.default_macro_classpath,
+ )
+ ]
+
+declare_scalac_provider = rule(
+ implementation = _declare_scalac_provider,
+ attrs = {
+ "scalac": attr.label(
+ executable = True,
+ cfg = "host",
+ allow_files = True,
+ mandatory = True),
+ "default_classpath": attr.label_list(allow_files = True),
+ "default_repl_classpath": attr.label_list(allow_files = True),
+ "default_macro_classpath": attr.label_list(allow_files = True),
+ })