Unify build tool integrations with Metals #942

tgodzik · 2019-06-25T17:14:30Z

Add an additional mechanism to save information about the SemanticDB plugin version for Metals. That information is then used to add scala options to compiler instances for the SemanticDB plugin.

Fixes #895

@jvican Is this what you meant? I still need figure out some tests and a way to simulate failing fatal warnings, although the current solution works the same as the one in this PR.

tgodzik · 2019-06-26T15:16:42Z

After talking with @jvican I think what we should do is:

Create a separate module something like integration-shared
Add functionality to fetch SemanticDB plugin via Coursier and save the paths to settings file - based on Scala version and SemanticDB version
Depend on that shared module in all integrations and just invoke for each project's scala version - that will save all needed paths to the settings path.
The mapper inside bloop would read that settings file and use the appropriate SemanticDB plugin path

We would need to add an additional parameter with SemanticDB version to all integrations (we don't have that while running bloopInstall), but that shouldn't be a problem.

Just a question to @jvican : is there a reason that we don't use Scala 2.12 in all of the integrations? At least in Gradle and Maven I don't see any reason to do that. And Metals doesn't work with 2.10 modules :P

jvican · 2019-07-01T07:28:11Z

I've given this a lot more thought and I think this PR is on the right track. Implementing specific semanticdb support in every build as I proposed in our call @tgodzik has the advantage that everything that requires downloading is done by the build tool but two big disadvantages:

By default, we enable semanticdb and we're exporting a build that is different from that in the build tool. That should be fine in theory but it has some compiler performance implications, especially for people using Bloop with IntelliJ.
It requires a lot more work. To be done safely, we need to shade ourselves the coursier dependency we pull in and we need to repeat this across all build tools.

By centralizing the download and setup in the server, we have two key advantages:

Faster to implement and reason about.
Build tools don't need to add support for semanticdb and apply the rest of changes in the build to play nicely with Metals.
By default, if Metals is not used, we don't experience the compiler performance overhead added by Semanticdb (important for CI workflows and IntelliJ users).

Therefore, we only need to tweak how this PR is implemented internally, I'll leave few code comments after this long comment.

jvican

Thanks a lot for working on this @tgodzik, I've left comments with suggestions and improvements, please let me know what you think! 😸

config/src/main/scala/bloop/config/Config.scala

frontend/src/main/scala/bloop/data/SettingsMapper.scala

frontend/src/main/scala/bloop/engine/BuildLoader.scala

tgodzik · 2019-07-03T16:07:36Z

@jvican Thanks for the comments, I just about starting to work on it, since I spent some time on Metal's issues.

frontend/src/main/scala/bloop/engine/SemanticDBCache.scala

mzarnowski

Just some very-minor nitpicks

frontend/src/it/scala/bloop/CommunityBuild.scala

frontend/src/main/scala/bloop/bsp/BloopBspServices.scala

frontend/src/main/scala/bloop/data/BuildSettings.scala

mzarnowski · 2019-07-09T14:50:26Z

frontend/src/main/scala/bloop/data/BuildSettings.scala

+      )
+      val contents = new String(bytes, StandardCharsets.UTF_8)
+      parser.parse(contents) match {
+        case Left(failure) => throw failure


why not None? or even better, chaning the return type to a Try?

…plugin version for Metals. That information is then used to add scala options to compiler instances for the SemanticDB plugin.

jvican

🎉 getting closer to the finish line, I have more questions/suggestions but the PR is looking slick for now 😄 good work!

backend/src/main/scala/bloop/BloopComponentsLock.scala

frontend/src/main/scala/bloop/bsp/BloopBspDefinitions.scala

shared/src/main/scala/bloop/io/AbsolutePath.scala

frontend/src/test/scala/bloop/util/TestUtil.scala

frontend/src/main/scala/bloop/data/WorkspaceSettings.scala

frontend/src/main/scala/bloop/engine/Build.scala

frontend/src/main/scala/bloop/bsp/BloopBspServices.scala

jvican · 2019-07-17T10:06:25Z

frontend/src/main/scala/bloop/data/WorkspaceSettings.scala

+import scala.util.Failure
+import scala.util.Success
+
+case class WorkspaceSettings(semanticDBVersion: String)


I'll add docs here and in several other places when we're doing with the review

frontend/src/main/scala/bloop/engine/Build.scala

tgodzik · 2019-07-22T18:28:45Z

backend/src/main/scala/bloop/DependencyResolution.scala

@@ -62,6 +63,7 @@ object DependencyResolution {
    }
    val fetch = ResolutionProcess.fetch(repositories, Cache.default.fetch)
    val resolution = start.process.run(fetch).unsafeRun()
+    if (shouldReportErrors) reportErrors(resolution, logger)


Tested with Metals and now we will show a warning like:

tgodzik · 2019-07-23T13:53:48Z

frontend/src/main/scala/bloop/engine/BuildLoader.scala

+          logger.info(s"SemanticDB plugin already added: ${containsSemanticDB.get}")
+          optionsSet
+        } else {
+          val workspaceDir = project.origin.path.getParent.getParent


@jvican Not sure if that is the best way of getting the workspace root directory.

The intent of this piece of code was good but after thinking about it and trying it out locally I've realized this is too verbose of a message and this information should be shown in the Metals doctor instead. Some of this code will be added back in a next commit in a different place without the `displayWarningToUser`, which IMO should only be used when the semanticdb plugin could **not** be resolved for a version we know it's compatible for. This is truly an scenario that is unlikely to happen and where good communication with users is key.

- Use new coursier API at the request of Alex, simplifies error handling - Add documentation to `enableMetalsSettings` and `detectWorkspaceDirectory`. - Cache `latest.release` resolution so that we only resolve the plugin once in the whole server session, otherwise there's too much overhead of resolving a `latest.release` artifact per project. - Refine the logic transforming the project so that we unconditionally apply range positions and we only emit warnings to users when a resolution error that we didn't expect happens.

These changes are not really important. They will be refined in an upcoming commit.

And lay the foundation for the big changes coming in the next commit.

This big commit rethinks how build load works to make it incremental. It adds more tests and documentation to the appropiate internal APIs to make this area easier to browse in the future. The previous logic was working perfectly fine but could incur some performance overhead because every time there was a change in the server the build load process would restart and that implies also resolving semanticdb plugins for all scala versions in a project. That is not a big problem because most of the times they are cached but the overhead of that operation is nevertheless too expensive, so the incremental build load process mitigates that.

The semantics to retry adding the semanticdb plugin to a project depend uniquely on whether the project has semanticdb disabled and whether the `newSettings` passed by the client are not empty. Whether there is a change in the settings or not should not affect these semantics.

Because `writeToFile` returned `Either`, there were places in our tests where we were completely swallowing any exception that could be happening when writing the workspace settings file. Now we will throw the exception instead. In the case of the internal build logic, we do swallow any exception thrown by this logic intentionally.

The workspace already implies that it's the root directory and we already use `Dir` as the suffix of many fields in the configuration, so this is a more idiomatic choice.

jvican

@tgodzik I've just pushed several changes that IMO make this PR already ready to be merged, it's your time now to review! I've left a few comments/explanations on the most important parts of the changes and the commit messages are also meant to be descriptive. Please let me know what you think about the changes 😄

jvican · 2019-07-19T19:54:56Z

backend/src/main/scala/sbt/internal/inc/ZincComponentManager.scala

 ) {
-  val log = new FullLogger(log0)


Happy to see this leaving 👍

jvican · 2019-07-30T15:57:09Z

backend/src/main/scala/bloop/DependencyResolution.scala

   */
-  def resolve(
+  def resolveWithErrors(


I updated this logic because Alex recommended me to update to the latest coursier API.

jvican · 2019-07-30T16:10:51Z

frontend/src/main/scala/bloop/data/LoadedProject.scala

@@ -0,0 +1,30 @@
+package bloop.data
+
+sealed trait LoadedProject {


@tgodzik This is a new key abstraction I have added to our build load infrastructure so that we can distinguish internally whether a project has been configured or it hasn't. Even if semanticdb is not supported in a project, we still consider it configured because we always add the range positions, so in checkForChange we detect configured projects that don't have semanticdb enabled and we retry the resolution when newSettings are passed.

jvican · 2019-07-30T16:32:00Z

frontend/src/main/scala/bloop/data/Project.scala

+      workspaceDir: AbsolutePath,
+      logger: Logger
+  ): Project = {
+    def enableSemanticDB(options: List[String], pluginPath: AbsolutePath): List[String] = {


Instead of adding semanticdb and range pos in one pass, we do it in two different methods so that by default we add range pos and only if we have a plugin path we enable the semanticdb.

jvican · 2019-07-30T16:32:30Z

frontend/src/main/scala/bloop/data/WorkspaceSettings.scala

+  final case object SemanticDBVersionChange extends DetectedChange
+
+  /** File name to store Metals specific settings*/
+  private[bloop] val settingsFileName = RelativePath("bloop.settings.json")


Nit: i relative path instead of a string is the most idiomatic way of writing file names 😄

jvican · 2019-07-30T16:40:37Z

frontend/src/main/scala/bloop/engine/caches/SemanticDBCache.scala

+      }
+    }
+
+    if (version == "latest.release") {


Mostly left this as you made it @tgodzik with the exception that if the version is latest.release we only resolve it once per instance of the bloop server.

jvican · 2019-07-30T16:41:40Z

frontend/src/test/scala/bloop/BuildLoaderSpec.scala

+      }
+  }
+
+  testLoad("reload if two file contents changed with same settings", Some(sameSettings)) {


I added a few more tests here.I really appreciate that you were the first one to add tons of tests here, I had almost forgotten about this test suite 😅

jvican · 2019-07-30T16:41:50Z

frontend/src/test/scala/bloop/BuildLoaderSpec.scala

+        case Build.ReturnPreviousState =>
+          sys.error(s"Expected return updated state, got previous state")
+        case action: Build.UpdateState =>
+          assert(action.deleted.size == 0)


I made some tests more specific

jvican · 2019-07-30T16:43:06Z

frontend/src/test/scala/bloop/bsp/BspMetalsClientSpec.scala

+
+      loadBspState(workspace, projects, logger, "Metals", bloopExtraParams = extraParams) { state =>
+        assertNoDiff(logger.warnings.mkString(System.lineSeparator), "")
+        assertNoDiffInSettingsFile(


In general, I really like testing against strings and showing diffs to know what's wrong opposed to doing concrete assertions. This way, it's obvious when reading the tests what the output of workspace settings should be. Just a FYI 😄

jvican · 2019-07-30T16:46:09Z

integrations/gradle-bloop/src/test/scala/bloop/integrations/gradle/ConfigGenerationSuite.scala

@@ -22,11 +22,11 @@ import bloop.engine.BuildLoader

 import scala.collection.JavaConverters._

-class ConfigGenerationSuite481 extends ConfigGenerationSuite{
+class ConfigGenerationSuite481 extends ConfigGenerationSuite {


Looks like these sources were not formatted before and my commit formatted them instead

tgodzik

Couple of early comments, I would want to go through the settings change detection again.

tgodzik · 2019-07-31T08:42:39Z

backend/src/main/scala/bloop/DependencyResolution.scala

-      sys.error(
-        s"Resolution of module $moduleInfo failed with:${System.lineSeparator}${prettyFileErrors}"
-      )
+    try Right(fetch.run().map(f => AbsolutePath(f.toPath)).toArray)


Maybe:

Try(fetch.run().map(f => AbsolutePath(f.toPath)).toArray).toEither

Try catches everything, I now that coursier can only throw CoursierError so the try-catch only catches that

tgodzik · 2019-07-31T09:02:37Z

frontend/src/main/scala/bloop/data/Project.scala

+    semanticDBPlugin match {
+      case None => projectWithRangePositions
+      case Some(pluginPath) =>
+        // Recognize 2.12.8-abdcddd as supported if 2.12.8 exists in supported versions


We don't check it here, is the comment needed?

tgodzik · 2019-07-31T09:18:28Z

frontend/src/main/scala/bloop/engine/Build.scala

+          if currentSettings.semanticDBVersion != newSettings.semanticDBVersion =>
+        Build.ForceReload(newSettings, List(WorkspaceSettings.SemanticDBVersionChange))
+      case (Some(_), Some(newSettings)) => Build.AvoidReload(Some(newSettings))
+      case (None, Some(newSettings)) => Build.AvoidReload(Some(newSettings))


Shouldn't we force it here?

Good catch!

tgodzik · 2019-07-31T09:46:30Z

frontend/src/main/scala/bloop/engine/BuildLoader.scala

+                )
+              }
+
+              val scalaVersion = instance.version


Since all paths end with metalsEnable, can't we put this into a function at least:

def enablePlugin[T](metalsEnable: Option[Path] => T): T

a bit less of a duplciated code.

tgodzik · 2019-07-31T11:59:06Z

LGTM

jvican · 2019-07-31T12:00:08Z

Et voilà

tgodzik requested a review from jvican June 25, 2019 17:22

tgodzik force-pushed the automatically-add-semanticDB-plugin branch from 5528107 to 47d671c Compare June 25, 2019 17:31

jvican changed the title ~~WiP Unify build tool integrations with Metals~~ Unify build tool integrations with Metals Jun 26, 2019

jvican added build server Any issue or pull request that has to do with hot compilers or BSP. enhancement integrations labels Jun 26, 2019

jvican suggested changes Jul 1, 2019

View reviewed changes

tgodzik force-pushed the automatically-add-semanticDB-plugin branch 3 times, most recently from 6a08b48 to fff9e49 Compare July 5, 2019 18:36

tgodzik commented Jul 5, 2019

View reviewed changes

frontend/src/main/scala/bloop/engine/SemanticDBCache.scala Outdated Show resolved Hide resolved

tgodzik force-pushed the automatically-add-semanticDB-plugin branch 5 times, most recently from 976c4ef to 9a30e5c Compare July 8, 2019 15:34

tgodzik requested review from jvican and mzarnowski July 8, 2019 15:35

mzarnowski suggested changes Jul 9, 2019

View reviewed changes

tgodzik force-pushed the automatically-add-semanticDB-plugin branch 2 times, most recently from 54152c1 to 5ec42d9 Compare July 16, 2019 09:52

Add an additional mechanism to save information about the samenticDB …

47134dd

…plugin version for Metals. That information is then used to add scala options to compiler instances for the SemanticDB plugin.

tgodzik force-pushed the automatically-add-semanticDB-plugin branch from 5ec42d9 to 47134dd Compare July 16, 2019 09:58

jvican suggested changes Jul 17, 2019

View reviewed changes

tgodzik force-pushed the automatically-add-semanticDB-plugin branch 2 times, most recently from f4caeb0 to 96f6dc1 Compare July 22, 2019 18:26

tgodzik commented Jul 22, 2019

View reviewed changes

tgodzik commented Jul 23, 2019

View reviewed changes

tgodzik requested a review from jvican July 23, 2019 13:53

jvican and others added 17 commits July 24, 2019 12:25

Remove unnecessary import

f8bcfb2

Make mostly stylistic changes to the bsp metals test

184e306

Make test suite less verbose

380a028

Change read and write of workspace settings

32351b4

These changes are not really important. They will be refined in an upcoming commit.

Add LoadedProject abstraction

5020ef6

And lay the foundation for the big changes coming in the next commit.

Remove loaded build abstraction

7ff9cb3

Add an additional setting for workspace root

e3241d1

Rename workspaceRoot to workspaceDir

194cc16

The workspace already implies that it's the root directory and we already use `Dir` as the suffix of many fields in the configuration, so this is a more idiomatic choice.

Use real workspace directory to enable semanticdb

27ef85a

Remove unnecessary Project.pprint

745b154

Rename misnomer in pickSettingsForReload method

2972653

Use workspace dir from project in tests too

eba47e1

jvican approved these changes Jul 30, 2019

View reviewed changes

Fix error computing reattemptConfiguration

eb93389

tgodzik commented Jul 31, 2019

View reviewed changes

jvican added 3 commits July 31, 2019 12:03

Return ForceReload when new settings are added

30927d7

Use Coeval to abstract over logic applying semanticdb

3f0a295

Merge branch 'master' into HEAD

f45a972

jvican merged commit e307563 into scalacenter:master Jul 31, 2019

tgodzik deleted the automatically-add-semanticDB-plugin branch July 31, 2019 12:46

tindzk mentioned this pull request Aug 6, 2019

Metals support tindzk/seed#15

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify build tool integrations with Metals #942

Unify build tool integrations with Metals #942

tgodzik commented Jun 25, 2019

tgodzik commented Jun 26, 2019

jvican commented Jul 1, 2019

jvican left a comment

tgodzik commented Jul 3, 2019

mzarnowski left a comment

mzarnowski Jul 9, 2019

jvican left a comment

jvican Jul 17, 2019

tgodzik Jul 22, 2019

tgodzik Jul 23, 2019

jvican left a comment

jvican Jul 19, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

jvican Jul 30, 2019

tgodzik left a comment

tgodzik Jul 31, 2019

jvican Jul 31, 2019

tgodzik Jul 31, 2019

tgodzik Jul 31, 2019

jvican Jul 31, 2019

tgodzik Jul 31, 2019 •

edited

Loading

tgodzik commented Jul 31, 2019

jvican commented Jul 31, 2019

		@@ -0,0 +1,30 @@
		package bloop.data

		sealed trait LoadedProject {

Unify build tool integrations with Metals #942

Unify build tool integrations with Metals #942

Conversation

tgodzik commented Jun 25, 2019

tgodzik commented Jun 26, 2019

jvican commented Jul 1, 2019

jvican left a comment

Choose a reason for hiding this comment

tgodzik commented Jul 3, 2019

mzarnowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvican left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvican left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgodzik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgodzik Jul 31, 2019 • edited Loading

Choose a reason for hiding this comment

tgodzik commented Jul 31, 2019

jvican commented Jul 31, 2019

tgodzik Jul 31, 2019 •

edited

Loading