Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactored BSP module and use isolated classloading for implementations #2245

Merged
merged 5 commits into from
Jan 13, 2023

Conversation

lefou
Copy link
Member

@lefou lefou commented Jan 9, 2023

I also excluded bsp4j from transitive Ammonite dependencies, as we don't use Ammonites BSP support and it brought older versions into the tree.

I removed any use of bsp4j API in scalalib, which was only there previously by accident, as I overlooked it's presence through Ammonites transitive dependencies. scalalib now does not depend on bsp4j. I added more case classes (equivalents to the bsp4j classes) if the BSP support required it.

The bsp module is now only a thin module, which provides the entry point to BSP and also manages the classloading of the implementation, which is now contained in bsp.worker module.

I also minimized the use of reflection in MillMain to start the BSP server, which should reduce the potential for errors which can only be catched in (manual) tests.

@lefou lefou changed the title Refactored BSP module and use isolated classloading for implemenations Refactored BSP module and use isolated classloading for implementations Jan 9, 2023
@lefou lefou force-pushed the bsp-isolate branch 2 times, most recently from aee1228 to 350b47e Compare January 10, 2023 09:54
main/src/mill/MillMain.scala Outdated Show resolved Hide resolved
bsp/src/mill/bsp/Constants.scala Show resolved Hide resolved
bsp/worker/src/mill/bsp/worker/BspTestReporter.scala Outdated Show resolved Hide resolved
bsp/worker/src/mill/bsp/worker/BspTestReporter.scala Outdated Show resolved Hide resolved
bsp/worker/src/mill/bsp/worker/BspWorkerImpl.scala Outdated Show resolved Hide resolved
bsp/worker/src/mill/bsp/worker/MillJvmBuildServer.scala Outdated Show resolved Hide resolved
bsp/worker/src/mill/bsp/worker/MillScalaBuildServer.scala Outdated Show resolved Hide resolved
@lefou lefou requested a review from lolgab January 11, 2023 08:58
@lefou lefou added this to the 0.11.0-M2 milestone Jan 11, 2023
Copy link
Contributor

@ckipp01 ckipp01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, from a pure code point of view this looks good! I like the separation it introduces although I'm not super familiar with workers yet.

I'll try to use this locally a bit today to see if I hit on anything from a usability standpoint.

def generatedBuildInfo: T[Seq[PathRef]] = T {
val workerDep = worker.publishSelfDependency()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh interesting, I never noticed this pattern before. I didn't know publishSelfDependency existed.

def generatedBuildInfo: T[Seq[PathRef]] = T {
val workerDep = worker.publishSelfDependency()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh interesting, I never noticed this pattern before. I didn't know publishSelfDependency existed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only done this way because we needed in in our BuildInfo right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. As we want to load (and resolve) it at runtime and don't want to bundle it in the assembly, we need to know the correct dependency coordinates.

ZincWorkerUtil.scalaBinaryVersion(scalaVersion()),
ScalaPlatform.JVM,
scalaCompilerClasspath().map(_.path.toNIO.toUri.toString).iterator.toSeq.asJava
"scala",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"scala",
ScalaBuildTarget.dataKind,

"You need to run `mill mill.bsp.BSP/install` before you can use the BSP server"
)

// TODO: if outdated, we could regenerate the resource file and re-load the worker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be nice, detect the Mill version somehow and warn the user if there is a mismatch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, I see we actually do this down below, so is this a leftover comment?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's still valid. We detect that we are outdated, but at this point we can't do anything about it, except logging or failing. Failing is too rude here, as we would force the user to understand and handle it. Logging is only helpful to understand the issue when we get access to the logs.

At this stage (firing up BSP server without having a completely initialized Evaluator) we can't trigger a re-build of the BSP connection file, as we can't use the evaluator and the task graph.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One way to handle a detected outdated worker could be to:

  • remember that fact in the BSP service
  • Wait for an Evaluator
  • Trigger a rebuild of the connection file
  • Send an event to the client, that we need to re-start or (if possible) are going to restart the server

@lefou
Copy link
Member Author

lefou commented Jan 12, 2023

Nice, from a pure code point of view this looks good! I like the separation it introduces although I'm not super familiar with workers yet.

"Worker" is a somewhat overloaded term. I don't use a T.worker here. I'd like to, but as we start the BSP as early as possible (and therefore can't wait for Mill to resolve and initialize), we need to do it without Mill's task support.

I'll try to use this locally a bit today to see if I hit on anything from a usability standpoint.

That would be nice. We still have no proper automated test setup that can guarantee a workable server. I typically just delete all .bsp and .metals dirs and fire up a vscode to check whether Mill BSP somehow ends up with a reasonable modules list in vscode. As I don't use it daily, if find it hard to spot idicators for a working/non-working integration. It would be nice, if we had some server-independent test suite, which only requires a BSP connection file and allows to assert on the outcome.

@ckipp01
Copy link
Contributor

ckipp01 commented Jan 12, 2023

if we had some server-independent test suite, which only requires a BSP connection file and allows to assert on the outcome.

Sort of an aside, but we talked about this a bit at the Center. It's on our radar and something we'd like to explore and potentially work on to better ensure the BSP ecosystem as a whole is more tested and less error-prone. I'll loop you in when I have more information there.

@ckipp01
Copy link
Contributor

ckipp01 commented Jan 13, 2023

Just a head up, using this locally for a bit, everything seems to work expected when using Metals.

@lefou lefou merged commit 4fb3954 into com-lihaoyi:main Jan 13, 2023
@lefou lefou deleted the bsp-isolate branch January 13, 2023 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants