-
Notifications
You must be signed in to change notification settings - Fork 185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade to scalameta 3.7.0 with default settings. #675
Conversation
This commit is a major milestone in that Scalafix now uses the latest SemanticDB with default settings. Previously, scalafix relied on two features that are now no longer default: * -P:semanticdb:denotations:all, to lookup information about any symbol scalafix relied on semanticdb-scalac to persist full information for all referenced symbols in each compilation unit. Now, scalafix can load information directly from the classpath thanks to metacp and the new --dependency-classpath command line flag. * -P:semanticdb:signatures:old, these were pretty-printed versions of types that have been replaced with the new specification of `Type` in https://github.com/scalameta/scalameta/blob/master/semanticdb/semanticdb3/semanticdb3.md#type There's quite a bit happening in this PR, it's roughly broken down by: * New --dependency-classpath cli flag be processed by metacp. * New `SymbolTable` to lookup information about symbols from the metacp processed classpath. * Refactor ExplicitResultTypes to use s.Type instead of the deprecated s.SymbolInformation.signatur * Remove tests for Symbol.Multi, which are no longer emitted by semanticdb-scalac. These changes required updates to testkit, the cli and sbt-scalafix. Future work: * The tests pass but I noticed a couple errors while running ExplicitResultTypes on Slick. Given that this PR already does quite a lot I prefer to get it merged asap and fix those errors later. * --dependency-classpath may not be necessary. We might be able to away with only --classpath and be smarter about which files to load from the provided files to fix.
lazy val scalafixSbt1 = scalafixSbt( | ||
scala212, | ||
sbt1, | ||
_.dependsOn(testUtils212 % Test) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this formatting change intended?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, for consistency with the changes below
def resolveClasspath: Configured[Classpath] = | ||
classpath match { | ||
def resolveClasspath: Configured[Classpath] = { | ||
classDirectory match { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a blocker but I don't like that this name hints that the classpath is a single directory entry, which is not what I usually see for a classpath.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree this is confusing. Note that the cli still accepts --classpath as an alias for --class-directory. I renamed for consistency with sbt classDirectory
to distinguish it from dependencyClasspath
.
The public facing API is fully compatible with the previous behavior despite this change so we have wiggle room to improve on this in the future.
* @param sclasspath Regular classpath to process. | ||
* @param out The output stream to print out error messages. | ||
*/ | ||
def toMetaClasspath(sclasspath: Classpath, out: PrintStream): Classpath = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fact that these methods take a printstream looks really awkward.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
awkward sure, but what do you propose instead? This is not a public API
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use System.out directly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scalafix runs via nailgun, in IDEs with custom logging, etc,
It's not OK to use System.out
.withScalaLibrarySynthetics(true) | ||
val reporter = metacp.Reporter().withOut(out) | ||
val mclasspath = scala.meta.cli.Metacp.process(settings, reporter).get | ||
mclasspath |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it okay to crash if metacp returns None?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've refactored the code to propagate the error and report a helpful message on failure.
@@ -55,12 +55,20 @@ case class ScalafixOptions( | |||
sourceroot: Option[String] = None, | |||
@HelpMessage( | |||
"java.io.File.pathSeparator separated list of directories or jars containing " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this a list or a single directory?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It remains a list, unchanged from the previous behavior of --classpath. I've renamed it back to --classpath and added more details about --dependency-classpath.
* This utility is necessary because SymbolTable contains only global symbols and | ||
* SemanticdbIndex has local symbols. | ||
*/ | ||
class CombinedSymbolTable(index: SemanticdbIndex, table: SymbolTable) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to pass both index
and table
here, given that EagerInMemorySemanticdbIndex
already takes a symbol table?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, I've merged this into EagerSemanticdbIndex
avoiding the need for CombinedSymbolTable
.
class LazySymbolTable(mclasspath: Classpath) extends SymbolTable { | ||
|
||
// Map from symbols to the paths to their corresponding semanticdb file where they are stored. | ||
private val unloadedSymbols = TrieMap.empty[String, AbsolutePath] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"unloaded" as in "not yet loaded" or as in "previously loaded, but no longer available"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to notYetLoadedSymbols
val fs = newJarFileSystem(path) | ||
// NOTE(olafur): We don't fs.close() because that can affect another place where `FileSystems.getFileSystems` | ||
// was used due to a `FileSystemAlreadyExistsException`. I don't know what the best solution is for reading the | ||
// same zip file from multiple concurrent places. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe this is/was the source of deadlocks?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was my first suspicion as well! I continuously ran the test suite to see problems. This wasn't the cause of the deadlock, I checked the stack traces and they led to metacp.
shortenNames: Boolean, | ||
pos: Position)(implicit index: SemanticdbIndex): Option[(Type, Patch)] = { | ||
try { | ||
val table = index.asInstanceOf[EagerInMemorySemanticdbIndex].table |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Woah, this cast looks scary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seconded, why do we know more than the compiler here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The SemanticdbIndex
trait should only have a single implementation, I've kept the abstract trait to separate the public API from the implementation. I admit this is not an ideal situation but the alternative is to expose implementation details in the public API. We have the same situation in other places, for example for linter suppression. private[scalafix]
does not bite enough IMO compared to forcing a cast.
/* | ||
* Returns a scala.meta.Tree given a scala.meta.Symbol. | ||
* | ||
* NOTE: this method has become an utter mess and is in need of a clean reimplementation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, not gonna review then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation is unchanged from the previous TypeSyntax, the diff is mostly reformatting with minor changes.
result <- TypeSyntax.prettify( | ||
tpe, | ||
ctx, | ||
config.unsafeShortenNames, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is this config passed? https://scalacenter.github.io/scalafix/docs/users/configuration shows how to add rules to the scalafix config, but does not show if it is possible to pass rule specific config.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's configuration for ExplicitResultTypes documented here https://scalacenter.github.io/scalafix/docs/rules/ExplicitResultTypes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn’t able to put that together from the docs for some reason but I get it now, thanks.
* This utility is necessary because SymbolTable contains only global symbols and | ||
* SemanticdbIndex has local symbols. | ||
*/ | ||
class CombinedSymbolTable(index: SemanticdbIndex, table: SymbolTable) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is combined symbol table needed if shortenNames is false?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's needed both when shortenNames is true and false.
private val unloadedSymbols = TrieMap.empty[String, AbsolutePath] | ||
private val loadedSymbols = TrieMap.empty[String, s.SymbolInformation] | ||
private val semanticdb = "META-INF/semanticdb" | ||
private val semanticIdx = "META-INF/semanticdb.semanticidx" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would this be a good place to add
private val manifest = "META-INF/MANIFEST.MF"
in case there is a classpath manifest in a jar?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that should be handled elsewhere, for example scala.meta.Classpath
should be able to handle.
if (root.isDirectory) { | ||
loadIndex(root, Files.newInputStream(root.resolve(semanticIdx).toNIO)) | ||
} else if (PathIO.extension(root.toNIO) == "jar") { | ||
withJarFileSystem(root) { jarRoot => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not familiar enough with the NIO api to know the answer to this, but are you sure that these apis handle symlinks?
import org.langmeta.semanticdb.Symbol | ||
import scala.meta.internal.semanticdb3.SingletonType.{Tag => x} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This saves two characters and makes the code harder to read. Why not just Tag.?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've replaced x
with Tag
but kept the import local, Tag
is used in s.Type.Tag
/s.SingletonType.Tag
and a few others I think.
} | ||
|
||
def isFunctionN(symbol: String): Boolean = { | ||
symbol.startsWith("scala.Function") && |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if there this is a perforamance issue, but from an accuracy issue how about
"scala.Function(\\d)+#".r
to make sure we capture Function1 but not FunctionPure (not that such a thing currently exists).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there are no other scala.Function*
symbols so I thought this would be enough, regexes are such a heavy machine.
} | ||
symbol match { | ||
case Symbol.Global(owner, Signature.Term(_) | Signature.Type(_)) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I read this as "A symbol is stable only if it is A Type or Term owned by Symbol.None or owned by a chain of n Global Terms". Is that correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method suffers from the same problems as symbolToTree, it can be ignored.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the introduction of SymbolInformation one should no longer infer any information from the structure of symbols. Symbols are purely an ID and nothing more.
@@ -4,7 +4,7 @@ object ExplicitResultTypesPathDependent { | |||
class Path { | |||
class B { class C } | |||
implicit val x: Path.this.B = new B | |||
implicit val y: x.C = new x.C | |||
implicit val y: Path.this.x.C = new x.C | |||
def gimme(yy: x.C) = ???; gimme(y) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
??? indeed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a few questions but don't have enough familiarity with the codebase to call any of them a blocker.
* revert --class-directory, stick to --classpath * better error handling for metacp interaction * add --metacp-cache-directory * add --metacp-no-par for pending --par flag in metacp
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the review! I've addressed all comments, but I'm still investigating the CI failures.
lazy val scalafixSbt1 = scalafixSbt( | ||
scala212, | ||
sbt1, | ||
_.dependsOn(testUtils212 % Test) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, for consistency with the changes below
def resolveClasspath: Configured[Classpath] = | ||
classpath match { | ||
def resolveClasspath: Configured[Classpath] = { | ||
classDirectory match { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree this is confusing. Note that the cli still accepts --classpath as an alias for --class-directory. I renamed for consistency with sbt classDirectory
to distinguish it from dependencyClasspath
.
The public facing API is fully compatible with the previous behavior despite this change so we have wiggle room to improve on this in the future.
* @param sclasspath Regular classpath to process. | ||
* @param out The output stream to print out error messages. | ||
*/ | ||
def toMetaClasspath(sclasspath: Classpath, out: PrintStream): Classpath = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
awkward sure, but what do you propose instead? This is not a public API
.withScalaLibrarySynthetics(true) | ||
val reporter = metacp.Reporter().withOut(out) | ||
val mclasspath = scala.meta.cli.Metacp.process(settings, reporter).get | ||
mclasspath |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've refactored the code to propagate the error and report a helpful message on failure.
@@ -55,12 +55,20 @@ case class ScalafixOptions( | |||
sourceroot: Option[String] = None, | |||
@HelpMessage( | |||
"java.io.File.pathSeparator separated list of directories or jars containing " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It remains a list, unchanged from the previous behavior of --classpath. I've renamed it back to --classpath and added more details about --dependency-classpath.
shortenNames: Boolean, | ||
pos: Position)(implicit index: SemanticdbIndex): Option[(Type, Patch)] = { | ||
try { | ||
val table = index.asInstanceOf[EagerInMemorySemanticdbIndex].table |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The SemanticdbIndex
trait should only have a single implementation, I've kept the abstract trait to separate the public API from the implementation. I admit this is not an ideal situation but the alternative is to expose implementation details in the public API. We have the same situation in other places, for example for linter suppression. private[scalafix]
does not bite enough IMO compared to forcing a cast.
} | ||
|
||
def isFunctionN(symbol: String): Boolean = { | ||
symbol.startsWith("scala.Function") && |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there are no other scala.Function*
symbols so I thought this would be enough, regexes are such a heavy machine.
} | ||
symbol match { | ||
case Symbol.Global(owner, Signature.Term(_) | Signature.Type(_)) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method suffers from the same problems as symbolToTree, it can be ignored.
} | ||
symbol match { | ||
case Symbol.Global(owner, Signature.Term(_) | Signature.Type(_)) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the introduction of SymbolInformation one should no longer infer any information from the structure of symbols. Symbols are purely an ID and nothing more.
/* | ||
* Returns a scala.meta.Tree given a scala.meta.Symbol. | ||
* | ||
* NOTE: this method has become an utter mess and is in need of a clean reimplementation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation is unchanged from the previous TypeSyntax, the diff is mostly reformatting with minor changes.
Tests were failing with `java.nio.file.NoSuchFileException: /META-INF/semanticdb.semanticidx` on 2.11, but not 2.12. I was unable to reproduce the problem using `JarFile` from java.io so I'm gonna throw in the towel and use JarFile for reading zip files instead.
The Windows CI was failing due to URI encoding issues: ``` [info] scalafix.tests.rule.SemanticTests *** ABORTED *** (0 milliseconds) [info] java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/projects/scalafix/.cross/unit/target/scala-2.12/test-classes/ ``` My experience is that this is typically caused by stringly typed programming around URIs, so let's hope this commit fixes the issue.
I'm investigating the CI build failures in scalameta/scalameta#1476 I suspect they are caused by even more incorrect usage of nio FileSystems. The Travis 2.11 failures seem to be caused by differences is positions compared to 3.2.0 |
Semanticdb-scala no longer emits accurate dialects so we need to use other tricks to compute the scala version specific directory
.withCacheDir(cacheDirectory.getOrElse(default.cacheDir)) | ||
val reporter = metacp | ||
.Reporter() | ||
.withOut(devNull) // out prints classpath of proccessed classpath, which is not relevant for scalafix. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh, this is unfortunate, given that we're calling process
directly.
"On macOS the default cache directory is ~/Library/Caches/semanticdb. ") | ||
metacpCacheDir: Option[String] = None, | ||
@HelpMessage("Set this flag to disable parallel processing with metacp.") | ||
metacpNoPar: Boolean = false, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that the help message should explain that there is a danger of deadlocks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
"On macOS the default cache directory is ~/Library/Caches/semanticdb. ") | ||
metacpCacheDir: Option[String] = None, | ||
@HelpMessage("Set this flag to disable parallel processing with metacp.") | ||
metacpNoPar: Boolean = false, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ShaneDelmore We'll need to set this flag when we'll be upgrading Scalafix internally.
} | ||
} | ||
|
||
class TypeSyntax private ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you ultimately want parts of this code to end up in Metap? scalameta/scalameta#1479
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to rewrite this file from scratch first before considering that. I am not happy with the code quality here, but I am more eager to upgrade to scalameta v3.7 and this PR is already doing too much.
This should fix the CI error on Windows
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the review! Upgrading to scalameta v3.7.2 fixed the last blocking problem on Windows
"On macOS the default cache directory is ~/Library/Caches/semanticdb. ") | ||
metacpCacheDir: Option[String] = None, | ||
@HelpMessage("Set this flag to disable parallel processing with metacp.") | ||
metacpNoPar: Boolean = false, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
} | ||
} | ||
|
||
class TypeSyntax private ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to rewrite this file from scratch first before considering that. I am not happy with the code quality here, but I am more eager to upgrade to scalameta v3.7 and this PR is already doing too much.
Merging and releasing 0.6.0-M2 Note that ExplicitResultTypes in M2 will be a fairly broken until TypeSyntax gets a rewrite that I'm planning for this week. |
This commit is a major milestone in that Scalafix now uses the latest
SemanticDB with default settings. Previously, scalafix relied on two
features that are now no longer default:
scalafix relied on semanticdb-scalac to persist full information for
all referenced symbols in each compilation unit. Now, scalafix
can load information directly from the classpath thanks to metacp
and the new --dependency-classpath command line flag.
types that have been replaced with the new specification of
Type
inhttps://github.com/scalameta/scalameta/blob/master/semanticdb/semanticdb3/semanticdb3.md#type
There's quite a bit happening in this PR, it's roughly broken down by:
SymbolTable
to lookup information about symbols from the metacpprocessed classpath.
s.SymbolInformation.signatur
semanticdb-scalac.
These changes required updates to testkit, the cli and sbt-scalafix.
Future work:
ExplicitResultTypes on Slick. Given that this PR already does quite a
lot I prefer to get it merged asap and fix those errors later.
with only --classpath and be smarter about which files to load from
the provided files to fix.