[BUG] Compiling CLI to exe on Windows

[roman-petrov]

Today we discussed CLI analysis performance with @dkrutskikh and decided to try to compile CLI tool to exe on Windows (using dart compile exe) as a possible workaround.

Compilation works fine, but when I try to launch analysis with created executable like metrics.exe C:\Projects\dart-code-metrics i get runtime exceptions, here is the output:

[exception: FileSystemException(path=C:\Projects\lib\_internal\sdk_library_metadata\lib\libraries.dart; message=Cannot open file)][stackTrace: #0      _PhysicalFile.readAsStringSync (package:analyzer/file_system/physical_file_system.dart:184)
#1      FolderBasedDartSdk.initialLibraryMap (package:analyzer/src/dart/sdk/sdk.dart:473)
#2      new FolderBasedDartSdk (package:analyzer/src/dart/sdk/sdk.dart:371)
#3      ContextBuilder.findSdk.<anonymous closure> (package:analyzer/src/context/builder.dart:198)
#4      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:140)
#5      DartSdkManager.getSdk (package:analyzer/src/generated/sdk.dart:95)
#6      ContextBuilder.findSdk (package:analyzer/src/context/builder.dart:197)
#7      ContextBuilder.createSourceFactoryFromWorkspace (package:analyzer/src/context/builder.dart:167)
#8      ContextBuilder.buildDriver (package:analyzer/src/context/builder.dart:110)
#9      ContextBuilderImpl.createContext (package:analyzer/src/dart/analysis/context_builder.dart:91)
#10     new AnalysisContextCollectionImpl (package:analyzer/src/dart/analysis/analysis_context_collection.dart:63)
#11     LintAnalyzer.runCliAnalysis (package:dart_code_metrics/src/analyzers/lint_analyzer/lint_analyzer.dart:75)
#12     CliRunner.runAnalysis (package:dart_code_metrics/src/cli/cli_runner.dart:21)
<asynchronous suspension>
#13     main (file:///c:/projects/dart-code-metrics/bin/metrics.dart:4)
<asynchronous suspension>
]
[exception: FileSystemException(path=C:\Projects\lib\_internal\libraries.dart; message=Cannot open file)][stackTrace: #0      _PhysicalFile.readAsStringSync (package:analyzer/file_system/physical_file_system.dart:184)
#1      FolderBasedDartSdk.initialLibraryMap (package:analyzer/src/dart/sdk/sdk.dart:473)
#2      new FolderBasedDartSdk (package:analyzer/src/dart/sdk/sdk.dart:371)
#3      ContextBuilder.findSdk.<anonymous closure> (package:analyzer/src/context/builder.dart:198)
#4      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:140)
#5      DartSdkManager.getSdk (package:analyzer/src/generated/sdk.dart:95)
#6      ContextBuilder.findSdk (package:analyzer/src/context/builder.dart:197)
#7      ContextBuilder.createSourceFactoryFromWorkspace (package:analyzer/src/context/builder.dart:167)
#8      ContextBuilder.buildDriver (package:analyzer/src/context/builder.dart:110)
#9      ContextBuilderImpl.createContext (package:analyzer/src/dart/analysis/context_builder.dart:91)
#10     new AnalysisContextCollectionImpl (package:analyzer/src/dart/analysis/analysis_context_collection.dart:63)
#11     LintAnalyzer.runCliAnalysis (package:dart_code_metrics/src/analyzers/lint_analyzer/lint_analyzer.dart:75)
#12     CliRunner.runAnalysis (package:dart_code_metrics/src/cli/cli_runner.dart:21)
<asynchronous suspension>
#13     main (file:///c:/projects/dart-code-metrics/bin/metrics.dart:4)
<asynchronous suspension>
]

As you can see, the source of these exceptions is not dart_code_metrics itself, they are coming from analyzer package.

Not sure if anything can be done here from our side, it seems we will need changes from Dart SDK team.

I created this issue because @dkrutskikh suggested me to do so, please feel free to just close it if you like :)

[incendial]

@roman-petrov could you please recheck if the problem still here? (sorry for taking your time, but I don't have Windows 😔)

[incendial]

Thanks to Konstantin for pointing out that sdk path maybe wrong dart-lang/sdk#46914 (comment). Could you also check if the path is correct?

[roman-petrov]

@incendial I checked again if the issue is still there.

I see exactly the same result as before. I have two versions of Dart SDK installed on my machine. To be more precise, I have:

• Dart SDK installed at D:\SDK\dart-sdk: this is just unzipped archive from official Dart SDK downloads.
• Dart SDK from Flutter SDK: I use fvm for Flutter version management

So I tried to compile using both Dart SDK I have:

• Using official stable 2.13.0: dart compile exe dart-code-metrics/bin/metrics.dart
• Using Dart from Flutter SDK through fvm (Flutter almost equal to master channel): fvm dart compile exe dart-code-metrics/bin/metrics.dart

To run dart code metrics after compilation I use metrics.exe analyze dart-code-metrics. I got the same results for both cases, call stack for both cases is not the same, but quite similar:

Compiled with Dart 2.13.0

#1      FolderBasedDartSdk.initialLibraryMap (package:analyzer/src/dart/sdk/sdk.dart:473)
#2      new FolderBasedDartSdk (package:analyzer/src/dart/sdk/sdk.dart:371)
#3      ContextBuilder.findSdk.<anonymous closure> (package:analyzer/src/context/builder.dart:204)
#4      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:140)
#5      DartSdkManager.getSdk (package:analyzer/src/generated/sdk.dart:95)
#6      ContextBuilder.findSdk (package:analyzer/src/context/builder.dart:203)
#7      ContextBuilder.createSourceFactoryFromWorkspace (package:analyzer/src/context/builder.dart:173)
#8      ContextBuilder.buildDriver (package:analyzer/src/context/builder.dart:115)
#9      ContextBuilderImpl.createContext (package:analyzer/src/dart/analysis/context_builder.dart:93)
#10     new AnalysisContextCollectionImpl (package:analyzer/src/dart/analysis/analysis_context_collection.dart:65)
#11     LintAnalyzer.runCliAnalysis (package:dart_code_metrics/src/analyzers/lint_analyzer/lint_analyzer.dart:76)
#12     AnalyzeCommand.runCommand (package:dart_code_metrics/src/cli/commands/analyze.dart:59)
#13     BaseCommand._verifyThenRunCommand (package:dart_code_metrics/src/cli/commands/base_command.dart:93)
#14     BaseCommand.run (package:dart_code_metrics/src/cli/commands/base_command.dart:25)
#15     CommandRunner.runCommand (package:args/command_runner.dart:209)
#16     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:119)
#17     new Future.sync (dart:async/future.dart:218)
#18     CommandRunner.run (package:args/command_runner.dart:119)
#19     CliRunner.run (package:dart_code_metrics/src/cli/cli_runner.dart:28)
#20     main (file:///d:/projects/github/dart-code-metrics/bin/metrics.dart:4)
#21     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:281)
#22     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:184)
]
[exception: FileSystemException(path=D:\Projects\lib\_internal\libraries.dart; message=Cannot open file)][stackTrace: #0      _PhysicalFile.readAsStringSync (package:analyzer/file_system/physical_file_system.dart:184)
#1      FolderBasedDartSdk.initialLibraryMap (package:analyzer/src/dart/sdk/sdk.dart:473)
#2      new FolderBasedDartSdk (package:analyzer/src/dart/sdk/sdk.dart:371)
#3      ContextBuilder.findSdk.<anonymous closure> (package:analyzer/src/context/builder.dart:204)
#4      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:140)
#5      DartSdkManager.getSdk (package:analyzer/src/generated/sdk.dart:95)
#6      ContextBuilder.findSdk (package:analyzer/src/context/builder.dart:203)
#7      ContextBuilder.createSourceFactoryFromWorkspace (package:analyzer/src/context/builder.dart:173)
#8      ContextBuilder.buildDriver (package:analyzer/src/context/builder.dart:115)
#9      ContextBuilderImpl.createContext (package:analyzer/src/dart/analysis/context_builder.dart:93)
#10     new AnalysisContextCollectionImpl (package:analyzer/src/dart/analysis/analysis_context_collection.dart:65)
#11     LintAnalyzer.runCliAnalysis (package:dart_code_metrics/src/analyzers/lint_analyzer/lint_analyzer.dart:76)
#12     AnalyzeCommand.runCommand (package:dart_code_metrics/src/cli/commands/analyze.dart:59)
#13     BaseCommand._verifyThenRunCommand (package:dart_code_metrics/src/cli/commands/base_command.dart:93)
#14     BaseCommand.run (package:dart_code_metrics/src/cli/commands/base_command.dart:25)
#15     CommandRunner.runCommand (package:args/command_runner.dart:209)
#16     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:119)
#17     new Future.sync (dart:async/future.dart:218)
#18     CommandRunner.run (package:args/command_runner.dart:119)
#19     CliRunner.run (package:dart_code_metrics/src/cli/cli_runner.dart:28)
#20     main (file:///d:/projects/github/dart-code-metrics/bin/metrics.dart:4)
#21     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:281)
#22     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:184)
]
Oops; metrics has exited unexpectedly: "FileSystemException(path=D:\Projects\lib\_internal\allowed_experiments.json; message=Cannot open file)"

Compiled with Dart from latest Flutter

#1      FolderBasedDartSdk.initialLibraryMap (package:analyzer/src/dart/sdk/sdk.dart:473)
#2      new FolderBasedDartSdk (package:analyzer/src/dart/sdk/sdk.dart:371)
#3      ContextBuilder.findSdk.<anonymous closure> (package:analyzer/src/context/builder.dart:204)
#4      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:140)
#5      DartSdkManager.getSdk (package:analyzer/src/generated/sdk.dart:95)
#6      ContextBuilder.findSdk (package:analyzer/src/context/builder.dart:203)
#7      ContextBuilder.createSourceFactoryFromWorkspace (package:analyzer/src/context/builder.dart:173)
#8      ContextBuilder.buildDriver (package:analyzer/src/context/builder.dart:115)
#9      ContextBuilderImpl.createContext (package:analyzer/src/dart/analysis/context_builder.dart:93)
#10     new AnalysisContextCollectionImpl (package:analyzer/src/dart/analysis/analysis_context_collection.dart:65)
#11     LintAnalyzer.runCliAnalysis (package:dart_code_metrics/src/analyzers/lint_analyzer/lint_analyzer.dart:76)
#12     AnalyzeCommand.runCommand (package:dart_code_metrics/src/cli/commands/analyze.dart:59)
#13     BaseCommand._verifyThenRunCommand (package:dart_code_metrics/src/cli/commands/base_command.dart:93)
#14     BaseCommand.run (package:dart_code_metrics/src/cli/commands/base_command.dart:25)
#15     CommandRunner.runCommand (package:args/command_runner.dart:209)
#16     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:119)
#17     new Future.sync (dart:async/future.dart:218)
#18     CommandRunner.run (package:args/command_runner.dart:119)
#19     CliRunner.run (package:dart_code_metrics/src/cli/cli_runner.dart:28)
#20     main (file:///d:/projects/github/dart-code-metrics/bin/metrics.dart:4)
#21     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:281)
#22     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:184)
]
[exception: FileSystemException(path=D:\Projects\lib\_internal\libraries.dart; message=Cannot open file)][stackTrace: #0      _PhysicalFile.readAsStringSync (package:analyzer/file_system/physical_file_system.dart:184)
#1      FolderBasedDartSdk.initialLibraryMap (package:analyzer/src/dart/sdk/sdk.dart:473)
#2      new FolderBasedDartSdk (package:analyzer/src/dart/sdk/sdk.dart:371)
#3      ContextBuilder.findSdk.<anonymous closure> (package:analyzer/src/context/builder.dart:204)
#4      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:140)
#5      DartSdkManager.getSdk (package:analyzer/src/generated/sdk.dart:95)
#6      ContextBuilder.findSdk (package:analyzer/src/context/builder.dart:203)
#7      ContextBuilder.createSourceFactoryFromWorkspace (package:analyzer/src/context/builder.dart:173)
#8      ContextBuilder.buildDriver (package:analyzer/src/context/builder.dart:115)
#9      ContextBuilderImpl.createContext (package:analyzer/src/dart/analysis/context_builder.dart:93)
#10     new AnalysisContextCollectionImpl (package:analyzer/src/dart/analysis/analysis_context_collection.dart:65)
#11     LintAnalyzer.runCliAnalysis (package:dart_code_metrics/src/analyzers/lint_analyzer/lint_analyzer.dart:76)
#12     AnalyzeCommand.runCommand (package:dart_code_metrics/src/cli/commands/analyze.dart:59)
#13     BaseCommand._verifyThenRunCommand (package:dart_code_metrics/src/cli/commands/base_command.dart:93)
#14     BaseCommand.run (package:dart_code_metrics/src/cli/commands/base_command.dart:25)
#15     CommandRunner.runCommand (package:args/command_runner.dart:209)
#16     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:119)
#17     new Future.sync (dart:async/future.dart:218)
#18     CommandRunner.run (package:args/command_runner.dart:119)
#19     CliRunner.run (package:dart_code_metrics/src/cli/cli_runner.dart:28)
#20     main (file:///d:/projects/github/dart-code-metrics/bin/metrics.dart:4)
#21     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:281)
#22     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:184)
]
Oops; metrics has exited unexpectedly: "FileSystemException(path=D:\Projects\lib\_internal\allowed_experiments.json; message=Cannot open file)"

Thanks to Konstantin for pointing out that sdk path maybe wrong dart-lang/sdk#46914 (comment). Could you also check if the path is correct?

Not sure what should be done here, where and how to check sdk path... If you know what should be done here, could you please provide some step by step instructions?

[roman-petrov]

Just a bit more information, maybe I am repeating myself, but this is important: compiling dart code metrics to exe works fine, metrics.exe also seems to work fine (displays help, maybe some other functionality might work).

The crash happens only when I run the analysis as metrics.exe analyze <path> command.

[scheglov]

Ah, I think I know why this happens.

import 'dart:io';

main() {
  print(Platform.resolvedExecutable);
}

And run it:

scheglov@scheglov-macbookpro2:~/Source/Dart/sdk.git/sdk (master)$ dart compile exe /Users/scheglov/dart/test/bin/test3.dart
Info: Compiling with sound null safety
Generated: /Users/scheglov/dart/test/bin/test3.exe
scheglov@scheglov-macbookpro2:~/Source/Dart/sdk.git/sdk (master)$ /Users/scheglov/dart/test/bin/test3.exe
/Users/scheglov/dart/test/bin/test3.exe
scheglov@scheglov-macbookpro2:~/Source/Dart/sdk.git/sdk (master)$ dart /Users/scheglov/dart/test/bin/test3.dart
/Users/scheglov/Applications/dart-sdk/bin/dart

Well, yes, the executable is not bin/dart anymore, it is this compiled executable file. But we (the analyzer) still expect that it would be a binary in dart-sdk/bin for purposes of detecting the Dart SDK. Apparently we cannot do this anymore. And actually there is no way to detect the location of the Dart SDK now - if you don't need one to run this script, you might not have one at all. So, the client that creates AnalysisContextCollection must provide sdkPath, at least to be compatible with compilation.

If sdkPath is not provided, we try to infer it using:

/// Return the path to the current Dart SDK.
String getSdkPath() => path.dirname(path.dirname(Platform.resolvedExecutable));

[incendial]

@roman-petrov thank you for the detailed comment! By checking the path I meant checking that path=D:\Projects\lib\_internal\libraries.dart is a valid path to the Dart SDK, but I see that it's not needed anymore.

So, the client that creates AnalysisContextCollection must provide sdkPath, at least to be compatible with compilation.

@scheglov that's exactly what I though when you mentioned that the path in the exception didn't look like a valid path to the Dart SDK, because we don't pass it and AnalysisServer does. Thank you very much for the explanation!

[roman-petrov]

@scheglov thank you for all your help!

@incendial I just created a small WIP PR trying to check if having sdk-path CLI option may help Windows users to speed up analysis. I developed a custom workflow tool (linting, monorepo, testing, etc) for my company and I will be interested in using this option if compilation to EXE will give significant performance improvement.

Do you think that having such an option will give value to the project? If yes - I will continue with my PR and provide necessary comparisons/measurements.

[incendial]

@roman-petrov we definitely need an ability to detect SDK correctly (especially, to unblock a use-case for compiled executable + package manager distribution), but I'm not sure, that it should be a CLI option (a quick look at dart analyze and flutter analyze showed that they don't take such option, but work properly).

Let's continue the discussion in PR.