Add ability to include generated code in compilation

[gaaclarke]

This tracker is for issues related to:

• Common Front End (CFE) and kernel

description

The flutter tool does the following invocation to compile the AOT app.dill:

/Users/aaclarke/dev/flutter/bin/cache/dart-sdk/bin/dart
  --disable-dart-dev
  /Users/aaclarke/dev/flutter/bin/cache/artifacts/engine/darwin-x64/frontend_server.dart.snapshot
  --sdk-root /Users/aaclarke/dev/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk_product/
  --target=flutter
  --no-print-incremental-dependencies
  -DFLUTTER_WEB_AUTO_DETECT=true
  -Ddart.vm.profile=false
  -Ddart.vm.product=true
  --aot
  --tfa
  --packages /Users/aaclarke/dev/flutter-examples/foo/.dart_tool/package_config.json
  --output-dill /Users/aaclarke/dev/flutter-examples/foo/.dart_tool/flutter_build/e8a7daa1dfd64733ed9757f1817652a6/app.dill
  --depfile /Users/aaclarke/dev/flutter-examples/foo/.dart_tool/flutter_build/e8a7daa1dfd64733ed9757f1817652a6/kernel_snapshot.d
  package:foo/main.dart

When the Flutter team implemented the Dart plugin registrant the Flutter tool started generating Dart code to be compiled at build time. Flutter developers got the generated code incorporated into the app.dill by replacing the package:foo/main.dart argument with a path to the generated code. This worked, but messes up the behavior of Dart_RootLibrary so that it broke custom entry points (see flutter/flutter#91841).

proposal

Add the ability to specify additional files that will be compiled into the app.dill via a flag. For example:

/Users/aaclarke/dev/flutter/bin/cache/dart-sdk/bin/dart
  --output-dill /foo/bar/app.dill
  --aot
  --depfile foo.d
  --include /foo/bar/generated.dart
  package:foo/main.dart

generated.dart will live in the package that is being compiled as if it were at lib/generated.dart.

alternatives considered

1. Generated the code into the lib/ directory - We don't want to fiddle with users source code, they shouldn't have to see it or maintain it. We want it to be ephemeral and hidden in the build process.

cc @alexmarkov

[alexmarkov]

@johnniwinther We might need to extend CFE APIs (kernelForProgram or CompilerOptions) to support this use case.

[alexmarkov]

129304e added an option --source to the front-end server.

A few caveats:

• Specifying multiple sources requires specifying --packages option (otherwise front-end complains that it cannot infer its location).
• The front-end server communication protocol was not changed. That means that compile command in the front-end server protocol still takes a single source as input, but extra sources specified on the command line are included into compilation.

@gaaclarke Please try out the new option when the change rolls into Flutter.

[gaaclarke]

@alexmarkov This works in that it fixes the entrypoint bug we had and I can see the code pointed to by --source is compiled and in the output binary. However when we try to find that symbol with Dart_GetClass there's an error Class '_PluginRegistrant' not found in library ''. here: https://github.com/flutter/engine/blob/4a3e7a5b72363c1f363d3000d04719c6938d963f/runtime/dart_isolate.cc#L703

I'm only using one --source parameter and it doesn't complain. It seems like the engine is using the RootLibrary but the code from --source isn't associated with that library?

[alexmarkov]

No, those are different libraries.

If you're not using part files, then library is just a Dart source file. So, if you compile using --source foo.dart bar.dart you will end up with 2 distinct libraries, foo.dart and bar.dart. Main library, returned by Dart_RootLibrary is bar.dart. If you're looking for a class in foo.dart, then you need to find foo.dart library using Dart_LookupLibrary.

So, you probably need to change code in https://github.com/flutter/engine/blob/4a3e7a5b72363c1f363d3000d04719c6938d963f/runtime/dart_isolate.cc#L722-L730
and pass a different library to InvokeDartPluginRegistrantIfAvailable.

[gaaclarke]

@alexmarkov thanks, Dart_LookupLibrary doesn't have a docstring, what are my options for finding my file passed in by --source. (Also, I found that if I supplied the wrong argument I get a Dart_IsError which later interferes with dart execution. Can I clear out that error from the C API?)

[alexmarkov]

There are other ways of obtaining library such as Dart_ClassLibrary and Dart_GetLoadedLibraries, but if you don't have anything from that library, then the best way is to find it by URI (fully qualified name of the library which you can also use to import it, e.g. package:foo/bar.dart) with Dart_LookupLibrary. For example Dart_LookupLibrary(Dart_NewStringFromCString("package:foo/bar.dart")).

I think if you're not propagating an error and don't use error handle then it will be ignored. But that's discouraged (similarly to catching all exceptions and ignoring them).

[gaaclarke]

FYI I printed out all the loaded libraries and the one compiled with --source has a library associated with file:///Users/aaclarke/temp/test_plugin/example/.dart_tool/flutter_build/generated_main.dart, not associated with the package.

This is fine, I'll just make some code that will look for the presence of this.

[alexmarkov]

@gaaclarke generated_main.dart has file:/ URI because it is not inside lib directory of the package. If you'd like to go an extra mile, it is probably possible to make a separate package for the generated main and add that package to .dart_tool/package_config.json.