Source generation fails with a package declaration

[AdrianRaFo]

I have this error when I add a package declaration on a scala protocol and try to generate avro or proto from it.

[error] scala.tools.reflect.ToolBoxError: reflective compilation has failed:
[error] 
[error] '{' expected.
[error]         at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.throwIfErrors(ToolBoxFactory.scala:320)
[error]         at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.parse(ToolBoxFactory.scala:295)
[error]         at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.$anonfun$parse$1(ToolBoxFactory.scala:421)
[error]         at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.apply(ToolBoxFactory.scala:359)
[error]         at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.parse(ToolBoxFactory.scala:418)
[error]         at higherkindness.mu.rpc.idlgen.IdlGenerator.generateFrom(IdlGenerator.scala:44)
[error]         at higherkindness.mu.rpc.idlgen.IdlGenerator.generateFrom$(IdlGenerator.scala:37)
[error]         at higherkindness.mu.rpc.idlgen.avro.AvroIdlGenerator$.generateFrom(AvroIdlGenerator.scala:27)
[error]         at higherkindness.mu.rpc.idlgen.Generator.$anonfun$generateFrom$1(Generator.scala:30)
[error]         at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:937)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:937)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error]         at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
[error]         at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
[error]         at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
[error]         at higherkindness.mu.rpc.idlgen.Generator.generateFrom(Generator.scala:29)
[error]         at higherkindness.mu.rpc.idlgen.Generator.generateFrom$(Generator.scala:25)
[error]         at higherkindness.mu.rpc.idlgen.avro.AvroIdlGenerator$.generateFrom(AvroIdlGenerator.scala:27)
[error]         at higherkindness.mu.rpc.idlgen.GeneratorApplication.generateFrom(GeneratorApplication.scala:63)
[error]         at higherkindness.mu.rpc.idlgen.IdlGenPlugin$.$anonfun$idlGenTask$1(IdlGenPlugin.scala:186)
[error]         at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:73)
[error]         at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:147)
[error]         at sbt.util.Difference.apply(Tracked.scala:313)
[error]         at sbt.util.Difference.apply(Tracked.scala:293)
[error]         at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:143)
[error]         at sbt.util.Difference.apply(Tracked.scala:313)
[error]         at sbt.util.Difference.apply(Tracked.scala:288)
[error]         at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:142)
[error]         at higherkindness.mu.rpc.idlgen.IdlGenPlugin$.$anonfun$taskSettings$1(IdlGenPlugin.scala:127)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:67)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:278)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (server_protocol / idlGen) scala.tools.reflect.ToolBoxError: reflective compilation has failed:
[error] 
[error] '{' expected.

[juanpedromoreno]

@AdrianRaFo Could you please add more context about what are you trying to do? Configs and code would be helpful. Thanks!

[AdrianRaFo]

code

package quickstart

import freestyle.rpc.protocol._

import shapeless.{:+:, CNil}
@option("java_multiple_files", true)
@option("java_outer_classname", "Quickstart")
@outputName("GreeterService")
@outputPackage("quickstart")
object service {
  sealed trait People extends Product with Serializable

}

config

idlType := "avro",
srcGenSerializationType := "AvroWithSchema",
sourceGenerators in Compile += (srcGen in Compile).taskValue,
libraryDependencies ++= Seq(
  "io.higherkindness" %% "mu-rpc-channel" % V.muRPC
)

Just the same config and same scala protocol works without declaring package quickstart and fails with the above error when I add it

[juanpedromoreno]

Thanks @AdrianRaFo , I see what you mean now. You want to generate IDL files from the Scala code..

That reminds me that we need to deprecate and remove that feature. Scala protocols are discouraged because of the binary issues, hence this wouldn't make sense.

[SemanticBeeng]

Scala protocols are discouraged because of the binary issues

@juanpedromoreno Is such a shapeless coproduct based protocol supposed to work ?

This is powerful and very useful for mapping protocol @message s to the domain objects.

Since the Scala from IDL generated code uses shapeless I expect the mu runtime would also work with same kind of Scala types handcrafted (no IDL involved): perform a runtime interpretation of these types to protobuf binary that is consistent with the idl-gen as suggested in a few places like https://higherkindness.io/mu/idl-generation and http://frees.io/docs/rpc/idl-generation.

Not sure what you mean by "Scala protocols" since even when starting with "*.proto" the end result is Scala objects.

The answer should be in how the mu runtime itself handles this Scala <-> protobuf serialization.
Please advise.

[juanpedromoreno]

Not sure what you mean by "Scala protocols" since even when starting with "*.proto" the end result is Scala objects.

That's fine and that's the ideal pattern to use.

What I meant is that we shouldn't define the messages and services directly in Scala, by hand. If you do so, you might need to use them as jar dependencies in the client side, for example. This highly dangerous since you would be carrying on all the transitive dependencies from where the Scala protocol was defined. Therefore this might end with evictions and binary incompatibilities.

What would be the recommended pattern? Defining the protocol in IDL, so sbt-mu-idlgen plugin would generate the sources in every place, with the local dependencies in the project where you import the IDL definitions.

[SemanticBeeng]

define the messages and services directly in Scala, by hand. If you do so, you might need to use them as jar dependencies in the client side, for example.

Please forgive me for missing something maybe important.
If the api semantics is to use protobuf according to an IDL can they not be called from a called from any client, JVM or not, regardless of how server runtime uses custom Scala classes ?

If "yes" then why would I need custom Scala objects on the client.
I am actually looking to call from browser.
Sorry I worked late and maybe blind to something. :-)

[juanpedromoreno]

No problem :)

Taking into account that we are on a grpc ecosystem, if you have IDL definitions (.proto files) for your services and messages, that would mean that everything would work across languages and platforms: C/C++, C#, Dart, Go, Java, Node.js, PHP (beta), Python, Ruby, and of course, Scala. Obviously, the tooling in every case would be different but in the end, semantically should be the same. I hope this makes sense to you.

[SemanticBeeng]

I think "yes". :-)
Let me say in different words to make it more clear what I mean by "runtime semantics".

I expect that any @message and all dependent types (included nested) are interpreted to protobuf at runtime by mu, in a way that is consistent with the semantics of the corresponding IDL, regardless if generated, source or handcrafted.

Is that a "yes" too :-)
Thank you and @AdrianRaFo !

[sander]

Thanks @AdrianRaFo , I see what you mean now. You want to generate IDL files from the Scala code..

That reminds me that we need to deprecate and remove that feature. Scala protocols are discouraged because of the binary issues, hence this wouldn't make sense.

@juanpedromoreno, do you mean that the future-proof way to use Mu in all cases is to (1) define the protocols in IDL (e.g. .proto files) and (2) use srcGen to generate Mu/Scala code from that?

From the response by @fedefernandez on #599 I had the impression that idlGen is still supported and being developed further – or are the @service and @message annotations really designed to be written by srcGen?

@AdrianRaFo, I had the same ToolBoxError and fixed it by using package quickstart { … } instead of package quickstart\n…, i.e. wrapping the package contents with { and }.

[fedefernandez]

@sander you're right in your second assumption. The annotation is supposed to be written by the srcGen. I'm still working on giving support to the issue #599 at plugin level, this is why I didn't suggested that approach for your case.

[sander]

Thanks for your quick reply @fedefernandez. Just to be sure I understand well: Mu has

• idlGen to generate IDL from Scala source files;
• srcGen to generate Scala source files from IDL.

You write that the annotation is supposed to be written by idlGen, but do you in fact mean srcGen in that statement?

[fedefernandez]

Yes, you're right, edited 😊

[juanpedromoreno]

idlGen has been removed and this issue shouldn't be reproducible 👍