Framework-less, generalized, annotation-based RPC #57

ghik · 2018-05-28T08:15:52Z

Generalized RPC/proxy/API manipulation framework.

Example code using stuff described below: NewRawRpc

`AsRaw` and `AsReal`

RpcFramework is no longer required. For backwards compatibility it's kept as a special case of the new framework.
Two global conversion typeclasses, AsReal and AsRaw (and a bidirectional conversion, AsRealRaw) have been introduced. All real-raw conversions are realized through these typeclasses. This includes both RPC trait conversion and parameter and return value serialization and deserialization.
There are macros available that materialize AsReal and AsRaw for real and raw RPC traits. Implicit macros have been removed - all macro materialized instances must be explicitly declared in companion objects of real RPC traits.
Macros that materialize AsReal and AsRaw for RPC traits do it solely based on how raw RPC trait looks like. In order to guide and customize the materialization, raw RPC, it's methods and parameters may be annotated with various meta annotations, e.g. @multi, @verbatim, @tagged.
Raw RPC trait contains raw methods. Every real method must be matched to a single raw method. If a raw method is annotated as @multi, it's supposed to handle multiple real methods. This is the most common situation. Whether a real method matches a raw method is determined solely by their signatures.
- If raw method is not annotated as @multi then its name must be the same as real method's name (with respect to @rpcName). If raw method is annotated as @multi then real method's name doesn't matter. Raw method must take that name as the only parameter in its first parameter list.
- Return types of raw and real method must match. If raw method is @verbatim then return types must be the same. If it's @encoded then there must be an appropriate AsRaw or AsReal conversion between raw and real method's return type. The macro performs an ad-hoc implicit search for that conversion and deems the methods non-matching when it cannot be found. Non-@multi raw methods are @verbatim by default while @multi raw methods are @encoded by default.
- Raw and real parameters must match. Without any meta-annotations put on raw parameters, they must match one-by-one, i.e. have the same types and be in the same order (names don't matter). Raw parameter annotated as @multi may match multiple real parameters (spanning multiple parameter lists). Its type must then be a collection: either some subclass of Iterable[Raw] or PartialFunction[String,Raw] where Raw is a type that will be matched against each real parameter type. By default, @multi raw parameters are @encoded which means that the macro will serialize and deserialize each real parameter value to Raw value using AsRaw and AsReal instances for these types. However - contrary to how method return types are treated - if one of these instance is not available, it will not affect macro's decision about whether raw and real method match. Implicit search errors for parameters are deferred until the very end, after whole macro has been expanded.
- In other words, raw method's signature serves as a simplistic grammar that real method's signature is parsed with.
Both methods and parameters may be explicitly filtered using tag annotations. Example: annotations for HTTP methods, e.g. @GET, @POST, @PUT, etc.

The design above removes several limitations:

There is no longer a single RawValue with single serialization strategy for all parameters and return types - each raw method and parameter may use its own type for encoded value with dedicated typeclasses for serialization
Parameters may be grouped, encoded and collected in various ways instead of just List[List[RawValue]].
RPC name of real methods must be unique only in scope of a single raw method that they all match to - in previous version RPC name had to be unique in the entire RPC trait

New RPC metadata

RPC metadata materialization has been reworked in largely similar way as raw-real RPC translation.

There is no longer any fixed RPCMetadata trait. RPC metadata may now be any class. Macro engine reflects on the constructor of that class and based on its parameters determines how it should be materialized. Metadata constructor parameters are annotated in the same way as raw RPC methods and parameters are annotated (i.e. with @multi, @verbatim, @encoded, etc.). This way metadata class mirrors the structure of raw RPC trait. Every raw method is reflected by a constructor parameter in metadata class. That parameter, in turn, holds another metadata class whose constructor parameters mirror raw parameters of that method.
Apart from the constructor parameters that mirror raw methods or raw parameters, every metadata class may also accept a set additional parameters reified specifically for that particular symbol (real RPC trait, method or parameter). There is another set of annotations that tell the macro engine how to reify these parameters:
- @infer - makes the macro engine search for implicit value of parameter's type. Metadata parameters may simply be declared as implicit by themselves and will also be inferred by macro, but with explicit @infer annotation you can make non-implicit parameters effectively implicit.
- @reifyAnnot - makes the macro engine reify some annotation(s) from real RPC trait/method/param. By default, macro expects the type of metadata parameter to be a subtype of StaticAnnotation. Metadata parameter with @reifyAnnot may also be specified as @optional or @multi to reify optional or repeated annotations.
- @hasAnnot[A] - makes the macro engine reify a Boolean flag telling if given real RPC trait/method/param is annotated as A.
- @reifyName - makes the macro engine reify a String containing RPC trait/method/param name (either with or without respect to @rpcName)
- @reifyPosition - makes the macro engine reify an instance of ParamPosition which contains detailed information about real parameter's index, index of its parameter list, etc.
- @reifyFlags - makes the macro engine reify an instance of ParamFlags which contains a set of flags telling if a real parameter is declared as implicit, by-name, repeated (varargs), has default value or is synthetic (e.g. comes from a context bound).
Metadata class for real method or real parameter must be a subtype of TypedMetadata[T] where T must will be matched against real method result type or real parameter type. This way matching of real methods/params to metadata params can mirror matching of real methods/params to raw methods/params.

Starzu · 2018-05-30T07:39:15Z

commons-core/src/main/scala/com/avsystem/commons/rpc/OptionLike.scala

+  def some(value: Value): O
+  def fold[B](opt: O, ifEmpty: => B)(f: Value => B): B
+}
+trait BaseOptionLike[O, A] extends OptionLike[O] {


Do you need both of these traits? If yes, could you put a comment why?
Besides that shouldn't BaseOptionLike be sealed too?

I use the base trait to do implicit search in macros. This way I avoid using wildcards and the implicit search is simpler. At the same time, I want to avoid the base trait to be extended directly because someone might forget about setting the type member to concrete type. For this reason, OptionLike is sealed while BaseOptionLike is not.

Starzu · 2018-05-30T08:48:57Z

commons-core/src/main/scala/com/avsystem/commons/rpc/rpc.scala

+  def materializeForRpc[M[_], T]: M[T] = macro macros.rpc.RPCMacros.rpcMetadata[M[T], T]
+}
+
+trait TypedMetadata[T]


Could you describe the role of this trait?

Yep, it needs docs.

In short: metadata class for method or parameter must extend TypedMetadata[X] where X is either the result type of a real method or the type of real parameter. For example:

class FunctionMetadata[T](...) extends TypedMetadata[Future[T]]

and then:

class RpcMetadata[T]( functions: Map[String, FunctionMetadata[_]] ... )

The macro will then match Future[_] against each function actual result type and determine what to substitute for the wildcard.

Starzu · 2018-05-30T09:13:27Z

commons-core/src/test/scala/com/avsystem/commons/rpc/NewRawRpc.scala

+case class GET() extends RestMethod
+case class PUT() extends RestMethod
+
+@methodTag[RestMethod, RestMethod]


@methodTag[RestMethod, GET/POST/PUT]?

RestMethod (the base trait) as the default means pretty much "no default". This is a test, anyway.

Starzu · 2018-05-30T09:14:16Z

commons-core/src/test/scala/com/avsystem/commons/rpc/NewRawRpc.scala

+  implicit def futureAsRealRawFromGenCodec[T: GenCodec]: AsRealRaw[Future[String], Future[T]] = ???
+}
+
+@methodTag[RestMethod, RestMethod]


@methodTag[RestMethod, GET/POST/PUT]?

Starzu · 2018-05-30T12:59:26Z

commons-annotations/src/main/scala/com/avsystem/commons/rpc/RPC.scala

-  * RPC interfaces may also have non-abstract members - these will be invoked locally. However, they may invoke
-  * remote members in their implementations.
-  */
-class RPC extends StaticAnnotation


Can we deprecate it first and remove in the next release?

Hmm, do you have a subclass of RPC in Udash? Can we leave that one instead and deprecate it?

Starzu · 2018-05-30T12:59:30Z

commons-annotations/src/main/scala/com/avsystem/commons/rpc/RPCName.scala

-/**
-  * You can use this annotation on overloaded RPC methods to give them unique identifiers for RPC serialization.
-  */
-class RPCName(val name: String) extends StaticAnnotation


Can we deprecate it first and remove in the next release?

I actually renamed it to rpcName to retain some naming convention. To be honest, I'd like to have a single annotation for both GenCodec (which currently uses @name) and RPC. So I will probably introduce some deprecated type aliases (although I'm not sure if they work with annotations).

I just want to get deprecation warning (with migration instructions) instead of class not found error. :)

I made an alias in Udash PR

Starzu · 2018-06-07T05:51:00Z

commons-core/src/main/scala/com/avsystem/commons/rpc/AsRealRaw.scala

+import scala.annotation.implicitNotFound
+
+@implicitNotFound("don't know how to encode ${T} as ${R}, appropriate AsRaw instance not found")
+trait AsRaw[R, T] {


Could you replace R and T with more meaningful names? Maybe Raw and Real?

Starzu · 2018-06-07T07:25:07Z

commons-core/src/main/scala/com/avsystem/commons/rpc/AsRealRaw.scala

+}
+
+@implicitNotFound("don't know how to encode and decode between ${T} and ${R}, appropriate AsRealRaw instance not found")
+trait AsRealRaw[R, T] extends AsReal[R, T] with AsRaw[R, T]


I think this name is misleading: AsRealRaw[Raw, Real]. In my opinion we should keep Raws and Reals in the same order.

ddworak

Can you compare RpcFramework performance before and after these changes? Would be a nice sanity check.

ddworak · 2018-06-07T06:27:04Z

commons-akka/src/main/scala/com/avsystem/commons/rpc/akka/client/ClientRawRPC.scala

@@ -4,7 +4,7 @@ package rpc.akka.client
 import akka.actor.ActorSystem
 import akka.pattern.ask
 import akka.util.Timeout
-import com.avsystem.commons.rpc.akka.AkkaRPCFramework.RawRPC
+import com.avsystem.commons.rpc.akka.AkkaRPCFramework.{RawInvocation => _, _}


import com.avsystem.commons.rpc.akka.AkkaRPCFramework.{RawRPC, RawValue}

would suffice

ddworak · 2018-06-07T06:32:44Z

commons-annotations/src/main/scala/com/avsystem/commons/annotation/AnnotationAggregate.scala

+  *
+  * In order to specify aggregated annotations, the class that extends `AnnotationAggregate` must
+  * redefine the `Implied` dummy type member and apply the aggregated annotations on it. Macro engines
+  * used in `GenCodec` materialization and RPC framework will automatically pick up these annotations.


I suggest we start with the fact that other macros won't, or move this to a more GenCodec-specific package.

Why? Annotation processing is a part of MacroCommons and it should work everywhere.

Non-AVSystem macros.

I specifically listed GenCodec macros and RPC macros - what's the problem?

Must have misread something 😢

ddworak · 2018-06-07T06:34:52Z

commons-annotations/src/main/scala/com/avsystem/commons/annotation/AnnotationAggregate.scala

  *
  *   case class SomeMongoEntity(@mongoId id: String, data: String)
  * }}}
  *
  * In the above example, applying `@mongoId` annotation on the `id` field has the same effect as if
  * annotations `@name("_id") @outOfOrder` were applied directly on that field.
+  *
+  * NOTE: thanks to the fact that aggregated annotations are applied on a type member
+  * you can pass the arguments of original annotation to aggregated annotations, e.g.


Do I need to know why when I ctrl+q for a quick overview? The type member note should be a comment on Implied (and it is).

IntelliJ cannot into Scaladoc. Among other problems, it removes annotations from code snippets, like in this case. I'm not going to work around it. Just read the sources or API reference.

ddworak · 2018-06-07T06:35:39Z

commons-annotations/src/main/scala/com/avsystem/commons/annotation/AnnotationAggregate.scala

+trait AnnotationAggregate extends StaticAnnotation {
+  /**
+    * Dummy type member meant to be redefined in order to have aggregated annotations applied on it.
+    * These annotations will be automatically picked up by macro engines each time they encounter


Macro engines or our macros?

Definitely not all macro engines in the world.

ddworak · 2018-06-07T07:55:05Z

commons-annotations/src/main/scala/com/avsystem/commons/rpc/rpcAnnotations.scala

+  * (see [[paramTag]] for more information on param tagging).
+  *
+  * Raw parameters marked as [[optional]] must be typed as `Option[T]` (or `Opt[T]`, `OptArg[T]` or whatever type that
+  * has an instance of `OptionLike`). By default, [[optional]] raw parameters are [[verbatim]] which means that the


Note where that instance should be provided (imported) when using a custom one (obviously T companion object works, but sometimes you don't want to clutter it).

That should be obvious to everyone who knows how implicits work in Scala 😎

ddworak · 2018-06-07T09:34:40Z

commons-core/src/main/scala/com/avsystem/commons/rpc/RpcUtils.scala

+/**
+  * @author ghik
+  */
+object RpcUtils {


These are only used in macro-generated code - maybe we could move this to the macro module?

Macro module is supposed to contain macro implementations only so that it can be a compile-time only dependency if necessary.

ddworak · 2018-06-07T09:37:01Z

commons-core/src/main/scala/com/avsystem/commons/rpc/rpcMetadata.scala

+  * Information about real parameter flags and modifiers as defined in Scala code.
+  */
+@transparent
+final case class ParamFlags(rawFlags: Int) extends AnyVal {


Enumeration + ValueSet would be more object friendly and less off-by-one error-prone.

Did you just suggest using scala.Enumeration to me?

I'd rather Enumeration than C-style bit-fiddling. ValueSet is actually quite a nice wrapper around BitSet.

I refuse to use Enumeration which uses reflection and is widely disgusted. I'm going to introduce some safety measures to avoid off-by-one errors in refactorings.

Conscience clause of Scala developers

ddworak · 2018-06-07T09:45:13Z

commons-macros/src/main/scala/com/avsystem/commons/macros/rpc/RPCMacros.scala

+
+import scala.reflect.macros.blackbox
+
+abstract class RPCMacroCommons(ctx: blackbox.Context) extends AbstractMacroCommons(ctx) {


wiedziałem

ddworak · 2018-06-07T09:45:39Z

commons-macros/src/main/scala/com/avsystem/commons/macros/rpc/RPCMappings.scala

+import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
+
+trait RPCMappings { this: RPCMacroCommons with RPCSymbols =>


ddworak · 2018-06-07T09:46:19Z

commons-macros/src/main/scala/com/avsystem/commons/macros/rpc/RPCMetadatas.scala

+
+import scala.annotation.StaticAnnotation
+
+trait RPCMetadatas { this: RPCMacroCommons with RPCSymbols with RPCMappings =>


# Conflicts: # version.sbt

ghik assigned mkej May 28, 2018

ghik requested review from Starzu, ddworak and mkej May 28, 2018 08:15

ghik added 26 commits May 29, 2018 11:22

first prototype of one RPC macro to rule them all

232ddff

AsRaw for RPC macro prototype

185195d

recursive implicit reuse

a3b8418

RPC support for default parameter values

1f7fd1c

RPC support for varargs and by-name parameters

fd0f2f4

@RPCName for params, relaxed uniqueness constraint for methods

e2c358e

AnnotationAggregate reworked to allow argument passing

aa17fc0

error reporting reworked

3793232

grouping real parameters with annotations

a905611

first probably working prototype of generalized RPC

5720e63

refactored RPC param encoding defaults

ab3df30

injecting raw RPC companion implicits, removal of duplicated implicits

f77610d

improved error reporting in RPC macros

1034c03

nice and detailed error messages from RPC macros

1e222e1

RPCFramework is now a bridge over new RPC engine

16af0a3

reworked method and param matching in RPC macros, tag annotations

f7a1db5

scaladocs for RPC annotations, cosmetic refactors

6077ebf

adjusted annotation extraction order

9e6901a

reverted order of type params in AsRaw/AsReal

5021726

repeated renamed to multi

d2f48ac

changed required signature of raw methods

ee288b3

refactors in RPCMacros

2269f55

more refactors in RPC macros in preparation for RPC metadata

6c317ad

minor refactors in RPCMacros

6f63589

split RPCMacros into subtraits

bfd2c1d

minor

e3686a6

ghik added 5 commits May 29, 2018 11:22

legacy RPCFramework now uses new RPC metadata

a9bc652

trailing comma

2269a96

MetadataAnnotation is toplevel

dcd58af

@reifyName, @reifyPosition, @reifyFlags and documentation, refactors

f98f72a

@whenAbsent annotation for GenCodec and RPC

899a5ff

ghik force-pushed the uberpc branch from 2502644 to 899a5ff Compare May 29, 2018 09:23

GenCodec docs updated, some tests for RPC metadata

2b65b11

Starzu reviewed May 30, 2018

View reviewed changes

ghik added 7 commits June 1, 2018 15:15

RPCMacros is not final

7d465a3

raw method arity

51ff4e2

optimized asReal/asRaw conversion trees

6a43f49

commons-annotations cross compilation

b7e4215

renamed @reify to @reifyAnnot, introduced @hasAnnot

8dc6510

more docs, allowed raw methods to take no parameter lists

4f9642b

cosmetic

ba2123c

Starzu reviewed Jun 7, 2018

View reviewed changes

more descriptive type parameter names for AsReal/AsRaw

3620fd0

Starzu reviewed Jun 7, 2018

View reviewed changes

ghik added 2 commits June 7, 2018 09:56

AsRealRaw renamed to AsRawReal

15ccd49

cosmetic consistency

eb6d656

Starzu approved these changes Jun 7, 2018

View reviewed changes

ddworak reviewed Jun 7, 2018

View reviewed changes

ghik added 3 commits June 7, 2018 13:09

RPC -> Rpc outside of RPCFramework

dd0a515

moved indent extension method to tests

ee8e578

safer ParamFlags

675cf21

ddworak approved these changes Jun 7, 2018

View reviewed changes

mkej approved these changes Jun 7, 2018

View reviewed changes

Merge remote-tracking branch 'origin/master' into uberpc

eb9f678

# Conflicts: # version.sbt

ghik merged commit 45ab789 into master Jun 7, 2018

ghik deleted the uberpc branch June 7, 2018 15:08


		import scala.reflect.macros.blackbox

		abstract class RPCMacroCommons(ctx: blackbox.Context) extends AbstractMacroCommons(ctx) {


		import scala.annotation.StaticAnnotation

		trait RPCMetadatas { this: RPCMacroCommons with RPCSymbols with RPCMappings =>

Framework-less, generalized, annotation-based RPC #57

Framework-less, generalized, annotation-based RPC #57

Conversation

ghik commented May 28, 2018 • edited Loading

Generalized RPC/proxy/API manipulation framework.

AsRaw and AsReal

New RPC metadata

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddworak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghik commented May 28, 2018 •

edited

Loading

`AsRaw` and `AsReal`