☂️ JNIgen transformations and possibly YAML config removal

[HosseinYousefi]

Note

All of this could be used for package:jni itself, as it already contains a lot of hand-written code for List, Map, ByteBuffer, … I plan to add support for exceptions and it's really useful to write transformers for common exception types and have them either in package:jni or even another package:java_library.

Use Cases

Differences between Java and Dart styles

For example, Java naming convention recommends constants to be SCREAMING_SNAKE_CASE, while Dart uses lowerCamelCase.

=> A way to easily change the case.

Some words are keywords in Dart

For example, the word in is a keyword in Dart and cannot be used as an argument name in a method, but it can be in Java. The current behavior is to add 0 to the end of these reserved names. So in becomes in0.

=> A way to specify certain words to rename.

Some methods and fields are used in Object

Like hashCode and toString. toString is also used in Java, however the return type of the method in Java is JString. The current behavior is to add 1 to the end of the existing methods. A prettier renaming would be to change toString to toJString or toJavaString.

=> A way to change method names, this of course needs to take the subclasses into account as well.

Some names have a different meaning in Dart semantics

In Dart, Errors are classes that we normally don't catch, in contrast to Exceptions which we do catch. This is largely the same in Java, however the exceptions that are subclasses of RuntimeException are more similar to Dart's Errors as they're "unchecked exceptions". One might want to rename classes such as NullPointerException to NullPointerError to better align with Dart semantics.

=> A way to change class names.

Some classes have the same names as dart:core classes

Even though it's technically possible to either import dart:core as core, or import <generated> as gen, it's probably easier to have a different name for classes like Object and String that exist both in Dart and Java. For example, renaming Object to JObject.

=> A way to change class names.

Purely for aesthetics

Self explanatory!

Java supports method overloading, but Dart does not

This means that int foo(), int foo(int a), and int foo(int a, int b) will turn into foo, foo1, and foo2 in Dart. An alternative for this would be to let users rename each foo or to write a custom logic with optional parameters.

=> A way to change method names.

Dart supports operator overloading, but Java does not

A good example of this is java.lang.BigInteger. Since Java doesn't support operator overloading, it has methods like add, multiply, divide, … These methods could be changed to their respective operators in Dart to match the language better. List's get and put also correlate with operators [] and []=.

=> A way to convert methods to operators.

Dart has getters and setters, but Java does not

Java is full of getFoos and setFoos. We can change these methods to getters and setters in Dart. We might only want to do this if we have a private foo field. Similarly some methods are better suited to be getters in Dart, an obvious example of this is hashCode() that can be converted to hashCode getter.

=> A way to convert methods to setters/getters.

We might not want to include certain classes/methods/fields

So, we need to have an option to exclude elements from the generated bindings.

=> A way to exclude elements.

Ability to add methods/fields to classes

For example, a toDartString could be a good method to add to JString, which is not already available. Of course, users could use extensions but those have their own limitations for example in case of static methods or constructors. This can be as simple as the ability to add any user-defined string to any class.

=> A way to add custom code.

Argument names are not available in the bytecode

When parsing from the bytecode, the argument names in methods are not available. This means that a method like int foo(int bar, double baz) gets turned into int foo(int i, double d).

=> A way to rename argument names.

Ability to implement a Dart class

For example we might want java.util.List to implement or mixin ListMixin as it is already the case for package:jni's JList.

=> A way to add custom code and to modify the class definition.

Approaches

Yaml config with regex support

This is the approach already used by ffigen. The syntax is something like:

The point here is not to specify an exact syntax for the yaml, but to give you a general idea of how it can exclude and rename.

Pros

• Relatively easy to do easy things.
• Declarative.

Cons

• Impossible to do complicated things, such as checking for some classes' structure or superclass when deciding about renaming. One such example is when trying to change all (.*)Exception classes to $1Error classes when these classes extend RuntimeException. This and many other such examples are simply not possible to do with a yaml config.
• Regex is notoriously difficult to debug.
• It's not obvious in which order the changes happen. Does only one change happen like a switch statement? Or will they happen in a chain from top to bottom?
• Fully qualified names in Java include periods and dollar signs (for inner classes) like a.b.Foo$Bar which need to be escaped since we allow regex: a\.b\.Foo\$Bar which makes it ugly (and error-prone) to do simple things.

Overall, I think it's not a good idea to have all of the transformation logic in yaml.

Passing functions to Dart config object

Another option would be to ditch the yaml configuration for more complex tasks and write Dart code. We already support this for normal use cases, and we can extend it to do transformations as well.

We could consider exposing a Method object instead of only the name to access things such as the name of the enclosing class.

This is better, we wrap the actual internal Method into a wrapper (called Method again!) with only the useful information and actions to be exposed to the user. This way any changes to the internal representation of Method will not break users' config code.

Pros

• Still easy to do easy things.
• Possible to do more complicated things.

Cons

• Having a single method for transforming methods, could lead to a messy code with a bunch of branchings.

Using transformer visitors

This is basically the same as passing functions to Dart config but could be chained together and can suggest better code organization habits.

Pros

• Almost the same as passing functions, just packaged differently

Cons

• Could be a bit unfamiliar for users who are not used to the visitor pattern.

Another Question, is yaml useful?

Suppose we go with the transformer option, will yaml be still useful? It would be weird to start off with yaml, and transfer everything to a Dart config to add a transformer. Another solution would be to "import" a dart file within the Yaml config like

But what do we gain from keeping the yaml around? Is it that much simpler than having a Dart file for beginners? Is JNIgen ever going to be used by beginners?

I would argue that most actual JNIgen use cases are going to be complicated enough to need some form of transformation.

Things to keep in mind

Imported bindings

Users cannot modify the imported bindings as they are already generated. This will impose certain limitations on what they can and cannot do. For example, one cannot rename an imported class, so we probably will have two classes Class and MutableClass, where Class is not renamable. Or have an imported boolean getter and throw when trying to modify an imported class.

What about fields and methods? The same method could be used in an imported class, and we cannot change its name if we're extending or implementing that class.

Reserved Names

Users might want to add certain names to the reserved namespace of a class. These names should not be reused in the subclasses. This happens when a package author adds custom code to introduce useful fields/methods.

Issues to tackle

• ☂️ Idiomatic conversions #612
• Import prefixes should have $ in them not to conflict with names. #614
• Incorrect renaming when the method/field already has a numeric suffix #571
• Renamer unit tests #569
• Remove yaml in favor of dart scripts for binding generation
• Keywords are too restrictive #664
• Add tests for class/method/field exclusion #694
• Name collision avoidance for typeclasses #703
• Hide JObjectPtr and others for a cleaner namespace #726
• Class & method filtering #751
• Interop between generated bindings #746
• Method / class / field renaming #787

[HosseinYousefi]

More use cases:

Renaming without breaking user's code

We could rename and deprecate. The previously used names (stored in a json file), can still work but be annotated with @Deprecated. (Could we also add a quick fix for this?)

Generate CHANGELOG

Renaming public API breaks any binding that depends on our binding, and any code that uses it. Transformers could produce a changelog to communicate the breaking changes.