Interop between generated bindings

[dcharkes]

We should support interop between 2 or more jnigen-generated packages.

_Originally posted by @mahesh-hegde in #761

[mahesh-hegde]

Approach 1:

type_map a la ffigen

pseudo-config

type_map:
    'com.apache.pdfbox.PDDOcument': 'pdfbox'
    'android.os.Process': 'android_os'

import_libraries:
    'pdfbox': 'package:pdfbox/pdfbox.dart'
    'android_os': 'package:android/os.dart'

Approach 2: similar to above, but infer library names by package names.

import_map:
    'com.apache.pdfbox.PDDOcument': 'pdfbox'       // library imported as `pdfbox_`
    'android.os.Process': 'package:android/os.dart'   // library imported as `os_`

Advantage: simpler config
Disadvantage: less control over import names in bindings, if that's an issue.

[dcharkes]

Cross linking the ffigen issue:

• Sharing shared definitions #539

cc @mannprerak2 any suggestions from thinking about this in the ffigen context?

[mahesh-hegde]

As detailed in #521 , We don't have transitive dependency resolution at least now. Only specified packages are traversed.

So in the scenario, {A, B, C}, B and C import A.T, both B and C have to specify A.T using import map only.

But yeah, the fundamental issue persists even if the dependencies aren't implicitly included.

If b.yaml explicitly binds B.X, A.Y and c.yaml binds C.Z, A.Y, Y from both packages isn't the same.

In jni there's no structs, everything is a fragile, weakly typed pointer so you can cast. But for the fundamental issue, I think there's no solution.

In such cases, manually refactoring A into seperate jnigen config, and linking using some import map is best bet.

[dcharkes]

Interaction with output format

With a mapping from Java type to generated dart binding (both approaches), the API gets tied to whether we chose to generate single file or multi file.
If the multi-file has approach still has a central file that exports everything, then it doesn't matter.

Config

I like approach 2 better. I don't think we care about the imported library name in the generated file.

How about we make the keys regexes or prefixes? That way we can match all types in a package:

# Map<QualifiedJavaType, DartLibraryUrl>
imports:
  'com.apache.pdfbox.*': 'package:pdfbox/pdfbox.dart'

[dcharkes]

In such cases, manually refactoring A into seperate jnigen config, and linking using some import map is best bet.

Agreed. That is the only way that this will work. (Same for ffigen.)

[mahesh-hegde]

I suppose regex is not required, if we specify

'com.apache.pdfbox': 'package:pdfbox/pdfbox.dart'

and jnigen needs to resolve com.apache.pdfbox.pdmodel.PDDocument

it should first check for com.apache.pdfbox.pdmodel.PDDocument, no specification,
then for com.apache.pdfbox.pdmodel
then for com.apache.pdfbox - match found.

Any pitfalls of this approach?

[dcharkes]

Yeah in the context of Java a prefix might make more sense than a regex. 👌

[mannprerak2]

cc @mannprerak2 any suggestions from thinking about this in the ffigen context?

For ffigen, libclang provides USR (id), which are supposed to be consistent across translation unit (individual parsing).
Maybe we can export a custom yaml/json file which can contain type-map for declarations with their USR for a parsed structures (Or maybe add annotations in the generated code itself)
Any new library can simply specify this (In library-import config), and since the USR would be same, we can reuse the definition similar to how type-map works.

Not sure if this is useful for jnigen.

[mahesh-hegde]

@mannprerak2 It's same in Java actually. The fully qualified name is supposed to be same across all libraries loaded in single process.

Does ffigen have any mechanism to export USRs automatically, or is it manually specified through type map?

[mannprerak2]

Does ffigen have any mechanism to export USRs automatically, or is it manually specified through type map?

Not yet, But it should be relatively easy to add to ffigen.

[dcharkes]

(note to self: USR = Unified Symbol Resolution)

We should design a format.

What information should be in there?

1. The USIs (Unique Symbol Identifiers)
2. The file in which the a USI is defined. Or should it be the Uri in the package? For that we need to define where in a package and in which package (package name).

If we have multiple USIs for one Uri, we should probably nest the USIs in Uris.

symbols.yaml

symbols:
  'package:pdfbox/pdfbox.dart':
    - 'com.apache.pdfbox.*'

or

symbols:
  'package:pdfbox/pdfbox.dart':
    - 'com.apache.pdfbox.pdmodel.PDDocument'
    # ...

For jnigen it might be nicer to just do

symbols:
  'com.apache.pdfbox' : 'package:pdfbox/pdfbox.dart'

But if we spell out all symbols

symbols:
  'com.apache.pdfbox.pdmodel.PDDocument' : 'package:pdfbox/pdfbox.dart'
  'com.apache.pdfbox.pdmodel.Foo' : 'package:pdfbox/pdfbox.dart'
  'com.apache.pdfbox.pdmodel.Bar' : 'package:pdfbox/pdfbox.dart'

because we're unlikely to have multiple regexes in a single .dart.

However, for ffigen, it is a bit more likely.

One question is: should we output an exact list of symbols, or a regex (or prefix)?

A prefix from the config in jnigen is probably fine, it is unlikely to collide with other names.

However, in the ffigen context, taking the method/class include filters is likely to not work. So there we should probably output the full list of symbols.

If we'd like to keep the ffigen and jnigen symbols.yaml output consistent with each other, it should be

symbols:
  # Map<DartPackageUri, List<UniqueSymbolIdentifer>>
  'package:pdfbox/pdfbox.dart':
    - 'com.apache.pdfbox.pdmodel.PDDocument'
    - 'com.apache.pdfbox.pdmodel.Foo'
    - 'com.apache.pdfbox.pdmodel.Bar'
    # ...

wdyt?

[mahesh-hegde]

In java, binaryName is unambiguous and uniquely maps to a class. FQN may be ambiguous in rare cases. In jnigen we use binaryName everywhere.

I am not familiar with USRs.

1. Are they serializable in some way?
2. Are they same across different invocations / parses of same set of files?
3. Are they same for a given symbol across different sets of files? For example, if you have {A.h, X.h} in one ffigen config1, {C.h, X.h} in config2, is the USR for symbol x from X same in both? Assuming same system of course.

I think it should also be possible to semi-manually work around this, by having the user provide a mapping which asserts X.h in config2 is equivalent to X.h in config1.

In the rest of this comment I am assuming we have a way to identify each binding symbol in string format, and there's some hierarchical structure. Fortunately this is almost always true in Java.

Re: YAML format

I think it's logical to have a mapping from required symbol to required import and not vice versa, because we lookup by symbol, and it's generated file && read by program, even if it looks tidier to have a tree with each import listing what it provides. Of course you can convert to different internal representation with a simple for loop. So matter of preference. Should this file be human readable? Does this file's size matter?

Re: lookup

Assuming a hierarchical structure among bindings, i.e given an identifier you can tell it's parent (eg java package or header file), we can easily implement lookup in most-specific to most-general order. (As I gave an example above).

a.b -> 'package:a/b.dart';

resolve(a.b.C) will look for a.b.C first, then a.b, and matches the above spec.

Assuming b encompasses all subpackages as well, a.b.c.D will match the above import spec. Maybe if we don't want this behavior, we can use a '/' or other symbol to explicitly specify that 'a.b' is recursively encompassing, and change the lookup code accordingly. That's not a problem.

Now, assuming some classes were filtered out using config, we need to explicitly write the filtered out class with null mapping, so an explicitly skipped class a.b.X won't match the above spec. Code should stop at seeing null.

There are more implementation details to still flesh out, but I think the basic idea is sufficient.

I think regular expressions may be harder to do constant time lookup, unless maybe you sort the list of all regexes and binary search the ones closest to what you are looking for. with prefixes there will be a single digit number of combinations you have to check in a map. Is there any specific advantage of using regex in ffigen context?

But yeah above all is leetcode. If it's only a machine read-write file I don't mind writing out every type either, it will still be smaller than generated binding file. And reading some JSON/YAML should be pretty fast compared to parsing C/Java.

https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_computer_science

[mannprerak2]

Are they serializable in some way?
Are they same across different invocations / parses of same set of files?
Are they same for a given symbol across different sets of files? Assuming same system of course.

Yes to all.