Add __INTRODUCED_IN annotations to all APIs

[camillol]

Description

All NDK APIs (specifically, all symbols that are imported from system libraries) should be annotated with an __INTRODUCED_IN macro. This is currently missing from several APIs introduced in recent NDK versions, e.g. it's not in the NdkMedia*.h headers, hardware_buffer.h, etc.

Looking at the unified headers from r17, several files just have large sections wrapped by #if blocks, e.g. #if __ANDROID_API__ >= 21. This is not a good substitute for annotating the individual symbol. With individual symbol annotations, we can change the definition of __INTRODUCED_IN to insert availability attributes, enable weak-linking, etc.; with #if blocks, we cannot do any of that.

Environment Details

Applies to all versions of the NDK.

[wang-bin]

May i ask how to use weak linking without linking to the library, for example libmediandk? For apple OSes, it's quit easy, app continues if the weak linked library can not be loaded. But on linux, app can not start without the dependent library.

[DanAlbert]

This isn't something we can support currently. After this bug is fixed it will be possible for us to add that option.

[enh]

all the __INTRODUCED_IN annotations should be in r18.

[rprichard]

I think we want the fix for http://b/110994763 in r18.

[enh]

true, but they're not regressions :-)

[enh]

marcone forgot to add the Bug: line to his fixes, but i think that's fixed as of this morning.

[rprichard]

We've added the __INTRODUCED_IN annotations to many APIs, but the __ANDROID_API__ guards are still there. We have an unscheduled item on the roadmap for allowing weak linking to symbols: https://android.googlesource.com/platform/ndk/+/master/docs/Roadmap.md#weak-symbols-for-api-additions.

[camillol]

If __INTRODUCED_IN is overrideable (e.g. check if EXPERIMENTAL_CUSTOM_INTRODUCED_IN is defined, use it if so), then applications that need it can start experimenting with weak linking now. Could we please have that? 2018年8月10日(金) 17:20 Ryan Prichard <notifications@github.com>:

…

We've added the __INTRODUCED_IN annotations to many APIs, but the __ANDROID_API__ guards are still there. We have an unscheduled item on the roadmap for allowing weak linking to symbols: https://android.googlesource.com/platform/ndk/+/master/docs/Roadmap.md#weak-symbols-for-api-additions . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#706 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL-sMe4H3POyXfYbF7atqUz0fpDiH5Iks5uPiNFgaJpZM4UOhnD> .

[DanAlbert]

There's still an ifdef around the whole header. We need to make some changes to one of our told before that's possible.

[camillol]

Well, annotating the individual APIs was the large part of the job. The rest of the work to make weak-linking possible (at least experimentally) should not take long. I can send a patch if you point me to the right repo (not this one, it seems). 2018年8月11日(土) 9:24 Dan Albert <notifications@github.com>:

…

There's still an ifdef around the whole header. We need to make some changes to one of our told before that's possible. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#706 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL-sMyY5mS5VvovwO3PI1klEhbxrH_Dks5uPwVOgaJpZM4UOhnD> .

[DanAlbert]

Merged into r18.

[cpsauer]

@camillol, can I ask if you ever got the weak linking working? We're looking at the same thing.

[camillol]

Weak linking on Android works, I've used it for other APIs. For this one, I think we ended up calling the Java API via JNI, which IIRC gave us compatibility with earlier API levels. Anyway, if the header hasn't been updated to support weak linking your can make your own copy and add the attributes.

[cpsauer]

@camillol, any chance I could get you to share a few more details of your weak linking fix?
I'm particularly curious what you ended up swapping out __INTRODUCED_IN for, how you did so, and how you loaded libraries that were conditionally available.

In general, did you find JNI to be a better solution? Or did you use both?

Thanks so, so much.

[camillol]

You need to add __attribute__((weak_import)) to each declaration (which can be done by defining a macro before including the header if the header is set up for it). The library will be linked and loaded as normal, but the absence of weak-linked symbols won't prevent your program from loading: the missing symbols will just be left as null. You need to check that they're actually available at runtime before using them. This is generally much more convenient than using JNI. However, NDK APIs tend to be released in a later API level than the Java equivalent, which may or may not be important to you.

[cpsauer]

@camillol, few last things, if that's okay

• Mechanically, how did you end up doing the macro redefinition?
  • Best option I'm currently thinking of is overriding versioning.h using -I. Is that what you did?
• Did you investigate the availability attributes option at all? And if so what did you learn? If not, no worries.
  • Seems like a good way to do it, but I'm not sure how one would add support for a new platform.
• How did you deal with the case where .so files themselves might not be available to load? If this didn't come up, that's equally interesting.

Again, thanks so much. Helps a ton to get to draw on your learnings.

[enh-google]

Did you investigate the availability attributes option at all? And if so what did you learn? If not, no worries.

this is something we're working on in the OS right now. once we've gotten the NDK off libgcc we'll be able to make it available in the NDK too.

[cpsauer]

Ooh, that's awesome to hear. @enh-google, just to confirm, that means an official fix to #837 is coming down the line? Do you have a rough sense of when? We'd love to use an official solution.

[DanAlbert]

r23 is going to have a #define that you can set to get largely untested, undocumented, not recommended weak symbols for the NDK. I wouldn't call that official, but it'll exist.

[camillol]

@cpsauer you can see an example in MediaPipe of how it can be done for OpenGL, whose headers include a GL_APICALL macro that can be overridden. __INTRODUCED_IN is not currently overridable (it's defined unconditionally), so you'd have to use a different way to get the compiler to see the declaration you want. There's a number of options, can't comment on what works best.

Regarding what happens if the .so is entirely missing, the entire library can also be marked as a weak import, in which case you can run even if it can't be located. I think the linker does that automatically when all the symbols imported from it are weak, but it's possible that you have to tell it separately.

[cpsauer]

Thanks so much, all.

[enh-google]

Regarding what happens if the .so is entirely missing, the entire library can also be marked as a weak import, in which case you can run even if it can't be located.

really? i've never seen that, and i don't think we implement that if it does exist. do you have a reference for this? (the lack of this is one of the things that makes this way of working less useful than it might otherwise be, since the existence of a DT_NEEDED can mean [if the whole library is new, not just one API] that your library won't load on older OS versions.)

[cpsauer]

@camillol, to settle the issue above about the behavior when the .so is missing: Any chance you've got the ability to test, a memory of testing it, or any recollection of docs about it? If the Android loader did (like iOS) have the behavior you're saying, then that'd be game-changingly useful.