Initial migration to NDK r19 (Part I - The core)

[opacam]

The time has come to move into ndk r19, there is big changes in android's ndk r19 which will make live easier at compile time because (extracted from android docs: https://developer.android.com/ndk/guides/other_build_systems):

As of NDK r19, the toolchains installed by default with the NDK may be used in-place. The make_standalone_toolchain.py script is no longer needed for interfacing with arbitrary build systems.

So, this implies that we can remove some cflags that we were using (isystem, sysroot, gcc-toolchain...). I think that this is good for python-for-android maintainability....but we will loose the gcc compiler because has been deprecated in android's ndk (partially removed in ndk r18 as far as I know), so this will mean the beginning of the end of python2legacy recipe (openssl will not work anymore for python2legacy, pygame will be removed soon, python2legacy needs the gcc compiler...)

Things done here:

• Rework of the arch environment to work with ndk r19+
  • Modifies target so we can use the toolchains in-place (as if we were using a standalone toolchain)
  • Removes some cflags that we don't need anymore
  • Move macros and includes from CFLAGS to CPPFLAGS (because this is the way it's supposed to be)
  • Move libraries from LDFLAGS to LDLIBS (again, the same as above)
  • Remove hardcoded cpu count in MAKE
  • Remove gcc of environment (because it has been partially removed in ndk r18)
  • Group environment variables and add some comments (to put some order and make easier for us to maintain it)
• Adapt python2 and python3 recipes to new clang triplet (requires an android ndk version >= r19)
• Add libz recipe Read android's ndk zlib version and add flags to python's recipe env, to solve build problems with python's zlib module (I think that this could also solve the zip/unzip issues reported in issue Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file #1388)
• Update docker files to use ndk r19b
• Fix numpy for ndk r19
• Update documentation
• Update test_archs and test_bootstrap

Those points it will be done later in another prs:

• Remove pygame bootstrap (already started by @inclement in Delete the pygame bootstrap #1670)
• Remove python2legacy (we introduced python2legacy recipe in order to made it work the pygame's bootstrap...without pygame...there is no point to maintain it..plus it won't get built with ndk r19)
• Fix a minimal set of recipes that has cflags that we don't need anymore in ndk r19 (and may cause build failures...I noticed that issue for recipes: cffi, pycrypto, pymunk...and probably more)
• Fix recipes that uses gnustl_shared (gnustl_shared has been removed in ndk r18...so all recipes that uses gnustl_shared will fail).

I've test this with arch armeabi-v7a and p4a's testapp of sqlite3/openssl and works as expected 😄

Note: sorry @AndreMiras, I know that you don't like those long prs...I already have more work done regarding ndk r19 that will go into another prs, this is only the core 😉

As a side note: I'm wondering, could this fix the python's linkage problems that we got in Mac Os #1647?

I leave this in WIP status because:

• We need to discuss some points of this pr
• We should test this with more recipes because the changes made in here affects the build of all the ecosystem, so...we shouldn't merge this, at least until we all are sure that it is the way to go

Resolves: #1613 and #1525

[opacam]

This change causes the red X in travis...I modified the recipe locally and it builded and worked fine at runtime... but even with those changes travis still fails. So no changes applied here to fix this recipe...@AndreMiras, maybe we should add pycrypto to BROKEN RECIPES in this pr?

Also see: https://travis-ci.org/kivy/python-for-android/jobs/495925991#L5227-L5247

[AndreMiras]

Yes please do so then 👍

[opacam]

@inclement, it seems that we finally can remove this, but why it happened?...your comment about sysroot and LDFLAGS may be the cause of this...anyway...ndk r19 solves this issue 😄...so I removed it

[inclement]

I think I discovered more recently that the crtbegin_so.o and crtend_so.o issues were due to an NDK bug when using clang as part of the build process, so it makes sense that this is fixed now. The other stuff I never went back to, but certainly happy if it's working now :p

[opacam]

@KeyWeeUsr, I was thinking with you when I changed this line 😉, we still have pending work related to this (#1558)...

Note: I put this in here as a reminder

[opacam]

Aaa..I think that I forgot to mention this change in the commit...anyway...I think that we better use this than the sh.cp command, don't you think?

[opacam]

One last thing, I suspect that some administrators that I not mentioned in the comments, may be interested in this pull request, so I mention it here, to make him know about this: @Jonast, @hackalog, @rnixx and @tito

Note: sorry if I exclude someone, it's not my intention, everyone is welcome , only mentioned the ones I think that could be interested 😉

[ghost]

@opacam nice work, I love the getting rid of extra options like --sysroot a lot.

While we're speaking about getting rid of options, something else that is vaguely related:

python-for-android/pythonforandroid/recipe.py

Line 1058 in 9313e33

def get_recipe_env(self, arch, with_flags_in_cc=True):

I strongly think this entire function really should be blank. The CythonRecipe seems to exist 99% because the base setup is just wrong, we should fix that. If you're interested maybe this is something that might fit into your work as you poke around in these parts, otherwise I will look into it soon

(This would also allow me to remove an ugly reference to the CythonRecipe I made here because the base setup is wrong:

python-for-android/pythonforandroid/build.py

Line 633 in 9313e33

standard_recipe = CythonRecipe()

This should not be required if things were set up correctly, sadly right now they don't appear to be)

This is just a fun note for additional future things, I'm not saying anything is missing from your pull request here 👍

[inclement]

Thanks @opacam. I won't have time to look at this in detail until the weekend probably.

[pax0r]

Adapt python2 and python3 recipes to new clang triplet (requires a minimal ndk version of r19)

would this mean that kivy application will work only on r19 devices?

[opacam]

Maybe I forgot the android word in there...so:

the python2 and python3 recipes will require an android ndk version >= r19

Note: also changed in first post, thanks @pax0r

As a side note: Only affects our build tools, the minimum and target api of the device it will remain as it is right now (minimun ndk-api of 21 and an android-api equal ot greater than that)

[pax0r]

Maybe I forgot the android word in there...so:

sorry my fault, I mistook ndk for api level

[opacam]

Update:

• Rebased onto top of the current develop branch (85207a0)
• Fixed merge conflicts
• Adapted test_archs to new ndk-r19 build system
• Tested on device with testapp (openssl/sqlite3) for arch: armeabi-v7a and arm64-v8a (this last one with numpy)
• Reworked some touched lines to be more readable (archs.py and python.py)
• removed a couple of commits that weren't essential to make it work the changes of this pr (to be easier to review when the time comes):
  • remove of openssl legacy libraries (Remove legacy version of openssl #1870)
  • a modification to ci.constants which it seems that we don't need to successfully pass the ci tests

[inclement]

Thanks for updating this @opacam, I think now is a good time to move towards merging it.

The changes look solid to me on a quick review. I might have a few comments when I have time to look more closely, but nothing major I expect.

The real problem with this will be how we minimise user issues, since this will break just about everybody's build (I think?). We need to somehow:

• clearly signal to the user when the build fails due to NDK version being too old
• update buildozer so it knows it needs to download a new NDK (and configure it to do so

I don't have a clear view about how to best do these, although both are certainly possible. Any thoughts?

[ghost]

So does this pull request make r19 work, but at the same time also break the older NDK version that works with the current latest python-for-android release? Or what exactly are the results of this getting merged?

That would be in so far a bit problematic that when e.g. providing docker images (like via p4a-build-spaces) they would now need to be tied be to some specific p4a version so things work. Is it at all feasible to support a new NDK and also the old one for at least one release or something?

[opacam]

@inclement,

clearly signal to the user when the build fails due to NDK version being too old

Oh yes, totally agree, I took a look in our code and actually we already check the ndk version (I think that you did that part of the code yourself in recommendations.py). So I think that we should modify it to raise an BuildInterruptingException instead of only checking the ndk, I already wrote some lines about this and I willl create a separate pr so we can discuss it in there, and this way, we will avoid the addition of more code in here... which would make more difficult to review this pr...unless you guys prefer that I do it in here...

update buildozer so it knows it needs to download a new NDK (and configure it to do so

I must admit, I didn't though about buildozer (nor p4a-build-spaces), also I don't know the internals of this project, I don't usually touch it, I use an outdated version of buildozer and I manually configure all the ndk/sdk paths manually in spec file...so changing ndk it's not a problem for me, but you are right this should be done. Probably we could use the modifications I did in dockerfiles to use as a model to update buildozer. Anyone can do this? If not, I can take a look at it...but I would really prefer that someone with more knowledge of buildozer perform this tasks...mostly because it would do it faster than me.

[opacam]

@Jonast,

...but at the same time also break the older NDK version that works with the current latest python-for-android release?

The thing is that the compilation method we introduce in here, is introduced in android's ndk-r19, so yes, older versions of ndk it can't work. I don't thing that this is a big deal as far as we make sure that the user have a clear message as @inclement pointed. Besides, why do you want to use an older version of ndk? I really prefer to use an updated one...which probably fix issues detected in older versions. And moreover, I don't think that you can success in building any apk with the current p4a and an ndk != 17...so for me it's only an dependency upgrade which should enhance the current situation (I still not tested but probably ndk-r20 it will also be usable...so we could use two ndk versions instead of one 😉).

Or what exactly are the results of this getting merged?

I cannot tell you the exactly result, because I don't know, what I can tell you is:

• Any recipe that uses stl library it won't work (because 'gnustl_shared has been removed in ndk 19)...this is not a problem, I already have some wrotten code to face this, but I still not pushed it... and that should be properly tested because I only make sure that the recipes correctly builds with different archs
• Some of the recipes should be updated because has some flags that will cause conflicts...I detected this problem in: cffi, pycrypto, pymunk, boost, libtorrent and some others (I also solved this in those mentioned recipes plus some others that I don't mention..but we could have other recipes with this problem...)
• I tested the testapp_pillow and it seems to work, so no graphic library issues (libpng, libjpeg, libfreetype and libharfbuzz seems to build fine and the app works at runtime)
• once merged, it will simplify the compilation because the new build system takes care of setting some of the needed compilation flags...so less code on our side...which will make the maintenance easier 😄

As a side note: I have a branch with all the changes (the ones from here and the ones I mentioned that I not pushed and I calculate that there is around 30 commits, so once discounted the commits from this pr, probably almost each of this commits could be a pr)

[opacam]

How could we organize us with changes that cannot be done with a few prs...like this ndk-r19 pr?

We had some talk with @AndreMiras about how to face this kind of big changes and the idea of creating an upstream branch appeared. Maybe we could create it and all the prs related to ndk-r19 should be merged in there, so we (and users) can test it without breaking develop or master branches. Once we are happy with the status of this we could safely merged into develop. Of course that will add an extra level of complexity which I'm not sure if we should introduce, but maybe we could use it sporadically to deal with big changes that cannot be done with a few commits, like the ones we are talking about.

Apart from this...It would be great if we can coordinate things between us with this kind of big changes ...so I'm thinking that maybe we could use the github feature projects.In there we could put all the known tasks to do for this ndk-r19 migration (even those ones that are external but related with the ndk-r19 thing, like buildozer or p4a-build-spaces that we talked in here). So this way, we would have a global view of the status of the project, so once we got all the known things to do solved, we will know that we can safely merge it into develop. To see what I'm talking about see a completed project in kivy projects

If all of you like the idea, I can create this project and put in there all the things that I think that should be done and also all of you can put in there all the things that you think that we should do...I think that will improve our work flow with this kind of stuff...

Also I would like to know the opinion of @tito and @tshirtman of this projects thing...so...it was positive for you guys this github feature? do you think that it will improve our workflow for this ndk-r19 migration?

Note: I suppose that I have permissions to do this project thing...the button to create a new project is in there... Regarding the new branch creation, this should be done by @inclement, @AndreMiras or any owner

As a side note: In order to all of us knows what issue/pr is related with this ndk-r19 migration I created a Label named ndk-r19...you all already probably seen that

So guys, what do you thing about all this thoughs?

[ghost]

besides, why do you want to use an older version of ndk?

The problem is that many users will either use the latest stable of p4a, or the develop version. A docker image can only install & preconfigure one version of the NDK at a time. So far, p4a-build-spaces always works no matter whether you pick the latest p4a stable or develop, as long as you keep it up-to-date - if things break, just update p4a-build-spaces and p4a to latest stable/update or to latest develop and it usually works again. This is impossible to maintain if the latest stable and develop don't have a common NDK version, so now for one group of the users things will end up broken.

There is a solution for this of course, which is that I make p4a-build-versions check which version of p4a ends up installed and what version of the NDK it needs. That is completely doable, but it also isn't trivial. That is why it would help a lot if it could be done in such a way that there's always an NDK version that works both with latest stable and whatever is the current develop, it just simplifies providing a build environment that works for the majority of users

But I can really work around this if there is no common NDK version, so if that's not doable then I'll make it work somehow anyway so don't worry

Edit: summed up, just let me know if it's doable to keep a common NDK version with reasonable effort, and if it isn't I'll sit down and make the changes in p4a-build-spaces to deal with it

can test it without breaking develop

I say, it's fine risking to break develop. I don't think it should be intentionally broken but if you keep the pull requests such that it's likely to stay in a somehow usable state but things occasionally go wrong for someone I don't mind. After all, that's why we have develop now separate from master, this is after all a dev version with all the risks & breakages that come with it

So I'd only make a separate branch if you already know after this pull request things will definitely end up broken for a while, that'd be a bit bad. But if it's likely to work just merge it, and we'll see what happens 😄 and deal with it if it happens to not work

[opacam]

@Jonast, I'm afraid that make this compatible with older NDKs it's complicated because we need to touch several core files as well as recipes, this will require to maintain two different versions of the compile flags depending on the used NDK which will lead to a code more difficult to maintain and also we will loose one of the benefits of this pr which will be that simplifies the compilation ... IMHO, I don't think that it's worth it the effort.

About the branch creation, you are right, probably there is no need to create a new one, at least, I do not expect that this pr breaks the develop branch in such that way that we have the need to create this new branch, but there will be some recipes issues. The good news is that I have more work in my computer already completed that I want to push in separate prs ...so we can mitigate possible complains that we could have.

[AndreMiras]

Yes I was also wondering, is CrystaX still supported/working?
It was dropped from the CI and I don't think any of the p4a maintainers would test things against CrystaX or even try to fix if it's not working.
So yes I would not bother adding theses lines, however I would not mix concerns too much would not address killing CrystaX here either.
Does this sound reasonable?

[opacam]

Yes I was also wondering, is CrystaX still supported/working?

I don't know if `crystax is still working, probably not...

I would not mix concerns too much would not address killing CrystaX here either.
Does this sound reasonable?

Yes, I totally agree with you, I would keep this as simple as we can and I would do the killing CrystaX in another pr

[opacam]

Yeah, just what I though...os x fails when compilng kivy with error:

/Users/travis/.android/android-ndk-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: build/temp.macosx-10.14-x86_64-3.7/kivy/_event.o: requires unsupported dynamic reloc R_ARM_REL32; recompile with -fPIC
clang: error: linker command failed with exit code 1 (use -v to see invocation)

See at: https://travis-ci.org/kivy/python-for-android/jobs/576204088#L1105

Note: -fPIC in cflags solves the problem

[opacam]

Ok, added a couple of commits, tests are fine now

the first commit is the libpng fix, I added it because:

• @AndreMiras can run the app test again (just be sure that you don't use any recipe dependant on gnustl_shared or it will fail...)
• we already talked about the libpng error in here...and it does not add any complexity at this point, since we are removing code, but we can deal with libpng later, so let me know and I will remove that commit, no problem 😉

The last commit..well...it does the job but I have my concerns...maybe it would be better to attack the problem per recipe/OS...you tell me guys

[AndreMiras]

Yes I confirm png is passing, well done. Now cffi is failing with the same error. But yes I'd say let's fix them one by one later.
Here is the error for records:

[INFO]:    Building cffi for armeabi-v7a
[INFO]:    cffi apparently isn't already in site-packages
[INFO]:    Building compiled components in cffi
[INFO]:    -> directory context /home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    -> running python3 setup.py clean --all
[INFO]:    -> running python3 setup.py build_ext -v
Exception in thread background thread for pid 26829:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 2459, in background_thread
    handle_exit_code(exit_code)
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 2157, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1:

  RAN: /home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python3 setup.py build_ext -v

  STDOUT:
running build_ext
building '_cffi_backend' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/c
/usr/bin/ccache /home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_
21/python3/Include -DANDROID -D__ANDROID_API__=21 -I/home/andre/.buildozer/android/platform/android-ndk-r19b/sysroot/usr/include/arm-linux-androideabi -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/python-installs/pywallet/include/python3.7 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/Include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
c/_cffi_backend.c:4197:31: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        *p_printable_filename = PyText_AsUTF8(s);
                              ^ ~~~~~~~~~~~~~~~~
c/_cffi_backend.c:5911:9: warning: 'ffi_prep_closure' is deprecated [-Wdeprecated-declarations]
    if (ffi_prep_closure(closure, &cif_descr->cif,
        ^
/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include/ffi.h:339:18: note: 'ffi_prep_closure' has been explicitly marked deprecated here
  __attribute__((deprecated))
                 ^
2 warnings generated.
creating build/lib.linux-x86_64-3.7
/usr/bin/ccache /home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/arme
abi-v7a__ndk_target_21/python3/Include -L/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/libs_collections/pywallet/armeabi-v7a -L/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/bootstrap_builds/sdl2-python3/libs/armeabi-v7a -L/home/andre/.buildozer/android/platform/android-ndk-r19b/platforms/android-21/arch-arm/usr/lib --sysroot=/home/andre/.buildozer/android/platform/android-ndk-r19b/platforms/android-21/arch-arm -L/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/android-build -lpython3.7m -target armv
7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Include -DANDROID -D__ANDROID_API__=21 -I/home/andre/.buildozer/android/platform/android-ndk-r19b/sysroot/usr/include/arm-linux-androideabi -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/python-installs/pywallet/include/python3.7 build/temp
.linux-x86_64-3.7/c/_cffi_backend.o -lffi -o build/lib.linux-x86_64-3.7/_cffi_backend.cpython-37m-x86_64-linux-gnu.so
/home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lunwind
/home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lunwind
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/ccache' failed with exit status 1


  STDERR:


Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 1192, in <module>
    main()
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/entrypoints.py", line 18, in main
    ToolchainCL()
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 668, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 153, in wrapper_func
    build_dist_from_args(ctx, dist, args)
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 206, in build_dist_from_args
    args, "ignore_setup_py", False
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/build.py", line 577, in build_recipes
    recipe.build_arch(arch)
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/recipe.py", line 893, in build_arch
    self.build_compiled_components(arch)
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/recipe.py", line 905, in build_compiled_components
    _env=env, *self.setup_extra_args)
  File "/home/andre/workspace/PyWallet/.buildozer/android/platform/python-for-android/pythonforandroid/logger.py", line 180, in shprint
    for line in output:
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 863, in next
    self.wait()
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/andre/.local/lib/python3.6/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1:

  RAN: /home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python3 setup.py build_ext -v

  STDOUT:
running build_ext
building '_cffi_backend' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/c
/usr/bin/ccache /home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_
21/python3/Include -DANDROID -D__ANDROID_API__=21 -I/home/andre/.buildozer/android/platform/android-ndk-r19b/sysroot/usr/include/arm-linux-androideabi -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/python-installs/pywallet/include/python3.7 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/Include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
c/_cffi_backend.c:4197:31: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        *p_printable_filename = PyText_AsUTF8(s);
                              ^ ~~~~~~~~~~~~~~~~
c/_cffi_backend.c:5911:9: warning: 'ffi_prep_closure' is deprecated [-Wdeprecated-declarations]
    if (ffi_prep_closure(closure, &cif_descr->cif,
        ^
/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include/ffi.h:339:18: note: 'ffi_prep_closure' has been explicitly marked deprecated here
  __attribute__((deprecated))
                 ^
2 warnings generated.
creating build/lib.linux-x86_64-3.7
/usr/bin/ccache /home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -target armv7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/arme
abi-v7a__ndk_target_21/python3/Include -L/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/libs_collections/pywallet/armeabi-v7a -L/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/bootstrap_builds/sdl2-python3/libs/armeabi-v7a -L/home/andre/.buildozer/android/platform/android-ndk-r19b/platforms/android-21/arch-arm/usr/lib --sysroot=/home/andre/.buildozer/android/platform/android-ndk-r19b/platforms/android-21/arch-arm -L/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/android-build -lpython3.7m -target armv
7a-linux-androideabi21 -fomit-frame-pointer -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Include -DANDROID -D__ANDROID_API__=21 -I/home/andre/.buildozer/android/platform/android-ndk-r19b/sysroot/usr/include/arm-linux-androideabi -I/home/andre/workspace/PyWallet/.buildozer/android/platform/build/build/python-installs/pywallet/include/python3.7 build/temp
.linux-x86_64-3.7/c/_cffi_backend.o -lffi -o build/lib.linux-x86_64-3.7/_cffi_backend.cpython-37m-x86_64-linux-gnu.so
/home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lunwind
/home/andre/.buildozer/android/platform/android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lunwind
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/ccache' failed with exit status 1

[AndreMiras]

What a revamp, congrats!
Let's merge it before it grows any bigger

[opacam]

@AndreMiras, thanks for the approval 😄

...but I will not push the merge button of this one unless @inclement tell me to do so...IMHO, this needs his approval 😉

BTW the second part is already prepared to be reviewed #1944, with the changes requested applied (the third part needs to be adapted to the changes made in the second part).

[inclement]

Thanks @opacam :)

[opacam]

Ooooo, 🎆🎉😂...taking a 🍺 right now to celebrate 🎉 ...this has been a long road...

¡¡¡Thanks to all for your reviews, comments and patience with this PR 😄!!!

Notes:

• @AndreMiras, the last cffi error is already mentioned in the initial post alongsidepycrypto and pymunk...I also forgot that fact sorry...btw...we should take a 🍺 someday soon, that is easy for us 😄 (others invited too but I have the feeling that it will be more difficult...)
• I may have some solutions to those recipes mentioned above, have to check because We started this some time ago. What I'm sure is that I have more work related with the stl lib refactor that we make in the third part (still needs update...tomorrow I will try to put some order and push the remaining things I have)
• @jtc0de , I'm thinking in you, hope that your p4a-build-spaces repo is not being to much affected with this PR merged in develop...also thinking in your sdl2 upgrade...I have the feeling that we will get there soon...at least I hope so... 💪
• @inclement, we will discuss this buildozer issue later if you don't mind, thanks to clarify your concerns in that issue and also thank you to merge this PR!!! 😄

[CikoXp]

Nuovo documento di testo.txt

Hi guys.. i'm getting trouble with the above cffi error during my apk build process.
Buildozer give this error but only if i iclude "cryptography" as requirement.
My app is based on "paramiko" that need "cryptography" to work properly.
Trying to buid the apk without "cryptography" (so only python3,kivy,paramiko) it works but apk crashes on android.
With adb logcat i discovered the "cryptography" dependancie for "paramiko" but it can't include it because the cffi error... i'm stuck... can anyone help pls?
Thnx at all anyway.

[opacam]

You probably are using latest p4a release or our master branch, try with our develop branch since this should have been fixed by #2130 and #2131 (still not merged in master branch). And please...the next time, open a new issue or ask for help in our support channels at discord, pull requests are supposed to be used for discuss/review the code changes, not for posting issues or asking for help...:wink:

[CikoXp]

Wow it's incredible what we can learn simply reading with enough attention.

A moment before your reply I've figured out that i'm using master branch instead develop.. simply checking my .buoldozer folder in pythonforandroid recipes my cffi init.py says "versoin 1.11.5"... something is wrong... and checking my buildozer.specs file i discovered the p4a branch pointed to master. simply pointed to develop and started a new clean buildozer process... i'm waiting for the result.

I'm sorry for the help req. in pull thread but after hours of google search i founded here my exact cffi error and replyed without starting my brain. Sorry again.. next time i'll try to think before write.