Comprehensive list of broken python3 recipes #1514
Comments
|
sqlalchemy is broken with python 3.7.1 cwd is /home/strubbi77/.local/share/python-for-android/build/other_builds/sqlalchemy-python3-python3/armeabi-v7a__ndk_target_21/sqlalchemy Traceback (most recent call last): |
ghost
mentioned this issue
|
[Previously posted in #1576 , suggested more appropriate here] I tried to build all the user modules for python3, kivy. Of course a successful The following dependencies are required in addition to those for a simple .py build I use python-for-android (master, installed 1/12/2019) The following modules successfully build an apk: BUT, of these, the following fail with error ".. _ctypes.."" Which confirms that order can be important. The following do not build because they have no dependency on 'python3', but The following does not build because of a dependency I can't identify: The following does not build because: The following does not build because of cmake errors The following does not build because (seems strange): The following does not build because has dependency on 'incremental', 'constantly' The following does not build because NDK v18 needed |
|
Next I looked in the .apk for .so files in addition to the default set of .so EDIT: some are in 'site-packages' directory, I was not looking there. Pillow NONE |
|
A little more detail.... Build environment:
.p4a Build command:
These respectively contain: Pillow :
Lots of reasons here, not clear to me I can correctly diagnose, but I see things like:
Build Log Files: |
|
There must be something extra going on here, your list of silent fails contains several modules that I build all the time and know to work. It's completely possible that there's some issue that causes them to not be built in some circumstances that don't match mine, which could well be a bug, but I also don't think I've ever seen other people hit issues with them. I'm thinking in particular of sdl2_image, sdl2_mixer, sdl2_ttf and pyjnius, which are core Kivy dependencies and so included in almost every apk anyone has ever built with p4a (a bit less so recently :) ), though I'd also strongly expect jedi, six and pysdl2 to work. I'll try to replicate your method at some point to see what happens. |
|
I can confirm pysdl2, sdl2_image, sdl2_ttf, pyjnius, png if that's used through PIL (pretty sure it is), jpeg (again through PIL), and libxml2 (used through lxml/python-docx) work for me, as in I actually used them to do things on Android. |
|
@Ham-Merhead You keep mentioning extra .so files, are you aware that several of these recipes are not expected to build include any? In particular, python modules like pysdl2 and jedi. Also, none of your "Contains object files" examples mention the libs directory which is where the object files end up for most recipes (e.g. sdl2_image leads to |
|
@inclement
Yes, I speculate on that for enum34 [stdlib contains enum, is that the same thing?], Where do I find that 'not expected' information? And if a requirement doesn't add anything to the apk, is it a requirement?
My editing error, those 3 ['sdl2_image', 'sdl2_mixer', 'sdl2_ttf'] I was not automatically looking in 'lib/armeabi-v7a', I've rerun the failed leveldb : Also I see 'ipaddress' is in stdlib, perhaps this is never a requirement?
Is 'PIL' built in? Where is it?
Yes, the best thing would be automated testing on Android; presumably via adb. 'libxml2' is really interesting, because init.py builds the '.la' but The object files I still can't find are: EDIT: Update I just found the ffmpeg files in lib, but not named ffmpeg, no others with this issue. |
It needs to be hardcoded for each recipe, they all invoke different build systems so without knowing how it's supposed to looked like afterwards, there is no automated way to determine whether they would normally be expected to create native code, or python code, or whatever else.
That is a misunderstanding, they all add something to the apk. It's just not always native code (no |
|
@Jonast Thanks for the help, I'm getting there slowly. Have found some .py and additional .so The one that really bugs me is static libraries '.a'. I can see them (mostly) being built and copied |
|
@Ham-Merhead regarding static libraries, that is normal/expected, since static libraries are by nature compiled statically into the app main binary. This means that 1. they are only useful with a compiler and not at runtime and hence the So the most direct effect without closer binary examination should only be that Edit: also yes that means it is pretty much impossible to check them. I would just put these recipes on a separate list that can't be automatically verified. The only way to do it practically that I can think of is write a small test program that actually tries to use them and runs in an emulator as part of the test, but of course that would be a lot more effort than just peeking into the |
|
Thanks again. |
|
@Ham-Merhead thanks for looking into this 👍 helping out with questions is the least I can do |
|
@Jonast These fail in Android with:
With the exception of 'snappy' all build .a files. Bug Rumor Example: .p4a |
That is expected, none of these are Python modules.
Does the size of the APK change, or more specifically the size of any of the other included .so files? I don't think all of these would be linked with libmain.so, but maybe with sdl2_image or other. |
|
You are a genius: Just a matter of looking a little deeper ........ |
|
@inclement Also I did not mention earlier, but builds for So that is 10 recipes with no clear extra contents, and identical sized '_python_bundle' Of course this '_python_bundle' size is not generally the case, for example: EDIT: and the lib files are all the same size |
This really doesn't matter, not everything is in the _python_bundle folder. On quick inspection, this list includes things like freetype and harfbuzz which generate static object files that won't do anything unless something else uses them (maybe sdl2_ttf). Same for png ang jpeg, probably used by sdl2_image. The ffpyplayer_codecs are probably used only by ffpyplayer. It looks likely that most or all of these are building fine. |
|
OK, we agree this is mostly about static object files. But I do not see that these builds are copied to the apk. I can't find them in an apk, and I cant find the code that copies Where do you think these files are in the apk? This is not magic (though it might be obtuse), it is rules based, Btw, I previously referenced this 'lib' size data but didn't show it. We know the the .a/.o are not in "_python_bundle", or "lib" So where are they? |
|
The .a files are static object files. Their purpose is to be included in binaries compiled by other programs, not to be copied into the APK on their own. I haven't checked, but it wouldn't be surprising if some of these recipes are expected to do nothing else on their own and exist only to be used by others (e.g. the png static object files are probably used by sdl2_image). If you want to understand this, I recommend learning C, or at least reading about what these files are. A quick google returned https://stackoverflow.com/questions/30186256/what-is-the-difference-between-o-a-and-so-files as the type of question you may be interested in. |
|
Ah. The lights go on. A readelf on libSLD2_image.so shows in part a symbol from 'png'
Testable :) Though the way static files are used here, standalone testing is not that meaningful. Is there any way to tell from the I have had perhaps hundreds of thousands of lines of C under my fingers. So some inkling of the basics. |
|
The big picture Recipes looking good: Recipes with APK build error: Details: |
|
Errors like sqlalchemy : are easy to overcome, the dependencies in every file under recipes/packagename/_init.py depends = [('python2', 'python3crystax'), 'setuptools'] have to be changed to: Then probably other errors will occur |
Update of Recipes for test of broken recipes because of invalid dependency graphs, refs #1514
Also bumps few versions.
Removed gevent, cffi and pycryptodome recipes from it since they now compile fine. Also bumped few versions.
Removed gevent, cffi and pycryptodome recipes from it since they now compile fine. Also bumped few versions.
…pdate_broken_recipes_list_bis Updates broken recipes list, refs #1514
|
I've updated ci/constants.py narrowing down the |
|
Getting better. Results of my attempt to build the recipes follow. Recipes looking good: Recipes with no APK created: Test .p4a Test depends Test log from failed builds: |
|
For what it's worth, |
|
Awesome research @Ham-Merhead ! |
This means that the recipe, or its dependencies, pull in something incompatible with your other requirements. In most cases it's because something is written to depend on 'python2' or 'hostpython2'. |
|
@AndreMiras Thanks As @Jonast pointed out 'pil' is a case that will never occur and can be removed from the list. It seems I should be able to build the Equally interesting (to me) is what additional files a good build produces. A snapshot now might be valuable information in the future. This is an excuse to post the log of all tests: |
* update to add dependency on python3 to opencv recipe numpy/opencv fails with latest p4a at runtime (import) #1678 * Further recipe changes for python3 #1514 * Update __init__.py Removed unnecessary hostpython* * Update __init__.py Removed unnecessary hostpython* * Update __init__.py Removed unnecessary python* * Update __init__.py Removed unnecessary python* * Update __init__.py Removed conflicts because it was wrong * Update __init__.py
|
Update , p4a installed 1/9/2019 Recipes looking good: Recipes with APK build error: Fail log: |
|
Thanks for the update @Ham-Merhead ! |
|
@AndreMiras I was one page behind (hard to keep up...). Reinstalled p4a and all is good: pyleveldb : |
|
Even better, I reran the previous fails and 3 more successes in addition to pyleveldb: And the log: |
Just like it was done before for
python2andpython3crystax, we need a list of brokenpython3recipes #1402The text was updated successfully, but these errors were encountered: