Psycopg2 error after the apk installation.

[ghost]

I got this error while debugging the android apk. I associate this to Buildozer because I specified into the buildozer.spec requirements the psycopg2 library. It means that is not working.

How do I fix it? I know that is a recipe for psycopg2 here: https://github.com/kivy/python-for-android/blob/master/pythonforandroid/recipes/psycopg2/__init__.py

How can I add this recipe to my project, to buildozer compile it successfully ?

[AndreMiras]

Usually you should share as much info as possible. e.g. your buildozer.spec your main.py code, your build log...
But yes it's more like a p4a related things to me. Have you tried extracting the APK to see what's in the site-packages?

[AndreMiras]

I could reproduce the issue, after extracting the APK it turns out that psycopg is here, but in the wrong location _python_bundle/site-packages/lib/python2.7/site-packages/psycopg2/_psycopg.so
It's then obvious looking at the recipe code that the python path is hardcoded and it should not.
@Piontk you can try your luck contributing the p4a by editing the recipe to fix it.
See here https://github.com/kivy/python-for-android/blob/9465952/pythonforandroid/recipes/psycopg2/__init__.py#L44 the site-package is hardcoded while it should be more generic to the Python version.

[ghost]

I extracted the APK and inside the lib directory (the "site-packages" directory do not exist) I could note that all the libraries where packaged, except the psycopg2. I'm using python 3 to run my app, it could be a problem?

[AndreMiras]

You need to extract two things, first unzip the .apk then tar xf assets/private.mp3
Probably just chaning --install-lib=lib/python2.7/site-packages to --install-lib=. in the recipe should be enough actually.
When you change it, you may need to buildozer android p4a -- clean_recipe_build psycopg2 before rebuilding

[ghost]

After running the tar xf assets/private.mp3 command line I got it. Seems that psycopg2 was packaged, but my app can't recognize it.

[AndreMiras]

Yes it's because the path is wrong, see it says _python_bundle/site-packages/lib/python2.7/site-package/psycopg2/. It's because the --install-lib=lib/python2.7/site-packages arg is wrong

[ghost]

Ok, so to fix it I have to edit the init.py, inside the psycopg2 recipe directory. I tried to edit it to: --install-lib=. as you said in a older comment. I had the same error on android debugging. After I tried to edit to --install_lib=_python_bundle/site-packages, but when I ran the tar xf assets/private.mp3 command the result is the same. The psycopg2 is located at _python_bundle/site-packages/lib/python2.7/site-packages/psycopg2

P.S: After every change I made I ran the buildozer android p4a -- clean_recipe_build psycopg2 before the buildozer android debug command

[ghost]

After a lot of attemptings I have found the error: I had specified a directory for customs recipes, and I was editting the wrong psycopg2 init.py file. After edit the right one I had this error message:

[AndreMiras]

Great yes that new error is way better!
By the way when you're sharing errors, I think it's better not to share the screenshot, but the error text (using triple backquotes).
So now it seems like the module compiles and is loaded, but we're missing some symbols. I'm not sure yet why, but that's something to dig

[AndreMiras]

Digging into it things feel more clear, it actually is a "feature" of the Android NDK.
When I look into ~/.buildozer/android/platform/android-ndk-r17c/sysroot/usr/include/langinfo.h I see the following:

#if __ANDROID_API__ >= 26
char* nl_langinfo(nl_item __item) __INTRODUCED_IN(26);
char* nl_langinfo_l(nl_item __item, locale_t __l) __INTRODUCED_IN(26);
#endif /* __ANDROID_API__ >= 26 */

So basically the header is available and will work at compile time, but the definition is not available if your ANDROID_API is below 26, hence you get undefined symbols runtime errors.
So I see two ways around it. One would be to try with higher API (ndk-api I guess), the other way would be to patch psycopg2 to skip nl_langinfo calls

[AndreMiras]

I gave it a try and it worked. I compiled from Docker with:

python setup_testapp_psycopg2.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --ndk-api 26 --bootstrap sdl2 --requirements python3,kivy,psycopg2

I used the following setup_testapp_psycopg2.py file:

from distutils.core import setup
from setuptools import find_packages

options = {'apk': {'requirements': 'python3,sdl2,kivy,psycopg2',
                   'android-api': 27,
                   'ndk-api': 21,
                   'dist-name': 'bdisttest_python3_psycopg2',
                   'ndk-version': '10.3.2',
                   'permission': 'VIBRATE',
                   }}

package_data = {'': ['*.py',
                     '*.png']
                }

packages = find_packages()
print('packages are', packages)

setup(
    name='testapp_python3_psycopg2',
    version='1.1',
    description='p4a setup.py test',
    author='Alexander Taylor',
    author_email='alexanderjohntaylor@gmail.com',
    packages=find_packages(),
    options=options,
    package_data={'testapp': ['*.py', '*.png']}
)

And the following testapp/main.py :

print('main.py was successfully called')
print('importing psycopg2')
import psycopg2
print('psycopg2: {}'.format(psycopg2))
print('psycopg2.__version__: {}'.format(psycopg2.__version__))
print('psycopg2._psycopg: {}'.format(psycopg2._psycopg))
print('DONE!')

See below relevant part of the adb trace:

02-20 23:47:05.240 15843 15865 I python  : AND: Ran string
02-20 23:47:05.240 15843 15865 I python  : Run user program, change dir and execute entrypoint
02-20 23:47:05.240 15843 15865 I python  : main.py was successfully called
02-20 23:47:05.240 15843 15865 I python  : importing psycopg2
02-20 23:47:05.390 15843 15865 I python  : psycopg2: <module 'psycopg2' from '/data/user/0/org.test.testapp_python3_psycopg2/files/app/_python_bundle/site-packages/psycopg2/__init__.pyc'>
02-20 23:47:05.390 15843 15865 I python  : psycopg2.__version__: 2.7.7 (dt dec pq3 ext lo64)
02-20 23:47:05.390 15843 15865 I python  : psycopg2._psycopg: <module 'psycopg2._psycopg' from '/data/user/0/org.test.testapp_python3_psycopg2/files/app/_python_bundle/site-packages/psycopg2/_psycopg.so'>
02-20 23:47:05.391 15843 15865 I python  : DONE!
02-20 23:47:05.391 15843 15865 I python  : Python for android ended.

There's a PR ready here #1723

[ghost]

The problem is: I need to install this app in a device that has an ANDROID_API = 22, so it won't be able to find the nl_langinfo. You mention that I can patch the psycopg2 library to avoid the nl_langinfo calls. How exactly can I do that? (I'm novice at this kind of operation).

I'm reading about and seems that the problem is inside the _psycopg.so file. The point is: how do I acess the code inside this .so file?

[AndreMiras]

The _psycopg.so file is compiled thanks to the psycopg2 recipe. So the idea would be to patch the source code before compiling.
Another way might be to try your luck with ac_cv_header_ in order to completely disable the langinfo header. Something like ac_cv_header_langinfo_h=no at configure time. https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Generic-Headers.html
For instance cpython implementation is relying on this macro https://github.com/python/cpython/blob/v3.7.2/configure.ac#L2153 and that's how we disabled it. I'm not sure about psycopg2, but that's the idea

[ghost]

So I need to change this configure.ac from cpython, trying to eliminate the langinfo.h references and try to compile ?

Edit: I found the configure.ac file into my VM. Now I need to found a command that makes the autoconf ignore the langinfo.h?