NumPy | ImportError: dlopen failed: cannot locate symbol "log10f" referenced by "_multiarray_tests.so"

[electricalgorithm]

Checklist

• the issue is indeed a bug and not a support request
• issue doesn't already exist: https://github.com/kivy/python-for-android/issues
• I have a short, runnable example that reproduces the issue
• I reproduced the problem with the latest development version (p4a.branch = develop)
• I used the grave accent (aka backticks) to format code or logs when appropriated

Versions

• Python: 3.7.9
• OS: Ubuntu 20.04
• Kivy: 1.11.1
• Cython: 0.29.21
• OpenJDK: 1.8.0_282

Description

I built program's debug APK without any error and try to run it with logcat. I can see the first screen -login screen- and when I log in, there should be a livestream video source on the main screen. But there is not and the logcat outputs an error about the module I used for livestream code block. The other functionalities of the program are working. I have asked help on Discord, there were no response. You can see the mobile-application's codes here in my repo.

buildozer.spec and others

Command:

~$ buildozer android deploy run logcat

Spec file (deleted commented lines):

[app]

# (str) Title of your application
title = Invisible Defender

# (str) Package name
package.name = invdef

# (str) Package domain (needed for android/ios packaging)
package.domain = org.ballilab

# (str) Source code where the main.py live
source.dir = .

# (list) Source files to include (let empty to include all the files)
source.include_exts = 

# (list) Source files to exclude (let empty to not exclude anything)
source.exclude_exts = log, bak

# (str) Application versioning (method 2)
version.regex = __version__ = ['"](.*)['"]
version.filename = %(source.dir)s/main.py

# (list) Application requirements
requirements = python3, kivy, kivymd, appdirs, pycryptodome, distlib, filelock, importlib-metadata, six, zipp, numpy, idna, chardet, certifi


# (str) Supported orientation (one of landscape, sensorLandscape, portrait or all)
orientation = all


#
# OSX Specific
#
# author = © Copyright Info

# change the major version of python used by the app
osx.python_version = 3

# Kivy version to use
osx.kivy_version = 1.9.1


# Android specific

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 0

# (list) Permissions
android.permissions = INTERNET

android.accept_sdk_license = True

# (str) Android logcat filters to use
android.logcat_filters = *:S python:D

# (str) The Android arch to build for, choices: armeabi-v7a, arm64-v8a, x86, x86_64
android.arch = armeabi-v7a


# Python for android (p4a) specific

# (str) python-for-android branch to use, defaults to master
p4a.branch = develop


# iOS specific

# Alternately, specify the URL and branch of a git checkout:
ios.kivy_ios_url = https://github.com/kivy/kivy-ios
ios.kivy_ios_branch = master

# Another platform dependency: ios-deploy
# Uncomment to use a custom checkout
#ios.ios_deploy_dir = ../ios_deploy
# Or specify URL and branch
ios.ios_deploy_url = https://github.com/phonegap/ios-deploy
ios.ios_deploy_branch = 1.7.0

[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 2

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1

Logs

buildozer android deploy run logcat
# Check configuration tokens
# Ensure build layout
# Check configuration tokens
# Preparing build
# Check requirements for android
# Run 'dpkg --version'
# Cwd None
Debian 'dpkg' package management program version 1.19.7 (amd64).
This is free software; see the GNU General Public License version 2 or
later for copying conditions. There is NO warranty.
# Search for Git (git)
#  -> found at /usr/bin/git
# Search for Cython (cython)
#  -> found at /home/xyz49/Documents/invisible-defender-v2/bin/cython
# Search for Java compiler (javac)
#  -> found at /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
# Search for Java keytool (keytool)
#  -> found at /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool
# Install platform
# Run 'git config --get remote.origin.url'
# Cwd /home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/python-for-android
https://github.com/kivy/python-for-android.git
# Run 'git branch -vv'
# Cwd /home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/python-for-android
* develop ab65c485 [origin/develop] Merge pull request #2338 from vesellov/develop
# Run '/usr/bin/python3 -m pip install -q  \'appdirs\' \'colorama>=0.3.3\' \'jinja2\' \'six\' \'enum34; python_version<"3.4"\' \'sh>=1.10; sys_platform!="nt"\' \'pep517<0.7.0\' \'toml\''
# Cwd None
# Apache ANT found at /home/xyz49/.buildozer/android/platform/apache-ant-1.9.4
# Android SDK found at /home/xyz49/.buildozer/android/platform/android-sdk
# Recommended android's NDK version by p4a is: 19c
# Android NDK found at /home/xyz49/.buildozer/android/platform/android-ndk-r19c
# Check application requirements
# Check garden requirements
# Compile platform
# Run '/usr/bin/python3 -m pythonforandroid.toolchain create --dist_name=invdef --bootstrap=sdl2 --requirements=python3,kivy,kivymd,appdirs,pycryptodome,distlib,filelock,importlib-metadata,six,zipp,numpy,idna,chardet,certifi --arch armeabi-v7a --copy-libs --color=always --storage-dir="/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a" --ndk-api=21'
# Cwd /home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/python-for-android
[INFO]:    Will compile for the following archs: armeabi-v7a
[INFO]:    Found Android API target in $ANDROIDAPI: 27
[INFO]:    Available Android APIs are (27)
[INFO]:    Requested API target 27 is available, continuing.
[INFO]:    Found NDK dir in $ANDROIDNDK: /home/xyz49/.buildozer/android/platform/android-ndk-r19c
[INFO]:    Found NDK version 19c
[INFO]:    Getting NDK API version (i.e. minimum supported API) from user argument
[INFO]:    ccache is missing, the build will not be optimized in the future.
[INFO]:    Found the following toolchain versions: ['4.9']
[INFO]:    Picking the latest gcc toolchain, here 4.9
[INFO]:    Of the existing distributions, the following meet the given requirements:
[INFO]:    	invdef: min API 21, includes recipes (hostpython3, libffi, openssl, sdl2_image, sdl2_mixer, sdl2_ttf, sqlite3, python3, sdl2, setuptools, cython, pycparser, six, cffi, numpy, pyjnius, android, kivy, pycryptodome, idna, certifi, chardet, appdirs, importlib-metadata, filelock, kivymd, distlib, zipp), built for archs (armeabi-v7a)
[INFO]:    invdef has compatible recipes, using this one
# Run '/home/xyz49/.buildozer/android/platform/android-sdk/platform-tools/adb devices'
# Cwd None
List of devices attached
e4b7ee14	device

# Deploy on e4b7ee14
# Run '/home/xyz49/.buildozer/android/platform/android-sdk/platform-tools/adb install -r "/home/xyz49/Documents/invisible-defender-v2/bin/invdef-2.0.0-armeabi-v7a-debug.apk"'
# Cwd /home/xyz49/.buildozer/android/platform
Performing Push Install
/home/xyz49/Documents/invisible-defender-v2/bin/invdef-2.0.0-armeabi-v7a-debug.apk: 1 file pushed, 0 skipped. 4.1 MB/s (84687290 bytes in 19.568s)
	pkg: /data/local/tmp/invdef-2.0.0-armeabi-v7a-debug.apk
Success
# Application pushed.
# Run on e4b7ee14
# Run '/home/xyz49/.buildozer/android/platform/android-sdk/platform-tools/adb shell am start -n org.ballilab.invdef/org.kivy.android.PythonActivity -a org.kivy.android.PythonActivity'
# Cwd /home/xyz49/.buildozer/android/platform
Starting: Intent { act=org.kivy.android.PythonActivity cmp=org.ballilab.invdef/org.kivy.android.PythonActivity }
# Application started.
# Run 'dpkg --version'
# Cwd None
Debian 'dpkg' package management program version 1.19.7 (amd64).
This is free software; see the GNU General Public License version 2 or
later for copying conditions. There is NO warranty.
# Search for Git (git)
#  -> found at /usr/bin/git
# Search for Cython (cython)
#  -> found at /home/xyz49/Documents/invisible-defender-v2/bin/cython
# Search for Java compiler (javac)
#  -> found at /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
# Search for Java keytool (keytool)
#  -> found at /usr/lib/jvm/java-11-openjdk-amd64/bin/keytool
# Run '/home/xyz49/.buildozer/android/platform/android-sdk/platform-tools/adb logcat *:S python:D'
# Cwd /home/xyz49/.buildozer/android/platform
--------- beginning of crash
--------- beginning of system
--------- beginning of main
I/python  ( 6731): Initializing Python for Android
I/python  ( 6731): Setting additional env vars from p4a_env_vars.txt
I/python  ( 6731): Changing directory to the one provided by ANDROID_ARGUMENT
I/python  ( 6731): /data/data/org.ballilab.invdef/files/app
I/python  ( 6731): Preparing to initialize python
I/python  ( 6731): _python_bundle dir exists
I/python  ( 6731): calculated paths to be...
I/python  ( 6731): /data/data/org.ballilab.invdef/files/app/_python_bundle/stdlib.zip:/data/data/org.ballilab.invdef/files/app/_python_bundle/modules
I/python  ( 6731): set wchar paths...
I/python  ( 6731): Initialized python
I/python  ( 6731): AND: Init threads
I/python  ( 6731): testing python print redirection
I/python  ( 6731): Android path ['.', '/data/data/org.ballilab.invdef/files/app/_python_bundle/stdlib.zip', '/data/data/org.ballilab.invdef/files/app/_python_bundle/modules', '/data/data/org.ballilab.invdef/files/app/_python_bundle/site-packages']
I/python  ( 6731): os.environ is environ({'SECONDARY_STORAGE': '/storage/sdcard1', 'PATH': '/sbin:/vendor/bin:/system/sbin:/system/bin:/system/xbin', 'ANDROID_BOOTLOGO': '1', 'ANDROID_ROOT': '/system', 'ANDROID_ASSETS': '/system/app', 'ANDROID_DATA': '/data', 'ANDROID_STORAGE': '/storage', 'ASEC_MOUNTPOINT': '/mnt/asec', 'LOOP_MOUNTPOINT': '/mnt/obb', 'BOOTCLASSPATH': '/system/framework/core-libart.jar:/system/framework/conscrypt.jar:/system/framework/okhttp.jar:/system/framework/core-junit.jar:/system/framework/bouncycastle.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/ims-common.jar:/system/framework/mms-common.jar:/system/framework/android.policy.jar:/system/framework/apache-xml.jar:/system/framework/qcmediaplayer.jar:/system/framework/WfdCommon.jar:/system/framework/qcom.fmradio.jar:/system/framework/oem-services.jar:/system/framework/vcard.jar:/system/framework/tcmiface.jar', 'SYSTEMSERVERCLASSPATH': '/system/framework/services.jar:/system/framework/ethernet-service.jar:/system/framework/wifi-service.jar', 'LD_PRELOAD': 'libsigchain.so:libNimsWrap.so', 'EXTERNAL_STORAGE': '/storage/emulated/legacy', 'EMULATED_STORAGE_SOURCE': '/mnt/shell/emulated', 'EMULATED_STORAGE_TARGET': '/storage/emulated', 'ANDROID_PROPERTY_WORKSPACE': '8,0', 'ANDROID_SOCKET_zygote_secondary': '9', 'ANDROID_ENTRYPOINT': 'main.pyc', 'ANDROID_ARGUMENT': '/data/data/org.ballilab.invdef/files/app', 'ANDROID_APP_PATH': '/data/data/org.ballilab.invdef/files/app', 'ANDROID_PRIVATE': '/data/data/org.ballilab.invdef/files', 'ANDROID_UNPACK': '/data/data/org.ballilab.invdef/files/app', 'PYTHONHOME': '/data/data/org.ballilab.invdef/files/app', 'PYTHONPATH': '/data/data/org.ballilab.invdef/files/app:/data/data/org.ballilab.invdef/files/app/lib', 'PYTHONOPTIMIZE': '2', 'P4A_BOOTSTRAP': 'SDL2', 'PYTHON_NAME': 'python', 'P4A_IS_WINDOWED': 'True', 'P4A_ORIENTATION': 'sensor', 'P4A_NUMERIC_VERSION': 'None', 'P4A_MINSDK': '21', 'LC_CTYPE': 'C.UTF-8'})
I/python  ( 6731): Android kivy bootstrap done. __name__ is __main__
I/python  ( 6731): AND: Ran string
I/python  ( 6731): Run user program, change dir and execute entrypoint
I/python  ( 6731): [INFO   ] [Logger      ] Record log in /data/data/org.ballilab.invdef/files/app/.kivy/logs/kivy_21-02-19_2.txt
I/python  ( 6731): [INFO   ] [Kivy        ] v2.0.0
I/python  ( 6731): [INFO   ] [Kivy        ] Installed at "/data/data/org.ballilab.invdef/files/app/_python_bundle/site-packages/kivy/__init__.pyc"
I/python  ( 6731): [INFO   ] [Python      ] v3.8.5 (default, Feb 19 2021, 11:47:02) 
I/python  ( 6731): [Clang 8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec7462
I/python  ( 6731): [INFO   ] [Python      ] Interpreter at ""
I/python  ( 6731): [INFO   ] [Factory     ] 186 symbols loaded
I/python  ( 6731): [INFO   ] [Image       ] Providers: img_tex, img_dds, img_sdl2 (img_pil, img_ffpyplayer ignored)
I/python  ( 6731): [INFO   ] [Window      ] Provider: sdl2
I/python  ( 6731): [INFO   ] [GL          ] Using the "OpenGL ES 2" graphics system
I/python  ( 6731): [INFO   ] [GL          ] Backend used <sdl2>
I/python  ( 6731): [INFO   ] [GL          ] OpenGL version <b'OpenGL ES 3.0 V@100.0 AU@  (GIT@)'>
I/python  ( 6731): [INFO   ] [GL          ] OpenGL vendor <b'Qualcomm'>
I/python  ( 6731): [INFO   ] [GL          ] OpenGL renderer <b'Adreno (TM) 306'>
I/python  ( 6731): [INFO   ] [GL          ] OpenGL parsed version: 3, 0
I/python  ( 6731): [INFO   ] [GL          ] Texture max size <4096>
I/python  ( 6731): [INFO   ] [GL          ] Texture max units <16>
I/python  ( 6731): [INFO   ] [Window      ] auto add sdl2 input provider
I/python  ( 6731): [INFO   ] [Window      ] virtual keyboard not allowed, single mode, not docked
I/python  ( 6731): [INFO   ] [KivyMD      ] v0.104.1
I/python  ( 6731): [INFO   ] [Text        ] Provider: sdl2
I/python  ( 6731): [INFO   ] [GL          ] NPOT texture support is available
I/python  ( 6731): [WARNING] [Base        ] Unknown <android> provider
I/python  ( 6731): [INFO   ] [Base        ] Start application main loop
I/python  ( 6731):  Exception in thread Thread-1:
I/python  ( 6731):  Traceback (most recent call last):
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/threading.py", line 932, in _bootstrap_inner
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/python-installs/invdef/jnius/__init__.py", line 84, in jnius_thread_hook
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/threading.py", line 870, in run
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/app/main.py", line 774, in camera_receiver
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/python-installs/invdef/numpy/__init__.py", line 142, in <module>
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/python-installs/invdef/numpy/core/__init__.py", line 104, in <module>
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/python-installs/invdef/numpy/core/_add_newdocs.py", line 4441, in <module>
I/python  ( 6731):    File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/platform/build-armeabi-v7a/build/python-installs/invdef/numpy/core/function_base.py", line 506, in add_newdoc
I/python  ( 6731):  ImportError: dlopen failed: cannot locate symbol "log10f" referenced by "_multiarray_tests.so"...

The function_base.py file Line 508

def add_newdoc(place, obj, doc, warn_on_python=True):
    """
    Add documentation to an existing object, typically one defined in C

    The purpose is to allow easier editing of the docstrings without requiring
    a re-compile. This exists primarily for internal use within numpy itself.

    Parameters
    ----------
    place : str
        The absolute name of the module to import from
    obj : str
        The name of the object to add documentation to, typically a class or
        function name
    doc : {str, Tuple[str, str], List[Tuple[str, str]]}
        If a string, the documentation to apply to `obj`

        If a tuple, then the first element is interpreted as an attribute of
        `obj` and the second as the docstring to apply - ``(method, docstring)``

        If a list, then each element of the list should be a tuple of length
        two - ``[(method1, docstring1), (method2, docstring2), ...]``
    warn_on_python : bool
        If True, the default, emit `UserWarning` if this is used to attach
        documentation to a pure-python object.

    Notes
    -----
    This routine never raises an error if the docstring can't be written, but
    will raise an error if the object being documented does not exist.

    This routine cannot modify read-only docstrings, as appear
    in new-style classes or built-in functions. Because this
    routine never raises an error the caller must check manually
    that the docstrings were changed.

    Since this function grabs the ``char *`` from a c-level str object and puts
    it into the ``tp_doc`` slot of the type of `obj`, it violates a number of
    C-API best-practices, by:

    - modifying a `PyTypeObject` after calling `PyType_Ready`
    - calling `Py_INCREF` on the str and losing the reference, so the str
      will never be released

    If possible it should be avoided.
    """
    new = getattr(__import__(place, globals(), {}, [obj]), obj)  #  <--<--<--- This is the line 506
    if isinstance(doc, str):
        _add_docstring(new, doc.strip(), warn_on_python)
    elif isinstance(doc, tuple):
        attr, docstring = doc
        _add_docstring(getattr(new, attr), docstring.strip(), warn_on_python)
    elif isinstance(doc, list):
        for attr, docstring in doc:
            _add_docstring(getattr(new, attr), docstring.strip(), warn_on_python)

[RobertFlatt]

This isn't the right place either. (somebody will probably close this soon)
Usage questions are best asked here https://groups.google.com/g/kivy-users

I/python ( 6731): File "/home/xyz49/Documents/invisible-defender-v2/.buildozer/android/app/main.py", line 774, in camera_receiver

Are you using a camera?
If you choose to go there add what camera software you are using on Android.
Most camera software doesn't keep up with Android changes and doesn't work.

Also pycrypotodome is not pure Python so that is probably going to be an issue too. Perhaps pycrypto ?

Any followup please go to Google Groups.

[electricalgorithm]

No, I'm not using telephone's (Android) camera. That function is to receive the camera shots/data which is taken by the server. You can see the camera_reciever function in this repo. I have been asked on Discord channel, since there were no answer, I thought that it is a bug.

Edit: There were some pycrpto/pycryptodome errors before, I fixed them in someway. The error which I published here is not about any encryption library because the camera_reciever function doesn't encrypt or decrypt anything. Also, other crypted things are working very well. I think it is not a problem about pycrpytodome library.

[oukiar]

This bug is related to #kivy/buildozer#1020

In this case, the problem comes from the numpy library that has missing the linking to math lib.

The workaround to fix this issue is adding the math flag at linking time

As i can see exist already a patch for numpy adding the math flag in library dependencies, but seems that not all compilation parts of numpy are patched, so you need modify the file .buildozer/android/platform/python-for-android/pythonforandroid/recipes/numpy/__init__.py for patch using the environment LDFLAGS

and update with the next code:

from pythonforandroid.recipe import CompiledComponentsPythonRecipe
from multiprocessing import cpu_count
from os.path import join


class NumpyRecipe(CompiledComponentsPythonRecipe):

    version = '1.18.1'
    url = 'https://pypi.python.org/packages/source/n/numpy/numpy-{version}.zip'
    site_packages_name = 'numpy'
    depends = ['setuptools', 'cython']

    patches = [
        join('patches', 'add_libm_explicitly_to_build.patch'),
        join('patches', 'do_not_use_system_libs.patch'),
        join('patches', 'remove_unittest_call.patch'),
        ]

    call_hostpython_via_targetpython = False

    def get_recipe_env(self, arch=None, with_flags_in_cc=True):
        env = super().get_recipe_env(arch, with_flags_in_cc)
        
        env['LDFLAGS'] += ' -lm'
        
        return env

    def build_compiled_components(self, arch):
        self.setup_extra_args = ['-j', str(cpu_count())]
        super().build_compiled_components(arch)
        self.setup_extra_args = []

    def rebuild_compiled_components(self, arch, env):
        self.setup_extra_args = ['-j', str(cpu_count())]
        super().rebuild_compiled_components(arch, env)
        self.setup_extra_args = []


recipe = NumpyRecipe()

clean the python-for-android and rebuild.

buildozer android clean
buildozer android debug