Trying to build a wheel file for netcdf4 and getting errors #823
Comments
|
Please create a fork of this repository, and post the link here. Make sure you include:
If anyone else wants this package too, let us know by clicking the thumbs-up button above. |
|
Took a second... Created a fork of the chaquopy GitHub site. It is at located at https://github.com/githubweather/chaquopy. Uploaded my meta.yaml, and a file called seeit.build_wheel.changes which is from the command 'diff build-wheel.py.orig build-wheel.py > seeit.build_wheel.changes'. The latter file, build-wheel.py being the file with my changes. Also created a file called README.githubweather which is a general summary of what I stated here. |
|
Forgot to add - I have made no edits to the netcdf4 source to date. |
|
Your changes to build-wheel are very difficult to understand in this format. Please actually modify build-wheel in the repository and commit it, instead of adding a separate patch file. It would also be useful if your changes included comments explaining why they were necessary. But I suspect most of them actually won't be necessary if you follow the instructions below correctly. Based on a quick look at the netcdf4 setup.py, you should probably be listing numpy as a Since this is a Python interface to a native library, and the library doesn't appear to come bundled with the Python package, you'll have to build the library as a separate wheel. The two wheels will have the same relationship as pyzmq and chaquopy-libzmq, as described in the README. Finally, it looks like this package may require the hdf5 library. We've already built this for Android, so you should add that to the Sorry about how complicated this is, but some packages are just like that. |
|
Greatly appreciate your assistance with this project. Currently trying to implement your suggestions. Downloaded numpy-1.23.3-0-cp310-cp310_21_arm64_v8a.whl and placed in server/pypi/dist/numpy. Edited meta.yaml, moving numpy reference to 'host' section as '- numpy 1.23.3-0'. Also tried '- numpy 1.23.3' and keep getting cannot find wheel file for requirement numpy 1.23.3-0 (or numpy 1.23.3). What am I doing wrong? |
|
If you need more help, please post the full build log, including the build-wheel.py command line, and your current meta.yaml file. |
|
I was able to get things right regarding the whl files for numpy and h5py since I needed the ones for python 3.8 and arm64_v8a. Thx for that advice. Now I'm trying to grasp the other part you mentioned -- "Since this is a Python interface to a native library, and the library doesn't appear to come bundled with the Python package, you'll have to build the library as a separate wheel. The two wheels will have the same relationship as pyzmq and chaquopy-libzmq, as described in the README." Which README are you referring to? |
|
Based on your suggestions I have broken things out as follows -- chaquopy-netcdf4 directory -- build.sh -- ======================================================================================== ======================================================================================= The current errors are: These look to be required by the native netcdf-c code and are defined in zlib (current version is 1.2.13). How can I include zlib in the build? |
|
I am so sorry. I thought I had inadvertently closed this issue earlier but now find it is not. Good. Pls disregard the 'new' issue of the same title that I created. Now, back to this issue, since I failed with the pasting of my build.log file (it was too many characters), is there any. way to attach a file? |
You can attach any file by dragging it into the comment box. |
|
The build.log above comes from executing: "build-wheel.py --python 3.8 --abi arm64-v8a chaquopy-netcdf4". Let me know how I can incorporate zlib into the native build of netcdf-c. In a 'regular' build on my VirtualBox Debian virtual machine I would add the LDFLAGS and LIB flags to configure. But not sure what to do within chaquopy. Does a 'wheel' file need to be created for zlib? Thanks for any clues. |
|
zlib comes with Android itself, so all that should be necessary is to add However, I then got this error: This probably means that one of its libraries requires another one of its libraries, and it's not bothering to provide an I also changed the netcdf build.sh file to be more consistent with the libzmq example. If you do need to deviate from the example, it would be useful if you explain why. Here's the resulting build.sh file: Finally, I've just pushed an update to build-wheel to let it detect With those changes, the build of chaquopy-netcdf4 succeeds. However, I'm suspicious about some of those |
|
Thanks for this awesome progress. I tested build.sh without the --disable-netcdf4 flag and it yields "can't find or link to the hdf5 library". Apparently. netcdf uses hdf5 in some cases for compression, etc. Seems would definitely be optimal if hdf5 is part of the build. What would be the best way to incorporate hdf5 in this build? |
|
See above: #823 (comment). |
|
yes - thanks for all your comments. I was trying to explain why I added the --disable-netcdf4 flag. When I looked this flag up (at the UCAR netcdf-c site) it is supposedly equivalent to adding the --enable-hdf5 flag, but I have not verified this yet. I noticed there is a wheel file for hdf5 which I added to this build. Not sure if it changes the build at all. |
|
Now I'm back on my Android Studio project and am trying to include the *.whl file generated thx to a lot of help from you. I have something wrong in getting the AS project to 'see' the .whl file now. I have uploaded the wheel file to my GitHub site (githubweather). Here are the particulars in the AS code: The resulting error upon build attempt: I clearly have not captured the nomenclature needed in the AS code. Pls let me know what I've got wrong here. |
|
I'm not sure whether a GitHub website link will work with Rather than putting it on a web server, the easiest way to test your package would be to copy your Alternatively, if your |
|
I went with your tip in paragraph 2 above. My AS code now looks like: where chaquopy-netcdf4 is the directory under /Users/me/TEMPY/home/me/WHEEL-TEST/chaquopy/server/pypi/dist/ and where the wheel file resides. When I try to rebuild, it simply says: "Exception:" with no information as to what the exception is. I hope I'm getting closer but still am missing something... |
|
That should be fine, but you could also try removing the By the way, to mark code blocks you need three backticks like this: |
|
I see a build log in the Android Studio window (Build Output tab). Was wondering where/if this same build log information is written to a file? When I run "adb logcat" it does not capture the build information that I see in the Android Studio window. |
|
The build log is shown in the Build Output window, and it isn't written to a file as far as I know. Unfortunately, the window defaults to showing individual error messages, which are often useless without context. To see the full build log, click the 'Build ... failed' caption to the left of the log. |
|
|
|
Pls note the screen capture of the build log above. |
|
Like I said, click the "Build ... failed" line in bold, then you'll see the full log in the text view to the right. If you don't see anything, you may have shrunk the text view to nothing, in which case you'll need to drag it out again. If you need more help, please post:
|
|
I see that when I right-click the "Build ... failed" line in bold, it allows one to "show warnings" and/or "show successful steps". The screenshot I posted were the warnings. I could screenshot the warnings and the successful steps. Would it be more hopeful if I post both? Here's the python section of build.gradle: Underneath the "--extra-index-url" directory is the directory "chaquopy-netcdf4". And in that directory is the wheel file, chaquopy_netcdf4-4.9.2-1-py3-none-android_21_arm64_v8a.whl |
|
should read "helpful" rather than "hopeful" in post above. |
|
@vmitro: Thanks, that all looks reasonable. @githubweather: You're using too many
The |
|
Thx @vmitro and @mhsmith. I have updated my meta.yaml file, including the numpy version, as you mentioned. Also created the chaquopy.patch file as recommended. My current issue is with the build.sh file. This is the build.sh file -- For reference here is the chaquopy.patch file, as in the info you provided -- Here is the result of running build-wheel.py -- Lastly, when I clone cftime.git, create the configure file (as done in build.sh), and run configure, this is the issue-- |
|
@githubweather: I don't think you need a |
|
Tried that. But still get the errors upon running build-wheel.py --- |
|
Strange, it should not fail so early in the build process. What's the content of the directory for Python target? Adjust when needed, you target Python 3.8 IIRC. Edit: It would maybe beneficial if you didn't deviate from the standard peocedure; could you try from scratch by reseting the repo, only copying the Python 3.8 target and adding the cftime directory to |
|
Yes - not sure what is happening. I created a new cloned tree of chaquopy (good idea btw). And the same thing is happening. My chaquopy-netcdf4 package still builds perfectly (wheel file generated just fine). And the chaquopy-cftime tree is dying on the build-wheel.py call. Here is the meta.yaml file for chaquopy-cftime -- Here is the tail end of the build-wheel.py call -- Here is the maven directory -- and the patch file -- |
Have you built the zip yourself or have you downloaded it from the Maven repo? Because the sizes don't match. The one in the repo is 5571890 bytes long and yours is 5701885. |
|
@vmitro - excellent catch! Indeed the maven file size mismatch was the key. The issue was I had been downloading the maven file to my Mac which would unzip it automatically. Thus, I then had to re-compress to a zip file so apparently it was getting mangled. A straight download to my VM fixed things. Also you were right about not needing a build.sh file, although it is somewhat unclear to me as to why one is not needed? I still have a bit of testing to go, i.e., including this cftime wheel file in my package build which includes netCDF4, but this is a big step forward. Thx. |
|
@githubweather I've got amazing powers of observation, haha. Now the reason why it doesn't need a separate build script is because the package's |
|
Indeed - thx again for catching that maven issue! Makes good sense regarding why a build.sh is not needed. I 'think' I'm on the last part of this -- now trying to build the netcdf4-python wheel file. When I run build-wheel.py I am seeing this:
When I do ldd on _netCDF4.so, I get, in part: Indeed when this _netCDF4.so is. built this is the call:
(Note: later on in the build-wheel.py run, _netCDF4.cpython-38-x86_64-linux-gnu.so is renamed to _netCDF4.so) So that the problem is that -L/usr/lib/x86_64-linux-gnu is in the call above, however, the ldd on _netCDF4.so, shows that the libm.so.6 is actually in /lib/x86_64-linux-gnu. I somehow need to have build-wheel.py add -L/lib/x86_64-linux-gnu so that it can find libm.so.6. What's the best way to do. this? Thx. |
|
The problem here is the /usr/lib references on the clang command line. This causes Linux libraries to be linked into the Android build, which will never work. Most likely, there's something in a build script (such as setup.py) which detects that it's building on Linux, and adds these arguments to the command unconditionally. You'll need to edit the build script to remove these arguments. Since build-wheel already adds all the necessary -L arguments to For how to go about doing making these edits, see "If any changes are needed to make the build work" in the README. I also notice that you haven't renamed your package to follow this comment. Please do that, or you will have even more problems. |
|
Thx for the suggestions. I had to make two edits in the setup.py file in order for it to finish and have build-wheel.py create a wheel file. Namely, commenting out these 2 lines -- However, it is creating a linux rather than an android wheel file. The file name generated is: Q1 - I'm not sure why build-wheel.py is building a linux wheel file rather than an Android wheel file? Do I need to find even more setup.py edits? Of note here, I had to use chaquopy-hdf5 1.10.2 in the host requirements, rather than hdf5 1.10.2 in the host requirements which is what I used successfully in building the chaquopy-netcdf4 4.9.2 wheel file. Q2 - not sure why I had to do this? (I'm using the same directory tree for the build) build-wheel-call-chaquopy-netcdf4-python.log And this is the output from the build-wheel call -- Note the error line at the end of the build-wheel execution--
Q3 - Please let me know what is needed to rectify this issue and would this issue play a role in the wheel file being linux-based rather than android-based? Thx for any suggestions. |
I don't know how I can make this any clearer than my previous comment:
This means the Python package name should be If you need any more help, please update your fork with the current state of your recipes, and use the same directory layout as the existing packages. Save any changes to the package source code as patches, again using the same format as the existing packages. But if you're still using a modified copy of build-wheel itself, please check that in directly, so I can see the changes in the GitHub diff viewer. |
|
Thanks for that. Have a new question regarding the numpy portion. When I run my project with the build.gradle as below: I am seeing this error: Some research showed that if numpy is installed using python3.10 as follows -- the error goes away. And I was able to prove this on my Mac. So my question is, as my build.gradle stands, python3.8 is employed. How can I force it to be python3.10? Also it needs to have the qualifier, as above, "--no-binary=:all:" to avoid the numpy.core._multiarray_umath error, on install. Can that arg be incorporated in the build.gradle, "numpy install"? When I try to add the version 3.10 qualifier in the python block of build.gradle it is erroring out saying the version qualifier is not allowed. How to circumvent this issue? |
|
One clarification on something I wrote above -- when I say I was able to prove this on my Mac. What I mean is, at the command line, I could run the relevant python script, using python3.10, and the 2 required arguments. Also, one other thing, in a prior Android project, I was able to declare "version 3.8" in the python block in build.gradle. But in my new Android project (the one I am referencing in my questions), the compiler states "version 3.10" is not a valid argument in the python block of build.gradle. It seems to be a versioning issue (Java/Android?) but I am not sure of the root cause. |
Please post exact error messages, otherwise it'll be hard for me to guess where they're coming from. As shown here, Python versions newer than 3.8 have been supported since Chaquopy 13.0. I recommend you update to the current Chaquopy version if you haven't already. But the packages you've built are for Python 3.8 (as shown by the The only NumPy builds we currently have are:
But some of your comments refer to version 1.24.2, as does the current version of your fork, which hasn't been updated for 2 months. So if you want more help, you must:
|
|
Finally updated my fork at https://github.com/githubweather/chaquopy/tree/master with recipes for chaquopy-netcdf4, chaquopy-cftime, chaquopy-netcdf4-python, and an updated numpy. When I run my APK in Android Studio, I am seeing this in the logcat: This is the build.gradle: The logcat correctly shows Python version as 3.10. But shows numpy version incorrectly as 1.23.5. As seen in my build.gradle, I install numpy-1.25.0rc1-0 from a wheel file. Not sure what is happening. From the build it looks like this newer version of numpy fixes the error seen in the logcat: Pls let me know what could be going wrong here. |
|
My main question is how in the logcat, it shows: when I run my APK, but on my Mac, where I am running Android Studio, it shows this at the command line: And I updated my fork so these package recipes that I'm using (some of them depend on numpy) reference numpy 1.25.0: chaquopy-cftime, chaquopy-netcdf4, numpy, and chaquopy-netcdf4-python. Plus the numpy wheel that I built and install via the build.gradle as seen below: [@mhsmith edit: same build.gradle file as above except for the following diff] // version "3.8"
pip {
// options "--extra-index-url","file:///Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/"
- install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/numpy/numpy-1.25.0rc1-0-cp310-cp310-android_21_x86_64.whl"
+ install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/numpy/numpy-1.25.0-0-cp310-cp310-android_21_x86_64.whl"
install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/chaquopy-netcdf4/chaquopy_netcdf4-4.9.2-1-py3-none-android_21_x86_64.whl"
install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/cftime/cftime-1.6.2-0-cp310-cp310-android_21_x86_64.whl"
- install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/netcdf4/netCDF4-1.6.3-0-cp310-cp310-android_21_x86_64.whl"
+ install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/netcdf4/netCDF4-1.6.4-0-cp310-cp310-android_21_x86_64.whl"
// install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/h5py/h5py-2.10.0-1-cp38-cp38-android_21_x86_64.whl"
}
}is based on numpy 1.25.0. |
|
Your Mac's Python installation is completely independent from your app's Python installation. Changing one will have no effect on the other. I downloaded these packages from the current version of your fork: And when I built an app with them I was able to import and use numpy just fine. So I still don't know why it isn't working for you, or where the number 1.23.5 came from.
If you need more help, please do this:
Also, your fork still contains a patch file against build.wheel.py, but I can't understand any of it. Please remove the patch file, and if you are still using a modified copy of build-wheel.py, check it in directly so I can see the changes in the GitHub diff viewer. And include comments explaining why the modifications were necessary. Finally, to simplify the situation, I strongly recommend you use one of our pre-built numpy versions listed here rather than building your own. If you need to use another version, please explain why. |
|
I have fixed a few things and now have an issue regarding building h5py for cp310. But first, to clarify, I am using the original build-wheel.py and removed the show.* file, which looked like a patch file, from my fork. Sorry for that confusion. [Remainder moved to #918] |
|
[Moved to #918] |
|
There are too many things in this issue already, so I've moved your comments about h5py to a separate issue: #918.
After several reports of this problem from other users, I eventually solved it in #909. To get the fix, please merge the current master branch of this repository into your fork. |
|
Thanks for your efforts here. To update my fork, is it as simple as "Sync fork" ==> "Update Branch". I assume this will preserve what I have plus import the changes you have made? Just wanted to verify as this effort is my first venture into Github. |
|
Yes, I think that will work. |
I am trying to build an app which requires python scripts that need to import netcdf4. When I do "pip install netcdf4" as seen in the python block in the build.gradle file below and try to build the app in Android Studio, I get:
"CCompiler.compile: Chaquopy cannot compile native code"
among other errors.
As such it was recommended to me by mhsmith that I build a wheel file for netcdf4 following the instructions at:
https://github.com/chaquo/chaquopy/tree/master/server/pypi
For reference, I am creating the wheel file on this VM:
Linux debian 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 GNU/Linux
I have had to make a few minor mods to the build-wheel.py file and get to this part of build-wheel.py --
and it outputs a string of errors such as the following --
However, I do see libmkl_rt.so.2 in /home/me/PYTHON3.8/lib. How can I make build-wheel.py see it? Via "LDFLAGS", or "LD_LIBRARY_PATH"? Regarding openblas, I built it 'by hand' and it installed to /opt/OpenBLAS/lib. Again, how to make build-wheel.py see this path?
Below is the portion of build.gradle that should be relevant--
Build log error from Android Studio:
Let me know if I need to add more to make this understandable.
Thanks much.
The text was updated successfully, but these errors were encountered: