Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0 #32

Open
jerryfat opened this issue Feb 20, 2024 · 32 comments · May be fixed by #38
Labels
Priority: Critical Issue is critical to the working of the system Target: pyqtdeploy The target of the issue is pyqtdeploy Type: Bug Issue related to the unexpected behaviour of an existing feature

Comments

@jerryfat
Copy link

Hi Achille Martin:
Thank you for the readme and pictures,
I was successfull building the demo app and running on emulator and tablet
Your demo app worked on emulator and phone I have
Great Instructions !

Looked like in previous 2.5.1 pyqtdeploy looks like user was able to add the site-packages directory and then include the packages.
I tried using the 2.5.1 pyqtdeploy and it cant open the pdy and find sysroot etc etc

when I build using 3.3.0 pyqtdelpoy my app code builds but apk app crashes ,
I tried to build my app but I dont see "Other packages" and installed packages like yaml and lxlm site-packages using pip install in venv or not venv too
my app runs on linux ok, but not emulator or tablet
I am assumming my app crashes on android since it cant find the imported packages that should have been frozen etc

can I add or scan directories for installed Other packages in your new 3.3.0 pyqtdeploy version ?

i didnt see how to add a sysroot directory like in 2.5.1 that looked and worked nice
I was trying to use the 2.5.1 ver and (wouldnt build) but
that was so nice in 2.5.1 , check marking the packages to include in build ,

In your readme maybe include another picture of the Other packages and add package if that feature works.
If there is a way to add packages manually let me know or workarounds
I appreciate your work very much, so nice the demo and readme work
Thanks so very much in advance,
Jerry Fat
my project on github:
https://github.com/jerryfat/FIX-Gateway/blob/master/README-MAVPX4.md

@achille-martin
Copy link
Owner

achille-martin commented Feb 21, 2024

Thank you for your interest @jerryfat!

Let's try to unpack what you are asking about and let's figure out the answer together.

I haven't created a template to write issues yet because I didn't actually expect this project to attract developers and programmers. However, since it seems that you have found some interest in the project, I'll bump this task up!

For the title, I would focus on: <keyword>: <problem / topic>

<keyword> could be:

  • Bug (=problem spotted)
  • Bugfix (=solution to a bug)
  • Feature (=request)
  • Research (=open question)

In your case, it seems that a good title would be: Bug: cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0

Moving on to the content of the bug/request, I would break it down into:

Bug

Description

Objective: What is the expected behaviour / What are you trying to achieve

Observations: Describe the observed unexpected behaviour, adding as many details as possible (including screenshots, when needed).

Version: [Component or package version(s) affected by this issue; it can be identified by a tag, or a commit SHA](Link to the tag/commit)

Reporter: Name Surname <email> of the person reporting the bug.

Linked Issues: [Name of the linked issue)](Link to related issue)

Replication

Provide a comprehensive list of step-by-step instructions to reproduce the issue.

Explanation

After investigating the reported bug, give a brief explanation of the reason which is believed to cause the issue.

Proposed solution

Describe the proposed solution to address the bug.

Notes:

Additional notes about the proposed solution.

@achille-martin
Copy link
Owner

achille-martin commented Feb 21, 2024

Your issue actually gathers all the elements that I have listed above, which is great.

Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0

Description

Objective: You want to add yaml and lxlm to your Android app using the .pdt

Observations:

  • You have installed packages yaml and lxlm site-packages using pip install in venv (and not venv as well)
  • You tried to include them to the .pdt
  • You built the Android app
  • When you launch your Android app, it crashes
  • When you launch your app on Linux, it works fine

Version: pyqtdeploy 3.3.0; yaml version?; lxlm version?

Reporter: @jerryfat

Linked Issues: None

Replication

Do you have an example PyQt5 script with yaml to share here?
As well as the steps you took to install yaml in your venv?

Explanation

Initial research

  • You have used pyqtdeploy 2.5.1 in the past and you were able to add the site-packages directory of your host Python interpreter through the Other packages tab of your .pdy. It worked fine.
  • You are wondering where this feature is or if it is still available in pyqtdeploy 3.3.0.

Existing resources

  • I haven't tried to include packages from the site-packages folder yet because I didn't need them yet, so I haven't been able to confirm the feature in pyqtdeploy 3.3.0.
  • Phil Thompson is the creator of pyqtdeploy and will be able to help you on that topic.
  • Phil has created a very brief documentation on pyqtdeploy which you can find HERE. However, it does not really cover your question. This is actually why I decided to create this repo PyQt-CroM: to promote the tool and complement the documentation, making it easier to access and understand.
  • After a bit of digging into pyqtdeploy 3.3.0, it looks like this page about sysroot definition is relevant to your question: by specifying the Python wheel, you can install a specific component from the sysroot, as explained HERE.

Going further

  • If the answer is still not satisfying, I suggest you have a look HERE.
  • Add your name to the PyQt mailing list and ask your question to Phil directly.

Proposed solution

We will update this section and then add the whole bug / solution note to the PyQt-CroM troubleshooting note (if needed).

Notes:

  • You would like an updated README with the feature clearly expressed.

Personal notes: Your project looks great, I had a very brief experience with the Mavlinks during a competition at Uni.

@achille-martin
Copy link
Owner

As a bonus, I have added your feedback to the specific discussion thread if you want to complete it.

@jerryfat
Copy link
Author

jerryfat commented Feb 21, 2024 via email

@achille-martin
Copy link
Owner

achille-martin commented Feb 22, 2024

  • Thanks for your feedback @jerryfat! Really appreciated

  • If you wish to support my little project, please leave a message or comment here: https://www.buymeacoffee.com/achille_martin

  • I'll have a look at how to add external python packages into the.pdy (testing with yaml first as an example). In the meantime, if you figure out the solution, let me know.

  • The design choices for pyqtdeploy are explained here (but it's not complete yet): https://www.riverbankcomputing.com/static/Docs/pyqtdeploy/

  • It is possible to revert back to pyqtdeploy v2.5.1 if you pick a tag (like 1.2.0) of PyQt-CroM repo. However, I try to promote the latest versions are you might be limited by PyQt and Python options.

  • What is the issue you have with "printf"? Did you copy-paste the snippet into your terminal and it bugged your bashrc?
    Can you send a screenshot of your bashrc after running the command? Thanks.

@jerryfat
Copy link
Author

jerryfat commented Feb 22, 2024 via email

@achille-martin
Copy link
Owner

achille-martin commented Feb 24, 2024

  • The pyqtdeploy documentation is hard to read sometimes but it is useful. When I struggle to understand it, I look into the source code and try to figure out what Phil Thompson did! I don't understand everything either about pyqtdeploy but I enjoy the tool.
    Pyqtdeploy is actually exploiting other tools from Qt (like androiddeploy) and from PyQt (to convert the python into Cpp).
    I will write another type of documentation once I get more information about the design choices for pyqtdeploy.

  • Test with yaml as site-package is ongoing. Someone else asked about including javascript code into the Android app. Not sure yet how it is supposed to look like / work but there is definitely an interest around adding external modules into the generated app.

  • No worries about the printf issue. If it comes up again, open a new issue and we'll look at it.

@achille-martin achille-martin changed the title How to add packages from installed site-packages, can I add or scan directories for installed Other packages in your new 3.3.0 pyqtdeploy version ? Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0 Feb 25, 2024
@jerryfat
Copy link
Author

jerryfat commented Feb 27, 2024 via email

@jerryfat
Copy link
Author

jerryfat commented Feb 27, 2024 via email

@achille-martin
Copy link
Owner

achille-martin commented Feb 27, 2024

Impressive, thanks for figuring this out!

Glad we are now able to import non-standard python packages through wheels.

I'll create a merge request so we can fully test the feature and then merge when happy.

In the meantime, please make use of your script to get the python packages you need!

Thanks again

@jerryfat
Copy link
Author

jerryfat commented Feb 27, 2024 via email

@jerryfat
Copy link
Author

jerryfat commented Feb 28, 2024 via email

@achille-martin
Copy link
Owner

There is unfortunately nothing to my knowledge that deals with .apk building and pyqtdeploy-2.5.1.

This is actually one of the reasons why I started PyQt-CroM: so that we can build on any platform quickly and easily.

I am not sure about the change from .pdy to .pdt, only the great Phil Thompson will know...
I haven't investigated the code fully yet, but I hope to get more insights as I use pyqtdeploy tool.

In the meantime, if you are looking for more documentation on pyqtdeploy-2.5.1, I suggest that you:

  • Download the .tar.gz from PyPI
  • Read the internal documentation by opening pyqtdeploy-2.5.1/doc/html/index.html
  • Navigate to other files within pyqtdeploy-2.5.1 if you cannot find the information you need at first

Additional note

Thinking about the documentation, I have double checked the ChangeLog for pyqtdeploy-3.3.0 (located in pyqtdeploy-3.3.0/ChangeLog) and it looks like there has been a full switch from .pdy to .pdt for all related files (even test files). This could mean that the upgrade from pyqtdeploy project file .pdy to pyqtdeploy project file .pdt might have been for consistency across the package.

@jerryfat
Copy link
Author

jerryfat commented Feb 28, 2024 via email

@achille-martin
Copy link
Owner

This is work in progress, I'll have a look but I can't say when it will be fully fixed.

I can do some testing before end of March, but it will be tight to get the feature completely ready.

The best way to get ultra quick feedback is to send an email to Riverbank / the PyQt maillist: https://www.riverbankcomputing.com/mailman/listinfo/pyqt

I think you should be able to send an email directly to pyqt@riverbankcomputing.com.

@achille-martin
Copy link
Owner

Hi @jerryfat,

Intro

I have created a small example on how to integrate PyYAML into an Android app.

The example pyqt5 app can be found on branch 32-bug-cannot-add-python-packages-from-site-packages-directory-of-my-python-interpreter-to-the-pdt-of-pyqtdeploy-330.

Question

Can you try to generate an Android app from the pyqt5 script and tell me if it all works as you expect?

Notes

You might need to:

cd $PYQT_CROM
git pull
git checkout 32-bug-cannot-add-python-packages-from-site-packages-directory-of-my-python-interpreter-to-the-pdt-of-pyqtdeploy-330
source $PYQT_CROM_DIR/venv/pyqt-crom-venv/bin/activate
pip3 install -r requirements.txt
cd $PYQT_CROM_DIR/utils
&& python3 build_app.py --pdt $PYQT_CROM_DIR/examples/external/external_python_project/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose

Going further

If you confirm that it's all going as expected, I will add a little tutorial on how to add any external python package to the sysroot.toml and config.pdt.

@jerryfat
Copy link
Author

jerryfat commented Mar 21, 2024 via email

@mrRaduga
Copy link

Hello @achille-martin

I am trying to use your tool now and I concluded that I also need the possibility to add extra packages, like requests, numpy, pandas. So, this why now I am here.

I have already tested your proposal and first I got the following error in the build step:

Screenshot 2024-04-18 145658

And I realized that the correct yaml package name is PyYaml and changed it to sysroot.toml and config.pdt.
After that the .apk file builds successfully but unfortunately it doesn't run on Android.

@achille-martin
Copy link
Owner

Hi @mrRaduga,

I have created a pull request that demonstrates the capability you are looking for, for the yaml case.

I am in the process of adding a bit of documentation to the PR so that anyone can add any non-standard python module to a PyQt5 app.

I'll keep adding some documentation and will ask you later on to verify that you can access the capability you need.

I expect that I will be done by the end of the month (April 2024).

@achille-martin
Copy link
Owner

achille-martin commented Apr 18, 2024

After re-reading your message, I realised that you have started testing the PR.

I will review the issue with PyYAML. This is in fact an exception of the whole process.

The name on PyPI (PyYAML) does not match the name of the package locally installed (yaml).

To make sure that we are on the same page:

  1. Can you tell me what you have done? Like all the steps

  2. Have you used this sysroot and this pdt to build this example app with yaml?

@jerryfat
Copy link
Author

jerryfat commented Apr 18, 2024 via email

@achille-martin
Copy link
Owner

Changing the name of the dependency in the sysroot only ignores the module. This is because the build process will look for something called PyYAML, but the only module that is installed locally is yaml. If you look closely at the debug prints of pyqtdeploy, you will realise that yaml is never built or never included at any point. It is simply skipped.

I might have missed an instruction or step somewhere, because I can definitely create an Android app with yaml.

I have attached the Android app example, which corresponds to this PyQt5 app. You can run the app in an Emulator or on an Android phone to validate.

ExternalPythonApp.zip

However, that is not the interesting bit. The interesting bit is that other users cannot replicate the steps, so I need to figure out where it went wrong.

@mrRaduga
Copy link

For testing external packages feature I did the following steps:

  1. Download your example external/external_python_project from the second branch
  2. Install the PyYAML library by the command pip install PyYAML in the enviroment
  3. Tried to deploy to android by command python3 build_app.py --pdt $HOME/Documents/pyqt-crom/examples/external/external_python_project/config.pdt --jobs 1 --target android-32 --qmake $QT_DIR/android/bin/qmake --verbose
  4. Received the error showed in my last message
  5. Updated the name of library in sysroot.toml file, as bellow:
    image
  6. Update the name of library in config.pdt file, as bellow:
    Screenshot 2024-04-19 100452
  7. Tried again to deploy to Android, which builds successfully, .apk file installed successfully but didn't run.

You are right, I checked the logs of building process and yaml is never included. And what is interesting the pip list command shows me the yaml module with PyYAML name, but the files locally are installed in yaml folder, no clue why. Maybe take sense to try with another modules?

Based on that I tried again with your config.pdt and sysroot.toml files and see in the console the rows bellow:
image
But I didn't see the package yaml in the build-android or sysroot-android folder after building process.

@achille-martin
Copy link
Owner

achille-martin commented Apr 22, 2024

Hi @jerryfat and @mrRaduga,

I am still in the process of trying to figure out what is preventing you from building with yaml on your machines.

I have created a help script that collects all necessary information for debugging purposes.

Can you please run the following commands and paste / link the output on this issue thread?

  1. Get wheel name to validate platform and version
chmod +x $PYQT_CROM_DIR/utils/bash/get_python_package_wheel.sh &&
$PYQT_CROM_DIR/utils/bash/get_python_package_wheel.sh pyyaml 6.0.1

Note: the output will be on screen. Please copy-paste the output here.

  1. Get errors linked to building and background information about OS and repo (change <absolute_path_to_your_pdt>)
chmod +x $PYQT_CROM_DIR/utils/bash/get_command_output_help.sh &&
$PYQT_CROM_DIR/utils/bash/get_command_output_help.sh "cd $PYQT_CROM_DIR/utils/python && python3 build_app.py --pdt <absolute_path_to_your_pdt>/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose"

If you would like to add your sysroot.toml and config.pdt to the output of the script (so that you don't have to copy-paste them here):

chmod +x $PYQT_CROM_DIR/utils/bash/get_command_output_help.sh &&
$PYQT_CROM_DIR/utils/bash/get_command_output_help.sh "cd $PYQT_CROM_DIR/utils/python && python3 build_app.py --pdt <absolute_path_to_your_pdt>/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose" -f "<absolute_path_to_sysroot>/sysroot.toml <absolute_path_to_pdt>/config.pdt"

WARNING: there are 3 breaking changes now in the PR (that will be released as v3.0.0).

  • Location of path_setup.sh has changed and is now called setup_path.sh -> might need to update your ~/.bashrc
  • Similar for download_sources.sh which has been moved too
  • The script build_app.py has also been moved

@mrRaduga
Copy link

mrRaduga commented Apr 23, 2024

Good morning @achille-martin,

I have successfully run your commands, and you can check the output in the attached file.
For testing I have used the external_package_project_2 without any changes for config.pdt and sysroot.toml file.
The .apk file successfully installed and launched on Android device with API 31, see the screenshot below and attached log from the device.
Screenshot_20240423_100920

command_output_help_2024_04_23-09_59_41.md

pyqt5_app_with_yaml.log

Physically located in this path on the device: /data/data/org.qtproject.example.ExternalPyApp/files/pyqt5_app_data/pyqt5_app_with_yaml/pyqt5_app_with_yaml.log

@achille-martin
Copy link
Owner

Thank you @mrRaduga, that is very promising!

I will keep updating the README and add more instructions to let people know how to add their own non-standard python modules.

I might give it a try with pandas in the meantime because it has various python package dependencies (while PyYAML had none).

I will also sort out this issue, probably once this issue is merged.

This will overall take a bit of time and I am away for a few weeks (so I would estimate the updates to be done by mid May 2024).

@mrRaduga
Copy link

Servus @achille-martin

Nice to find you well and thanks for your job

Based on your updates, I tried again to implement the request package in the external example, but unfortunately without any good result.

Getting the wheels for a request package using the get_python_package_wheel.sh script returns me the following result, which should not be underestimated. Why do I have 5 wheels founded and downloaded, are these dependencies for the requests package? If yes, why then are there no others?
image

After that I tried to get the dependencies for each package using the py_package_dependent_collector.py script. Finally, the sysroot file looks like this, which is a little off-putting in my opinion:

image

As I understand it, from this point of view, each dependency that is a non-standard Python library also requires specifying all the wheels and dependencies in the sysroot file. If I understand this correctly, this can be very tedious, recursive, and hard to debug which packages are missing as a dependency.

P.S. I also tried to just specify the requests package as a wheel in the sysroot file, also without success.

I'll be glad if you come back with a successful example of using the requests library, I see it as a rather complex example.

@achille-martin
Copy link
Owner

achille-martin commented May 17, 2024

@mrRaduga Thanks for giving it a try.

I am currently testing with numpy (no required dependencies but complex) and then pandas (required dependencies and complex so should be equivalent to requests). I am also in the process of creating a script that will search for all the dependencies and list them so they can be added to the sysroot directly.

So far, I am having a few issues with some python built-in packages (potential need to move to Python3.11.0). I'll update you when I can get the numpy package working and I'll share the handy scripts.

@jerryfat
Copy link
Author

jerryfat commented May 17, 2024 via email

@achille-martin
Copy link
Owner

@jerryfat Thank you for your contribution.

The wheels do not need to be downloaded actually, pyqtdeploy will take care of that and build straight away.

I am currently trying to understand whether specific standard python library dependencies need to be specified for a non-standard python package or if it would be better to include ALL standard python libraries into the app (at the cost of the heavy weight of the app).

To remind everyone, pyqt-crom is still in its early stages, so there is a lot of work required around understand the "build process" (especially through Qt tools).

@achille-martin achille-martin added Priority: Critical Issue is critical to the working of the system Target: pyqtdeploy The target of the issue is pyqtdeploy Type: Bug Issue related to the unexpected behaviour of an existing feature labels May 22, 2024
@pierre75013
Copy link

Hi Achille,
As I said on your fiverr account, I am also interested to add Python packages from site-package like numpy and scipy.
Thank you very much for your interesting contribution with the project pyqt-crom which works very well on the demo example and with a simple project of mine that I succesfully transforms in an Android apk application.
Pierre

@achille-martin
Copy link
Owner

Hi Pierre,
Thank you for the support, I am working in the background on the topic so will let you know (and everyone interested) when I have some more progress on general integration of python packages!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: Critical Issue is critical to the working of the system Target: pyqtdeploy The target of the issue is pyqtdeploy Type: Bug Issue related to the unexpected behaviour of an existing feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants