-
-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0 #32
Comments
Thank you for your interest @jerryfat! Let's try to unpack what you are asking about and let's figure out the answer together. I haven't created a template to write issues yet because I didn't actually expect this project to attract developers and programmers. However, since it seems that you have found some interest in the project, I'll bump this task up! For the title, I would focus on:
In your case, it seems that a good title would be: Moving on to the content of the bug/request, I would break it down into: BugDescriptionObjective: What is the expected behaviour / What are you trying to achieve Observations: Describe the observed unexpected behaviour, adding as many details as possible (including screenshots, when needed). Version: [Component or package version(s) affected by this issue; it can be identified by a tag, or a commit SHA](Link to the tag/commit) Reporter: Name Surname <email> of the person reporting the bug. Linked Issues: [Name of the linked issue)](Link to related issue) ReplicationProvide a comprehensive list of step-by-step instructions to reproduce the issue. ExplanationAfter investigating the reported bug, give a brief explanation of the reason which is believed to cause the issue. Proposed solutionDescribe the proposed solution to address the bug. Notes:Additional notes about the proposed solution. |
Your issue actually gathers all the elements that I have listed above, which is great. Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0DescriptionObjective: You want to add yaml and lxlm to your Android app using the .pdt Observations:
Version: pyqtdeploy 3.3.0; yaml version?; lxlm version? Reporter: @jerryfat Linked Issues: None ReplicationDo you have an example PyQt5 script with ExplanationInitial research
Existing resources
Going further
Proposed solutionWe will update this section and then add the whole bug / solution note to the PyQt-CroM troubleshooting note (if needed). Notes:
Personal notes: Your project looks great, I had a very brief experience with the Mavlinks during a competition at Uni. |
As a bonus, I have added your feedback to the specific discussion thread if you want to complete it. |
Hi Achille:
Thank you for creating detailed emails for "issues" message I left on your github
Do you want me to do anything more, I can test or give you opinions lol
I appreciate your patience in detailing such a complicated issue in your emails in response to my newbie questions,
You "hit the nail on the head" as they say or "Bullseye"
Any pics are so helpful but difficult to create,
maybe a flowchart of the the overall process and or the vars you use once it works
it so cool you got all this working and the readme too thanks for a very good and concise document
you write very well .
I stumbled upon your github by chance
I have tried the necesitas method of pyqt into android and it was really clumsy and didnt really seem to work right
I can test or give you opinions lol
or add issues or whatever you want
I have some time if I can help you somehow...
btw from your readme somehow the printf "" of vars into .bashrc is in a loop on my machine
I had to stop the command and hand remove the entries at end of .bashrc
I dont know why its doing that
Do you want me to see why that >> .bashrc command in your readme is doing that here ?
it looks correct in readme but didnt work here in xterm for me ubuntu 20.1 mint
only problem I had with your readme
Can I use your 2.5.1 version to build with apps with "Other site-packages" ?
the sysroot you create is really cool concept, trying to understand what you are doing
and I dont yet know how does sysroot get set there in 2.5.1 or works ?
i see the change from xml to python config file (?) python style of config vars from 2.5.1 to 3.3.0
and I see why you changed from .pdy to .pdt suffix,
good idea
Is there a link or description somewhere to older working build and version 2.5.1 that I can try with other packages to create android apk?
Is there a readme or working pyqtdeploy using the 2.5.1 ? using which Python ? that can be used ?
and are there any manual ways I can add Other packages to your 3.3.0 ?
or is that feature still needs to be added ? it was in 2.5.1 and works great , nice interface in 2.5.1
Finest Regards, and thanks again
<edited by AM to not show the reply>
|
|
Hi Achille:
Appreciate pyqtdeploy3.3.0 document of the 'Other packages' or 'site-packages' for my issue
how did you make such a nice doc btw ? very nice and captures this info well
Appreciate all those links at end of doc, and how I could add a package or module to the pyqtdeply3.3.0 pdt ,
did not know you could do that
I will also try this and let you know if it works here, but this getting kinda "outside my wheelhouse" lol as they say
I am assumming (!) yaml or lxml or any other site-packages package should do same thing as a test for the python binary from the pdt ?,
so any imported module (from site-packages or from pip install)? of venv target python binary ? (from the binary python attached venv sysroot ?
my hope is issue should somewhow automagically (for me) be detected and the site packages appear in pyqtdeploy3.3.0 Other packages tab
I am guessing that by specifying the python version in pyqtdeploy3.3.0 and the .pdt, the build process downloads and builds a binary which then needs all the site packages as static(freeze) and then linked into app , way over my head right now.
also guessing so by adding the packages from a wheel in the pdt they will get built as binaries and linked into final app binary
I was going to add these test import yaml or lxml to the demo project in the instructions that builds ok and will add a line or two in app code that calls
will send to you
I dont know how to look under the hood of the pyqtdeploy app or pyQt or python, still a mystery and learning
This is getting way over my understanding of the magic you guys like Phil Thompson make
I had no idea the complexity and now struggling to understand and make stuff work on apk
I sent an email to Qt inf at riverbank email aaddress, only after trying to create a new login for their forums. which didnt seem to work for me.
I hope I didnt anger someone like Phil Thompson , my new hero.
I could not reproduce 'my problem' on command line in .bashrc from your most excellent instructions,
I now have no idea why my .bashrc filled up with thousands of those text lines.
Finest Regards, and Sincerely,
<edited by AM to not show the replies>
|
|
Hi Achille:
I wrote an app that checks for the .whl for imports in a .py file
I have attached the code
i will git hub it soon
if no -f on command line it uses the app imports
if -f filename.py it checks filename.py for imports and gets whl files
app tries to get .whl using pip download and puts into ./wheels directory
it then creates the sysroot.toml style formatted text for inclusion into the pyqtdeploy3.3.0
I will try a few demos to see if this brings in the wheels for each import and dependencies packages
python 3.10
`list-modules-imported.py`
*************************************************
Cf. file `list-modules-imported-v1.txt`
[list-modules-imported-v1.txt](https://github.com/achille-martin/pyqt-crom/files/15358119/list-modules-imported-v1.txt)
*******************************************************************************
<edited by AM to group the code>
|
I put on github for now:
https://raw.githubusercontent.com/jerryfat/FIX-Gateway/master/list-modules-imported.py
https://raw.githubusercontent.com/jerryfat/FIX-Gateway/master/list-pip-module-deps.py
$ pythonlist-pip-module-deps.py -m module arg lists module deps
$ python list-modules-imported.py if no args creates 'sysroot.toml style 'component text of itself and if any dependency packages also formatted as components for sysroot.toml
with args $ list-modules-imported.py -p package or $ list-modules-imported.py -f filename.py , either creates sysroot.toml style [component] text
…-Jerry Fat
On Friday, February 23, 2024 at 10:48:26 PM EST, Achille MARTIN ***@***.***> wrote:
-
The pyqtdeploy documentation is hard to read sometimes but it is useful. When I struggle to understand it, I look into the source code and try to figure out what Phil Thompson did! I don't understand everything either about pyqtdeploy but I enjoy the tool.
Pyqtdeploy is actually using other tools from Qt (like androiddeploy) and from PyQt (to convert the python into Cpp).
I will write another type of documentation once I get more information about the design choices for pyqtdeploy.
-
Test with yaml as site-package is ongoing. Someone else asked about including javascript code into the Android app. Not sure yet how it is supposed to look like / work but there is definitely an interest around adding external modules into the generated app.
-
No worries about the print issue. If it comes up again, open a new issue and we'll look at it.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Impressive, thanks for figuring this out! Glad we are now able to import non-standard python packages through wheels. I'll create a merge request so we can fully test the feature and then merge when happy. In the meantime, please make use of your script to get the python packages you need! Thanks again |
Hi Achille:
I just fixed a bug after testing list-modules-imported,py ; error when if wheels dir not existing
also tried all the cmd line args, -f -m none and with wheels dir existing
you;ll need to pull new copy of list-modules-imported,py
https://raw.githubusercontent.com/jerryfat/FIX-Gateway/master/list-modules-imported.py
if you find issues let me know
Thanks and Finest Regards,
-Jerry Fat
<edited by AM to remove the replies>
|
Hi:
Do you know if is there is a set of instruction like yours but using pyqtdeploy2.5.1 and .pdy for building an apk ?
The pyqtdeply2.5.1 looked great and it generates a huge json file that looks like it has tons of things going on inside.
any idea when did the change occur from .pdy to .pdt config files ?
Thanks in advance,
-Jerry Fat
<edited by AM to remove replies>
|
There is unfortunately nothing to my knowledge that deals with This is actually one of the reasons why I started PyQt-CroM: so that we can build on any platform quickly and easily. I am not sure about the change from In the meantime, if you are looking for more documentation on
Additional noteThinking about the documentation, I have double checked the ChangeLog for |
Hi Achille:
Does this sound correct ?
to test if the pyqtdeploy3.3.0 docs and instructions work for adding 'component' certifi in sysroot.toml
I added 'import certifi' into your demo_app.py
downloaded .wh
built binariesl
works in linux nbut ot in android and crashes
summary, from
~/Documents/pyqt-crom/examples/demo/demo_project/
sysroot.toml file:
[certifi]
plugin = "wheel"
wheel = "certifi-2021.10.8-py2.py3-none-any.whl"
dependencies = ["Python:importlib.resources", "Python:os"]
exclusions = ["__main__.py"]
I tried to test adding componet by importing into demo app:
I added "import certifi" to demo app.py
and downloaded the .whl
$ cd ~/Documents/pyqt-crom/examples/demo/demo_project/
$ python -m pip download --only-binary :all: --dest . --no-cache certifi
to check .whl:
$ ls ~/Documents/pyqt-crom/examples/demo/demo_project/certifi-2024.2.2-py3-none-any.whl -als
$ 160 -rw-rw-r-- 1 jf jf 163774 Feb 28 17:44 /home/jf/Documents/pyqt-crom/examples/demo/demo_project/certifi-2024.2.2-py3-none-any.whl
using your instructions
pyqtdeploy3.3.0 build succeeds , demo_app.py works in linux with the import certifi
demo_app . apk crashes in android (linux android studio emulator and my phone)
demo_app.py witout the import certifi wokrks on android and linux
I dont know how to move forward now.
btw in that app list-modules-imported.py i wrote that gets .whl files on github ,
it still needs dependencies added from the list-pip-module-deps.py test qapp I wrote on github
I was going to combine but,
If the import certifi in demo_app.py fails then I dont know what to do now.
I tried to build using the .pdy and pqyqtdeply2.5.1 and it failed to build sysroot and errored
Do you have a way to contact author ? I tried to get a qt forum login but no one ever responded, and login doesnt work
Thanks
-Jerry Fat
<edited by AM to remove replies>
|
This is work in progress, I'll have a look but I can't say when it will be fully fixed. I can do some testing before end of March, but it will be tight to get the feature completely ready. The best way to get ultra quick feedback is to send an email to Riverbank / the PyQt maillist: https://www.riverbankcomputing.com/mailman/listinfo/pyqt I think you should be able to send an email directly to pyqt@riverbankcomputing.com. |
Hi @jerryfat, IntroI have created a small example on how to integrate PyYAML into an Android app. The example pyqt5 app can be found on branch QuestionCan you try to generate an Android app from the pyqt5 script and tell me if it all works as you expect? NotesYou might need to:
Going furtherIf you confirm that it's all going as expected, I will add a little tutorial on how to add any external python package to the |
HI:
Thanks for the info on building android app with python package, I will try and get back to you friday
THANKS
<edited by AM to remove unnecessary reply>
|
Hello @achille-martin I am trying to use your tool now and I concluded that I also need the possibility to add extra packages, like requests, numpy, pandas. So, this why now I am here. I have already tested your proposal and first I got the following error in the build step: And I realized that the correct yaml package name is PyYaml and changed it to sysroot.toml and config.pdt. |
Hi @mrRaduga, I have created a pull request that demonstrates the capability you are looking for, for the I am in the process of adding a bit of documentation to the PR so that anyone can add any non-standard python module to a PyQt5 app. I'll keep adding some documentation and will ask you later on to verify that you can access the capability you need. I expect that I will be done by the end of the month (April 2024). |
After re-reading your message, I realised that you have started testing the PR. I will review the issue with The name on PyPI ( To make sure that we are on the same page:
|
Hello Achille:
I was able to compile after changing package name too
the app compiled, but the app and apk did not work on Android or simulator in linux.
Was this techinque ever tested ?
It doesnt work here too.
IMHO This is not a useabale solution for running python packages on Android.
my $0.02
-Jerry Fat
On Thursday, April 18, 2024 at 11:54:44 AM EDT, Achille MARTIN ***@***.***> wrote:
After re-reading your message, I realised that you have started testing the PR.
I will review the issue with PyYAML. This is in fact an exception of the whole process.
The name on PyPI does not match the name of the package locally installed.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Changing the name of the dependency in the sysroot only ignores the module. This is because the build process will look for something called I might have missed an instruction or step somewhere, because I can definitely create an Android app with yaml. I have attached the Android app example, which corresponds to this PyQt5 app. You can run the app in an Emulator or on an Android phone to validate. However, that is not the interesting bit. The interesting bit is that other users cannot replicate the steps, so I need to figure out where it went wrong. |
I am still in the process of trying to figure out what is preventing you from building with yaml on your machines. I have created a help script that collects all necessary information for debugging purposes. Can you please run the following commands and paste / link the output on this issue thread?
Note: the output will be on screen. Please copy-paste the output here.
If you would like to add your
WARNING: there are 3 breaking changes now in the PR (that will be released as v3.0.0).
|
Good morning @achille-martin, I have successfully run your commands, and you can check the output in the attached file. command_output_help_2024_04_23-09_59_41.md Physically located in this path on the device: /data/data/org.qtproject.example.ExternalPyApp/files/pyqt5_app_data/pyqt5_app_with_yaml/pyqt5_app_with_yaml.log |
Thank you @mrRaduga, that is very promising! I will keep updating the README and add more instructions to let people know how to add their own non-standard python modules. I might give it a try with I will also sort out this issue, probably once this issue is merged. This will overall take a bit of time and I am away for a few weeks (so I would estimate the updates to be done by mid May 2024). |
Servus @achille-martin Nice to find you well and thanks for your job Based on your updates, I tried again to implement the request package in the external example, but unfortunately without any good result. Getting the wheels for a request package using the get_python_package_wheel.sh script returns me the following result, which should not be underestimated. Why do I have 5 wheels founded and downloaded, are these dependencies for the requests package? If yes, why then are there no others? After that I tried to get the dependencies for each package using the py_package_dependent_collector.py script. Finally, the sysroot file looks like this, which is a little off-putting in my opinion: As I understand it, from this point of view, each dependency that is a non-standard Python library also requires specifying all the wheels and dependencies in the sysroot file. If I understand this correctly, this can be very tedious, recursive, and hard to debug which packages are missing as a dependency. P.S. I also tried to just specify the requests package as a wheel in the sysroot file, also without success. I'll be glad if you come back with a successful example of using the requests library, I see it as a rather complex example. |
@mrRaduga Thanks for giving it a try. I am currently testing with So far, I am having a few issues with some python built-in packages (potential need to move to Python3.11.0). I'll update you when I can get the |
Hello Achille:
I also am unable to include packages in the .pdt file
I have also written two python apps that would get package and download them, but they become recursive after getting all the packages and deps in python literally, and besides includiing them in sysroot never worked either
there is an already existing package (excilim) and it does not get built into android binary either
that should be first test
try date package and print the time or date in android using the example, it wont work either in android as package never gets built and then linked into android binary
that should be the first test
-Jerry Fat
list-modules-imported-for-pdt.py
Cf. file `list-modules-imported-for-pdt-v1.txt`
[list-modules-imported-for-pdt-v1.txt](https://github.com/achille-martin/pyqt-crom/files/15358133/list-modules-imported-for-pdt-v1.txt)
<edit by AM to make more compact>
|
@jerryfat Thank you for your contribution. The wheels do not need to be downloaded actually, I am currently trying to understand whether specific standard python library dependencies need to be specified for a non-standard python package or if it would be better to include ALL standard python libraries into the app (at the cost of the heavy weight of the app). To remind everyone, pyqt-crom is still in its early stages, so there is a lot of work required around understand the "build process" (especially through |
Hi Achille, |
Hi Pierre, |
Hi Achille Martin:
Thank you for the readme and pictures,
I was successfull building the demo app and running on emulator and tablet
Your demo app worked on emulator and phone I have
Great Instructions !
Looked like in previous 2.5.1 pyqtdeploy looks like user was able to add the site-packages directory and then include the packages.
I tried using the 2.5.1 pyqtdeploy and it cant open the pdy and find sysroot etc etc
when I build using 3.3.0 pyqtdelpoy my app code builds but apk app crashes ,
I tried to build my app but I dont see "Other packages" and installed packages like yaml and lxlm site-packages using pip install in venv or not venv too
my app runs on linux ok, but not emulator or tablet
I am assumming my app crashes on android since it cant find the imported packages that should have been frozen etc
can I add or scan directories for installed Other packages in your new 3.3.0 pyqtdeploy version ?
i didnt see how to add a sysroot directory like in 2.5.1 that looked and worked nice
I was trying to use the 2.5.1 ver and (wouldnt build) but
that was so nice in 2.5.1 , check marking the packages to include in build ,
In your readme maybe include another picture of the Other packages and add package if that feature works.
If there is a way to add packages manually let me know or workarounds
I appreciate your work very much, so nice the demo and readme work
Thanks so very much in advance,
Jerry Fat
my project on github:
https://github.com/jerryfat/FIX-Gateway/blob/master/README-MAVPX4.md
The text was updated successfully, but these errors were encountered: