From 901f9e4edb8c6a4f965e49c5bcc90f1e98cd4c73 Mon Sep 17 00:00:00 2001 From: Jonas Thiem Date: Thu, 25 Jul 2019 20:19:07 +0200 Subject: [PATCH] Add a document describing how p4a interacts with pip & python packages --- doc/source/contribute.rst | 151 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) diff --git a/doc/source/contribute.rst b/doc/source/contribute.rst index 2fde4daf17..1de0883c4c 100644 --- a/doc/source/contribute.rst +++ b/doc/source/contribute.rst @@ -79,3 +79,154 @@ Release checklist - [ ] `armeabi-v7a` - [ ] `arm64-v8a` - [ ] Check that the version number is correct + + + +How python-for-android uses `pip` +--------------------------------- + +*Last update: July 2019* + +This section is meant to provide a quick summary how +p4a (=python-for-android) uses pip and python packages in +its build process. +**It is written for a python +packagers point of view, not for regular end users or +contributors,** to assist with making pip developers and +other packaging experts aware of p4a's packaging needs. + +Please note this section just attempts to neutrally list the +current mechanisms, so some of this isn't necessarily meant +to stay but just how things work inside p4a in +this very moment. + + +Basic concepts +~~~~~~~~~~~~~~ + +*(This part repeats other parts of the docs, for the sake of +making this a more independent read)* + +p4a builds & packages a python application for use on Android. +It does this by providing a Java wrapper, and for graphical applications +an SDL2-based wrapper which can be used with the kivy UI toolkit if +desired (or alternatively just plain PySDL2). Any such python application +will of course have further library dependencies to do its work. + +p4a supports two types of package dependencies for a project: + +**Recipe:** install script in custom p4a format. Can either install +C/C++ or other things that cannot be pulled in via pip, or things +that can be installed via pip but break on android by default. +These are maintained primarily inside the p4a source tree by p4a +contributors and interested folks. + +**Python package:** any random pip python package can be directly +installed if it doesn't need adjustments to work for Android. + +p4a will map any dependency to an internal recipe if present, and +otherwise use pip to obtain it regularly from whatever external source. + + +Install process regarding packages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The install/build process of a p4a project, as triggered by the +`p4a apk` command, roughly works as follows in regards to python +packages: + +1. The user has specified a project folder to install. This is either + just a folder with python scripts and a `main.py`, or it may + also have a `pyproject.toml` for a more standardized install. + +2. Dependencies are collected: they can be either specified via + ``--requirements`` as a list of names or pip-style URLs, or p4a + can optionally scan them from a project folder via the + pep517 library (if there is a `pyproject.toml` or `setup.py`). + +3. The collected dependencies are mapped to p4a's recipes if any are + available for them, otherwise they're kept around as external + regular package references. + +4. All the dependencies mapped to recipes are built via p4a's internal + mechanisms to build these recipes. (This may or may not indirectly + use pip, depending on whether the recipe wraps a python package + or not and uses pip to install or not.) + +5. **If the user has specified to install the project in standardized + ways,** then the `setup.py`/whatever build system + of the project will be run. This happens with cross compilation set up + (`CC`/`CFLAGS`/... set to use the + proper toolchain) and a custom site-packages location. + The actual comand is a simple `pip install .` in the project folder + with some extra options: e.g. all dependencies that were already + installed by recipes will be pinned with a `-c` constraints file + to make sure pip won't install them, and build isolation will be + disabled via ``--no-build-isolation`` so pip doesn't reinstall + recipe-packages on its own. + + **If the user has not specified to use standardized build approaches**, + p4a will simply install all the remaining dependencies that weren't + mapped to recipes directly and just plain copy in the user project + without installing. Any `setup.py` or `pyproject.toml` of the user + project will then be ignored in this step. + +6. Google's gradle is invoked to package it all up into an `.apk`. + + +Overall process / package relevant notes for p4a +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Here are some common things worth knowing about python-for-android's +dealing with python packages: + +- Packages will work fine without a recipe if they would also build + on Linux ARM, don't use any API not available in the NDK if they + use native code, and don't use any weird compiler flags the toolchain + doesn't like if they use native code. The package also needs to + work with cross compilation. + +- There is currently no easy way for a package to know it is being + cross-compiled (at least that we know of) other than examining the + `CC` compiler that was set, or that it is being cross-compiled for + Android specifically. If that breaks a package it currently needs + to be worked around with a recipe. + +- If a package does **not** work, p4a developers will often create a + recipe instead of getting upstream to fix it because p4a simply + is too niche. + +- Most packages without native code will just work out of the box. + Many with native code tend not to, especially if complex, e.g. numpy. + +- Anything mapped to a p4a recipe cannot be just reinstalled by pip, + specifically also not inside build isolation as a dependency. + (It *may* work if the patches of the recipe are just relevant + to fix runtime issues.) + Therefore as of now, the best way to deal with this limitation seems + to be to keep build isolation always off. + + +Ideas for the future regarding packaging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- We in overall prefer to use the recipe mechanism less if we can. + In overall the recipes are just a collection of workarounds. + It may look quite hacky from the outside, since p4a + version pins recipe-wrapped packages usually to make the patches reliably + apply. This creates work for the recipes to be kept up-to-date, and + obviously this approach doesn't scale too well. However, it has ended + up as a quite practical interims solution until better ways are found. + +- Obviously, it would be nice if packages could know they are being + cross-compiled, and for Android specifically. We aren't currently aware + of a good mechanism for that. + +- If pip could actually run the recipes (instead of p4a wrapping pip and + doing so) then this might even allow build isolation to work - but + this might be too complex to get working. It might be more practical + to just gradually reduce the reliance on recipes instead and make + more packages work out of the box. This has been done e.g. with + improvements to the cross-compile environment being set up automatically, + and we're open for any ideas on how to improve this. +