Dependency specification and testing should be cleaned up.

[Julian-O]

This is a review of the way Buildozer's (pre-run) dependencies are set-up, and how that set-up is tested.

I hope to follow up with a PR shortly.

Ideally:

• pip should be able to find all the package dependencies of a project (and only those package dependencies).
• If there are version dependencies, pip should be able to find them to select the best version - especially when there are multiple packages to install.
• If dependencies only apply to some uses (testing, generation of documentation, building for a particular platform, etc.) they should be specified in extras_require. They will only be installed for users who want that functionality, and allow those users to install all dependencies with relative ease.
• The documentation should reflect this, so people know how to set-up their environment.
• The integration tests should confirm that all the necessary dependencies are installed.
• [Outside the scope here: It should be PEP 517 compatible.]

The current system falls short of this ideal in the following ways:

• Buildozer does not specify the version of Cython.

  • There is a known issue that Buildozer doesn't work with Cython V3 and above. See Use a pinned version of Cython for now, as most of the recipes are incompatible with Cython==3.x.x #1637
  • The tests pre-install a version to pin to 0.29.36.
  • However, the specification of acceptable versions should be done in setup.py, so anyone who doesn't pre-install Cython still gets the right version.
  • Thus, the integration tests should NOT pre-install it. They should be testing that the setup process gets it right.

• Buildozer does not specify that it needs pytest to run the unit-tests in the /tests directory.

  • This triggers warnings in some static-checkers, such as those in PyCharm
  • pytest is currently named as a dependency tox.ini, but you can run unit-tests without tox. The dependency should be added to extras_require for "tests" in setup.py.
  • coverage is also installed by tox, but it is not required to run the tests, so need not be mentioned in setup.py.
  • Tox still needs to install pytest because it doesn't install Buildozer; it just copies the files.

• The Android integration test installs automake. This is not required for Android. It should be removed to both save time and to show it is not required.

• The Android integration messes with ssl links on MacOs:

    sudo ln -sfn /usr/local/opt/openssl /usr/local/ssl
  

  This is not documented and not required. Notably, it doesn't appear in the ios version. It should be removed.

• setup.py unnecessarily requires virtualenv.

  • It isn't used by Buildozer. venv has been part of Python since 3.3, and we no longer support earlier versions than that.
  • It can be removed. (See also existing PR dependencies: rm virtualenv, already using python3 -m venv #1515.)
  • The documentation tells ios developers to manually pip install virtualenv. That instruction should be removed.

• setup.py unnecessarily requires sh.

  • It isn't used by Buildozer.
  • However, removing it can cause IOS builds to fail! Because kivy-ios needs it. So, the dependency should be from kivy-ios, not Buildozer. Unexpectedly, kivy-ios does list sh in its requires. It turns out Buildozer's TargetIos attempts to install kivy-ios from Github, but doesn't actually install it (with pip). It just copies the files across. The dependencies are never resolved.
  • Remove 'sh' from setup.py. Replace it with an extras_require for ios, that installs kivy-ios. That will install its dependencies, such as sh.
  • The documentation tells ios developers are to manually pip install kivy-ios. That instruction should be replaced with installing buildozer[ios], and let the system work out the individual dependencies.

• The ios integration test unnecessarily installs cookiecutter and pbxproject.

  • They appear to be spurious.
  • Remove.

• Buildozer does not specify that it needs sphinx to generate the documents in the /docs directory.

  • Add it to extra_requires for docs.
  • The tests of the documentation (added in Add tests of the documentation. #1683) should be updated to install buildozer[docs], and not directly install sphinx.

• Tox doesn't need to special-case Python 3 any more.

• The tests shouldn't install Buildozer as an "editable install" (i.e. the -e option on pip install)

  • They should test the behaviour the client will see.
  • It is incompatible with PEP 517 - and hopefully one day Buildozer will be compatible with PEP 517.

[Julian-O]

I have realised that Cython is actually not directly needed by Buildozer, but only by p4a and kivy-ios.

In an ideal world, the requirement should be moved into them, and they should be pip installed, simplifying Buildozer's task.

[RobertFlatt]

This would be great, a significant portion of user questions are new users who watch a YouTube video then in Buildozer crash and burn in some obscure way because they didn't look at the install instructions. Trapping these errors at the start would greatly improve the user experience.

This has to be in p4a, as these are p4a dependencies not Buildozer dependencies. Tests in Buildozer will produce false results as the Buildozer version and the p4a version are not related.

Also most of the dependencies are not pip dependencies, they are system libraries. These should ideally be tested without requiring another Python package (e.g. python-apt), adding this would break every existing installation. There is apparently some variability in system library names between Linux flavors. We don't document this, but any testing would have to take this into account. Automatic installation would require sudo, which is the opposite of what is required for an apk build on most platforms.

That said, this would be great.

[Julian-O]

@RobertFlatt: There is some discussion in #1686 that may be helpful, and a promise of further discussion.

Note sure how adding python-apt or similar would break existing installations, but that discussion can happen later.

some variability in system library names between Linux flavors.

Ugh! That'll be a nightmare.

Automatic installation would require sudo, which is the opposite of what is required for an apk build on most platforms.

Sure. Hopefully we can limit it to just the apt-get/homebrew portions. [I am not aware if the warning against running as root is a sensible precaution against overwriting important files, or it leads to known issues. If the latter, we should put that in a comment in warn_if_root().]

[RobertFlatt]

Note sure how adding python-apt or similar would break existing installations,

It would become a required dependency, which (presumably) nobody currently has installed.

Ugh! That'll be a nightmare.

Just some research that nobody has bothered to document. These projects are not famous for the quality of their documentation.

the warning against running as root is a sensible precaution

There are three reasons not to run as root.

1. Some Python packages behave differently if installed (by p4a) as root.

2. A program with root permissions that pulls stuff from all over the net is a security risk.

3. On traditional Linux OSes this is not the way it is designed to be used.

[RobertFlatt]

One more for the list, Java version

[RobertFlatt]

One more for the list: test cwd is not on an NTFS partition.

I've just seen a new twist on this, with the latest version of git reporting this if the project is on an NTFS partition

fatal: detected dubious ownership in repository at '/mnt/....

Git presumably thinks it is on windows :(

[Julian-O]

@RobertFlatt: I know that the Linux Android SDK doesn't like being installed on NTFS (symbolic links break) and now Git complains, so such a check makes sense if platform is linux/macos.

However, if the platform is win32 (I know, I know; not yet supported - p4a is the big bottleneck), the Windows versions of the SDK and Git are fine with NTFS.

[RobertFlatt]

@Julian-O The common case we see is WSL, and setting cwd to the Windows partition (the default ☹️).
The case today was a Linux (not WSL) has a mounted disk that happened to be NTFS and was only historically a Windows disk, there was no Windows OS involved.

The issue with the latest Git on NTFS (and also with building some Python packages on NTFS) is they see they are on NTFS and choose to operate as if they are under the Windows OS, despite being executed in a Linux environment. This combination results in obscure fails.

It not the currently supported host OS that is the issue, it is the formatting of the disk containing the directory that is the cwd.

So for the current OS an NTFS test would be very helpful.

---

Separately, when considering adding a Windows (not WSL) port of p4a, this is also an NTFS port. The question becomes what OS related issues will there be? I hope for your sake not many.

In the specific case of Git, one could argue the port will be OK (it is the OS combination of a built app that is the issue)- we'll know when you test it. The recipes are a separate issue, and again we'll know when you test.

Exactly what the issues for a Windows port of p4a will be would be foolish to predict. But it is normal to expect to have to address such issues when porting.

Any need for the above NTFS test for a Windows (not WSL) port of p4a would imply the project is dead in the water, so the above test would not be applied. Which does raise the question how does one check for Windows OS without checking the disk format? I expect there is an answer somewhere.

[Julian-O]

@RobertFlatt - short version: I completely agree - that is all consistent with my understanding.

So for the current OS an NTFS test would be very helpful.

Right. I think it would a great idea to reduce support requests to have a check that says:

 if platform in ("darwin", "linux"):   # <- I am asking for this line
     if drive_is_ntfs(root_dir) or drive_is_ntfs(global_buildozer_dir):
         warn()

I have been working to make Buildozer itself platform-independent, and reacted to this suggestion to make sure we don't go backwards.

The question becomes what OS related issues will there be? I hope for your sake not many.
[...] But it is normal to expect to have to address such issues when porting.

Fully expecting this to be a nightmare. I have been focussed on Buildozer to date, but p4a is much scarier. Just did an info dump in #1651 if you want to know more about where I am at there.

Which does raise the question how does one check for Windows OS without checking the disk format? I expect there is an answer somewhere.

sys.platform will give you the OS. Checking for the disk format appears to be, of course, platform-specific. I imagine there is a small library out there that hides the details.

[RobertFlatt]

@Julian-O
We are on the same page.