To make android build through buildozer work in native Windows without WSL2/ VirtualBox etc

[Samael-TLB]

Description

Currently, buildozer can only build for android on Linux OS. For usage through Windows, its suggested to use either WSL2 or VirtualBox, etc. But, on reviewing the internal codebase, it seems we can support native Windows build for android without using any hacks. We have to just have the commands issued to the shell to be platform compliant. Either we can use cmd or powershell for that. The only other part that I am not fully covered up for to achieve this , is the requirements of buildozer. Once, we can figure the list of dependencies that might be needed for native Windows or can port the Linux dependencies counterparts for Windows, we can safely have native Windows build for android.

To further support the claim, we can see that beehive project supports android building natively without any OS specific dependencies(beg to differ?). Also, as we are using the cmdline-tools to build for android and as these tools are also released for Windows by Android project so we can safely do the compilation on native Windows.

Moreover, this feature will help a lot of Windows user who may be unfamiliar with Linux or WSL2 and they can just as comfortably do the build without doing anything specific, just like how Android Studio does it on Windows. This will give extra ease of use and for future, we can have a cross-platform Android Studio like project using Kivy resources for build.

[Julian-O]

I have been working on addressing this issue for a few weeks now.

It is a non-trivial change, especially for python-for-android. My development motto has been "It probably won't happen in my lifetime, but I want to make it possible in my grandchildren's lifetime."

I started with Buildozer. In my prototyping, I am getting as far as the first python-for-android call.

To achieve the goal, I have been breaking apart the Buildozer object into separate files: #1632, #1639, #1647 (still awaiting review)

The latter is the key one. It provides platform-agnostic infrastructure for most of the key operations (excluding python-for-android). Once it is merged, I will be issuing a number of PRs to slowly migrate across from the old operations to the new ones.

Apart from that, there have been some specific points where the Windows code wasn't working: #1628. #1648 (awaiting review)

When all of that is finished, I expect to have Buildozer able to perform all of the operations on native Windows EXCEPT:

• Anything to do with macos and ios
• Automatically accepting the license agreement
• The key work done by Python For Android, which is a big showstopper!

If you would like to help out, I can certainly find stuff to do, but the bottleneck is PR reviews, so I am not sure extra coders will help.

[Julian-O]

This is a status report/info dump about the state of play in allowing Buildozer to support the Android target on Windows:

Brief Summary: A small dent has been made in the mountain of work over the past few months. It clears the path for future work (and has also improved the quality and performance of a lot of code unrelated to Windows), but hasn't delivered anything useful to the user.

Background for those new to the project:

• Python-For-Android (p4a) is a tool to port the Python interpreter to Android, and package your Python project so the CPython can run it. There are a lot of messy details (like recipes and cross-compilation); things can easily go wrong. It takes a lot of care to set it up right, and select the right parameters.

• Buildozer is a tool that takes a config file, and uses it to set up all the dependencies that p4a has. (It also supports other targets; iOS and OSX.)

• When run on Windows, Buildozer offers very limited options. Basically, it can be used to debug Android apps running on your phone, but not build them.

What needs to be done to make Buildozer work on Windows?

• The biggest issue, by far, is that p4a doesn't support Windows. It is filled with platform-dependent code. I will discuss that further below. Even if Buildozer supported Windows, it would fail when p4a is called to do the core work. I need to emphasize this, so I am not giving false hope: While I have been tinkering with Buildozer, p4a is where the biggest effort and risk is in terms of Windows support.

• There were many calls in Buildozer to command-line utilities that don't exist on Windows. What was worse, the call to subprocess was performed in a way that didn't work on Windows. I have made some significant improvements via a series of PRs to migrate to Buildops - a set of utilities that are largely platform-independent.

  • Known shortcoming of Buildops: There is a cmd_expect function that is not platform independent. [Want to help out? This would be good side project.] The only consequence is that the automated acceptances of licenses is not possible on Windows.
  • Known shortcoming of Buildops: At the time of writing, unzipping isn't working but Add alternate unzip implementation #1692 should deal with that.
  • Known shortcoming of Buildops: If it is asked to file_extract a .bin file, it will try to execute it, which won't work on Windows. However, in practice, this is never happens on a Windows platform, so it isn't a problem. (I am not even sure if it is ever happens on the other platforms.)

• There were many small fixes that needed to be done (e.g. sdkmanager is called sdkmanager.bat on Windows). I have fixed a few, but expect a few more.

  • aidl is called aidl.exe on Windows. Unfortunately, which won't recognise that the explicit path is fine on Windows, so the full path needs to be corrected - this is a trivial fix that I haven't submitted yet, hoping to bundle it with something more substantial.

• The Android target code raises a NotImplementedError exception on import on Windows machines, to make it clear it is not supported. I added a backdoor - if you set an environment variable, it will allow Android as a target to allow experimenting.

• _ensure_virtual_env() is another "Won't work on Windows, but is never called while building Android apps, so it doesn't matter" situations.

• Just making the code platform independent isn't enough. The tests for the code often are not platform-independent, assuming paths and similar.

  • First step: Add pytest "@skipif" decorators to skip them on Windows, with a clear comment about what in the test needs to be fixed to remove it.
  • Second step: Add CI tests to run tests on Windows. Perhaps with appveyor? We don't want our gains to be lost by non-Windows developers breaking them without knowing.
    • Tip: Ensure they run in parallel and don't slow down the already-slow test cycle, to avoid inconveniencing developers working on unrelated code.
  • Third step: Update the tests themselves to be cross-platform, and remove the @skipifs.

• Once all the above is done, the Android target should stop raising NotImplementedException on Windows platforms.

• Once all the above is done, the documentation will need to be updated.

Right now, with the experimental environment variable, a patch for the AIDL name and with no SDK license acceptance, Buildozer can successfully run init/help/version/clean and all other non-target-specific commands. When asked to build, it can successfully:

• Ensure build layout
• Check config tokens.
• Prepare the build
• Check requirements
• Install p4a
• Install Apache ANT
• Install Android NDK
• Install Android SDK, and update
• Download the platform API target

It falls over as soon as it is time to call p4a.

P4a is the most important next step.

• The great news is with Buildozer able to be used to drive it, it be easier to work on.
• p4a has many calls to command line utilities that don't exist on Windows.
  • Some are trivially replaced by pure platform-independent Python. I have submitted a few PRs related to this, but that work is largely on the fringe.
  • There is extensive use of the "sh" package to make these calls, which used to work on Windows, but they haven't supported Windows for some time.
  • There are Windows-compatible forks, which involves having a dependency on another project.
  • Alternatively, could move away from using the "sh" module to something more akin to subprocess (e.g. buildops.cmd in Buildozer), but sh has a particular API style that means there will be a lot of work to rip it out.
• Other developers have vaguely warned that there will be much deeper difficulties in cross-compiling. I haven't hit these issues yet.
• The list of platforms supported is entrenched in a lot of code, especially recipes. I see this as an area that should be refactored to be more extensible, and then extended to add Windows. Recipes look like they could gain from using subclassing more cleanly. I've had some ideas in that area, but no action yet.