[revamp] Twisted

[str4d]

I'm creating this issue because my meta-discussion doesn't belong in the other tickets I have opened. If it wasn't already obvious, my motivation is to get Twisted working with the revamp, so I can use it in a new project.

In #453 it was suggested that Twisted will require a recipe. I just tried the same process as #453 again, but using --requirements setuptools,zope.interface,twisted and with a Twisted CompiledComponentsPythonRecipe (but no recipes for setuptools or zope.interface). This fails (even after manually fixing the build env after #450 occurs) because the Python modules are installed after the recipes, and of course Twisted depends on the Python modules.

So I am currently stuck between #453 (preventing use of Python modules only), #449 (preventing use of a recipe depending on Python modules) and #451 (preventing use of recipes only). I'm not sure which is the "right way" to be doing this, which doesn't help either. Given that the deprecated Twisted recipe required compilation, I agree that Twisted probably requires a recipe, which implies that #453 is not the right way forward, but that still leaves me stuck 😫

[inclement]

Have you also made a zope recipe (as in p4a master) and made it a dependency of the twisted one? I guess that's how p4a master avoids this problem.

[inclement]

Also, is the setuptools requirement part of the twisted problem? It doesn't seem that the old twisted recipe depends on it, but maybe I missed it.

[str4d]

Yes, I have a zope.interface recipe (a trivial PythonRecipe), and it builds fine. But Twisted also depends on setuptools:

[INFO]:    # Building recipes
[INFO]:    Building hostpython2 for armeabi
[INFO]:    Building sdl2_image for armeabi
[INFO]:    Building sdl2_mixer for armeabi
[INFO]:    Building sdl2_ttf for armeabi
[INFO]:    Building python2 for armeabi
[INFO]:    Building sdl2 for armeabi
[INFO]:    Building zope_interface for armeabi
name is None <class 'pythonforandroid.recipes.zope_interface.ZopeRecipe'>
[INFO]:    zope_interface apparently isn't already in site-packages
[INFO]:    Installing zope_interface into site-packages
[INFO]:    -> directory context /home/str4d/.local/share/python-for-android/build/other_builds/zope_interface/armeabi/zope_interface
[INFO]:    running python.host setup.py install -O2
           working ... Writing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site ...
[INFO]:    <- directory context /home/str4d/dev/git/tahoe-lafs/kivytest
Should remove zope tests etc. here, but skipping for now
[INFO]:    Building twisted for armeabi
name is None <class 'pythonforandroid.recipes.twisted.TwistedRecipe'>
[INFO]:    twisted apparently isn't already in site-packages
[INFO]:    Building compiled components in twisted
[INFO]:    -> directory context /home/str4d/.local/share/python-for-android/build/other_builds/twisted/armeabi/twisted
[INFO]:    running python.host setup.py build_ext -v
           working ... ImportError: No module named setuptools ...                                                 Traceback (most recent call last):
  File "/home/str4d/.local/bin/p4a", line 11, in <module>
    sys.exit(main())
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2797, in main
    ToolchainCL()
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2442, in __init__
    getattr(self, args.command)(unknown)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 169, in wrapper_func
    args = build_dist_from_args(ctx, dist, args)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2221, in build_dist_from_args
    build_recipes(build_order, python_modules, ctx)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2036, in build_recipes
    recipe.build_arch(arch)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 1901, in build_arch
    self.build_compiled_components(arch)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 1910, in build_compiled_components
    shprint(hostpython, 'setup.py', 'build_ext', '-v')
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 129, in shprint
    for line in output:
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 565, in next
    self.wait()
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_1: 

  RAN: '/home/str4d/.local/share/python-for-android/build/python-installs/twisted/bin/python.host setup.py build_ext -v'

  STDOUT:
Traceback (most recent call last):
  File "setup.py", line 13, in <module>
    import setuptools
ImportError: No module named setuptools

and when I made a trivial PythonRecipe for setuptools it caused #451.

I don't know how the old Twisted recipe was working without depending on setuptools. I am running ./distribute.sh -m "twisted" now to find out.

[inclement]

Just another observation, Twisted should probably be a CythonRecipe, not a CompiledComponentsPythonRecipe. Also, the old zope recipe is for all of zope (and includes compilation), not just zope.interface. If you can't work out the problem, maybe moving further to precisely duplicate the old structure would help.

[str4d]

It appears that Twisted 15.2.1 did not depend on setuptools, but Twisted 15.4.0 does (since this ticket was closed). I am trying again now with 15.2.1 (which I'm sure wasn't working either, but I've tried so many things and have so many red herrings that I'm losing track).

AFAICT the old zope recipe does not include compilation, and is only for zope.interface. I'm looking at recipes/zope/recipe.sh in branch master, and its URL is for zope.interface specifically. Is this the wrong recipe?

I had gone with a CompiledComponentsPythonRecipe for Twisted precisely because it appeared to be the closest to the old Twisted recipe, but I will try using a CythonRecipe.

[inclement]

The twisted recipe.sh does the cython stuff at https://github.com/kivy/python-for-android/blob/master/recipes/twisted/recipe.sh#L34 - this is basically the difference between the Cython- and CompiledComponentsPython- recipes; the former does the manual cythoning.

[inclement]

I thought the zope recipe was doing compilation because it bothers setting the ldflags etc (https://github.com/kivy/python-for-android/blob/master/recipes/zope/recipe.sh#L26). Are you doing this? If there's no compilation then I agree it shouldn't be important.

[str4d]

Re: zope, I see it sets the ldflags, but all it does in between them is hostpython setup.py install, so unless I'm misunderstanding something there is no compilation step?

Re: Twisted - ah, my mistake, I missed the CYTHON line. Incidentally, using CompiledComponentsPythonRecipe with Twisted 15.2.1 does get past the dependency stage, and actually throws the same wrong ELF class as PythonReciped setuptools in #451.

[str4d]

Changing to CythonRecipe doesn't help, Twisted tries to import io via its setup.py and throws wrong ELF class. AFAICT the new Twisted recipe is identical to the old one, so this is blocking on whatever environment issue is causing #451.

[inclement]

The problem is that the old twisted recipe actually calls hostpython from its build dir at the install stage, not from the normal hostpython location. I think that's why it specifies the root dir for installation, which wouldn't be necessary otherwise. I think this also means that several of the other commands fail and (I suppose) are ignored in the normal recipe, but in the revamp would raise sh exceptions.

Here's my twisted recipe investigating this, which gets past the ELF problem but hits something else relating to compile settings. I couldn't immediately see the problem but maybe it can help you.

from pythonforandroid.toolchain import CythonRecipe, shprint, current_directory, ArchAndroid, info, Recipe
from os.path import exists, join
import sh
import glob


class TwistedRecipe(CythonRecipe):
    # version = 'stable'
    version = '15.2'
    url = 'http://twistedmatrix.com/Releases/Twisted/{version}/Twisted-{version}.1.tar.bz2'

    depends = ['zope']

    def prebuild_arch(self, arch):
        super(TwistedRecipe, self).prebuild_arch(arch)

        # Need to whitelist tty.pyo and termios.so here

    def get_recipe_env(self, arch):
        env = super(TwistedRecipe, self).get_recipe_env(arch)
        env['PYTHONPATH'] = ':'.join([self.ctx.get_site_packages_dir()])
        print('env is', env)
        return env

    def build_cython_components(self, arch):
        info('Cythonizing anything necessary in {}'.format(self.name))
        env = self.get_recipe_env(arch)
        with current_directory(self.get_build_dir(arch.arch)):
            hostpython = sh.Command(self.ctx.hostpython)
            info('Trying first build of {} to get cython files: this is '
                 'expected to fail'.format(self.name))
            try:
                shprint(hostpython, 'setup.py', 'build_ext', _env=env)
            except sh.ErrorReturnCode_1:
                print()
                info('{} first build failed (as expected)'.format(self.name))

            info('Running cython where appropriate')
            shprint(sh.find, self.get_build_dir('armeabi'), '-iname', '*.pyx', '-exec',
                    self.ctx.cython, '{}', ';', _env=env)
            info('ran cython')

            # shprint(hostpython, 'setup.py', 'build_ext', '-v', _env=env)

            print('stripping')
            build_lib = glob.glob('./build/lib*')
            shprint(sh.find, build_lib[0], '-name', '*.o', '-exec',
                    env['STRIP'], '{}', ';', _env=env)
            print('stripped!?')
            # exit(1)

            # Here we do *not* use the normal hostpython binary in the
            # target python dir, because twisted tries to import
            # _io.so which would fail.
            hostpython_build = sh.Command(join(
                Recipe.get_recipe('hostpython2', self.ctx).get_build_dir('armeabi'),
                'hostpython'))
            shprint(hostpython_build, 'setup.py', 'install', '-O2',
                    '--root={}'.format(self.ctx.get_python_install_dir()),
                    '--install-lib=/lib/python2.7/site-packages', _env=env)


    def postbuild_arch(self, arch):
                sup

recipe = TwistedRecipe()

[str4d]

Thanks for the explanation! It prompted me to delve deeper into what env vars the old recipe was setting, and I finally figured it out. In your get_recipe_env() above I added env['BUILDLIB_PATH'], which contains the correct _io.so; between that and the distinction you pointed out between hostpython and hostpython_build, the wrong ELF class issue is solved.

I now have a built distribution and APK, which I need to test. Once I've verified it is working, I will make a PR. Thanks for all your help!

[str4d]

One problem I forsee is that dists/twisted/private/lib/python2.7/lib-dynload (which contains the wrong ELF class .so files) is being bundled into assets/private.mp3. So this may need to be removed, or maybe the PYTHONPATH on Android will handle it fine. TBC...

[inclement]

The ELF class isn't wrong, it's because they target 32 bit ARM, hence the problem when hostpython tries to import them on the desktop. This happens because the hostpython is really an x86 python binary in an arm python-install, and the workaround here is to use the hostpython from its own (x86) install dir but target the arm one. This is normal and should (barring other issue) run fine on an actual android device.

[str4d]

You're right, and I was able to run Twisted fine on Android as-is. I've made a PR with the recipes.