Problems installing mitmproxy (requires cryptography>=3.3,<3.5 and zstandard>=0.11,<0.16)

[emanuele-f]

Chaquopy version

11.0.0

Relevant parts of your build.gradle file

        ndk {
            abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
        }
        python {
            pip {
                install "mitmproxy"
            }
            pyc {
                src true
            }
        }

Describe your issue

I'm trying to install mitmproxy with your plugin but it fails when trying to install cryptography>=36,<37 and zstandard>=0.11,<0.17 .

Log for cryptography:

Collecting cryptography<37,>=36 (from mitmproxy==8.0.0.dev0)
  Using cached https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f/cryptography-36.0.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.8 -S -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9we8ppnw/cryptography/setup.py'"'"'; __file__='"'"'/tmp/pip-install-9we8ppnw/cryptography/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-9we8ppnw/cryptography/
    Complete output (14 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-9we8ppnw/cryptography/setup.py", line 14, in <module>
        from setuptools_rust import RustExtension
    ModuleNotFoundError: No module named 'setuptools_rust'
    
    ERROR: Command errored out with exit status 1:

            =============================DEBUG ASSISTANCE==========================
            If you are seeing an error here please try the following to
            successfully install cryptography:
    
            Upgrade to the latest pip and try again. This will fix errors for most
            users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
            =============================DEBUG ASSISTANCE==========================
    
    ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Failed to install cryptography<37,>=36 from https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f/cryptography-36.0.1.tar.gz (from mitmproxy==8.0.0.dev0).
For assistance, please raise an issue at https://github.com/chaquo/chaquopy/issues.
Or try using one of the following versions, which are available as pre-built wheels: ['2.8', '3.3.2'].
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Failed to install cryptography<37,>=36 from https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f/cryptography-36.0.1.tar.gz (from mitmproxy==8.0.0.dev0).

Chaquopy: Exit status 1

> Task :app:generateDebugPythonRequirements FAILED
Execution failed for task ':app:generateDebugPythonRequirements'.
> Process 'command 'python3.8'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Log for zstandard:

Collecting zstandard<0.17,>=0.11 (from mitmproxy==8.0.0.dev0)
  Downloading https://files.pythonhosted.org/packages/7d/74/50521e915f23ea5fecee190e933527174a34e8623317369d3b9a13ba109c/zstandard-0.16.0.tar.gz (563kB)
    ERROR: Command errored out with exit status 1:
    ERROR: Command errored out with exit status 1:

     command: /usr/bin/python3.8 -S -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-u4_zlr4a/zstandard/setup.py'"'"'; __file__='"'"'/tmp/pip-install-u4_zlr4a/zstandard/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-u4_zlr4a/zstandard/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-u4_zlr4a/zstandard/setup.py", line 90, in <module>
        setup_zstd.get_c_extension(
      File "/tmp/pip-install-u4_zlr4a/zstandard/setup_zstd.py", line 75, in get_c_extension
        raise Exception("unhandled compiler type: %s" % compiler.compiler_type)
    Exception: unhandled compiler type: disabled
    ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

ERROR: Failed to install zstandard<0.17,>=0.11 from https://files.pythonhosted.org/packages/7d/74/50521e915f23ea5fecee190e933527174a34e8623317369d3b9a13ba109c/zstandard-0.16.0.tar.gz (from mitmproxy==8.0.0.dev0).
Failed to install zstandard<0.17,>=0.11 from https://files.pythonhosted.org/packages/7d/74/50521e915f23ea5fecee190e933527174a34e8623317369d3b9a13ba109c/zstandard-0.16.0.tar.gz (from mitmproxy==8.0.0.dev0).

For assistance, please raise an issue at https://github.com/chaquo/chaquopy/issues.
Chaquopy: Exit status 1

> Task :app:generateDebugPythonRequirements FAILED
Execution failed for task ':app:generateDebugPythonRequirements'.
> Process 'command 'python3.8'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

[mhsmith]

Thanks for the report. This package isn't currently a priority, but I'll update this issue if we add it in the future. And if anyone else wants it too, please click the thumbs-up button above.

[emanuele-f]

Thank you for your feedback. I hope this issue can be fixed soon as chaquopy looks very promising for my project in its ability to easily bundle python modules. Here I explain how I wish to integrate it in PCAPdroid.

I don't know if you have already evaluated this and if it's feasible in your environment, but it would be great if people could contribute to your project by testing and fixing such packaging issues locally and provide PR. This would be a win for your project as you would expand compatibility and visibility and also a possibility for developers to speed up their integration process.

[mhsmith]

You could try using our package build tool to build cryptography and zstandard yourself. Normally this would cost $250 (#175), but because you're an established open-source project, I'll give you access for free. It's in a private GitHub repository, so please check your GitHub notifications for the link.

I've included our existing build files for cryptography. It didn't require any patches, so hopefully all you'll have to do is update the version number and rerun the build.

I don't know anything about zstandard, but I've included instructions for how to add a new package, and several other examples.

If you have any questions, please search the existing issues in the private repository, and create a new one if you still need help.

[emanuele-f]

Thank you, I'm checking this

[emanuele-f]

I've updated the requirements in this issue title as they were taken from the master branch of mitmproxy, whereas the latest mitmproxy version on pip (7.0.4) requires some older versions. In particular, I've built cryptography 3.4.8 and zstandard 0.15.2.

The building process is well documented and smooth. Only cryptography was a little problematic as it requires rust. In order to avoid patching it, I've tried adding setuptools_rust as a build dependency, however it fails at the linker stage as it tries to link against libgcc_s.so.1 . Luckily, it was possible to exclude the rust dependency by setting CRYPTOGRAPHY_DONT_BUILD_RUST via a patch file, however this is not possible in newer cryptography versions. I can post/send you the recipes if they can be useful (e.g. integrating them in your repo), just let me know where.

I've successfully built a poc app which bundles mitmproxy and the PCAPdroid plugin to export the decrypted TLS data. I'm making some adjustments and I will publish it on github soon. Using chaquopy feel like magic, almost everything works out of the box and python errors are reported with a detailed stacktrace which make it fairly easy to troubleshoot them. I only miss these two features:

• lack of ability to run a python script directly. In particular, I had to inspect how the mitmdump program was implemented and created a python script to import it and run it via functions calls. This is feasible, however it would be great to run python scripts directly.
• lack of ability to reset the python environment. I have a problem with mitmdump which does not close the TCP listening socket upon stop, which requires restarting the app on every run. I have to troubleshoot this, however the ability to destroy/recreate or at least reset the python environment could have solved this problem immediately. Also the python environment will probably get dirtier and dirtier on subsequent method invocations.

[mhsmith]

I can post/send you the recipes if they can be useful (e.g. integrating them in your repo), just let me know where.

That would be great, you can make a PR in the other repository, or email them to me if you prefer.

lack of ability to run a python script directly. In particular, I had to inspect how the mitmdump program was implemented and created a python script to import it and run it via functions calls.

In this case the script is auto-generated by the entry_points section in setup.py, which tells you exactly what function you need to call. I agree it might be useful to provide an API to call entry-point scripts in a more command-line-like way. But as you discovered, these scripts often assume they're the only thing that will ever be run in the lifetime of the process, so calling them multiple times doesn't work and you end up needing to use the function call interface anyway.

lack of ability to reset the python environment

Because many native modules store state in static variables, the only reliable way to reset the Python environment is to exit the whole process. This is generally discouraged on Android, but it's still possible with the standard Java or Python APIs. Then you can restart the process with a new intent, either from a different app, or a different process in the same app.

[mhsmith]

Thanks very much for sending me the recipes: I've now built the packages myself and added them to the public repository.

[emanuele-f]

That's great!

I've just published an initial version of the PCAPdroid mitm addon. There are a lot of things to refine but I'm satisfied with the result. Thank you for providing this great framework!