Cannot make install

[malibu1966]

Hi there, when I try to make install crossbar I get the following error:

ERROR: Cannot install autobahn[compress,encryption,scram,serialization,twisted,xbr]==24.4.2, crossbar and crossbar==23.1.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    crossbar 23.1.2 depends on eth-abi 4.0.0b2 (from git+https://github.com/ethereum/eth-abi.git@v4.0.0-beta.2#egg=eth-abi)
    py-eth-sig-utils 0.4.0 depends on eth-abi>=1.1.1
    autobahn[compress,encryption,scram,serialization,twisted,xbr] 24.4.2 depends on eth-abi>=4.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

root@d1ae3f62f327:/code/repos/crossbar# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

UPDATE: At first I thought the answer was to change the line in requirements-latest.txt for eth-abi from 4.0.0-beta.2 to 4.0.0 but then I get the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/code/repos/crossbar/crossbar/__init__.py", line 19, in <module>
    import eth_abi
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/__init__.py", line 3, in <module>
    from eth_abi.abi import (  # NOQA
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/abi.py", line 1, in <module>
    from eth_abi.codec import (
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/codec.py", line 12, in <module>
    from eth_abi.decoding import (
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 8, in <module>
    from eth_utils import (
  File "/opt/venv/lib/python3.9/site-packages/eth_utils/__init__.py", line 86, in <module>
    from .network import (
  File "/opt/venv/lib/python3.9/site-packages/eth_utils/network.py", line 7, in <module>
    from eth_typing import ChainId
ImportError: cannot import name 'ChainId' from 'eth_typing' (/opt/venv/lib/python3.9/site-packages/eth_typing/__init__.py)
make: *** [Makefile:41: install] Error 1

I tried to remove references to eth_typing from the rest of the code but it seems fairly pervasive.

[Skully17]

See #2091
unfortunately, I believe the change is being blocked by the automated tests not working.

[malibu1966]

Thanks for the response. Do you happen to know what was the last working version? I only need wampcra, not doing any of this bitcoin stuff.

I tried 23.1.1 as well and that had the same problem.

Also I tried 22.6.1 which is the next version down and also in pypi. The pip install worked but I get this error on a config.json that works on the crossbar docker image:

Traceback (most recent call last):
  File "/opt/venv/bin/crossbar", line 5, in <module>
    from crossbar import run
  File "/opt/venv/lib/pypy3.9/site-packages/crossbar/__init__.py", line 15, in <module>
    from crossbar._util import hl
  File "/opt/venv/lib/pypy3.9/site-packages/crossbar/_util.py", line 22, in <module>
    from crossbar.common.checkconfig import InvalidConfigException
  File "/opt/venv/lib/pypy3.9/site-packages/crossbar/common/checkconfig.py", line 22, in <module>
    from autobahn.wamp.message import _URI_PAT_STRICT_NON_EMPTY, _URI_PAT_STRICT_LAST_EMPTY, \
ImportError: cannot import name 'identity_realm_name_category' from 'autobahn.wamp.message' (/opt/venv/lib/pypy3.9/site-packages/autobahn/wamp/message.py)

This error happens even when running crossbar with no parameters. I guess I will keep trying lower versions.

I'm on the verge of using this in a commercial product so very interested in getting it going.

UPDATE: I tried pip install of the latest of every major version back to 17.12.1. Unfortunately it looks like autobahn.wamp.cryptosign lost the class SigningKey at some point and it seems to be pulling in the new cryptosign instead of the one that worked with that version so it is erring on not being able to find that for a long way back.

[malibu1966]

In case anyone needs a database back end for their workers, the mysql proper and the mysql-connector-python package which is pure python works on the crossbar container. Unfortunately the standard and recommended mysqlclient and psycopg2 (for postgresql) won't work on the crossbar container because it is built with pypy and modules with C extensions need to be completely recompiled for pypy as far as I understand. Since the mysql driver won't work with mariadb then mariadb is out as an option as well.

[Skully17]

The issue is that Autobahn has been updated with the new eth_abi 4.0.0 version and Crossbar hasn't. This means none of the current Crossbar versions will install/work as they still require the old eth_abi 4.0.0b2 version.
I've been using my own copy of crossbar with the dependency update and others that I need (also in pending PRs into Crossbar). you could do the same or install the branch in the PR with the fix like this:

pip install git@github.com:oberstet/crossbar.git@updating_dependencies

[malibu1966]

So I install crossbar and then I do this?

[Skully17]

This is to install Crossbar. With this version of Crossbar it shouldn't clash with autobahn

[malibu1966]

Thank you very much for your help. This is what I got:

root@b56f8af996e6:/code# pip install git@github.com:oberstet/crossbar.git@updating_dependencies
WARNING: Ignoring invalid distribution -rossbar (/opt/venv/lib/python3.9/site-packages)
Collecting git@ github.com:oberstet/crossbar.git@updating_dependencies
ERROR: Could not install packages due to an OSError: No connection adapters were found for 'github.com:oberstet/crossbar.git@updating_dependencies'

I think updating_dependencies is supposed to be a tag right? I don't see it in the pulldown.

[oberstet]

Just a quick note seeing this flying by: I hope the Ethereum world - dependencies and all that - is finally stabilizing=) It's not quite yet perfect in Crossbar.io, I mean all the build integration and test automation .. but it is on the right track and making progress over time. In my eyes.

psycopg2 (for postgresql) won't work on the crossbar container because it is built with pypy and modules with C extensions need to be completely recompiled for pypy

I've had success in the past using https://pypi.org/project/psycopg2cffi/ This is a proper library for PG which uses CFFI and thus works great on PyPy. I am not currently using PG together with Python a lot, but there also is https://www.psycopg.org/psycopg3/ and of course the general question has been asked before;) https://stackoverflow.com/questions/9350422/how-do-you-get-pypy-django-and-postgresql-to-work-together

[Skully17]

Thank you very much for your help. This is what I got:

Sorry, got the command wrong. Try this:

pip install git+https://github.com/oberstet/crossbar.git@updating_dependencies

I've actually run it myself this time (like I should have done before 🙂 ) and it installs

[malibu1966]

First of all, thanks everyone for providing input and being so helpful.

About psycopg2cffi, thanks for the recommendation. I do remember seeing some stuff on that and I will try it, but it doesn't seem to be very popular and there wasn't a lot on it that I found. Even the link you gave questions in a comment how well supported psycopg2cffi is.

[malibu1966]

So I tried the pip install into a venv and it seems to install fine but when I enter crossbar start I get the following error:

Traceback (most recent call last):
  File "/opt/venv/bin/crossbar", line 5, in <module>
    from crossbar import run
  File "/opt/venv/lib/python3.9/site-packages/crossbar/__init__.py", line 19, in <module>
    import eth_abi
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/__init__.py", line 5, in <module>
    from eth_abi.abi import (
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/abi.py", line 1, in <module>
    from eth_abi.codec import (
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/codec.py", line 13, in <module>
    from eth_abi.decoding import (
  File "/opt/venv/lib/python3.9/site-packages/eth_abi/decoding.py", line 9, in <module>
    from eth_utils import (
  File "/opt/venv/lib/python3.9/site-packages/eth_utils/__init__.py", line 86, in <module>
    from .network import (
  File "/opt/venv/lib/python3.9/site-packages/eth_utils/network.py", line 7, in <module>
    from eth_typing import ChainId
ImportError: cannot import name 'ChainId' from 'eth_typing' (/opt/venv/lib/python3.9/site-packages/eth_typing/__init__.py)

This was a fresh docker image built from python:3.9.12-slim. I also tried python:latest with the same result.

[oberstet]

I don't know .. I am usually using CPython built from raw vanilla upstream sources .. which works

(.venv) oberstet@intel-nuci7:~/scm/typedefint/rfminer$ python
Python 3.12.1 (main, Dec 23 2023, 01:01:09) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from eth_typing import ChainId
>>> 
(.venv) oberstet@intel-nuci7:~/scm/typedefint/rfminer$ pip show eth_typing
Name: eth-typing
Version: 3.5.2
Summary: eth-typing: Common type annotations for ethereum python packages
Home-page: https://github.com/ethereum/eth-typing
Author: The Ethereum Foundation
Author-email: snakecharmers@ethereum.org
License: MIT
Location: /home/oberstet/scm/typedefint/rfminer/.venv/lib/python3.12/site-packages
Requires: typing-extensions
Required-by: eth-abi, eth-keys, eth-utils, py-ecc, web3
(.venv) oberstet@intel-nuci7:~/scm/typedefint/rfminer$ python -V
Python 3.12.1

[malibu1966]

I did pip install psycopg2cffi but it still doesn't work. I think I would need to recompile it.

[oberstet]

whether it is worth the work getting PyPy and PostgreSQL, plus your own Python code work, of course is sth for you to decide! but what I will say is: I have used Autobahn/Crossbario on PyPy and PostgreSQL before - and it is a perfect choice and very hard to beat in various dimensions. At least my experience in real world "big enterprise applications" ...

maybe I should add, there are different ways of integration ... including stuff like writing PL/pgSQL stored procedures and register them as WAMP procedures, or publish WAMP events right from PL/pgSQL procedural code .. all using native database types ... all of them=) it is pretty powerful, but yeah, I am a big PostgreSQL fan ..

[malibu1966]

Well then I would like to know how to get it to work. I did setup.py install for psycopg2cffi and that didn't work either. I'm intending to use the Django ORM so I can't use raw commands.

[oberstet]

how to get it to work. I did setup.py install for psycopg2cffi and that didn't work either.

so one way to approach seems like: reuse the official Docker image for PyPy3.9

docker pull pypy:3.9-7.3.16

and then pip install psycopg2cffi within the installed Python there.

once you have that, you might archive your build image as a Docker file that derives from above PyPy image, and then may or may not also bundle your actual application code ... just my 2cts.

[malibu1966]

I tried this. I made a docker file based on pypy as you recommended. I did pip install Django and pip install psycopg2cffi. Still get the same error:

  File "/opt/pypy/lib/pypy3.9/site-packages/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 or psycopg module")
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 or psycopg module

I found a comment for some code to add to the settings.py to be compatible with psycopg2 and psycopg2cffi:

try:
    import psycopg2
except ImportError:
    # Fall back to psycopg2cffi
    from psycopg2cffi import compat
    compat.register()

But then I get this error:

  File "/opt/pypy/lib/pypy3.9/site-packages/django/db/backends/postgresql/psycopg_any.py", line 77, in <module>
    from psycopg2 import errors, extensions, sql  # NOQA
ImportError: cannot import name 'errors' from 'psycopg2cffi' (/opt/pypy/lib/pypy3.9/site-packages/psycopg2cffi/__init__.py)

It still doesn't look to me like psycopg2cffi is up to date.

[oberstet]

It still doesn't look to me like psycopg2cffi is up to date.

Can you post your Dockerfile? Maybe I see the issue ..

[malibu1966]

Thanks for sticking with this. I haven't had a lot of time in the last bit.

Note that I wasn't doing pip install with the Dockerfile but rather I was signing into the container and running it manually for the sake of investigation, but it should be all the same.

Here is my Dockerfile (paraphrased a bit):

FROM pypy:3.9-7.3.16
EXPOSE 8000
EXPOSE 9000

RUN apt-get -y update && apt-get -y install libpq-dev python3-dev
RUN pip install --upgrade pip
RUN pip install django
RUN pip install psycopg2cffi

I also tried pip install psycopg2-binary. Not sure if that is needed with psycopg2cffi or not, but neither works anyway.

[samspade0069]

In install version 22.6.1 2 files hava a error in imports.

site-packages/crossbar/common/checkconfig.py and site-packages/crossbar/router/auth/cryptosign.py have identity_realm_name_category change to identify_realm_name_category and run correct crossbar

[oberstet]

Here is my Dockerfile (paraphrased a bit):

fwiw, I was checking to see if I might be able to actually look at what you do, or even docker-build try it;) so, nope. anyways, doesn't matter. also, a complete docker file for building has sth more like 140 lines ..

[samspade0069]

Not install in docker image. Install in debian 11 with command pip install crossbar. I had to change the two files to get it to start.

[malibu1966]

With all due respect, why all the manual pip install lines? Shouldn't that be handled through dependencies? Thanks for posting the file, it will be very helpful.