Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolving Home Assistant takes 12 hours with new resolver #10788

Closed
1 task done
balloob opened this issue Jan 12, 2022 · 16 comments
Closed
1 task done

Resolving Home Assistant takes 12 hours with new resolver #10788

balloob opened this issue Jan 12, 2022 · 16 comments
Labels
C: dependency resolution About choosing which dependencies to install type: support User Support

Comments

@balloob
Copy link
Contributor

balloob commented Jan 12, 2022

Description

The new resolver introduced in pip takes 12 hours to resolve Home Assistant, making it unusable for Home Assistant. Our current workaround is using the legacy resolver.

(Time measured on AMD Ryzen 7, 32GB memory)

The legacy resolver is going to be removed in #9631 making pip unusable for Home Assistant.

Home Assistant focuses on privacy and local control and is world’s largest open source home automation platform. In 2020 it was the 2nd most active Python project on GitHub (per State of the Octoverse 2020). We integrate over 1000 different APIs and each API is integrated via a Python package installed via pip (requirements.txt).

All our dependency versions are pinned, but the libraries that we use don’t pin their dependencies. This leads to long resolving time as the new resolver will download all the different potentially compatible versions to find the right one. Libraries don’t pin their dependencies because a library tries to be compatible with as many other projects as possible (we always make sure we don’t rely on HA specific API implementations).

I understand you don’t want to keep the legacy resolver around forever, but we would like to see that the new resolver is usable prior to it’s removal.

Expected behavior

Done in less than 15 minutes.

pip version

21.3.1

Python version

3.10

OS

Debian 11

How to Reproduce

git clone https://www.gitHub.com/home-assistant/core home-assistant-core
cd home-assistant-core
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements_all.txt

Output

No response

Code of Conduct
@balloob balloob added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jan 12, 2022
@pradyunsg pradyunsg added C: dependency resolution About choosing which dependencies to install and removed S: needs triage Issues/PRs that need to be triaged labels Jan 13, 2022
@notatallshaw
Copy link
Member

I can not reproduce, I get ResolutionImpossible pretty quickly (bar download time).

I tested on Ubuntu 20.04.3 LTS and changed the commands a little bit so they could be easily repeated without causing the state of the environment to change:

git clone https://www.gitHub.com/home-assistant/core home-assistant-core
cd home-assistant-core
python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install pip --upgrade
python -m pip download -r requirements_all.txt -d downloads

The ResolutionImpossible seems to related to aiohttp which is set to 3.8.1 in the top level requirements and constraints file but at least one of the requirements discord-py specifies it as aiohttp<3.8.0 and >=3.6.0. Full conflict error I get:

The conflict is caused by:
    The user requested aiohttp==3.8.1
    adax-local 0.1.3 depends on aiohttp>=3.0.6
    accuweather 0.3.0 depends on aiohttp
    adax 0.2.0 depends on aiohttp>=3.0.6
    adguardhome 0.5.0 depends on aiohttp>=3.0.0
    advantage-air 0.2.5 depends on aiohttp
    afsapi 0.0.4 depends on aiohttp>=3.3.2
    aio-geojson-geonetnz-quakes 0.13 depends on aiohttp<4 and >=3.7.4
    aio-geojson-geonetnz-volcano 0.6 depends on aiohttp<4 and >=3.7.4
    aio-geojson-nsw-rfs-incidents 0.4 depends on aiohttp<4 and >=3.7.4
    aioambient 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aioaseko 0.0.1 depends on aiohttp
    aioazuredevops 1.3.5 depends on aiohttp>=3.6.2
    aiobotocore 1.2.2 depends on aiohttp>=3.3.1
    aioeafm 0.1.2 depends on aiohttp<4.0.0 and >=3.6.1
    aioeagle 1.1.0 depends on aiohttp
    aioemonitor 1.0.5 depends on aiohttp>=3.7.0
    aioflo 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aiogithubapi 21.11.0 depends on aiohttp<4.0 and >=3.6.1
    aioguardian 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4post0
    aioharmony 0.2.9 depends on aiohttp
    aiohttp-cors 0.7.0 depends on aiohttp>=1.1
    aiohue 3.0.11 depends on aiohttp
    aiohwenergy 0.6.0 depends on aiohttp
    aiolookin 0.1.0 depends on aiohttp>=3.7.4
    aiolyric 1.0.8 depends on aiohttp>=3.7.3
    aiomodernforms 0.1.8 depends on aiohttp>=3.0.0
    aiomusiccast 0.14.3 depends on aiohttp<4.0.0 and >=3.7.4
    aionanoleaf 0.1.1 depends on aiohttp
    aionotion 3.0.2 depends on aiohttp<4.0.0 and >=3.7.4
    aiopvapi 1.6.19 depends on aiohttp<4 and >=3.7.4
    aiopvpc 3.0.0 depends on aiohttp>=3.7.4.post0
    aiorecollect 1.0.8 depends on aiohttp<4.0.0 and >=3.7.4
    aioridwell 2021.12.2 depends on aiohttp>=3.8.0
    aioshelly 1.0.7 depends on aiohttp
    aiosteamist 0.3.1 depends on aiohttp<4.0.0 and >=3.8.1
    aiosyncthing 0.5.1 depends on aiohttp>=3.7.4
    aiotractive 0.5.2 depends on aiohttp>=3.7.4
    aiounifi 29 depends on aiohttp
    aiowatttime 0.1.1 depends on aiohttp<4.0.0 and >=3.7.4
    aioymaps 1.2.2 depends on aiohttp>=3.0.0
    airly 1.1.0 depends on aiohttp>=3.5.4
    airthings-cloud 0.1.0 depends on aiohttp>=3.0.6
    alpha-vantage 2.3.1 depends on aiohttp
    ambee 0.4.0 depends on aiohttp>=3.0.0
    ambiclimate 0.2.1 depends on aiohttp>=3.0.6
    asmog 0.0.6 depends on aiohttp
    async-upnp-client 0.23.3 depends on aiohttp>=3.7.4
    asyncpysupla 0.0.5 depends on aiohttp
    blebox-uniapi 1.3.3 depends on aiohttp>=3
    bond-api 0.1.15 depends on aiohttp>=3.6.1
    brunt 1.1.1 depends on aiohttp
    bsblan 0.4.0 depends on aiohttp>=3.0.0
    connect-box 0.2.8 depends on aiohttp
    coronavirus 1.1.1 depends on aiohttp>=3.0.0
    crownstone-cloud 1.4.9 depends on aiohttp~=3.7
    crownstone-sse 2.0.3 depends on aiohttp~=3.7
    directv 0.4.0 depends on aiohttp>=3.6.2
    discord-py 1.7.3 depends on aiohttp<3.8.0 and >=3.6.0
    The user requested (constraint) aiohttp==3.8.1

@pradyunsg
Copy link
Member

Thanks for filing this @balloob! ^>^

While I haven't started using Home Assistant yet, it's been on my radar and I appreciate the work that you and other volunteers who have worked on it, have put into it!

Much like @notatallshaw, I'm unable to reproduce this with pip 21.3.1 on MacOS, with a ResolutionImpossible error after 4m 4s with unpopulated cache (i.e. everything had to download). With a populated cache, that takes 2m 0s.

The error is that exact same as @notatallshaw, as far as I can tell.

The conflict is caused by:
    The user requested aiohttp==3.8.1
    adax-local 0.1.3 depends on aiohttp>=3.0.6
    accuweather 0.3.0 depends on aiohttp
    adax 0.2.0 depends on aiohttp>=3.0.6
    adguardhome 0.5.0 depends on aiohttp>=3.0.0
    advantage-air 0.2.5 depends on aiohttp
    afsapi 0.0.4 depends on aiohttp>=3.3.2
    aio-geojson-geonetnz-quakes 0.13 depends on aiohttp<4 and >=3.7.4
    aio-geojson-geonetnz-volcano 0.6 depends on aiohttp<4 and >=3.7.4
    aio-geojson-nsw-rfs-incidents 0.4 depends on aiohttp<4 and >=3.7.4
    aioambient 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aioaseko 0.0.1 depends on aiohttp
    aioazuredevops 1.3.5 depends on aiohttp>=3.6.2
    aiobotocore 1.2.2 depends on aiohttp>=3.3.1
    aioeafm 0.1.2 depends on aiohttp<4.0.0 and >=3.6.1
    aioeagle 1.1.0 depends on aiohttp
    aioemonitor 1.0.5 depends on aiohttp>=3.7.0
    aioflo 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aiogithubapi 21.11.0 depends on aiohttp<4.0 and >=3.6.1
    aioguardian 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4post0
    aioharmony 0.2.9 depends on aiohttp
    aiohttp-cors 0.7.0 depends on aiohttp>=1.1
    aiohue 3.0.11 depends on aiohttp
    aiohwenergy 0.6.0 depends on aiohttp
    aiolookin 0.1.0 depends on aiohttp>=3.7.4
    aiolyric 1.0.8 depends on aiohttp>=3.7.3
    aiomodernforms 0.1.8 depends on aiohttp>=3.0.0
    aiomusiccast 0.14.3 depends on aiohttp<4.0.0 and >=3.7.4
    aionanoleaf 0.1.1 depends on aiohttp
    aionotion 3.0.2 depends on aiohttp<4.0.0 and >=3.7.4
    aiopvapi 1.6.19 depends on aiohttp<4 and >=3.7.4
    aiopvpc 3.0.0 depends on aiohttp>=3.7.4.post0
    aiorecollect 1.0.8 depends on aiohttp<4.0.0 and >=3.7.4
    aioridwell 2021.12.2 depends on aiohttp>=3.8.0
    aioshelly 1.0.7 depends on aiohttp
    aiosteamist 0.3.1 depends on aiohttp<4.0.0 and >=3.8.1
    aiosyncthing 0.5.1 depends on aiohttp>=3.7.4
    aiotractive 0.5.2 depends on aiohttp>=3.7.4
    aiounifi 29 depends on aiohttp
    aiowatttime 0.1.1 depends on aiohttp<4.0.0 and >=3.7.4
    aioymaps 1.2.2 depends on aiohttp>=3.0.0
    airly 1.1.0 depends on aiohttp>=3.5.4
    airthings-cloud 0.1.0 depends on aiohttp>=3.0.6
    alpha-vantage 2.3.1 depends on aiohttp
    ambee 0.4.0 depends on aiohttp>=3.0.0
    ambiclimate 0.2.1 depends on aiohttp>=3.0.6
    asmog 0.0.6 depends on aiohttp
    async-upnp-client 0.23.3 depends on aiohttp>=3.7.4
    asyncpysupla 0.0.5 depends on aiohttp
    blebox-uniapi 1.3.3 depends on aiohttp>=3
    bond-api 0.1.15 depends on aiohttp>=3.6.1
    brunt 1.1.1 depends on aiohttp
    bsblan 0.4.0 depends on aiohttp>=3.0.0
    connect-box 0.2.8 depends on aiohttp
    coronavirus 1.1.1 depends on aiohttp>=3.0.0
    crownstone-cloud 1.4.9 depends on aiohttp~=3.7
    crownstone-sse 2.0.3 depends on aiohttp~=3.7
    directv 0.4.0 depends on aiohttp>=3.6.2
    discord-py 1.7.3 depends on aiohttp<3.8.0 and >=3.6.0
    The user requested (constraint) aiohttp==3.8.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

You are likely getting a warning from the legacy resolver about dependency conflicts as well -- it just does not fail when it encounters a dependency conflict.

@notatallshaw
Copy link
Member

notatallshaw commented Jan 13, 2022 •
edited
Loading

I experimented to see if I could "fix" the ResolutionImpossible, I tried slowly loosening the requirements to allow pip to find compatible versions. This revealed a lot of conflicting requirements based on aiohttp, websockets, click, boto3, etc..

Here are two cases I found which seemed unresolvable:

  1. mycroftapi==2.0 depends on the very old websocket-client==0.44.0. But mycroftapi but it hasn't been updated in over 3 years and the home page link returns a 404: https://pypi.org/project/mycroftapi/ . Rather than trying to find very old versions of the 4 projects that require newer than websocket-client==0.44.0 I just dropped the mycroftapi requirement.
  2. pysmarty 0.8 requires pymodbus==1.5.2 and pystiebeleltron 0.0.1.dev2 requires pymodbus>=2.1.0. This doesn't seem resolvable as pysmarty doesn't seem to be updated any more (no update in 2-3 years) and pystiebeleltron only has 2 non-release versions and no updates in over 1 year. I drop[ed pystiebeleltron from the requirements as it is an unreleased version.

I continued loosening requirements but unfortunately this eventually led to very long backtracking times. I used an experimental version of Pip I have which implements backjumping to try and speed up these very complex scenarios and continued trying to loosen the requirements.

Eventually I was able to find a set of requirements that didn't conflict (minus the two packages above), and after switching to Python 3.9 to build them (I could not get homeassistant-pyozw to build on 3.10). Here is the output of pip freeze, perhaps you can compare it to a pip freeze in a working environment you have, and update them 1 at a time to see what causes conflicts:

Pip Freeze Output 
abodepy==1.2.0
accuweather==0.3.0
acme==1.22.0
Adafruit-Blinka==5.5.1
adafruit-circuitpython-bmp280==3.1.1
adafruit-circuitpython-busdevice==5.1.1
adafruit-circuitpython-dht==3.7.0
adafruit-circuitpython-mcp230xx==2.2.2
adafruit-circuitpython-pca9685==3.3.2
adafruit-circuitpython-register==1.9.7
Adafruit-GPIO==1.0.3
Adafruit-PlatformDetect==3.19.2
Adafruit-PureIO==1.1.9
Adafruit-SHT31==1.0.2
adal==1.2.7
adax==0.2.0
Adax-local==0.1.3
adb-shell==0.4.0
adext==0.4.2
adguardhome==0.5.0
advantage-air==0.2.5
AEMET-OpenData==0.2.1
aenum==3.1.5
afsapi==0.0.4
agent-py==0.0.23
aio-geojson-client==0.15
aio-geojson-geonetnz-quakes==0.13
aio-geojson-geonetnz-volcano==0.6
aio-geojson-nsw-rfs-incidents==0.4
aio-georss-client==0.8
aio-georss-gdacs==0.5
aioambient==2.0.0
aioaseko==0.0.1
aioasuswrt==1.4.0
aioazuredevops==1.3.5
aiobotocore==2.1.0
aiocache==0.11.1
aiocoap==0.4.3
aioconsole==0.3.3
aiodiscover==1.4.5
aiodns==3.0.0
aioeafm==0.1.2
aioeagle==1.1.0
aioemonitor==1.0.5
aioesphomeapi==10.6.0
aiofiles==0.8.0
aioflo==2021.11.0
aioftp==0.12.0
aiogithubapi==21.11.0
aioguardian==2021.11.0
aioharmony==0.2.9
aiohomekit==0.6.10
aiohttp==3.7.4.post0
aiohttp-cors==0.7.0
aiohue==3.0.11
aiohwenergy==0.6.0
aioimaplib==0.9.0
aioitertools==0.8.0
aiokafka==0.6.0
aiokef==0.2.16
aiolifx==0.7.0
aiolifx-effects==0.2.2
aiolookin==0.1.0
aiolyric==1.0.8
aiomodernforms==0.1.8
aiomusiccast==0.14.3
aionanoleaf==0.1.1
aionotify==0.2.0
aionotion==3.0.2
aiooncue==0.3.0
aiopulse==0.4.3
aiopvapi==1.6.19
aiopvpc==3.0.0
aiopylgtv==0.4.0
aiorecollect==1.0.8
aioridwell==2021.10.0
aiosenseme==0.5.5
aioserial==1.3.0
aioshelly==1.0.7
aioshutil==1.1
aiosqlite==0.17.0
aiosteamist==0.1.0
aioswitcher==2.0.6
aiosyncthing==0.5.1
aiotractive==0.5.2
aiounifi==29
aiovlc==0.1.0
aiowatttime==0.1.1
aioymaps==1.2.2
airly==1.1.0
airthings-cloud==0.1.0
airtouch4pyapi==1.0.5
alabaster==0.7.12
aladdin-connect==0.3
alarmdecoder==1.13.11
alpha-vantage==2.3.1
ambee==0.4.0
amberelectric==1.0.3
Ambiclimate==0.2.1
amcrest==1.9.3
android-backup==0.2.0
androidtv==0.0.60
anel-pwrctrl-homeassistant==0.0.1.dev2
anthemav==1.2.0
anyio==3.5.0
apcaccess==0.0.13
apns2==0.3.0
appdirs==1.4.4
apprise==0.9.6
aprslib==0.6.46
APScheduler==3.6.3
aqualogic==2.6
arcam-fmj==0.12.0
argcomplete==2.0.0
arris-tg2492lg==1.1.0
arrow==1.2.1
asmog==0.0.6
asn1==2.4.2
asterisk-mbox==0.5.0
astral==2.2
astroid==2.9.3
async-timeout==3.0.1
async-upnp-client==0.23.3
asyncclick==7.1.2.3
asynccmd==0.2.4
asyncio==3.4.3
asyncio-dgram==2.1.1
asyncio-mqtt==0.11.1
asyncio-throttle==1.0.2
asyncpysupla==0.0.5
asyncssh==2.8.1
asyncstdlib==3.10.2
asynctest==0.13.0
atenpdu==0.3.2
atomicwrites==1.4.0
attrs==21.2.0
auroranoaa==0.0.2
aurorapy==0.2.6
Authlib==0.15.5
av==8.1.0
awesomeversion==22.1.0
axis==44
azure-common==1.1.27
azure-core==1.21.1
azure-eventhub==5.5.0
azure-servicebus==0.50.3
Babel==2.9.1
backoff==1.11.1
baidu-aip==1.6.6.0
base36==0.1.1
batinfo==0.4.2
bcrypt==3.1.7
beautifulsoup4==4.10.0
bellows==0.29.0
bimmer-connected==0.8.7
bitarray==2.3.5
bitstring==3.1.9
bizkaibus==0.1.1
bleak==0.14.1
blebox-uniapi==1.3.3
blinkpy==0.18.0
BlinkStick==1.2.0
blockchain==1.4.4
blurhash==1.1.4
bond-api==0.1.15
boschshcpy==0.2.28
boto3==1.20.24
botocore==1.23.24
bravia-tv==1.0.11
broadlink==0.18.0
brother==1.1.0
brotlipy==0.7.0
brottsplatskartan==0.0.1
brunt==1.1.1
bs4==0.0.1
bsblan==0.4.0
bt-proximity==0.2.1
bthomehub5-devicelist==0.1.1
btlewrap==0.0.10
btsmarthub-devicelist==0.2.0
buienradar==1.0.5
cached-property==1.5.2
cachetools==4.2.4
caldav==0.8.2
casttube==0.2.1
cattrs==1.10.0
cbor2==5.4.2
cchardet==2.1.7
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
chardet==4.0.0
charset-normalizer==2.0.10
circuit-webhook==1.0.1
ciscomobilityexpress==0.3.9
ciso8601==2.2.0
clearpasspy==1.0.2
click==7.1.2
click-log==0.3.2
click-plugins==1.1.1
clx-sdk-xms==1.0.0
CO2Signal==0.4.2
coinbase==2.1.0
colorama==0.4.4
coloredlogs==15.0.1
colorlog==6.6.0
colorthief==0.2.1
colorzero==2.0
colour==0.1.5
commentjson==0.9.0
commonmark==0.9.1
concord232==0.15
config==0.5.1
ConfigArgParse==1.5.3
configparser==5.2.0
connect-box==0.2.8
construct==2.10.56
convertdate==2.3.2
coronavirus==1.1.1
coverage==6.2
crc16==0.1.1
crccheck==1.1
crcmod==1.7
croniter==1.0.6
crownstone-cloud==1.4.9
crownstone-core==3.1.0
crownstone-sse==2.0.3
crownstone-uart==2.1.0
cryptography==35.0.0
cssselect==1.1.0
cycler==0.11.0
Cython==0.29.26
dacite==1.6.0
dataclasses-json==0.5.3
datadog==0.15.0
datapoint==0.9.8
dateparser==1.1.0
DateTime==4.3
dbus-next==0.2.3
debugpy==1.5.1
decorator==5.1.1
deepmerge==1.0.1
defusedxml==0.7.1
deluge-client==1.7.1
demjson3==3.0.5
denonavr==0.10.9
Deprecated==1.2.13
devolo-home-control-api==0.17.4
devolo-plc-api==0.7.1
dicttoxml==1.7.4
directv==0.4.0
discogs-client==2.3.0
discord.py==1.7.3
discovery30303==0.2.1
distlib==0.3.4
dlipower==0.7.165
dnspython==2.1.0
docopt==0.6.2
docutils==0.17.1
DoorBirdPy==2.1.0
dovado==0.4.1
dsmr-parser==0.32
DTLSSocket==0.1.12
dwdwfsapi==1.0.4
dweepy==0.3.0
dynalite-devices==0.1.46
ebusdpy==0.0.17
ecdsa==0.17.0
ecoaliface==0.4.0
elgato==3.0.0
eliqonline==1.2.2
elkm1-lib==1.0.0
elmax-api==0.0.2
emoji==1.5.0
emulated-roku==0.2.1
enocean==0.50.0
enturclient==0.2.2
enum-compat==0.0.3
env-canada==0.5.20
envoy-reader==0.20.1
envoy-utils==0.0.1
envs==1.4
ephem==3.7.7.0
epson-projector==0.4.2
epsonprinter==0.0.9
esprima==4.0.1
eternalegypt==0.0.12
evohome-async==0.3.15
faadelays==0.0.7
fake-useragent==0.1.11
fastdotcom==0.0.3
feedparser==6.0.2
fiblary3==0.1.7
filelock==3.4.2
fints==1.0.1
fitbit==0.3.1
fixerio==1.0.0a0
fjaraskupan==1.0.2
Flask==2.0.2
flipr-api==1.4.1
flux-led==0.28.2
fnvhash==0.1.0
fonttools==4.28.5
foobot-async==1.0.0
forecast-solar==2.1.0
fortiosapi==1.0.5
freebox-api==0.0.10
freesms==0.2.0
fritzconnection==1.8.0
future==0.18.2
garages-amsterdam==3.0.0
gcloud==0.18.3
geniushub-client==0.6.30
geographiclib==1.52
geojson==2.5.0
geojson-client==0.6
geopy==2.1.0
georss-client==0.14
georss-generic-client==0.6
georss-ign-sismologia-client==0.3
georss-qld-bushfire-alert-client==0.5
get-mac==0.8.3
getmac==0.8.2
gios==2.1.0
gitterpy==0.1.7
glances-api==0.2.0
gntp==1.0.3
goalzero==0.2.1
goodwe==0.2.10
google-api-core==1.31.5
google-api-python-client==1.6.4
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-cloud-pubsub==2.9.0
google-cloud-texttospeech==0.4.0
google-nest-sdm==1.3.0
googleapis-common-protos==1.54.0
googlemaps==2.5.1
goslide-api==0.5.1
gpiozero==1.5.1
gps3==0.33.3
graphql-subscription-manager==0.4.3
graphviz==0.19.1
greeclimate==1.0.1
greeneye-monitor==3.0.1
greenlet==1.1.2
greenwavereality==0.5.1
growattServer==1.1.0
grpc-google-iam-v1==0.12.3
grpcio==1.43.0
gstreamer-player==1.1.2
gTTS==2.2.3
guppy3==3.1.2
h11==0.12.0
h2==2.6.2
ha-ffmpeg==3.0.2
ha-philipsjs==2.7.6
habitipy==0.2.0
hangups==0.4.17
HAP-python==4.4.0
hass-nabucasa==0.51.0
hass-splunk==0.1.1
HATasmota==0.3.1
haversine==2.5.1
hbmqtt==0.9.6
hdate==0.10.4
heatmiserV3==1.1.18
herepy==2.0.0
hijri-converter==2.2.2
hikvision==0.4
hkavr==0.0.5
hlk-sw16==0.0.9
hole==0.6.0
holidays==0.12
home-assistant-frontend==20211229.1
homeassistant-pyozw==0.1.10
homeconnect==0.6.3
homematicip==1.0.1
homepluscontrol==0.0.5
horimote==0.4.1
hpack==3.0.0
html-table-parser-python3==0.1.5
httmock==1.4.0
http-ece==1.1.0
httpcore==0.14.4
httplib2==0.19.0
httpsig==1.3.0
httpx==0.21.0
huawei-lte-api==1.4.18
huisbaasje-client==0.1.0
humanfriendly==10.0
Hydrawiser==0.2
hyper==0.7.0
hyperframe==3.2.0
hyperion-py==0.7.4
iammeter==0.1.7
iaqualink==0.4.1
ibm-cloud-sdk-core==3.13.2
ibm-watson==5.2.2
ibmiotf==0.3.4
icmplib==3.0
identify==2.4.3
idna==3.3
ifaddr==0.1.7
iglo==1.2.7
ihcsdk==2.7.0
imageio==2.13.5
imagesize==1.3.0
importlib-metadata==4.10.0
incomfort-client==0.4.4
inflection==0.5.1
influxdb==5.3.1
influxdb-client==1.24.0
iniconfig==1.1.1
intelhex==2.3.0
iotawattpy==0.1.0
iperf3==0.1.11
ismartgate==4.0.4
iso4217==1.7.20211001
iso8601==1.0.2
isodate==0.6.1
isort==5.10.1
itsdangerous==2.0.1
jellyfin-apiclient-python==1.7.2
Jinja2==3.0.3
jmespath==0.10.0
josepy==1.12.0
Js2Py==0.71
jsonpatch==1.32
jsonpath==0.82
jsonpickle==2.0.0
jsonpointer==2.2
jsonrpc-async==2.1.0
jsonrpc-base==2.1.0
jsonrpc-websocket==3.1.1
jsonschema==4.4.0
justbackoff==0.6.0
jwcrypto==1.0
kafka-python==2.0.2
kaiterra-async-client==0.0.2
keba-kecontact==1.1.0
kiwiki-client==0.1.1
kiwisolver==1.3.2
konnected==1.2.0
korean-lunar-calendar==0.2.1
kostal-plenticore==0.2.0
krakenex==2.1.0
lakeside==0.12
lark-parser==0.7.8
lazy-object-proxy==1.7.1
libcst==0.4.0
libpyfoscam==1.0
libpyvivotek==0.4.0
librouteros==3.2.0
libsoundtouch==0.8.0
life360==4.1.1
lightify==1.0.7.3
lightwave==0.19
limitlessled==1.1.3
linode-api==4.1.9b1
lmnotify==0.0.4
locationsharinglib==4.1.5
logi-circle==0.2.2
loguru==0.5.3
lomond==0.3.3
london-tube-status==0.2
luftdaten==0.7.1
lupupy==0.0.24
lw12==0.9.2
lxml==4.7.1
mac-vendor-lookup==0.1.11
magicseaweed==1.0.3
Markdown==3.3.6
MarkupSafe==2.0.1
marshmallow==3.14.1
marshmallow-dataclass==8.5.3
marshmallow-enum==1.5.1
Mastodon.py==1.5.1
matplotlib==3.5.1
matrix-client==0.4.0
maxcube-api==0.4.3
mbddns==0.1.2
mccabe==0.6.1
mcstatus==6.0.0
MechanicalSoup==0.12.0
mediafile==0.9.0
messagebird==1.2.0
metar==1.9.0
meteoalertapi==0.2.0
meteofrance-api==1.0.2
mficlient==0.3.0
micloud==0.5
miflora==0.7.0
mill-local==0.1.1
millheater==0.9.0
miniaudio==1.46
minio==5.0.10
mitemp-bt==0.0.5
mock==4.0.3
motionblinds==0.5.8
motioneye-client==0.3.12
ms-cv==0.1.1
msgpack==1.0.3
msrest==0.6.21
msrestazure==0.6.4
mt-940==4.23.0
mullvad-api==1.0.0
multidict==5.2.0
mutagen==1.45.1
mutesync==0.0.1
mypy-extensions==0.4.3
nad-receiver==0.3.0
natsort==8.0.2
nclib==1.0.1
ndms2-client==0.1.1
nessclient==0.9.15
netaddr==0.8.0
netdata==1.0.1
netdisco==3.0.0
netifaces==0.11.0
netmap==0.7.0.2
nettigo-air-monitor==1.2.1
neurio==0.3.1
nexia==0.9.13
nextcloudmonitor==1.1.0
niko-home-control==0.2.1
niluclient==0.1.2
noaa-coops==0.1.8
nodeenv==1.6.0
noiseprotocol==0.3.1
notifications-android-tv==0.1.3
notify-events==1.0.4
nsapi==3.0.5
nsw-fuel-api-client==1.1.0
nuheat==0.3.0
numato-gpio==0.10.0
numpy==1.21.4
oasatelematics==0.3
oauth2client==4.1.3
oauthlib==3.1.1
objgraph==3.4.1
oemthermostat==1.1.1
omnilogic==0.4.5
ondilo==0.2.0
onkyo-eiscp==1.2.7
onvif-zeep-async==1.2.0
open-garage==0.2.0
open-meteo==0.0.1
openerz-api==0.1.0
openevsewifi==1.1.0
openhomedevice==2.0.1
opensensemap-api==0.1.5
openwebifpy==3.2.7
openwrt-luci-rpc==1.1.11
openwrt-ubus-rpc==0.0.2
OPi.GPIO==0.5.2
oru==0.1.11
orvibo==1.1.1
ovoenergy==1.1.12
oyaml==1.0
p1monitor==1.0.1
packaging==21.3
paho-mqtt==1.6.1
panacotta==0.1
panasonic-viera==0.3.6
pandas==1.3.5
paramiko==2.9.2
passlib==1.7.4
pbr==5.8.0
pcal9535a==0.7
pdunehd==1.3.2
pencompy==0.0.3
pexpect==4.6.0
phone-modem==0.1.1
pi1wire==0.1.0
pi4ioe5v9xxxx==0.0.2
pifacecommon==4.2.2
pifacedigitalio==3.0.5
piglow==1.2.4
pigpio==1.78
pilight==0.1.1
Pillow==9.0.0
pizzapi==0.0.3
pkce==1.0.3
platformdirs==2.4.1
PlexAPI==4.7.1
plexauth==0.0.6
plexwebsocket==0.0.13
pluggy==1.0.0
plugwise==0.8.5
plumbum==1.7.2
plumlightpad==0.0.11
ply==3.11
pmsensor==0.4
poolsense==0.0.8
pprintpp==0.4.0
praw==7.4.0
prawcore==2.3.0
prayer-times-calculator==0.0.5
pre-commit==2.16.0
prettytable==3.0.0
ProgettiHWSW==0.1.1
proliphix==0.4.1
prometheus-client==0.7.1
proto-plus==1.19.8
protobuf==3.19.3
proxmoxer==1.1.1
psutil==5.8.0
ptyprocess==0.7.0
pubnub==5.5.0
pulsectl==20.2.4
pure-pcapy3==1.0.1
pure-python-adb==0.3.0.dev0
pushbullet.py==0.11.0
pushover-complete==1.1.1
pvo==0.2.0
pwmled==1.6.7
py==1.11.0
py-canary==0.5.1
py-cpuinfo==8.0.0
py-melissa-climate==2.1.4
py-nextbusnext==0.1.5
py-nightscout==1.2.2
py-schluter==0.1.7
py-synologydsm-api==1.0.5
py-vapid==1.8.2
py-zabbix==1.1.7
py17track==2021.12.0
pyads==3.2.2
pyaehw4a1==0.3.9
pyaes==1.6.1
pyaftership==0.1.2
pyairnow==1.1.0
pyairvisual==5.0.9
pyalmond==0.0.2
pyarlo==0.2.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyatag==0.3.5.3
pyatmo==6.2.2
pyAtome==0.1.1
pyatv==0.9.8
pybalboa==0.13
pybbox==0.0.5a0
pyblackbird==0.5
pybotvac==0.0.23
pycares==4.1.2
pycarwings2==2.13
pyCEC==0.5.1
pycfdns==1.2.2
pychannels==1.0.0
PyChromecast==10.2.3
pycketcasts==1.0.0
pyclimacell==0.18.2
pycmus==0.1.1
pycocotools==2.0.1
pycognito==2022.1.0
pycomfoconnect==0.4
pyControl4==0.0.6
pycoolmasternet-async==0.1.2
pycountry==22.1.10
pycountry-convert==0.7.2
pycparser==2.21
pycryptodome==3.12.0
pycryptodomex==3.12.0
pycsspeechtts==1.0.4
pydaikin==2.7.0
pydanfossair==0.1.0
pydantic==1.9.0
pydeconz==85
pydelijn==0.6.1
pydexcom==0.2.2
PyDispatcher==2.0.5
pydoods==1.0.2
pydroid-ipcam==0.8
pyebox==1.1.4
pyeconet==0.1.14
pyedimax==0.2.1
pyefergy==0.1.5
pyEight==0.2.0
pyEmby==1.8
pyenvisalink==4.0
pyephember==0.3.1
pyeverlights==0.1.0
pyevilgenius==1.0.0
pyezviz==0.2.0.5
pyfido==2.1.1
pyfireservicerota==0.0.43
pyflic==2.0.3
PyFlick==0.0.2
PyFlume==0.6.5
pyflunearyou==2.0.2
pyfnip==0.2
pyforked-daapd==0.1.11
pyfreedompro==1.1.0
pyfritzhome==0.6.2
PyFronius==0.7.1
pyftdi==0.53.3
pyfttt==0.3
pygatt==4.0.5
Pygments==2.11.2
pygtfs==0.1.6
pygti==0.9.2
pyhaversion==21.11.1
pyheos==0.7.2
pyHik==0.3.0
pyhiveapi==0.4.2
pyhomematic==0.1.77
pyhomeworks==0.0.6
pyhumps==3.5.0
pyialarm==1.9.0
pyicloud==0.7.3
pyinsteon==1.0.14
pyintesishome==1.7.6
pyipma==2.0.5
pyipp==0.11.0
pyiqvia==2021.11.0
pyirishrail==0.0.2
pyiss==1.0.1
pyisy==3.0.1
pyitachip2ir==0.0.7
pyjsparser==2.7.1
PyJWT==2.1.0
pykira==0.1.1
pykmtronic==0.3.0
pykodi==0.2.7
pykrakenapi==0.1.8
pykulersky==0.5.2
pykwb==0.0.8
pylacrosse==0.4
pylast==4.2.1
pylaunches==1.2.1
pylgnetcast==0.3.7
pylibrespot-java==0.1.0
pylint==2.12.2
pylitejet==0.3.0
pylitterbot==2021.12.0
pylutron==0.2.8
pylutron-caseta==0.13.0
pymailgunner==1.4
pymata-express==1.19
pymazda==0.2.2
pymediaroom==0.6.4.1
PyMeeus==0.5.11
pymelcloud==2.5.6
PyMetEireann==2021.8.0
pymeteoclimatic==0.0.6
PyMetno==0.9.0
pymfy==0.11.0
pymitv==1.4.3
pymochad==0.2.0
pymodbus==1.5.2
pymonoprice==0.3
pymsteams==0.1.12
PyMVGLive==1.1.4
pymyq==3.1.4
pymysensors==0.22.1
PyNaCl==1.4.0
pynetgear==0.8.0
pynetio==0.1.9.1
PyNINA==0.1.4
pynuki==1.5.2
pynut2==2.1.2
pynws==1.3.2
pynx584==0.5
pynzbgetapi==0.2.0
pyobihai==1.3.1
pyoctoprintapi==0.1.7
pyombi==0.1.10
pyOpenSSL==21.0.0
pyopenuv==2021.11.0
pyopnsense==0.2.0
pyoppleio==1.0.5
pyotgw==1.1b1
pyotp==2.6.0
pyoverkiz==1.1.1
pyowm==3.2.0
pyownet==0.10.0.post1
pyparsing==2.4.7
pypca==0.0.7
pypck==0.7.12
pypjlink2==1.2.1
pyplaato==0.0.15
pypoint==2.2.1
pyprof2calltree==1.4.5
pyprosegur==0.0.5
pyps4-2ndscreen==1.2.0
Pypubsub==4.0.3
PyQRCode==1.2.1
pyquery==1.4.3
pyqvrpro==0.52
pyqwikswitch==0.93
pyrail==0.0.3
pyrainbird==0.4.3
Pyrebase4==4.5.0
pyrecswitch==1.0.2
pyrepetierng==0.1.0
pyRFC3339==1.1
pyRFXtrx==0.27.0
pyrisco==0.3.1
pyrituals==0.0.6
PyRMVtransport==0.3.2
pyroute2==0.6.5
pyroute2.core==0.6.5
pyroute2.ethtool==0.6.5
pyroute2.ipdb==0.6.5
pyroute2.ipset==0.6.5
pyroute2.ndb==0.6.5
pyroute2.nftables==0.6.5
pyroute2.nslink==0.6.5
pyrsistent==0.18.0
pyruckus==0.12
pysabnzbd==1.1.0
pysaj==0.0.16
pySDCP==1
pysensibo==1.0.3
pyserial==3.5
pyserial-asyncio==0.5
pysesame2==1.0.1
Pysher==1.0.1
pysiaalarm==3.0.2
pysignalclirestapi==0.3.4
pyskyqhub==0.1.4
pysma==0.6.10
pysmappee==0.2.29
pysmartapp==0.3.3
pysmartthings==0.7.6
pysmarty==0.8
pysmb==1.2.7
pysmi==0.3.4
pysml==0.0.5
pysnmp==4.4.12
PySocks==1.7.1
pysoma==0.0.10
pyspcwebgw==0.4.0
pysqueezebox==0.5.5
pysuez==0.1.19
PySyncThru==0.7.10
pytankerkoenig==0.0.6
pytautulli==21.11.0
pytest==6.2.5
pytest-cov==2.12.1
pytest-mock==3.6.1
pytest-rerunfailures==10.2
pytest-runner==5.3.1
pytfiac==0.4
pythinkingcleaner==0.0.3
python-awair==0.2.1
python-blockchain-api==0.0.2
python-clementine-remote==1.0.1
python-dateutil==2.8.2
python-didl-lite==1.3.2
python-digitalocean==1.13.2
python-dotenv==0.19.2
python-ecobee-api==0.2.14
python-engineio==3.14.2
python-etherscan-api==0.0.3
python-family-hub-local==0.0.2
python-forecastio==1.4.0
python-gc100==1.0.3a0
python-gitlab==1.6.0
python-hpilo==4.3
python-http-client==3.3.5
python-izone==1.2.3
python-join-api==0.0.6
python-jose==3.3.0
python-juicenet==1.0.2
python-jwt==3.3.2
python-kasa==0.4.0
python-magic==0.4.24
python-miio==0.5.9.2
python-mpd2==3.0.4
python-mystrom==1.1.2
python-nest==4.1.0
python-opendata-transport==0.3.0
python-openzwave-mqtt==1.4.0
python-picnic-api==1.1.0
python-qbittorrent==0.4.2
python-ripple-api==0.0.3
python-singleton==0.1.2
python-slugify==4.0.1
python-smarttub==0.0.29
python-sochain-api==0.0.2
python-socketio==4.6.1
python-songpal==0.12
python-tado==0.12.0
python-telegram-bot==13.1
python-twitch-client==0.6.0
python-vlc==1.1.2
python-whois==0.7.3
pythonegardia==1.0.40
pyTibber==0.21.7
pytile==2021.10.0
pytouchline==0.7
pytraccar==0.10.0
pytradfri==7.2.1
pytrafikverket==0.1.6.2
PyTransportNSW==0.1.1
PyTurboJPEG==1.6.5
pytz==2021.3
pytz-deprecation-shim==0.1.0.post0
pyudev==0.22.0
pyunifiprotect==1.6.3
pyuptimerobot==21.11.0
pyusb==1.2.1
pyvera==0.3.13
pyversasense==0.0.6
pyvesync==1.4.2
PyViCare==2.15.0
pyvizio==0.1.57
pyvlx==0.2.19
pyvolumio==0.1.3
pyW215==0.7.0
pyW800rf32==0.1
pywebpush==1.9.2
pywemo==0.7.0
pywilight==0.0.70
pywinusb==0.4.2
pyxeoma==1.4.1
PyXiaomiGateway==0.13.4
PyYAML==6.0
pyzbar==0.1.7
pyzerproc==0.4.8
qnapstats==0.4.0
quantum-gateway==0.0.6
RachioPy==1.0.3
radiotherm==2.1.0
raincloudy==0.0.7
raspyrfm-client==1.2.8
ratelimit==2.2.1
readlike==0.1.3
regenmaschine==2021.10.0
regex==2021.11.10
renault-api==0.1.4
ReParser==1.4.3
repoze.lru==0.7
requests==2.27.1
requests-file==1.5.1
requests-futures==1.0.0
requests-oauth==0.4.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
responses==0.17.0
RestrictedPython==5.2
rfc3986==1.5.0
rfk101py==0.0.1
rflink==0.0.58
rich==10.16.2
ring-doorbell==0.7.2
ritassist==0.9.2
rjpl==0.3.6
rocketchat-API==0.6.1
rokuecp==0.10.0
roombapy==1.6.5
roonapi==0.0.38
rova==0.2.1
rpi-bad-power==0.1.0
rsa==4.8
RtmAPI==0.7.2
rtsp-to-webrtc==0.4.0
russound==0.1.9
russound-rio==0.1.7
Rx==3.2.0
rxv==0.7.0
s3transfer==0.5.0
samsungctl==0.7.1
samsungtvws==1.6.0
satel-integra==0.3.4
scapy==2.4.5
schedule==1.1.0
schiene==0.23
screenlogicpy==0.5.4
scsgate==0.1.0
semver==2.13.0
sendgrid==6.8.2
sense-energy==0.9.3
sense-hat==2.2.0
sentry-sdk==1.5.2
sepaxml==2.0.0
sgmllib3k==1.0.0
sharkiqpy==0.1.8
sharp-aquos-rc==0.3.2
shodan==1.26.0
simplehound==0.3
simplejson==3.17.6
simplepush==1.1.4
simplisafe-python==2021.11.2
siobrultech-protocols==0.5.0
sisyphus-control==3.1.2
six==1.16.0
skybellpy==0.6.3
slackclient==2.5.0
sleekxmppfs==1.3.4
sleepyq==0.8.1
slixmpp==1.7.1
smart-meter-texas==0.4.7
SmartHab==0.21
smbus-cffi==0.5.1
smbus2==0.4.1
smhi-pkg==1.0.15
sn3218==1.2.7
snapcast==2.1.3
sniffio==1.2.0
snitun==0.30.0
snowballstemmer==2.2.0
soco==0.25.3
solaredge==0.0.2
solaredge-local==0.2.0
solax==0.2.9
somecomfort==0.8.0
somfy-mylink-synergy==1.0.6
sonarr==0.3.0
soupsieve==2.3.1
speak2mary==1.4.0
speedtest-cli==2.1.3
Sphinx==4.3.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
spiderpy==1.6.1
spidev==3.5
spotipy==2.19.0
SQLAlchemy==1.4.27
sqlitedict==1.7.0
srpenergy==1.3.2
srptools==1.0.1
sseclient-py==1.7.2
starkbank-ecdsa==2.0.3
starline==0.1.5
starlingbank==3.2
statsd==3.2.1
stdiomask==0.0.6
steamodd==4.21
stevedore==3.5.0
stookalert==0.1.4
streamlabswater==1.0.1
stringcase==1.2.0
subarulink==0.3.12
sucks==0.9.4
sunwatcher==0.2.1
surepy==0.7.2
swisshydrodata==0.1.0
synology-srm==0.2.0
systembridge==1.1.5
tabulate==0.8.9
tailer==0.4.1
tailscale==0.1.6
tank-utility==1.4.0
tapsaff==0.2.1
tellcore-net==0.4
tellcore-py==1.1.2
tellduslive==0.10.11
tellsticknet==0.1.2
temescal==0.3
temperusb==1.5.3
tenacity==8.0.1
tesla-powerwall==0.3.12
tesla-wall-connector==1.0.1
text-unidecode==1.3
thermoworks-smoke==0.1.8
thingspeak==1.0.0
tikteck==0.4
titlecase==2.3
tmb==0.0.4
todoist-python==8.0.0
tololib==0.1.0b3
toml==0.10.2
toonapi==0.2.1
tornado==6.1
total-connect-client==2021.12
tp-connected==0.0.4
tqdm==4.62.3
transitions==0.8.10
transmissionrpc==0.11
TravisPy==0.3.5
ttls==1.4.2
tuya-iot-py-sdk==0.6.6
twentemilieu==0.5.0
twilio==6.32.0
TwitterAPI==2.7.5
typer==0.4.0
typing-inspect==0.7.1
typing_extensions==4.0.1
tzdata==2021.5
tzlocal==4.1
uamqp==1.5.1
uEagle==0.0.2
ujson==4.3.0
unasync==0.5.0
unifiled==0.11
untangle==1.1.1
upb-lib==0.4.12
upcloud-api==2.0.0
update-checker==0.18.0
uplink==0.9.5
uplink-protobuf==0.1.0
uritemplate==3.0.1
url-normalize==1.4.1
urllib3==1.26.8
urwid==2.1.2
uscisstatus==0.1.1
uvcclient==0.11.0
vallox-websocket-api==2.6.0
vehicle==0.3.1
velbus-aio==2021.11.7
venstarcolortouch==0.15
vilfo-api-client==0.3.2
vincenty==0.1.4
virtualenv==20.13.0
vobject==0.9.6.1
vol==0.1.1
volkszaehler==0.2.1
voluptuous==0.12.2
voluptuous-serialize==2.5.0
volvooncall==0.9.1
vsure==1.7.3
vtjp==0.1.14
vultr==0.1.2
wakeonlan==2.0.1
wallbox==0.4.4
waqiasync==1.0.0
warrant-lite==1.0.4
watchdog==2.1.6
waterfurnace==1.1.0
WazeRouteCalculator==0.14
wcwidth==0.2.5
webcolors==1.11.1
webexteamssdk==1.1.1
websocket-client==1.1.0
websockets==8.1
Werkzeug==2.0.2
whirlpool-sixth-sense==0.15.1
wiffi==1.1.0
wirelesstagpy==0.8.1
withings-api==2.3.2
wled==0.11.0
wolf-smartset==0.1.11
wrapt==1.13.3
WSDiscovery==2.0.0
XBee==2.3.2
xbee-helper==0.0.7
xbox-webapi==2.0.11
xboxapi==2.0.1
xknx==0.18.15
XlsxWriter==3.0.2
xmltodict==0.12.0
xs1-api-client==3.0.0
yalesmartalarmclient==0.3.7
yalexs==1.1.18
yarl==1.7.2
yeelight==0.7.8
yeelightsunflower==0.0.10
youless-api==0.16
youtube-dl==2021.12.17
zeep==4.1.0
zengge==0.2
zeroconf==0.38.1
zha-quirks==0.0.65
zhong-hong-hvac==1.0.9
ziggo-mediabox-xl==1.1.0
zigpy==0.42.0
zigpy-deconz==0.14.0
zigpy-xbee==0.14.0
zigpy-zigate==0.7.3
zigpy-znp==0.6.4
zipp==3.7.0
zm-py==0.5.2
zope.interface==5.4.0
zwave-js-server-python==0.33.0

@notatallshaw
Copy link
Member

notatallshaw commented Jan 14, 2022 •
edited
Loading

Actually I rebuilt the requirements files with the above freeze information.

Using these and fixing/removing the constraints file you can install using pip 21.3.1. Though you will need to figure out which libraries are too old for your requirements and look at why they have conflicts by updating them and see what the conflicts are.

Here is a diff on the requirements_all.txt:

-PyRMVtransport==0.3.3
+PyRMVtransport==0.3.2
-aioambient==2021.11.0
+aioambient==2.0.0
-aiobotocore==1.2.2
+aiobotocore==2.1.0
-aioridwell==2021.12.2
+aioridwell==2021.10.0
-aiosteamist==0.3.1
+aiosteamist==0.1.0
-boto3==1.16.52
+boto3==1.20.24
-hole==0.7.0
+hole==0.6.0
-# homeassistant.components.mycroft
-mycroftapi==2.0
-open-meteo==0.2.1
+open-meteo==0.0.1
-py17track==2021.12.2
+py17track==2021.12.0
-pyicloud==0.10.2
+pyicloud==0.7.3
-pymodbus==2.5.3
+pymodbus==1.5.2
-pyoverkiz==1.1.0
+pyoverkiz==1.1.1
-pysml==0.0.6
+pysml==0.0.5
-# homeassistant.components.stiebel_eltron
-pystiebeleltron==0.0.1.dev2
-pytile==2021.12.0
+pytile==2021.10.0
-simplisafe-python==2021.12.2
+simplisafe-python==2021.11.2
-systembridge==2.2.3
+systembridge==1.1.5
-vallox-websocket-api==2.8.1
+vallox-websocket-api==2.6.0

And the diff on requirements.txt:

-aiohttp==3.8.1
+aiohttp==3.7.4.post0
-async_timeout==4.0.2
+async_timeout==3.0.1
-pip>=8.0.3,<20.3

@pradyunsg pradyunsg added type: support User Support and removed type: bug A confirmed bug or unintended behavior labels Jan 14, 2022
@balloob
Copy link
Contributor Author

balloob commented Jan 14, 2022

Thanks for digging in!

Is there a place where we can find your experimental pip version?

We definitely have some old packages around for integrations that haven't been touched in years. Sometimes integration requirements get disabled when we raise our minimum Python version. Looks like we'll have to do so here too.

It's too bad that our lax handling of dependencies of our integrations is finally catching up on us. It means we'll need to pro-actively take care of it. Ccurrently we do it once an issue is raised that something doesn't work.

The requirements diff doesn't look too bad although I would hate to have to downgrade things (because it means we can't move forward!)

@pradyunsg
Copy link
Member

pradyunsg commented Jan 14, 2022 •
edited
Loading

You can blame reach out to packages like discord-py regarding some of the pain here, asking them to remove the relevant problematic pins. You're basically already seeing the problems described in https://iscinumpy.dev/post/bound-version-constraints/#tldr.

If those packages are using poetry or using something like ~= without good reason, point them to the aforementioned blog post.

@notatallshaw
Copy link
Member

notatallshaw commented Jan 14, 2022 •
edited
Loading

Is there a place where we can find your experimental pip version?

Yes but it's not stable so I don't guarantee anything about it.

To explain a little bit about what it's doing, a backtracking optimization landed on pip 21.3: #10479 . This optimization is very good at handling the cases where the solution seems "obvious", i.e. A and B don't agree on what version C should be used, so it checks different versions of A and B until they do. However it's not very good at situations where you add a dependency A and it conflicts with multiple preexisting dependencies e.g. X, Y, and Z.

In the experimental version I am testing I use backjumping in an attempt to make those scenarios more solvable. But it still has many limitations and my implementation at the moment is very hacky and I've not verified it other than testing it in complex scenarios and seeing how it performs.

Here's a tag of the version I used to help here: https://github.com/notatallshaw/pip/tree/homeassistant . I think you can install it like so: python -m pip install git+https://github.com/notatallshaw/pip@homeassistant

It also only helps to a certain extent, allowing to make some of your requirements a little bit more flexible. You can't for example remove every version requirement from the hundreds of requirements and expect it to find a solution, I still had to very manually and iteratively loosen up the requirements. You might want to check how other large projects like apache-airflow handle this? I think there are tools like pip compile that are supposed to help? (never had to look in to it myself).

We definitely have some old packages around for integrations that haven't been touched in years. Sometimes integration requirements get disabled when we raise our minimum Python version. Looks like we'll have to do so here too.

What makes it tricky to find the conflicting packages here is that everything is added at once. I wonder if it's possible you can break your integrations down in to logical sub groups and split your requirements in to those groups? Then at least you can test the issues in the groups and then once resolved see if you can install the groups together.

It's too bad that our lax handling of dependencies of our integrations is finally catching up on us. It means we'll need to pro-actively take care of it. Ccurrently we do it once an issue is raised that something doesn't work.

Python is a tricky language to manage such a large number of requirements. I wonder if you'll have to ultimately take some sort of vendoring approach so you can have different dependencies have different versions of their dependencies.

The requirements diff doesn't look too bad although I would hate to have to downgrade things (because it means we can't move forward!)

Yeah most of them look fine, but there's a few like open-meteo where the downgrade looks very challenging :(.

@jensens
Copy link

jensens commented Jan 19, 2022

We had the same problem with our CMS Plone which has lots of dependencies. With current pip main branch and this #10574 PR merged these issues are solved. Did you try it already?
I installed pip like so pip install git+https://github.com/pypa/pip.git#egg=pip

@pradyunsg
Copy link
Member

This is different, since they have dependency conflicts in the graph, rather than a dense graph post-resolve.

@ashb
Copy link

ashb commented Jan 20, 2022

FWI we in apache-airflow don't handle this very well either, other than carefully (and somewhat blindly) altering our deps when we run into a backtracking issue.

@notatallshaw notatallshaw mentioned this issue Jan 20, 2022
Closed
@balloob
Copy link
Contributor Author

balloob commented Jan 20, 2022

In the past we used our constraints file to "solve" conflicts. That is similar to what is requested in pypa/pipenv#4530 to work with the new resolver. Having that feature means we can move fast again with shipping security updates when needed without having to rush to ship a security fix and update X other packages that cause backtracking issues.

@potiuk
Copy link
Contributor

potiuk commented Jan 20, 2022 •
edited
Loading

In the past we used our constraints file to "solve" conflicts. That is similar to what is requested in pypa/pipenv#4530 to work with the new resolver. Having that feature means we can move fast again with shipping security updates when needed without having to rush to ship a security fix and update X other packages that cause backtracking issues.

Yep - we have very sophisticated mechanism of both preparing and using constraints. In fact we are publishing a set of constraints for our users so that they can install airflow reliably. For example this is the only "official" way you can install 3.6 version of Airlfow 2.2.3 (from https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html)

pip install "apache-airflow[celery]==2.2.3" \
   --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.3/constraints-3.6.txt

The problem here is that it backtracked when we tried to upgrade the constraints. We have > 500 dependencies, so we do not upgrade constraints manually - we automated it. In our CI, the constraints will be automatically upgraded in the main build and whenever dependencies change. We are using --eager-upgrade for that. After the --eager-upgrade succeds, we run tests and only when tests pass we publish new constraints in "main" (which later are frozen when we release Airflow).

So in our case the backtracking happens (sometimes) when we attempt to generate new constraints with --eager-upgrade (which excludes using existing constraints effectively).

BTW. It's pretty sophisticated mechanism with Airflow's >500 deps - I gave recently presentation on that where you can see most of the why's and how's if you are interested: https://www.youtube.com/watch?v=_SjMdQLP30s&t=2549s

@mweinelt mweinelt mentioned this issue Feb 4, 2022
Merged
13 tasks
@potiuk potiuk mentioned this issue Mar 2, 2022
Open
1 task
@milahu
Copy link

milahu commented Apr 25, 2022

Expected behavior

Done in less than 15 minutes.

wow, that's still long

pipenv's Pipfile.lock could help. this should lock all dependencies, including transitive dependencies

at least this way, you can cache the resolution step

@notatallshaw
Copy link
Member

Expected behavior

Done in less than 15 minutes.

wow, that's still long

pipenv's Pipfile.lock could help. this should lock all dependencies, including transitive dependencies

at least this way, you can cache the resolution step

This only helps if you know a working set of dependencies to start with. The original set of requirements linked here didn't have a resolvable set of dependencies.

@pradyunsg
Copy link
Member

@balloob Are things better now, with the current release of pip?

@balloob
Copy link
Contributor Author

balloob commented Oct 6, 2022 •
edited
Loading

We're still using the legacy resolver. We do have a GitHub Action in place to make sure we don't introduce new conflicts. However, we have no idea how we're going to handle upgrading components in the future once the legacy resolver is gone without being able to force a resolution (there was some recent discussion about this here can't find it right now). The atomicwrites incident really scared us. With the modern resolver we would not have been able to publish a release until all dependencies (which are mostly volunteer-led) with conflicts had done a new release to resolve it.

As reported above, the resolving can be done in 15 minutes. So I guess this issue resolved 🥲

@balloob balloob closed this as completed Oct 6, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
C: dependency resolution About choosing which dependencies to install type: support User Support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ashb @jensens @potiuk @balloob @pradyunsg @notatallshaw @milahu