Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Orquesta to v1.6.0 #6050

Merged
merged 12 commits into from
Nov 20, 2023
Merged

Update Orquesta to v1.6.0 #6050

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
633ec10
Bump orquesta to v1.6.0
arm4b Oct 25, 2023
880e5d2
Try to use python-version dependent requirements
arm4b Oct 25, 2023
4e13147
Update fixate-requirements to handle markers
arm4b Oct 26, 2023
aa4e19e
Alternative: Unpin networkx as it's pinned by orquesta depending on t…
arm4b Oct 26, 2023
7f47b36
Add a Changelog for #6050
arm4b Oct 27, 2023
abf9a80
Use networkx<3 to match orquesta
arm4b Oct 27, 2023
6cd25e8
Revert requirements for decorator, it doesn't affect anything
arm4b Oct 27, 2023
f0e0bfa
Revert networkx version for pants
arm4b Oct 27, 2023
0e73c4c
Merge branch 'master' into update/orquesta
arm4b Nov 6, 2023
e1b0938
Merge branch 'master' into update/orquesta
arm4b Nov 6, 2023
cbd0ee0
Merge branch 'master' into update/orquesta
arm4b Nov 11, 2023
16f27aa
Merge branch 'master' into update/orquesta
arm4b Nov 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ Fixed

* Update cryptography 3.4.7 -> 39.0.1, pyOpenSSL 21.0.0 -> 23.1.0, paramiko 2.10.5 -> 2.11.0 (security). #6055

* Bumped `eventlet` to `0.33.3` and `gunicorn` to `21.2.0` to fix `RecursionError` bug in setting `SSLContext` `minimum_version` property. #6061
* Bumped `eventlet` to `0.33.3` and `gunicorn` to `21.2.0` to fix `RecursionError` bug in setting `SSLContext` `minimum_version` property. (security) #6061
Contributed by @jk464

* Update orquesta to v1.6.0 to fix outdated dependencies (security). #6050

Added
~~~~~

Expand Down
2 changes: 1 addition & 1 deletion contrib/runners/orquesta_runner/in-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.5.0
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.6.0
2 changes: 1 addition & 1 deletion contrib/runners/orquesta_runner/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
# If you want to update depdencies for a single component, modify the
# in-requirements.txt for that component and then run 'make requirements' to
# update the component requirements.txt
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.5.0
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.6.0
4 changes: 2 additions & 2 deletions fixed-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ lockfile==0.12.2
# >=0.23 was from jinja2
MarkupSafe<2.1.0,>=0.23
mongoengine==0.23.0
# networkx v2.6 does not support Python3.6. Update networkx to match orquesta
networkx>=2.5.1,<2.6
# required by orquesta (networkx<2.6 for py3.6, networkx<3 for py3.8)
networkx<3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@armab what happens if you change networkx here to match orquesta, e.g. >=2.6 & < 3
and then re-generate the requirements files
and then re-generate pants.

Wondering if that is why it complains that it can't rectifiy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though also orquesta looks a bit more complicated...
requirements.txt:networkx>=2.5.1,<2.6; python_version < '3.7'
requirements.txt:networkx>=2.6,<3; python_version >= '3.7'

So we have different dependencies for different python versions, I wonder if the pants generate lockfiles copes with that?

Copy link
Member Author

@arm4b arm4b Oct 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amanda11 Yeah, I tried that in my first approach. See the build failures here: #6050 (comment)

Outside of issues in st2 with fixate-requirements script not allowing duplicate records of networkx, I've found later that pip under EL7 didn't work as expected with the environment markers during the s2-packages build: #6050 (comment)

After all the tries in st2-packages and st2 the current approach is the way of less resistance and at least I got the packages build working (with additional #6050 (comment) fix)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the pants generate lockfiles copes with that?

On the pants lockfile side, I saw

st2/lockfiles/st2.lock

Line 134 in 8f6cb46

"requires_python": ">=3.6",

so technically it may support things, but I experienced issues with pip too, so 🤷‍♂️

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pants does not support using the legacy resolver. I'm not sure why the legacy resolver is required in this case, as it seems clear to me. I'll have to play with it, hopefully this week.

Copy link
Member Author

@arm4b arm4b Oct 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cognifloyd Thanks for looking into it 👍 Can't wait and I'm looking forward to merge this PR so I can continue updating the dependencies for the v3.8.1 patch.

Some context on the issue with dependencies.
For some unknown reason, the new pip resolver went into an infinite loop resolving the dependencies: https://app.circleci.com/pipelines/github/StackStorm/st2/4158/workflows/11838dac-5adc-4554-9a87-f7bf506bedb6/jobs/15940?invite=true#step-109-978547_123 and couldn't process that networkx with conditional python_version env markers failing the EL7 build:

[package: st2] [18:59:15]      ERROR: Cannot install orquesta==1.6.0 and st2==3.9.dev0 because these package versions have conflicting dependencies.
[package: st2] [18:59:15]      
[package: st2] [18:59:15]      The conflict is caused by:
[package: st2] [18:59:15]          st2 3.9.dev0 depends on networkx
[package: st2] [18:59:15]          orquesta 1.6.0 depends on networkx<3 and >=2.6

In this ^^ case orquesta 1.6.0 depends on networkx<3 and >=2.6 is a wrong condition chosen, as EL7 runs on py3.6 and should choose a different networkx for dependency. So that might be a bug in pip.

I've tried to use the latest pip version available for py3.6, but it didn't help.
From the pip changelog there might be a relevant fix to try: (When checking for conflicts in the build environment, correctly skip requirements containing markers that do not match the current environment.) available in pip 22.1, but we can't use it because py3.6 support was dropped in pip 22.0.

For pants build, something pip buggy is happening as well, but more 👀 would be appreciated.

My thinking that's the worst case, for the upcoming st2 v3.8.1 patch we'll live with it and take what worked in this PR for st2-packages. The new pants build system is not going to go live for this patch anyway.
In st2 v3.9.0 we can drop python 3.6, switch to a new pip dependency resolver, latest pip and remove python env markers alltogether. That'll be quite an exercism that should potentially fix some bugs and technical debt 😈

# networkx requires decorator>=4.3,<5 which should resolve to version 4.4.2
# but the wheel on pypi does not say it supports python3.8, so pip gets
# confused. For now, pin decorator to work around pip's confusion.
Expand Down
2 changes: 1 addition & 1 deletion requirements-pants.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ mongoengine
# networkx version is constrained in orquesta.
networkx
orjson
orquesta @ git+https://github.com/StackStorm/orquesta.git@v1.5.0
orquesta @ git+https://github.com/StackStorm/orquesta.git@v1.6.0
Copy link
Member Author

@arm4b arm4b Oct 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to play a bit with pants.

@cognifloyd For ./pants generate-lockfiles --resolve=st2 I've got the following error:

ERROR: Cannot install networkx and orquesta==1.6.0 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
 
 The conflict is caused by:
     The user requested networkx
     orquesta 1.6.0 depends on networkx<3 and >=2.6
 
 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict

the dependency conflict made no sense to me

Copy link
Member Author

@arm4b arm4b Oct 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remove networkx from requirements-pants.txt I'm getting the following during the pants lock generation:

ERROR: Could not find a version that satisfies the requirement networkx<3,>=2.6 (from orquesta)
ERROR: No matching distribution found for networkx<3,>=2.6

weird 🤔

# NOTE: Recent version substantially affect the performance and add big import time overhead
# See https://github.com/StackStorm/st2/issues/4160#issuecomment-394386433 for details
oslo.config>=1.12.1,<1.13
Expand Down
4 changes: 2 additions & 2 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ lockfile==0.12.2
logshipper@ git+https://github.com/StackStorm/logshipper.git@stackstorm_patched ; platform_system=="Linux"
mock==4.0.3
mongoengine==0.23.0
networkx>=2.5.1,<2.6
networkx<3
nose
nose-parallel==0.4.0
nose-timer==1.0.1
orjson==3.5.2
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.5.0
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.6.0
oslo.config>=1.12.1,<1.13
oslo.utils<5.0,>=4.0.0
paramiko==2.11.0
Expand Down
2 changes: 1 addition & 1 deletion st2common/in-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ mongoengine
networkx
# used by networkx
decorator
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.5.0
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.6.0
st2-rbac-backend@ git+https://github.com/StackStorm/st2-rbac-backend.git@master
oslo.config
paramiko
Expand Down
4 changes: 2 additions & 2 deletions st2common/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ jsonschema==2.6.0
kombu==5.0.2
lockfile==0.12.2
mongoengine==0.23.0
networkx>=2.5.1,<2.6
networkx<3
orjson==3.5.2
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.5.0
orquesta@ git+https://github.com/StackStorm/orquesta.git@v1.6.0
oslo.config>=1.12.1,<1.13
paramiko==2.11.0
pyOpenSSL==23.1.0
Expand Down
Loading