Skip to content

Commit

Permalink
Adjust deps to avoid incompatible versions of s3fs and aiobotocore
Browse files Browse the repository at this point in the history
Affects fresh installs of nextstrain-cli with the latest aiobotocore
version, 2.0.0.

The latest aiobotocore, 2.0.0, does not fulfill the dep declarations of
the latest s3fs, 2021.10.1.  As a result, pip fulfills our s3fs dep with
2021.7.0 instead, which while declared to be compatible with
aiobotocore>=1.4.1 is not actually compatible in practice.  The result
is a runtime AttributeError on AioSession during upload to S3 in
`nextstrain build --aws-batch`.

When our previous deps on s3fs and aiobotocore[boto3] were introduced,
s3fs did not yet support its own [boto3] extra.  It does now, which
means we can let it declare the aiobotocore[boto3] dep itself and not
end up with conflicting dep declarations.  This results in fresh
installs using an older version of aiobotocore (1.4.x), but that's what
we want since it works.  Good to get the transitive dep out of our
declarations too.

Still a mess.  Dependencies are hard.

Resolves #133.
  • Loading branch information
tsibley committed Nov 3, 2021
1 parent 83c5049 commit e9a9aa6
Showing 1 changed file with 6 additions and 5 deletions.
11 changes: 6 additions & 5 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,15 +99,16 @@ def find_namespaced_packages(namespace):
# declarations.
#
# Resolve the issue by using a specially-provided package extra from
# aiobotocore which causes them to declare explicit dependencies on
# boto3 so that the dependency resolver can figure it out properly.
# s3fs (first introduced with 2021.4.0) which causes them to declare an
# explicit dependency on aiobotocore's specially-provided package extra
# on boto3 so that dependency resolver can figure it out properly.
#
# See <https://github.com/dask/s3fs/issues/357> for more background.
# See <https://github.com/dask/s3fs/issues/357> and
# <https://github.com/nextstrain/cli/issues/133> for more background.
#
# What a mess.
"fsspec",
"s3fs",
"aiobotocore[boto3]",
"s3fs[boto3] >=2021.04.0",

# We use pkg_resources.parse_version(), which (confusingly) is provided
# by setuptools. setuptools is nearly ever-present, but it can be
Expand Down

0 comments on commit e9a9aa6

Please sign in to comment.