Benefit of output separation - bring back everything under aws-sdk-cpp?

[h-vetinari]

I'm wondering what gains we're getting from having

aws-c-auth
aws-c-cal
aws-c-common
aws-c-compression
aws-c-event-stream
aws-c-http
aws-c-io
aws-c-mqtt
aws-c-s3
aws-c-sdkutils
aws-checksums
aws-crt-cpp
s2n

all being migrated separately. It's generating a lot of churn, and I'm not sure what gain we get in return?

It might make sense to rather just implicitly migrate through this package?

W.r.t. to the costs that any benefits would need to offset (IMO) the very frequent migrations for all of the above packages. That alone creates a lot of maintenance on the pinning repo, but worse than that, they often logjam because one of the packages involved automerges a new version while another migration is still running, requiring manual intervention.

While the above migrations are mostly internal (i.e. don't cause rebuilds of packages outside of aws-*), and we finally reduced the number of "external" migrations due to aws-sdk-cpp in #662, there are still ways this amps up the maintenance elsewhere.

For example, I added aws-crt-cpp as a host-dep to arrow because the link check complained otherwise, which worked without too much hassle for some time, but now generated 3 migrations (times 4 maintenance branches) in a very short succession (conda-forge/arrow-cpp-feedstock#1224, conda-forge/arrow-cpp-feedstock#1229, conda-forge/arrow-cpp-feedstock#1240)

CC @conda-forge/aws-sdk-cpp @conda-forge/core

[xhochy]

My initial idea was to make this package here a split package so that we only include the features needed for arrow in one part and providing a fatter package for other use cases.

[xhochy]

Another benefit is that aws-sdk-cpp and awscrt both use the libraries and they appear together in some of my environments.

[h-vetinari]

I think the points you mention are clear advantages, the question to me is just if they are important enough to justify the near-constant migration effort in the aws-* space.

Another benefit is that aws-sdk-cpp and awscrt both use the libraries and they appear together in some of my environments.

Couldn't we build awscrt on top of a hypothetically unified libaws-sdk? That way, there wouldn't be any duplicated libraries in the environment (just more than necessary; the .conda for aws-sdk-cpp currently has 3MB, which isn't nothing, but also not gigantic; e.g. the gcp bindings are >10x larger).

[xhochy]

Couldn't we build awscrt on top of a hypothetically unified libaws-sdk?

I'm not sure that maintenance would actually be less.

the .conda for aws-sdk-cpp currently has 3MB, which isn't nothing, but also not gigantic; e.g. the gcp bindings are >10x larger

We only build a subset at the moment:

aws-sdk-cpp-feedstock/recipe/build.sh

Line 15 in 7891af2

-DBUILD_ONLY='s3;core;transfer;config;identity-management;sts;sqs;sns;monitoring;logs' \

[jaimergp]

What was the initial reason to split them to begin with? CI usage? Does the build go over 6h?

[xhochy]

I don't remember the circumstances anymore but I have split it during the osx-arm64 migration: #126 It was probably down to easier cross-compiling/patching of dependencies.