"Out of files" (sockets) after repeated use of aws-api with default-http-connection

[puredanger]

See: https://clojure.atlassian.net/browse/TDEPS-199 for the original case.

Dependencies

{:deps
  {com.cognitect.aws/api {:mvn/version "0.8.505"}
   com.cognitect.aws/endpoints {:mvn/version "1.1.11.969"}
   com.cognitect.aws/s3 {:mvn/version "811.2.858.0"}}}

Description with failing test case

tools.deps uses the aws-api to access S3 for downloading deps from s3 maven repos (like the datomic-maven repo).

Reproducible with this deps.edn from https://clojure.atlassian.net/browse/TDEPS-199:

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.10.3"}
            com.datomic/ion {:mvn/version "0.9.50"}
            software.amazon.awscdk/codebuild {:mvn/version "1.115.0"}}
 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}}

You can repro by using ambient AWS credentials that support s3 operations and Clojure CLI version 1.10.3.933. The 3 deps above expand to 57 libs in the full transitive tree. Use of the s3 maven repo is of course necessary here and these transitive deps happen to use many maven version ranges, which necessitate metadata requests to all repos (including s3) so there are a large number of s3 requests being sent in the course of expansion. This use case fails with "Too many open files" (really sockets).

The connection code is here:

https://github.com/clojure/tools.deps.alpha/blob/v0.12.1003/src/main/clojure/clojure/tools/deps/alpha/util/s3_transporter.clj#L93

The user reported that not retrieving the default-http-connection and passing it to aws-client "fixed" the problem. This seems odd though and I think it's worth understanding why this matters.

I did apply that change, but also made some additional changes to run this connection code only once, rather than on every download, which would presumably also have fixed the problem in isolation.

Note that these and other changes have been applied in newer versions of the Clojure CLI so this issue will not be reproducible in newer versions.