1.7.3 release [also 1.8.0-rc1 and 1.6.6 LTS] - post install | Pkg.update() → "could not download registries (thread failed to start)"

[Audrius-St]

Upgraded to Julia 1.7.3.

julia> versioninfo()
Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

Post upgrade

julia> using Pkg
julia> Pkg.update()
    Updating registry at `C:\Users\Audrius-St\.julia\registries\General.toml`
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = getaddrinfo() thread failed to start while requesting https://pkg.julialang.org/registries

Do not know if this is related to the new release, but have not encountered this error message previously.

julia> using Downloads

julia> request("https://pkg.julialang.org/registries")
ERROR: getaddrinfo() thread failed to start while requesting https://pkg.julialang.org/registries
Stacktrace:
 [1] (::Downloads.var"#9#18"{Base.DevNull, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Bool, String, Bool, Bool})(easy::Downloads.Curl.Easy)
   @ Downloads C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\Downloads\src\Downloads.jl:387
 [2] with_handle(f::Downloads.var"#9#18"{Base.DevNull, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Bool, String, Bool, Bool}, handle::Downloads.Curl.Easy)
   @ Downloads.Curl C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\Downloads\src\Curl\Curl.jl:88
 [3] #8
   @ C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\Downloads\src\Downloads.jl:328 [inlined]
 [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Bool, String, Bool, Bool}, arg::Base.DevNull)
   @ ArgTools C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\ArgTools\src\ArgTools.jl:112
 [5] #7
   @ C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\Downloads\src\Downloads.jl:327 [inlined]
 [6] arg_read
   @ C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\ArgTools\src\ArgTools.jl:61 [inlined]
 [7] request(url::String; input::Nothing, output::Nothing, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Nothing, verbose::Bool, debug::Nothing, throw::Bool, downloader::Nothing)
   @ Downloads C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\Downloads\src\Downloads.jl:326
 [8] request(url::String)
   @ Downloads C:\Users\Audrius-St\AppData\Local\Programs\Julia-1.7.3\share\julia\stdlib\v1.7\Downloads\src\Downloads.jl:306
 [9] top-level scope
   @ REPL[4]:1

[KristofferC]

Could you also try with eg the nightly julia version and see if the behavior is the same there?

[StefanKarpinski]

Quoting what I wrote on discourse, where this was originally reported:

That looks like an issue with libuv, which internally uses a dedicated thread for DNS lookups in order to avoid blocking since getaddrinfo is one of the few C APIs that has no non-blocking alternative. Looks like libuv is unable to start that thread for some reason. That inability could be related to the Downloads bump, but it’s hard to see how.

[KristofferC]

Looking at what got backported in #44189, it doesn't see that so much other things are likely to have impacted this though.

[Audrius-St]

Could you also try with eg the nightly julia version and see if the behavior is the same there?

Nightly Julia release:

julia> versioninfo()
Julia Version 1.9.0-DEV.650
Commit 4f9483cb6d (2022-05-26 14:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 12 × Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 12 virtual cores

Connection to pkg.julialang.org/registries appears to work as expected:

julia> using Downloads

julia> request("https://pkg.julialang.org/registries")
Response("https", "https://us-east.pkg.julialang.org/registries.conservative", 200, "HTTP/1.1 200 OK", ["server" => "nginx/1.21.6", "date" => "Thu, 26 May 2022 23:02:48 GMT", "content-type" => "text/plain", "content-length" => "88", "connection" => "keep-alive", "last-modified" => "Thu, 26 May 2022 22:15:26 GMT", "etag" => "\"628ffbfe-58\"", "accept-ranges" => "bytes", "x-lb-strategy" => "pkgservers_roundrobin"])

[KristofferC]

There is some discussion about this error here curl/curl#6495. Is there something "special" about your system? Running with some aggressive firewall or something like that?

[Audrius-St]

Yes, thank you. Missing the forest for the trees.
As I’ve never encountered any firewall access/download blocking issues before with the security software I use, it did not occur to me. Once I updated to the most recent version of the security software the issue was resolved.

[StefanKarpinski]

I was wrong about this being a libuv issue—it's actually a libcurl issue (as indicated by @KristofferC above). It seems that Julia's usage of libcurl can easily hit this issue since it's simple for someone to start an unlimited number of concurrent downloads. Since we don't want to artificially limit the number of concurrent downloads, I've worked around this libcurl issue by adding a semaphore to limit the number of concurrent connections. This allows arbitrarily many concurrent downloads but throttles the number of incomplete connections to servers. Since establishing connections should be quick, this avoids many problems without slowing things down. Specifically, since the DNS resolution happens during the estblishment of the connection, this means that libcurl won't ever start more than 16 threads for getaddrinfo calls.

[vtjnash]

Since establishing connections should be quick

IIRC, this took an average of about 100ms on my machine per connection, mostly spent waiting for libmbedtls to parse the certificate information on disk, based on the "concurrent requests" test duration