release-1.6: Backports for julia-1.6.4

[KristofferC]

• REPL: fix #27184, ensure macro existence before lowering #40621
• MINSIGSTKSZ is no longer constant in glibc #41860
• Only do NFD normalization for latex tab-completion if needed #41870
• fix #41908, inference error in subst_trivial_bounds #41976
• (#42139) Fixes _is_mailto in resolution of autolink in Markdown module #42140
• eagerly do boundscheck when indexing into CartesianIndices #42119
• Base.julia_cmd(): correctly forward the --sysimage-native-code=no flag if it is provided #42185
• fix #42220: macro hygiene with unary :: #42221
• Slightly loosen a test so that linalg tests pass when using MKL #42315
• Fix problem with union spliting during inlining, add regression test #42347
• fix verifier to allow GlobalRef as assignment RHS #42417
• tests: add help and reflection abilities to choosetests option parser #42462
• Distributed: improve some of the tests in the Distributed test suite when multithreading is enabled #42499
• [buildkite] Fix env leakage of signed tokens #42526
• make tests ignore root more robustly #42533
• Add option to allow stale_cachefile to ignore loaded modules (for Pkg.precompile) #42545
• Fix libgit2 error throw #42576
• Base: fix tryparse(Bool, " ") #42623
• fix #42590, long attribute loop compiling cfunction #42635
• Add cryptic secret for docs deploy key #42667
• Disable DL_LOAD_PATH prepending for @-paths on Darwin #42721
• choosetests: return a NamedTuple (instead of a Tuple), to make it easier to add more information in the future #42723
• fix unique over a dimension to use isequal over == #42737
• Distributed test suite: increase the timeout in the poll_while function from 60 seconds to 120 seconds #42753
• Distributed test suite: if Threads.nthreads() > 1, skip certain tests #42764
• Optimize show(io::IO, m::Module) #42773
• Test.jl: no longer explicitly print some things in white #42864
• Test suite: improve the warning that we print when we skip the Profile tests on ARM #42862
• Use the module specific names in names() instead of the binding #42532
• choosetests: add the --force-net option, which will throw an error if networking is unavailable #42889
• choosetests: Pkg/pkg is no longer relevant #42890
• Eliminate @loader_path/julia element of DL_LOAD_PATH #42930
• Distributed test suite: mark another test as thread-unsafe #42941
• Update BinaryBuilder libuv to 2.0.1+4 #41900
• Fix libunwind segfaults from JIT frames on FreeBSD #42970
• Add missing type parameter to TakeWhile #42958
• file tests: in the "tempname with parent" test, temporarily unset the TMPDIR environment variable while running the test #43012

Backported CI PRs:

• CI: Work around registry corruption issues #41934
• Fix the cmdlineargs tests on Buildkite #42118
• CI (Buildkite, code coverage): increase the value of JULIA_WORKER_TIMEOUT on the code coverage job #42193
• CI (Buildkite, code coverage): Fail the code coverage job if the coverage is suspiciously low #42213
• CI (Buildkite, GHA): Allow any user with triage or commit permissions to retry all failed Buildkite jobs #42138
• CI (Buildkite): in the experimental asan job, remove the "Test that ASAN is enabled" step (but keep the "Build julia-debug with ASAN" step) #42229
• CI (Buildkite): Add package_linux and tester_linux for more Linux architectures #41707
• CI (GHA): Only run the Statuses workflow when a pull request is opened or synchronized #42389
• CI (GHA): update to the latest commit of the retry-buildkite action #42386
• CI (Buildkite): explicitly set the OPENBLAS_NUM_THREADS environment variable #42470
• CI (Buildkite): in the Buildkite cache, don't cache the "registries" directory in the default depot #42475
• CI (Buildkite): move the asan job from "experimental" to "main" #42478
• CI (Buildkite): fix the signature for the scheduled coverage pipeline #42488
• CI (Buildkite): add package_musl64 and tester_musl64 (Linux, musl libc, 64-bit, x86_64 architecture) #42476
• CI: add TSAN to the sanitizers pipelines #42444
• CI (buildkite): ensure passing --output-sync when passing -j #42511
• CI (Buildkite): delete an empty YAML file #42531
• CI (Buildkite): reorganize some of the logic in the tester_linux.yml file #42514
• CI (Buildkite): TEMPORARILY allow the asan job to fail #42541
• CI (Buildbot, GHA): Rename "Statuses" to "Create Buildbot Statuses" #42640
• CI (Buildkite, GHA): Rename "Retry" to "Retry Failed Buildkite Jobs" #42641
• CI (Buildkite, GHA): in the "Retry Failed Buildkite Jobs" workflow, remove the labeled trigger #42642
• CI (Buildkite): Add a once-daily scheduled job that builds Julia with USE_BINARYBUILDER=0 (and runs the test suite) on linux64 #42592
• Add a comment trigger for rerunning failed Buildkite jobs #42655
• CI (Buildkite, GHA): Fix some bugs in the "Retry" workflow #42657
• CI (Buildkite): Run the Distributed test suite with multithreading enabled #42479
• CI (Buildkite): fix the timeout for the tester_linux job that runs under rr #42724
• CI (Buildkite): make sure to gitignore any unencrypted repo keys, regardless of where they are located in the repository #42803
• CI (Buildkite): Update all rootfs images to the latest versions #42802
• CI (Buildkite): add a separate Buildkite job that runs the tests of the network-related stdlibs (e.g. Downloads.jl), and automatically retries that job up to a maximum number of tries #42861
• CI (Buildkite): choosetests: preparations for setting up a separate Buildkite job that runs the Pkg test suite #42859
• CI (Buildkite): add Pkg, Artifacts, and LazyArtifacts to the list of "network-related stdlibs" that we test in a separate Buildkite job #42885
• CI (GHA, Buildbot): temporarily, don't create the linuxaarch64 Buildbot status #42954
• CI (Buildkite): increase the default timeouts #42953
• CI (Buildkite): Fix the Buildkite once-daily scheduled job #42976
• CI (Buildkite): increase the timeouts for the Buildkite jobs that run under rr #42979
• CI (GHA, Buildbot): Update the list of commit statuses created by the "Create Buildbot Statuses" workflow #42985
• CI (Buildkite): fix another bug in the once-daily scheduled Buildkite job #42983
• CI (Buildkite): increase the timeout of the code coverage job #42994
• CI (Buildkite): use a TMPDIR that is backed by real storage #42995

Need manual backport:

• inference: bail from const-prop if const-prop fails #42112
• coverage: ensure topline is recorded during inlining #42170
• External stdlibs: Move *_URL stdlib definitions into .version files #42474
• External stdlibs: stop using the unauthenticated git:// protocol in the external stdlib URLs #42906
• Remove some other uses of the unauthenticated git:// protocol in Git URLs #42907
• Running make testall should run all of the tests #42904
• Timer: handle timeout correctly #42854

Contains multiple commits, manual intervention needed:

• fix collect on stateful iterators #42172
• fixed Float16 from Float64 and BigFloat #42837

Non-merged PRs with backport label:

• disallow unbalanced bidirectional formatting in strings and comments #42918
• Fix @simd for non 1 step CartesianPartition #42736
• add a missing transpose in LinearAlgebra.copy_transpose! to handle matmul of isbits array-elements with non-identity transpose #42715
• CI (Buildkite): upload musl64 nightly binaries to S3 #42495
• CI (Buildkite): on pull request builds, build the merge commit (e.g. refs/pull/1234/merge) #42403
• Update mbedtls to final 2.x release #42311
• fix some issues with equality of factorizations #41228
• fix command literals with trailing backslashes #41041
• Fix log message id from at-deprecate to be a Symbol. #40820

[DilumAluthge]

Why isn't #41707 showing up in the list?

[KristofferC]

Might be because it is old. The way the PR works is that it scans all PRs looking for a label and then it stops at some hardcoded date. I don't know if there is a way to get all PRs with a certain label. I can adjust the date or include it manually.

[DilumAluthge]

I don't know if there is a way to get all PRs with a certain label.

Hmmm. Yeah, the GitHub API docs don't mention any such functionality.

[DilumAluthge]

I can adjust the date or include it manually.

Yeah, set the date to something like July 20, 2021. That should be sufficient to include #41707. If for some reason #41707 doesn't backport automatically, we can set the date even earlier to look for PRs on which #41707 depends.

[DilumAluthge]

I don't know if there is a way to get all PRs with a certain label.

Maybe using the search API? Here's an example from their docs:

q=windows+label:bug+language:python+state:open&sort=created&order=asc

https://docs.github.com/en/rest/reference/search#search-issues-and-pull-requests

[vchuravy]

Ah sorry I did #42564 against the release-1.6 branch instead of backports-release-1.6, causing these conflicts.

[KristofferC]

Oops. Revert and push it here?

[vchuravy]

Oops. Revert and push it here?

Done.

[DilumAluthge]

The Buildkite tests are failing because #42533 has not yet been backported.

[DilumAluthge]

The Buildbot testers are failing due to two test failures in the errorshow test set:

errorshow                          (7) |        started at 2021-10-19T13:50:19.980
errorshow                          (7) |         failed at 2021-10-19T13:50:34.302
Test Failed at /buildworker/worker/tester_linux64/build/share/julia/test/errorshow.jl:319
  Expression: err_str == "TypeError: in TypeWithIntParam, in A, expected Vector{Int64}<:A<:(AbstractArray{Int64}), got Type{Vector{Float64}}"
   Evaluated: "TypeError: in TypeWithIntParam, in A, expected Vector{Int64}<:A<:(AbstractArray{Int64, N} where N), got Type{Vector{Float64}}" == "TypeError: in TypeWithIntParam, in A, expected Vector{Int64}<:A<:(AbstractArray{Int64}), got Type{Vector{Float64}}"
Test Failed at /buildworker/worker/tester_linux64/build/share/julia/test/errorshow.jl:321
  Expression: err_str == "TypeError: in TypeWithIntParam, in A, expected Vector{Int64}<:A<:(AbstractArray{Int64}), got Type{Vector{Float64}}"
   Evaluated: "TypeError: in TypeWithIntParam, in A, expected Vector{Int64}<:A<:(AbstractArray{Int64, N} where N), got Type{Vector{Float64}}" == "TypeError: in TypeWithIntParam, in A, expected Vector{Int64}<:A<:(AbstractArray{Int64}), got Type{Vector{Float64}}"

[KristofferC]

ada1cd2 probably needs a tweak

[DilumAluthge]

Contains multiple commits, manual intervention needed:

• CI (Buildkite): Update all rootfs images to the latest versions #42802

😭 I thought I'd squashed when merging.

If #42802 had multiple commits, then all of them should be backported.

[DilumAluthge]

Hmmm. It definitely looks like I squashed when merging. Why is it showing up under "contains multiple commits"?

[KristofferC]

Maybe some bug in the script :) It does however have a conflict.

[DilumAluthge]

It does however have a conflict.

That PR only touches Buildkite files. So if there's a conflict, that means I forgot to add the backport label to another CI PR.

Any chance that you have a log or anything that gives details on the merge conflicts?

[KristofferC]

It was #42444.

[DilumAluthge]

It was #42444.

Thanks for tracking that down!

[jeremiahpslewis]

Hi! (Not entirely sure if this is the right place to ask this question, please let me know.) Would it make sense for #39769, e.g. GMP 6.2.0 -> 6.2.1 to be added to the 1.6 backports?

[KristofferC]

@nanosoldier runtests(["ADI", "AlgamesDriving", "AlgorithmsCollection", "ArrayLayouts", "BLASBenchmarksCPU", "BayesianQuadrature", "BlockArrays", "BytePairEncoding", "ClassicalOrthogonalPolynomials", "ControlSystemIdentification", "CorrelationFunctions", "Cropbox", "DIVAnd", "DataDeps", "DecisionTree", "DynamicalSystemsBase", "EmailScraper", "FSInteraction", "FilesystemDatastructures", "FlashWeave", "GigaSOM", "Graph500", "InfiniteLinearAlgebra", "KrigingEstimators", "LazyArrays", "MCIntegration", "MarketData", "MiniLoggers", "NumericalAlgorithms", "Onda", "PackageAnalyzer", "ParquetFiles", "PastaQ", "QuantumESPRESSOExpress", "QuasiArrays", "RRTMGP", "Reactive", "RoME", "SortMark", "SyntheticDatasets", "TensorKit", "Turing", "VoronoiGraph", "YAXArrayBase"], vs = ":release-1.6")

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.