The artifacts override tests fail for me locally

[DilumAluthge]

Steps to reproduce:

delete!(ENV, "JULIA_PKG_SERVER");
empty!(Base.DEPOT_PATH)
pushfirst!(Base.DEPOT_PATH, mktempdir(; cleanup = true))
import Pkg
cd(joinpath(dirname(dirname(pathof(Pkg))), "test"))
include("utils.jl")
include("artifacts.jl")

The error that I get:

Override.toml: Test Failed at /Users/dilum/local/julia/share/julia/stdlib/v1.8/Pkg/test/artifacts.jl:690
Expression: arty_path == barty_override_path
 Evaluated: "/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_25KvpZ/depot2/artifacts/64d0b4f8d9c004b862b38c4acfbd74988226995c" == "/var/folders/jy/7hh5zyw95cg2lh66lfpthqq80000gn/T/jl_25KvpZ/a_wild_barty_appears"
Stacktrace:
[1] macro expansion
 @ ~/local/julia/share/julia/stdlib/v1.8/Test/src/Test.jl:460 [inlined]
[2] (::Main.ArtifactTests.var"#126#142"{String})()
 @ Main.ArtifactTests ~/local/julia/share/julia/stdlib/v1.8/Pkg/test/artifacts.jl:690
Test Summary: | Pass  Fail  Total
Override.toml |   20     1     21
ERROR: LoadError: Some tests did not pass: 20 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /Users/dilum/local/julia/share/julia/stdlib/v1.8/Pkg/test/artifacts.jl:1

[DilumAluthge]

If I clone the Pkg.jl repository, and change the UUID in the project file, and run julia --project -e 'import Pkg; Pkg.instantiate(); Pkg.test()', the tests pass.

But if I then change the UUID back to the original UUID, and then I run julia --project -e 'import Pkg; Pkg.instantiate(); Pkg.test()', I get the test failure as described above.

@staticfloat I'm probably going to need your help to debug this. We will need to figure out how to get these tests passing without changing the UUID in the Pkg.jl project file, because I don't think we'll be able to do the "change the UUID" trick when running the tests with Base.runtests("Pkg").

[staticfloat]

The reason the tests don't pass is that if you don't change the UUID, you load the wrong Pkg; changing the UUID dodges the "is this an stdlib that I'm loading" check.

It should be fine without changing the UUID when testing in Base, because we will be running a Julia that has the correct Pkg since we just built it.

[DilumAluthge]

It should be fine without changing the UUID when testing in Base, because we will be running a Julia that has the correct Pkg since we just built it.

Hmmmm. I am still seeing the following test failure on Buildkite:

Test Failed at /cache/build/amdci7-5/julialang/julia-master/julia-b9e5bf8ce9/share/julia/stdlib/v1.8/Pkg/test/artifacts.jl:690
  Expression: arty_path == barty_override_path
   Evaluated: "/tmp/jl_do4N4X/depot2/artifacts/64d0b4f8d9c004b862b38c4acfbd74988226995c" == "/tmp/jl_do4N4X/a_wild_barty_appears"

The full Buildkite log is here: https://buildkite.com/julialang/julia-master/builds/5345#6729df4a-5129-4510-a957-940bc1ed473c

This is a Buildkite job triggered on the following pull request: JuliaLang/julia#42885

[DilumAluthge]

Yeah, I am continuing to see this consistently on JuliaLang/julia#42885

Another example log (from https://buildkite.com/julialang/julia-master/builds/5411#a702e08a-5079-489b-8697-83ad9e989913, which has an rr trace attached):

Pkg             (7) |         failed at 2021-11-02T13:00:25.853
Test Failed at /cache/build/amdci4-4/julialang/julia-master/julia-da505fec8e/share/julia/stdlib/v1.8/Pkg/test/artifacts.jl:690
Expression: arty_path == barty_override_path
Evaluated: "/tmp/jl_TJWFl1/depot2/artifacts/64d0b4f8d9c004b862b38c4acfbd74988226995c" == "/tmp/jl_TJWFl1/a_wild_barty_appears"

[DilumAluthge]

As far as I can tell, this is the last remaining Pkg failure when running the Pkg tests as part of Base Julia CI.