Pkg3: Separated Test Environments

[ChrisRackauckas]

Will one be able to set a separate environment for testing? I don't think that the current REQUIRE setup is sufficient for the current organization-by-organizations setup. Sometimes there are large changes that are happening in tandem, and it would be helpful to have a designated way to say "this package needs to be on master for tests".

One case where this shows up is in organizations where there's a change to a main package which cases changes in all dependents. Another case where this shows up is where you create features which are dependent on a large change in some other package, like getting your package ready for DataFrames' nullable-array interface and wanting the proper CI. Currently this is done with little hacks around it (calling an extra script to checkout packages only when the test is run on CI), but I think making it standard and explicit would be beneficial for both developers and those who are reviewing packages (@tkelman).

[tkelman]

I think this is better handled by staged tags. Nothing should depend on "master," that's a moving target and bad for reproducibility. If you depend on an unreleased sha, then you're not testing against what your users will be installing once it gets released. So this is inherently a temporary thing, and would be better handled by directing the test version of the package to use a test registry to get its dependency version info. Then if the changes from the test registry make it through as is to the release registry, you're testing the same thing users will get.

[ChrisRackauckas]

Nothing should depend on "master," that's a moving target and bad for reproducibility.

Yes, for tagged versions I agree. But that's irrelevant. But I was wondering if it can be standardized how/where branches are chosen for testing. When checking someone's master or dev branch, this stuff can be hidden: it's some script called in the .yml, or sprinkled into the master for the tests. It's up to the developer to pull this all out / comment it out when changes in downstream packages occur and when you eventually want to tag. But it also means that this information is scattered around the repositories if you're trying to find out how to get some development branch working. If it was just in the test/REQUIRE file how everything is being tested, it would be clear (and it would be clear that everything needs to be testing on releases for any tag).

[tkelman]

There should probably be a test-only manifest file, or a test-only section of the main manifest file. test/REQUIRE should be made more capable the same way as REQUIRE

[StefanKarpinski]

There could be test/{Config,Manifest}.toml files corresponding to the current test/REQUIRE.

[tkelman]

if we're using a richer format here, then no need for it to be a separate file really. there would be some advantage to recording test dependency info in the registry for a package as well, so all that information is available without having to download every version of every package to get it

[StefanKarpinski]

True, it can simply be a separate section. There's space for various kinds of targets but I'm unclear on the design there. @wildart had some thoughts on it so perhaps he can chime in here.