Isolate each test in a parentless module.

[yuyichao]

Replaces #9798.
This uses a parentless module (similar to #9798) since I was interested in the memory saving (not much but I haven't measured too carefully).

There are two test breakage that I need to workaround (instead of fixing).

1. Defining a new module on one process is not visible on another one, which breaks the DictChannel test in examples.
2. The dictionary key completion doesn't work for qualified names.

There might be a way to fix the first one (or just disable isolation for it as what I'm doing now is probably good enough). The second one is likely a bug that should be fixed.

[yuyichao]

I pushed a change to make the example test pass while being isolated by running the remote on the same worker. I'm not sure if it is still testing everything needed though. @amitmurthy

[tkelman]

Was it @dhoegh who implemented the dict key completion? Pretty recently iirc.

[yuyichao]

It looks reasonably easy to fix. I'll try later.

[dhoegh]

Nope, but a sensible guess. It was @bjarthur in #17017.

[tkelman]

sorry, should have actually checked blame!

[yuyichao]

Dict completion fixed. Rebased to the first commit so it can be merged independently if necessary.

[amitmurthy]

I think it is important to actually have the example test run with a remote worker. That test has also uncovered an as yet open issue - #16091

[tkelman]

if this doesn't take too long to run, should we make it a loop over both the CompletionFoo.test_dict version and the no-module test_dict version?

[yuyichao]

The same applies to almost all other tests in the same file though (this was basically the only one that uses a non-qualified name in the replcompletion test before). It's mainly to avoid polluting the namespace but making the name slightly longer could also be fine.

[tkelman]

this was basically the only one that uses a non-qualified name in the replcompletion test before

Then shouldn't this be kept for the sake of test coverage?

[yuyichao]

Well, I think it makes sense to add it back with a longer name but no it won't help test coverage since this one uses a completely different code path that isn't shared with anything else.

[tkelman]

module qualified vs not qualified by a module. it seems like you're no longer testing the unqualified version? nevermind having it as a function works great

[tkelman]

I'd like to do the first commit before branching (it's a bugfix) but hold off on the second until after.

[yuyichao]

I'd like to do the first commit before branching (it's a bugfix) but hold off on the second until after.

Sure. FWIW the kwarg error one is probably more important but there isn't a test for it without the second commit now. I'll split this two out into a separate PR shortly.

[amitmurthy]

One option is to split examples.jl and move the clustermanager and dictchannel tests to parallel_exec.jl. With an appropriate comment in examples.jl explaining the rationale for the move.

[yuyichao]

Still waiting on branching and the appveyor failure is the git network issue...

I switch to normal module now since anonymous modules are going to be kept alive by the types defined in it anyway....

[tkelman]

typo, methodswith ?

[yuyichao]

Comment addressed local tests passed and CI re-enabled. This should be ready to go.

I've also tested locally that turning off isolation still pass tests and running all the tests on a single process also works without additional warnings.

All the test files can be run with both mode now. A few things that needs to be run on Main are explicitly evald in Main (e.g. the REPL, the serializer and the TestHelper module). I've also removed a few modules in the dates tests since they are not needed anymore.

[yuyichao]

Failure was #19132 rebased and will merge when CI passed.