Parallelizing pytest

[byte-sculptor]

This PR:

1. Invokes pytest with a -n auto argument to make multiple tests run in parallel (num parallel jobs = num cores)
2. Installs pytest-xdist in all our pipelines to make the above possible
3. Shortens the duration of TestOptimizerEvaluator
4. Modifies the tests that use gRPC to attempt to start the service on 100 different ports before giving up. This is the easiest way to make sure that all tests requiring gRPC can run in parallel. The alternative would be to have them all talk to a single instance, but it would turn an easy parallelization problem into a hard one as we'd need to manage the lifetime of that single instance.
5. Relaxes the check in TestSmartCacheWithRemoteOptimizerV3.py as the current one was a bit too ambitious and lead to some flakiness.
6. Puts a band-aid on test_optimization_with_context. Turn Gaussian Blob With Context into an ObjectiveFunction #207 hints at a long-term fix.

[bpkroth]

Hmm, questions ...
Is this only necessary when running tests in parallel?
Or is it not cleaning up after itself? Or maybe some delay after the teardown is necessary?
Can a single remote optimizer not handle requests from multiple clients?

[bpkroth]

In fd8ef08 I think we had to add some additional code to the teardown_method:
e29697b

[byte-sculptor]

It's so that multiple services can run in parallel. Having one handle multiple requests would work too.

[byte-sculptor]

*one service - each test instantiates its own optimizer.

[bpkroth]

Thanks, makes sense.
The teardown changes (specifically closing the client channel explicitly) may still be necessary in the other one. I think I only made the change in the one that was throwing errors at the time.

[bpkroth]

If I recall correctly it leaves a socket open that causes conflicts later on when it goes to reuse it.

[byte-sculptor]

I thought it would only happen on the server side. Is there any repro that would tell us when we've fixed the bug?

[bpkroth]

main...bpkroth:demonstrate-missing-grpc-client-close-bug
bpkroth@cadc29c

[bpkroth]

Hmm, the command to execute them got dropped in the commit message (-k test_echo is optional - it's there just to try and keep the repro as simple as possible and works mostly because I added a second test_echo2 - the bug only shows up after the second test run since there needs to be cruft left over from the first):

# pytest -svxl -k test_echo source/Mlos.Python/mlos/unit_tests/TestBayesianOptimizerGrpcClient.py

[byte-sculptor]

awesome, thanks

[bpkroth]

Couple minor suggestions.

[amueller]

didn't look into the issue @bpkroth pointed out in detail but otherwise looks good.

[bpkroth]

didn't look into the issue @bpkroth pointed out in detail but otherwise looks good.

@amueller it was just another place to apply the grpc client close issue you found and I helped debug when converting to pytest (exactly references are in the thread).