Set default test order to alphabetical

[mikeharder]

• appconfiguration: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385339&view=results
• authorization: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385340&view=results
• core: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385143&view=results
• cosmos: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385343&view=results
• eventhubs: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385346&view=results
• formrecognizer: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385353&view=results
• identity: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385354&view=results
• keyvault: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385338&view=results
• search: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385355&view=results
• servicebus: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385356&view=results
• storage: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385333&view=results
• textanalytics: https://dev.azure.com/azure-sdk/public/_build/results?buildId=385360&view=results

[sima-zhu]

This will be override by client setting. Storage has its own surefire setting.

[alzimmermsft]

Maven by default merges configuration, since this isn't set in Storage it should use this value. Another question is will Spock respect this configuration.

[JimSuplizio]

I'm not OK with this change and I do not think it should be checked in. Tests should be able to run in any order. If test A fails because test B ran first, then the tests should be fixed, not the ordering.

[mikeharder]

I'm not OK with this change and I do not think it should be checked in. Tests should be able to run in any order. If test A fails because test B ran first, then the tests should be fixed, not the ordering.

Ideally yes, and if we really wanted to enforce this we would set the default to random instead of either alphabetical or the default filesystem, and aggressively disable and fix tests until they all pass consistently. However, random has the downside of non-repeatable builds, so builds may randomly flip from passing to failing due to test order until all the tests have been fixed.

Between filesystem and alphabetical I definitely prefer alphabetical. alphabetical is completely deterministic regardless of OS or culture settings. filesystem is "mostly deterministic" meaning the order shouldn't change on a given OS and culture setting, but it can change across OS/culture. Thus filesystem is the worst of both worlds, tests may pass consistently on one OS/culture but fail on a different one (as we recently saw with ServiceBus tests on Ubuntu 18 with C culture).

So I propose we move from filesystem to alphabetical now as one step forward, then separately consider if we want to try enabling random with its pros and cons.

[mikeharder]

All pipelines pass with this change, so it should be safe if the team agrees this is a step forward.