Improve Azure Pipelines support #1991
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dscho
force-pushed
the
azure-pipelines-g4w
branch
7 times, most recently
from
31c30cf to
70c3380
Compare
dscho
force-pushed
the
azure-pipelines-g4w
branch
20 times, most recently
from
451b670 to
2665ad0
Compare
dscho
force-pushed
the
azure-pipelines-g4w
branch
from
2665ad0 to
0009277
Compare
TASKDEFINITIONSURI is not set in parallel tasks... And COLLECTIONURI is not set in the Linux32 job... Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For some reason, the indentation of the PowerShell tasks got all messed up. Fix that. While at it, do not define `c()` for only one user, and do not initialize `/etc/passwd` (it does not seem to have a noticeable effect). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
At some point we switched from Hosted Linux Preview to Ubuntu 16.04, at which point it became unnecessary to install the `docker` package for the Linux-32 job. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Remove a trailing empty line. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Let's use the new YAML schema. See https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=vsts&tabs=schema Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Some improvements (suggested via review on the Git mailing list) have not been incorporated into Git for Windows' `master` yet, other changes were left-overs from debugging that need reverting. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We do not need build-extra. So let's not clone it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
force-pushed
the
azure-pipelines-g4w
branch
from
d9a49a6 to
a5b21d3
Compare
It is not strictly necessary to start a command in PowerShell with `&`, but it is safer. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Windows job currently takes a whopping ~1h20m to complete. Which is *far* longer than the next-longest job takes (linux-gcc, ~35m). As such, it makes sense to start the Windows job first, to minimize the overall run time (which is now pretty safely the run time of the Windows job). This affects only the Azure Pipelines configuration, not the Travis one, of course, as Travis cannot run our Windows job: 1h20m is distinctly longer than the 50 minute timeout of Travis' free tier. This commit is best viewed with `--color-moved`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
force-pushed
the
azure-pipelines-g4w
branch
6 times, most recently
from
73cbff7 to
e4facb6
Compare
Instead of a shallow fetch followed by a sparse checkout, we are better off by using a separate, dedicated Pipeline that bundles the SDK as a build artifact, and then consuming that build artifact here. In fact, since this artifact will be used a lot, we spent substantial time on figuring out a minimal subset of the Git for Windows SDK, just enough to build and test Git. The result is a size reduction from around 1GB (compressed) to around 55MB (compressed). This also comes with the change where we now call `usr\bin\bash.exe` directly, as `git-cmd.exe` is not included in the minimal SDK. That reduces the time to initialize Git for Windows' SDK from anywhere between 2m30s-7m to a little over 1m. Note: in theory, we could also use the DownloadBuildArtifacts@0 task here. However, restricted permissions that are in effect when building from forks would let this fail for PR builds, defeating the whole purpose of the Azure Pipelines support for git.git. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Every once in a while, the Azure Pipeline fails with some semi-random error: timer thread did not terminate timely This error message means that the thread that is used to emulate the setitimer() function did not terminate within 1,000 milliseconds. The most likely explanation (and therefore the one we should assume to be true, according to Occam's Razor) is that the timeout of one second is simply not enough because we try to run so many tasks in parallel. So let's give it ten seconds instead of only one. That should be enough. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While working on parallelizing the tests in Azure Pipelines, an issue was discovered with the `is_msys2_sh()` function: it expects the path components to be separated by exactly one dir separator. That does not need to be the case, though, e.g. when the components in the `PATH` variable have trailing slashes. Let's make the code much more robust in this respect. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make sure to write the .xml in UTF-8 encoding. We also need to make sure that invalid UTF-8 encoding is turned into valid UTF-8 (using the Replacement Character, \uFFFD) because t9902's trace contains such invalid byte sequences, and the task that uploads the test results would refuse to do anything if it was asked to parse an .xml file with invalid UTF-8 in it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is a bit ridiculous to spin up a full-blown Perl instance (especially on Windows, where that means spinning up a full POSIX emulation layer, AKA the MSYS2 runtime) just to tell how large a given file is. So let's just use the test-tool to do that job instead. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
… failure Let's use the new `test-tool path-utils file-size` command in the new JUnit XML feature. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add the comment that we added to the patch series intended for upstream git.git. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
force-pushed
the
azure-pipelines-g4w
branch
from
fd09704 to
6ee6efe
Compare
The fact that Git's test suite is implemented in Unix shell script that is as portable as we can muster, combined with the fact that Unix shell scripting is foreign to Windows (and therefore has to be emulated), results in pretty abysmal speed of the test suite on that platform, for pretty much no other reason than that language choice. For comparison: while the Linux build & test is typically done within about 8 minutes, the Windows build & test typically lasts about 80 minutes in Azure Pipelines. To help with that, let's use the Azure Pipeline feature where you can parallelize jobs, make jobs depend on each other, and pass artifacts between them. The tests are distributed using the following heuristic: listing all test scripts ordered by size in descending order (as a cheap way to estimate the overall run time), every Nth script is run (where N is the total number of parallel jobs), starting at the index corresponding to the parallel job. This slicing is performed by a new function that is added to the `test-tool`. To optimize the overall runtime of the entire Pipeline, we need to move the Windows jobs to the beginning (otherwise there would be a very decent chance for the Pipeline to be run only the Windows build, while all the parallel Windows test jobs wait for this single one). We use Azure Pipelines Artifacts for both the minimal Git for Windows SDK as well as the built executables, as deduplication and caching close to the agents makes that really fast. For comparison: while downloading and unpacking the minimal Git for Windows SDK via PowerShell takes only one minute (down from anywhere between 2.5 to 7 when using a shallow clone), uploading it as Pipeline Artifact takes less than 30s and downloading and unpacking less than 20s (sometimes even as little as only twelve seconds). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
force-pushed
the
azure-pipelines-g4w
branch
from
6ee6efe to
3346fdc
Compare
|
This is now in sync with gitgitgadget#31 (which I just submitted as v3). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Let's backport a few changes from the review on the Git mailing list, really drop the
test_atexitcommits, drop the apt lock hack, parallelize the test suite on Windows, etcEventually all of this PR should be squashed into the next iteration of gitgitgadget#31