Add PyYAML 5.3.1 as a dependency to Helix machines running the Python scripts

[ivdiazsa]

Overview and Motivation

As part of the effort for the Merge-on-Red project, a new YAML-based logger was implemented to the runtime repo's test source generators. Now, each test run outputs the usual Xml results file, as well as a new Yaml one alongside it. The motivation for this effort is that when a test run crashes and/or freezes terribly badly, we end up with an incomplete Xml file, which is not readable by parsers. On the other hand, even if it's incomplete, a YAML file is still valid and parseable thanks to its lack of closing indicators.

What we need from Arcade

Add PyYAML version 5.3.1 to the list of requirements that have to be satisfied by pip in the Helix machines.

Why we need this

Now that we have our YAML-based logger, we have to "teach" the Helix machines how to interpret it (this will be done in a separate PR currently in the works). However, Python does not have native support for YAML processing, so we need an external library. For this purpose, we are opting to use PyYAML, which is one of the most stable and widely used YAML libraries for Python. And therefore, we need to have it installed on the machines that run the tests.

Detailed Information on the Project

• Merge-on-Red is thoroughly explained here: [User Story] CI Health: Redefining CI investigations and Health runtime#75243
• As for this particular work item, we are tracking it here: [Merge-on-Red] - Accurately log catastrophic test failures and freezes in source-gened test infrastructure runtime#77735 (This one is also linked in the Merge-on-Red issue specified above)

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Add PyYaml to python packages preinstalled on Helix agents

[ivdiazsa]

Hi @MattGal! Here's a detailed explanation of the motivation behind adding this PyYAML dependency, alongside with links to the project it's part of.

[MattGal]

https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/27390

[MattGal]

@ivdiazsa unfortunately as is often the case with adding python dependencies in Helix we must deal with a set of dependencies that works across the oldest possible Linux machines supported. In the case of trying to add this dependency, I've run smack into the fact that ubuntu 18.04 (and likely others; this was a first-ring validation test) comes with PyYaml 3.12 which being installed as by the "real" package manager cannot be uninstalled by Pip, and if it does get uninstalled / upgraded it may well bring knock-on effects to Ubuntu 18.04 OS components that we don't notice for some time.

I will keep poking at the problem, just wanted to share that this is turning out to be trickier than first expected (unless you can be satisfied with 3.12, that is...)

[ivdiazsa]

That's unfortunate @MattGal. I think there should be no problem whatsoever if we use 3.12 in this case. Let me try to test it.

[ivdiazsa]

Follow up question. What version of Python are the Helix machines using? This just to try to emulate that environment as closely as possible, and find other potential issues if any.

[ivdiazsa]

@MattGal Sorry for the spam. Just tested and so far, it seems 3.12 works just fine so we can move on with that for the time being.

[MattGal]

@ivdiazsa I ended up with pyYaml 3.11 and to answer your questions, windows machines use 3.10 currently while linux and MacOS use what the OS comes with specifically because changing this in any significant way leads to OS problems. You can assume we have always Python 3 >= 3.4 if you like using fancy new language features.

[ivdiazsa]

@ivdiazsa I ended up with pyYaml 3.11 and to answer your questions, windows machines use 3.10 currently while linux and MacOS use what the OS comes with specifically because changing this in any significant way leads to OS problems. You can assume we have always Python 3 >= 3.4 if you like using fancy new language features.

Perfect! That's completely fine. Thanks for your help Matt! Let me know when it's ready, so we can close this issue and I can submit the PR for the next phase of the project.

[MattGal]

@ivdiazsa to set expectations, I'd guess not until 11/30, as it didn't make today's rollout and we'll be skipping next week's rollout. I'll close this issue once it has been made generally available.

[MattGal]

This is rolled out and should be usable on all Helix agents, please let us know if you hit issues.

[ivdiazsa]

Thanks a lot @MattGal! I'll let you know if we hit any issues.