Upgrade PyYAML package from PyYAML-5.1.2.tar.gz to PyYAML 5.4.1

[mend-bolt-for-github]

This Bug is now used to upgrade PyYAML library from v 5.1.2 to 5.4.1

There are multiple vulnerabilities with following CVEs, 

 

CVE-2020-1747

CVE-2019-20477

CVE-2020-14343 

CVE-2019-20477

 

These vulnerabilities are expected to be fixed because of this upgrade

Description of sample vulnerability is mentioned below.

---

 

CVE-2020-14343 - High Severity Vulnerability

Vulnerable Library - PyYAML-5.1.2.tar.gz

YAML parser and emitter for Python

Library home page: [https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz]

Path to dependency file: cortx-ha/jenkins/pyinstaller/v1/requirements.txt

Path to vulnerable library: cortx-ha/jenkins/pyinstaller/v1/requirements.txt,cortx-ha/jenkins/pyinstaller/v2/requirements.txt

Dependency Hierarchy:

• ❌ PyYAML-5.1.2.tar.gz (Vulnerable Library)

Found in HEAD commit: 489a85b33aee06bc85dc7f2b7c71262cada47dd9

Found in base branch: main

Vulnerability Details

A vulnerability was discovered in the PyYAML library in versions before 5.4, where it is susceptible to arbitrary code execution when it processes untrusted YAML files through the full_load method or with the FullLoader loader. Applications that use the library to process untrusted input may be vulnerable to this flaw. This flaw allows an attacker to execute arbitrary code on the system by abusing the python/object/new constructor. This flaw is due to an incomplete fix for CVE-2020-1747.

Publish Date: 2021-02-09

URL: CVE-2020-14343

CVSS 3 Score Details (9.8)

Base Score Metrics:

• Exploitability Metrics:
• Attack Vector: Network
• Attack Complexity: Low
• Privileges Required: None
• User Interaction: None
• Scope: Unchanged
• Impact Metrics:
• Confidentiality Impact: High
• Integrity Impact: High
• Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14343]

Release Date: 2021-02-09

Fix Resolution: PyYAML - 5.4

---

⛑️ Automatic Remediation is available for this issue

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @indrajitzagade @ajaysrivas for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @indrajitzagade @ajaysrivas for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @indrajitzagade @ajaysrivas for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[cortx-admin]

Subhash Arya commented in Jira Server:

All components needs to bump up the Py-Yaml version. RE needs to use this version in the third party repo.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @s-arya @ajaysrivas for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[cortx-admin]

Swanand Gadre commented in Jira Server:

Initial build after making changes to custom build pipeline worked.

PyYAML 5.4.1 library is now uploaded for custom build.

 

Thanks [~936455]

 

Besides changes to PyYAML version in cortx-re, then next step required to update PyYAML version in cortx-utils repository as well. 

Next build with changes to cortx-re and cortx-utils build is failing

 

[~938581] and [~729494] are working on correcting Jenkin pipeline and will update once pipeline is corrected

[cortx-admin]

Nikhil Patil commented in Jira Server:

[~936818], there are a few changes required in the hare repo as well. because if we build a job using the latest 5.4.1 PyYAML version. In the "hare" repo, it will be trying to install the 5.4.3 PyYAML package which is present in - [https://github.com/Seagate/cortx-hare/blob/main/hax/requirements.txt] and [https://github.com/Seagate/cortx-hare/blob/main/provisioning/miniprov/requirements.txt.] and also we have to do the same thing in "cortx-utils" repo as well -  https://github.com/Seagate/cortx-utils/blob/main/py-utils/python_requirements.txt

[cortx-admin]

Nikhil Patil commented in Jira Server:

[~729494]

successful build job URL -  [http://eos-jenkins.colo.seagate.com/job/Release_Engineering/job/re-workspace/job/custom-ci-test-EOS-25657/68/console]
PR - [https://github.com/Seagate/cortx-re/pull/710]

[cortx-admin]

Gaurav Chaudhari commented in Jira Server:

{panel:bgColor=#c1c7d0}h2. cortx-ha - main branch build pipeline SUCCESS
h3. Build Info:

• Author: noreply

• Component Build : 89

• Release Build : 664

  h3. Artifact Location :

• http://cortx-storage.colo.seagate.com/releases/cortx/github/main/centos-7.9.2009/664
  {panel}

[cortx-admin]

Nikhil Patil commented in Jira Server:

done with the changes

[cortx-admin]

Nikhil Patil commented in Jira Server:

changes got merge to main

[cortx-admin]

Swanand Gadre commented in Jira Server:

Reopening this bug

---

[cortx-admin]

Swanand Gadre commented in Jira Server:

Work done so far

 

• PyYAML version 5.4.1 is uploaded to a custom location.
• custom ci pipeline was failing since last 2-3 weeks, which is now fixed.
• This jenkin pipeline is now able to build cortx build with PyYAML version changes in following repositories
• RE
• Utils
• Ha
• Hare
• After successful build, the deployment is triggered.
• This deployment is triggered with single VM - Non clustered environment.
• This deployment completed successfully.
• However PyYAML version installed on the system was still 5.1.2

 

• In order to troubleshoot this issue further, cortx-prereq was installed manually on a separate VM
• cortx-prereq indeed installed PyYAML version 5.4.1.
• So the conclusion is that some component is overriding PyYAML version 5.1.2 above (already installed 5.4.1 version).

 

I wanted to check on following questions

 

1 - Is this approach of deploying build in non clustered environment correct ? OR should we try deployment in kubernetes environment (which is relevant for LC)?
2 - If we deploy in kubernetes package, then custom image ghcr.io/seagate/cortx-all:2.0.0-1237-k8s-package is built with PyYAML 5.4.1. But how will it help in debugging the problem that some component is overriding PyYAML installation ?
3 - I am still not clear, is PyYAML version is really getting overridden, because typically python complains, if
someone tries to downgrade a version.
4 - How do we address this problem of some component overriding PyYAML version 5.1.2 above (already installed 5.4.1 version)?
5 - For subsequent testing, shall we do deployment in kubernetes environment ?

[cortx-admin]

Swanand Gadre commented in Jira Server:

From the earlier build experience, we know that changes in terms of 5.1.2 to 5.4.1 to following repos are required.
Which means these repos do not relie on RE (Which needs to be corrected).

But for this experiement , lets make changes to following repos

RE
https://github.com/swanand-gadre/cortx-re
File updated is scripts/third-party-rpm/python_requirements.txt

Utils
https://github.com/swanand-gadre/cortx-utils
File updated is cortx-utils/py-utils/python_requirements.txt

Ha
https://github.com/swanand-gadre/cortx-ha
File updated are
cortx-ha/jenkins/requirements.txt
cortx-ha/jenkins/pyinstaller/v1/requirements.txt
cortx-ha/jenkins/pyinstaller/v2/requirements.txt

Hare
https://github.com/swanand-gadre/cortx-hare
Following file already refers to 5.4.1
cortx-hare/cfgen/requirements.txt

Following files refers to 5.4.3 , so I updated them to 5.4.1
Changes are for types-PyYAML=5.4.3 to types-PyYAML=5.4.1
hax/requirements.txt
provisioning/miniprov/requirements.txt

Till this point build and deployment was tested and build was getting successful, but deployment was showing 5.1.2
So I also made changes to prvsnr, as next steps

prvsnr
https://github.com/swanand-gadre/cortx-prvsnr
Changes are made in following files - 5.1.2

srv/components/system/files/cortx_py_utils_requirements.txt
test_requirements.txt

---

An image is build based on changes to these repos.

Then image is run to check PyYAML version and it gave following output

 

[root@ssc-vm-g3-rhev4-2777 ~]# docker run --rm ghcr.io/seagate/cortx-all:2.0.0-1265-custom-ci pip3 show PyYAML
Name: PyYAML
Version: 5.4.1
Summary: YAML parser and emitter for Python
Home-page: https://pyyaml.org/
Author: Kirill Simonov
Author-email: xi@resolvent.net
License: MIT
Location: /usr/local/lib64/python3.6/site-packages
Requires:

 

This means, when files in pvrsnr were changed, final PyYAML version remains 5.4.1

So changes in prvsnr doe upgrade of PyYAML..

Further in order to confirm this theory, I am changing pvrsnr to PyYAML 5.1.2 and will build the image.

Meanwhile, I will also continue deployment of the image which has 5.4.1 to continue for subsequent tests

 

 

[cortx-admin]

Swanand Gadre commented in Jira Server:

Testing for this library upgrade is complete
PRs will be ready for review / approval (expected today or tomorrow)

PRs will have code changes for

• RE,
• Utils and
• prvsnr components

[cortx-admin]

Swanand Gadre commented in Jira Server:

Closing this bug now, as the libraries are upgraded

[cortx-admin]

Swanand Gadre commented in Jira Server:

This library is now upgraded to 5.4.1

So closing this issue.

[cortx-admin]

Swanand Gadre commented in Jira Server:

This library is upgraded , so closing this issue now.

[cortx-admin]

Gaurav Chaudhari commented in Jira Server:

{panel:bgColor=#c1c7d0}h2. cortx-ha - main branch build pipeline SUCCESS
h3. Build Info:

• Author: noreply

• Component Build : 303

• Release Build : 790

  h3. Artifact Location :

• http://cortx-storage.colo.seagate.com/releases/cortx/github/main/centos-7.9.2009/790
  {panel}

[cortx-admin]

Swanand Gadre commented in Jira Server:

Reopening this issue, as cortx-test repository refers to PyYAML 5.1.2 , which needs to be upgraded

Regards
Swanand

[cortx-admin]

Swanand Gadre commented in Jira Server:

PyYAML vulnerability is no more reported in Mend for cortx-test.

So closing this defect as fixed

[cortx-admin]

Swanand Gadre commented in Jira Server:

PyYAML vulnerability is no more reported in Mend for cortx-test.

So closing this defect as fixed

[cortx-admin]

Swanand Gadre commented in Jira Server:

PyYAML vulnerability is no more reported in Mend for cortx-test.

So closing this defect as fixed

[cortx-admin]

Swanand Gadre commented in Jira Server:

PyYAML vulnerability is no more reported in Mend for cortx-test.

So closing this defect as fixed

[cortx-admin]

Donald R Bloyer commented in Jira Server:

[~521878]  can you close this if there is a agreement this is addressed.

[~744427] Do we need someone from Opensource or something (because this appears to be BoardGenius) to take action to close?

[cortx-admin]

Sonal Kalbende commented in Jira Server:

Cortx-test has updated PyYaml version and we are using  pyyaml==6.0.0 now.

 

[cortx-admin]

Sonal Kalbende commented in Jira Server:

Closing as per last comment. cc: [~932497]