Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CP-46944: Update yum plugins to dnf plugins #5526

Merged
merged 3 commits into from
Mar 26, 2024

Conversation

liulinC
Copy link
Collaborator

@liulinC liulinC commented Mar 22, 2024

XS9 does not have yum command, and yum plugin is used to add HTTP headers automatically for yum commands when

  • access pool master repo. (ptoken)
  • access remote repo for CDN udpate (accesstoken)

This is updating the yum plugins to corresponding DNF plugins, and enable dnf with the same functionality.

Test done:

  • XenRT test
    • XS9: 3961239. (The suite cover sync updates from pool cordinator, thus covered the plugins)
    • XS8: 3961239 (XS8 is not related, just run for sure)

Manul test:

  • Add print in the plugins to make sure it is called. For example.
    print("set ptoken: ", f'cookie:{secret}')
    print("set accesstoken: ", access_token, "tokenid: ", referer)
dnf --disablerepo=* --enablerepo=local-6192a1a6-cda9-4ba1-e4a6-e36148fcb239,local-e6445302-c68d-8154-647c-785d93e59307 check-update
set ptoken:  cookie:pool_secret=70d0ef24-832b-adef-afec-f06011eed6d9/7993fa04-59e1-5ea3-016d-8908c63764cb/05feb002-bb82-8b7f-a025-db6805ea5525
set ptoken:  cookie:pool_secret=70d0ef24-832b-adef-afec-f06011eed6d9/7993fa04-59e1-5ea3-016d-8908c63764cb/05feb002-bb82-8b7f-a025-db6805ea5525
local-6192a1a6-cda9-4ba1-e4a6-e36148fcb239                                                                52 kB/s | 991  B     00:00    
local-e6445302-c68d-8154-647c-785d93e59307 ```


  • run xe pool-sync-updates token=this_is_token token-id=this_is_token_id
/usr/bin/dnf --disablerepo=* --enablerepo=remote-6192a1a6-cda9-4ba1-e4a6-e36148fcb239 -y makecache succeeded [ output = 'set ptoken:  cookie:pool_secret=70d0ef24-832b-adef-afec-f06011eed6d9/7993fa04-59e1-5ea3-016d-8908c63764cb/05feb002-bb82-8b7f-a025-db6805ea5525\x0Aset ptoken:  cookie:pool_secret=70d0ef24-832b-adef-afec-f06011eed6d9/7993fa04-59e1-5ea3-016d-8908c63764cb/05feb002-bb82-8b7f-a025-db6805ea5525\x0Aset accesstoken:  X-Access-Token:this_is_token tokenid:  Referer:this_is_token_id\x0Aremote-6192a1a6-cda9-4ba1-e4a6-e36148fcb239     5.0 kB/s | 991  B     00:00    \x0AMetadata cache created.\x0A' ]

call path: sync_updates --> set_available_updates -> get_hosts_updates -> http_get_host_updates_in_json --> Xapi_http.http_request Constants.get_host_updates_uri

Signed-off-by: Lin Liu <lin.liu@citrix.com>
Copy link

codecov bot commented Mar 22, 2024

Codecov Report

Merging #5526 (266edbc) into feature/xs9 (3d5f055) will increase coverage by 1.9%.
Report is 1 commits behind head on feature/xs9.
The diff coverage is 100.0%.

Additional details and impacted files
@@              Coverage Diff              @@
##           feature/xs9   #5526     +/-   ##
=============================================
+ Coverage         51.8%   53.7%   +1.9%     
=============================================
  Files               19      23      +4     
  Lines             2586    2694    +108     
=============================================
+ Hits              1341    1449    +108     
  Misses            1245    1245             
Files Coverage Δ
python3/dnf_plugins/accesstoken.py 100.0% <100.0%> (ø)
python3/dnf_plugins/ptoken.py 100.0% <100.0%> (ø)
python3/tests/stubs/dnf.py 100.0% <100.0%> (ø)
python3/tests/test_dnf_plugins.py 100.0% <100.0%> (ø)
Flag Coverage Δ
python2.7 53.6% <ø> (ø)
python3.11 60.0% <100.0%> (+1.8%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@liulinC liulinC force-pushed the private/linl/dnfp branch 7 times, most recently from 970632a to eb59b89 Compare March 22, 2024 07:50
@liulinC
Copy link
Collaborator Author

liulinC commented Mar 22, 2024

Investigate the python CI failures, It is somehow NOT scanned, try to re-open a new one.
Edit: Issue resolved by providing mock stub.

@liulinC liulinC closed this Mar 22, 2024
@liulinC liulinC reopened this Mar 22, 2024
@liulinC liulinC force-pushed the private/linl/dnfp branch 4 times, most recently from dde0d5a to 2b1133f Compare March 22, 2024 09:38
github CI has issues with
- ubuntu-22.04 container
- python11
- python3-dnf

Instead of patching the upstreams, we just provide a stub for
the fix

Signed-off-by: Lin Liu <lin.liu@citrix.com>
def test_repo_without_access_token(self, mock_grabber):
"""If repo has not accestoken, it should not be blocked"""
mock_repo = _mock_repo()
accesstoken.AccessToken(mock_repo.base, MagicMock()).config()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need a assert_not_called ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comments, the target is to test repo without accesstoken still work, so this case does not care whether it is called or not,
but it has no bad to put a no called there anyway.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit: just updated to follow the suggestion.

repo = self.base.repos[repo_name]

if len(repo.baseurl) > 0 and repo.baseurl[0].startswith("http://127.0.0.1") \
and repo.ptoken:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a newly added config item for dnf? May I please know the reason to add this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this require changes on xapi code which populate the repo configurations?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you mean ptoken?

No, this is an optional configure item for the repo, to decide whether we should enable pool token for a repo.
it is used by yum, and I patched dnf to support this.

No extra xapi code change is required for this, on the other side, dnf is pathed to support this as xapi use it.

def _mock_repo(a_token=None, p_token=None, baseurl=None):
mock_repo = MagicMock()
mock_repo.accesstoken = a_token
mock_repo.ptoken = p_token
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is a good reason to add this "ptoken" config in dnf repo, then what information it would contain?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a boolean to indicate whether enable pool token for this repo. refer to https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/repository_helpers.ml#L432


class Ptoken(dnf.Plugin):
"""ptoken plugin class"""
name = "ptoken"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this "name" is required by dnf plugin?
(Would be better to keep a consistent formating style for both plugins, e.g. the blank lines here.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it just follow the dnf plugin protocols, find this in the comments:
refer to https://dnf.readthedocs.io/en/latest/api_plugins.html

Signed-off-by: Lin Liu <lin.liu@citrix.com>
@liulinC liulinC merged commit d1a3219 into xapi-project:feature/xs9 Mar 26, 2024
28 checks passed
Copy link

pytype_reporter extracted 50 problem reports from pytype output

.

You can check the results of the job here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants