Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daemon: unit-testable rpm-ostree wrapper and karg improvements #2360

Closed
wants to merge 1 commit into from
Closed

daemon: unit-testable rpm-ostree wrapper and karg improvements #2360

wants to merge 1 commit into from

Conversation

darkmuggle
Copy link
Contributor

This is a first-step towards unit-testable rpm-ostree commands by
creating a mockable rpm-ostree wrapper. To demonstrate the idea, kernel
arguments were plumbed into the new facility.

The NodeUpdateClient is the top level interface used by the daemon for
doing node updates and configuration. However, not all rpm-ostree
commands were using the CoreOS implementation of RpmOstreeClient. To
facilate mockability, all rpm-ostree commands are now routed
through RpmOstreeClient. Mocking of rpm-ostree command is done
through the new struct field of runRpmOstreeFunc. Unit tests, can use
this faculty to mock output of rpm-ostree.

The first unit-test that uses this mocking feature is kernel arguments.
rpm-ostree kargs --delete will only delete kargs that it knows about,
and attempting to delete a run-time or missing karg results in an
error. By mocking rpm-ostree the logic is of adding, removing and
updating kernel arguments is provable.

Finally, this does introduce a behavior change. Previously if an admin
adjusted the kargs outside of the MCD and upgrade would fail due to the
kargs being missing. The MCD will now tolerate the missing kargs and
apply the expected state. Unit tests have been added deal with
expecting, albeit runtime missing kargs.

Signed-off-by: Ben Howard ben.howard@redhat.com

- What I did

- How to verify it

- Description for the changelog

@darkmuggle darkmuggle added 4.8 Work deferred for 4.8 WIP labels Jan 23, 2021
@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2021
@openshift-ci-robot
Copy link
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 23, 2021
@darkmuggle
Copy link
Contributor Author

Throwing this up for early review. If the team likes the approach for unit-testable executables such as rpm-ostree, I can work out more of this. Obviously, this won't help with the CI situation but can help make testing faster by testing the logic and not needing an actual cluster. In the case of kargs I was able to prove my prior PR was logically incorrect due to splitting strings.

Anyway, if the team likes this approach I can continue with the work.

@darkmuggle
Copy link
Contributor Author

/test all

@darkmuggle darkmuggle changed the title [WIP] daemon: unit-testable rpm-ostree wrapper and karg improvements daemon: unit-testable rpm-ostree wrapper and karg improvements Feb 18, 2021
@darkmuggle darkmuggle removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 18, 2021
@darkmuggle
Copy link
Contributor Author

Okay, this is ready now that 4.8 is opened.

@darkmuggle darkmuggle marked this pull request as ready for review February 18, 2021 15:35
@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 18, 2021
Copy link
Member

@ashcrow ashcrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like having the functionality to be able to unit test! There is a lot in this PR however and I don't feel confident in my ability to be the only reviewer :-). I recommend having at least two other folks take a look as well. 👍

Copy link
Member

@cgwalters cgwalters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks sane to me, but I only gave this a surface level review, didn't deep dive into the new logic. Makes total sense to make some of this unit testable of course!

@cgwalters
Copy link
Member

Related to this we have coreos/rpm-ostree#2389 and I hope to add Go rpm-ostree bindings at some point and take some of the trickier interaction logic there. That might naturally lead to making things easier to mock.

That said I think by far the biggest MCO simplification would be if rpm-ostree implemented coreos/rpm-ostree#2326 (this is also related to ye olde #1190 ) - basically the MCO would just translate the MachineConfig bits into e.g. /etc/rpm-ostree.conf.d/mco.conf and run rpm-ostree rebuild - no need to manually diff packages/kargs etc, let rpm-ostree take care of reconciling all that.

Anyways I think the rpm-ostree Go bindings would be extracted from this repo, and it makes total sense from that PoV to have things like "reconcile kernel args" be part of that.

@darkmuggle darkmuggle removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 18, 2021
Copy link
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am generally in favour of splitting it up into interfaces, however with this PR I was unable to test the details due to an error (see below comments).

Also in general there are some points that may change behaviour (e.g. not allowing you to update OS now, whereas before it was a silent no-op, see below) that would block rhel nodes from updating. I think we should be very careful not to change that explicit behaviour.

@darkmuggle
Copy link
Contributor Author

@yuqi-zhang I dropped the errors for now to preserve the behavior. If this merges this week (no pressure -- it merges when it merges or not) I'll follow up with PR's that build on it.

@yuqi-zhang
Copy link
Contributor

/retest

// commands. These checks are not meant to be exhaustive.
switch noun {
// Argless commands
case "cancel", "rollblack":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rollblack -> rollback

// known error conditions
case "":
return nil, errNotEnoughArgs
default:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"status" is still not supported in this switch case right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah with this PR I get Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: unsupported command 'rpm-ostree status'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworked this to include full command validation, but this will need some serious testing.

@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ashcrow, cgwalters, darkmuggle

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@darkmuggle
Copy link
Contributor Author

/hold (again)

Putting this back into hold state pending validation on RHEL 7.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 23, 2021
This is a first-step towards unit-testable rpm-ostree commands by
creating a mockable rpm-ostree wrapper. To demonstrate the idea, kernel
arguments were plumbed into the new facility.

The NodeUpdateClient is the top level interface used by the daemon for
doing node updates and configuration. However, not all `rpm-ostree`
commands were using the CoreOS implementation of `RpmOstreeClient.` To
facilate mockability, all `rpm-ostree` commands are now routed
through `RpmOstreeClient`. Mocking of `rpm-ostree` command is done
through the new struct field of `runRpmOstreeFunc`. Unit tests, can use
this faculty to mock output of `rpm-ostree`.

The first unit-test that uses this mocking feature is kernel arguments.
`rpm-ostree kargs --delete` will only delete kargs that it knows about,
and attempting to delete a run-time or missing karg results in an
error. By mocking `rpm-ostree` the logic is of adding, removing and
updating kernel arguments is provable.

Finally, this does introduce a behavior change. Previously if an admin
adjusted the kargs outside of the MCD and upgrade would fail due to the
kargs being missing. The MCD will now tolerate the missing kargs and
apply the expected state. Unit tests have been added deal with
expecting, albeit runtime missing kargs.

Signed-off-by: Ben Howard <ben.howard@redhat.com>
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 21, 2021

@darkmuggle: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/okd-e2e-aws c23e9cb link /test okd-e2e-aws
ci/prow/e2e-aws-workers-rhel7 c23e9cb link /test e2e-aws-workers-rhel7
ci/prow/e2e-aws c23e9cb link /test e2e-aws
ci/prow/e2e-ovn-step-registry c23e9cb link /test e2e-ovn-step-registry
ci/prow/e2e-metal-ipi c23e9cb link /test e2e-metal-ipi
ci/prow/e2e-gcp-op c23e9cb link /test e2e-gcp-op
ci/prow/images c23e9cb link /test images
ci/prow/e2e-aws-serial c23e9cb link /test e2e-aws-serial
ci/prow/e2e-agnostic-upgrade c23e9cb link /test e2e-agnostic-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@darkmuggle
Copy link
Contributor Author

Withdrawling as stale.

@darkmuggle darkmuggle closed this May 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4.8 Work deferred for 4.8 approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. team-mco
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants