Use an uncached direct client to read the kcp and md to bypass delay in cache

[rahulbabu95]

Issue #, if available:
Some of the InPlace tests fail intermittently after a successful upgrade during the validation phase as some of the InPlace CRs are not cleaned up. We use an in-place-upgrade-needed on the kcp and md objects to identify the InPlace flow and remove those annotations once the upgrade is successful. In some cases, after the InPlace upgrade is successful the cache used to read the state of objects doesn't get updated quick enough. This leads to our controllers creating the InPlace custom resources again which leads to another upgrade on the node. Moreover since the annotation is removed, cache does get updated eventually leaving these objects in an untracked state as the kcp and md controller do not reconcile for InPlace flow anymore after the removal of annotation.

Description of changes:
Use an uncached direct client to read kcp and md, so that we track these objects on realtime. This way we can rely on annotations on the objects to reconcile for the InPlace flow.

Testing (if applicable):
Ran all InPlace tests locally.

Documentation added/planned (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 73.61%. Comparing base (4583834) to head (5dbcc13).
Report is 171 commits behind head on main.

Files	Patch %	Lines
controllers/kubeadmcontrolplane_controller.go	82.60%	3 Missing and 1 partial ⚠️
controllers/machinedeployment_controller.go	82.60%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7779      +/-   ##
==========================================
+ Coverage   73.48%   73.61%   +0.13%     
==========================================
  Files         579      588       +9     
  Lines       36357    37173     +816     
==========================================
+ Hits        26718    27366     +648     
- Misses       7875     8015     +140     
- Partials     1764     1792      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[abhinavmpandey08]

/approve
/lgtm
/woof

[eks-distro-bot]

@abhinavmpandey08:

In response to this:

/approve
/lgtm
/woof

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[eks-distro-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavmpandey08

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhinavmpandey08]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rahulbabu95]

/cherrypick release-0.19

[eks-distro-pr-bot]

@rahulbabu95: new pull request created: #7786

In response to this:

/cherrypick release-0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.