Agent update refactor supports GA versioning #2810
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nagworld9
requested review from
narrieta,
ZhidongPeng and
maddieford
as code owners
Codecov Report
@@ Coverage Diff @@
## develop #2810 +/- ##
===========================================
+ Coverage 71.96% 72.08% +0.12%
===========================================
Files 104 106 +2
Lines 15881 15952 +71
Branches 2274 2287 +13
===========================================
+ Hits 11428 11499 +71
+ Misses 3930 3919 -11
- Partials 523 534 +11
|
|
test |
narrieta
reviewed
Apr 27, 2023
tests_e2e/tests/lib/retry.py
Outdated
| @@ -57,3 +57,23 @@ def retry_ssh_run(operation: Callable[[], Any]) -> Any: | |||
| raise | |||
| log.warning("The operation failed, retrying in 30 secs.\n%s", e) | |||
| time.sleep(30) | |||
|
|
|||
|
|
|||
| def retry_if_not_found(operation: Callable[[], bool], attempts: int = 5) -> bool: | |||
There was a problem hiding this comment.
"if_not_found" implies some kind of search, how about "retry_if_false". Let's also make the sleep time a parameter with a default value.
narrieta
reviewed
Apr 27, 2023
| goal_state_properties=GoalStateProperties.ExtensionsGoalState) | ||
|
|
||
| attempts = 5 | ||
| while attempts > 0: |
There was a problem hiding this comment.
should this be using your new retry function?
There was a problem hiding this comment.
Let me see If I can reuse
narrieta
reviewed
Apr 27, 2023
| print("RSM requested version GS not available yet to the agent, checking again in 30 secs.") | ||
| attempts -= 1 | ||
| time.sleep(30) | ||
|
|
There was a problem hiding this comment.
should this fail if we never get the expected goal state?
There was a problem hiding this comment.
yes, it should fail. I'll update.
narrieta
reviewed
Apr 27, 2023
| poller.wait(timeout=timeout) | ||
|
|
||
| if not poller.done(): | ||
| raise TimeoutError(f"Failed to restart {self._identifier.name} after {timeout} seconds") |
There was a problem hiding this comment.
Good catch, failed to restart doesn't make sense. I'll change
narrieta
reviewed
May 8, 2023
| # BVT for the agent update scenario | ||
| # | ||
| # The test verifies agent update for rsm workflow. This test covers three scenarios downgrade, upgrade and no update. | ||
| # For each scenario, we intiaite the rsm request with target version and then verify agent updated to that target version. |
| found: bool = retry_if_not_found(_check_agent_supports_versioning) | ||
|
|
||
| if not found: | ||
| raise Exception("Agent failed to report supported feature flag, so skipping agent update validations") |
There was a problem hiding this comment.
Test failures should be reported with AssertionErrors (i.e. the verification should be done using assertpy)
| RSM update rely on supported flag that agent sends to CRP.So, checking if GA reports feature flag from the agent log | ||
| """ | ||
| def _check_agent_supports_versioning() -> bool: | ||
| found: str = self._ssh_client.run_command("grep -q 'Agent.*supports GA Versioning' /var/log/waagent.log && echo true || echo false").rstrip() |
There was a problem hiding this comment.
this should be checking the agent status file, rather than the agent log
| return ignore_rules | ||
|
|
||
| def run(self) -> None: | ||
| # Allow agent to send supported feature flag |
There was a problem hiding this comment.
we should also add a test to verify agent update was attempted on VM creation
There was a problem hiding this comment.
RSM update will not be attempted on vm creation as CRP requires feature flag to drive rsm request.
| self._verify_agent_reported_supported_feature_flag() | ||
|
|
||
| log.info("*******Verifying the Agent Downgrade scenario*******") | ||
| self._mock_rsm_update("1.3.0.0") |
There was a problem hiding this comment.
We shouldn't use "mock" in the names/comments, since the end-to-end tests do not use any mocks. This is triggering an actual request.
There was a problem hiding this comment.
Previous PRs I was suggested to rename to mock.
| @@ -0,0 +1,66 @@ | |||
| #!/usr/bin/env pypy3 | |||
There was a problem hiding this comment.
probably a better name for the script would be "wait_for_rsm_goal_state.py"
| goal_state = protocol.client.get_goal_state() | ||
| requested_version = get_requested_version(goal_state) | ||
| if requested_version == args.version: | ||
| print("Latest GS includes rsm requested version : {0}.".format(requested_version)) |
There was a problem hiding this comment.
these should be logger.info rather than print to include a timestamp
There was a problem hiding this comment.
will this log to agent log If I use logger.info? I don't want to add to agent log. This will create confusion with agent rsm related messages.
| def _check_rsm_gs(self, requested_version: str) -> None: | ||
| # This checks if RSM GS available to the agent after we mock the rsm update request | ||
| output = self._ssh_client.run_command(f"rsm_goal_state.py --version {requested_version}", use_sudo=True) | ||
| log.info('Verifying requested version GS available to the agent \n%s', output) |
There was a problem hiding this comment.
if we don't get the RSM goal state we should raise an exception
| @@ -0,0 +1,33 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
we need a better name for this script. "agent-update-config" is too generic
| self._check_rsm_gs("1.3.1.0") | ||
| self._verify_guest_agent_update("1.3.1.0") | ||
|
|
||
| # verify no version update. There is bug in CRP and will enable once it's fixed |
There was a problem hiding this comment.
CRP not populating GA family in GS if requested version is same as last successful attempted updated version.
Before I was on vacation, Kashif mentioned change was merged, waiting for next CRP deployment
narrieta
reviewed
May 9, 2023
| def get_ga_updates_enabled(conf=__conf__): | ||
| """ | ||
| If True, the agent go through update logic to look for new agents otherwise it will stop agent updates. | ||
| NOTE: This option is needed in e2e tests to control agent updates. |
There was a problem hiding this comment.
If only needed by the tests, this should be a Debug/experimental option
There was a problem hiding this comment.
How is this different than auto update?
There was a problem hiding this comment.
we can't use the auto update flag because of legacy behavior associated with it where agent fallback to daemon version which we don't want as our e2e tests need to run on test version
There was a problem hiding this comment.
@nagworld9 - did we decide this to expose this option to the user? if yes, let's add it to README, if not, let's make it Debug
There was a problem hiding this comment.
sorry, what is the difference between GAUpdates.Enabled and Debug.EnableGAVersioning
There was a problem hiding this comment.
No, this is not exposed to the user. I'll change to debug.
Debug.EnableGAVersioning- This flag is like kill switch. Once this is trun off our code will go to self update path.
GAUpdates.Enabled- This flag is to stop the agent updates. Only used in e2e tests.
There was a problem hiding this comment.
I'll change as part of next work item.
narrieta
reviewed
May 9, 2023
azurelinuxagent/common/event.py
Outdated
| @@ -127,6 +127,7 @@ class WALAEventOperation: | |||
| Update = "Update" | |||
| VmSettings = "VmSettings" | |||
| VmSettingsSummary = "VmSettingsSummary" | |||
| FeatureFlag = "FeatureFlag" | |||
There was a problem hiding this comment.
"FeatureFlag" is too generic, can we use a different name?
There was a problem hiding this comment.
This is meant to reuse for multiple features flags reporting not just for agent update.
There was a problem hiding this comment.
If it is meant for multiple features I think the message should be json or something like that so that we can parse it for individual features. At this point it is used only for agent versioning. If you want to use it for all features, you should also use it for multi-config and fastrack.
I think you should make this one specific to versioning
narrieta
reviewed
May 9, 2023
| @@ -0,0 +1,316 @@ | |||
| import json | |||
There was a problem hiding this comment.
this module needs a better name, "guestagent.py" is too generic
narrieta
reviewed
May 15, 2023
azurelinuxagent/ga/agent_update.py
Outdated
| @@ -0,0 +1,340 @@ | |||
| import datetime | |||
There was a problem hiding this comment.
let's rename this file to "agent_update_handler.py"
azurelinuxagent/ga/update.py
Outdated
| @@ -635,7 +549,11 @@ def _process_goal_state(self, exthandlers_handler, remote_access_handler): | |||
| CGroupConfigurator.get_instance().check_cgroups(cgroup_metrics=[]) | |||
|
|
|||
| # report status before processing the remote access, since that operation can take a long time | |||
| self._report_status(exthandlers_handler) | |||
| agent_update_status = agent_update_handler.get_vmagent_update_status() | |||
| self._report_status(exthandlers_handler, agent_update_status) | |||
There was a problem hiding this comment.
self._report_status should call agent_update_handler.get_vmagent_update_status() instead of receiving agent_update_status as argument
azurelinuxagent/ga/update.py
Outdated
| @@ -737,6 +618,24 @@ def _report_extensions_summary(self, vm_status): | |||
| logger.warn(msg) | |||
| add_event(op=WALAEventOperation.GoalState, is_success=False, message=msg) | |||
|
|
|||
| def _log_agent_supports_versioning_or_not(self): | |||
There was a problem hiding this comment.
Do we need to log this at all? Once this code is merged the feature will always be supported.
There was a problem hiding this comment.
Feature is supported by default but cx can turn it off if GA versioning flag is disabled.
and also, we introduced this logging when e2e need for it to know feature is supported or not by the agent.
There was a problem hiding this comment.
(see above comment: the test can check the status that the agent reports rather than the log)
If we want to log that the customer disabled the feature let's add a message specific for that. we have several of those messages during agent initialization at the beginning of the run() method
azurelinuxagent/ga/update.py
Outdated
| agent_update_status = agent_update_handler.get_vmagent_update_status() | ||
| self._report_status(exthandlers_handler, agent_update_status) | ||
|
|
||
| # Logging after agent reports supported feature flag so this msg in sync with report status |
There was a problem hiding this comment.
I'm not sure I understand this comment. We would always log that the agent supports versioning, right?
There was a problem hiding this comment.
yes, we do but we are logging this after we report status because e2e tests rely on this in agent update scenario
There was a problem hiding this comment.
i see... i think your test should instead check the status that we save to the history folder, would be more robust than checking the log
| MAX_FAILURE = 3 # Max failure allowed for agent before declare bad agent | ||
|
|
||
|
|
||
| class GAUpdateReportState(object): |
There was a problem hiding this comment.
These 2 variables are globals. We should not introduce globals.
| report_expected_version = FlexibleVersion("0.0.0.0") | ||
|
|
||
|
|
||
| class GuestAgent(object): |
There was a problem hiding this comment.
We should split this class in code to download the agent and code to get the information of agents that are already downloaded. We can follow offline.
There was a problem hiding this comment.
I'll refactor in different work item
narrieta
reviewed
May 16, 2023
azurelinuxagent/ga/agent_update.py
Outdated
| self._protocol = protocol | ||
| self._ga_family = conf.get_autoupdate_gafamily() | ||
| self._autoupdate_enabled = conf.get_autoupdate_enabled() | ||
| self._gs_id = self._protocol.get_goal_state().extensions_goal_state.id |
There was a problem hiding this comment.
Why initialize the goal state ID in init, when we are passing the goal state as parameter to run?
There was a problem hiding this comment.
we are updating the gs_id in the run with goal state parameter. The initialization here is just placeholder value.
There was a problem hiding this comment.
maybe initialize it to "not-initialized", "unknown" or something like that? we are going to remove the instance of the goal state from the protocol class
azurelinuxagent/ga/agent_update.py
Outdated
| self._autoupdate_enabled = conf.get_autoupdate_enabled() | ||
| self._gs_id = self._protocol.get_goal_state().extensions_goal_state.id | ||
| self._is_requested_version_update = True # This is to track the current update type(requested version or self update) | ||
| self.persistent_data = AgentUpdateHandlerUpdateState() |
There was a problem hiding this comment.
can we rename this to "update_state" or similar? "persistent" gives the impression it is saved to disk
azurelinuxagent/ga/agent_update.py
Outdated
|
|
||
| return next_hotfix_time, next_normal_time | ||
|
|
||
| def __get_agent_family_from_last_gs(self, goal_state): |
There was a problem hiding this comment.
rename to "__get_agent_family_manifests" or similar
azurelinuxagent/ga/agent_update.py
Outdated
| family = self._ga_family | ||
| agent_families = goal_state.extensions_goal_state.agent_families | ||
| agent_family_manifests = [m for m in agent_families if m.name == family and len(m.uris) > 0] | ||
| if len(agent_family_manifests) == 0: |
There was a problem hiding this comment.
can we report the 2 possible conditions separately? (family not found vs emtpty uris)
azurelinuxagent/ga/agent_update.py
Outdated
| def __get_requested_version(agent_family): | ||
| """ | ||
| Get the requested version from agent family | ||
| Returns: Requested version if supported and available |
There was a problem hiding this comment.
Not "supported", but "enabled"
There was a problem hiding this comment.
sorry, i meant the comment should read "Returns: Requested version if enabled and available" (not sure about the "available" part, though. what does it mean that agent versioning is available?
azurelinuxagent/ga/agent_update.py
Outdated
| if requested_version is None: | ||
| if conf.get_enable_ga_versioning(): # log the warning only when ga versioning is enabled | ||
| warn_msg = "Missing requested version in agent family: {0} for incarnation: {1}, fallback to largest version update".format(self._ga_family, self._gs_id) | ||
| self.__log_event(LogLevel.WARNING, warn_msg) |
There was a problem hiding this comment.
rather than logging this event every 6 hours, it should be done on new goal state
There was a problem hiding this comment.
This could swamp the agent log with this event. It's possible that every goal state missing requested version if there is any bug in crp
There was a problem hiding this comment.
i don't think this would swamp the log, if goal states were that frequent then we have bigger problems, since we log quite a few messages in each goal state.
now, coming to think about it, we should probably log it when we actually attempt the update, otherwise i think the message is not interesting
narrieta
reviewed
May 18, 2023
azurelinuxagent/ga/agent_update.py
Outdated
| warn_msg = "Missing requested version in agent family: {0} for incarnation: {1}, fallback to largest version update".format(self._ga_family, self._gs_id) | ||
| self.__log_event(LogLevel.WARNING, warn_msg) | ||
| GAUpdateReportState.report_error_msg = warn_msg | ||
| agent_manifest = goal_state.fetch_agent_manifest(agent_family.name, agent_family.uris) |
There was a problem hiding this comment.
If requested_version is None we would be fetching the manifest every 6 seconds
There was a problem hiding this comment.
yes, that's the case if we want to have largest version as fallback.
There was a problem hiding this comment.
we shouldn't do a download every 6 seconds
There was a problem hiding this comment.
I'll change it in different task
azurelinuxagent/ga/agent_update.py
Outdated
| next_hotfix_time, next_normal_time = self.__get_next_upgrade_times(now) | ||
| upgrade_type = self.__get_agent_upgrade_type(requested_version) | ||
|
|
||
| if next_hotfix_time > now and next_normal_time > now: |
There was a problem hiding this comment.
is this condition needed? i think the condition below already covers this
azurelinuxagent/ga/agent_update.py
Outdated
|
|
||
| msg_ = "Goal state {0} is requesting a new agent version {1}, will update the agent before processing the goal state.".format( | ||
| self._gs_id, str(requested_version)) | ||
| self.__log_event(LogLevel.INFO, msg_) |
There was a problem hiding this comment.
Please avoid the use of periodic logs/events in log_event(), they have caused problems in the past (e.g if messages change). It seems that from this point on they are not even needed since the call to should_update_agent already checks the update periods
There was a problem hiding this comment.
This periodic log intentionally built on message change and timestamp only.
There was a problem hiding this comment.
periodic messages have produced issues in the past. we should avoid them (i've been removing them as i touch code)
if the messages change as the code evolves, it easy to miss that, and it can create very large logs
azurelinuxagent/ga/agent_update.py
Outdated
| path = os.path.join(conf.get_lib_dir(), "{0}-*".format(AGENT_NAME)) | ||
| return [GuestAgent.from_installed_agent(path=agent_dir) for agent_dir in glob.iglob(path) if os.path.isdir(agent_dir)] | ||
|
|
||
| def __log_event(self, level, msg_, success_=True): |
There was a problem hiding this comment.
Why the trailing underscore in msg_ and success_? the convention in Python is to do that when using a name that collides with a built-in name, but that does not seem to be the case here
azurelinuxagent/ga/agent_update.py
Outdated
| try: | ||
| agent = self.__download_and_get_agent(goal_state, agent_family, agent_manifest, requested_version) | ||
|
|
||
| if not agent.is_available: |
There was a problem hiding this comment.
When would this condition be True?
There was a problem hiding this comment.
For example, if download agent is missing agent manifest path
There was a problem hiding this comment.
why would it be missing? if we are concerned about that we should check during publishing, probably in the agent-update test
There was a problem hiding this comment.
My concern is while extracting the zip, it may tamper the folder and mess up the files
There was a problem hiding this comment.
and also, it helps avoid going in loop if version is blacklisted
azurelinuxagent/ga/agent_update.py
Outdated
| path = os.path.join(conf.get_lib_dir(), "{0}-*".format(AGENT_NAME)) | ||
|
|
||
| known_versions = [agent.version for agent in known_agents] | ||
| known_versions.append(CURRENT_VERSION) |
There was a problem hiding this comment.
I think you should pass CURRENT_VERSION explicitly as part of known_agents to make the purpose of this method clearer
azurelinuxagent/ga/agent_update.py
Outdated
| current_agent = next(agent for agent in agents_on_disk if agent.version == CURRENT_VERSION) | ||
| msg = "Marking the agent {0} as bad version since a downgrade was requested in the GoalState, " \ | ||
| "suggesting that we really don't want to execute any extensions using this version".format(CURRENT_VERSION) | ||
| logger.info(msg) |
There was a problem hiding this comment.
can you send this to telemetry, too?
azurelinuxagent/ga/agent_update.py
Outdated
| msg = "Marking the agent {0} as bad version since a downgrade was requested in the GoalState, " \ | ||
| "suggesting that we really don't want to execute any extensions using this version".format(CURRENT_VERSION) | ||
| logger.info(msg) | ||
| current_agent.mark_failure(is_fatal=True, reason=msg) |
There was a problem hiding this comment.
The purge method should not delete blacklisted versions (it can delete the code, but leave the directory and error file on disk), otherwise the daemon may download and use them again
There was a problem hiding this comment.
why would it download again if rsm didn't ask for it. We only download if the requested version ask for it. That's what we want to do it.
There was a problem hiding this comment.
the agent baked in on the image could download it again
maddieford
reviewed
May 24, 2023
| def get_ga_updates_enabled(conf=__conf__): | ||
| """ | ||
| If True, the agent go through update logic to look for new agents otherwise it will stop agent updates. | ||
| NOTE: This option is needed in e2e tests to control agent updates. |
There was a problem hiding this comment.
How is this different than auto update?
azurelinuxagent/ga/agent_update.py
Outdated
| next_hotfix_time, next_normal_time = self.__get_next_upgrade_times(now) | ||
| upgrade_type = self.__get_agent_upgrade_type(requested_version) | ||
|
|
||
| if next_hotfix_time > now and next_normal_time > now: |
azurelinuxagent/ga/agent_update.py
Outdated
| return pkg | ||
|
|
||
| raise Exception("No matching package found in the agent manifest for requested version: {0} in goal state incarnation: {1}, " | ||
| "skipping agent update".format(str(version), self._gs_id)) |
There was a problem hiding this comment.
Is there an extra set of " here?
There was a problem hiding this comment.
sorry, what do you mean?
There was a problem hiding this comment.
raise Exception("No matching package found in the agent manifest for requested version: {0} in goal state incarnation: {1}, " "skipping agent update".format(str(version), self._gs_id))
I see two separate strings:
- "No matching package found in the agent manifest for requested version: {0} in goal state incarnation: {1}, "
- "skipping agent update".format(str(version), self._gs_id)
the second string is taking two values in format() but it's the first string that needs to be formatted
Maybe this is some syntax i'm unfamiliar with
There was a problem hiding this comment.
What is this xml used for?
There was a problem hiding this comment.
This would be used in one of our test case which checks for missing GA family case.
* agent update refactor * address PR comments * updated available agents * fix pylint warn * updated test case warning * added kill switch flag * fix pylint warning * move last update attempt variables
* control agent updates in e2e tests and fix uts (#2743) * disable agent updates in dcr and fix uts * address comments * fix uts * report GA versioning feature
* agent versioning test_suite * address PR comments * fix pylint warning * fix update assertion * fix pylint error
…gent update. (#2778) * improve logging and don't log same error until next period * address comments * update comment * update comment
* Added self-update time window * address comment
* wait for rsm goal state * address comments
…2809) * add own vm property * add agent_update to daily run
narrieta
approved these changes
Jun 9, 2023
maddieford
approved these changes
Jun 14, 2023
| update_handler.run(debug=True) | ||
|
|
||
| self.__assert_exit_code_successful(update_handler) | ||
| self.assertEqual(1, update_handler.get_iterations(), "Update handler should've exited after the first run") | ||
| self.__assert_agent_directories_available(versions=["99999.0.0.0"]) | ||
| self.__assert_agent_directories_available(versions=["9.9.9.10"]) |
There was a problem hiding this comment.
why this change in version?
There was a problem hiding this comment.
In this test I have mocked the ext_conf goal state to ext_conf_requested_version.xml which has requested version as 9.9.9.10. That's why we are checking agent updated to that version.
<GAFamily>
<Name>Prod</Name>
<Version>9.9.9.10</Version>
<Uris>
<Uri>http://mock-goal-state/manifest_of_ga.xml</Uri>
</Uris>
azurelinuxagent/ga/agent_update.py
Outdated
| return pkg | ||
|
|
||
| raise Exception("No matching package found in the agent manifest for requested version: {0} in goal state incarnation: {1}, " | ||
| "skipping agent update".format(str(version), self._gs_id)) |
There was a problem hiding this comment.
raise Exception("No matching package found in the agent manifest for requested version: {0} in goal state incarnation: {1}, " "skipping agent update".format(str(version), self._gs_id))
I see two separate strings:
- "No matching package found in the agent manifest for requested version: {0} in goal state incarnation: {1}, "
- "skipping agent update".format(str(version), self._gs_id)
the second string is taking two values in format() but it's the first string that needs to be formatted
Maybe this is some syntax i'm unfamiliar with
|
@maddieford Somehow, I can't reply to your comment. Anyway, that's not two strings. It's single string break into two lines. I do have a test case which testing that specific error msg. We don't have issues and it's coming as single formatted string. |
nagworld9
added a commit
that referenced
this pull request
Mar 27, 2024
* Add support for Azure Clouds (#2795) * Add support for Azure Clouds --------- Co-authored-by: narrieta <narrieta> * Check certificates only if certificates are included in goal state and update test-requirements to remove codecov (#2803) * Update version to dummy 1.0.0.0' * Revert version change * Only check certificats if goal state includes certs * Fix code coverage deprecated issue * Move condition to function call * Add tests for no outbound connectivity (#2804) * Add tests for no outbound connectivity --------- Co-authored-by: narrieta <narrieta> * Use cloud when validating test location (#2806) * Use cloud when validating test location --------- Co-authored-by: narrieta <narrieta> * Redact access tokens from extension's output (#2811) * Redact access tokens from extension's output * python 2.6 --------- Co-authored-by: narrieta <narrieta> * Add @GabstaMSFT as code owner (#2813) Co-authored-by: narrieta <narrieta> * Fix name of single IB device when provisioning RDMA (#2814) The current code assumes the ipoib interface name is ib0 when single IB interface is provisioned. This is not always true when udev rules are used to rename to other names like ibPxxxxx. Fix this by searching any interface name starting with "ib". * Allow tests to run on random images (#2817) * Allow tests to run on random images * PR feedback --------- Co-authored-by: narrieta <narrieta> * Bug fixes for end-to-end tests (#2820) Co-authored-by: narrieta <narrieta> * Enable all Azure clouds on end-to-end tests (#2821) Co-authored-by: narrieta <narrieta> * Add Azure CLI to container image (#2822) Co-authored-by: narrieta <narrieta> * Fixes for Azure clouds (#2823) * Fixes for Azure clouds * add debug info --------- Co-authored-by: narrieta <narrieta> * Add test for extensions disabled; refactor VirtualMachine and VmExtension (#2824) * Add test for extensions disabled; refactor VirtualMachine and VmExtension --------- Co-authored-by: narrieta <narrieta> * Fixes for end-to-end tests (#2827) Co-authored-by: narrieta <narrieta> * Add test for osProfile.linuxConfiguration.provisionVMAgent (#2826) * Add test for osProfile.linuxConfiguration.provisionVMAgent * add files * pylint * added messages * ssh issue --------- Co-authored-by: narrieta <narrieta> * Enable suppression rules for waagent.log (#2829) Co-authored-by: narrieta <narrieta> * Wait for service start when setting up test VMs; collect VM logs when setup fails (#2830) Co-authored-by: narrieta <narrieta> * Add vm arch to heartbeat telemetry (#2818) (#2838) * Add VM Arch to heartbeat telemetry * Remove outdated vmsize heartbeat tesT * Remove unused import * Use platform to get vmarch (cherry picked from commit 66e8b3d) * Add regular expression to match logs from very old agents (#2839) Co-authored-by: narrieta <narrieta> * Increase concurrency level for end-to-end tests (#2841) Co-authored-by: narrieta <narrieta> * Agent update refactor supports GA versioning (#2810) * agent update refactor (#2706) * agent update refactor * address PR comments * updated available agents * fix pylint warn * updated test case warning * added kill switch flag * fix pylint warning * move last update attempt variables * report GA versioning supported feature. (#2752) * control agent updates in e2e tests and fix uts (#2743) * disable agent updates in dcr and fix uts * address comments * fix uts * report GA versioning feature * Don't report SF flag idf auto update is disabled (#2754) * fix uts (#2759) * agent versioning test_suite (#2770) * agent versioning test_suite * address PR comments * fix pylint warning * fix update assertion * fix pylint error * logging manifest type and don't log same error until next period in agent update. (#2778) * improve logging and don't log same error until next period * address comments * update comment * update comment * Added self-update time window. (#2794) * Added self-update time window * address comment * Wait and retry for rsm goal state (#2801) * wait for rsm goal state * address comments * Not sharing agent update tests vms and added scenario to daily run (#2809) * add own vm property * add agent_update to daily run * merge conflicts * address comments * address comments * additional comments addressed * fix pylint warning * Add test for FIPS (#2842) * Add test for FIPS * add test * increase sleep * remove unused file * added comment * check uptime --------- Co-authored-by: narrieta <narrieta> * Eliminate duplicate list of test suites to run (#2844) * Eliminate duplicate list of test suites to run * fix paths * add agent update --------- Co-authored-by: narrieta <narrieta> * Port NSBSD system to the latest version of waagent (#2828) * nsbsd: adapt to recent dns.resolver * osutil: Provide a get_root_username function for systems where its not 'root' (like in nsbsd) * nsbsd: tune the configuration filepath * nsbsd: fix lib installation path --------- Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Fix method name in update test (#2845) Co-authored-by: narrieta <narrieta> * Expose run name as a runbook variable (#2846) Co-authored-by: narrieta <narrieta> * Collect test artifacts as a separate step in the test pipeline (#2848) * Collect test artifacts as a separate step in the test pipeline --------- Co-authored-by: narrieta <narrieta> * remove agent update test and py27 version from build (#2853) * Fix infinite retry loop in end to end tests (#2855) * Fix infinite retry loop * fix message --------- Co-authored-by: narrieta <narrieta> * Remove empty "distro" module (#2854) Co-authored-by: narrieta <narrieta> * Enable Python 2.7 for unit tests (#2856) * Enable Python 2.7 for unit tests --------- Co-authored-by: narrieta <narrieta> * Skip downgrade if requested version below daemon version (#2850) * skip downgrade for agent update * add test * report it in status * address comments * revert change * improved error msg * address comment * update location schema and added skip clouds in suite yml (#2852) * update location schema in suite yml * address comments * . * pylint warn * comment * Do not collect LISA logs by default (#2857) Co-authored-by: narrieta <narrieta> * Add check for noexec on Permission denied errors (#2859) * Add check for noexec on Permission denied errors * remove type annotation --------- Co-authored-by: narrieta <narrieta> * Wait for log message in AgentNotProvisioned test (#2861) * Wait for log message in AgentNotProvisioned test * hardcoded value --------- Co-authored-by: narrieta <narrieta> * Always collect logs on end-to-end tests (#2863) * Always collect logs * cleanup --------- Co-authored-by: narrieta <narrieta> * agent publish scenario (#2847) * agent publish * remove vm size * address comments * deamom version fallback * daemon versionfix * address comments * fix pylint error * address comment * added error handling * add time window for agent manifest download (#2860) * add time window for agent manifest download * address comments * address comments * ignore 75-persistent-net-generator.rules in e2e tests (#2862) * ignore 75-persistent-net-generator.rules in e2e tests * address comment * remove * Always publish artifacts and test results (#2865) Co-authored-by: narrieta <narrieta> * Add tests for extension workflow (#2843) * Update version to dummy 1.0.0.0' * Revert version change * Basic structure * Test must run in SCUS for test ext * Add GuestAgentDCRTest Extension id * Test stucture * Update test file name * test no location * Test location as southcentralus * Assert ext is installed * Try changing version for dcr test ext * Update expected message in instance view * try changing message to string * Limit images for ext workflow * Update classes after refactor * Update class name * Refactor tests * Rename extension_install to extension_workflow * Assert ext status * Assert operation sequence is expected * Remove logger reference * Pass ssh client * Update ssh * Add permission to run script * Correct permissions * Add execute permissions for helper script * Make scripts executable * Change args to string * Add required parameter * Add shebang for retart_agent * Fix arg format * Use restart utility * Run restart with sudo * Add enable scenario * Attempt to remove start_time * Only assert enable * Add delete scenario * Fix uninstall scenario * Add extension update scenario * Run assert scenario on update scenario * Fix reference to ext * Format args as str instead of arr * Update test args * Add test case for update without install * Fix delete * Keep changes * Save changes * Add special chars test case * Fix dcr_ext issue{ * Add validate no lag scenario * Fix testguid reference * Add additional log statements for debugging * Fix message to check before encoding * Encode setting name * Correctly check data * Make check data executable * Fix command args for special char test * Fix no lag time * Fix ssh client reference * Try message instead of text * Remove unused method * Start clean up * Continue code cleanup * Fix pylint errors * Fix pylint errors * Start refactor * Debug agent lag * Update lag logging * Fix assert_that for lag * Remove typo * Add readme for extension_workflow scenario * Reformat comment * Improve logging * Refactor assert scenario * Remove unused constants * Remove unusued parameter in assert scenario * Add logging * Improve logging * Improve logging * Fix soft assertions issue * Remove todo for delete polling * Remove unnecessary new line * removed unnecessary function * Make special chars log more readable * remove unnecessary log * Add version to add or update log * Remove unnecessary assert instance view * Add empty log line * Add update back to restart args to debug * Add update back to restart args to debug * Remove unused init * Remove test_suites from pipeline yml * Update location in test suite yml * Add comment for location restriction * Remove unused init and fix comments * Improve method header * Rename scripts * Remove print_function * Rename is_data_in_waagent_log * Add comments describing assert operation sequence script * add comments to scripts and type annotate assert operation sequence * Add GuestAgentDcrExtension source code to repo * Fix typing.dict error * Fix typing issue * Remove outdated comment * Add comments to extension_workflow.py * rename scripts to match test suite name * Ignore pylint warnings on test ext * Update pylint rc to ignore tests_e2e/GuestAgentDcrTestExtension * Update pylint rc to ignore tests_e2e/GuestAgentDcrTestExtension * disable all errors/warnings dcr test ext * disable all errors/warnings dcr test ext * Run workflow on debian * Revert to dcr config distros * Move enable increment to beginning of function * Fix gs completed regex * Remove unnessary files from dcr test ext dir * Update agent_ext_workflow.yml to skip China and Gov clouds (#2872) * Update agent_ext_workflow.yml to skip China and Gov clouds * Update tests_e2e/test_suites/agent_ext_workflow.yml * fix daemon version (#2874) * Wait for extension goal state processing before checking for lag in log (#2873) * Update version to dummy 1.0.0.0' * Revert version change * Add sleep time to allow goal state processing to complete before lag check * Add retry logic to gs processing lag check * Clean up retry logic * Add back empty line * Fix timestamp parsing issue * Fix timestamp parsing issue * Fix timestamp parsing issue * Do 3 retries{ * Extract tarball with xvf during setup (#2880) In a pipeline run we saw the following error when extracting the tarball on the test node: Adding v to extract the contents with verbose * enable agent update in daily run (#2878) * Create Network Security Group for test VMs (#2882) * Create Network Security Group for test VMs * error handling --------- Co-authored-by: narrieta <narrieta> * don't allow downgrades for self-update (#2881) * don't allow downgrades for self-update * address comments * update comment * add logger * Supress telemetry failures from check agent log (#2887) Co-authored-by: narrieta <narrieta> * Install assertpy on test VMs (#2886) * Install assertpy on test VMs * set versions --------- Co-authored-by: narrieta <narrieta> * Add sample remote tests (#2888) * Add sample remote tests * add pass * review feedback --------- Co-authored-by: narrieta <narrieta> * Enable Extensions.Enabled in tests (#2892) * enable Extensions.Enabled * address comment * address comment * use script * improve msg * improve msg * Reorganize file structure of unit tests (#2894) * Reorganize file structure of unit tests * remove duplicate * add init * mocks --------- Co-authored-by: narrieta <narrieta> * Report useful message when extension processing is disabled (#2895) * Update version to dummy 1.0.0.0' * Revert version change * Fail GS fast in case of extensions disabled * Update extensions_disabled scenario to look for GS failed instead of timeout when extensions are disabled * Update to separate onHold and extensions enabled * Report ext disabled error in handler status * Try using GoalStateUnknownFailure * Fix indentation error * Try failing ext handler and checking logs * Report ext processing error * Attempt to fail fast * Fix param name * Init error * Try to reuse current code * Try to reuse current code * Clean code * Update scenario tests * Add ext status file to fail fast * Fail fast test * Report error when ext disabled * Update timeout to 20 mins * Re enable ext for debugging * Re enable ext for debugging * Log agent status update * Create ext status file with error code * Create ext status file with error code * We should report handler status even if not installed in case of extensions disabled * Clean up code change * Update tests for extensions disabled * Update test comment * Update test * Remove unused line * Remove ununsed timeout * Test failing case * Remove old case * Remove unused import * Test multiconfig ext * Add multi-config test case * Clean up test * Improve logging * Fix dir for testfile * Remove ignore error rules * Remove ununsed imports * Set handler status to not ready explicitly * Use OS Util to get agent conf path * Retry tar operations after 'Unexpected EOF in archive' during node setup (#2891) * Update version to dummy 1.0.0.0' * Revert version change * Capture output of the copy commands during setup * Add verbose to copy command * Update typing for copy to node methods * Print contents of tar before extracting * Print contents of tar before extracting * Print contents of tar before extracting * Print contents of tar before extracting * Retry copying tarball if contents on test node do not match * Revert copy method def * Revert copy method def * Catch EOF error * Retry tar operations if we see failure * Revert target_path * Remove accidental copy of exception * Remove blank line * tar cvf and copy commands overwrite * Add log and telemetry event for extension disabled (#2897) * Update version to dummy 1.0.0.0' * Revert version change * Add logs and telemetry for processing extensions when extensions disabled * Reformat string * Agent status scenario (#2875) * Update version to dummy 1.0.0.0' * Revert version change * Create files for agent status scenario * Add agent status test logic * fix pylint error * Add comment for retry * Mark failures as exceptions * Improve messages in logs * Improve comments * Update comments * Check that agent status updates without processing additional goal states 3 times * Remove unused agent status exception * Update comment * Clean up comments, logs, and imports * Exception should inherit from baseexception * Import datetime * Import datetime * Import timedelta * instance view time is already formatted * Increse status update time * Increse status update time * Increse status update time * Increase timeout * Update comments and timeoutS * Allow retry if agent status timestamp isn't updated after 30s * Remove unused import * Update time value in comment * address PR comments * Check if properties are None * Make types & errors more readable * Re-use vm_agent variable * Add comment for dot operator * multi config scenario (#2898) * Update version to dummy 1.0.0.0' * Revert version change * multi config scenario bare bones * multi config scenario bare bones * Stash * Add multi config test * Run on arm64 * RCv2 is not supported on arm64 * Test should own VM * Add single config ext to test * Add single config ext to test * Do not fail test if there are unexpected extensions on the vm * Update comment for accuracy * Make resource name parameter optional * Clean up code * agent and ext cgroups scenario (#2866) * agent-cgroups scenario * address comments * address comments * fix-pylint * pylint warn * address comments * improved logging" * improved ext cgroups scenario * new changes * pylint fix * updated * address comments * pylint warn * address comment * merge conflicts * agent firewall scenario (#2879) * agent firewall scenario * address comments * improved logging * pylint warn * address comments * updated * address comments * pylint warning * pylint warning * address comment * merge conflicts * Add retry and improve the log messages in agent update test (#2890) * add retry * improve log messages * merge conflicts * Cleanup common directory (#2902) Co-authored-by: narrieta <narrieta> * improved logging (#2893) * skip test in mooncake and usgov (#2904) * extension telemetry pipeline scenario (#2901) * Update version to dummy 1.0.0.0' * Revert version change * Barebones for etp * Scenario should own VM because of conf change * Add extension telemetry pipeline test * Clean up code * Improve log messages * Fix pylint errors * Improve logging * Improve code comments * VmAccess is not supported on flatcar * Address PR comments * Add support_distros in VmExtensionIdentifier * Fix logic for support_distros in VmExtensionIdentifier * Use run_remote_test for remote script * Ignore logcollector fetch failure if it recovers (#2906) * download_fail unit test should use agent version in common instead of 9.9.9.9 (#2908) (#2912) (cherry picked from commit ed80388) * Download certs on FT GS after check_certificates only when missing from disk (#2907) (#2913) * Download certs on FT GS only when missing from disk * Improve telemetry for inconsistent GS * Fix string format (cherry picked from commit c13f750) * Update pipeline.yml to increase timeout to 90 minutes (#2910) Runs have been timing out after 60 minutes due to multiple scenarios sharing VMs * Fix agent memory usage check (#2903) * fix memory usage check * add test * added comment * fix test * disable ga versioning changes (#2917) * Disable ga versioning changes (#2909) * disbale rsm changes * add flag (cherry picked from commit 5a4fae8) * merge conflicts * fix the ignore rule in agent update test (#2915) (#2918) * ignore the agent installed version * address comments * address comments * fixes (cherry picked from commit 8985a42) * Use Mariner 2 in FIPS test (#2916) * Use Mariner 2 in FIPS test --------- Co-authored-by: narrieta <narrieta> * Change pipeline timeout to 90 minutes (#2925) * fix version checking (#2920) Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * mariner container image (#2926) * mariner container image * added packages repo * addressed comments * addressed comments * Fix for "local variable _COLLECT_NOEXEC_ERRORS referenced before assignment" (#2935) * Fix for "local variable _COLLECT_NOEXEC_ERRORS referenced before assignment" * pylint --------- Co-authored-by: narrieta <narrieta> * fix agent manifest call frequency (#2923) (#2932) * fix agent manifest call frequency * new approach (cherry picked from commit 6554032) * enable rhel/centos cgroups (#2922) * Add support for EC certificates (#2936) * Add support for EC certificates * pylint * pylint * typo --------- Co-authored-by: narrieta <narrieta> * Add Cpu Arch in local logs and telemetry events (#2938) * Add cpu arch to telem and local logs * Change get_vm_arch to static method * update unit tests * Remove e2e pipeline file * Remove arch from heartbeat * Move get_vm_arch to osutil * fix syntax issue * Fix unit test * skip cgorup monitor (#2939) * Clarify support status of installing from source. (#2941) Co-authored-by: narrieta <narrieta> * agent cpu quota scenario (#2937) * agent_cpu_quota scenario * addressed comments * addressed comments * skip test version install (#2950) * skip test install * address comments * pylint * local run stuff * undo * Add support for VM Scale Sets to end-to-end tests (#2954) --------- Co-authored-by: narrieta <narrieta> * Ignore dependencies when the extension does not have any settings (#2957) (#2962) * Ignore dependencies when the extension does not have any settings * Remove message --------- Co-authored-by: narrieta <narrieta> (cherry picked from commit 79bc12c) * Cache daemon version (#2942) (#2963) * cache daemon version * address comments * test update (cherry picked from commit 279d557) * update warning message (#2946) (#2964) (cherry picked from commit 33552ee) * fix self-update frequency to spread over 24 hrs for regular type and 4 hrs for hotfix (#2948) (#2965) * update self-update frequency * address comment * mark with comment * addressed comment (cherry picked from commit f15e6ef) * Reduce the firewall check period in agent firewall tests (#2966) * reduce firewall check period * reduce firewall check period * undo get daemon version change (#2951) (#2967) * undo daemon change * pylint (cherry picked from commit fabe7e5) * disable agent update (#2953) (#2968) (cherry picked from commit 9b15b04) * Change agent_cgroups to own Vm (#2972) * Change cgroups to own Vm * Agent cgroups should own vm * Check SSH connectivity during end-to-end tests (#2970) Co-authored-by: narrieta <narrieta> * Gathering Guest ProxyAgent Log Files (#2975) * Remove debug info from waagent.status.json (#2971) * Remove debug info from waagent.status.json * pylint warnings * pylint --------- Co-authored-by: narrieta <narrieta> * Extension sequencing scenario (#2969) * update tests * cleanup * . * . * . * . * . * . * . * . * . * Add new test cases * Update scenario to support new tests * Scenario should support failing extensions and extensions with no settings * Clean up test * Remove locations from test suite yml * Fix deployment issue * Support creating multiple resource groups for vmss in one run * AzureMonitorLinuxAgent is not supported on flatcar * AzureMonitor is not supported on flatcar * remove agent update * Address PR comments * Fix issue with getting random ssh client * Address PR Comments * Address PR Comments * Address PR comments * Do not keep rg count in runbook * Use try/finally with lock * only check logs after scenario startS * Change to instance member --------- Co-authored-by: narrieta <narrieta> * rename log file for agent publish scenario (#2956) * rename log file * add param * address comment * Fix name collisions on resource groups created by AgentTestSuite (#2981) Co-authored-by: narrieta <narrieta> * Save goal state history explicitly (#2977) * Save goal state explicitly * typo * remove default value in internal method --------- Co-authored-by: narrieta <narrieta> * Handle errors when adding logs to the archive (#2982) Co-authored-by: narrieta <narrieta> * Timing issue while checking cpu quota (#2976) * timing issue * fix pylint" * undo * Use case-insentive match when cleaning up test resource groups (#2986) Co-authored-by: narrieta <narrieta> * Update supported Ubuntu versions (#2980) * Fix pylint warning (#2988) Co-authored-by: narrieta <narrieta> * Add information about HTTP proxies (#2985) * Add information about HTTP proxies * no_proxy --------- Co-authored-by: narrieta <narrieta> * agent persist firewall scenario (#2983) * agent persist firewall scenario * address comments * new comments * GA versioning refactor plus fetch new rsm properties. (#2974) * GA versioning refactor * added comment * added abstract decorator * undo abstract change * update names * addressed comments * pylint * agent family * state name * address comments * conf change * Run remote date command to get test case start time (#2993) * Run remote date command to get test case start time * Remove unused import * ext_sequencing scenario: get enable time from extension status files (#2992) * Get enable time from extension status files * Check for empty array * add status example in comments * ssh connection retry on restarts (#3001) * Add e2e test scenario for hostname monitoring (#3003) * Validate hostname is published * Run on distro without known issues * Add comment about debugging network down * Create e2e scenario for hostname monitoring * Remove unused import * Increase timeout for hostname change * Add password to VM and check for agent status if ssh fails * run scenario on all endorsed distros * Use getdistro() to check distro * Add comment to get_distro * Add publish_hostname to runbook * Make get_distro.py executable * Address first round of PR comments * Do not enable hostname monitoring on distros where it is disabled * Skip test on ubuntu * Update get-waagent-conf-value to remove unused variable * AMA is not supported on cbl-mariner 1.0 (#3002) * Cbl-mariner 1.0 is not supported by AMA * Use get distro to check distro * Add comment to get_distro * log update time for self updater (#3004) * add update time log * log new agent update time * fix tests * Fix publish hostname in china and gov clouds (#3005) * Fix regex to parse china/gov domain names * Improve regex * Improve regex * Self update e2e test (#3000) * self-update test * addressed comments * fix tests * log * added comment * merge conflicts * Lisa should not cleanup failed environment if keep_environment=failed (#3006) * Throw exception for test suite if a test failure occurs * Remove unused import * Clean up * Add comment * fix(ubuntu): Point to correct dhcp lease files (#2979) From Ubuntu 18.04, the default dhcp client was systemd-networkd. However, WALA has been checking for the dhclient lease files. This PR seeks to correct this bug.Interestingly, it was already configuring systemd-networkd but checking for dhclient lease files. Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Use self-hosted pool for automation runs (#3007) Co-authored-by: narrieta <narrieta> * Add distros which use Python 2.6 (for reference only) (#3009) Co-authored-by: narrieta <narrieta> * Move cleanup pipeline to self-hosted pool (#3010) Co-authored-by: narrieta <narrieta> * NM should not be restarted during hostname publish if NM_CONTROLLED=y (#3008) * Only restart NM if NM_controlled=n * Clean up code * Clean up code * improve logging * Make check on NM_CONTROLLED value sctrict * Install missing dependency (jq) on Azure Pipeline Agents (#3013) * Install missing dependency (jq) on Azure Pipeline Agents * use if statement * remove if statement --------- Co-authored-by: narrieta <narrieta> * Do not reset the mode of a extension's log directory (#3014) Co-authored-by: narrieta <narrieta> * Daemon should remove stale published_hostname file and log useful warning (#3016) * Daemon should remove published_hostname file and log useful warning * Clean up fast track file if vm id has changed * Clean up initial_goal_state file if vm id has changed * Clean up rsm_update file if vm id has changed * Do not report TestFailedException in test results (#3019) Co-authored-by: narrieta <narrieta> * skip agent update run on arm64 distros (#3018) * Clean test VMs older than 12 hours (#3021) Co-authored-by: narrieta <narrieta> * honor rsm update with no time when agent receives new GS (#3015) * honor rsm update immediately * pylint * improve msg * address comments * address comments * address comments * added verbose logging * Don't check Agent log from the top after each test suite (#3022) * Don't check Agent log from the top after each test suite * fix initialization of override --------- Co-authored-by: narrieta <narrieta> * update the proxy agenet log folder for logcollector (#3028) * Log instance view before asserting (#3029) * Add config parameter to wait for cloud-init (Extensions.WaitForCloudInit) (#3031) * Add config parameter to wait for cloud-init (Extensions.WaitForCloudInit) --------- Co-authored-by: narrieta <narrieta> * Revert changes to publish_hostname in RedhatOSModernUtil (#3032) * Revert changes to publish_hostname in RedhatOSModernUtil * Fix pylint bad-super-call * Remove agent_wait_for_cloud_init from automated runs (#3034) Co-authored-by: narrieta <narrieta> * Adding AutoUpdate.UpdateToLatestVersion new flag support (#3020) * support new flag * address comments * added more info * updated * address comments * resolving comment * updated * Retry get instance view if only name property is present (#3036) * Retry get instance view if incomplete during assertions * Retry getting instance view if only name property is present * Fix regex in agent extension workflow (#3035) * Recover primary nic if down after publishing hostname in RedhatOSUtil (#3024) * Check nic state and recover if down: * Fix typo * Fix state comparison * Fix pylint errors * Fix string comparison * Report publish hostname failure in calling thread * Add todo to check nic state for all distros where we reset network * Update detection to check connection state and separate recover from publish * Pylint unused argument * refactor recover_nic argument * Network interface e2e test * e2e test for recovering the network interface on redhat distros * Only run scenario on distros which use RedhatOSUtil * Fix call to parent publish_hostname to include recover_nic arg * Update comments in default os util * Remove comment * Fix comment * Do not do detection/recover on RedhatOSMOdernUtil * Resolve PR comments * Make script executable * Revert pypy change * Fix publish hostname paramters * Add recover_network_interface scenario to runbook (#3037) * Implementation of new conf flag AutoUpdate.UpdateToLatestVersion support (#3027) * GA update to latest version flag * address comments * resloving comments * added TODO * ignore warning * resolving comment * address comments * config present check * added a comment * Fix daily pipeline failures for recover_network_interface (#3039) * Fix daily pipeline failures for recover_network_interface * Clear any unused settings properties when enabling cse --------- Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Keep failed VMs by default on pipeline runs (#3040) * enable RSM e2e tests (#3030) * enable RSM tests * merge conflicts * Check for 'Access denied' errors when testing SSH connectivity (#3042) Co-authored-by: narrieta <narrieta> * Add Ubuntu 24 to end-to-end tests (#3041) * Add Ubuntu 24 to end-to-end tests * disable AzureMonitorLinuxAgent --------- Co-authored-by: narrieta <narrieta> * Skip capture of VM information on test runs (#3043) Co-authored-by: narrieta <narrieta> * Create symlink for waagent.com on Flatcar (#3045) Co-authored-by: narrieta <narrieta> * don't allow agent update if attempts reached max limit (#3033) * set max update attempts * download refactor * pylint * disable RSM updates (#3044) * Skip test on alma and rocky until we investigate (#3047) * fix agent update UT (#3051) * version update to 2.10.0.8 (#3050) * modify agent update flag (#3053) --------- Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> Co-authored-by: maddieford <93676569+maddieford@users.noreply.github.com> Co-authored-by: Long Li <longli@microsoft.com> Co-authored-by: sebastienb-stormshield <sebastien.bini@stormshield.eu> Co-authored-by: Zheyu Shen <arsdragonfly@gmail.com> Co-authored-by: Zhidong Peng <zpeng@microsoft.com> Co-authored-by: d1r3ct0r <mwadimemakokha@gmail.com> Co-authored-by: narrieta <narrieta>
nagworld9
added a commit
that referenced
this pull request
Nov 13, 2024
* Add support for Azure Clouds (#2795) * Add support for Azure Clouds --------- Co-authored-by: narrieta <narrieta> * Check certificates only if certificates are included in goal state and update test-requirements to remove codecov (#2803) * Update version to dummy 1.0.0.0' * Revert version change * Only check certificats if goal state includes certs * Fix code coverage deprecated issue * Move condition to function call * Add tests for no outbound connectivity (#2804) * Add tests for no outbound connectivity --------- Co-authored-by: narrieta <narrieta> * Use cloud when validating test location (#2806) * Use cloud when validating test location --------- Co-authored-by: narrieta <narrieta> * Redact access tokens from extension's output (#2811) * Redact access tokens from extension's output * python 2.6 --------- Co-authored-by: narrieta <narrieta> * Add @GabstaMSFT as code owner (#2813) Co-authored-by: narrieta <narrieta> * Fix name of single IB device when provisioning RDMA (#2814) The current code assumes the ipoib interface name is ib0 when single IB interface is provisioned. This is not always true when udev rules are used to rename to other names like ibPxxxxx. Fix this by searching any interface name starting with "ib". * Allow tests to run on random images (#2817) * Allow tests to run on random images * PR feedback --------- Co-authored-by: narrieta <narrieta> * Bug fixes for end-to-end tests (#2820) Co-authored-by: narrieta <narrieta> * Enable all Azure clouds on end-to-end tests (#2821) Co-authored-by: narrieta <narrieta> * Add Azure CLI to container image (#2822) Co-authored-by: narrieta <narrieta> * Fixes for Azure clouds (#2823) * Fixes for Azure clouds * add debug info --------- Co-authored-by: narrieta <narrieta> * Add test for extensions disabled; refactor VirtualMachine and VmExtension (#2824) * Add test for extensions disabled; refactor VirtualMachine and VmExtension --------- Co-authored-by: narrieta <narrieta> * Fixes for end-to-end tests (#2827) Co-authored-by: narrieta <narrieta> * Add test for osProfile.linuxConfiguration.provisionVMAgent (#2826) * Add test for osProfile.linuxConfiguration.provisionVMAgent * add files * pylint * added messages * ssh issue --------- Co-authored-by: narrieta <narrieta> * Enable suppression rules for waagent.log (#2829) Co-authored-by: narrieta <narrieta> * Wait for service start when setting up test VMs; collect VM logs when setup fails (#2830) Co-authored-by: narrieta <narrieta> * Add vm arch to heartbeat telemetry (#2818) (#2838) * Add VM Arch to heartbeat telemetry * Remove outdated vmsize heartbeat tesT * Remove unused import * Use platform to get vmarch (cherry picked from commit 66e8b3d) * Add regular expression to match logs from very old agents (#2839) Co-authored-by: narrieta <narrieta> * Increase concurrency level for end-to-end tests (#2841) Co-authored-by: narrieta <narrieta> * Agent update refactor supports GA versioning (#2810) * agent update refactor (#2706) * agent update refactor * address PR comments * updated available agents * fix pylint warn * updated test case warning * added kill switch flag * fix pylint warning * move last update attempt variables * report GA versioning supported feature. (#2752) * control agent updates in e2e tests and fix uts (#2743) * disable agent updates in dcr and fix uts * address comments * fix uts * report GA versioning feature * Don't report SF flag idf auto update is disabled (#2754) * fix uts (#2759) * agent versioning test_suite (#2770) * agent versioning test_suite * address PR comments * fix pylint warning * fix update assertion * fix pylint error * logging manifest type and don't log same error until next period in agent update. (#2778) * improve logging and don't log same error until next period * address comments * update comment * update comment * Added self-update time window. (#2794) * Added self-update time window * address comment * Wait and retry for rsm goal state (#2801) * wait for rsm goal state * address comments * Not sharing agent update tests vms and added scenario to daily run (#2809) * add own vm property * add agent_update to daily run * merge conflicts * address comments * address comments * additional comments addressed * fix pylint warning * Add test for FIPS (#2842) * Add test for FIPS * add test * increase sleep * remove unused file * added comment * check uptime --------- Co-authored-by: narrieta <narrieta> * Eliminate duplicate list of test suites to run (#2844) * Eliminate duplicate list of test suites to run * fix paths * add agent update --------- Co-authored-by: narrieta <narrieta> * Port NSBSD system to the latest version of waagent (#2828) * nsbsd: adapt to recent dns.resolver * osutil: Provide a get_root_username function for systems where its not 'root' (like in nsbsd) * nsbsd: tune the configuration filepath * nsbsd: fix lib installation path --------- Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Fix method name in update test (#2845) Co-authored-by: narrieta <narrieta> * Expose run name as a runbook variable (#2846) Co-authored-by: narrieta <narrieta> * Collect test artifacts as a separate step in the test pipeline (#2848) * Collect test artifacts as a separate step in the test pipeline --------- Co-authored-by: narrieta <narrieta> * remove agent update test and py27 version from build (#2853) * Fix infinite retry loop in end to end tests (#2855) * Fix infinite retry loop * fix message --------- Co-authored-by: narrieta <narrieta> * Remove empty "distro" module (#2854) Co-authored-by: narrieta <narrieta> * Enable Python 2.7 for unit tests (#2856) * Enable Python 2.7 for unit tests --------- Co-authored-by: narrieta <narrieta> * Skip downgrade if requested version below daemon version (#2850) * skip downgrade for agent update * add test * report it in status * address comments * revert change * improved error msg * address comment * update location schema and added skip clouds in suite yml (#2852) * update location schema in suite yml * address comments * . * pylint warn * comment * Do not collect LISA logs by default (#2857) Co-authored-by: narrieta <narrieta> * Add check for noexec on Permission denied errors (#2859) * Add check for noexec on Permission denied errors * remove type annotation --------- Co-authored-by: narrieta <narrieta> * Wait for log message in AgentNotProvisioned test (#2861) * Wait for log message in AgentNotProvisioned test * hardcoded value --------- Co-authored-by: narrieta <narrieta> * Always collect logs on end-to-end tests (#2863) * Always collect logs * cleanup --------- Co-authored-by: narrieta <narrieta> * agent publish scenario (#2847) * agent publish * remove vm size * address comments * deamom version fallback * daemon versionfix * address comments * fix pylint error * address comment * added error handling * add time window for agent manifest download (#2860) * add time window for agent manifest download * address comments * address comments * ignore 75-persistent-net-generator.rules in e2e tests (#2862) * ignore 75-persistent-net-generator.rules in e2e tests * address comment * remove * Always publish artifacts and test results (#2865) Co-authored-by: narrieta <narrieta> * Add tests for extension workflow (#2843) * Update version to dummy 1.0.0.0' * Revert version change * Basic structure * Test must run in SCUS for test ext * Add GuestAgentDCRTest Extension id * Test stucture * Update test file name * test no location * Test location as southcentralus * Assert ext is installed * Try changing version for dcr test ext * Update expected message in instance view * try changing message to string * Limit images for ext workflow * Update classes after refactor * Update class name * Refactor tests * Rename extension_install to extension_workflow * Assert ext status * Assert operation sequence is expected * Remove logger reference * Pass ssh client * Update ssh * Add permission to run script * Correct permissions * Add execute permissions for helper script * Make scripts executable * Change args to string * Add required parameter * Add shebang for retart_agent * Fix arg format * Use restart utility * Run restart with sudo * Add enable scenario * Attempt to remove start_time * Only assert enable * Add delete scenario * Fix uninstall scenario * Add extension update scenario * Run assert scenario on update scenario * Fix reference to ext * Format args as str instead of arr * Update test args * Add test case for update without install * Fix delete * Keep changes * Save changes * Add special chars test case * Fix dcr_ext issue{ * Add validate no lag scenario * Fix testguid reference * Add additional log statements for debugging * Fix message to check before encoding * Encode setting name * Correctly check data * Make check data executable * Fix command args for special char test * Fix no lag time * Fix ssh client reference * Try message instead of text * Remove unused method * Start clean up * Continue code cleanup * Fix pylint errors * Fix pylint errors * Start refactor * Debug agent lag * Update lag logging * Fix assert_that for lag * Remove typo * Add readme for extension_workflow scenario * Reformat comment * Improve logging * Refactor assert scenario * Remove unused constants * Remove unusued parameter in assert scenario * Add logging * Improve logging * Improve logging * Fix soft assertions issue * Remove todo for delete polling * Remove unnecessary new line * removed unnecessary function * Make special chars log more readable * remove unnecessary log * Add version to add or update log * Remove unnecessary assert instance view * Add empty log line * Add update back to restart args to debug * Add update back to restart args to debug * Remove unused init * Remove test_suites from pipeline yml * Update location in test suite yml * Add comment for location restriction * Remove unused init and fix comments * Improve method header * Rename scripts * Remove print_function * Rename is_data_in_waagent_log * Add comments describing assert operation sequence script * add comments to scripts and type annotate assert operation sequence * Add GuestAgentDcrExtension source code to repo * Fix typing.dict error * Fix typing issue * Remove outdated comment * Add comments to extension_workflow.py * rename scripts to match test suite name * Ignore pylint warnings on test ext * Update pylint rc to ignore tests_e2e/GuestAgentDcrTestExtension * Update pylint rc to ignore tests_e2e/GuestAgentDcrTestExtension * disable all errors/warnings dcr test ext * disable all errors/warnings dcr test ext * Run workflow on debian * Revert to dcr config distros * Move enable increment to beginning of function * Fix gs completed regex * Remove unnessary files from dcr test ext dir * Update agent_ext_workflow.yml to skip China and Gov clouds (#2872) * Update agent_ext_workflow.yml to skip China and Gov clouds * Update tests_e2e/test_suites/agent_ext_workflow.yml * fix daemon version (#2874) * Wait for extension goal state processing before checking for lag in log (#2873) * Update version to dummy 1.0.0.0' * Revert version change * Add sleep time to allow goal state processing to complete before lag check * Add retry logic to gs processing lag check * Clean up retry logic * Add back empty line * Fix timestamp parsing issue * Fix timestamp parsing issue * Fix timestamp parsing issue * Do 3 retries{ * Extract tarball with xvf during setup (#2880) In a pipeline run we saw the following error when extracting the tarball on the test node: Adding v to extract the contents with verbose * enable agent update in daily run (#2878) * Create Network Security Group for test VMs (#2882) * Create Network Security Group for test VMs * error handling --------- Co-authored-by: narrieta <narrieta> * don't allow downgrades for self-update (#2881) * don't allow downgrades for self-update * address comments * update comment * add logger * Supress telemetry failures from check agent log (#2887) Co-authored-by: narrieta <narrieta> * Install assertpy on test VMs (#2886) * Install assertpy on test VMs * set versions --------- Co-authored-by: narrieta <narrieta> * Add sample remote tests (#2888) * Add sample remote tests * add pass * review feedback --------- Co-authored-by: narrieta <narrieta> * Enable Extensions.Enabled in tests (#2892) * enable Extensions.Enabled * address comment * address comment * use script * improve msg * improve msg * Reorganize file structure of unit tests (#2894) * Reorganize file structure of unit tests * remove duplicate * add init * mocks --------- Co-authored-by: narrieta <narrieta> * Report useful message when extension processing is disabled (#2895) * Update version to dummy 1.0.0.0' * Revert version change * Fail GS fast in case of extensions disabled * Update extensions_disabled scenario to look for GS failed instead of timeout when extensions are disabled * Update to separate onHold and extensions enabled * Report ext disabled error in handler status * Try using GoalStateUnknownFailure * Fix indentation error * Try failing ext handler and checking logs * Report ext processing error * Attempt to fail fast * Fix param name * Init error * Try to reuse current code * Try to reuse current code * Clean code * Update scenario tests * Add ext status file to fail fast * Fail fast test * Report error when ext disabled * Update timeout to 20 mins * Re enable ext for debugging * Re enable ext for debugging * Log agent status update * Create ext status file with error code * Create ext status file with error code * We should report handler status even if not installed in case of extensions disabled * Clean up code change * Update tests for extensions disabled * Update test comment * Update test * Remove unused line * Remove ununsed timeout * Test failing case * Remove old case * Remove unused import * Test multiconfig ext * Add multi-config test case * Clean up test * Improve logging * Fix dir for testfile * Remove ignore error rules * Remove ununsed imports * Set handler status to not ready explicitly * Use OS Util to get agent conf path * Retry tar operations after 'Unexpected EOF in archive' during node setup (#2891) * Update version to dummy 1.0.0.0' * Revert version change * Capture output of the copy commands during setup * Add verbose to copy command * Update typing for copy to node methods * Print contents of tar before extracting * Print contents of tar before extracting * Print contents of tar before extracting * Print contents of tar before extracting * Retry copying tarball if contents on test node do not match * Revert copy method def * Revert copy method def * Catch EOF error * Retry tar operations if we see failure * Revert target_path * Remove accidental copy of exception * Remove blank line * tar cvf and copy commands overwrite * Add log and telemetry event for extension disabled (#2897) * Update version to dummy 1.0.0.0' * Revert version change * Add logs and telemetry for processing extensions when extensions disabled * Reformat string * Agent status scenario (#2875) * Update version to dummy 1.0.0.0' * Revert version change * Create files for agent status scenario * Add agent status test logic * fix pylint error * Add comment for retry * Mark failures as exceptions * Improve messages in logs * Improve comments * Update comments * Check that agent status updates without processing additional goal states 3 times * Remove unused agent status exception * Update comment * Clean up comments, logs, and imports * Exception should inherit from baseexception * Import datetime * Import datetime * Import timedelta * instance view time is already formatted * Increse status update time * Increse status update time * Increse status update time * Increase timeout * Update comments and timeoutS * Allow retry if agent status timestamp isn't updated after 30s * Remove unused import * Update time value in comment * address PR comments * Check if properties are None * Make types & errors more readable * Re-use vm_agent variable * Add comment for dot operator * multi config scenario (#2898) * Update version to dummy 1.0.0.0' * Revert version change * multi config scenario bare bones * multi config scenario bare bones * Stash * Add multi config test * Run on arm64 * RCv2 is not supported on arm64 * Test should own VM * Add single config ext to test * Add single config ext to test * Do not fail test if there are unexpected extensions on the vm * Update comment for accuracy * Make resource name parameter optional * Clean up code * agent and ext cgroups scenario (#2866) * agent-cgroups scenario * address comments * address comments * fix-pylint * pylint warn * address comments * improved logging" * improved ext cgroups scenario * new changes * pylint fix * updated * address comments * pylint warn * address comment * merge conflicts * agent firewall scenario (#2879) * agent firewall scenario * address comments * improved logging * pylint warn * address comments * updated * address comments * pylint warning * pylint warning * address comment * merge conflicts * Add retry and improve the log messages in agent update test (#2890) * add retry * improve log messages * merge conflicts * Cleanup common directory (#2902) Co-authored-by: narrieta <narrieta> * improved logging (#2893) * skip test in mooncake and usgov (#2904) * extension telemetry pipeline scenario (#2901) * Update version to dummy 1.0.0.0' * Revert version change * Barebones for etp * Scenario should own VM because of conf change * Add extension telemetry pipeline test * Clean up code * Improve log messages * Fix pylint errors * Improve logging * Improve code comments * VmAccess is not supported on flatcar * Address PR comments * Add support_distros in VmExtensionIdentifier * Fix logic for support_distros in VmExtensionIdentifier * Use run_remote_test for remote script * Ignore logcollector fetch failure if it recovers (#2906) * download_fail unit test should use agent version in common instead of 9.9.9.9 (#2908) (#2912) (cherry picked from commit ed80388) * Download certs on FT GS after check_certificates only when missing from disk (#2907) (#2913) * Download certs on FT GS only when missing from disk * Improve telemetry for inconsistent GS * Fix string format (cherry picked from commit c13f750) * Update pipeline.yml to increase timeout to 90 minutes (#2910) Runs have been timing out after 60 minutes due to multiple scenarios sharing VMs * Fix agent memory usage check (#2903) * fix memory usage check * add test * added comment * fix test * disable ga versioning changes (#2917) * Disable ga versioning changes (#2909) * disbale rsm changes * add flag (cherry picked from commit 5a4fae8) * merge conflicts * fix the ignore rule in agent update test (#2915) (#2918) * ignore the agent installed version * address comments * address comments * fixes (cherry picked from commit 8985a42) * Use Mariner 2 in FIPS test (#2916) * Use Mariner 2 in FIPS test --------- Co-authored-by: narrieta <narrieta> * Change pipeline timeout to 90 minutes (#2925) * fix version checking (#2920) Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * mariner container image (#2926) * mariner container image * added packages repo * addressed comments * addressed comments * Fix for "local variable _COLLECT_NOEXEC_ERRORS referenced before assignment" (#2935) * Fix for "local variable _COLLECT_NOEXEC_ERRORS referenced before assignment" * pylint --------- Co-authored-by: narrieta <narrieta> * fix agent manifest call frequency (#2923) (#2932) * fix agent manifest call frequency * new approach (cherry picked from commit 6554032) * enable rhel/centos cgroups (#2922) * Add support for EC certificates (#2936) * Add support for EC certificates * pylint * pylint * typo --------- Co-authored-by: narrieta <narrieta> * Add Cpu Arch in local logs and telemetry events (#2938) * Add cpu arch to telem and local logs * Change get_vm_arch to static method * update unit tests * Remove e2e pipeline file * Remove arch from heartbeat * Move get_vm_arch to osutil * fix syntax issue * Fix unit test * skip cgorup monitor (#2939) * Clarify support status of installing from source. (#2941) Co-authored-by: narrieta <narrieta> * agent cpu quota scenario (#2937) * agent_cpu_quota scenario * addressed comments * addressed comments * skip test version install (#2950) * skip test install * address comments * pylint * local run stuff * undo * Add support for VM Scale Sets to end-to-end tests (#2954) --------- Co-authored-by: narrieta <narrieta> * Ignore dependencies when the extension does not have any settings (#2957) (#2962) * Ignore dependencies when the extension does not have any settings * Remove message --------- Co-authored-by: narrieta <narrieta> (cherry picked from commit 79bc12c) * Cache daemon version (#2942) (#2963) * cache daemon version * address comments * test update (cherry picked from commit 279d557) * update warning message (#2946) (#2964) (cherry picked from commit 33552ee) * fix self-update frequency to spread over 24 hrs for regular type and 4 hrs for hotfix (#2948) (#2965) * update self-update frequency * address comment * mark with comment * addressed comment (cherry picked from commit f15e6ef) * Reduce the firewall check period in agent firewall tests (#2966) * reduce firewall check period * reduce firewall check period * undo get daemon version change (#2951) (#2967) * undo daemon change * pylint (cherry picked from commit fabe7e5) * disable agent update (#2953) (#2968) (cherry picked from commit 9b15b04) * Change agent_cgroups to own Vm (#2972) * Change cgroups to own Vm * Agent cgroups should own vm * Check SSH connectivity during end-to-end tests (#2970) Co-authored-by: narrieta <narrieta> * Gathering Guest ProxyAgent Log Files (#2975) * Remove debug info from waagent.status.json (#2971) * Remove debug info from waagent.status.json * pylint warnings * pylint --------- Co-authored-by: narrieta <narrieta> * Extension sequencing scenario (#2969) * update tests * cleanup * . * . * . * . * . * . * . * . * . * Add new test cases * Update scenario to support new tests * Scenario should support failing extensions and extensions with no settings * Clean up test * Remove locations from test suite yml * Fix deployment issue * Support creating multiple resource groups for vmss in one run * AzureMonitorLinuxAgent is not supported on flatcar * AzureMonitor is not supported on flatcar * remove agent update * Address PR comments * Fix issue with getting random ssh client * Address PR Comments * Address PR Comments * Address PR comments * Do not keep rg count in runbook * Use try/finally with lock * only check logs after scenario startS * Change to instance member --------- Co-authored-by: narrieta <narrieta> * rename log file for agent publish scenario (#2956) * rename log file * add param * address comment * Fix name collisions on resource groups created by AgentTestSuite (#2981) Co-authored-by: narrieta <narrieta> * Save goal state history explicitly (#2977) * Save goal state explicitly * typo * remove default value in internal method --------- Co-authored-by: narrieta <narrieta> * Handle errors when adding logs to the archive (#2982) Co-authored-by: narrieta <narrieta> * Timing issue while checking cpu quota (#2976) * timing issue * fix pylint" * undo * Use case-insentive match when cleaning up test resource groups (#2986) Co-authored-by: narrieta <narrieta> * Update supported Ubuntu versions (#2980) * Fix pylint warning (#2988) Co-authored-by: narrieta <narrieta> * Add information about HTTP proxies (#2985) * Add information about HTTP proxies * no_proxy --------- Co-authored-by: narrieta <narrieta> * agent persist firewall scenario (#2983) * agent persist firewall scenario * address comments * new comments * GA versioning refactor plus fetch new rsm properties. (#2974) * GA versioning refactor * added comment * added abstract decorator * undo abstract change * update names * addressed comments * pylint * agent family * state name * address comments * conf change * Run remote date command to get test case start time (#2993) * Run remote date command to get test case start time * Remove unused import * ext_sequencing scenario: get enable time from extension status files (#2992) * Get enable time from extension status files * Check for empty array * add status example in comments * ssh connection retry on restarts (#3001) * Add e2e test scenario for hostname monitoring (#3003) * Validate hostname is published * Run on distro without known issues * Add comment about debugging network down * Create e2e scenario for hostname monitoring * Remove unused import * Increase timeout for hostname change * Add password to VM and check for agent status if ssh fails * run scenario on all endorsed distros * Use getdistro() to check distro * Add comment to get_distro * Add publish_hostname to runbook * Make get_distro.py executable * Address first round of PR comments * Do not enable hostname monitoring on distros where it is disabled * Skip test on ubuntu * Update get-waagent-conf-value to remove unused variable * AMA is not supported on cbl-mariner 1.0 (#3002) * Cbl-mariner 1.0 is not supported by AMA * Use get distro to check distro * Add comment to get_distro * log update time for self updater (#3004) * add update time log * log new agent update time * fix tests * Fix publish hostname in china and gov clouds (#3005) * Fix regex to parse china/gov domain names * Improve regex * Improve regex * Self update e2e test (#3000) * self-update test * addressed comments * fix tests * log * added comment * merge conflicts * Lisa should not cleanup failed environment if keep_environment=failed (#3006) * Throw exception for test suite if a test failure occurs * Remove unused import * Clean up * Add comment * fix(ubuntu): Point to correct dhcp lease files (#2979) From Ubuntu 18.04, the default dhcp client was systemd-networkd. However, WALA has been checking for the dhclient lease files. This PR seeks to correct this bug.Interestingly, it was already configuring systemd-networkd but checking for dhclient lease files. Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Use self-hosted pool for automation runs (#3007) Co-authored-by: narrieta <narrieta> * Add distros which use Python 2.6 (for reference only) (#3009) Co-authored-by: narrieta <narrieta> * Move cleanup pipeline to self-hosted pool (#3010) Co-authored-by: narrieta <narrieta> * NM should not be restarted during hostname publish if NM_CONTROLLED=y (#3008) * Only restart NM if NM_controlled=n * Clean up code * Clean up code * improve logging * Make check on NM_CONTROLLED value sctrict * Install missing dependency (jq) on Azure Pipeline Agents (#3013) * Install missing dependency (jq) on Azure Pipeline Agents * use if statement * remove if statement --------- Co-authored-by: narrieta <narrieta> * Do not reset the mode of a extension's log directory (#3014) Co-authored-by: narrieta <narrieta> * Daemon should remove stale published_hostname file and log useful warning (#3016) * Daemon should remove published_hostname file and log useful warning * Clean up fast track file if vm id has changed * Clean up initial_goal_state file if vm id has changed * Clean up rsm_update file if vm id has changed * Do not report TestFailedException in test results (#3019) Co-authored-by: narrieta <narrieta> * skip agent update run on arm64 distros (#3018) * Clean test VMs older than 12 hours (#3021) Co-authored-by: narrieta <narrieta> * honor rsm update with no time when agent receives new GS (#3015) * honor rsm update immediately * pylint * improve msg * address comments * address comments * address comments * added verbose logging * Don't check Agent log from the top after each test suite (#3022) * Don't check Agent log from the top after each test suite * fix initialization of override --------- Co-authored-by: narrieta <narrieta> * update the proxy agenet log folder for logcollector (#3028) * Log instance view before asserting (#3029) * Add config parameter to wait for cloud-init (Extensions.WaitForCloudInit) (#3031) * Add config parameter to wait for cloud-init (Extensions.WaitForCloudInit) --------- Co-authored-by: narrieta <narrieta> * Revert changes to publish_hostname in RedhatOSModernUtil (#3032) * Revert changes to publish_hostname in RedhatOSModernUtil * Fix pylint bad-super-call * Remove agent_wait_for_cloud_init from automated runs (#3034) Co-authored-by: narrieta <narrieta> * Adding AutoUpdate.UpdateToLatestVersion new flag support (#3020) * support new flag * address comments * added more info * updated * address comments * resolving comment * updated * Retry get instance view if only name property is present (#3036) * Retry get instance view if incomplete during assertions * Retry getting instance view if only name property is present * Fix regex in agent extension workflow (#3035) * Recover primary nic if down after publishing hostname in RedhatOSUtil (#3024) * Check nic state and recover if down: * Fix typo * Fix state comparison * Fix pylint errors * Fix string comparison * Report publish hostname failure in calling thread * Add todo to check nic state for all distros where we reset network * Update detection to check connection state and separate recover from publish * Pylint unused argument * refactor recover_nic argument * Network interface e2e test * e2e test for recovering the network interface on redhat distros * Only run scenario on distros which use RedhatOSUtil * Fix call to parent publish_hostname to include recover_nic arg * Update comments in default os util * Remove comment * Fix comment * Do not do detection/recover on RedhatOSMOdernUtil * Resolve PR comments * Make script executable * Revert pypy change * Fix publish hostname paramters * Add recover_network_interface scenario to runbook (#3037) * Implementation of new conf flag AutoUpdate.UpdateToLatestVersion support (#3027) * GA update to latest version flag * address comments * resloving comments * added TODO * ignore warning * resolving comment * address comments * config present check * added a comment * Fix daily pipeline failures for recover_network_interface (#3039) * Fix daily pipeline failures for recover_network_interface * Clear any unused settings properties when enabling cse --------- Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Keep failed VMs by default on pipeline runs (#3040) * enable RSM e2e tests (#3030) * enable RSM tests * merge conflicts * Check for 'Access denied' errors when testing SSH connectivity (#3042) Co-authored-by: narrieta <narrieta> * Add Ubuntu 24 to end-to-end tests (#3041) * Add Ubuntu 24 to end-to-end tests * disable AzureMonitorLinuxAgent --------- Co-authored-by: narrieta <narrieta> * Skip capture of VM information on test runs (#3043) Co-authored-by: narrieta <narrieta> * Create symlink for waagent.com on Flatcar (#3045) Co-authored-by: narrieta <narrieta> * don't allow agent update if attempts reached max limit (#3033) * set max update attempts * download refactor * pylint * disable RSM updates (#3044) * Skip test on alma and rocky until we investigate (#3047) * Ext_sequencing scenario should check agent log for extension enable order (#3049) * Ext_sequencing scenario should check agent log for extension enable order * Format timestamp for ignore errors before timestamp * If test is skipped, scenario start will be datetime min * Remove unnecessary log * Make none check explicit * update canary region (#3056) * Add Python 3.10 to the pylint matrix (#3057) Co-authored-by: narrieta <narrieta> * reset network service unit file if python version changes (#3058) * Ignore network unreachable errors in publish hostname (#3060) * Address pylint warning deprecated-method (#3059) Co-authored-by: narrieta <narrieta> * fix agent update UT (#3051) (#3054) (cherry picked from commit d9f7ed4) * modify agent update flag (#3053) (#3055) (cherry picked from commit 049de5c) * skip run on flatcar (#3061) * retry on agent cgroups tracking check (#3062) * retry on agentcgroups check * address comments * Recognize SLE-Micro as a SLE based distribution (#3048) Using the agent with SLE-Micro forces the agent to fallback to the common default implementation for nominally distribution specific behavior. This misses the SUSE specific implementations. Co-authored-by: Nageswara Nandigam <84482346+nagworld9@users.noreply.github.com> * Retry ssh check if connection reset (#3065) * Add distutils/version.py to azurelinuxagent (#3063) * Add distutils/version.py to azurelinuxagent --------- Co-authored-by: narrieta <narrieta> * Run pylint on Python 3.11 (#3067) * Run pylint on Python 3.11 --------- Co-authored-by: narrieta <narrieta> * Fix pylint warnings (#3069) * Fix pylint warnings * Update .github/workflows/ci_pr.yml Co-authored-by: maddieford <93676569+maddieford@users.noreply.github.com> --------- Co-authored-by: narrieta <narrieta> Co-authored-by: maddieford <93676569+maddieford@users.noreply.github.com> * reset uphold setting for agent service in flatcar distro (#3066) * reset uphold settings for flatcar images * updated comment * stop the rebbot service * address comments * retry on quota reset check (#3068) * Use legacycrypt instead of crypt on Python >= 3.13 (#3070) * Use legacycrypt instead of crypt on Python >= 3.13 * remove ModuleNotFound --------- Co-authored-by: narrieta <narrieta> * Skip network unreachable error in publish hostname test (#3071) Co-authored-by: narrieta <narrieta> * Fix osutil/default route_add to pass string array. (#3072) Co-authored-by: narrieta <narrieta> * Fix argument to GoalState.__init__ (#3073) Co-authored-by: narrieta <narrieta> * Ignore network unreachable error in hostname test (#3074) * Ignore network unreachable error in hostname test --------- Co-authored-by: narrieta <narrieta> * Add lock around access to fast_track.json (#3076) Co-authored-by: narrieta <narrieta> * added retries for agent cgroups test (#3075) * retries for agent cgroups test * pylint warn * addressed comment * cron job script (#3077) * Fix mock for cgroup unit test (#3079) * Fix mock for cgroup unit test --------- Co-authored-by: narrieta <narrieta> * Add DistroVersion class to compare distro versions (#3078) * Add DistroVersion class to compare distro versions * comment * python 2 --------- Co-authored-by: narrieta <narrieta> * enable GA versioning (#3082) * Run unit tests with pytest on Python >= 3.10 (#3081) * Run unit tests with pytest on Python >= 3.10 --------- Co-authored-by: narrieta <narrieta> * Fix pytest warnings (#3084) Co-authored-by: narrieta <narrieta> * update setup (#3088) * Add keyvault test to daily run + Specify tests suite as a list (#3089) Co-authored-by: narrieta <narrieta> * ignore case (#3093) * Add retry on keyvault test (#3095) * Add retry on keyvault test * newline --------- Co-authored-by: narrieta <narrieta> * Reboot Vm if CSE timesout so logs are collected (#3097) * LogCollector should skip and log warning for files that don't exist (#3098) * Skip collection on files that do not exist * Fix pylint * Separate error handling * log file to collect * wait for provision to complete before install test agent (#3094) * wait for provision to complete * address comments * agent publish refactor (#3091) * agent publish refactor * support arm 64vm * convert dict to str * address comments * pylint * new comments * updated comment * Add EnableFirewall to README (#3100) * Add EnableFirewall to README * change phrasing --------- Co-authored-by: narrieta <narrieta> * Add Ubuntu minimal to test run (#3102) * Add ubuntu minimal to test run * typo * suppress warnings --------- Co-authored-by: narrieta <narrieta> * check for unexpected process in agent cgroups before cgroups enabled (#3103) * check for unexpected process in cgroup before enable * agent restart * move the process check * fix unit tests * address comments * pylint * Cgroups api refactor for v2 (#3096) * Cgroups api refactor (#6) * Initial changes for log collector cgroups v2 support * Fix pylint issues * Fix pylint issues * Fix pylint issues * Check that both controllers are mounted in the chosen cgroups version for log collector * Fix regex * Update test_agent unit tests * Fix unit tests * Update format strings * Fix broken cgroupconfigurator unit tests * pyling * Fix cgroups api unit tests * Ignore unused args * Ignore unused args * Add cgroup configurator tests * v2 required check in parent cgroup * unit tests is_controller_enabled * Fix test failure and pylint: * pylint * Update agent checks * Fix controller enable logic and unit tests * Remove changes to collect logs * Fix pylint * Add e2e test for v2 * Fix log warnings * Add cgroups v2 disabled scenario to daily runbook * Address PR comments (#7) * get_cgroup_api should raise exception when controllers not mounted * Combine cgroups_supported() check * Combine SystemdCgroupsApi and CGroupApi classes * fix pylint and tests with sudo * Rename SystemdCgroupsApi to SystemdCgroupApi * Cgroup should be singular when referring to the APi * Unimpleneted methods should raise NotImplementederror * Check for cpu,cpuacct * v2 start extension command should not be implemented * log_cgorup_info and log_cgroup_warning should be in cgroupapi * Systemd check should come before api * Explicitly check for empty dict * Only check if controllers are enabled at root for v2 * Remove unnecessary mocked paths in mock cgroup env * V2 does not have concept of mounting controllers * Fix super call for python 2 * get_cgroup_api should be function * Move logging functions up * Use stat -f to get cgroup mode * Mock hybrid path * Fix unit tests: * Debug tests * Debug tests * Debug unit tests * Fix unit tests * Fix pylint * Fix e2e test for v2 * Fix e2e test * Fix e2e test * Fix e2e test * Combine common implementations * Improve comments * Pylint * Address PR comments (#8) * Run unit tests * Clean up drop in files if cgroups are disabled * Init values for cgroup apis * Rever test change * get_cgroup_api should check if mountpoints are correct (#9) * Fix conflict after merge * Merge issues (#10) * Fix merge issues * Fix unit tests * get_cgroup_api raises InvalidCroupMountpointException (#11) * get_cgroup_api can raise InvalidCgroupMountpointException * Add unit test for agent * Address PR comments (#12) * Increase timeout for agent to start and provisioning to complete (#3105) * Keep whole goal state in log (#3104) * Log cgroup if process found in unexpected slice (#3107) * Allow retries for ifdown and add comments (#3106) * Collect telemetry for firewall settings changed (#3110) (#3112) Co-authored-by: narrieta <narrieta> (cherry picked from commit 468cf81) * Update agent_publish test to check for new agent update pattern (#3114) (#3119) * Add new agent update pattern * Use record message * Need to update log record timestamp (cherry picked from commit 1d91c14) * remove secret and use cert for aad app in e2e pipeline (#3116) * remove secret and use cert * address comment * wait for rg creation in e2e tests (#3117) * wait for rg creation * update param * check for rg existance * input rg name * Reduce the frequency of firewall telemetry (#3124) (#3127) * Reduce the frequency of firewall telemetry * python 2: timespan.total_seconds() does not exist * fix unit test --------- Co-authored-by: narrieta <narrieta> (cherry picked from commit 5302651) * suppress pylint warn contextmanager-generator-missing-cleanup (#3138) * suppress pylint warn * addressed comments * Switching to SNI based authentication for aad app (#3137) * SNI auth * new env var * pylint * new namespace (#3139) * support dict/list resources type for lisa template (#3140) * support dict/list for resources schema * addressed comment * eFix multi config (#16) (#3145) * Use runcommand api for runcommand multiconfig operations * remove rc * Fix comments * Remove comment * Fix rc * pylint * Add line * refactor cgroup controllers (#3135) * refactor cgroup controllers (#13) * Refactor Cgroup, CpuCgroup, MemoryCgroup to ControllerMetrics, CpuMetrics, MemoryMetrics * Create methods to get unit/process cgroup representation * Refactoring changes * Refactoring changes * Fix e2e test * Fix unintentional comment change * Remove unneeded comments * Clean up comments and make code more readable * Simplify get controller metrics * Clean up cgroupapi * Cleanup cgroup -> controllermetrics changes * Clean up cgroup configurator * Fix unit tests for agent.py * Fix cgroupapi tests * Fix cgroupconfigurator and tests * Rename controller metrics tests * Ignore pylint issues * Improve test coverage for cgroupapi * Rename cgroup to metrics * Update cgroup.procs to accurately represent file * Do not track metrics if controller is not mounted * We should set cpu quota before tracking cpu metrics * Pylint * address pr comments (#14) * Address Nag's comments * pyling * pylint * remove lambda (#15) * updated PR template (#3144) * fixing custom image test run (#3147) * Avoiding mocked exception from being lost on test (#3149) If another exception arises (that's the case here when python 3.12 is used due to the changes in os.shutil.rmtree), the mocked exception is lost because it is incomplete (neither errno nor strerror are set: it goes to args). * Add more useful logging for agent unit properties (#3154) * Remove wireserver fallback for imds calls (#3152) * Remove wireserver fallback for imds calls * remove unused method * remove obsolete unit test * remove unused import --------- Co-authored-by: narrieta@microsoft <narrieta> * Remove unused import (#3155) Co-authored-by: narrieta@microsoft <narrieta> * Expand support for backend ethernet (#3150) IBManager will continue to be used for a new ethernet-backend offering from AzureHPC. While the key name remains the same (IPoIB_data), the interfaces will be of the format ethXX. Removing the check that skips anything that isn't ibXX. We are not at the risk of proceeding for any other nics since the IPoIB_data will only have the backend RDMA ones, and despite reading from the system for the loop, we match it against the array parsed from the IPoIB_data KVP. IB interfaces have padded virtual macs, non-IB interfaces won't. Adding if-else to only do the padded-octet check for IB. Everything else will use the standard 6-octet pattern. * Allow use of node 16 (#3160) Co-authored-by: narrieta@microsoft <narrieta> * Fix Ubuntu version codename for 24.04 (#3159) 24.04 is noble, not focal Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> * Fix regex pattern for ext seq scenario (#3162) * Update test certificate data (#3166) Co-authored-by: narrieta@microsoft <narrieta> * Remove extension status only on extension delete (#3167) * Remove extension status only on extension delete * . * . --------- Co-authored-by: narrieta@microsoft <narrieta> * Add support for Azure Linux 3 (#3183) * . * Add Azure Linux 3 to test runs * . * . * . * . * Update setup.py --------- Co-authored-by: narrieta@microsoft <narrieta> * Use self-update for initial update (#3184) * use self-update for initial update * addressing comments * cleanup files * state files * remove comment * send updatemode in heartbeat and don't send RSM supported feature flag if versioning disabled in agent (#3189) * rsm changes (cherry picked from commit d73cef5) * addressed comment (cherry picked from commit 1ab9122) * updated comment * addressed comments * added semicolon * Disable multi-config test in AzureCloud (#3192) Co-authored-by: narrieta@microsoft <narrieta> * Add cgroupv2 support for log collector (#3188) * Lc v2 implementation branch (#18) * memory experimentation changes * Initial changes * obvious issues * Fix e2e test * First round of unit test fixes * Fix existing unit tests * Remove unneeded cpu files * Get memory usage should return tuple * Fix log for tracking cgroup * Add unit tests * Add unit tests * Address pylint comments * Clean up code * clean up code * Fix unit tests (#19) * Fix unit tests * Fix unit tests * Revisions (#20) * Respond to comments * Test failures * Fix type issue * Revisions * Additional revisions (#21) * Revisions * Remove unit test for sending telem * final fixes * add config flag * Fix e2e tests * workaround for python3.5 UTs build setup and replace assert_called_once mock method (#3191) * python3.5 workaround * replace assert_called_once * addressing comment * Fix log collector unit tests on 3.5 (#3193) * Fix unit tests 3.5 * Fix ut * Fix JIT for FIPS 140-3 (#3190) * . * . * ., --------- Co-authored-by: narrieta@microsoft <narrieta> * Capture logcollector pattern only once (#3194) * Capture logcollector pattern only once * Add comment * Check agent Slice unit property before setting up azure.slice (#3196) (#3198) (cherry picked from commit bdd4a4b) * version update to 2.12.0.0 (#3195) * fixing attribute error (#3202) * version update to 2.12.0.1" (#3203) * supress too-many-positional-args pylint warn (#3224) (#3225) (cherry picked from commit 4dcf95c) * move setupslice after cgroupsv2 check, remove unit file for log collector and remove fiirewall daemon-reload (#3223) (#3226) * move daemon reload * test fix * UT test * firewall daemon-reload * address comments * address comments (cherry picked from commit 47e969a) * Ubuntu 24 image (#25) (#3229) (#3230) * Update ubuntu 24 * Add ubuntu 24 to nat clouds * Add arm64 ubuntu 24 * Update all ubuntu images * Skip arm64 in nat clouds * Fix syntax issues (cherry picked from commit 31adf25) * Add controller/cgroup path telemetry (#3231) * version update to 2.12.0.2 (#3233) --------- Co-authored-by: Norberto Arrieta <narrieta@users.noreply.github.com> Co-authored-by: maddieford <93676569+maddieford@users.noreply.github.com> Co-authored-by: Long Li <longli@microsoft.com> Co-authored-by: sebastienb-stormshield <sebastien.bini@stormshield.eu> Co-authored-by: Zheyu Shen <arsdragonfly@gmail.com> Co-authored-by: Zhidong Peng <zpeng@microsoft.com> Co-authored-by: d1r3ct0r <mwadimemakokha@gmail.com> Co-authored-by: Robert Schweikert <rjschwei@suse.com> Co-authored-by: Miriam España Acebal <miriam.espana@canonical.com> Co-authored-by: Anam Ahmad <Anam9@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Issue #
PR information
Quality of Code and Contribution Guidelines