[MAC]: Agent gets uninstalled without uninstall-token after upgrade(8.11.3>8.12.0) when Tamper protection is enabled. #3926

amolnater-qasource · 2023-12-19T10:49:30Z

Kibana Build details:

VERSION: 8.12.0 BC2
BUILD: 69899
COMMIT: 15a6cc8236b4828b97da733746ec36bd33f03bba
Artifact Link: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.11.3-darwin-x86_64.tar.gz

Host OS: MAC14

Preconditions:

8.12.0 BC2 Kibana cloud environment should be available.
8.11.3 Agent should be installed using agent policy.
Agent tamper protection should be enabled.

Steps to reproduce:

Trigger upgrade for 8.11.3 mac agent.
Observe agent gets upgraded to latest version.
Wait for more than 20 minutes.
Run agent uninstall command without uninstall-token and observe agent is uninstalled.

NOTE:

Issue is not reproducible on Windows and Linux agents.

Screenshot:

Expected Result:
Agent shouldn't get uninstalled without uninstall-token after upgrade(8.11.3>8.12.0) when Tamper protection is enabled.

Logs:
elastic-agent-diagnostics-2023-12-19T10-10-03Z-00.zip

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-12-19T10:49:32Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

amolnater-qasource · 2023-12-19T10:49:39Z

@manishgupta-qasource Please review.

manishgupta-qasource · 2023-12-19T11:32:36Z

Secondary review for this ticket is Done

pierrehilbert · 2023-12-19T11:58:35Z

@aleksmaus do you have an idea here?

aleksmaus · 2023-12-19T14:06:31Z

The tamper protection implementation is not platform specific on the agent side, so if it worked on Windows and Linux it should in theory work the same on Mac.
The tamper protection is implemented by Endpoint, if Endpoint is properly protected after upgrade it can't be uninstalled on it's own and the Agent will not uninstall if Endpoint fails to uninstall.
Since this uninstall after upgrade, and during upgrade process Endpoint unprotects itself temporarily in order to be able to be uninstalled, I wonder if the protection was not reenabled by Endpoint for some reason.

The Agent logs (from the attached diags file) contain Endpoint logs that point to the fact that the protection was enabled at 2023-12-19T09:33:52.270Z

{"log.level":"error","@timestamp":"2023-12-19T09:33:52.270Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 09:33:52: debug: Utilities.cpp:420 Tamper protection enabled","context":"command output","ecs.version":"1.6.0"}

followed by

{"log.level":"error","@timestamp":"2023-12-19T09:33:52.270Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 09:33:52: info: InstallLib.cpp:885 Failed to read os section of tamper-protection-config, continuing","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-19T09:33:52.270Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 09:33:52: info: InstallLib.cpp:975 Finished checking installed uninstall protection artifacts with result deny","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-19T09:33:52.270Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 09:33:52: info: InstallLib.cpp:1047 Finished checking command line provided uninstall resource result deny","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-19T09:33:52.270Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 09:33:52: error: InstallLib.cpp:1237 Invalid uninstall token","context":"command output","ecs.version":"1.6.0"}

And then 10 minutes later it looks like Endpoint is not protected

{"log.level":"error","@timestamp":"2023-12-19T09:44:59.771Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 09:44:43: info: InstallLib.cpp:1153 Skipping uninstall token validation as tamper protection is not enabled.","context":"command output","ecs.version":"1.6.0"}

I'll test some more, and also CCing @intxgo on Endpoint side, cause he would know the Endpoint implementation details better. Leszek if you have theories why the protection was not reenabled after upgrade please comment.

aleksmaus · 2023-12-19T15:24:46Z

Could you please provide more details on the test environment?
Where is the agent 8.12.0 package is downloaded from?
I used the cloud deployment 8.12.0, installed the agent 8.11.3, and don't see any upgrades available for that agent.

I tried to install the agent 8.12.0 from the snapshot builds
https://snapshots.elastic.co/8.12.0-83badc78/downloads/beats/elastic-agent/elastic-agent-8.12.0-SNAPSHOT-darwin-aarch64.tar.gz

That fails to install Endpoint on code signature verification

184 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.531Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:306 Invalid signature for path: /Library/Elastic/Agent/data/elastic-agent-828420/components/endpoint-security","context":"command output","ecs.version":"1.6.0"}
185 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.532Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:339 Signature identifier: endpoint_installer-55554944c94fac6709ed32ca8cb80b6f13156f88","context":"command output","ecs.version":"1.6.0"}
186 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.532Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:219 Checking /Applications/ElasticEndpoint.app/Contents/MacOS/ElasticEndpoint","context":"command output","ecs.version":"1.6.0"}
187 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.541Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:293 Valid signature for path: /Applications/ElasticEndpoint.app/Contents/MacOS/ElasticEndpoint","context":"command output","ecs.version":"1.6.0"}
188 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.541Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:339 Signature identifier: co.elastic.endpoint","context":"command output","ecs.version":"1.6.0"}
189 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.541Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:103 Extracted signer info: Apple Development: Douglas Weyrauch (WNASJ7283B)","context":"command output","ecs.version":"1.6.0"}
190 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.541Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:103 Extracted signer info: Apple Worldwide Developer Relations Certification Authority","context":"command output","ecs.version":"1.6.0"}
191 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.541Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:103 Extracted signer info: Apple Root CA","context":"command output","ecs.version":"1.6.0"}
192 {"log.level":"error","@timestamp":"2023-12-19T14:59:13.541Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-12-19 14:59:13:
    debug: CodeSignature.cpp:71 Extracted entitlement: com.apple.developer.system-extension.install","context":"command output","ecs.version":"1.6.0"}   
    ```

amolnater-qasource · 2023-12-20T06:05:45Z

Hi @aleksmaus

Thank you for looking into this.

Please find below more details for the same.

We have installed the 8.11.3 released artifact for mac with agent policy having tamper protection enabled and after that no changes were made to it.
Added agent binary: https://staging.elastic.co/8.12.0-f6277726/downloads/ so that the agent gets upgraded successfully.
- This is the artifact link to the BC2 kibana.
The upgrade is not available under the Fleet UI, for exactly previous version until latest version is not released globally.
- Some changes were made under [Fleet] Agent upgrade available should use latest agent version kibana#167410
- Due to this for agent upgrade to a BC build from exactly last version released we need to use API to trigger upgrade.

POST kbn:/api/fleet/agents/<agent-id>/upgrade
{
  "version": "8.12.0"
}

We upgraded this agent to latest version and then waited for more than 20 minutes, and then attempted to uninstall[without token] with already enabled tamper protection.

Further, we are not able to reproduce this on Windows or Linux agent.
However we reproduced this issue multiple times on MAC on agent upgrade.
- Issue is not reproducible by directly installing 8.12.0 Agent.

Please let us know if we are still missing anything here.

Thanks!!

intxgo · 2023-12-20T12:51:51Z

I'll check the logs. When trying to reproduce this, please also check what's the state of Endpoint, was it already upgraded?
sudo /Library/Elastic/Endpoint/elastic-endpoint version
When Agent was uninstalled, was Endpoint also gone, or does it stay on the system?

intxgo · 2023-12-20T12:55:04Z

the version.txt from attached diagnostics

build_time: 2023-12-14T16:27:23Z
commit: 82842070a93cc09a7a18ee021fa7ebcd6f3974d7
snapshot: false
version: 8.12.0

intxgo · 2023-12-20T13:36:24Z

The attached diagnostics indicate that Endpoint was properly protected, version 8.12.0
Then in the logs:

{"log.level":"info","@timestamp":"2023-12-19T09:33:50.212Z","log.logger":"component.runtime.endpoint-default.service_runtime","log.origin":{"file.name":"runtime/service.go","file.line":318},"message":"failed check endpoint service: 2023-12-19 09:33:50: notice: InstallLib.cpp:624 Installed endpoint is a different version; found: [version: 8.12.0, compiled: Wed Dec 13 11:00:00 2023, branch: HEAD, commit: c307bc90ff84ad6f1412c048219bb720a85e5815], expected: [version: 8.11.3, compiled: Thu Dec 7 13:00:00 2023, branch: HEAD, commit: a6bfc3ce0ece32f6d1da8e5623f7183445e4c14e]: exit status 2, try install","ecs.version":"1.6.0"}

which initiates

InstallLib.cpp:309 Running [/Library/Elastic/Agent/data/elastic-agent-f4f6fb/components/previous/elastic-endpoint.app/Contents/MacOS/elastic-endpoint] [uninstall --keepstate --log stdout]
...
InstallLib.cpp:1237 Invalid uninstall token"
InstallLib.cpp:516 Failed to uninistall with preserved state, attempting full uninstall"

and it seems the last line did uninstall Endpoint without any tamper protection checks. I'm digging in Endpoint code, either we have a bug ~~or Agent sent us UPGRADE action in the meant time.~~

~~Is it possible that Agent attempts to upgrade Endpoint on it's own and in parallel proceeds with uninstall initiated by command line?~~

The log appears linear, my theory is that Endpoint installer did not read process exit code correctly from launched subprocess. I'm going to reproduce it locally

intxgo · 2023-12-20T16:07:46Z

I confirm it's a bug in macOS version of Elastic Defend.

aleksmaus · 2023-12-20T16:15:21Z

I confirm it's a bug in macOS version of Elastic Defend.

Leszek, thank you very much for the help!

ricardoungureanu · 2024-01-15T11:44:01Z

Hi @amolnater-qasource!

I have tried reproducing this and I was unable to. From the logs above, it seems that there were 2 updates, one at 9:33, the other at 9:44. I can see both of them complete successfully.

The log Skipping uninstall token validation as tamper protection is not enabled is because Agent invalidates the config during the update, temporary disabling TP.

Here is the output when I try to uninstall endpoint

Elastic Agent will be uninstalled from your system at /Library/Elastic/Agent. Do you want to continue? [Y/n]:Y
[=   ] Failed to uninstall service  [0s] failed to uninstall component "endpoint-default": error uninstalling service: 2024-01-15 11:26:16: error: InstallLib.cpp:1237 Invalid uninstall token: exit status 28
[=   ] Failed to uninstall agent  [0s] Error: error uninstalling agent: error uninstalling components: error uninstalling component: error uninstalling service: 2024-01-15 11:26:16: error: InstallLib.cpp:1237 Invalid uninstall token: exit status 28
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.12/fleet-troubleshooting.html
sh: git_branch: command not found

I used the same versions as indicated by you.

Thanks!

amolnater-qasource · 2024-01-16T09:51:53Z

Hi Team,

We have revalidated this issue on latest 8.12.0 BC6 kibana cloud environment and found it fixed now.

Observations:

Agent doesn't get uninstalled without uninstall-token after upgrade(8.11.4>8.12.0-BC6) when Tamper protection is enabled.

Build details:
VERSION: 8.12.0 BC6
BUILD: 70088
COMMIT: e9092c0a17923f4ed984456b8a5db619b0a794b3
Artifact Link: https://staging.elastic.co/8.12.0-3eba7f46/summary-8.12.0.html

Screenshot:

Hence we are closing this issue and marking as QA:Validated.

Thanks!

harshitgupta-qasource · 2024-01-25T11:13:32Z

`Bug Conversion`

Test-Case not required as this particular checkpoint is already covered in the following testcase:
- https://elastic.testrail.io/index.php?/cases/view/271165

Thanks!

amolnater-qasource added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team impact:high Short-term priority; add to current release, or definitely next. labels Dec 19, 2023

amolnater-qasource closed this as completed Jan 16, 2024

amolnater-qasource added the QA:Validated Validated by the QA Team label Jan 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MAC]: Agent gets uninstalled without uninstall-token after upgrade(8.11.3>8.12.0) when Tamper protection is enabled. #3926

[MAC]: Agent gets uninstalled without uninstall-token after upgrade(8.11.3>8.12.0) when Tamper protection is enabled. #3926

amolnater-qasource commented Dec 19, 2023

elasticmachine commented Dec 19, 2023

amolnater-qasource commented Dec 19, 2023

manishgupta-qasource commented Dec 19, 2023

pierrehilbert commented Dec 19, 2023

aleksmaus commented Dec 19, 2023 •

edited

Loading

aleksmaus commented Dec 19, 2023

amolnater-qasource commented Dec 20, 2023

intxgo commented Dec 20, 2023

intxgo commented Dec 20, 2023

intxgo commented Dec 20, 2023 •

edited

Loading

intxgo commented Dec 20, 2023

aleksmaus commented Dec 20, 2023

ricardoungureanu commented Jan 15, 2024

amolnater-qasource commented Jan 16, 2024

harshitgupta-qasource commented Jan 25, 2024

[MAC]: Agent gets uninstalled without uninstall-token after upgrade(8.11.3>8.12.0) when Tamper protection is enabled. #3926

[MAC]: Agent gets uninstalled without uninstall-token after upgrade(8.11.3>8.12.0) when Tamper protection is enabled. #3926

Comments

amolnater-qasource commented Dec 19, 2023

elasticmachine commented Dec 19, 2023

amolnater-qasource commented Dec 19, 2023

manishgupta-qasource commented Dec 19, 2023

pierrehilbert commented Dec 19, 2023

aleksmaus commented Dec 19, 2023 • edited Loading

aleksmaus commented Dec 19, 2023

amolnater-qasource commented Dec 20, 2023

intxgo commented Dec 20, 2023

intxgo commented Dec 20, 2023

intxgo commented Dec 20, 2023 • edited Loading

intxgo commented Dec 20, 2023

aleksmaus commented Dec 20, 2023

ricardoungureanu commented Jan 15, 2024

amolnater-qasource commented Jan 16, 2024

harshitgupta-qasource commented Jan 25, 2024

Bug Conversion

aleksmaus commented Dec 19, 2023 •

edited

Loading

intxgo commented Dec 20, 2023 •

edited

Loading

`Bug Conversion`