(bugfix) log level does not change when standalone agent is reloaded or when otel runtime is used

[khushijain21]

What does this PR do?

When agent.logging.level config is changed during runtime in standalone elastic-agent, the new log level was not respected. (Note: This works in managed mode. This is the code path followed by managed mode for ref https://github.com/khushijain21/elastic-agent/blob/loglevel/internal/pkg/agent/application/actions/handlers/handler_action_settings.go#L54-L94 )

This PR fixes this bug and ensures this works both for process and OTel runtime.

Note: If both agent.logging.level and service::telemetry::logs::level are set - then the agent log level is applied to process runtime and telemetry::logs is applied to OTel runtime. If not, all components use agent.logging.level

Why is it important?

This is important so that we don't have to restart elastic-agent to update its logging level

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None

How to test this PR locally

Start standalone elastic-agent with monitoring enabled and change the logging level without restarting. Observe the debug logs.

I tested this PR manually for both process and otel mode. And by switching from INFO-DEBUG-INFO.

Related issues

• Closes Log level changes do not affect the elastic-otel-collector sub-process #11959

[mergify]

This pull request does not have a backport label. Could you fix it @khushijain21? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pierrehilbert]

@khushijain21 just to confirm my understanding, this is only happening with OTel right?

[khushijain21]

@khushijain21 just to confirm my understanding, this is only happening with OTel right?

I was able to reproduce this bug even on 8.18 and 9.2 in standalone elastic-agent. So, no, it doesn't happen with OTel only.

[pierrehilbert]

We should definitively add a changelog entry then

[cmacknz]

(Note: This works in managed mode. This is the code path followed by managed mode for ref https://github.com/khushijain21/elastic-agent/blob/loglevel/internal/pkg/agent/application/actions/handlers/handler_action_settings.go#L54-L94 )

For context there are (not obviously) three paths for managed mode:

1. The SETTINGS action you linked, which lets you set the log level for an individual agent without generating a policy change action. This is what happens if you change the log level in the agent details view in the Fleet UI.
2. A policy level log level setting to change the log level for every agent in a policy at once, which was introduced as the second mechanism in Support log level setting from policy #3090.
3. The initial log level can be set per agent in the elastic-agent.yml, prior to the introduction of the policy level log level this is how people would intialize the log level for each agent consistently without having to send each of them a SETTINGS action. This is why you will see some logic around falling back to the locally stored agent level.

[cmacknz]

Thanks! I tested this locally and it worked with a Fleet managed agent to change the log level via the policy or via the settings action.

There are a couple of improvements to make, and we need some test coverage for this as well showing that the cases that didn't work before still do.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 992db44

History

• 💔 Build #32962 failed a711fd5
• 💔 Build #32874 failed eeca2e6
• 💔 Build #32866 failed 6044a3e
• 💔 Build #32861 failed de36b8d
• 💔 Build #32852 failed 0833da4

cc @khushijain21

[cmacknz]

LGTM, thanks for sticking with this!

[github-actions]

@Mergifyio backport 8.19 9.1 9.2 9.3

[mergify]

backport 8.19 9.1 9.2 9.3

✅ Backports have been created

Details

• #12124 [8.19] (backport #11998) (bugfix) log level does not change when standalone agent is reloaded or when otel runtime is used has been created for branch 8.19 but encountered conflicts
• #12125 [9.1] (backport #11998) (bugfix) log level does not change when standalone agent is reloaded or when otel runtime is used has been created for branch 9.1 but encountered conflicts
• #12126 [9.2] (backport #11998) (bugfix) log level does not change when standalone agent is reloaded or when otel runtime is used has been created for branch 9.2 but encountered conflicts
• #12127 [9.3] (backport #11998) (bugfix) log level does not change when standalone agent is reloaded or when otel runtime is used has been created for branch 9.3

[cmacknz]

We can technically backport this as it's a bug fix but let's keep it only in 9.3 so we have time to soak test it first. There's no urgency to fix this and it is not a simple change. I've closed the backport PRs to other branches.