Skip to content

Attempt to make test more reliable by querying ES directly #8422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jun 24, 2025
Merged

Attempt to make test more reliable by querying ES directly #8422

merged 6 commits into from
Jun 24, 2025

Conversation

@michel-laterman
Copy link
Contributor

@michel-laterman michel-laterman commented Jun 9, 2025

What does this PR do?

Attempt to make the TestFIPSAgentConnectingToFIPSFleetServerInECHFRH test more reliable by querying ES for agents running the cloud policy instead of gathering agents from Kibana.

Related issues

@michel-laterman
Attempt to make test more reliable by querying ES directly
7db3303 
Attempt to make the TestFIPSAgentConnectingToFIPSFleetServerInECHFRH
test more reliable by querying ES for agents running the cloud policy
instead of gathering agents from Kibana.
@michel-laterman michel-laterman requested a review from a team as a code owner June 9, 2025 22:59
@michel-laterman michel-laterman requested a review from swiatekm June 9, 2025 22:59
@michel-laterman michel-laterman added flaky-test Unstable or unreliable test cases. backport-8.19 Automated backport to the 8.19 branch labels Jun 9, 2025
@ycombinator
Copy link
Contributor

ycombinator commented Jun 9, 2025

I don't love that we are reaching into ES docs to determine whether the Agent is FIPS-capable. I would prefer to instead use the Kibana Fleet API and check its response to determine whether the Agent is FIPS-capable. I'm trying this out in #8421 with a require.Eventually wrapped around the Kibana Fleet API call and test assertion. That said, I'm happy to abandon my PR if the flakiness persists with it and go with this PR here if it's reliably not flaky.

@swiatekm
Copy link
Contributor

swiatekm commented Jun 10, 2025

I like the solution in Shaunak's PR more as well. The FIPS agent should eventually be accessible via Kibana's API. Going around it to ES itself feels unnecessarily hacky.

@mergify
Copy link
Contributor

mergify bot commented Jun 10, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b integration-fips-status-es upstream/integration-fips-status-es
git merge upstream/main
git push upstream integration-fips-status-es

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 10, 2025

Closing as the other solution seems to work well enough for the moment

This was referenced Jun 12, 2025
Closed
Open
@mergify
Copy link
Contributor

mergify bot commented Jun 17, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b integration-fips-status-es upstream/integration-fips-status-es
git merge upstream/main
git push upstream integration-fips-status-es

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 17, 2025

As discussed in the FIPS sync, i've fixed this PR to work with the eventually loop so that we can eliminate calls to Kibana as it is a component that is only introduced for testing.

ycombinator
ycombinator previously approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's run this in CI at least 10 times to make sure there's no flakiness.

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 17, 2025

buildkite test this

@michel-laterman michel-laterman requested review from a team as code owners June 17, 2025 20:36
@michel-laterman michel-laterman force-pushed the integration-fips-status-es branch from 094c116 to bb0f434 Compare June 17, 2025 20:37
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jun 18, 2025
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 18, 2025

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@v1v v1v removed request for a team June 18, 2025 07:44
@v1v
Copy link
Member

v1v commented Jun 18, 2025

I have unassigned my teams from the review. It seems that the issue is not related to the CI itself. Please let me know if we can assist with anything else. Thanka

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 19, 2025

buildkite test this

swiatekm
swiatekm previously approved these changes Jun 20, 2025
View reviewed changes
Copy link
Contributor

@swiatekm swiatekm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this actually fixes the flakiness, I'd be interested to find out why going through Kibana introduces it in the first place.

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 20, 2025

buildkite test this

2 similar comments
@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 23, 2025

buildkite test this

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jun 23, 2025

buildkite test this

@mergify
Copy link
Contributor

mergify bot commented Jun 24, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b integration-fips-status-es upstream/integration-fips-status-es
git merge upstream/main
git push upstream integration-fips-status-es

@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Jun 24, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 24, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

cc @michel-laterman

@michel-laterman michel-laterman merged commit d1d9e16 into elastic:main Jun 24, 2025
19 checks passed
@michel-laterman michel-laterman deleted the integration-fips-status-es branch June 24, 2025 20:44
mergify bot pushed a commit that referenced this pull request Jun 24, 2025
@michel-laterman @mergify
Attempt to make test more reliable by querying ES directly (#8422)
3534203 
Attempt to make the TestFIPSAgentConnectingToFIPSFleetServerInECHFRH
test more reliable by querying ES for agents running the cloud policy
instead of gathering agents from Kibana.

(cherry picked from commit d1d9e16)
@mergify mergify bot mentioned this pull request Jun 24, 2025
Merged
v1v added a commit that referenced this pull request Jun 25, 2025
@v1v
Merge branch 'feature/hosted-stack-using-oblt-cli' into test/oblt-cli…
b0720e7 
…-hosted

* feature/hosted-stack-using-oblt-cli: (26 commits)
  Use the current official docker image for oblt-cli
  Mark the elasticinframetrics processor as deprecated and schedule for removal (#8659)
  [main][Automation] Update versions (#8668)
  chore: Update create_deployment_csp_configuration.yaml (#8669)
  Attempt to make test more reliable by querying ES directly (#8422)
  [test] split up ess and beats serverless integration tests (#8551)
  Remove resource/k8s processor and use k8sattributes processor for service attributes (#8599)
  fix: use --force-confold for deb tests in TestUpgradeAgentWithTamperProtectedEndpoint_DEB (#8649)
  [main][Automation] Bump stack images versions to 9.1.0-ea0b7542 (#8612)
  chore: Update to elastic/beats@f6594fb72670 (#8640)
  [deb/rpm] restart endpoint with tamper protection after elastic-agent  (#8637)
  ci: don't preinstall fleet packages on retried CI steps (#8636)
  chore: Update to elastic/beats@6b6941eed496 (#8619)
  [main][Automation] Bump VM Image version to 1750467641 (#8617)
  flaky: skip TestUpgradeAgentWithTamperProtectedEndpoint_RPM (#8626)
  Add skip-changelog PR label for bump VM PRs (#8627)
  build(deps): bump github.com/elastic/go-seccomp-bpf from 1.5.0 to 1.6.0 (#8611)
  [ci] fix k8s integration tests flakiness (#8575)
  bump apmconfig Otel extension to v0.3.0 (#8600)
  Enhancement/6394 allow deb rpm to upgrade with endpoint tamper protection (#6907)
  ...
pkoutsovasilis pushed a commit that referenced this pull request Jul 2, 2025
@mergify @michel-laterman
Attempt to make test more reliable by querying ES directly (#8422) (#…
59ff169 
…8662)

Attempt to make the TestFIPSAgentConnectingToFIPSFleetServerInECHFRH
test more reliable by querying ES for agents running the cloud policy
instead of gathering agents from Kibana.

(cherry picked from commit d1d9e16)

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>
Co-authored-by: michel-laterman <michel.laterman@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ycombinator ycombinator ycombinator approved these changes

@swiatekm swiatekm Awaiting requested review from swiatekm

Assignees

@michel-laterman michel-laterman

Labels

backport-8.19 Automated backport to the 8.19 branch flaky-test Unstable or unreliable test cases. skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@michel-laterman @ycombinator @swiatekm @elasticmachine @v1v @pierrehilbert