Push the total request-body-bytes to usage-api #194429

ersin-erdal · 2024-09-30T12:47:10Z

Resolves: https://github.com/elastic/response-ops-team/issues/209

This PR is a follow-on of #186804.

Creates a new task that runs every 1 hour to push the total connector-request-body-bytes that have been saved in the event log to usage-api.

kibana-ci · 2024-09-30T16:15:45Z

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

💔 Build #238021 failed 134918c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ersin-erdal

# Conflicts: # x-pack/plugins/actions/server/config.ts

pmuellr · 2024-10-17T16:06:00Z

x-pack/plugins/actions/server/config.ts

@@ -72,6 +72,8 @@ const connectorTypeSchema = schema.object({
  maxAttempts: schema.maybe(schema.number({ min: MIN_MAX_ATTEMPTS, max: MAX_MAX_ATTEMPTS })),
 });

+export const DEFAULT_USAGE_API_URL = 'https://usage-api.elastic-system/api/v1/usage';


We can't do this, as this way of hard-coding the URL broke, when the URL changed :-)

We're in the process of figuring out what to do with these, but we likely need a central location (core? new plugin?) to maintain these things, including the ca certs that we've already added.

github-actions · 2024-10-17T16:46:43Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…o 209-request-body-bytes-task

elasticmachine · 2024-10-28T10:28:44Z

Pinging @elastic/response-ops (Team:ResponseOps)

…kibana into 209-request-body-bytes-task

pmuellr

LGTM, but left a comment about the const DEFAULT_USAGE_API_URL - that URL doesn't seem right.

I'm also curious what the story is for core providing us a plugin that does some of this stuff. Is there an issue open for that, assigned to core (or whomever), we can link this to? And do we have an issue open to change this code to start using that future code?

pmuellr · 2024-10-28T13:34:42Z

x-pack/plugins/actions/server/config.ts

@@ -72,6 +72,8 @@ const connectorTypeSchema = schema.object({
  maxAttempts: schema.maybe(schema.number({ min: MIN_MAX_ATTEMPTS, max: MAX_MAX_ATTEMPTS })),
 });

+export const DEFAULT_USAGE_API_URL = 'https://usage-api.usage-api/api/v1/usage';


Is this a URL that actually works? Seems like security uses usage-api.elastic-system? Typo?

This is the new URL.
They rolled this back after the incident then rolled out up to QA for testing.
I will check if it is already on prod before merging the PR.

usage-api namespace is correct and what we're using in prod now for security solution: https://github.com/elastic/kibana-controller/blob/621b30031f8a0a7205ea79eb9ad2ac2dbbff03c5/helm/values.yaml#L69. I believe the elastic-system namespace for usage-api has already been deleted.

x-pack/plugins/actions/server/usage/connector_usage_reporting_task.test.ts

…kibana into 209-request-body-bytes-task

js-jankisalvi

LGTM 👍

ersin-erdal · 2024-10-29T12:16:09Z

I'm also curious what the story is for core providing us a plugin that does some of this stuff. Is there an issue open for that, assigned to core (or whomever), we can link this to? And do we have an issue open to change this code to start using that future code?

I mentioned this PR ^^^ in the issue to create the plugin.

afharo

LGTM

afharo · 2024-10-30T10:48:19Z

x-pack/plugins/actions/server/config.ts

-    })
-  ),
+  usage: schema.object({
+    url: schema.string({ defaultValue: DEFAULT_USAGE_API_URL }),


nit: May I suggest that this is schema.maybe(schema.string())?

This way, we've got a way to disable the feature by not providing the URL in the config.

WDYT?

I am planning to use the ca for that purpose.
And this is temporary, there is an issue to create a plugin that provides ca and url from kibana-controller.

…kibana into 209-request-body-bytes-task

Aran-K

One comment on ID, but other than that the usage record structure looks good

Aran-K · 2024-10-31T14:42:37Z

x-pack/plugins/actions/server/usage/connector_usage_reporting_task.ts

+    timestamp.setMilliseconds(0);
+
+    return {
+      id: `connector-request-body-bytes-${timestamp.toISOString()}`,


You might want to consider making this more unique. id needs to be globally unique, so adding in project_id or something similar could help ensure that

elasticmachine · 2024-10-31T17:22:49Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b557370
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-194429-b55737026fb6

Failed CI Steps

FTR Configs #79

Metrics [docs]

✅ unchanged

History

💚 Build #247117 succeeded 83760d7
💚 Build #246731 succeeded 7e74f86
💔 Build #246570 failed d3e4f3a
💛 Build #246260 was flaky 7987940
💚 Build #246127 succeeded 038d0e6

cc @ersin-erdal

kibanamachine · 2024-10-31T19:30:37Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11618611909

Resolves: elastic/response-ops-team#209 This PR is a follow-on of elastic#186804. Creates a new task that runs every 1 hour to push the total connector-request-body-bytes that have been saved in the event log to usage-api. (cherry picked from commit 216f899)

kibanamachine · 2024-10-31T19:35:38Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [Push the total request-body-bytes to usage-api (#194429)](#194429)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com>

Resolves: elastic/response-ops-team#209 This PR is a follow-on of elastic#186804. Creates a new task that runs every 1 hour to push the total connector-request-body-bytes that have been saved in the event log to usage-api.

Push the total request-body-bytes to usage-api

134918c

ersin-erdal self-assigned this Sep 30, 2024

ersin-erdal added ci:cloud-redeploy Always create a new Cloud deployment and removed ci:cloud-deploy Create or update a Cloud deployment labels Sep 30, 2024

Merge branch 'main' into 209-request-body-bytes-task

7f423e3

This was referenced Oct 1, 2024

Add CA certificate path to actions config #194528

Merged

Push the request-body-bytes collected by the ConnectorUsageCollector to usage-api #191259

Closed

ersin-erdal added 6 commits October 13, 2024 19:22

Merge branch 'refs/heads/main' into 209-request-body-bytes-task

f24eee7

# Conflicts: # x-pack/plugins/actions/server/config.ts

move url to config

0a5472d

fix unit tests

698b39b

fix config test

f71606c

test with missing project id

fc98df8

fix test

3c210b1

ersin-erdal requested review from Zacqary and removed request for Zacqary October 15, 2024 16:18

Merge branch 'main' into 209-request-body-bytes-task

563e87e

ersin-erdal added ci:project-deploy-elasticsearch Create an Elasticsearch Serverless project and removed ci:cloud-redeploy Always create a new Cloud deployment labels Oct 17, 2024

Merge branch 'main' into 209-request-body-bytes-task

b122027

pmuellr requested changes Oct 17, 2024

View reviewed changes

ersin-erdal added ci:project-deploy-observability Create an Observability project and removed ci:project-deploy-elasticsearch Create an Elasticsearch Serverless project labels Oct 17, 2024

ersin-erdal added 2 commits October 17, 2024 21:44

handle undefined aggregation result

02d746c

Merge remote-tracking branch 'origin/209-request-body-bytes-task' int…

14a9a5f

…o 209-request-body-bytes-task

Merge branch 'main' into 209-request-body-bytes-task

038d0e6

ersin-erdal marked this pull request as ready for review October 28, 2024 10:28

ersin-erdal requested review from a team as code owners October 28, 2024 10:28

ersin-erdal and others added 3 commits October 28, 2024 11:59

Set interval to 1h

9d99ce9

Merge branch '209-request-body-bytes-task' of github.com:ersin-erdal/…

21081c5

…kibana into 209-request-body-bytes-task

Merge branch 'main' into 209-request-body-bytes-task

7987940

pmuellr approved these changes Oct 28, 2024

View reviewed changes

ersin-erdal added 2 commits October 29, 2024 11:20

fix typo

66f08d0

Merge branch '209-request-body-bytes-task' of github.com:ersin-erdal/…

d3e4f3a

…kibana into 209-request-body-bytes-task

js-jankisalvi approved these changes Oct 29, 2024

View reviewed changes

Merge branch 'main' into 209-request-body-bytes-task

7e74f86

afharo approved these changes Oct 30, 2024

View reviewed changes

ersin-erdal added 2 commits October 30, 2024 15:58

make missing project it log a warning

d9f82b9

Merge branch '209-request-body-bytes-task' of github.com:ersin-erdal/…

83760d7

…kibana into 209-request-body-bytes-task

Aran-K reviewed Oct 31, 2024

View reviewed changes

ersin-erdal and others added 2 commits October 31, 2024 17:14

Add project id to the report id

2a76a29

Merge branch 'main' into 209-request-body-bytes-task

b557370

ersin-erdal merged commit 216f899 into elastic:main Oct 31, 2024
26 checks passed

kibanamachine added the v9.0.0 label Oct 31, 2024

kibanamachine mentioned this pull request Oct 31, 2024

[8.x] Push the total request-body-bytes to usage-api (#194429) #198614

Merged

kibanamachine added the v8.17.0 label Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Push the total request-body-bytes to usage-api #194429

Push the total request-body-bytes to usage-api #194429

ersin-erdal commented Sep 30, 2024 •

edited by kibanamachine

Loading

kibana-ci commented Sep 30, 2024 •

edited

Loading

pmuellr Oct 17, 2024

github-actions bot commented Oct 17, 2024

elasticmachine commented Oct 28, 2024

pmuellr left a comment

pmuellr Oct 28, 2024

ersin-erdal Oct 29, 2024

joeypoon Oct 29, 2024 •

edited

Loading

js-jankisalvi left a comment

ersin-erdal commented Oct 29, 2024

afharo left a comment

afharo Oct 30, 2024

ersin-erdal Oct 30, 2024

Aran-K left a comment

Aran-K Oct 31, 2024

elasticmachine commented Oct 31, 2024 •

edited

Loading

kibanamachine commented Oct 31, 2024

kibanamachine commented Oct 31, 2024

Push the total request-body-bytes to usage-api #194429

Push the total request-body-bytes to usage-api #194429

Conversation

ersin-erdal commented Sep 30, 2024 • edited by kibanamachine Loading

kibana-ci commented Sep 30, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

pmuellr Oct 17, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 17, 2024

🤖 GitHub comments

elasticmachine commented Oct 28, 2024

pmuellr left a comment

Choose a reason for hiding this comment

pmuellr Oct 28, 2024

Choose a reason for hiding this comment

ersin-erdal Oct 29, 2024

Choose a reason for hiding this comment

joeypoon Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

js-jankisalvi left a comment

Choose a reason for hiding this comment

ersin-erdal commented Oct 29, 2024

afharo left a comment

Choose a reason for hiding this comment

afharo Oct 30, 2024

Choose a reason for hiding this comment

ersin-erdal Oct 30, 2024

Choose a reason for hiding this comment

Aran-K left a comment

Choose a reason for hiding this comment

Aran-K Oct 31, 2024

Choose a reason for hiding this comment

elasticmachine commented Oct 31, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

kibanamachine commented Oct 31, 2024

kibanamachine commented Oct 31, 2024

💚 All backports created successfully

Questions ?

ersin-erdal commented Sep 30, 2024 •

edited by kibanamachine

Loading

kibana-ci commented Sep 30, 2024 •

edited

Loading

joeypoon Oct 29, 2024 •

edited

Loading

elasticmachine commented Oct 31, 2024 •

edited

Loading