-
Notifications
You must be signed in to change notification settings - Fork 464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rally benchmark kubernetes.pod #8443
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
6582a2c
bump github.com/elastic/package-spec/v3@v3.0.1
ac9b2c6
Rally benchmark kubernetes.pod
4791f9b
Merge branch 'main' into rally_benchmark_kubernetes.pod
d57bc4a
timestamp two days period
f67d633
match pod.uid and pod.name
3e2d345
fix description
1ebd17a
refactor node id
99351cf
use negative period of timestamp
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
14 changes: 14 additions & 0 deletions
14
packages/kubernetes/_dev/benchmark/rally/pod-benchmark.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| --- | ||
| description: Benchmark 864000 kubernetes.pod events ingested | ||
| data_stream: | ||
| name: pod | ||
| corpora: | ||
| generator: | ||
| total_events: 864000 | ||
| template: | ||
| type: gotext | ||
| path: ./pod-benchmark/template.ndjson | ||
| config: | ||
| path: ./pod-benchmark/config.yml | ||
| fields: | ||
| path: ./pod-benchmark/fields.yml |
42 changes: 42 additions & 0 deletions
42
packages/kubernetes/_dev/benchmark/rally/pod-benchmark/config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| fields: | ||
| - name: timestamp | ||
| period: -48h # two days | ||
| - name: cloud.availabilit_zone | ||
| value: "europe-west1-d" | ||
| - name: agent.id | ||
| value: "12f376ef-5186-4e8b-a175-70f1140a8f30" | ||
| - name: agent.name | ||
| value: "kubernetes-scale-123456" | ||
| - name: agent.ephemeral_id | ||
| value: "f94220b0-2ca6-4809-8656-eb478a66c541" | ||
| - name: agent.version | ||
| value: "8.7.0" | ||
| - name: agent.snasphost | ||
| value: false | ||
| - name: metricset.period | ||
| value: 10000 | ||
| - name: event.duration | ||
| range: | ||
| min: 1 | ||
| max: 4000000 | ||
| - name: container.network.ingress.bytes | ||
| range: | ||
| min: 0 | ||
| max: 15000 | ||
| - name: container.network.egress.bytes | ||
| range: | ||
| min: 0 | ||
| max: 10000 | ||
| - name: Bytes | ||
| range: | ||
| min: 1 | ||
| max: 3000000 | ||
| - name: rangeofid | ||
| range: | ||
| min: 9000 | ||
| max: 10000 | ||
| - name: Percentage | ||
| range: | ||
| min: 0 | ||
| max: 5000 | ||
| fuzziness: 0.005 |
32 changes: 32 additions & 0 deletions
32
packages/kubernetes/_dev/benchmark/rally/pod-benchmark/fields.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| - name: timestamp | ||
| type: date | ||
| - name: agent.id | ||
| type: keyword | ||
| - name: agent.name | ||
| type: keyword | ||
| - name: agent.version | ||
| type: keyword | ||
| - name: agent.ephemeral_id | ||
| type: keyword | ||
| - name: container.network.ingress.bytes | ||
| type: integer | ||
| - name: container.network.egress.bytes | ||
| type: integer | ||
| - name: event.duration | ||
| type: long | ||
| - name: event.ingested | ||
| type: date | ||
| - name: metricset.period | ||
| type: long | ||
| - name: Offset | ||
| type: integer | ||
| - name: Bytes | ||
| type: long | ||
| - name: rangeofid | ||
| type: integer | ||
| - name: Ip | ||
| type: ip | ||
| - name: Percentage | ||
| type: double | ||
| - name: agent.snapshot | ||
| type: boolean |
199 changes: 199 additions & 0 deletions
199
packages/kubernetes/_dev/benchmark/rally/pod-benchmark/template.ndjson
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,199 @@ | ||
| {{- $period := generate "metricset.period" }} | ||
| {{- $timestamp := generate "timestamp" }} | ||
| {{- $agentId := generate "agent.id" }} | ||
| {{- $agentVersion := generate "agent.version" }} | ||
| {{- $agentName := generate "agent.name" }} | ||
| {{- $agentEphemeralid := generate "agent.ephemeral_id" }} | ||
| {{- $rxbytes := generate "container.network.ingress.bytes" }} | ||
| {{- $txbytes := generate "container.network.egress.bytes" }} | ||
| {{- $rangeofid := generate "rangeofid" }} | ||
| {{- $nodeid := div $rangeofid 110 -}} | ||
| {{- $pct := generate "Percentage" }} | ||
| { "@timestamp": "{{$timestamp.Format `2006-01-02T15:04:05.999999Z07:00`}}", | ||
| "container":{ | ||
| "network":{ | ||
| "ingress":{ | ||
| "bytes": {{ $rxbytes }} | ||
| }, | ||
| "egress":{ | ||
| "bytes": {{ $txbytes }} | ||
| } | ||
| } | ||
| }, | ||
| "kubernetes": { | ||
| "node":{ | ||
| "uid":"host-{{ $nodeid }}" , | ||
| "hostname":"host-{{ $nodeid }}", | ||
| "name":"host-{{ $nodeid }}", | ||
| "labels":{ | ||
| "cloud_google_com/machine-family":"e2", | ||
| "cloud_google_com/gke-nodepool":"kubernetes-scale-nl", | ||
| "kubernetes_io/hostname":"host-{{ $nodeid }}", | ||
| "cloud_google_com/gke-os-distribution":"cos", | ||
| "topology_kubernetes_io/zone":"europe-west1-d", | ||
| "topology_gke_io/zone":"europe-west1-d", | ||
| "topology_kubernetes_io/region":"europe-west1", | ||
| "kubernetes_io/arch":"amd64", | ||
| "cloud_google_com/gke-cpu-scaling-level":"4", | ||
| "env":"kubernetes-scale", | ||
| "failure-domain_beta_kubernetes_io/region":"europe-west1", | ||
| "cloud_google_com/gke-max-pods-per-node":"110", | ||
| "cloud_google_com/gke-container-runtime":"containerd", | ||
| "beta_kubernetes_io/instance-type":"e2-standard-4", | ||
| "failure-domain_beta_kubernetes_io/zone":"europe-west1-d", | ||
| "node_kubernetes_io/instance-type":"e2-standard-4", | ||
| "beta_kubernetes_io/os":"linux", | ||
| "cloud_google_com/gke-boot-disk":"pd-balanced", | ||
| "kubernetes_io/os":"linux", | ||
| "cloud_google_com/private-node":"false", | ||
| "cloud_google_com/gke-logging-variant":"DEFAULT", | ||
| "beta_kubernetes_io/arch":"amd64" | ||
| } | ||
| }, | ||
| "pod":{ | ||
| "uid": "demo-pod-{{ $rangeofid }}", | ||
| "start_time": "{{$timestamp.Format `2006-01-02T15:04:05.999999Z07:00`}}", | ||
| "memory":{ | ||
| "rss":{ | ||
| "bytes":"{{generate `Bytes`}}" | ||
| }, | ||
| "major_page_faults":0, | ||
| "usage":{ | ||
| "node":{ | ||
| "pct": "{{divf $pct 1000000}}" | ||
| }, | ||
| "bytes": "{{generate `Bytes`}}", | ||
| "limit":{ | ||
| "pct":"{{divf $pct 1000000}}" | ||
| } | ||
| }, | ||
| "available":{ | ||
| "bytes":0 | ||
| }, | ||
| "page_faults":1386, | ||
| "working_set":{ | ||
| "bytes": "{{generate `Bytes`}}", | ||
| "limit":{ | ||
| "pct": "{{divf $pct 1000000}}" | ||
| } | ||
| } | ||
| }, | ||
| "ip":"{{generate `Ip`}}", | ||
| "name":"demo-pod-{{ $rangeofid }}", | ||
| "cpu":{ | ||
| "usage":{ | ||
| "node":{ | ||
| "pct":0 | ||
| }, | ||
| "nanocores":0, | ||
| "limit":{ | ||
| "pct":0 | ||
| } | ||
| } | ||
| }, | ||
| "network":{ | ||
| "tx":{ | ||
| "bytes": {{ $txbytes }}, | ||
| "errors":0 | ||
| }, | ||
| "rx":{ | ||
| "bytes": {{ $rxbytes }}, | ||
| "errors":0 | ||
| } | ||
| } | ||
| }, | ||
| "namespace":"demo-{{ $rangeofid }}", | ||
| "namespace_uid":"demo-{{ $rangeofid }}", | ||
| "replicaset":{ | ||
| "name":"demo-deployment-{{ $rangeofid }}" | ||
| }, | ||
| "namespace_labels":{ | ||
| "kubernetes_io/metadata_name":"demo-{{ $rangeofid }}" | ||
| }, | ||
| "labels":{ | ||
| "app":"demo", | ||
| "pod-template-hash":"{{ $rangeofid }}", | ||
| "app-2":"demo-2", | ||
| "app-1":"demo-1" | ||
| }, | ||
| "deployment":{ | ||
| "name":"demo-deployment-{{ $rangeofid }}" | ||
| } | ||
| }, | ||
| "cloud": { | ||
| "provider": "gcp", | ||
| "availability_zone": "europe-west1-d", | ||
| "instance":{ | ||
| "name": "{{ $agentName }}" , | ||
| "id": "{{ $agentId }}" | ||
| }, | ||
| "machine":{ | ||
| "type":"e2-standard-4" | ||
| }, | ||
| "service":{ | ||
| "name":"GCE" | ||
| }, | ||
| "project":{ | ||
| "id":"elastic-obs-integrations-dev" | ||
| }, | ||
| "account":{ | ||
| "id":"elastic-obs-integrations-dev" | ||
| } | ||
| }, | ||
| "orchestrator":{ | ||
| "cluster":{ | ||
| "name":"kubernetes-scale", | ||
| "url":"https://{{generate `Ip`}}" | ||
| } | ||
| }, | ||
| "service":{ | ||
| "address": "https://{{ $agentName }}:10250/stats/summary", | ||
| "type":"kubernetes" | ||
| }, | ||
| "data_stream":{ | ||
| "namespace":"default", | ||
| "type":"metrics", | ||
| "dataset":"kubernetes.pod" | ||
| }, | ||
| "ecs": { | ||
| "version": "8.2.0" | ||
| }, | ||
| "agent": { | ||
| "id": "{{ $agentId}}", | ||
| "name": "{{ $agentName }}" , | ||
| "type": "metricbeat", | ||
| "version": "{{ $agentVersion }}", | ||
| "ephemeral_id": "{{ $agentEphemeralid }}" | ||
| }, | ||
| "elastic_agent": { | ||
| "id": "{{ $agentId }}" , | ||
| "version": "{{ $agentVersion }}", | ||
| "snapshot": {{ generate `agent.snapshot` }} | ||
| }, | ||
| "metricset":{ | ||
| "period": "{{ $period }}" , | ||
| "name":"pod" | ||
| }, | ||
| "event":{ | ||
| "duration": "{{generate "event.duration"}}", | ||
| "agent_id_status": "verified", | ||
| "module":"kubernetes", | ||
| "dataset":"kubernetes.pod" | ||
| }, | ||
| "host":{ | ||
| "hostname":"host-{{ $nodeid }}", | ||
| "os":{ | ||
| "kernel":"5.10.161+", | ||
| "codename":"focal", | ||
| "name":"Ubuntu", | ||
| "type":"linux", | ||
| "family":"debian", | ||
| "version":"20.04.5 LTS (Focal Fossa)", | ||
| "platform":"ubuntu" | ||
| }, | ||
| "containerized": false, | ||
| "name": "host-{{ $nodeid }}", | ||
| "id": "host-{{ $nodeid }}", | ||
| "architecture":"x86_64" | ||
| } | ||
| } | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me understand a thing. If you run the tool to generate let's say 1000 events, then there will be 90 different node names in these events. But there will be 1000? different node uids ? So the the same nodes(with same name) will have a different uid. Is this assumption correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I copied the assets from @gizas elastic/elastic-integration-corpus-generator-tool#111
but yes, according to the config this will be correct.
if need to have 90
$nodeIdand 90$uIdand they always must match we had to addcardinality: 90to both of themThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cardinality doesn't it mean that they will just be 90 different ? How can we know that they will match? Or we could directly link the name with the uid by setting the name:host-{{uid}}.
The reason I am mentioning this, is that if the uid and name don't match it cannot actually be a realistic test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does it affect the test if we just remove one of the fields to avoid this scenario?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but i agree, we are going to come up against these types of 'correlated' fields all the time. we should probably find a way to deal with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They will match because cardinal values are generated sequentially and reiterated at the same time:
since
uidis a call to a function, we cannot apply cardinality indeed (it's not a field defined basically, sorry for the initial suggestion we could apply it).we could still have
name:host-{{uid}}but we will end up with as many host as number of events generated.uidmust a uid, must it? (sorry for the dump question).we could make it a field in the format
word1-word2-word3-word4and makenameandhostnamebe a "link" to it.having
uidas a field we could apply a cardinalityThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Late to the game but trying to catch up.
Summarising to the above:
1)For now the number of nodes did not affect the test.
2) The assumption we made is that the specific test will produce a result of around 10.000 unique pod names and cosidering that we have the limit of 110 pods per node, that is why we resulted to 90 nodes. (See https://github.com/elastic/elastic-integration-corpus-generator-tool/pull/111/files#diff-d34fb1cf4866915a96be5b7ef33896bfecb1d98d1571dc32204396cc8aef5255R38)
So I would say to keep it like that for now
3) We have the same problem with pod.name and pod.uid. We need to correlate specific pod names with pod.uids and the problem there afffects the tests because uids are dimensions https://github.com/elastic/rally-tracks/blob/master/tsdb_k8s_queries/pod-template.json#L663. I will update pod.name to be the a "link" to pod.uid, is simpler and should work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am running some more tests and will update the uids shortly.
To add one above:
See below block that will produce 10000 pod names (based on rangeofid) and will distribute to 90 nodes (based on limit we have that 1node has 110 maximum pods)
Then the key is that we produce the result with
gotext -t 8640000, which is the number of events when we scrape every 10 sec.See info
So the ideal would be based on the number of events (and the user gives eg number of pods to calculate number of nodes)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aspacca and @gizas I believe the having each generated pod event having a unique node uid does not make sense. We need to apply cardinality. I don't know why uid needs to be a real uid in these tests.
we could make it a field in the format word1-word2-word3-word4 and make name and hostname be a "link" to it.This makes sense to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aspacca , @MichaelKatsoulis here is my update template where I removed nodeid and just i divide the rangeofid of pod with 110. This gives me a fix node id for specific group of pods
elastic/elastic-integration-corpus-generator-tool@fe73f69
This way 0-109 pods are inside node-0
110-219 node-1 and goes one!