Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net_host_name of etcd metrics has different data formats in ipv4 and ipv6 #15400

Closed
B-Betty opened this issue Mar 3, 2023 · 6 comments
Closed
Assignees
Labels

Comments

@B-Betty
Copy link

B-Betty commented Mar 3, 2023

What happened?

The value of net_host_name in Etcd metrics is the IP in the ipv4 environment, and in the ipv6 environment: ip:2379.
Is this a bug? How to solve it?

ipv4:
etcd_server_is_leader{http_scheme="http", instance="177.177.77.22:8889", job="otel-collector", kubernetes_node_name="node154", net_host_name="10.99.xx.xx", net_host_port="2379", service_instance_id="10.99.xx.xx:2379", service_name="etcd"}

ipv6:
etcd_server_is_leader{http_scheme="http", instance="[fd00:177:177::a68a]:8889", job="otel-collector", kubernetes_node_name="node1", net_host_name="202:202::xx:xx:2379", service_instance_id="202:202::xx:xx:2379", service_name="etcd"}

What did you expect to happen?

Is this a bug? How to solve it?

How can we reproduce it (as minimally and precisely as possible)?

curl http://ip:2379/metrics

Anything else we need to know?

No response

Etcd version (please run commands below)

</details>


### Etcd configuration (command line flags or environment variables)

<details>

# paste your configuration here

</details>


### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

<details>

```console


</details>


### Relevant log output

_No response_
@tjungblu
Copy link
Contributor

tjungblu commented Mar 3, 2023

I might be mistaken, but net_host_name is populated from Prometheus/opentelemetry?

@jmhbnz
Copy link
Member

jmhbnz commented Mar 3, 2023

Agree I think this is upstream of etcd.

Here is the open telemetry conventions ref: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/span-general/#nethostname

Which version of etcd did you encounter this in? I ask as it looks like this issue may have been fixed upstream but we may have been using an older version of otel-go for a while?

For example release-3.5 branch has go.opentelemetry.io/otel v1.0.1 in etcd/server/go.mod which won't have included this upstream fix: open-telemetry/opentelemetry-go@1f4b606.

Refer: https://pkg.go.dev/go.opentelemetry.io/otel?tab=versions

Edit: @ahrtr If I'm on the right track should we backport a version bump for otel dependency? I'm happy to raise if so.

@ahrtr
Copy link
Member

ahrtr commented Mar 6, 2023

For this case, the labels/attributes (e.g. net_host_name) are not populated by etcd itself. Instead, they may be populated by otel, since the job is otel-collector?

We keep bumping dependencies for main (3.6), but not for release-3.5 and release-3.4. In general, I think we should only bump dependencies on demand for both 3.5 and 3.4. Specially we only need to bump dependencies for 3.5/3.4 if we see any issues or CVE. I agree to bump otel for 3.5/3.4 to address this minor issue. WDYT? cc @ptabor @serathius @spzala

@jmhbnz
Copy link
Member

jmhbnz commented May 6, 2023

Hey @B-Betty - I'm working on recreating this now, can you please provide snippets of your prometheus & open telemetry collector configs to confirm how you are consuming the etcd metrics? Thanks!

@stale
Copy link

stale bot commented Aug 12, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

@jmhbnz
Copy link
Member

jmhbnz commented Dec 29, 2023

Closing - This was fixed upstream in OpenTelemetry-Go 1.1.0: https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md#110---2021-10-27

The "go.opentelemetry.io/otel/semconv/v1.4.0".NetAttributesFromHTTPRequest function correctly handles IPv6 addresses as IP addresses

Last month we updated release-3.5 to OpenTelemetry-Go 1.20 via f26074a. This change was included in etcd release v3.5.11.

@jmhbnz jmhbnz closed this as completed Dec 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

4 participants