frontendproxy- error initializing config ' envoy.yaml'- Address: value length must be at least 1 characters

[ragigeo]

Hello Team,

We are deploying demo app using kubectl apply.
But opentelemetry-demo-frontendproxy giving below error. Attached is the details pod log. The yaml file I am using

I am using
Ubuntu 22.04.5 LTS
k3s version v1.31.4+k3s1

[2025-01-27 08:09:20.509][8][info][main] [source/server/server.cc:432] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory [2025-01-27 08:09:20.519][8][critical][main] [source/server/server.cc:414] error initializing config ' envoy.yaml': Proto constraint validation failed (BootstrapValidationError.StaticResources: embedded message failed validation | caused by StaticResourcesValidationError.Clusters[5]: embedded message failed validation | caused by ClusterValidationError.LoadAssignment: embedded message failed validation | caused by ClusterLoadAssignmentValidationError.Endpoints[0]: embedded message failed validation | caused by LocalityLbEndpointsValidationError.LbEndpoints[0]: embedded message failed validation | caused by LbEndpointValidationError.Endpoint: embedded message failed validation | caused by EndpointValidationError.Address: embedded message failed validation | caused by AddressValidationError.SocketAddress: embedded message failed validation | caused by SocketAddressValidationError.Address: value length must be at least 1 characters):

frontendproxy-pod_logs.txt

There are other pods also not coming up. Are all interrelated ?

NAME READY STATUS RESTARTS AGE opentelemetry-demo-accountingservice-776654b9f-wf5hq 1/1 Running 0 57m opentelemetry-demo-adservice-7698678c9-96x8d 1/1 Running 0 57m opentelemetry-demo-cartservice-6959789c8d-grt6c 1/1 Running 0 57m opentelemetry-demo-checkoutservice-7d44b97f97-nh4f5 1/1 Running 0 57m opentelemetry-demo-currencyservice-5598d9c4f-7t7lz 1/1 Running 0 57m opentelemetry-demo-emailservice-557c447648-f6czt 1/1 Running 0 57m opentelemetry-demo-flagd-7f5c9b7875-qn47m 1/1 Running 0 57m opentelemetry-demo-frauddetectionservice-8bc759b5b-cnp4j 0/1 CrashLoopBackOff 13 (5m9s ago) 57m opentelemetry-demo-frontend-7b4b4c5585-gj56h 1/1 Running 0 57m opentelemetry-demo-frontendproxy-5d99f9d5bb-rkr2x 0/1 CrashLoopBackOff 15 (4m58s ago) 57m opentelemetry-demo-grafana-c4f5d8bb8-rlnxh 1/1 Running 0 57m opentelemetry-demo-grafana-test 0/1 Error 0 56m opentelemetry-demo-imageprovider-7c94b8875c-vgjqc 0/1 CrashLoopBackOff 15 (4m31s ago) 56m opentelemetry-demo-jaeger-7d6cb5dd87-tdshh 1/1 Running 0 57m opentelemetry-demo-kafka-84d55469c9-pz6pd 1/1 Running 0 56m opentelemetry-demo-loadgenerator-c6f76fb4f-p9pzv 1/1 Running 0 56m opentelemetry-demo-otelcol-69dddd9586-8xg2t 1/1 Running 0 57m opentelemetry-demo-paymentservice-65fd87c78f-hn4ld 1/1 Running 0 56m opentelemetry-demo-productcatalogservice-7f58968db4-r8qpf 1/1 Running 0 56m opentelemetry-demo-prometheus-server-5b6995c858-t692x 1/1 Running 0 57m opentelemetry-demo-quoteservice-6c87b7b46d-rfh29 1/1 Running 0 56m opentelemetry-demo-recommendationservice-86d4d9c599-9z4wd 1/1 Running 0 56m opentelemetry-demo-shippingservice-6cf99559fb-nt2bl 1/1 Running 0 56m opentelemetry-demo-valkey-f9cf475b6-tt4fj 1/1 Running 0 56m otel-demo-opensearch-0 1/1 Running 0 56m

[julianocosta89]

@ragigeo are you able to use the latest k8s manifests?
https://github.com/open-telemetry/opentelemetry-demo/blob/main/kubernetes/opentelemetry-demo.yaml

I've just tried it out and most services started fine, I just got an issue with flagd, which I've reported in here: #1983.

BTW, the Helm installation worked fine and I got all services up and running.

[ragigeo]

Thanks @julianocosta89 ,
After taking the latest , My above pro fixed.
But below two pods still not able to come up. Could you please help/suggest. Thanks

opentelemetry-demo-frauddetectionservice-8bc759b5b-rk2r6 0/1 CrashLoopBackOff 15 (110s ago) 55m opentelemetry-demo-imageprovider-7c94b8875c-686hd 0/1 CrashLoopBackOff 15 (2m33s ago) 55m

Logs for imageprovider

`kubectl logs -n otel-demo opentelemetry-demo-imageprovider-7c94b8875c-686hd
load_module modules/ngx_otel_module.so;

events {
worker_connections 1024;
}

http {
otel_exporter {
endpoint opentelemetry-demo-otelcol:4317;
}
otel_trace on;
otel_trace_context propagate;
otel_service_name imageprovider;
otel_span_name imageprovider; include mime.types;
sendfile on;
server {
listen 8081;
listen [::]:8081;resolver 127.0.0.11;
autoindex off; server_name _;
server_tokens off; root /static;
gzip_static on;
}
}
2025/01/31 10:48:29 [emerg] 1#1: socket() [::]:8081 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8081 failed (97: Address family not supported by protocol)
`

Logs frauddetectionservice

kubectl logs -n otel-demo opentelemetry-demo-frauddetectionservice-8bc759b5b-rk2r6 Defaulted container "frauddetectionservice" out of: frauddetectionservice, wait-for-kafka (init) Picked up JAVA_TOOL_OPTIONS: -javaagent:/app/opentelemetry-javaagent.jar OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended [otel.javaagent 2025-01-31 10:49:14:932 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.4.0 WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. SLF4J(W): No SLF4J providers were found. SLF4J(W): Defaulting to no-operation (NOP) logger implementation SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details. 2025-01-31 10:49:35 - frauddetectionservice - Consumed record with orderId: 51954a30-dfc0-11ef-a5db-2ea051d1adf2, and updated total count to: 1 trace_id=8a4af6574c84343f93709b1bba0c2437 span_id=a71a57366dff98eb trace_flags=01 2025-01-31 10:49:36 - frauddetectionservice - Consumed record with orderId: 6b5a64dc-dfc0-11ef-a5db-2ea051d1adf2, and updated total count to: 2 trace_id=7f4bb5628bf8dcaeeebf481701cfd3f2 span_id=01530f07fd072e29 trace_flags=01 2025-01-31 10:49:36 - frauddetectionservice - Consumed record with orderId: 6e4cb381-dfc0-11ef-a5db-2ea051d1adf2, and updated total count to: 3 trace_id=38e67b82229b2f8be3319f24c6c12bf3 span_id=b8413c929fc3238b trace_flags=01

[julianocosta89]

Fraud detection seems to be working and processing fine (from the logs).

[julianocosta89]

Did you just apply the manifests or have you changed anything?

[ragigeo]

t apply the manifests or have you changed anything?

I have changed below along with updated manifests

otlphttp/example:
endpoint: http://my endpoint ip:4318

For loadgenerator

• name: LOCUST_WEB_HOST
  value: "0.0.0.0"

[julianocosta89]

@ragigeo Could you try to run without any change?

[ragigeo]

Hi @julianocosta89 ,
I ran without any changes and saw loadgenerator, imageprovider, frauddetectionservice for failed to start.
So I have added name: LOCUST_WEB_HOST value: "0.0.0.0" for loadgenerator and it started but. But other two are still not started attached is the logs.
Could you please help thanks

Logs for imageprovider

kubectl logs opentelemetry-demo-imageprovider-7c94b8875c-grdvs -n otel-demo

load_module modules/ngx_otel_module.so;

events {
    worker_connections 1024;
}

http {
    otel_exporter {
        endpoint opentelemetry-demo-otelcol:4317;
    }
    otel_trace on;
    otel_trace_context propagate;
    otel_service_name imageprovider;
    otel_span_name imageprovider;


    include mime.types;
    sendfile on;
    server {
        listen 8081;
        listen [::]:8081;

        resolver 127.0.0.11;
        autoindex off;

        server_name _;
        server_tokens off;

        root /static;
        gzip_static on;
    }
}
2025/02/03 11:12:41 [emerg] 1#1: socket() [::]:8081 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8081 failed (97: Address family not supported by protocol)

Logs for frauddetectionservice

kubectl logs opentelemetry-demo-frauddetectionservice-8bc759b5b-z4j2h -n otel-demo

Defaulted container "frauddetectionservice" out of: frauddetectionservice, wait-for-kafka (init)
Picked up JAVA_TOOL_OPTIONS: -javaagent:/app/opentelemetry-javaagent.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2025-02-03 11:12:44:472 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.4.0
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
2025-02-03 11:12:58 - frauddetectionservice - Consumed record with orderId: 17a9730d-e21f-11ef-a1cf-8614fdc32a90, and updated total count to: 1 trace_id=4e38d6d9a5f0a11e24d044f6dd8f9d48 span_id=5dd9db0379df1e13 trace_flags=01 
2025-02-03 11:12:58 - frauddetectionservice - Consumed record with orderId: 3544e6a9-e21f-11ef-a1cf-8614fdc32a90, and updated total count to: 2 trace_id=9bf7b0e43f0c84137010c75cf4bbc985 span_id=3131544bebb46607 trace_flags=01

[julianocosta89]

I can't reproduce the error on my end, but I googled the error and it seems to be something related to IPv6:
digitalocean/nginxconfig.io#151

Regarding frauddetection though, from the logs it seems it is running fine. We even have orders being processed.