sidechain worker crashed because rpc communication to parentchain failed

[brenzi]

2025-01-15 07:26:16.662	
[2025-01-15T06:26:16.661Z INFO  integritee_service::main_impl] [TargetB] Received finalized header update (7996350), syncing parent chain...
2025-01-15 07:26:16.678	
[2025-01-15T06:26:16.677Z INFO  integritee_service::parentchain_handler] [TargetB] Syncing blocks from 7996349 to 7996350
2025-01-15 07:26:16.738	
[2025-01-15T06:26:16.738Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:17.058	
[2025-01-15T06:26:17.058Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:20.222	
[2025-01-15T06:26:20.222Z ERROR integritee_service::ocall_bridge::ffi::worker_request] Worker request failed: NodeApiFactory(FailedToCreateNodeApi(RpcClient(MaxConnectionAttemptsExceeded)))
2025-01-15 07:26:20.222	
[2025-01-15T06:26:20.222Z ERROR integritee_service::sidechain_setup] Sgx(SGX_ERROR_UNEXPECTED)
2025-01-15 07:26:20.236	
[2025-01-15T06:26:20.235Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:20.881	
[2025-01-15T06:26:20.880Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:21.738	
[2025-01-15T06:26:21.738Z ERROR integritee_service::prometheus_metrics] Failed to update integritee wallet metrics: ApiClient(RpcClient(MaxConnectionAttemptsExceeded))
2025-01-15 07:26:21.766	
[2025-01-15T06:26:21.766Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:22.060	
[2025-01-15T06:26:22.060Z ERROR integritee_service::main_impl] [Integritee] parentchain block syncing terminated with a failure: ApiClient(RpcClient(MaxConnectionAttemptsExceeded))
2025-01-15 07:26:22.060	
[!] [Integritee] parentchain block syncing has terminated
2025-01-15 07:26:24.919	
[L1Event:TargetB] BalanceTransfer { from: 6d6f646c506f745374616b650000000000000000000000000000000000000000 (5EYCAe5c...), to: 6ca8fe9e98a7d7fb4269fe93c638a2e388c6085e74c18bc220c125fd7f0b1b68 (5EXBEKy7...), amount: 21371232 }
2025-01-15 07:26:25.092	
[L1Event:TargetB] BalanceTransfer { from: 6d6f646c506f745374616b650000000000000000000000000000000000000000 (5EYCAe5c...), to: 6ca8fe9e98a7d7fb4269fe93c638a2e388c6085e74c18bc220c125fd7f0b1b68 (5EXBEKy7...), amount: 21371232 }
2025-01-15 07:26:25.248	
[2025-01-15T06:26:25.248Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:25.893	
[2025-01-15T06:26:25.892Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:26.779	
[2025-01-15T06:26:26.778Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:30.261	
[2025-01-15T06:26:30.261Z WARN  substrate_api_client::rpc::tungstenite_client::client] Connection attempt failed due to Io(Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })
2025-01-15 07:26:30.893	
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RpcClient(MaxConnectionAttemptsExceeded)', /home/ubuntu/work/worker/app-libs/parentchain-interface/src/event_subscriber.rs:45:52
2025-01-15 07:26:30.893	
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[brenzi]

The crash occurred both on the Paseo (machine: integritee-1) and AssetHubPolkadot (machine: integritee-3) shards at the same time.

• node-exporter reported metrics without interruption, so the nodes were alright

[brenzi]

the same happened again today at 7:30 on all shards simultaneously (same time of day as last time). I guess this is an issue with our kube cluster?

again: no gap in node-exporter metrics

dmesg on machines shows nothing suspicious