[BUG] Request with JSON-RPC: Connection timed out

[2-towns]

Seen in testnet running v0.2.0.

There are several error logs Failed to send POST Request with JSON-RPC: Connection timed out showing up in the Kibana dashboard.

Here is the a full record:

{
  "@timestamp": [
    "2025-02-28T12:40:20.414Z"
  ],
  "container_image": [
    "codexstorage/nim-codex:0.2.0-dist-tests"
  ],
  "container_image.keyword": [
    "codexstorage/nim-codex:0.2.0-dist-tests"
  ],
  "file": [
    "/var/log/pods/codex_codex-4-1_2ac2c56a-6ad5-4298-9dca-298a98936b1e/codex/0.log"
  ],
  "file.keyword": [
    "/var/log/pods/codex_codex-4-1_2ac2c56a-6ad5-4298-9dca-298a98936b1e/codex/0.log"
  ],
  "hostname": [
    "vector-dh9qw"
  ],
  "hostname.keyword": [
    "vector-dh9qw"
  ],
  "log_counter": [
    0
  ],
  "log_level": [
    "ERR"
  ],
  "log_level.keyword": [
    "ERR"
  ],
  "log_timestamp": [
    "2025-02-28T12:40:20.412Z"
  ],
  "log_topic": [
    "marketplace sales initial-proving"
  ],
  "log_topic.keyword": [
    "marketplace sales initial-proving"
  ],
  "message": [
    "ERR 2025-02-28 12:40:20.412+00:00 Error during SaleInitialProving.run        topics=\"marketplace sales initial-proving\" tid=1 error=\"Failed to send POST Request with JSON-RPC: Connection timed out\" count=2400062"
  ],
  "message.keyword": [
    "ERR 2025-02-28 12:40:20.412+00:00 Error during SaleInitialProving.run        topics=\"marketplace sales initial-proving\" tid=1 error=\"Failed to send POST Request with JSON-RPC: Connection timed out\" count=2400062"
  ],
  "pod_ip": [
    "10.244.2.18"
  ],
  "pod_ip.keyword": [
    "10.244.2.18"
  ],
  "pod_name": [
    "codex-4-1"
  ],
  "pod_name.keyword": [
    "codex-4-1"
  ],
  "pod_node_name": [
    "codex-s-4vcpu-8gb-jmy2d"
  ],
  "pod_node_name.keyword": [
    "codex-s-4vcpu-8gb-jmy2d"
  ],
  "_id": "AMySTJUBq12jZ1xPiIHT",
  "_index": "codex-storage-2025.02.28",
  "_score": null
}

Mentioning @markspanbroek due to previous issues with the subscriptions.

[2-towns]

Reproduced in my local node using Testnet

ERR 2025-03-06 13:46:28.047+01:00 There was an error in SlotFilled subscription topics="marketplace onchain market" tid=558574 msg="HTTP polling: There was an exception while getting subscription changes: Failed to send POST Request with JSON-RPC: Connection timed out"

[2-towns]

@veaceslavdoina any clue on this ?

[veaceslavdoina]

This could be because node is overloaded but on the chart it is probably not reflected

[veaceslavdoina]

I see a lot of 499 from Nginx(sidecar) which is in front of the Public Geth - so, Codex close connection prematurely and as I know, we do not have any options to set a custom timeout.

However, $request_time=0.000 is very suspicious

Codex --> rpc.testnet.codex.storage --> Nginx --> Nginx (sidecar) --> Geth RPC

[veaceslavdoina]

And to mention - just one 502 for last 30 days, in February

[dryajov]

I'm seeing this locally as well, while debugging the issues with the #1239, lots of timeout errors coming from the json rpc.

[veaceslavdoina]

I'm seeing this locally as well, while debugging the issues with the #1239, lots of timeout errors coming from the json rpc.

That is suspicious...

Or local Hardhat is not able to handle Codex connections or in bot cases (in-cluster Geth and local Hardhat) we have same/similar issue.

[gmega]

This is in the backburner until work on #808 is done, apparently @markspanbroek suspects this might be caused by the prover blocking things and causing requests to timeout. If this persists after that, it has to be looked into.