Subscribe Very Slow / Hanging [Rust & Python]

[hub-il]

Subscribing to a stream is very very to return a result. Sometimes it will return in 45 seconds and sometimes it returns nothing after 3 hours. I'm running Solana 2.0.18 and Yellowstone v3.0.1+solana.2.0.18.

How fast should blocks be returned and what could be causing such a significant delay?

Example can run for hours and return nothing:

./client-ubuntu-22.04 -e http://127.0.0.1:10000 subscribe --blocks

Responses not requiring a subscription return instantly:

./client-ubuntu-22.04 -e http://127.0.0.1:10000 get-block-height
[2024-12-11T18:02:03Z INFO  client] Connected
[2024-12-11T18:02:03Z INFO  client] response: GetBlockHeightResponse { block_height: 285119088 }

Yellowstone Config:

{
  "libpath": "/nodedata/solana_data/yellowstone-grpc-geyser-release_v3.0.1+solana.2.0.18/lib/libyellowstone_grpc_geyser.so",
  "log": {
    "level": "info"
  },
  "grpc": {
    "address": "127.0.0.1:10000",
    "max_decoding_message_size": "100_000_000",
    "snapshot_plugin_channel_capacity": null,
    "snapshot_client_channel_capacity": "50_000_000",
    "channel_capacity": "100_000",
    "unary_concurrency_limit": 100,
    "unary_disabled": false
  },
  "prometheus": {
    "address": "0.0.0.0:8999"
  }
}

[Juanito87]

What's the latency between producer and consumer? do you have enough bandwith for your filters?

[hub-il]

@Juanito87 thanks for the reply. Both producer and consumer are on the same machine if that's what you mean. Client is pointing to 127.0.0.1.

As far as bandwidth for your filters I'm not sure what you mean. Are you talking about IO bandwidth, network bandwidth, CPU, RAM? Network, RAM, and IO should all be good. CPU shows 32 cores in HTOP.

Using the rust client my request looks just like this ./client-ubuntu-22.04 -e http://127.0.0.1:10000 subscribe --blocks

Using Python my request looks like this:

blocks_filter = geyser_pb2.SubscribeRequestFilterBlocks(include_transactions=True, include_accounts=True, include_entries=True)
            stream_request = [geyser_pb2.SubscribeRequest(
                slots=[],
                accounts={},
                transactions={},
                transactions_status={},
                entry={},
                blocks={"client": blocks_filter},
                blocks_meta={},
                commitment=CommitmentLevel.FINALIZED,
                accounts_data_slice=[],
                ping=None
            )]

[hub-il]

@Juanito87 I have been able to get some data to stream but not consistently. The machine I am using now has 64 cores and 512 GB of ram. The cores are AMD 2.5GZ boosting to 3.0GZ. Local NVME drives stripped and separate drives for accounts and the ledger.

I am having the same issues with Websockets, the Rust client, and Python. Seems like others are running with these configs just fine. I just don't know what the issue could be.

My agave-validator config looks like this:

agave-validator \
--identity /nodedata/solana_data/validator-keypair.json \
--log - \
--ledger /nodedata/solana_data/ledger_data \
--accounts /nodedata/solana_data/accounts_data \
--entrypoint entrypoint.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
--known-validator Certusm1sa411sMpV9FPqU5dXAYhmmhygvxJ23S6hJ24 \
--known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
--known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
--known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
--expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
--expected-shred-version 50093 \
--only-known-rpc \
--full-rpc-api \
--no-voting \
--no-wait-for-vote-to-start-leader \
--private-rpc \
--enable-cpi-and-log-storage \
--no-skip-initial-accounts-db-clean \
--dynamic-port-range 8100-8200 \
--rpc-bind-address 0.0.0.0 \
--rpc-port 7699 \
--no-port-check \
--enable-rpc-transaction-history \
--wal-recovery-mode skip_any_corrupted_record \
--use-snapshot-archives-at-startup when-newest \
--limit-ledger-size \
--rpc-pubsub-enable-block-subscription \
--enable-extended-tx-metadata-storage \
--geyser-plugin-config /nodedata/solana_data/yellowstone_grpc_config_latest.json

And my Yellowstone config looks like this:

{
    "libpath": "/nodedata/solana_data/yellowstone-grpc-geyser-release_v3.0.1+solana.2.0.18/lib/libyellowstone_grpc_geyser.so",
    "log": {
        "level": "info"
    },
    "grpc": {
        "address": "0.0.0.0:7505",
        "compression": {
        "accept": [
            "gzip"
        ],
        "send": [
            "gzip"
        ]
        },
        "max_decoding_message_size": "50_000_000",
        "snapshot_plugin_channel_capacity": null,
        "snapshot_client_channel_capacity": "50_000_000",
        "channel_capacity": "100_000",
        "unary_concurrency_limit": 100,
        "unary_disabled": false
    },
    "prometheus": {
        "address": "0.0.0.0:8999"
    }
}