Skip to content

A thrift_client_retry_interval_ms config should be added in be to avoid avalanche disaster in fe thrift server #4023

@caiconghui

Description

@caiconghui

Although we have max_running_txn_num_per_db config to prevent excessive system load in fe, but frequent retry may also cause excessive system load in fe, in our production environment, we see many logs like
retrying call frontend service, address=TNetworkAddress(hostname=10.118.32.48, port=9020), reason=THRIFT_EAGAIN (timed out) within a short time, and find high system load in fe, some times cause fe core, the main reason is that there are already many backlogs in fe, which cause be thrift client rpc timeout, but be thrift client retry immediately without any breaks, which cause things more worse, finally make avalanche disaster in fe.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions