Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Panicked when table client is initialing #21

Merged
merged 3 commits into from
Mar 10, 2023

Conversation

MichaelLeeHZ
Copy link
Contributor

@MichaelLeeHZ MichaelLeeHZ commented Mar 9, 2023

What changes were proposed in this pull request?

Fix table client init error when one OB server is invalid:

  1. Failure to connection an server is allowed
  2. Panic when all server are invalid

Why are the changes needed?

Table client will init timeout when one OB server is invalid, eg, OUT_OF_MEM, SERVER_IS_INIT.

2023-03-09 15:16:59.749 ERRO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/conn_pool.rs:189] ConnPool::add_connection_background::bg_add fail to build a connection after 3 retries, err:Common error, code:ObException(OB_SERVER_IS_INIT), err:rcode:OB_SERVER_IS_INIT, message:, addr:33.144.36.65:2882, trace_id:YA8D28AC2-0000000000000000, server_trace_id:YB4221902441-0005F66EFC146757
2023-03-09 15:16:59.749 ERRO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/conn_pool.rs:171] ConnPool::add_connection_background::bg_add fail to build connection after 4 retries
2023-03-09 15:16:59.750 INFO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/mod.rs:148] Close write stream for connection to 33.144.36.65:2882
2023-03-09 15:16:59.751 INFO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/mod.rs:333] Connection::process_reading_data read zero bytes, the connection to 33.144.36.65:2882 was closed by remote.
2023-03-09 15:16:59.751 WARN [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/mod.rs:358] Connection::process_reading_data fail to close read stream to 33.144.36.65:2882, err:Transport endpoint is not connected (os error 107)
2023-03-09 15:16:59.751 INFO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/mod.rs:260] Close read stream for connection to 33.144.36.65:2882
2023-03-09 15:16:59.751 ERRO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/conn_pool.rs:189] ConnPool::add_connection_background::bg_add fail to build a connection after 3 retries, err:Common error, code:ObException(OB_SERVER_IS_INIT), err:rcode:OB_SERVER_IS_INIT, message:, addr:33.144.36.65:2882, trace_id:YD603D981-0000000000000000, server_trace_id:YB4221902441-0005F66EFC146759
2023-03-09 15:16:59.751 ERRO [/root/.cargo/git/checkouts/obkv-table-client-rs-4fa64c39e1be7389/211e571/src/rpc/conn_pool.rs:171] ConnPool::add_connection_background::bg_add fail to build connection after 4 retries

2023-03-09 15:04:56.736 ERRO [common_util/src/panic.rs:42] thread 'main' panicked 'Failed to setup analytic engine: OpenObkv { source: InitClient { full_user_name: "obceresdbother:ceresdbother0_4016:ceresdbother", source: Common(ConnPool, "ConnPool::wait_for_initialized create connection timeout_ms:25000"), backtrace: Backtrace(   0: <snafu::backtrace_shim::Backtrace as snafu::GenerateBacktrace>::generate

Will break the compatibility? How if so?

Does this PR introduce any user-facing change?

No

How was this patch tested?

Integration test

Checklist

  • I've run the tests to see all new and existing tests pass.
  • If this Pull Request resolves an issue, I linked to the issue in the text above.
  • I've informed the technical writer about the documentation change if necessary.

@CLAassistant
Copy link

CLAassistant commented Mar 9, 2023

CLA assistant check
All committers have signed the CLA.

@MichaelLeeHZ MichaelLeeHZ force-pushed the fix_table_client_init_err branch from 16f980a to 71e8ad6 Compare March 9, 2023 10:48
@MichaelLeeHZ MichaelLeeHZ force-pushed the fix_table_client_init_err branch from 71e8ad6 to a783329 Compare March 9, 2023 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Continuous access to abnormal ObServer which is restarting
3 participants