Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unexptected TCP RST from azurite side #2053

Open
Slach opened this issue Jul 13, 2023 · 3 comments · May be fixed by #2454
Open

unexptected TCP RST from azurite side #2053

Slach opened this issue Jul 13, 2023 · 3 comments · May be fixed by #2454
Assignees

Comments

@Slach
Copy link

Slach commented Jul 13, 2023

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

latest docker image

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

I created reproducible docker-compose
https://gist.github.com/Slach/043c85f759d4422a75fceab6dccb1f8d

What problem was encountered?

TCP RST after make client requests
image
image

Steps to reproduce the issue?

git clone https://gist.github.com/Slach/043c85f759d4422a75fceab6dccb1f8d
./reproduce.sh

Have you found a mitigation/solution?

No

also look details ClickHouse/ClickHouse#52088

Slach added a commit to Altinity/clickhouse-backup that referenced this issue Jul 13, 2023
…ckup_restore: true`, it works, but slow, look ClickHouse/ClickHouse#52088 and Azure/Azurite#2053 for details

- clean Dockerfile for build-reace-docker and build-race-fips-docker parameters
@blueww blueww self-assigned this Jul 17, 2023
@blueww
Copy link
Member

blueww commented Jul 17, 2023

@Slach
Thanks for raising this issue!

Would you please share the Azurite debug log for this issue?
And please point out in which step of reproduce.sh does this issue repro?

Besides that, just to confirm this kind of issue not happen on product Azure?

@parshimers
Copy link

I've experienced this issue as well in a different context. The debug log shows nothing, because the connection is immediately terminated by the RST. It happens immediately after the client connects. I can't reproduce it on MacOS but it happens easily on Linux.

@blueww
Copy link
Member

blueww commented Aug 16, 2023

@parshimers

From the above wireshark screenshot, it looks the RST is expected, since:
There are 4 requests in the screenshot, the first request from Azurite to client has flag "FIN", means the connection is being torn down, and the 2nd request send from client to Azurite has flag "ACK" means agree to terminate the connection, so the 4th request is RST.

The problem looks is on why the 1st request send FIN. But we don't see the whole connection in the screenshot, so can't get the reason. Could you please send the wireshark trace file of the whole request (please use test data, don't include any confidential data in the network trace.)

Besides that, since you can't repro the issue in MacOS, do you use the same Azurite docker image version on linux and macOS? (Docker image with version latest might be cached in local. IF so you might can clearup the local cached Azurite latest docker image, and try again to make sure you use latest docker image.)

As you can't repro on MacOS but can repro on Linux (I assume you use same docker image), it look not be an Azurite code issue, but something related with OS or test environment.

Besides that, does the issue happen randomly, how frequently? If you add some retry logic on storage client side, can it workaround this issue?

Slach added a commit to Altinity/clickhouse-backup that referenced this issue Feb 26, 2024
Slach added a commit to Slach/Azurite that referenced this issue Aug 1, 2024
Slach added a commit to Slach/Azurite that referenced this issue Aug 2, 2024
Slach added a commit to Slach/Azurite that referenced this issue Aug 26, 2024
…t for fix Azure#2053 and workaround ClickHouse/ClickHouse#60447

Signed-off-by: Slach <bloodjazman@gmail.com>
Slach added a commit to Slach/Azurite that referenced this issue Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants