Random traffic lock in 9.0.x

[calavera]

I just realized that I never opened an issue about this problem, and we should look into it.

We deployed c4b7d10 in two canary boxes serving full production traffic and bump into an issue where TrafficServer stopped processing any transaction after some time. The time was aleatory, the first occurrence happened in one box after 8 hours, it happened sooner in that box after a restart, it was in less than 4 hours. The second box worked normally for about three days, until it arrived to this deadlock.

We saw this problem because in our graph, all TCP connections were suddenly blocked:

@shinrich mentioned that she was investigating something similar.

[ywkaras]

You don't see anything odd in the logs?

[calavera]

You don't see anything odd in the logs?

nothing with the default logs, all processing stalls, but we'll try to run with debug enabled in one box to see if we get more information.

[randall]

Can you run pstack on the traffic_server process before restarting?

[calavera]

Can you run pstack on the traffic_server process before restarting?

will do

[ywkaras]

Might be worth a try to cherry-pick ywkaras@ecde23c into 9 to make a load to run with debug output enabled.

[shinrich]

PR #6396 may help. But it implies there still is an initial thread that is blocking the others. In our case, we are sometimes seeing luajit get stuck in GC.

[calavera]

PR #6396 may help.

Thanks @shinrich. We can test it as soon as it lands in the 9.0.x branch

[calavera]

We have not had time to investigate this as much as I wanted, but I'm starting to think that this is caused by one of our plugins, and not the core.

I think this should not be considered a blocker for the 9.0.x release.

[zwoop]

@calavera Is this still an issue that needs to be addressed upstream?

[calavera]

We have not checked since May, so I don't know. We're working on being able to test again, and if we don't find this problem anymore, I'll close the issue.

[calavera]

Closing this because we've not seen it in new tests.