Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: tests for access log transmission to Clickhouse server #2269

Closed
ai-tmpst opened this issue Oct 28, 2024 · 3 comments
Closed

tests: tests for access log transmission to Clickhouse server #2269

ai-tmpst opened this issue Oct 28, 2024 · 3 comments
Assignees
Milestone

Comments

@ai-tmpst
Copy link
Contributor

ai-tmpst commented Oct 28, 2024

To verify the correct operation of the access log transmission from the Tempesta FW ring buffer, mapped to userspace, to a Clickhouse server via a user-space utility utils/tfw_logger. This test will ensure that access logs are correctly generated by Tempesta FW, and transmitted to the Clickhouse server without data loss or corruption.

Clickhouse has SQL-like query syntax.

Access log table creation:
CREATE TABLE IF NOT EXISTS access_log (timestamp DateTime64, address IPv6, method UInt8, version UInt8, status UInt16, response_content_length UInt32, response_time UInt32, vhost String, uri String, referer String, user_agent String, dropped_events UInt64, PRIMARY KEY(timestamp))

For tests I used Clickhouse client/server downloaded as curl https://clickhouse.com/ | sh. But I think it would be better to install the server from distro repositories.

@krizhanovsky
Copy link
Contributor

krizhanovsky commented Oct 28, 2024

I'll deploy the Clockhouse logs on our website, so there will be manual testing and we can move the task to 0.9.

Things to test:

  • access log with dmesg and clickouse or clickhouse only
  • test different values for mmap_log_buffer_size
  • restart Tempesta FW under load (e.g. h2load or wrk) with clickhouse access log
  • query Clickhouse that all the records are presented and correct (URL, IP addresses, Host, response status and size, timestamps are monotonically increasing and do represent correct current time)
  • send SIGSTOP for the user space daemon, wait a little under load, and send SIGCONT - check that Clickhouse entries represent the number of records, which couldn't be logged
  • with deproxy introduce a small delay on the server side and check that response time correctly represents the delay in this case and no-delay cases

@ai-tmpst
Copy link
Contributor Author

ai-tmpst commented Nov 14, 2024

for i in $(seq 1 30); do kill -SIGSTOP 8384 && sleep 10 && kill -SIGCONT 8384; done
Something like this we need in tests. Now it can lead to incorrect tail position.

@krizhanovsky
Copy link
Contributor

Done in tempesta-tech/tempesta-test#749

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants