Skip to content

Conversation

jstirnaman
Copy link
Contributor

@jstirnaman jstirnaman commented Sep 23, 2025

Closes #6403

Adds documentation building on the mentioned Slack discussion:

  • Add runtime architecture documentation covering thread pools, resource allocation, and concurrency model

@jstirnaman jstirnaman force-pushed the influxdb3-runtime-thread-alloc branch from 029c1cb to 68b6bc9 Compare September 23, 2025 21:46
@jstirnaman jstirnaman marked this pull request as ready for review September 24, 2025 15:43
@jstirnaman
Copy link
Contributor Author

jstirnaman commented Sep 24, 2025

Feedback:

After skimming the "performance tuning" page: I noticed that some of the example commands have flags in the wrong places. 3ent is sometimes peculiar about the ordering of flags and commands. For example, --num-io-threads flag only works before the serve command. I suggest breaking the page into a few parts: first part is general principles: must monitor cpu usage (per core), memory, io (network); then what few settings I might use to adjust cpu, threads, and memory; then as a final section the encyclopedic appendix, probably as a separate page. I found it hard to see what information might be useful to me and as an operator while I should understand the full performance tuning docs, usually I'm looking for something more succinct: "my ingesters are only using 2 cores --> increase io_threads in my ingesters --> done". Perhaps the "common performance issues" section should be first in the page. Less is more, and specifics are golden. For example, the docs suggest Increase WAL flush frequency but doesn't say how to do that (it does mention --wal-flush-interval but far in the page from where "Increase WAL flush frequency" is mentioned).

--io-runtime-type [env: INFLUXDB3_IO_RUNTIME_TYPE=] [default: multi-thread]: should be marked as a dev flag. Don't recommend using anything but multi-thread which is the default. The current-thread option will be gone in 3.5 3ent (running with just one thread is probably not a good idea generally anyway)

@jstirnaman jstirnaman marked this pull request as draft September 24, 2025 22:20
…:- IO runtime thread pool (line protocol parsing and validation)- DataFusion runtime thread pool (WAL persistence and queries)

- Recommendations and best practices for monitoring and configuring Core and Enterprise
@jstirnaman jstirnaman force-pushed the influxdb3-runtime-thread-alloc branch from 5df2e43 to f7f85c1 Compare September 29, 2025 20:16
@jstirnaman jstirnaman removed the request for review from philjb September 29, 2025 20:16
@jstirnaman jstirnaman changed the base branch from master to 6403-influxdb3-perf-tuning September 29, 2025 20:17
@jstirnaman jstirnaman marked this pull request as ready for review September 29, 2025 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Runtime usage and thread management recommendations
2 participants