Memory peak when UI is used

[osoykan]

Hi,

I’m using the db-scheduler UI extension and have developed a Ktor plugin for Kotlin based on this library. I haven’t tried the Spring Boot version yet.

In my Kotlin application, I utilize TaskLogic. During load testing with the showData option enabled, I observed that simply refreshing or accessing the /db-scheduler page causes memory usage to peak. I believe this is because db-scheduler eagerly populates a list through a consumer instead of exposing a Flux, as seen in Caching.java (Lines 48-53). Whereas pageSize should be considered before executing the query or limiting the Flux before it is collected.

For example, with 10k scheduled tasks each containing 150KB of data, the UI attempts to retrieve 150MB of data on every refresh and mouse hover unless showData is disabled. Even when showData is turned off, the UI still fetches task names and dates for all 10k items, effectively ignoring the pageSize setting.

Have you encountered this issue before, or is there a configuration option available to modify this behavior?

Thanks!

[geirsagberg]

I have the same issue myself. I suspect db-scheduler must be updated to support paging, alternatively db-scheduler-ui must implement a direct fetch from the databse, though that would be more brittle and DB-specific.

[geirsagberg]

In general, I try to keep the tasks as small as possible, and rather use IDs as data, with each tasks fetching the data it needs. We still run into an issue when we have 3 million tasks scheduled though 😅

[kagkarlsson]

Yeah the SchedulerClient sdk for fetching scheduled executions currently does not support pagination, which may cause some problems for use-cases with lots of executions.

Feel free to file a feature-request for it in db-scheduler 👍