DRILL-5270: Improve loading of profiles listing in the WebUI #1654
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: Closed the old PR #755 and #1250 and opening this.
When Drill is displaying profiles stored on the file system (Local or Distributed), it does so by loading the entire list of
.sys.drill
files in the profile directory, sorting and deserializing. This can get expensive, since only a single CPU thread does this.As an example, a directory of 120K profiles, the time to just fetch the list of files alone is about 6 seconds. After that, based on the number of profiles being rendered, the time varies. An average of 30ms is needed to deserialize a standard profile, which translates to an additional 3sec for the rendering of default 100 profiles.
A user reported issue confirms just that:
DRILL-5028 Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS
Additional JIRAs filed ask for managing these profiles
DRILL-2362 Drill should manage Query Profiling archiving
DRILL-2861 enhance drill profile file management
This PR brings the following enhancements to achieve that:
Reload & reconstruction of the profiles in the Tree is done in the event of any of the following states changing:
i. Modification Time of profile dir
ii. Number of profiles in the profile dir
iii. Number of profiles requested exceeds existing the currently available list
Access to the profiles requires a lock to ensure that at any time, only one of the requesting users is able to trigger a reconstruction of the TreeSet. If reconstruction is not required, it uses the existing cache rather than attempting to read the entire profile directory's contents all over again.