Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: This PR is a revamp of the work done for DRILL-5270 (PR #1250 and #1654). Those PRs were intended to improve the profile loading time for the WebUI, but did not address the fundamental problem of having too many profiles in the profiles directory.
When Drill is displaying profiles stored on the file system (Local or Distributed), it does so by loading the entire list of .sys.drill files in the profile directory, sorting and deserializing. This can get expensive, since only a single CPU thread does this.
As an example, a directory of 120K profiles, the time to just fetch the list of files alone is about 6 seconds. After that, based on the number of profiles being rendered, the time varies. An average of 30ms is needed to deserialize a standard profile, which translates to an additional 3sec for the rendering of default 100 profiles.
A user reported issue confirms just that:
DRILL-5028 Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS
Additional JIRAs filed ask for managing these profiles
DRILL-2362 Drill should manage Query Profiling archiving
DRILL-2861 enhance drill profile file management
This PR brings the following enhancements to ensure that profiles are better managed.
<profileDir>/yyyy/mm/dd
Reference: https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/user/UserWorker.java#L67
e.g. moving from
yyyy/mm/dd
toyyyy/mm/dd/hh
diorama
, also exists in the root of the profile directory. The purpose of this is to allow users to dump external profiles that can be then rendered and visualized in the WebUI. Currently, the profile needs to be dumped manually and cannot be done via the WebUI. This could be a future enhancement,