-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
APPS-2549 Bulk describe optimization in scatter-collect to cover files in arrays, structs #464
Conversation
…ers might not match because ids will be converted to set
@@ -414,7 +418,6 @@ abstract class JobMeta(val workerPaths: DxWorkerPaths, | |||
if (queryFiles.isEmpty) { | |||
queryFiles | |||
} else { | |||
logger.trace(s"Bulk describing ${queryFiles.size} files") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This logging is moved to dxApi (other PR).
} collect { | ||
case dxFile: DxFile => dxFile | ||
} flatMap { | ||
dxApi.flattenDxFileObjectsFromJson |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The part using the new implementation from dxApi
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! In the ticket please:
- Run a WF prior to changes, and provide a splunk stats showing file/describe from dxScala agent. Should be some.
- Run after changes applied showing splunk stats for the same WF (new analysis) and decreased/eliminated dxScala-made file/describe.
done -- APPS-2549 |
Would depend on dxScala change
https://github.com/dnanexus/dxScala/pull/129/files