Skip to content

Commit

Permalink
Fix Reduce task command runner script size [ci fast]
Browse files Browse the repository at this point in the history
This commit mitigate the problem of having a too big
command runner script, skipping the adding of input
files deletion commands in the script itself, when
the number of input files is greater than or equals 100.

Solve #2118
  • Loading branch information
pditommaso committed Mar 6, 2022
1 parent 1a91e73 commit 6930408
Showing 1 changed file with 12 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -114,14 +114,24 @@ class SimpleFileCopyStrategy implements ScriptFileCopyStrategy {
String getStageInputFilesScript(Map<String,Path> inputFiles) {
assert inputFiles != null

def len = inputFiles.size()
def delete = []
def links = []
for( Map.Entry<String,Path> entry : inputFiles ) {
final stageName = entry.key
final storePath = entry.value

// delete all previous files with the same name
delete << "rm -f ${Escape.path(stageName)}"
// Delete all previous files with the same name
// Note: the file deletion is only needed to prevent
// file name collisions when re-running the runner script
// for debugging purpose. However, this can cause the creation
// of a very big runner script when a large number of files is
// given due to the file name duplication. Therefore the rationale
// here is to keep the deletion only when a file input number is
// given (which is more likely during pipeline development) and
// drop in any case when they are more than 100
if( len<100 )
delete << "rm -f ${Escape.path(stageName)}"

// link them
links << stageInputFile( storePath, stageName )
Expand Down

0 comments on commit 6930408

Please sign in to comment.