Fixed bug resulting from prefix strings of less than 3 characters when creating temporary files in GermlineCNVCaller and improved documentation of corresponding utility methods.

[samuelklee]

Closes #7410.

[samuelklee]

@mwalker174 super quick one for you. Didn't think it was worth adding a regression test at this point (since this would involve editing sample names in some of the test resources/models), but feel free to file an issue if you think it might be worth addressing later.

[mwalker174]

Thanks @samuelklee, looks good! It would be nice to have methods calling File.createTempFile() automatically amend the filename if the prefix is too short, but that also has edge cases that consuming tools could get into trouble with… I think the parameter doc updates are sufficient here.

[ruqianl]

Thanks for fixing this! When is the fix expected to be included in the docker image?

[samuelklee]

No problem @ruqianl, thanks again for bringing it to our attention! The fix will be included in the official Docker image in the next release; however, I'm not sure if that will happen imminently, as the last release occurred just 18 days ago and there have only been 10 additional commits in the meantime. @droazen may have a better timeline for you.

I believe there are some nightly Docker builds at https://hub.docker.com/r/broadinstitute/gatk-nightly/, so give it enough time and this fix should show up there (if it hasn't already).

[ruqianl]

Thanks, @samuelklee I'd try out the nightly builds.

I have got another question about running multiple shards in parallel by submitting multiple slurm jobs each running GermlineCNVCaller with one shard. That seems to trigger this (INFO (theano.gof.compilelock).

INFO (theano.gof.compilelock): Waiting for existing lock by process '62379' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by process '3632551' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by unknown process (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by process '62379' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by process '3633423' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by process '3633419' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by process '3633423' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir INFO (theano.gof.compilelock): Waiting for existing lock by process '3633419' (I am process '61988') INFO (theano.gof.compilelock): To manually release the lock, delete /mnt/beegfs/mccarthy/scratch/general/rlyu/Projects/Snakemake_projects/yeln_2019_spermtyping/.theano/compiledir_Linux-4.18-el8_4.x86_64-x86_64-with-debian-buster-sid-x86_64-3.6.10-64/lock_dir

And there isn't a lock_dir under the specified path, but there are a bunch of tmp dirs like

tmpxmako3mg tmpxqfosn7p tmpxsqfj_t1

Would you recommend deleting these folders to release the compiler lock?

[samuelklee]

Thanks @ruqianl, you may want to read through the comments at #6235 and the corresponding PR #6244, which both address this issue. See also the following bit of documentation added in that PR:

Advanced users may wish to set the THEANO_FLAGS environment variable to override the GATK theano configuration. For example, by running THEANO_FLAGS="base_compiledir=PATH/TO/BASE_COMPILEDIR" gatk GermlineCNVCaller ..., users can specify the theano compilation directory (which is set to $HOME/.theano by default). See theano documentation at https://theano-pymc.readthedocs.io/en/latest/library/config.html.

So you can specify a unique compilation directory for each of your jobs to avoid the compilelock, e.g., THEANO_FLAGS="base_compiledir=PATH/TO/BASE_COMPILEDIR/FOR/JOB/0" gatk GermlineCNVCaller ..., THEANO_FLAGS="base_compiledir=PATH/TO/BASE_COMPILEDIR/FOR/JOB/1" gatk GermlineCNVCaller ..., etc.

Alternatively, you can increase config.compile.timeout as discussed in those comments.

[ruqianl]

Great! Thanks for the links! :)