Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: dcm2niix outname #351

Merged
merged 1 commit into from
May 23, 2019
Merged

fix: dcm2niix outname #351

merged 1 commit into from
May 23, 2019

Conversation

mgxd
Copy link
Member

@mgxd mgxd commented May 22, 2019

This changes the dcm2niix outname from an erroneous substring to the heuristic prefix. Specifically, it will change the call from

dcm2niix -b y -f func ...

to

dcm2niix -b y -f /path/to/bids/sub-1/func/sub-1_task-blah_bold ...

@codecov-io
Copy link

codecov-io commented May 22, 2019

Codecov Report

Merging #351 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #351   +/-   ##
=======================================
  Coverage   74.23%   74.23%           
=======================================
  Files          35       35           
  Lines        2682     2682           
=======================================
  Hits         1991     1991           
  Misses        691      691
Impacted Files Coverage Δ
heudiconv/convert.py 79.92% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75a4c82...c3c30b3. Read the comment docs.

@yarikoptic
Copy link
Member

Did you forget y in your example after -b? Just making sure I am not missing something

@yarikoptic
Copy link
Member

This suggests that you will be writing into a directory nearby original file... That one can be read only

What problem are you trying to fix?

@yarikoptic
Copy link
Member

Oh, scrape my comment about original directory... But knowing the problem/reason would help. I would have also tried to look into history of may be commit which changed that line tells why only the basename was considered. I would guess - to not leave junk around if conversion fails etc

@mgxd
Copy link
Member Author

mgxd commented May 23, 2019

@yarikoptic this will accomplish 2 things:

  1. make it clear to users which key is currently being converted
  2. avoid dcm2niix having to add prefixes to outputs with the same name if multiple outfiles are generated
    (-o func could generate func.nii func1.nii etc)

@yarikoptic
Copy link
Member

Re 2. -- sorry I don't get it since I think dcm2niix doesn't add prefixes but suffixes, and does it regardless either you provide it with just relative path or full path or directory for output:

(git-annex)hopa:…mouth-phantoms/bids_test6-PD+T2w[master]005-anat-PD+T2w_acq-tse-tra-3mm
$> dcm2niix -b y -f something . 
Chris Rorden's dcm2niiX version v1.0.20190410  (JP2:OpenJPEG) GCC8.3.0 (64-bit Linux)
Found 70 DICOM file(s)
Warning: interpolated protocol 'anat-PD+T2w_acq-tse-tra-3mm' may be unsuitable for dwidenoise/mrdegibbs. ./000001.dcm
slices not stacked: echo varies (TE 9.4, 94; echo 1, 2). Use 'merge 2D slices' option to force stacking
Convert 35 DICOM as ./something_e1 (640x640x35x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "./something_e1.nii"
Convert 35 DICOM as ./something_e2 (640x640x35x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "./something_e2.nii"
Conversion required 2.147344 seconds (0.196041 for core code).
dcm2niix -b y -f something .  5.91s user 0.17s system 282% cpu 2.153 total
1 11283.....................................:Thu 23 May 2019 11:25:50 AM EDT:.
(git-annex)hopa:…mouth-phantoms/bids_test6-PD+T2w[master]005-anat-PD+T2w_acq-tse-tra-3mm
$> dcm2niix -b y -f /tmp/out/something . 
Chris Rorden's dcm2niiX version v1.0.20190410  (JP2:OpenJPEG) GCC8.3.0 (64-bit Linux)
Found 70 DICOM file(s)
Warning: interpolated protocol 'anat-PD+T2w_acq-tse-tra-3mm' may be unsuitable for dwidenoise/mrdegibbs. ./000001.dcm
slices not stacked: echo varies (TE 9.4, 94; echo 1, 2). Use 'merge 2D slices' option to force stacking
Convert 35 DICOM as ./tmp/out/something_e1 (640x640x35x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "./tmp/out/something_e1.nii"
Convert 35 DICOM as ./tmp/out/something_e2 (640x640x35x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "./tmp/out/something_e2.nii"
Conversion required 1.765559 seconds (0.144816 for core code).
1 11284.....................................:Thu 23 May 2019 11:27:10 AM EDT:.
(git-annex)hopa:…mouth-phantoms/bids_test6-PD+T2w[master]005-anat-PD+T2w_acq-tse-tra-3mm
$> dcm2niix -b y -f /tmp/out/ .         
Chris Rorden's dcm2niiX version v1.0.20190410  (JP2:OpenJPEG) GCC8.3.0 (64-bit Linux)
Found 70 DICOM file(s)
Warning: interpolated protocol 'anat-PD+T2w_acq-tse-tra-3mm' may be unsuitable for dwidenoise/mrdegibbs. ./000001.dcm
slices not stacked: echo varies (TE 9.4, 94; echo 1, 2). Use 'merge 2D slices' option to force stacking
Convert 35 DICOM as ./tmp/out/_e1 (640x640x35x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "./tmp/out/_e1.nii"
Convert 35 DICOM as ./tmp/out/_e2 (640x640x35x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "./tmp/out/_e2.nii"
Conversion required 1.748011 seconds (0.145586 for core code).
dcm2niix -b y -f /tmp/out/ .  5.88s user 0.11s system 341% cpu 1.752 total
1 11285.....................................:Thu 23 May 2019 11:27:40 AM EDT:.
(git-annex)hopa:…mouth-phantoms/bids_test6-PD+T2w[master]005-anat-PD+T2w_acq-tse-tra-3mm
$> dcm2niix --version
Chris Rorden's dcm2niiX version v1.0.20190410  (JP2:OpenJPEG) GCC8.3.0 (64-bit Linux)
 Error: invalid option '--version (null)'
Example output filename: '/func.nii.gz'

what am I missing?

@mgxd
Copy link
Member Author

mgxd commented May 23, 2019

erm by "prefix" I meant suffix, and this change should decrease the amount of collisions. Is there any advantage to having -f anat instead of -f ./full/path/to/output?

And I think the added clarity that 1) gives is enough to make the change.

@yarikoptic
Copy link
Member

Having intermediate results stored in a temporary place, not in the final location/naming helps to

  • avoid junk from crashed runs interfering (we would crash unless --overwrite is given)
  • cleanup by removing stale tmp/ directories

this change should decrease the amount of collisions.

how so? I thought that func is getting generated in the temporary directory. So collisions of what?

@yarikoptic
Copy link
Member

as for 1. -- which is imho indeed valuable, I wonder if we could take "git-annex" approach to urls (when addurl) and some other tools -- make a name out of the path, e.g. prefix.lstrip(os.sep).replace(os.sep, '_'), may be ideally first stripping away the dataset topdir from the prefix first. Then they can see what is being produced and get an idea later on from looking at left over tmp directories what has not completed correctly

@mgxd
Copy link
Member Author

mgxd commented May 23, 2019

Having intermediate results stored in a temporary place, not in the final location/naming

they still would be generated in a temp dir (we're still setting the nipype node's base_dir)

It would just generate a path that emulates the final path, i.e.

# invocation
heudiconv ... -o /code/heudiconv/localconv/multiecho

# current file generated after convertnode.run
/tmp/dcm2niix_5p2gey2/convert/func_e1.nii.gz

# proposed file generated after convertnode.run
/tmp/dcm2niixqk4u57yr/convert/code/heudiconv/localconv/multiecho/sub-ME/func/sub-ME_task-test_run-1_bold_e1.nii.gz

@yarikoptic
Copy link
Member

thanks for the explanation @mgxd -- makes full sense! I was misguided by the full path as prefix I guess thinking it would be the full output path.
I also dug through the history to make sure that original use of basename on dir there to see that may be there was some specific reason noted... but it seems it was not the case in 527f9f3 so this should only make it better ;) Thanks!

@yarikoptic yarikoptic merged commit 312cd2e into nipy:master May 23, 2019
@yarikoptic yarikoptic added this to the 0.6 milestone Nov 22, 2019
yarikoptic added a commit that referenced this pull request Dec 16, 2019
This is largely a bug fix.  Metadata and order of `_key-value` fields in BIDS
could change from the result of converting using previous versions, thus minor
version boost.
14 people contributed to this release -- thanks
[everyone](https://github.com/nipy/heudiconv/graphs/contributors)!

 Enhancement

- Use [etelemetry](https://pypi.org/project/etelemetry) to inform about most
  recent available version of heudiconv. Please set `NO_ET` environment variable
  if you want to disable it ([#369][])
- BIDS:
  - `--bids` flag became an option. It can (optionally) accept `notop` value
    to avoid creation of top level files (`CHANGES`, `dataset_description.json`,
    etc) as a workaround during parallel execution to avoid race conditions etc.
    ([#344][])
  - Generate basic `.json` files with descriptions of the fields for
    `participants.tsv` and `_scans.tsv` files ([#376][])
  - Use `filelock` while writing top level files. Use
    `HEUDICONV_FILELOCK_TIMEOUT` environment to change the default timeout value
    ([#348][])
  - `_PDT2` was added as a suffix for multi-echo (really "multi-modal")
    sequences ([#345][])
- Calls to `dcm2niix` would include full output path to make it easier to
  discern in the logs what file it is working on ([#351][])
- With recent [datalad]() (>= 0.10), created DataLad dataset will use
  `--fake-dates` functionality of DataLad to not leak data conversion dates,
  which might be close to actual data acquisition/patient visit ([#352][])
- Support multi-echo EPI `_phase` data ([#373][] fixes [#368][])
- Log location of a bad .json file to ease troubleshooting ([#379][])
- Add basic pypi classifiers for the package ([#380][])

 Fixed
- Sorting `_scans.tsv` files lacking valid dates field should not cause a crash
  ([#337][])
- Multi-echo files detection based number of echos ([#339][])
- BIDS
  - Use `EchoTimes` from the associated multi-echo files if `EchoNumber` tag is
    missing ([#366][] fixes [#347][])
  - Tolerate empty ContentTime and/or ContentDate in DICOMs ([#372][]) and place
    "n/a" if value is missing ([#390][])
  - Do not crash and store original .json file is "JSON pretification" fails
    ([#342][])
- ReproIn heuristic
  - tollerate WIP prefix on Philips scanners ([#343][])
  - allow for use of `(...)` instead of `{...}` since `{}` are not allowed
    ([#343][])
  - Support pipolar fieldmaps by providing them with `_epi` not `_magnitude`.
    "Loose" BIDS `_key-value` pairs might come now after `_dir-` even if they
    came first before ([#358][] fixes [#357][])
- All heuristics saved under `.heudiconv/` under `heuristic.py` name, to avoid
  discrepancy during reconversion ([#354][] fixes [#353][])
- Do not crash (with TypeError) while trying to sort absent file list ([#360][])
- heudiconv requires nipype >= 1.0.0 ([#364][]) and blacklists `1.2.[12]` ([#375][])

* tag 'v0.6.0': (60 commits)
  Version boost to 0.6.0
  DOC: populate detailed changelog for 0.6.0 and tune up formatting in previous one
  Fix miscellaneous typos in ReproIn heuristic file.
  BF: fix check for the sbatch (SLURM) not being available
  ENH: make test-compare-two-versions take any two worktrees, and just show diff if results already known
  Update heudiconv/convert.py
  apply @mgxd 's suggestions, adding a warning and a timeout environment variable
  need str typecast
  Use empty string not None
  Empty acq_time results in empty cell not 'n/a'
  DOC: Clarify tarball session handling
  remove repetitive import statement
  respond to review - add explicit py2 check - change file saving strategy - use logger instead of print
  fix remaning py2 errors
  MNT: Add Python support metadata to package
  fix some python2/3 incompatibilities
  add return data (accidently removed return)
  make content unicode
  test that load_json provides filename if invalid
  explicitly name invalid json
  ...
yarikoptic added a commit that referenced this pull request Dec 16, 2019
[0.6.0] - 2019-12-16

This is largely a bug fix.  Metadata and order of `_key-value` fields in BIDS
could change from the result of converting using previous versions, thus minor
version boost.
14 people contributed to this release -- thanks
[everyone](https://github.com/nipy/heudiconv/graphs/contributors)!

Enhancement

- Use [etelemetry](https://pypi.org/project/etelemetry) to inform about most
  recent available version of heudiconv. Please set `NO_ET` environment variable
  if you want to disable it ([#369][])
- BIDS:
  - `--bids` flag became an option. It can (optionally) accept `notop` value
    to avoid creation of top level files (`CHANGES`, `dataset_description.json`,
    etc) as a workaround during parallel execution to avoid race conditions etc.
    ([#344][])
  - Generate basic `.json` files with descriptions of the fields for
    `participants.tsv` and `_scans.tsv` files ([#376][])
  - Use `filelock` while writing top level files. Use
    `HEUDICONV_FILELOCK_TIMEOUT` environment to change the default timeout value
    ([#348][])
  - `_PDT2` was added as a suffix for multi-echo (really "multi-modal")
    sequences ([#345][])
- Calls to `dcm2niix` would include full output path to make it easier to
  discern in the logs what file it is working on ([#351][])
- With recent [datalad]() (>= 0.10), created DataLad dataset will use
  `--fake-dates` functionality of DataLad to not leak data conversion dates,
  which might be close to actual data acquisition/patient visit ([#352][])
- Support multi-echo EPI `_phase` data ([#373][] fixes [#368][])
- Log location of a bad .json file to ease troubleshooting ([#379][])
- Add basic pypi classifiers for the package ([#380][])

Fixed

- Sorting `_scans.tsv` files lacking valid dates field should not cause a crash
  ([#337][])
- Multi-echo files detection based number of echos ([#339][])
- BIDS
  - Use `EchoTimes` from the associated multi-echo files if `EchoNumber` tag is
    missing ([#366][] fixes [#347][])
  - Tolerate empty ContentTime and/or ContentDate in DICOMs ([#372][]) and place
    "n/a" if value is missing ([#390][])
  - Do not crash and store original .json file is "JSON pretification" fails
    ([#342][])
- ReproIn heuristic
  - tolerate WIP prefix on Philips scanners ([#343][])
  - allow for use of `(...)` instead of `{...}` since `{}` are not allowed
    ([#343][])
  - Support pipolar fieldmaps by providing them with `_epi` not `_magnitude`.
    "Loose" BIDS `_key-value` pairs might come now after `_dir-` even if they
    came first before ([#358][] fixes [#357][])
- All heuristics saved under `.heudiconv/` under `heuristic.py` name, to avoid
  discrepancy during reconversion ([#354][] fixes [#353][])
- Do not crash (with TypeError) while trying to sort absent file list ([#360][])
- heudiconv requires nipype >= 1.0.0 ([#364][]) and blacklists `1.2.[12]` ([#375][])

* tag 'v0.6.0':
  Boost perspective release date in changelog to today
  ENH(TST): Fix version to older pytest to ease backward compatibility testing
  RF: use tmpdir not tmp_path fixture
  FIX: minor typo in CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants