Default to using Pytorch APIs for monai.transforms (ETA TBD)

[wyli]

Is your feature request related to a problem? Please describe.
Based on our discussions, this ticket proposes to implement the MONAI transforms preferably using the Pytorch APIs instead of the current mainly numpy-based solutions.

[wyli]

as an initial step, the transform should support both tensor and ndarray types, and return value should have the same type as input.

[rijobro]

I had a thought about this. LoadImage would still return numpy and therefore all transforms that follow would be doing numpy<->torch conversions. So, to make the most of this suggested plan of returning the image in the same type as the input, we would want to move the ToTensor as early as possible, e.g.:

t = Compose([
    LoadImage,
    ToTensor,
    AllOtherTransforms,
    ...
])

Alternatively, we modify LoadImage to return torch.Tensor, but this would be a breaking change.

[wyli]

because this is a significant feature, how about we create a feature branch for it in the codebase? when it's entirely ready we merge it into dev. otherwise we keep the feature branch up-to-date with the dev branch. all of us can collaborate on the feature branch.

[rijobro]

If I do this as a branch on the Project-MONAI fork, will I have to get an approved code review every time I want to commit? If so, I think I'll just do it as a branch on my fork.

[wyli]

that's also fine, just two thoughts:

• from your fork , you wouldn't be able to trigger the full self-hosted ci pipelines (I feel we may need a full test before merging)
• we (at least @Nic-Ma and I) could help build this feature together, submitting PRs to a MONAI branch is easier than submitting them to your fork...

[rijobro]

Ok. Could we create a branch that has CI/CD testing, but doesn't require code reviews? Potentially also doesn't require the tests to pass before pushing to that particular branch (just to speed things up, we'd have to be careful not to break things of course).

[wyli]

sounds great, we should do the same for any major new features in the future...

[rijobro]

Great. I can't create branches on the main repository, could you take care of that?

[wyli]

sure, created feature/2231-transforms (https://github.com/Project-MONAI/MONAI/tree/feature/2231-transforms) I'll push update ci/cd config to include this feature/* pattern

[Spenhouet]

This is really a necessary change. I just wanted to create a discussion on this since I find my self constantly switching with ToTensord and ToNumpyd.

Alternatively, we modify LoadImage to return torch.Tensor, but this would be a breaking change.

I would vote for making this breaking change. MONAI claims to be a "PyTorch-based framework". For all transforms to always expect a torch Tensor and always return a torch Tensor just seems like the right thing to do.
Then all transforms which are currently not torch native would need to convert to numpy array and back.
This would then also help to highlight which transforms need a rewriting with a torch native solution.

Hopefully this makes it into version 0.7.0.

[rijobro]

I've drafted this PR for all transforms, and for the large part, I made the transform ambivalent to torch/numpy input (e.g., img[None] is the same API for both torch and numpy).

Now that I've finished the draft, I've started creating bite-sized PRs to get them into dev.

If you check the big PR, there's also functionality for calculating the number of times a conversion is required. Inevitably, some transforms can only operate on torch.Tensor (e.g., GaussianFilter is a nn.module, and conversely some interpolation uses scikit, requiring input to be numpy). But at least if you knew you had two numpy transforms, you could group them together to minimise conversion.

[rijobro]

And regardless of whether the udpated LoadImage defaults to numpy (backwards compatible) or torch (since more transforms will be torch-based), we'll give the user the option to return the opposite (e.g., LoadImage(..., as_tensor=True)), so if you prefer torch by default, that will be possible.

[Spenhouet]

@rijobro I get the decision to provide the option to group transforms based on numpy or torch. Is there some long term goal of reimplementing and deprecating all numpy based transforms to torch based transforms?

[rijobro]

I don't see any reason to deprecate them – if possible, we might as well try and support both. Anyway, most will be torch-based. Here's the list from the big PR, the only two numpy transforms are RandHistogramShift (uses np.interp) and Orientation (uses nibabel):

TorchOrNumpy:  Activations
TorchOrNumpy:  AddChannel
TorchOrNumpy:  AddCoordinateChannels
Torch:         AddExtremePointsChannel
TorchOrNumpy:  AdjustContrast
Torch:         Affine
Torch:         AffineGrid
TorchOrNumpy:  AsChannelFirst
TorchOrNumpy:  AsChannelLast
TorchOrNumpy:  AsDiscrete
Uncategorised: BatchInverseTransform
TorchOrNumpy:  BorderPad
TorchOrNumpy:  BoundingRect
TorchOrNumpy:  CastToType
TorchOrNumpy:  CenterScaleCrop
TorchOrNumpy:  CenterSpatialCrop
TorchOrNumpy:  ClassesToIndices
TorchOrNumpy:  ConvertToMultiChannelBasedOnBratsClasses
TorchOrNumpy:  CropForeground
TorchOrNumpy:  DataStats
Torch:         DetectEnvelope
TorchOrNumpy:  DivisiblePad
Torch:         EnsureChannelFirst
Uncategorised: EnsureType
TorchOrNumpy:  FgBgToIndices
Torch:         Flip
Torch:         GaussianSharpen
Torch:         GaussianSmooth
TorchOrNumpy:  GibbsNoise
TorchOrNumpy:  Identity
TorchOrNumpy:  KSpaceSpikeNoise
Torch:         KeepLargestConnectedComponent
Torch:         LabelToContour
TorchOrNumpy:  LabelToMask
TorchOrNumpy:  MaskIntensity
TorchOrNumpy:  MeanEnsemble
TorchOrNumpy:  NormalizeIntensity
Numpy:         Orientation
TorchOrNumpy:  Pad
TorchOrNumpy:  PadListDataCollate
Torch:         ProbNMS
Torch:         Rand2DElastic
Torch:         Rand3DElastic
TorchOrNumpy:  RandAdjustContrast
Torch:         RandAffine
Torch:         RandAffineGrid
Torch:         RandAxisFlip
TorchOrNumpy:  RandBiasField
TorchOrNumpy:  RandCoarseDropout
TorchOrNumpy:  RandCropByLabelClasses
TorchOrNumpy:  RandCropByPosNegLabel
TorchOrNumpy:  RandDeformGrid
Torch:         RandFlip
TorchOrNumpy:  RandGaussianNoise
Torch:         RandGaussianSharpen
Torch:         RandGaussianSmooth
TorchOrNumpy:  RandGibbsNoise
Numpy:         RandHistogramShift
TorchOrNumpy:  RandKSpaceSpikeNoise
Uncategorised: RandLambda
TorchOrNumpy:  RandRicianNoise
Torch:         RandRotate
TorchOrNumpy:  RandRotate90
TorchOrNumpy:  RandScaleCrop
TorchOrNumpy:  RandScaleIntensity
TorchOrNumpy:  RandShiftIntensity
TorchOrNumpy:  RandSpatialCrop
TorchOrNumpy:  RandSpatialCropSamples
TorchOrNumpy:  RandStdShiftIntensity
TorchOrNumpy:  RandWeightedCrop
Torch:         RandZoom
TorchOrNumpy:  RemoveRepeatedChannel
TorchOrNumpy:  RepeatChannel
Torch:         Resample
Torch:         Resize
TorchOrNumpy:  ResizeWithPadOrCrop
Torch:         Rotate
TorchOrNumpy:  Rotate90
TorchOrNumpy:  SaveImage
Torch:         SavitzkyGolaySmooth
TorchOrNumpy:  ScaleIntensity
TorchOrNumpy:  ScaleIntensityRange
TorchOrNumpy:  ScaleIntensityRangePercentiles
TorchOrNumpy:  ShiftIntensity
TorchOrNumpy:  SimulateDelay
Torch:         Spacing
TorchOrNumpy:  SpatialCrop
TorchOrNumpy:  SpatialPad
TorchOrNumpy:  SplitChannel
TorchOrNumpy:  SqueezeDim
TorchOrNumpy:  StdShiftIntensity
TorchOrNumpy:  ThresholdIntensity
TorchOrNumpy:  ToNumpy
Torch:         ToTensor
TorchOrNumpy:  Transpose
TorchOrNumpy:  VoteEnsemble
Torch:         Zoom

[wyli]

This is now finalised, more info via python monai/transforms/utils.py:

Total number of transforms: 220
Number transforms allowing both torch and numpy: 136
Number of TorchTransform: 49
Number of NumpyTransform: 20
Number of uncategorised: 15

Most IO and performance profiling utility transforms are uncategorised.
please feel free to open new tickets for any further issues/feature requests.

Thanks for leading this effort @rijobro.