Dlprof tool deprecated

[wyli]

Is your feature request related to a problem? Please describe.
Dlprof has been end-of-lifed, would be great to revise the tutorial to use pytorch native profiler or nsight directly

[yiheng-wang-nv]

I'm fine to take this task.

[yiheng-wang-nv]

Hi @drbeh , in https://github.com/Project-MONAI/tutorials/blob/master/performance_profiling/pathology/profiling_train_base_nvtx.md it says specific NVTX tags we added to track certain behaviors, but I cannot find any related code of nvtx directly in the py file, could you explain more about how to use nsys for the profiling? I wish to take your tutorial for reference to modify the radiology part and replace the dlprof. Thanks!

[bhashemian]

Hi @yiheng-wang-nv, we have implemented Range decorator and context manager in MONAI, which adds nvtx tags (for defining ranges) to transforms and code blocks as you can see in the code. On the other hand, nsys adds tags to many GPU kernel by default and you can use flags to disable or enable it.
Please let me know if you have any other question.

tutorials/performance_profiling/pathology/train_evaluate_nvtx.py

Lines 218 to 275 in ac88fee

	Range()(ToCupy()),
	Range("ColorJitter")(
	RandCuCIM(name="color_jitter", brightness=64.0 / 255.0, contrast=0.75, saturation=0.25, hue=0.04)
	),
	Range("RandomFlip")(RandCuCIM(name="image_flip", apply_prob=cfg["prob"], spatial_axis=-1)),
	Range("RandomRotate90")(
	RandCuCIM(name="rand_image_rotate_90", prob=cfg["prob"], max_k=3, spatial_axis=(-2, -1))
	),
	Range()(CastToType(dtype=np.float32)),
	Range("RandomZoom")(RandCuCIM(name="rand_zoom", min_zoom=0.9, max_zoom=1.1)),
	Range("ScaleIntensity")(
	CuCIM(name="scale_intensity_range", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range()(ToTensor(device=device)),
	]
	)
	preprocess_cpu_valid = Compose([ToTensorD(keys="label")])
	preprocess_gpu_valid = Compose(
	[
	Range("ValidToCupyAndCast")(ToCupy(dtype=np.float32)),
	Range("ValidScaleIntensity")(
	CuCIM(name="scale_intensity_range", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range("ValidToTensor")(ToTensor(device=device)),
	]
	)
	elif cfg["backend"] == "numpy":
	preprocess_cpu_train = Compose(
	[
	Range()(ToTensorD(keys=("image", "label"))),
	Range("ColorJitter")(
	TorchVisionD(
	keys="image",
	name="ColorJitter",
	brightness=64.0 / 255.0,
	contrast=0.75,
	saturation=0.25,
	hue=0.04,
	)
	),
	Range()(ToNumpyD(keys="image")),
	Range("RandomFlip")(RandFlipD(keys="image", prob=cfg["prob"], spatial_axis=-1)),
	Range("RandomRotate90")(RandRotate90D(keys="image", prob=cfg["prob"])),
	Range()(CastToTypeD(keys="image", dtype=np.float32)),
	Range("RandomZoom")(RandZoomD(keys="image", prob=cfg["prob"], min_zoom=0.9, max_zoom=1.1)),
	Range("ScaleIntensity")(
	ScaleIntensityRangeD(keys="image", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range()(ToTensorD(keys="image")),
	]
	)
	preprocess_cpu_valid = Compose(
	[
	Range("ValidCastType")(CastToTypeD(keys="image", dtype=np.float32)),
	Range("ValidScaleIntensity")(
	ScaleIntensityRangeD(keys="image", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range("ValidToTensor")(ToTensorD(keys=("image", "label"))),

[yiheng-wang-nv]

Hi @yiheng-wang-nv, we have implemented Range decorator and context manager in MONAI, which adds nvtx tags (for defining ranges) to transforms and code blocks as you can see in the code. On the other hand, nsys adds tags to many GPU kernel by default and you can use flags to disable or enable it. Please let me know if you have any other question.

tutorials/performance_profiling/pathology/train_evaluate_nvtx.py

Lines 218 to 275 in ac88fee

	Range()(ToCupy()),
	Range("ColorJitter")(
	RandCuCIM(name="color_jitter", brightness=64.0 / 255.0, contrast=0.75, saturation=0.25, hue=0.04)
	),
	Range("RandomFlip")(RandCuCIM(name="image_flip", apply_prob=cfg["prob"], spatial_axis=-1)),
	Range("RandomRotate90")(
	RandCuCIM(name="rand_image_rotate_90", prob=cfg["prob"], max_k=3, spatial_axis=(-2, -1))
	),
	Range()(CastToType(dtype=np.float32)),
	Range("RandomZoom")(RandCuCIM(name="rand_zoom", min_zoom=0.9, max_zoom=1.1)),
	Range("ScaleIntensity")(
	CuCIM(name="scale_intensity_range", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range()(ToTensor(device=device)),
	]
	)
	preprocess_cpu_valid = Compose([ToTensorD(keys="label")])
	preprocess_gpu_valid = Compose(
	[
	Range("ValidToCupyAndCast")(ToCupy(dtype=np.float32)),
	Range("ValidScaleIntensity")(
	CuCIM(name="scale_intensity_range", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range("ValidToTensor")(ToTensor(device=device)),
	]
	)
	elif cfg["backend"] == "numpy":
	preprocess_cpu_train = Compose(
	[
	Range()(ToTensorD(keys=("image", "label"))),
	Range("ColorJitter")(
	TorchVisionD(
	keys="image",
	name="ColorJitter",
	brightness=64.0 / 255.0,
	contrast=0.75,
	saturation=0.25,
	hue=0.04,
	)
	),
	Range()(ToNumpyD(keys="image")),
	Range("RandomFlip")(RandFlipD(keys="image", prob=cfg["prob"], spatial_axis=-1)),
	Range("RandomRotate90")(RandRotate90D(keys="image", prob=cfg["prob"])),
	Range()(CastToTypeD(keys="image", dtype=np.float32)),
	Range("RandomZoom")(RandZoomD(keys="image", prob=cfg["prob"], min_zoom=0.9, max_zoom=1.1)),
	Range("ScaleIntensity")(
	ScaleIntensityRangeD(keys="image", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range()(ToTensorD(keys="image")),
	]
	)
	preprocess_cpu_valid = Compose(
	[
	Range("ValidCastType")(CastToTypeD(keys="image", dtype=np.float32)),
	Range("ValidScaleIntensity")(
	ScaleIntensityRangeD(keys="image", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0)
	),
	Range("ValidToTensor")(ToTensorD(keys=("image", "label"))),

Hi @drbeh , thanks very much for the explain.