Improve performance of experimental.resize

[banasraf]

Category:

Refactoring

Description:

This PR improves experimental.resize operator to reduce the CPU overhead of the operator. It contains the following improvements (from most to least significant):

• using two workspace objects for the CV-CUDA op to get rid of synchronising between minibatches
• using custom allocator (taking memory from scratchpad) for the input/output TensorBatches
• improved method for creating input/output TensorBatch objects: moving all common parts out of the loop
• removed auxiliary TensorLists in favour of using the original i/os directly with PushFramesToBatch method

Additional information:

Affected modules and functionalities:

experimental.resize, nvcvop.h/cc

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A

[banasraf]

!build

[dali-automaton]

CI MESSAGE: [19099938]: BUILD STARTED

[mzient]

Does this actually help? How much? It's against the trend of more aggressive dynamic allocation.

[banasraf]

It helped A LOT. For some reason the destructors of these were taking a lot of time.
Anyway, I'm working on removing those auxiliary TensorLists completely, because the .ShareData and .Resize also take significant amount of time.

[dali-automaton]

CI MESSAGE: [19099938]: BUILD FAILED

[banasraf]

!build

[dali-automaton]

CI MESSAGE: [19870509]: BUILD FAILED

[banasraf]

!build

[dali-automaton]

CI MESSAGE: [19871581]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [19871581]: BUILD PASSED

[mzient]

Unused?

[mzient]

I think that combining the two loops and changing the outer loop condition to "while there are frames left to insert" makes it more readable:

Suggested change

	for (int64_t i = 0; i < num_frames; ++i) {
	if (frame_offset == sample_nframes) {
	frame_offset = 0;
	do {
	++sample_id;
	auto sample_shape = input_shape[sample_id];
	DALI_ENFORCE(sample_id < t_list.num_samples());
	std::copy(&sample_shape[first_spatial_dim], &sample_shape[input_shape.sample_dim()],
	frame_shape.begin());
	frame_stride = volume(frame_shape) * type_size;
	sample_nframes = calc_num_frames(sample_shape, first_spatial_dim);
	} while (sample_nframes * frame_stride == 0); // we skip empty samples
	data =
	static_cast<const uint8_t *>(t_list.raw_tensor(sample_id)) + frame_stride * frame_offset;
	}
	tensors.push_back(AsTensor(data, make_span(frame_shape), dtype, nvcv_layout));
	data += frame_stride;
	frame_offset++;
	int frames_left = num_frames;
	while (frames_left) {
	if (frame >= sample_nframes) {
	++sample_id;
	assert(sample_id < t_list.num_samples());
	auto sample_shape = input_shape[sample_id];
	std::copy(&sample_shape[first_spatial_dim], &sample_shape[input_shape.sample_dim()],
	frame_shape.begin());
	frame_stride = volume(frame_shape) * type_size;
	if (frame_stride == 0) { // this sample is (effectively) empty - skip
	sample_nframes = 0;
	continue;
	}
	sample_nframes = calc_num_frames(sample_shape, first_spatial_dim);
	data = static_cast<const uint8_t *>(t_list.raw_tensor(sample_id)) + frame_stride * frame_offset;
	}
	tensors.push_back(AsTensor(data, make_span(frame_shape), dtype, nvcv_layout));
	data += frame_stride;
	frame_offset++;
	frames_left--;
	}

[mzient]

I don't think that either user or faulty data could trigger this - it would be an internal error, so I'd recommend using an assert or throwing logic_error at worst.

[mzient]

This is a bug. Both inputs and outputs should be inserted in one go and skipping the empty samples should be based solely on the output size. The user may request resizing a non-empty tensor to (0, 0), which is not an error AFAIR. Resizing an empty input to non-empty shape is an error and should be thrown at some point.

[mzient]

This is a bug - the emptiness of a frame depends on the output shape, not input. At least in old DALI resize, you can resize a non-empty frame to size 0. I understand that such samples should be skipped (both at input and output).
Resizing an empty frame to a non-zero shape is impossible and should throw.