Review of frame tensor creation / allocation

[NicolasHug]

We create/allocate output frame tensors in different places. In particular, we determine the height and width of the output tensor from different sources:

• For batch APIs (CPU and GPU): from the stream metadata, which itself comes from the CodecContext
• For single frame APIs:
  • CPU: swscale and filtergraph: from the AVFrame
  • GPU: from the CodecContext

The info from the metadata / CodecContext are available as soon as we add a stream, e.g. right after we instantiate a Python VideoDecoder. The AVFrame is only available once we have decoded the frame with ffmpeg (this is the "raw output").

The source of truth really is the AVFrame. CondecContext may be wrong, and in particular we now know that some streams may have variable height and width #312.

---

Details:

• For batch APIs:

torchcodec/src/torchcodec/decoders/_core/VideoDecoder.cpp

Lines 165 to 187 in 41c6491

	VideoDecoder::BatchDecodedOutput::BatchDecodedOutput(
	int64_t numFrames,
	const VideoStreamDecoderOptions& options,
	const StreamMetadata& metadata)
	: ptsSeconds(torch::empty({numFrames}, {torch::kFloat64})),
	durationSeconds(torch::empty({numFrames}, {torch::kFloat64})) {
	if (options.dimensionOrder == "NHWC") {
	frames = torch::empty(
	{numFrames,
	options.height.value_or(*metadata.height),
	options.width.value_or(*metadata.width),
	3},
	{torch::kUInt8});
	} else if (options.dimensionOrder == "NCHW") {
	frames = torch::empty(
	{numFrames,
	3,
	options.height.value_or(*metadata.height),
	options.width.value_or(*metadata.width)},
	torch::TensorOptions()
	.memory_format(torch::MemoryFormat::ChannelsLast)
	.dtype({torch::kUInt8}));
	} else {

• For single frames APIs:

  torchcodec/src/torchcodec/decoders/_core/VideoDecoder.cpp

  Lines 887 to 890 in 41c6491

  	int width = streamInfo.options.width.value_or(frame->width);
  	int height = streamInfo.options.height.value_or(frame->height);
  	torch::Tensor tensor = torch::empty(
  	{height, width, 3}, torch::TensorOptions().dtype({torch::kUInt8}));

  
  and

  torchcodec/src/torchcodec/decoders/_core/VideoDecoder.cpp

  Lines 1279 to 1286 in 41c6491

  	std::vector<int64_t> shape = {filteredFrame->height, filteredFrame->width, 3};
  	std::vector<int64_t> strides = {filteredFrame->linesize[0], 3, 1};
  	AVFrame* filteredFramePtr = filteredFrame.release();
  	auto deleter = [filteredFramePtr](void*) {
  	UniqueAVFrame frameToDelete(filteredFramePtr);
  	};
  	torch::Tensor tensor = torch::from_blob(
  	filteredFramePtr->data[0], shape, strides, deleter, {torch::kUInt8});

• For CUDA APIs:
  

  torchcodec/src/torchcodec/decoders/_core/CudaDevice.cpp

  Lines 156 to 161 in 41c6491

  	int width = options.width.value_or(codecContext->width);
  	int height = options.height.value_or(codecContext->height);
  	NppiSize oSizeROI = {width, height};
  	Npp8u* input[2] = {src->data[0], src->data[1]};
  	torch::Tensor& dst = output.frame;
  	dst = allocateDeviceTensor({height, width, 3}, options.device);

[NicolasHug]

I'll close this issue. I've addressed this in a bunch of successive PRs, the state isn't 100% where we want it to be yet, but at least now we have assumption checks in place, and all strategies are documented

torchcodec/src/torchcodec/decoders/_core/VideoDecoder.h

Lines 430 to 473 in 84cef50

	// --------------------------------------------------------------------------
	// FRAME TENSOR ALLOCATION APIs
	// --------------------------------------------------------------------------
	// Note [Frame Tensor allocation and height and width]
	//
	// We always allocate [N]HWC tensors. The low-level decoding functions all
	// assume HWC tensors, since this is what FFmpeg natively handles. It's up to
	// the high-level decoding entry-points to permute that back to CHW, by calling
	// MaybePermuteHWC2CHW().
	//
	// Also, importantly, the way we figure out the the height and width of the
	// output frame tensor varies, and depends on the decoding entry-point. In
	// *decreasing order of accuracy*, we use the following sources for determining
	// height and width:
	// - getHeightAndWidthFromResizedAVFrame(). This is the height and width of the
	// AVframe, *post*-resizing. This is only used for single-frame decoding APIs,
	// on CPU, with filtergraph.
	// - getHeightAndWidthFromOptionsOrAVFrame(). This is the height and width from
	// the user-specified options if they exist, or the height and width of the
	// AVFrame *before* it is resized. In theory, i.e. if there are no bugs within
	// our code or within FFmpeg code, this should be exactly the same as
	// getHeightAndWidthFromResizedAVFrame(). This is used by single-frame
	// decoding APIs, on CPU with swscale, and on GPU.
	// - getHeightAndWidthFromOptionsOrMetadata(). This is the height and width from
	// the user-specified options if they exist, or the height and width form the
	// stream metadata, which itself got its value from the CodecContext, when the
	// stream was added. This is used by batch decoding APIs, for both GPU and
	// CPU.
	//
	// The source of truth for height and width really is the (resized) AVFrame: it
	// comes from the decoded ouptut of FFmpeg. The info from the metadata (i.e.
	// from the CodecContext) may not be as accurate. However, the AVFrame is only
	// available late in the call stack, when the frame is decoded, while the
	// CodecContext is available early when a stream is added. This is why we use
	// the CodecContext for pre-allocating batched output tensors (we could
	// pre-allocate those only once we decode the first frame to get the info frame
	// the AVFrame, but that's a more complex logic).
	//
	// Because the sources for height and width may disagree, we may end up with
	// conflicts: e.g. if we pre-allocate a batch output tensor based on the
	// metadata info, but the decoded AVFrame has a different height and width.
	// it is very important to check the height and width assumptions where the
	// tensors memory is used/filled in order to avoid segfaults.