MediaConvert CreateJobTemplate error for Baseline profiles

[tigrato]

Documentation

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug

When creating a simple encoding template job for H264 video output with Baseline profile the number NumberBFramesBetweenReferenceFrames has to be sent as 0 (Baseline does not support Bframes between reference frames).
The field NumberBFramesBetweenReferenceFrames: int32 is only serialized if the provided value is not zero so it is never sent here

Expected behavior

The system allow me to create the JobTemplate without errors

Current behavior

MediaConvert API returns an error mentioning that NumberBFramesBetweenReferenceFrames must be <=0 OR that the Profile must be changed to something different than Baseline (If I sent the NumberBFramesBetweenReferenceFrames <0 the media convert api returns other error)

MediaConvert: CreateJobTemplate, https response error StatusCode: 400, RequestID: 083e9ebb-36b3-414b-8dd3-cbc50d2edbf8, BadRequestException: /outputGroups/0/outputs/3/videoDescription/codecSettings/h264Settings/codecProfile: Should match all dependencies: See other errors for more details | /outputGroups/0/outputs/3/videoDescription/codecSettings/h264Settings: Should match exactly one schema defined in "oneOf" | /outputGroups/0/outputs/3/videoDescription/codecSettings/h264Settings: numberBFramesBetweenReferenceFrames is a required property | /outputGroups/0/outputs/3/videoDescription/codecSettings/h264Settings/codecProfile: Should be equal to one of the allowed values in ["HIGH","HIGH_10BIT","HIGH_422","HIGH_422_10BIT","MAIN"]

Steps to Reproduce

Create via GO API a template with the Following H264 config

H264Settings: &types.H264Settings{
                AdaptiveQuantization: "HIGH",
                Bitrate: 1768615,
                CodecLevel: "LEVEL_3_1",
                CodecProfile: "BASELINE",
                DynamicSubGop: "STATIC",
                EntropyEncoding: "CAVLC",
                FieldEncoding: "PAFF",
                FlickerAdaptiveQuantization: "DISABLED",
                FramerateControl: "SPECIFIED",
                FramerateConversionAlgorithm: "DUPLICATE_DROP",
                FramerateDenominator: 1,
                FramerateNumerator: 25,
                GopBReference: "DISABLED",
                GopClosedCadence: 1,
                GopSize: 50.0,
                GopSizeUnits: "FRAMES",
                HrdBufferInitialFillPercentage: 0,
                HrdBufferSize: 0,
                InterlaceMode: "PROGRESSIVE",
                MaxBitrate: 0,
                MinIInterval: 0,
                NumberBFramesBetweenReferenceFrames: 0,
                NumberReferenceFrames: 2,
                ParControl: "SPECIFIED",
                ParDenominator: 1,
                ParNumerator: 1,
                QualityTuningLevel: "SINGLE_PASS",
                QvbrSettings: nil,
                RateControlMode: "CBR",
                RepeatPps: "DISABLED",
                ScanTypeConversionMode: "",
                SceneChangeDetect: "ENABLED",
                Slices: 1,
                SlowPal: "DISABLED",
                Softness: 0,
                SpatialAdaptiveQuantization: "ENABLED",
                Syntax: "DEFAULT",
                Telecine: "NONE",
                TemporalAdaptiveQuantization: "ENABLED",
                UnregisteredSeiTimecode: "DISABLED",
              },

Possible Solution

Mark NumberBFramesBetweenReferenceFrames to be serialized if the Baseline profile is selected even if it is zero or change the media convert behaviour to set it as 0 if baseline profile is found.

AWS Go SDK version used

v1.16

Compiler and Version used

1.17

Operating System and version

MacOS 11.6

[vudh1]

Hi, is this still persisting with the latest version of SDK?

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[tigrato]

@vudh1 yes, it is still present

[RanVaknin]

Hi @tigrato ,

Sorry for the really long wait time. This issue took time to investigate.

The problem is that the default value 0, is being ignored even though you are trying to set it explicitly.
This is because of the way the mediaconvert service team exports their model does not work correctly with Smithy based SDKs (Kotlin, Rust, Go, Swift..) In specific, those exports lack the "Default" trait which indicates default values that should be serialized.

In your case NumberBFramesBetweenReferenceFrames is set to 0, but 0 is also the default value for it so the field is ignored in serialization.
Unfortunately right now there is no solution for this and these individual parameters will have to be patched by hand. A broad solution will be addressed in the future as part of a cross-SDK effort.

In the meantime, you can check out my workaround code. In short, what the code does is:

• Create a custom middleware
• In that middleware we get the request JSON body, and hard-insert the missing field and value and re-write the body.
• Lastly we append the custom middleware after the Serialize step.

⚠️ Please note that this code works with the parameters I have made up, and might look a bit different for you.

func main() {

	customResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
		if service == mediaconvert.ServiceID && region == "us-east-1" {
			return aws.Endpoint{
				PartitionID:   "aws",
				URL:           "https://q25wbt2lc.mediaconvert.us-east-1.amazonaws.com",
				SigningRegion: "us-east-1",
			}, nil
		}
		return aws.Endpoint{}, &aws.EndpointNotFoundError{}
	})

	myMiddleware := middleware.SerializeMiddlewareFunc("MyTestMiddleware",
		func(ctx context.Context, input middleware.SerializeInput, next middleware.SerializeHandler,
		) (
			output middleware.SerializeOutput, metadata middleware.Metadata, err error,
		) {
			req, ok := input.Request.(*smithyhttp.Request)
			if !ok {
				return output, metadata, fmt.Errorf("unexpected transport: %T", input.Request)
			}

			// get the body from the request
			bodyStream := req.GetStream()
			buf := new(strings.Builder)
			_, err = io.Copy(buf, bodyStream)
			actualBodyString := buf.String()

			// the location after which we want to insert the parameter.
			substring := `h264Settings":{`
			index := strings.Index(actualBodyString, substring)

			// re-constructing the body
			newBodyString := actualBodyString[:index+len(substring)] + `"NumberBFramesBetweenReferenceFrames":0,` + actualBodyString[len(substring)+index:]
			newStream := strings.NewReader(newBodyString)

			// matching the content length
			req.ContentLength = int64(len(newBodyString))
			input.Request = req

			// overriding the body
			input.Request, err = req.SetStream(newStream)
			if err != nil {
				panic(err)
			}

			return next.HandleSerialize(ctx, input)
		})

	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithEndpointResolverWithOptions(customResolver), config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody))
	if err != nil {
		log.Fatalf("unable to load SDK config, %v", err)
	}

	client := mediaconvert.NewFromConfig(cfg, func(options *mediaconvert.Options) {
		options.APIOptions = append(options.APIOptions, func(stack *middleware.Stack) error {
			return stack.Serialize.Add(myMiddleware, middleware.After)
		})

	})

	// this is a fake template, your input fields might look different
	out, err := client.CreateJobTemplate(context.Background(), &mediaconvert.CreateJobTemplateInput{
		Name: aws.String("1582_template"),
		Settings: &types.JobTemplateSettings{
			OutputGroups: []types.OutputGroup{
				{
					OutputGroupSettings: &types.OutputGroupSettings{
						Type: "CMAF_GROUP_SETTINGS",

						CmafGroupSettings: &types.CmafGroupSettings{
							SegmentLength:  1,
							FragmentLength: 1,
						},
					},
					Outputs: []types.Output{
						{
							ContainerSettings: &types.ContainerSettings{
								Container: "CMFC",
							},
							VideoDescription: &types.VideoDescription{
								CodecSettings: &types.VideoCodecSettings{
									Codec: types.VideoCodecH264,
									H264Settings: &types.H264Settings{
										AdaptiveQuantization:                "HIGH",
										Bitrate:                             1768615,
										CodecLevel:                          "LEVEL_3_1",
										CodecProfile:                        "BASELINE",
										DynamicSubGop:                       "STATIC",
										EntropyEncoding:                     "CAVLC",
										FieldEncoding:                       "PAFF",
										FlickerAdaptiveQuantization:         "DISABLED",
										FramerateControl:                    "SPECIFIED",
										FramerateConversionAlgorithm:        "DUPLICATE_DROP",
										FramerateDenominator:                1,
										FramerateNumerator:                  25,
										GopBReference:                       "DISABLED",
										GopClosedCadence:                    1,
										GopSize:                             50.0,
										GopSizeUnits:                        "FRAMES",
										HrdBufferInitialFillPercentage:      0,
										HrdBufferSize:                       0,
										InterlaceMode:                       "PROGRESSIVE",
										MaxBitrate:                          0,
										MinIInterval:                        0,
										NumberBFramesBetweenReferenceFrames: 0,
										NumberReferenceFrames:               2,
										ParControl:                          "SPECIFIED",
										ParDenominator:                      1,
										ParNumerator:                        1,
										QualityTuningLevel:                  "SINGLE_PASS",
										QvbrSettings:                        nil,
										RateControlMode:                     "CBR",
										RepeatPps:                           "DISABLED",
										ScanTypeConversionMode:              "",
										SceneChangeDetect:                   "ENABLED",
										Slices:                              1,
										SlowPal:                             "DISABLED",
										Softness:                            0,
										SpatialAdaptiveQuantization:         "ENABLED",
										Syntax:                              "DEFAULT",
										Telecine:                            "NONE",
										TemporalAdaptiveQuantization:        "ENABLED",
										UnregisteredSeiTimecode:             "DISABLED",
									},
								},
							},
						},
					},
				},
			},
		},
	})

	if err != nil {
		panic(err)
	}
}

Let me know if this helps at all.
I'm going to leave this issue open so we can address it down the line.

Thanks again for your patience,
Ran~

[tigrato]

Hey, @RanVaknin thanks for the response.
I ended up with something similar to your workaround to force the inclusion of the missing values.
Hope this gets fixed soon so I can get rid of this code.

[RanVaknin]

Hi @tigrato ,

We are tracking this in a separate ticket aws/aws-sdk#577

Closing this in favor of tracking it there.
Thanks,
Ran~

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.