Clarification of CTA WAVE conformance check results

[nicholas-fr]

When running the tool with the latest development branch (626fe38), on the CTA WAVE test vectors, with CTA WAVE checks enabled, the message in the results is not fully clear.

For the "BeforeAdaptationSet" test:
"Wave set conformance to CMAF Presentation Profile", with a PASS result the message is "No conformance found" which is ambiguous. Should this instead read something like "No conformance issues found"?

Also, does "Informative" refer to particular informative section(s) of the CTA WAVE content spec?
I would be grateful if you could clarify.

php Process_cli.php --cmaf --ctawave --segments https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/t34/2022-10-17/stream.mpd

"CTA-WAVE": {
	"verdict": "PASS",
	"BeforeMPD": {
		"verdict": "PASS"
	},
	"MPD": {
		"verdict": "PASS"
	},
	"BeforeRepresentation": {
		"verdict": "PASS"
	},
	"Representation": {
		"verdict": "PASS"
	},
	"BeforeAdaptationSet": {
		"verdict": "PASS",
		"test": [
			{
				"spec": "CTAWAVE",
				"section": "Presentation Checks",
				"test": "Attempting to open switching sets for adaptationSet ",
				"messages": [
					"Files exist"
				],
				"state": "PASS"
			},
			{
				"spec": "CTAWAVE",
				"section": "Informative",
				"test": "Wave set conformance to CMAF Presentation Profile",
				"messages": [
					"No conformance found"
				],
				"state": "PASS"
			}
		]
	},
	"AdaptationSet": {
		"verdict": "PASS"
	},
	"Period": {
		"verdict": "PASS"
	}
}

[Phencys]

This message was an informative message, meaning that there are no indications in the original (before refactor) codebase, that this is a requirement for WAVE confromance.

Therefore, this was always a PASS check, regardless of the outcome.

Pullrequest moves this message to a separate 'message' channel, and then only signals if conformance to a specific CMAF profile was found.

[dsilhavy]

Output in the current development branch looks like this and should address this issue:

{
    "CTA-WAVE": {
      "verdict": "FAIL",
      "BeforeMPD": {
        "verdict": "PASS"
      },
      "MPD": {
        "verdict": "PASS"
      },
      "BeforeRepresentation": {
        "verdict": "PASS",
        "test": [
          {
            "spec": "Segment Validations",
            "section": "analyze_results()",
            "test": "stderr filled??",
            "messages": [
              "Contents in stderr.txt found"
            ],
            "state": "PASS"
          }
        ]
      },
      "Representation": {
        "verdict": "PASS"
      },
      "BeforeAdaptationSet": {
        "verdict": "FAIL",
        "test": [
          {
            "spec": "CTAWAVE",
            "section": "Presentation Checks",
            "test": "Attempting to open switching sets for adaptationSet ",
            "messages": [
              "Files exist",
              "Files exist"
            ],
            "state": "PASS"
          },
          {
            "spec": "CTAWAVE",
            "section": "WAVE Content Spec 2018Ed-Section 4.2.1",
            "test": "Each WAVE video Media Profile SHALL conform to normative ref. listed in Table 1",
            "messages": [
              "NAL Units found for track 0 of switching set 0",
              "Valid profile for track 0 of switching set 0",
              "Valid color primaries for track 0 of switching set 0",
              "Valid transfer characteristics for track 0 of switching set 0",
              "Valid matrix coefficients for track 0 of switching set 0",
              "Valid height for track 0 of switching set 0",
              "Valid width for track 0 of switching set 0",
              "Valid framerate for track 0 of switching set 0",
              "NAL Units found for track 0 of switching set 0",
              "Valid profile for track 0 of switching set 0",
              "Valid color primaries for track 0 of switching set 0",
              "Valid transfer characteristics for track 0 of switching set 0",
              "Valid matrix coefficients for track 0 of switching set 0",
              "Valid height for track 0 of switching set 0",
              "Valid width for track 0 of switching set 0",
              "Valid framerate for track 0 of switching set 0"
            ],
            "state": "PASS"
          },
          {
            "spec": "CTAWAVE",
            "section": "WAVE Content Spec 2018Ed-Section 4.1",
            "test": "WAVE content SHALL include one or more Switching Sets conforming to at least one WAVE approved CMAF Media Profile",
            "messages": [
              "Switching Set 0 found with only tracks conforming to Media Profile H",
              "Video track found, but WAVE video track missing"
            ],
            "state": "FAIL"
          },
          {
            "spec": "CTAWAVE",
            "section": "WAVE Content Spec 2018Ed-Section 5",
            "test": "If a video track is included, then conforming Presentation will at least include that video in a CMAF SwSet conforming to required AVC (HD) Media Profile",
            "messages": [
              "Switching set not found"
            ],
            "state": "FAIL"
          }
        ],
        "info": [
          "Stream found to conform to a CMAF Presentation Profile: "
        ]
      },
      "AdaptationSet": {
        "verdict": "PASS"
      },
      "Period": {
        "verdict": "FAIL",
        "info": [
          "Stream found to conform to a CMAF Presentation Profile: "
        ],
        "test": [
          {
            "spec": "WAVE Content Spec 2018Ed",
            "section": "Section 6.2",
            "test": "WAVE CMFHD Baseline Program Shall contain a sequence of one or more CMAF Presentations conforming to CMAF CMFHD profile",
            "messages": [
              "Not all CMAF Switching sets are CMFHD conformant"
            ],
            "state": "FAIL"
          }
        ]
      }
    },
    
}

Please let us know if you see any more issues with this