Separate status from test case result

[TeddyCr]

Context

Failure status is currently stored alongside a testCaseResult. For each failure a new failure status is created and stored with the testCaseResult -- see below.

OpenMetadata/openmetadata-spec/src/main/resources/json/schema/tests/basic.json

Lines 89 to 150 in d551005

	"testCaseFailureStatus": {
	"description": "Schema to capture test case result.",
	"javaType": "org.openmetadata.schema.tests.type.TestCaseFailureStatus",
	"type": "object",
	"properties":
	{
	"testCaseFailureStatusType": {
	"description": "Status of Test Case Acknowledgement.",
	"javaType": "org.openmetadata.schema.tests.type.TestCaseFailureStatusType",
	"type": "string",
	"enum": ["Ack", "New", "Resolved"],
	"javaEnums": [
	{
	"name": "Ack"
	},
	{
	"name": "New"
	},
	{
	"name": "Resolved"
	}
	]
	},
	"testCaseFailureReason": {
	"description": "Reason of Test Case resolution.",
	"javaType": "org.openmetadata.schema.tests.type.TestCaseFailureReason",
	"type": "string",
	"enum": ["FalsePositive", "MissingData", "Duplicates", "OutOfBounds", "Other"],
	"javaEnums": [
	{
	"name": "FalsePositive"
	},
	{
	"name": "MissingData"
	},
	{
	"name": "Duplicates"
	},
	{
	"name": "OutOfBounds"
	},
	{
	"name": "Other"
	}
	]
	},
	"testCaseFailureComment": {
	"description": "Test case failure resolution comment.",
	"type": "string"
	},
	"updatedBy": {
	"description": "User who updated the test case failure status.",
	"type": "string"
	},
	"updatedAt": {
	"description": "Time when test case failure status was updated.",
	"$ref": "../type/basic.json#/definitions/timestamp"
	}
	}
	}
	},
	"additionalProperties": false

To prevent re writing the test case failure status to a testCaseResult, we'll be separating failure status as their own entities and link it to a testCaseResult (if status of testCaseResult if different than resolved).

A.C.

• Separate failure status into its own extension. Should be a time series extension linking failureStatus [Many] <-> [One] testCase
• Create assign and review statuses
• Enable listing of failure status by:
  • status
  • test case
  • assignee -> this should be an entity reference
  • reviewer -> this should be an entity reference

[harshach]

@TeddyCr so there will be global status at test case level. How do we determine a test case is failed or not , if we are not going to use the testcase result a way of checking it?

[TeddyCr]

@harshach I was thinking to have a reference to the testCaseFailureStatus in the testCaseResult. Let me know if you are thinking otherwise.

Basically, when we write a testCaseResult that failed, we check if the latest status is not Resolved, if it is not we return the testCaseFailure reference and attach it to the testCaseResult, if it is we create a new one and attach it to the testCaseResult.

I had this kind of relationship in mind

Where the ID of the testCaseFailureStatus stays the same until we have it as resolved -- like below. Of course ID won't be incremental but UUID.

now my initial thought was to keep the only 2 states that we have right in the threadTaskStatus Open or Close and to open new task (i.e. if a status goes from ack -> assigned then open a task, if it goes from assigned -> InReview then close the assigned task and open a new one for the inReview). Hence the ID of the testCaseFailureStatus stays the same until we have it as resolved so that we can display the timeline (see below) on each task.

[harshach]

@TeddyCr what happens if the test case is failed with false positives or some temporary issue. Lets say after 2 more runs its turned to green automatically without user doing anything, will this handle that case?

Instead of calling testCaseResultFailureStatus -> should we call it as testCaseResultStatus. Why not store the current status?
2.

[TeddyCr]

Why not store the current status?

@harshach what do you mean by this ^?

what happens if the test case is failed with false positives or some temporary issue. Lets say after 2 more runs its turned to green automatically without user doing anything, will this handle that case?
We had discussed that we would not close it automatically but that a user should close with a reason to keep an accurate view of the statuses. Let me know if you think we should do it in a different way.