feat: mrf be field validation

[kevin9foong]

Problem

BE validation is currently not performed for MRF submissions. This allows for invalid responses to be saved for MRF submissions.

For example:

• duplicate options selected for checkboxes

Closes FRM-1688

Solution

Add validator for V3 responses and use this V3 validator in the MRF middleware which uses V3 responses.

Breaking Changes

• No - this PR is backwards compatible

Tests

Regression test:

• Create an MRF form with all types of basic fields, try a combination of different field settings.
• Create a table with dropdown field and text field col.
• Fill in with valid values and ensure that we can still submit all fields without validation error.

Setup for next few tests:

• Set all fields in earlier MRF form to optional to prepare for the next few test cases.
• Go to web console (inspect -> find the XHR request -> copy as cURL), save this in a text editor to create invalidate submissions

Checkbox field validation - no duplicates with others:

• Create a option called Others: A
• Enabled others
• Submit the form with checkbox response selecting the option Others: A and a custom A value in others
• Should return a validation failure, which matches Storage/email mode validation.

Table field validation:

• From the cURL command in setup, add row to table field type value despite addMoreRows disabled, it should reject.
• From the cURL command in setup, add row to table field type value, enable addMoreRows, it should accept.
• From the cURL command in setup, add more rows than maxRowsAllowed to table field type value with addMoreRows enabled, it should reject.
• Make text field required, set the textField value in row to '' empty string, it should reject.
• Change dropdown to invalid value, it should reject.
• Set text field and dropdown to not required, send empty string values for both, it should accept.

Short text length enforced:

• From the cURL command in setup, change the value of 'textfield' to less than the minimum text length set and submit, it should reject.

Children field not allowed in MRF:

• From the cURL command in setup, change the fieldType of any to children and submit, it should reject.

Decimal field range is enforced:

• From the cURL command in setup, change the value of decimal fieldType to beyond the valid range and submit, it should reject.

Short text length is enforced:

• From the cURL command in setup, change the value of short fieldType to beyond the valid range and submit, it should reject.

Valid country/region only:

• From the cURL command in setup, change the value of country_region fieldType to invalid value and submit, it should reject.

NRIC must be valid:

• From the cURL command in setup, change the value of nric fieldType to invalid nric value and submit, it should reject.

Invalid field types not allowed:

• From the cURL command in setup, change the value of fieldType to invalid field type eg, abc and submit, it should reject.

[datadog-opengovsg]

Datadog Report

Branch report: feat/mrf-be-validation
Commit report: 2190435
Test service: formsg

✅ 0 Failed, 1085 Passed, 0 Skipped, 1m 46.57s Total duration (5m 51.94s time saved)

[linear]

FRM-1688 MRF v1.1 | Backend field validation for field-level checks (required/optional, custom field validations)

[kevin9foong]

Pending UT.

The core code is ready for review.

[KenLSM]

Since we're potentially blocking submissions here and we don't have full confidence that we're making the right blocks. Would recommend us to implement this through a "shadow"[1] instead.

Lmk if you have similar concerns on the confidence of the validation implementation.

[1] We log that a validation has failed, instead of block. Once we have sufficient information and confidence, we can officially swap that to a block.

[kevin9foong]

done, when a validation error for v3 occurs, ValidateFieldErrorV3 will be thrown which will be logged on DD and we can use that for monitoring for the next 1-2 weeks.

[kevin9foong]

Written all UT for test coverage:

Field types:
| HeaderResponseV3 > dont need to test since header is not submitted.
| RadioResponseV3
| CheckboxResponseV3
| AttachmentResponseV3
| TableResponseV3
| ChildBirthRecordsResponseV3 > dont need to test since not currently used.
| YesNoResponseV3
| EmailResponseV3
| MobileResponseV3
| NumberResponseV3
| DecimalResponseV3
| ShortTextResponseV3
| LongTextResponseV3
| HomeNoResponseV3
| DropdownResponseV3
| RatingResponseV3
| NricResponseV3
| UenResponseV3
| DateResponseV3
| CountryRegionResponseV3

TC for middleware to invoke validator:

• assert children field not supported for MRF
• assert isVisible is correctly detected

[KenLSM]

Good cleanup!

[KenLSM]

LGTM!