Include all Cat2/3 Errors on Critical Tab of Error Report

[elipe17]

Summary of Changes

• Updated error report generation to accept a boolean indicating if it is a S3 or S4 file
• Updated the prioritized queryset function to prioritize the queryset differently for S3 and S4 files
• Exclude header update indicator cat2 errors wrt S3/S4 files
• Updated spreadsheet writer to exclude the case number column for S3/S4 files since it will always be empty
  Pull request closes Move Cat 2 and Cat 3 errors from aggregate and stratum to critical worksheet #3341

Something to note. S3/S4 records/errors do NOT have a case number association. However, the error report always includes a column on the critical tab for case number. In the S3/S4 case, that column will be completely empty. I have reached out to Alex to see if I can handle that case under this PR or if it should live in another ticket.

How to Test

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

1. Open http://localhost:3000/ and sign in.
2. Update a Aggregate/Stratum file to have some cat2/cat3 errors
3. Submit the file
4. Verify the report displays all cat2/cat3 errors on the "Critical" tab

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• Any Cat 2 or Cat 3 errors detected in Aggregate data is included on critical worksheet
• Any Cat 2 or Cat 3 errors detected in Stratum data is included on critical worksheet
• The following cat 2 error message does not appear on critical worksheet: HEADER Update Indicator must be set to D instead of N. Please review Exporting Complete Data Using FTANF in the Knowledge Center.
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[codecov]

Codecov Report

Attention: Patch coverage is 73.07692% with 7 lines in your changes missing coverage. Please review.

Project coverage is 91.38%. Comparing base (ff01121) to head (e2e05cd).
Report is 5 commits behind head on develop.

Files with missing lines	Patch %	Lines
tdrs-backend/tdpservice/data_files/util.py	69.56%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3381      +/-   ##
===========================================
- Coverage    91.48%   91.38%   -0.10%     
===========================================
  Files          299      299              
  Lines         8595     8614      +19     
  Branches       636      640       +4     
===========================================
+ Hits          7863     7872       +9     
- Misses         615      622       +7     
- Partials       117      120       +3     

Flag	Coverage Δ
dev-backend	91.24% <73.07%> (-0.09%)	⬇️
dev-frontend	92.44% <ø> (-0.22%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-backend/tdpservice/data_files/views.py	88.99% <100.00%> (+0.10%)	⬆️
tdrs-backend/tdpservice/data_files/util.py	94.06% <69.56%> (-5.94%)	⬇️

... and 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f01cfef...e2e05cd. Read the comment docs.

[ADPennington]

@klinkoberstar

Section 3 and 4 error reports before this change:

2024-Q1-SSP Stratum Data (1).xlsx
2023.Q3.Tribal Aggregate Data (1).txt
2022-Q1-Aggregate Data (5).xlsx
2024-Q3-Stratum Data (1).xlsx

Section 3 and 4 error reports after this change:

2022-Q1-Aggregate Data (6).xlsx
2022-Q1-SSP Stratum Data (4).xlsx
2023-Q3-Tribal Aggregate Data (9).xlsx
2024-Q3-Stratum Data.xlsx

[ADPennington]

🚀