Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add media backup #2220

Merged
merged 7 commits into from
Sep 25, 2023
Merged

Add media backup #2220

merged 7 commits into from
Sep 25, 2023

Conversation

asteel-gsa
Copy link
Contributor

@asteel-gsa asteel-gsa commented Sep 22, 2023

Adds mediabackup command, and documentation regarding it

  • Make sure you’ve merged main into your branch shortly before creating the PR. (You should also be merging main into your branch regularly during development.)

  • Verify that no Git surgery was necessary, or, if it was necessary at any point, repeat the testing after it’s finished.

  • Once a PR is merged, keep an eye on it until it’s deployed to dev, and do enough testing on dev to verify that it deployed successfully, the feature works as expected, and the happy path for the broad feature area (such as submission) still works.

PR checklist: reviewers

  • Pull the branch to your local environment and run make docker-clean; make docker-first-run && docker compose up; then run docker compose exec web /bin/bash -c "python manage.py test"
  • Manually test out the changes locally, or check this box to verify that it wasn’t applicable in this case.
  • Check that the PR has appropriate tests. Look out for changes in HTML/JS/JSON Schema logic that may need to be captured in Python tests even though the logic isn’t in Python.
  • Verify that no Git surgery is necessary at any point (such as during a merge party), or, if it was, repeat the testing after it’s finished.

The larger the PR, the stricter we should be about these points.

@asteel-gsa asteel-gsa requested a review from a team September 22, 2023 12:42
@asteel-gsa asteel-gsa temporarily deployed to dev September 22, 2023 12:43 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 22, 2023 12:43 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Sep 22, 2023

Terraform plan for dev

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.dev.module.clamav.cloudfoundry_app.clamav_api will be updated in-place
  ~ resource "cloudfoundry_app" "clamav_api" {
      ~ docker_image                    = "ghcr.io/gsa-tts/fac/clamav@sha256:01b7d29c95a8c60a3355e1e0d7ea583b4a4f7aab1fe60039b8f5f5791c08c1a3" -> "ghcr.io/gsa-tts/fac/clamav@sha256:4b7513597a0e3cb7e71a1109e6fd63d53476ebf453928b23649176609b3ee0f7"
        id                              = "779bbc51-f78a-4186-90eb-5acb68d7d746"
        name                            = "fac-av-dev"
        # (15 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.dev.module.database.cloudfoundry_service_instance.rds,
  on /tmp/terraform-data-dir/modules/dev.database/database/main.tf line 14, in resource "cloudfoundry_service_instance" "rds":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 2 more similar warnings elsewhere)

✅ Plan applied in Deploy to Development and Management Environment #233

@github-actions
Copy link
Contributor

github-actions bot commented Sep 22, 2023

Terraform plan for meta

No changes. Your infrastructure matches the configuration.
No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

✅ Plan applied in Deploy to Development and Management Environment #233

@github-actions
Copy link
Contributor

github-actions bot commented Sep 22, 2023

File Coverage Missing
All files 89%
api/serializers.py 90% 125-126 131 136
api/test_views.py 96% 105
api/uei.py 96% 17-18 107-108
api/views.py 97% 196-197 204-205 226 362-363
audit/excel.py 87% 402 418 424 429 434 456-457 469 670-671 680-686 696 716
audit/forms.py 56% 22-29
audit/intake_to_dissemination.py 92% 70-71 200-206 256
audit/models.py 87% 185 218 389 407-408 416 438 514-515 519 527 536 546 549-555
audit/test_commands.py 87%
audit/test_mixins.py 90% 112-113 117-119 184-185 189-191
audit/test_validators.py 95% 434 438 606-607 846 853 860 867
audit/utils.py 76% 13 21 38
audit/validators.py 96% 269-270 298-299 314-315 325 516-525
audit/views.py 42% 86-107 130-131 205-206 251-252 263-264 266-270 317-330 333-347 352-365 382-388 393-413 440-444 449-478 521-525 530-550 577-581 586-615 658-662 667-679 682-692 697-709 732-733 738-787 790-830 833-850
audit/cross_validation/additional_ueis.py 93% 33
audit/cross_validation/check_award_ref_declaration.py 90%
audit/cross_validation/check_award_reference_uniqueness.py 93%
audit/cross_validation/check_certifying_contacts.py 87%
audit/cross_validation/check_findings_count_consistency.py 91%
audit/cross_validation/check_ref_number_in_cap.py 90%
audit/cross_validation/check_ref_number_in_findings_text.py 90%
audit/cross_validation/errors.py 78% 30 65
audit/cross_validation/naming.py 68% 178-182
audit/cross_validation/submission_progress_check.py 85% 64 79-82
audit/cross_validation/tribal_data_sharing_consent.py 86% 20
audit/cross_validation/validate_general_information.py 93% 28-29
audit/fixtures/single_audit_checklist.py 79% 156 232-241
audit/management/commands/load_fixtures.py 46% 39-45
audit/viewlib/submission_progress_view.py 96% 158-159
audit/viewlib/upload_report_view.py 30% 32-35 44 91-115 118-186
cms/views.py 57% 11-16 29-30
config/urls.py 71% 87
djangooidc/backends.py 78% 32 57-63
djangooidc/exceptions.py 66% 19 21 23 28
djangooidc/oidc.py 16% 32-35 45-51 64-70 92-149 153-199 203-226 230-275 280-281 286
djangooidc/views.py 80% 22 43 114
djangooidc/tests/common.py 96%
report_submission/forms.py 92% 35
report_submission/views.py 76% 84 216-217 219 241-242 261-262 288-398 401-411
report_submission/templatetags/get_attr.py 76% 8 11-14 18
support/admin.py 49% 9-10 13 34 37 74 77 82 89-95 98-100 103-104 107-108
support/cog_over.py 91% 30-33 93 140
support/signals.py 66% 23-24 29-30
support/test_cog_over.py 98% 134-135 224
tools/update_program_data.py 89% 96
users/auth.py 95% 39-40
users/fixtures/user_fixtures.py 91%

Minimum allowed coverage is 90%

Generated by 🐒 cobertura-action against 931f74c

@asteel-gsa asteel-gsa temporarily deployed to meta September 22, 2023 13:00 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 22, 2023 13:00 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 22, 2023 13:08 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 22, 2023 13:08 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 22, 2023 18:03 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 22, 2023 18:03 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 25, 2023 14:28 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 25, 2023 14:28 — with GitHub Actions Inactive
We have a race condition where if the service is unbound too quickly,
the backups fail
@asteel-gsa asteel-gsa temporarily deployed to dev September 25, 2023 17:21 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 25, 2023 17:21 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 25, 2023 17:21 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 25, 2023 17:21 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 25, 2023 17:23 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 25, 2023 17:23 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to dev September 25, 2023 20:55 — with GitHub Actions Inactive
@asteel-gsa asteel-gsa temporarily deployed to meta September 25, 2023 20:55 — with GitHub Actions Inactive
Copy link
Contributor

@danswick danswick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, @asteel-gsa. You may want to pull in main now that the unrelated failing tests have been resolved.

@tadhg-ohiggins tadhg-ohiggins temporarily deployed to dev September 25, 2023 22:04 — with GitHub Actions Inactive
@tadhg-ohiggins tadhg-ohiggins temporarily deployed to meta September 25, 2023 22:04 — with GitHub Actions Inactive
@asteel-gsa
Copy link
Contributor Author

Thank you @danswick and @tadhg-ohiggins

@asteel-gsa asteel-gsa added this pull request to the merge queue Sep 25, 2023
Merged via the queue into main with commit 8974656 Sep 25, 2023
14 checks passed
@asteel-gsa asteel-gsa deleted the as/add-mediabackup branch September 25, 2023 22:18
@asteel-gsa asteel-gsa restored the as/add-mediabackup branch September 25, 2023 22:19
asteel-gsa added a commit that referenced this pull request Sep 25, 2023
@asteel-gsa asteel-gsa deleted the as/add-mediabackup branch September 25, 2023 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement on-demand backup (db and media) tooling in the codebase
3 participants