Update data loading as part of app deployment #1998
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Not needed - invoke via bin/ops and call the wait-and-load.sh as part of the app deployment.
This is a requirement so that we can get psql to run the data loading script
We need to be able to pull down the data loading repo. NOTE: we need to scope down *.github to be the redirect url for git's hosting service in a subsequent PR.
This script will grab the historical data, and then execute it as part of our app deployments.
|
Terraform plan for meta Plan: 4 to add, 0 to change, 0 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# module.environments["dev"].local_file.cf_org will be created
+ resource "local_file" "cf_org" {
+ content = <<-EOT
cf_org_name = "gsa-tts-oros-fac"
EOT
+ content_base64sha256 = (known after apply)
+ content_base64sha512 = (known after apply)
+ content_md5 = (known after apply)
+ content_sha1 = (known after apply)
+ content_sha256 = (known after apply)
+ content_sha512 = (known after apply)
+ directory_permission = "0777"
+ file_permission = "0644"
+ filename = "./../dev/orgname.auto.tfvars"
+ id = (known after apply)
}
# module.environments["preview"].local_file.cf_org will be created
+ resource "local_file" "cf_org" {
+ content = <<-EOT
cf_org_name = "gsa-tts-oros-fac"
EOT
+ content_base64sha256 = (known after apply)
+ content_base64sha512 = (known after apply)
+ content_md5 = (known after apply)
+ content_sha1 = (known after apply)
+ content_sha256 = (known after apply)
+ content_sha512 = (known after apply)
+ directory_permission = "0777"
+ file_permission = "0644"
+ filename = "./../preview/orgname.auto.tfvars"
+ id = (known after apply)
}
# module.environments["production"].local_file.cf_org will be created
+ resource "local_file" "cf_org" {
+ content = <<-EOT
cf_org_name = "gsa-tts-oros-fac"
EOT
+ content_base64sha256 = (known after apply)
+ content_base64sha512 = (known after apply)
+ content_md5 = (known after apply)
+ content_sha1 = (known after apply)
+ content_sha256 = (known after apply)
+ content_sha512 = (known after apply)
+ directory_permission = "0777"
+ file_permission = "0644"
+ filename = "./../production/orgname.auto.tfvars"
+ id = (known after apply)
}
# module.environments["staging"].local_file.cf_org will be created
+ resource "local_file" "cf_org" {
+ content = <<-EOT
cf_org_name = "gsa-tts-oros-fac"
EOT
+ content_base64sha256 = (known after apply)
+ content_base64sha512 = (known after apply)
+ content_md5 = (known after apply)
+ content_sha1 = (known after apply)
+ content_sha256 = (known after apply)
+ content_sha512 = (known after apply)
+ directory_permission = "0777"
+ file_permission = "0644"
+ filename = "./../staging/orgname.auto.tfvars"
+ id = (known after apply)
}
Plan: 4 to add, 0 to change, 0 to destroy.✅ Plan applied in Deploy to Development and Management Environment #169 |
|
Terraform plan for dev Plan: 0 to add, 1 to change, 1 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
- destroy
Terraform will perform the following actions:
# module.dev.cloudfoundry_app.data_load will be destroyed
# (because cloudfoundry_app.data_load is not in configuration)
- resource "cloudfoundry_app" "data_load" {
- disk_quota = 64 -> null
- docker_image = "ghcr.io/gsa-tts/fac-historic-public-csvs/load-historic-public-data@sha256:bc012a8a2cde3044a6fc12b1b07c04a7abd6b3ab7c97ce1aeeb1ab7479027cd4" -> null
- enable_ssh = true -> null
- environment = (sensitive value) -> null
- health_check_invocation_timeout = 0 -> null
- health_check_timeout = 0 -> null
- health_check_type = "port" -> null
- id = "8e769637-5131-41c0-aabf-d5d5faae04f0" -> null
- id_bg = "8e769637-5131-41c0-aabf-d5d5faae04f0" -> null
- instances = 1 -> null
- memory = 64 -> null
- name = "dataload" -> null
- ports = [] -> null
- space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" -> null
- stopped = false -> null
- strategy = "rolling" -> null
- timeout = 180 -> null
}
# module.dev.module.https-proxy.cloudfoundry_app.egress_app will be updated in-place
~ resource "cloudfoundry_app" "egress_app" {
~ environment = (sensitive value)
id = "a6eb3870-2769-4aa9-a8f2-f1a31939f563"
~ id_bg = "************************************" -> (known after apply)
name = "https-proxy"
# (17 unchanged attributes hidden)
# (1 unchanged block hidden)
}
Plan: 0 to add, 1 to change, 1 to destroy.✅ Plan applied in Deploy to Development and Management Environment #169 |
|
Minimum allowed coverage is Generated by 🐒 cobertura-action against 23aed8c |
If we don't do this, then the default Ubuntu pg_wrapper script will be called, and it gets very confused because it's not written to be location-independent.
This will allow us to call wait-and-load as normal. (This is seperating it out from the dockerfile)
jadudm
reviewed
Sep 1, 2023
There was a problem hiding this comment.
Can we run tasks in production?
There was a problem hiding this comment.
yup:
Alex Steel@DESKTOP-NL4DO24 MINGW64 ~/Code/FAC (as/dataload-fix)
$ cf tasks gsa-fac
Getting tasks for app gsa-fac in org gsa-tts-oros-fac / space production as alexander.steel@gsa.gov...
id name state start time command
3 test RUNNING Fri, 01 Sep 2023 12:25:20 UTC echo foo;however, our bump to 4g should likely fix this error
Retrieving logs for app gsa-fac in org gsa-tts-oros-fac / space production as alexander.steel@gsa.gov...
2023-09-01T08:25:21.92-0400 [CELL/0] OUT Cell bb361af4-f289-4c7b-9a63-40a3d76dcca5 creating container for instance 8597041f-504a-4724-9dfb-15f5a61fa1d7
2023-09-01T08:25:22.84-0400 [CELL/0] OUT Security group rules were updated
2023-09-01T08:25:22.85-0400 [CELL/0] OUT Cell bb361af4-f289-4c7b-9a63-40a3d76dcca5 successfully created container for instance 8597041f-504a-4724-9dfb-15f5a61fa1d7
2023-09-01T08:25:53.00-0400 [CELL/0] ERR Copying into the container failed: stream-in: nstar: error streaming in: exit status 2. Output: tar: ./app/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/libGLESv2.so: Wrote only 1536 of 10240 bytes
2023-09-01T08:25:53.00-0400 [CELL/0] ERR tar: ./app/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/chrome_sandbox: Cannot open: No space left on device
2023-09-01T08:25:53.00-0400 [CELL/0] ERR tar: ./app/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/chrome_100_percent.pak: Cannot open: No space left on device
2023-09-01T08:25:53.00-0400 [CELL/0] ERR tar: ./app/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/nacl_irt_x86_64.nexe: Cannot open: No space left on device
2023-09-01T08:25:53.00-0400 [CELL/0] ERR tar: ./app/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/nacl_helper: Cannot open: No space left on device
2023-09-01T08:25:53.00-0400 [CELL/0] ERR tar: ./app/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/chrome_200_percent.pak: Cannot open: No space left on device
2023-09-01T08:25:53.32-0400 [CELL/0] OUT Cell bb361af4-f289-4c7b-9a63-40a3d76dcca5 stopping instance 8597041f-504a-4724-9dfb-15f5a61fa1d7
2023-09-01T08:25:53.32-0400 [CELL/0] OUT Cell bb361af4-f289-4c7b-9a63-40a3d76dcca5 destroying container for instance 8597041f-504a-4724-9dfb-15f5a61fa1d7
2023-09-01T08:25:55.12-0400 [CELL/0] OUT Cell bb361af4-f289-4c7b-9a63-40a3d76dcca5 successfully destroyed container for instance 8597041f-504a-4724-9dfb-15f5a61fa1d7
jadudm
reviewed
Sep 1, 2023
mogul
approved these changes
Sep 1, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Paired w/ @mogul on this, there are some future enhancements we need to do, namely scoping down the *.github here. The reason we are doing this is so we can use the proxy to pull the historical data directly onto the instance running the task. We should scope this down to only have the necessary redirects, instead of
*.githubas a whole.NOTE: This will have to be actively monitored, specifically
maininto your branch shortly before creating the PR. (You should also be mergingmaininto your branch regularly during development.)PR checklist: reviewers
make docker clean; make docker-first-run && docker compose up; then rundocker compose exec web /bin/bash -c "python manage.py test"The larger the PR, the stricter we should be about these points.