Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API-26761 Optimize VBADocuments::UploadRemover job #12839

Merged
merged 0 commits into from
May 31, 2023

Conversation

caseywilliams
Copy link
Contributor

Summary

  • This job is mostly well behaved, but occasionally (seven times in the past month, for example) SQL queries can time out, extending the length of the job beyond the accepted limits. See the DataDog stats on this job here.
  • These changes attempt to reduce the frequency that this happens by reducing the number of SQL queries and limiting the set of columns retrieved.

Related issue(s)

Testing done

  • Just updated specs

What areas of the site does it impact?

  • Attempts to optimize the job that removes old Benefits Intake uploads from S3

Acceptance criteria

  • I fixed|updated|added unit tests and integration tests for each feature (if applicable).
  • No error nor warning in the console.
  • Events are being sent to the appropriate logging solution
  • Documentation has been updated (link to documentation)
  • No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
  • Feature/bug has a monitor built into Datadog or Grafana (if applicable)
  • If app impacted requires authentication, did you login to a local build and verify all authenticated routes work as expected
  • I added a screenshot of the developed feature

@caseywilliams caseywilliams added Lighthouse lighthouse benefits-intake Lighthouse Benefits Intake API banana-peels Lighthouse Banana Peels Team labels May 30, 2023
@caseywilliams caseywilliams requested review from a team as code owners May 30, 2023 20:30
@va-vfs-bot va-vfs-bot temporarily deployed to API-26761/upload-remover-updates/main/main May 30, 2023 20:36 Inactive
@caseywilliams caseywilliams force-pushed the API-26761/upload-remover-updates branch from 6801606 to 0b8350a Compare May 31, 2023 15:33
@va-vsp-bot va-vsp-bot requested a deployment to API-26761/upload-remover-updates/main/main May 31, 2023 15:34 In progress
Copy link
Contributor

@mathisto mathisto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful. Nice call on the SELECT and batching.

@va-vfs-bot va-vfs-bot temporarily deployed to API-26761/upload-remover-updates/main/main May 31, 2023 15:45 Inactive
@caseywilliams caseywilliams merged commit 873f9fb into master May 31, 2023
@caseywilliams caseywilliams deleted the API-26761/upload-remover-updates branch May 31, 2023 16:31
ryan-mcneil pushed a commit that referenced this pull request Dec 11, 2023
Updates SQL in the job to hopefully reduce execution time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
banana-peels Lighthouse Banana Peels Team benefits-intake Lighthouse Benefits Intake API Lighthouse lighthouse
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants