Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update o&m checklists #4246

Merged
merged 2 commits into from
Mar 30, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 37 additions & 22 deletions .github/ISSUE_TEMPLATE/o-and-m.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,43 @@ assignees: ''
---
As part of day-to-day operation of Data.gov, there are many [Operation and Maintenance (O&M) responsibilities](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities). Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an [O&M Triage role](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#om-triage-rotation). One person on the team is assigned the Triage role which rotates each sprint. _This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time._


## Miscs
- Watch for user email requests
- Watch in [#datagov-alerts](https://gsa-tts.slack.com/archives/C4RGAM1Q8) and [Vulnerable dependency notifications (daily email reports)](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#vulnerable-dependency-notifications-daily-email-reports) for critical alerts.

- Watch for user email requests
- Watch in [#datagov-alerts](https://gsa-tts.slack.com/archives/C4RGAM1Q8) and [Vulnerable dependency notifications (daily email reports)](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#vulnerable-dependency-notifications-daily-email-reports) for critical alerts.
- Monitor and improve [Data.gov O&M Dashboard](https://onenr.io/0LREMrzdrRa)

## Acceptance criteria
You are responsible for all [O&M responsibilities](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities) this week. We've highlighted a few so they're not forgotten.

| Task | Friday | Monday | Tuesday | Wednesday | Thursday | Friday | Monday | Tuesday | Wednesday | Thursday | Weekly/Monthly |
|---------------------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| Check Deployments | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | ➖ |
| Check Restarts | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | ➖ |
| Check [Snyk Scans](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#automated-dependency-updates-ad-hoc-github-prs) | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | <ul><li>[ ] Catalog</li><li>[ ] Inventory</li></ul> | ➖ |
| Check Catalog Auto Tasks | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> |<ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> |<ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | <ul><li>[ ] DB-Solr Sync</li><li>[ ] Tracking Update</li><li>[ ] Stuck Jobs</li></ul> | ➖ |
| Check [Harvesting Emails](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#harvest-job-report-daily-email-report) | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | <ul><li>[ ] Catalog</li></ul> | ➖ |
| [New Relic Alerts](https://alerts.newrelic.com/accounts/1601367/incidents) Triaged | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | ➖ |
| Triage DMARC Report from Google | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | <ul><li>[ ] </li></ul> | ➖ |
| Check [Catalog Solr](https://github.com/GSA/data.gov/wiki/Operation-and-Maintenance-Responsibilities#solr) | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | <ul><li>[ ] Week 1</li><li>[ ] Week 2</li></ul> |
| [Audit Log](https://docs.google.com/spreadsheets/d/1z6lqmyNxC7s5MiTt9f6vT41IS2DLLJl4HwEqXvvft40/edit) [*AU-6*](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#au-6-log-auditing) | <ul><li>[ ] </li></ul> | ➖ | ➖ | ➖ | ➖ | <ul><li>[ ] </li></ul> | ➖ | ➖ | ➖ | ➖ | ➖ |
| [Catalog Dupe Check](https://github.com/GSA/data.gov/wiki/Operation-and-Maintenance-Responsibilities#duplicate-check) | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | <ul><li>[ ] Week 1</li><li>[ ] Week 2</li></ul> |
| [Invicti Scan](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#netsparker-compliance-scan-report-from-isso) | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | ➖ | <ul><li>[ ] Week 1</li><li>[ ] Week 2</li></ul> |

- [ ] Weekly [resources.data.gov link scan](https://app.circleci.com/pipelines/github/GSA/resources.data.gov?branch=main)
- [ ] Finishing the shift: Log the [number of alerts](https://docs.google.com/spreadsheets/d/1u1hSUAQW6FWzphog122stfB6MB9Wiq0NROT3PeicRoM/edit#gid=939071144)
You are responsible for all [O&M responsibilities](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities) this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

## Daily Checklist
Check Deployments
- [ ] Catalog
- [ ] Inventory

Check Restarts
- [ ] Catalog
- [ ] Inventory

Check [Snyk Scans](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#automated-dependency-updates-ad-hoc-github-prs)
- [ ] Catalog
- [ ] Inventory

Check Catalog Auto Tasks
> **Note**
btylerburton marked this conversation as resolved.
Show resolved Hide resolved
> You will need to update the chart values manually. Click the Action link in each issue and grab the values from `monitor task output` and `check runtime`.

- [ ] [DB-Solr Sync](https://github.com/GSA/catalog.data.gov/issues/848)
- [ ] [Tracking Update](https://github.com/GSA/catalog.data.gov/issues/847)

Check [Harvesting Emails](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#harvest-job-report-daily-email-report)
- [ ] Catalog

Other
- [ ] [New Relic Alerts](https://alerts.newrelic.com/accounts/1601367/incidents) Triaged
- [ ] Triage DMARC Report from Google

## Weekly Checklist
- [ ] [Audit Log](https://docs.google.com/spreadsheets/d/1z6lqmyNxC7s5MiTt9f6vT41IS2DLLJl4HwEqXvvft40/edit) (more info on [AU-3 and AU-6 Log auditing](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#au-3-and-au-6-log-auditing))
- [ ] Check [Catalog Solr](https://github.com/GSA/data.gov/wiki/Operation-and-Maintenance-Responsibilities#solr)
- [ ] [Catalog Dupe Check](https://github.com/GSA/data.gov/wiki/Operation-and-Maintenance-Responsibilities#duplicate-check)
- [ ] [Invicti Scan](https://github.com/gsa/data.gov/wiki/Operation-and-Maintenance-Responsibilities#netsparker-compliance-scan-report-from-isso)