O+M 2024-06-24

[FuhuXia]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• Check auto generated O&M tickets from no status column
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• DB-Solr Sync
• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check
• Check user management requests
• Check Snyk Dashboard and cleanup

Monthly Checklist

• Invicti Scan

ad-hoc checklist

• audit/review applications on cloud foundry and determine what can be stopped and/or deleted.

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[FuhuXia]

NewRelic shows catalog-web performed better after #4708.

For example:

transaction time reduced from ~1100ms to ~400 ms:

Apdex score:

Web request timeout percentage dropped from ~1% to ~0.1%:

POSTGRES DB tracking_summary SELECT query throughput:

tracking_summary SELECT query dropped from the top to no.6 in most time consuming ranking:

[FuhuXia]

Set the following harvest source to manual schedule until source url is fixed.

• /harvest/city-and-county-of-durham-north-carolina-data-json
• /harvest/santa-rosa-ca-data-json
• /harvest/new-mexico-resource-geographic-information-system-nm-rgis/

[FuhuXia]

Starting from June 5, 2024, our harvest agent is blocked by Institute of Museum and Library Services' web server, harvest source /harvest/imls-json can not be harvested.

[hkdctol]

Have reached out to the contact addresses I have for IMLS.

[FuhuXia]

Since 2024-06-25 03:50 EDT googlebot started to send nonsense traffic to catalog, doubling the total requests catalog receives, and doubling the catalog-web CPU usage. If this trends continues, we might have to block certain traffic based on the request pattern. Details in slack discussion.

[FuhuXia]

Reduced prod catalog-web instances from 5 to 3. Mem from 850M to 800M. The following two PRs save us 2050M memory.

• we can afford to use less catalog-web instances #4800
• use less catalog-web instances catalog.data.gov#1383

[FuhuXia]

Change harvest sources their original schedules for those that were paused due to repeated ParentNotHarvestedException error.

• /harvest/dot-socrata-data-json
• /harvest/healthdata-gov