O+M 2023-05-12

[nickumia-reisys]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Tracking Update
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[nickumia-reisys]

Day 1

• Catalog solr follower 0 restarted 5/8 @ 3:11a (normal)
• Catalog solr follower 1 restarted 5/8 @ 4:53a (normal)
• DMARC Report (outlook): 42% compliant (187 total emails)
  • Email domains in question:
    • epa.gov
    • vsmtpx-e107-02.localdomain (although this does pass...)
  • Failures that seem to be false positives:
    • dkim: fail spf: softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
• DMARC Report (google): 100% compliant (410 total emails)
• Worked through open PRs.
• Worked through Snyk triaging.
• Identified issue with inventory build failing.

[nickumia-reisys]

Day 2 + 3

• Fixed Catalog Solr Follower 2 (pairing with @FuhuXia and @btylerburton)

• Fixed Inventory Build issue

• Raised issue about catalog tracking update user visits
  

• Raised issue about increasing catalog db-solr-sync updates
  

• Added @rshewitt as a user for catalog, inventory, sysadmins-users

• Github Actions has been down since 9:40a on 5/10.. Still down.. will continue to monitor status, https://www.githubstatus.com/
  

[nickumia-reisys]

Day 4 + 5

• Github showed weakness until end of day 5/11. On 5/12, Github Status showed all green.
  
• Backfilled Tracking Update and revealed that the increased traffic started on 5/4 and has leveled at around 230k per day since 5/9
  
• 5/12: Catalog Solr Follower 0 restarted @8:11a
• 5/12: Catalog Solr Follower 1 restarted @8:12a
  • This caused a problem that @FuhuXia took care of.
    • Solr follower core locked after restart with multiple index folders found #4138
• Finally got all of Inventory Open PRs through
• Harvesting Status
  • Known broken harvest sources still broken.
  • State JSON harvest still broken.
  • FCC Data.json failed 5/12
    • ConnectionError getting json source: HTTPSConnectionPool(host='[opendata.fcc.gov](http://opendata.fcc.gov/)', port=443): Max retries exceeded with url: /data.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f30b6ec98b0>: Failed to establish a new connection: [Errno -2] Name or service not known')).
  • OEI Non-Geo Records
    • JSONDecodeError loading json. Invalid control character '\n' at: line 52 column 997 (char 3150)
• Investigated Tracking Update anamolies... No clear explanation of increased tracking update number...
  • Cloudfront requests
    
  • NewRelic requests (only _tracking route -- catalog-proxy)
    
  • NewRelic requests (all traffic -- catalog-proxy)
    

[nickumia-reisys]

Didn't get much official O&M done last week, but I'll catch up on things this week with the continuing O&M role

• O+M 05-19-2023 #4318