Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

O+M 04-14-2023 #4275

Closed
15 tasks
hkdctol opened this issue Apr 7, 2023 · 8 comments
Closed
15 tasks

O+M 04-14-2023 #4275

hkdctol opened this issue Apr 7, 2023 · 8 comments
Assignees
@nickumia-reisys
Labels
Explore O&M Operations and maintenance tasks for the Data.gov platform

Comments

@hkdctol
Copy link
Contributor

hkdctol commented Apr 7, 2023

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Miscs

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Deployments

  • Catalog
  • Inventory

Check Restarts

  • Catalog
  • Inventory

Check Snyk Scans

  • Catalog
  • Inventory

Check Catalog Auto Tasks

Note
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

Check Harvesting Emails

  • Catalog

Other

Weekly Checklist
@hkdctol hkdctol moved this to 🏗 In Progress [8] in data.gov team board Apr 7, 2023
This was referenced Apr 10, 2023
Merged
Merged
Merged
Closed
Closed
@nickumia-reisys
Copy link
Contributor

Two harvest sources failed yesterday:

  • BLS Data: https://www.bls.gov/data.json
  • GeoNode State CSW: 
    Error contacting the CSW server: HTTPSConnectionPool(host='[geonode.state.gov](http://geonode.state.gov/)', port=443): Max retries exceeded with url: /catalogue/csw?service=CSW&version=2.0.2&request=GetCapabilities (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f0662b25c10>, 'Connection to [geonode.state.gov](http://geonode.state.gov/) timed out. (connect timeout=10)'))

BLS had to be a glitch, but it is still pretty weird..
image
image

The CSW source has apparently been broken since Friday for reasons that we do not know currently.

@nickumia-reisys
Copy link
Contributor

nickumia-reisys commented Apr 11, 2023 •
edited
Loading

DMARC Failures from 4/10 (from enterprise.protection.outlook.com reporter):

x.x.x.x         dkim:  fail     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  fail     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  fail     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  fail     spf:  softfail ses-513xxxx.ssb.data.gov [] epa.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  none ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] vsmtpx-e107-01.localdomain
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
Success Rate 31/64 = 0.48

DMARC Failures from 4/10 (from google.com reporter):

x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
Success Rate 422/422 = 1.00

DMARC Failures from 4/11 (from google.com reporter): Success Rate 422/422 = 1.00

@nickumia-reisys
Copy link
Contributor

Catalog solr restarted twice last night:

  • Follower 2 @ 4:12a
  • Follower 1 @ 4:26a
  • Both seem okay.

@nickumia-reisys nickumia-reisys mentioned this issue Apr 12, 2023
Closed
@nickumia-reisys
Copy link
Contributor

BLS Data failed again today...

@nickumia-reisys
Copy link
Contributor

  • Tracking Update took realllyyy long last night... It wasn't even a lot of updates, just really long
    image

  • Solr Follower 0 restarted 4/14 @ 2:08a

  • Solr Follower 2 restarted 4/14 @ 4:51a

  • Solr Follower 1 restarted 4/14 @ 5:15a

@nickumia-reisys
Copy link
Contributor

The following harvests failed again... all were already documented https://github.com/GSA/data.gov/wiki/Broken-Harvest-Sources

Harvest Source: hawaii json
Harvest Source: Santa Rosa CA Data.json
Harvest Source: OEI Non-Geo Records
Harvest Source: City and County of Durham, North Carolina Data.json Harvest Source
Harvest Source: New Mexico Resource Geographic Information System (NM RGIS)
Harvest Source: OPM JSON
Harvest Source: Baltimore JSON

@nickumia-reisys
Copy link
Contributor

BLS Data has failed consistently since 4/7. Thanks to help from @FuhuXia, we confirmed that they are blocking us with increased bot security.

@hkdctol
Copy link
Contributor Author

hkdctol commented Apr 14, 2023

Just contacted BLS point of contact.

@nickumia-reisys nickumia-reisys mentioned this issue Apr 17, 2023
Merged
@robert-bryson robert-bryson moved this from 🏗 In Progress [8] to ✔ Done in data.gov team board Apr 17, 2023
@hkdctol hkdctol moved this from ✔ Done to Closed in data.gov team board May 1, 2023
@github-project-automation github-project-automation bot moved this from 🗄 Closed to ✔ Done in data.gov team board May 8, 2023
@nickumia-reisys nickumia-reisys moved this from ✔ Done to 🗄 Closed in data.gov team board May 11, 2023
@nickumia-reisys nickumia-reisys added O&M Operations and maintenance tasks for the Data.gov platform Explore labels Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nickumia-reisys nickumia-reisys

Labels
Explore O&M Operations and maintenance tasks for the Data.gov platform
Projects
data.gov team board
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hkdctol @nickumia-reisys