Check temporary data collection issues

[jpmckinney]

e.g. after a week, in case there's a temporary IP ban or similar. These are currently frozen until issue is resolved.

• https://contrataciones.gov.py/datos/api/v3/doc/oauth/token DNSLookupError from server, but not locally.
  • Note: In Sentry, can filter these out with !title:"DNSLookupError: DNS lookup failed: no results for hostname lookup: contrataciones.gov.py." or !title:"Gave up retrying <POST https://contrataciones.gov.py/datos/api/v3/doc/oauth/token> (failed 3 time...", depending on the issue.
• We also witnessed title:"DNSLookupError: DNS lookup failed: no results for hostname lookup: nocopo.bpp.gov.ng." but nigeria_portal succeeded more recently. In another issue, saw the corresponding title:"Gave up retrying <GET https://nocopo.bpp.gov.ng/ocdsjson.ashx?ocid=all> (failed 3 times): DNSLook..."

The following are not temporary:

• mexico_administracion_publica_federal_bulk: some releases are missing the ocid prefix, connection refused some times (2024-04) kingfisher-collect#1045
• chile_compra_api_records/chile_compra_api_releases return 502 kingfisher-collect#1037
• slovenia: 502 Bad Gateway (2024-09) kingfisher-collect#1022
• moldova: High error rate (2024-04) kingfisher-collect#1083

[yolile]

• The same issue was seen for Uruguay Retrying <GET http://comprasestatales.gub.uy/ocds/rss/2024/04> (failed 1 times): DNS lookup failed: no results for hostname lookup: comprasestatales.gub.uy. but then it worked again

[yolile]

• Buenos Aires returned 503 https://data.buenosaires.gob.ar/api/3/action/package_show?id=buenos-aires-compras but it is working locally on the server

[yolile]

• canada_montreal returned 520 but it is working on the server https://ville.montreal.qc.ca/vuesurlescontrats/api/releases.json?limit=10000

[yolile]

https://contrataciones.gov.py/datos/api/v3/doc/oauth/token DNSLookupError from server, but not locally.

I get curl: (6) Could not resolve host: contrataciones.gov.py in the server

I've frozen paraguay_dncp for now

[jpmckinney]

dig contrataciones.gov.py and nslookup contrataciones.gov.py also fail from the data support server (ocp23), but not from Linode. So, it's like an IP block is banned.

For Montreal, the data support server can collect it (in under 3 mins):

curl -n https://collect.kingfisher.open-contracting.org/schedule.json -d project=kingfisher -d spider=canada_montreal -d note="Started by James." -d steps=

On ocp13, setting a user agent with curl still succeeds:

curl -A 'Scrapy/2.11.1 (+https://scrapy.org)' 'https://ville.montreal.qc.ca/vuesurlescontrats/api/releases.json?limit=10000'

[jpmckinney]

I'm proxying Montreal now and it works. I unfroze it.

Not sure why you froze Buenos Aires. I now set validate_json = True, because it returns incomplete JSON. open-contracting/kingfisher-collect#874 It also occasionally returns 503.

[yolile]

Because Buenos Aires has only one file and it is broken

…

On Sat, Apr 27, 2024, 1:38 AM James McKinney ***@***.***> wrote: I'm proxying Montreal now and it works. I unfroze it. Not sure why you froze Buenos Aires. I now set validate_json = True, because it returns incomplete JSON. open-contracting/kingfisher-collect#874 <open-contracting/kingfisher-collect#874> — Reply to this email directly, view it on GitHub <#348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUEPSRKRLKGEVLHGOD6YVDY7M2VZAVCNFSM6AAAAABG3GKFY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBQGM3TGNJTGM> . You are receiving this because you were assigned.Message ID: ***@***.***>

[jpmckinney]

Closing here as now a deployment issue with Bob.