Reverse Geocoding jobs keep failing.

[dunc1ca]

I'm not sure if I'm doing something wrong but I have approximately 500 thousand points to reverse geocode. After 3 weeks, only about 20 thousand have been successful. The rest will fail and go through a retry cycle. I have cleared the queue and selected continue reverse geocoding since upgrading versions, but it still continues to fail most geocode jobs.

[dunc1ca]

	Next Retry	Retry Count	Queue	Job	Arguments	Error
	just now	0	reverse_geocoding	ReverseGeocodingJob	"Point", 618367	Geocoder::LookupTimeout: Geocoder::LookupTimeout
	right now	0	reverse_geocoding	ReverseGeocodingJob	"Point", 618397	Errno::ECONNREFUSED: Errno::ECONNREFUSED

[shaman007]

Seems like they're throttling and banning IP's. There are several Dawarich instanses seems ti be started and they see some IP's using the API.

I'd checked, to set up OSM locally about 128 GB RAM is needed and severe effort to be applied.

UPD:

the thing to handle this stuff woudl be about 100 Eur/month, better 150.

[shaman007]

Would be really cool to get in touch with @jonaskmt or @komoot if they have some API pricing for enthusiasts. I am sure they don't need my 100-200-300 eur badly to allow to geocode stuff, still I would rather pay a bit to have API access.

[shaman007]

Welp, downloading photon now. See komoot/photon#848

[Freika]

Added a section on that to the FAQ: https://dawarich.app/docs/FAQ#why-i-have-so-many-failed-jobs-in-sidekiq

Also, we have an instruction on how to set up your own Photon instance to be independent of Nominatim's API limits: https://dawarich.app/docs/tutorials/reverse-geocoding#setting-up-your-own-reverse-geocoding-service

[Freika]

@shaman007 Photon doesn't need 128GB of RAM, since it provides only geocoding/reverse geocoding api, it's heavy on disk space but not on RAM

[shaman007]

Yeah, I'd already tried my own. With only me as a user it's totaly managable.

[Freika]

Yeah, I see you got an answer in Photon's repo :)

[dunc1ca]

I seem to have fixed the reverse geocoding fails by changing BACKGROUND_PROCESSING_CONCURRENCY: from 10 to 1. It seems like komoot didn't like the request rate and just rejected me. It is now geocoding about 4000 points per hour.