Google Floods Part 3: Adding Inundation Maps; COUNTRY=mozambique

[lowriech]

This adds inundation polygons from the Google Floods API.

https://www.loom.com/share/5b430b093ee546139f5ccbe3299c60c9?sid=049273de-18ff-4b6a-8718-b1a164c82b52

[lowriech]

@gislawill this is updated to call gauges, rather than rely on a previous call passing its attributes

[ericboucher]

Please explicit the inputs here, esp. the cache_key and how it should be used.

[gislawill]

Added some add comments and updated the params to be more explicit

[ericboucher]

@gislawill thanks for this. Yes I think it could be useful to add the cache timeout mechanism for other requests as well.

And for the cache clearing, I think it would be worth implementing but not sure if there is an easy way. We could do something simple like delete any file that's more than 30 days old or something like this. @wadhwamatic any thoughts on that point?

[wadhwamatic]

And for the cache clearing, I think it would be worth implementing but not sure if there is an easy way. We could do something simple like delete any file that's more than 30 days old or something like this. @wadhwamatic any thoughts on that point?

@ericboucher - yes, we should delete older files in this case and 30 days sounds like a good initial period to try

[gislawill]

@wadhwamatic and @ericboucher, we have many different options for clearing this cache. I think whichever option we go with, it should be easy to trigger the cache clearing automatically and manually. We'll also want to be able to easily adjust the "minimum age to clear" to allow for experimentation with this cache age and for clearing caches after a bug fix rollout.

The two good options would be a shell script and an endpoint:

• Script
  • A Python script (could be Bash too) that loops through the cache directory and clears files based on age
  • The age would default to 30 days but could be provided as an environment variable
    • A provided age of 0 would clear all cached files
  • This could be triggered manually by SSH-ing into the container and running it
• Endpoint
  • A FastAPI endpoint that loops through the cache directory and clears files based on age
  • The age would default to 30 days but could be provided in the request body
  • This could be triggered manually by any request client (i.e. curl)
    • Very easy, though not secure (but can't think of any real risks associated with it)

Two options for automated triggers:
- A cron daemon in the Docker image to keep simple
- An ECS Scheduled Task which is a slightly more complicated set up but easier to monitor and more resilient (retries, alerting)

I'm leaning towards a script that's triggered by an ECS Scheduled Task. Curious for your thoughts. Also curious if either of you would have a preference between adding this cache clearing in this PR vs a follow-up?

[ericboucher]

@gislawill the endpoint option seems to be convenient and allows any of us to trigger it manually when necessary. We could "secure" it with a token or something simple, that should be enough to test things out.

I'd prefer to have this added in a follow-up PR as it would make it a bit easier to discuss implementation and triggers. If we go with the endpoint option, we could use a simple GitHub action to run daily and call this endpoint.

[gislawill]

Sounds good to me @ericboucher. @wadhwamatic I've created issue #1371 for this work.

This PR should be ready for review. It's blocked on rollout by the copy + translations questions here: #1328 (comment)

[ericboucher]

@gislawill can you resolve the conflicts when you get a chance? :)