Glacier support

[BuGlessRB]

I'm trying to use scaleway S3, and it supports pushing objects to glacier state, and retrieving them from glacier state.
I'd like to implement two things:
a. Support in S3 backer that when it notices that an object is in glacier state (and thus cannot be retrieved), it should retry the request, but now adding instructions to retrieve it from glacier state to standard state. After that it should periodically retry fetching the object until it has been restored from glacier to standard state (after which the object will be returned).
b. Have s3backer decide (depending on hot-ness of the object) when to push it into glacier state at writing.

a) should be easy. b) might be easy after a related cache-management split into three LRU queues.

[archiecobbs]

Seems like (a) would be useful but instead of implementing (b) maybe a better approach would be to rely on S3 Lifecycle policies to move cooler data into Amazon Glacier.

[BuGlessRB]

On Mon, Jun 15, 2020 at 5:22 PM Archie L. Cobbs ***@***.***> wrote: (b) maybe a better approach would be to rely on S3 Lifecycle policies to move cooler data into Amazon Glacier.

Not necessarily better. One would like to determine transition to Glacier state based on hot-ness. The lifecycle rules work on tags and last time written. But for hotness, you need last-time-read, as well. And if this information rolls out of a very simply LRU queue on s3backer side, there is no great complexity to be added for a substantial gain in performance (and saved cost). An added benefit is that the lifecycle rules often run with day-granularity; actively pushing to glacier could be instant (again saving cost). -- Stephen.

[archiecobbs]

I'm not convinced the savings would be worth the trouble... but of course, that's an opinion that will differ from person to person depending on usage patterns and how many pennies you're trying to pinch.