store on local filesystem for X (ex:days) and on S3 beyond X

[korenlev]

Is your proposal related to a problem?

highest performance for shorter-lived data (queries to local ssd)
lower performance for longer-lived data (queries to s3)

Describe the solution you'd like

define 2 types of object storage options , assign 1 for initial period and the other for beyond X

OR:
dynamically defined storage option based on certain criteria ?

Describe alternatives you've considered

none

Additional context

[GiedriusS]

This is already possible with options --min-time and --max-time just that queries will be actually entirely executed in memory, in a central location due to the lack of #305. Thus, I suggest closing this 😄

[korenlev]

just to be clear please:
is the current logic to check first the Prometheus local persistent disk ?
if the store is defined as aws s3 and Prometheus local disk has the requested metrics (Chunk/s) for that requested time period - will aws s3 be contacted at all ? will compactor upload those also to s3, even if they are all available locally ?

[yeya24]

If both s3 and Prometheus store are in the requested time range, then they will be contacted at the same time and querier will merge the results and do some deduplication.

[korenlev]

greatly appreciate your swift reply guys ! closing this