Downsampling, how it works ?

[mixman68]

Hello,

I try to configure a new thanos cluster

my Compactor is set as

  thanos-compactor:                             
    image: quay.io/thanos/thanos:v0.31.0                       
    volumes:                            
      - ./thanos/:/etc/thanos/                                 
      - compactor_data:/data/                 
    command:                                  
      - "compact"                                  
      - "--log.level=debug"                     
      - "--data-dir=/data"                                               
      - "--objstore.config-file=/etc/thanos/bucket_config.yaml" 
      - "--retention.resolution-raw=2d"                        
      - "--retention.resolution-5m=120d"                       
      - "--retention.resolution-1h=1y"                               
      - "--wait"                                               
      - "--wait-interval=3m"                                             
    restart: always

But i see no downsampling, the metric thanos_compact_downsample_total still 0.

Scrapes has beginned et 2023-04-15 @ 13:00

Can you help me please, i think it is a little issue in my config, but i don't found which

[mixman68]

Hello

Some news :

Thanos has compacted but forget the beginning series from 15 to 17

[v1jayr]

With --retention.resolution-raw=2d you can end up losing the data before compactor downsamples. You should set the retention to a longer period to give compactor enough buffer to compact + downsample. Note that Thanos does not begin downsample for 40 hours, 10 days for 5m and 1h downsampling respectively.

[mixman68]

Hello

thanks for reply

what is the minimum value I can put on each retention

i put 2d because 2d = 48h > 40 hours
I misunderstand the rule I think

[v1jayr]

NOTE: If retention at each resolution is lower than minimum age for the successive downsampling pass, data will be deleted before downsampling can be completed. As a rule of thumb retention for each downsampling level should be the same, and should be greater than the maximum date range (10 days for 5m to 1h downsampling).

from https://thanos.io/tip/components/compact.md/#tldr-on-how-thanos-downsampling-works

[mixman68]

Hello, thanks you
I set this :
- "--retention.resolution-raw=4d"
- "--retention.resolution-5m=30d"
- "--retention.resolution-1h=1y"
now it is good I think

[rkdutta]

Just a question, when you set 4d for raw resolution, did you see 5m downsampled data after 40 hours or its just the raw resolution data.

[mixman68]

Seems to be good with your recommendations, but I don't understand why I have raw blocks after 7 days with this policy.

[mixman68]

Hello back,

i suspect lost data again before 21 april, no downsampled data, still raw data since 21 apr 6:00 to today

Raw sampling

5m downsampling

And bucket view

The 4th level is only raw data, 3nd level contains downsampled and raw data from 27 April 02:00

I tried to understand rule, so I don't understand why I lost 5m downsampling data and raw data after 8 days

- "--retention.resolution-raw=4d"
- "--retention.resolution-5m=30d"
- "--retention.resolution-1h=1y"

Thanks

[ivanitlearning]

The settings --retention.resolution-X have nothing to do with configuring downsampling. By default, Thanos creates additional samples at 5m, 1h resolution according to these default settings

Compactor downsampling is done in two passes:

1. All raw resolution metrics that are older than 40 hours are downsampled at a 5m resolution
2. All 5m resolution metrics older than 10 days are downsampled at a 1h resolution

These are not modified by any of your settings. Instead what you have configured

- "--retention.resolution-raw=4d"
- "--retention.resolution-5m=30d"
- "--retention.resolution-1h=1y"

• deletes raw samples after 4 days
• deletes downsampling-created samples of 5m resolution after 30d
• deletes downsampling-created samples of 1h resolution after 1 year

The explanation in the flags make it clear what this actually does

      --retention.resolution-1h=0d
                                How long to retain samples of resolution 2 (1
                                hour) in bucket. Setting this to 0d will retain
                                samples of this resolution forever
      --retention.resolution-5m=0d
                                How long to retain samples of resolution 1 (5
                                minutes) in bucket. Setting this to 0d will
                                retain samples of this resolution forever
      --retention.resolution-raw=0d
                                How long to retain raw samples in bucket.
                                Setting this to 0d will retain samples of this
                                resolution forever

Unless you are paying a lot for bucket storage, there's no need to delete data, whether created by downsampling or ingested normally. You can keep them in the bucket indefinitely. The documentation says this much

By default, there is NO retention set for object storage data. This means that you store data forever, which is a valid and recommended way of running Thanos.

You can configure retention by using --retention.resolution-raw --retention.resolution-5m and --retention.resolution-1h flag. Not setting them or setting to 0s means no retention.

It's a little confusing here because no retention usually means data is deleted, but they actually mean its stored in the bucket forever.