Compactor: Compaction of Raw Data without cleaning local Disk

[Jakob3xD]

Thanos, Prometheus and Golang version used
Thanos: v0.7.0 Prometheus: latest (v2.12.0)

What happened
When compacting Raw data the local disk wont clean after a compaction is done.
In my case this will cause the compactor to use more then 1TB of Space when stating a lvl 4 compaction.

What you expected to happen
I expected that the compactor clears the data on his local disk after he uploaded the compacted block to the object storage like it is done on the down-sampled data.
It was expected that the compactor crashes once the 1TB disk was full.

How to reproduce it (as minimally and precisely as possible):
Some blocks form several Sidecars which needs to compact.

The one larger peak at 11:15 is the finish of the level 3 compaction.
Each small peak you see is a new compaction to level 4.
Full logs to relevant components
Once the disk is full:
level=error ts=2019-09-05T16:51:30.809934432Z caller=main.go:213 msg="running command failed" err="error executing compaction: compaction failed: compaction failed for group 0@{monitor="node",prom="prom-30",region="fsn1",replica="a"}: download block 01DM09QBNY0SDHGPT6J71FPST2: copy object to file: write /tmp/thanos/compact/0@{monitor="node",prom="prom-30",region="fsn1",replica="a"}/01DM09QBNY0SDHGPT6J71FPST2/chunks/000002: no space left on device"

Anything else we need to know

[krasi-georgiev]

When you run the sidecar to upload the blocks to remote storage you should have the compaction disabled
see: https://github.com/thanos-io/thanos/blob/master/docs/components/sidecar.md#example-basic-deployment

If you need the compaction enabled for some reason you can set the Prometheus size or time retention flag so that older blocks get deleted.

[Jakob3xD]

@krasi-georgiev
I wasn't talking about the Prometheus compactor. I meant the Thanos Compactor running over the object storage.

[krasi-georgiev]

@Jakob3xD oops, sorry for the confusion.
I just had a quick look and the compactor does indeed clear the tmp folder before every compaction.

thanos/pkg/compact/compact.go

Lines 1018 to 1021 in 865d5ec

	// Clean up the compaction temporary directory at the beginning of every compaction loop.
	if err := os.RemoveAll(c.compactDir); err != nil {
	return errors.Wrap(err, "clean up the compaction temporary directory")
	}

Are you sure that the compacted blocks don't exceed 1tb or if you don't have something else on that HDD that occupied the space?

The cleanup happens while the compactor downloads the blocks so in theory there could be a race where the old blocks are not removed while trying to download the new ones.
@bwplotka is there any reason why aren't these old blocks removed before running the Compact go routines?

[Jakob3xD]

@krasi-georgiev
I am sure that the compacted block does not exceed 1TB.
Each time a little peak occurred in the graph I could see in the web inspector that a new level 4 compaction was performed.
Beside that the compactor is using a separate disk volume where only the compactor writes onto.

I only see level=info but I can tell that each time the message

caller=compact.go:1035 msg="start of compaction

occur the folder gets fully cleaned up.

Just to be clear. My issue is that after the compactor performed the compaction and starts to delete the blocks on the object storage it does not fully clean them on his local folder. The compactor deletes about 40gb on the object storage and on his local folder but there are still 46gb of data in his local folder which does not get cleaned up.
So if after each compaction 46gb data remains I would need over 1426gb space for the compactor with about 31 Sidecars + Rulers.

Does the Compactor deletes the compacted block on his folder or just the blocks he compacted?

[krasi-georgiev]

The code is to delete everything in data/compact (with the default options)
But I see the problem. The tmp compact folder is deleted before every compaction and not when it completes. Maybe we should add a cleanup before and after the compaction.
cc @bwplotka

[Jakob3xD]

For my understanding and to clarify it.
When you say:

The tmp compact folder is deleted before every compaction[...]

Do you mean the moment where the Compactor starts the compaction of the whole level. So when the log message says caller=compact.go:1035 msg="start of compaction" or do you mean the the moment where a block gets compacted caller=compact.go:440 msg="compact blocks"?

[krasi-georgiev]

I little bit before: start of compaction
To be exact just before the start sync of metas log

[bwplotka]

Yes, so that was the simplificaitons we did. We have only one place to delete stuff (beginning of compactor iteration), in order to reduce cases of forgetting to delete. (:

I think this issue shows that we can't simplify like this and it would be nice to clean things ASAP (as well as on the beginning - in case of crash/restart)

[Jakob3xD]

@krasi-georgiev @bwplotka It seems like you guys misunderstood my issue.
I wasn't quite sure about this comment so I waited for the PR from @krasi-georgiev .

I little bit before: start of compaction
To be exact just before the start sync of metas log

Today a new level 4 compaction started with Thanos v0.8.1 and the issues is still present.
I try to clarify my issue.
As far as I understand the Compactor it is doing the following steps:

1. start sync of metas
2. start of GC
3. start of compaction
   3.1 Clear local compactor folder
   3.2 Download list of blocks from object storage
   3.3 Compact list of blocks and upload it to the object storage
   3.4 Delete list of blocks which were compacted from object storage
   3.5 Repeat step 3.2,3.3,3.4 for another list of blocks
   (3.6 Clear local compactor folder)
4. compaction iterations done
5. start first pass of downsampling
   5.1 Download compacted block
   5.2 Create down-sampled block out of the compacted block
   5.3 Upload the down-sampled block to the object storage
   5.? Clear the local down-sampling folder
   5.? Delete overlapping old down sampling block from object storage
   5.? Repeat the steps 5.1 till 5.? for another compacted block

I am note quite sure about all steps made by the down-sampling because not everything is displayed in the logs.
So as I understand you added "Clear local compactor folder" as step 3.6 in your PR.
But my issues is that it should be cleared after 3.4 and repeated with 3.5.

So that step 3 looks like this:
3. start of compaction
3.1 Clear local compactor folder
3.2 Download list of blocks from object storage
3.3 Compact list of blocks and upload it to the object storage
3.4 Delete list of blocks which were compacted from object storage
3.5 Clear local compactor folder
3.6 Repeat step 3.2,3.3,3.4,3.5 for another list of blocks
4. compaction iterations done

Do you guys understand my issue?

Followed are my current compactor settings:
compact
--data-dir /tmp/thanos
--objstore.config-file /thanos/s3.yml
-w
--retention.resolution-raw=90d
--retention.resolution-5m=90d
--retention.resolution-1h=90d
--compact.concurrency=4
--block-sync-concurrency=40
--consistency-delay=2h

[bwplotka]

Thanks, I made our e2e more strict and fixed this hopefully. Each compaction group run was still reusing disk potentially: #1666