Store: error parsing block

[joaosilva15]

Thanos version used
thanos, version 0.7.0 (branch: HEAD, revision: c6eaf68bec32aefd280318eaef6121d6ddb20d10) build user: root@aa355daaf987 build date: 20190902-15:31:30 go version: go1.12.5

What happened
Thanos store couldn't present all the data we were expecting as we have months of data and only one day was being shown. After looking at the logs all our thanos store instances were showing an error when trying to read the meta.json file from some folders in the data directory. After inspecting the folders we confirmed that they were empty.
After the clean up of the empty folders store returned to its normal behaviour.

What you expected to happen
The folders should not be empty so some process must have failed and the folders were not cleaned up. Also, thanos probably should keep reading the folders until a new healthy one is found so the query would be correctly precessed.

How to reproduce it (as minimally and precisely as possible):
Couldn't reproduce the problem

Full logs to relevant components
level=warn ts=2019-09-20T09:53:44.795446173Z caller=bucket.go:325 msg="error parsing block range" block=01DN3WMQJ1EQM192SSTXSBPS4V err="read meta.json: open /mnt/prometheus/store/01DN3WMQJ1EQM192SSTXSBPS4V/meta.json: no such file or directory"

Anything else we need to know
We use azure blob storage with thanos.
We use store in our prd environment with thanos 0.6.0 and the problem hasn't happened.

thanos: uname -a
Linux thanos 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29 14:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[GiedriusS]

Hello, seems like it was fixed by this: #1505. Please try out the newest (master) version! Shout loudly at me if it happens again with that commit.

[joaosilva15]

Tested with the newest version and it seems to be solved! Thanks 😄

[FUSAKLA]

Yes, we are running it in production and the issue does not happen anymore.
Thanks for verifying!