Cache flush doesn't work

[A-Harby]

I set the max_zdb_data_dir_size to only 2 GB and added files in the qsfs disk of 10 GB up to 6 GB and I didn't notice any change in the report.

So how to check or make sure it's working correct?

[scottyeager]

The fact that there is no used space reported in either metadata or data backends suggests that zstor is not working. Those number should start going up as soon as you start adding data to the qsfs mount and exceed the zdb datasize.

As for the question of enforcement of max_zdb_data_dir_size, probably the best thing to do is check the size of the folder where zdb is storing it's data directly. In case you used the default shown in some of our docs, that would be:

du -hs /data

But data blocks won't be deleted from local storage until they are successfully uploaded with zstor, so if that's not working it's the first thing to address.

[A-Harby]

This is my zstor-default.toml

minimal_shards = 3
expected_shards = 5
redundant_groups = 0
redundant_nodes = 0
zdbfs_mountpoint = "/mnt/qsfs"
socket = "/var/run/zstor.sock"
prometheus_port = 9100
zdb_data_dir_path = "/data/data/zdbfs-data"
max_zdb_data_dir_size = 2560

And as you can see, the max_zdb_data_dir_size is set to approximately 2 GB.

Then I tried du -hs /data and got this 2.9G /data.
Also tried df -h and got this zdbfs 10G 3.1G 7.0G 31% /mnt/qsfs

So i excepted that after adding 3 GB of data in the qsfs disk, it would flush since the max zdb is only 2 GB, but it didn't.

So what could be the error?

[A-Harby]

@scottyeager, can you help me here?

[scottyeager]

Hi @A-Harby, I've done some testing and observed the same as you. It seems that zstor is not removing the zdb data blocks.

Good catch here. I hadn't tested that part specifically yet. Will continue looking into this.

[scottyeager]

threefoldtech/0-stor_v2#122

[scottyeager]

I did some further testing and was able to see this feature working.

What I noticed is:

# du -hs /data/
2.9G	/data/

# du -hs /data/data/zdbfs-data/
2.6G	/data/data/zdbfs-data/

Zstor only clears data blocks from zdbfs-data, and there's a GB vs GiB thing going on here too I think. The output from du is telling us that zdbfs-data contains ~2600MB, which is what we expect.

I had noticed explicitly in the Zstor logs in the earlier test that it was not deleting data file, but it seems this could have been due to inability to connect with the backends. I will continue monitoring, but I don't have a clear consistently reproducible issue.