chunks are in a single directory #364
Comments
|
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
stale
bot
removed
the
stale
rfratto
added
the
keepalive
|
Hi all, this is an issue on our side as well. We are now storing >2M files in a single directory and loki is now unusable for us because it takes ~10 minutes to do an operation on the chunk directory. |
|
In my opinion, filesystem chunk storage now is only fit for small and local testing. For any large deployment or prod use-case, please choose other storage like S3. If you still need to use filesystem, maybe a NFS like CephFS, it's better to implement isolation by yourselves like isolate chunks by namespace or something else, and contribute to Cortex project if it works well in your side. |
|
@mizeng, thanks for the feedback! |
|
that's question to Cortex.. Quote from https://github.com/grafana/loki/tree/master/docs/operations/storage |
|
Not yet ! but we want filesystem to be production ready and friendly :) |
|
/srv/loki/chunks/ZmFrZS8yMDUyNTU2MmY5NmFlNTE5OjE2OTMwZjVlNzFkOjE2OTMwZjVlNzFkOmVlNjhmYmQ= In this case... https://github.com/grafana/loki/blob/v2.0.0/pkg/storage/stores/shipper/util/util.go#L74
Can anyone tell me where the uploader value comes from? Next, here is a good practice about the hashed directory structure. |
|
Hi all, is there any way to separate chunks per directory (not a single directory for all of them!) ? |
|
Any Idea when this is coming? We are losing now 100gb of logfiles due to this problem. |
|
This was implemented in https://github.com/grafana/loki/releases/tag/v2.5.0 |
[release-6.0] Backport PR grafana#14416
Describe the bug
The chunks are in a single, flat directory.
To Reproduce
276K /srv/loki/chunks/ZmFrZS8yOWNjNDg0M2RmZjhkNTlmOjE2OTM1ZDg1MDFkOjE2OTM1ZDg1MWRiOmUzMWMyNmJj
Dg1MzU4OjE2OTM1ZDg1Njk4Ojk4OWM1Mjk3
Expected behavior
The chunks are split in multiple subdirectories as it would be more efficient to list. If they represent timestamps why not using that.
Environment:
Screenshots, promtail config, or terminal output
The text was updated successfully, but these errors were encountered: