-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
failed to get token ranges for ingester err="zone not set" #13414
Comments
Had the same issue after a container update yesterday. Checked my I started up my promtail instances and received the following log entries:
Looks like I am still having issues with the inmemory ring key value store. Loki logs show no issues:
Accessing the Loki metrics page displays current metrics, so the instance is running. |
New install of 3.1 gives me the same error
|
I had the same issue with Loki 3.1.0 and the following configuration file: ---
auth_enabled: false
server:
http_listen_port: 3100
log_level: info
common:
replication_factor: 1
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
ring:
# instance_availability_zone: zone
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2024-05-28
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
compactor:
retention_enabled: true
retention_delete_delay: 2h
delete_request_cancel_period: 24h
delete_request_store: s3
ruler:
enable_api: true
enable_alertmanager_v2: true
alertmanager_url: http://alertmanager.monitoring.svc:9093
remote_write:
enabled: true
client:
url: http://prometheus.monitoring.svc:9090/api/v1/write
storage:
type: s3
s3:
endpoint: minio.cloud.svc:9000
s3forcepathstyle: true
bucketnames: loki
insecure: true
access_key_id: ${S3_USER}
secret_access_key: ${S3_PASS}
wal:
dir: /loki/ruler_wal
limits_config:
retention_period: 720h
max_query_length: 721h
max_entries_limit_per_query: 1000000
ingestion_rate_mb: 8 # MB, default is 4
ingestion_burst_size_mb: 12 # MB, default is 6
split_queries_by_interval: 15m
unordered_writes: true
retention_stream: []
shard_streams:
enabled: false
volume_enabled: true
discover_service_name: [ topic ]
discover_log_levels: true
query_range:
align_queries_with_step: true
cache_results: true
query_scheduler:
max_outstanding_requests_per_tenant: 2048
frontend:
max_outstanding_per_tenant: 2048
analytics:
reporting_enabled: false
ingester:
max_chunk_age: 8h
chunk_idle_period: 4h
pattern_ingester:
enabled: true
storage_config:
tsdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
aws:
s3: s3://${S3_USER}:${S3_PASS}@minio.cloud.svc:9000/loki
s3forcepathstyle: true However, when I un-comment the |
I ran into this yesterday and gave up looking and went back to 3.0.0. I was using a fresh install of the latest Helm chart. |
I'm hitting this with all 3.1 configurations I can think to try. Seems #13103 introduced Rolled back to 3.0.0 to get things online, but would love to be able to upgrade. :-) |
I see the same error for 3.1.0 , but the whole loki stack is working as expected. What is the cause of this error. |
I'm having the same problem. Can anyone solve the problem? |
Same error here with latest version of the official loki chart. |
experiencing the same error without zone awareness replication of ingesters for |
Same here, running single binary with filesystem storage and replication=1, so IIUC there are no zones in this configuration. |
I'm running the distributed-loki helm chart with image tag 3.2.1 and it seems setting If pattern_ingester is disabled then I have no errors. |
Also experiencing this error in 3.1.1 version
Could this be the reason the ingester is getting stuck in the ring in an Unhealthy state? |
Im getting the same error with 3.0.0 version.. Is it normal ? |
After last version update (3.1) I get the next error:
level=error ts=2024-07-04T12:19:47.79081202Z caller=recalculate_owned_streams.go:55 msg="failed to get token ranges for ingester" err="zone not set"
this is my config-yml file:
After last version update (3.1) I get the next error:
level=error ts=2024-07-04T12:19:47.79081202Z caller=recalculate_owned_streams.go:55 msg="failed to get token ranges for ingester" err="zone not set"
this is my config-yml file:
The text was updated successfully, but these errors were encountered: