Query endpoints fail with 404

[atombender]

Describe the bug
Well, this is odd. We've been runing 0.3.0 for a while on Kubernetes. Then I wanted to test a newer logcli (which has added --from and --to), and that returned 404. So I thought, fine, I guess I can't use a newer logcli with 0.3.0, and I reverted. But now all queries are returning 404.

The server is running happily (see log below). But:

$ curl http://loki.ops.svc.cluster.local:3100/loki/api/v1/query_range?query=%7Bapp%3D%22test%22%2Cnamespace%3D%22staging%22%7D&limit=30&start=1569602403782397228&end=1569606003782397228&direction=FORWARD
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Fri, 27 Sep 2019 17:39:11 GMT
Content-Length: 19

404 page not found

Or for that matter: logcli --addr http://loki.ops.svc.cluster.local:3100 query --forward '{app="test",namespace="staging"}'.

The log (debug enabled):

level=debug ts=2019-09-27T17:41:38.125608772Z caller=logging.go:44 traceID=21e2fac5c19d8231 msg="GET /loki/api/v1/query_range?query=%7Bapp%3D%22test%22%2Cnamespace%3D%22staging%22%7D&limit=30&start=1569602498097123118&end=1569606098097123118&direction=FORWARD (404) 52.259µs"

Other endpoints, like /ready, /metrics, /api/prom/query and the push endpoints work fine. Promtail seems to work fine pushing logs to Loki.

From looking at the code, the only possible way that Loki can return 404 is if the HTTP handler isn't mounted. I don't see how it couldn't be. There are no explicit 404 or http.StatusCodeNotFound writes in the code.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Screenshots, promtail config, or terminal output
If applicable, add any output to help explain your problem.

level=info ts=2019-09-27T17:27:12.026069275Z caller=loki.go:125 msg=initialising module=server
level=info ts=2019-09-27T17:27:12.026393566Z caller=server.go:120 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-09-27T17:27:12.026820327Z caller=loki.go:125 msg=initialising module=overrides
level=info ts=2019-09-27T17:27:12.02685289Z caller=override.go:48 msg="per-tenant overides disabled"
level=info ts=2019-09-27T17:27:12.026867938Z caller=loki.go:125 msg=initialising module=table-manager
level=info ts=2019-09-27T17:27:12.111137755Z caller=loki.go:125 msg=initialising module=store
level=debug ts=2019-09-27T17:27:12.111172762Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2581 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111283539Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2582 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111302003Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2583 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111321453Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2584 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111369666Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2585 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111396055Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2586 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111414377Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2587 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111426556Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2588 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111440973Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2589 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111462074Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2590 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111477859Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2591 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111494448Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2592 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111508791Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2593 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111525444Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2594 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=debug ts=2019-09-27T17:27:12.111555578Z caller=schema_config.go:378 msg="Expected Table" tableName=index_2595 provisionedRead=300 provisionedWrite=1 useOnDemandMode=false
level=info ts=2019-09-27T17:27:12.111607487Z caller=loki.go:125 msg=initialising module=ring
level=debug ts=2019-09-27T17:27:12.111618046Z caller=schema_config.go:399 msg="Table is Active" tableName=index_2595 provisionedRead=300 provisionedWrite=3000 useOnDemandMode=false useWriteAutoScale=false useReadAutoScale=false
level=info ts=2019-09-27T17:27:12.111756574Z caller=loki.go:125 msg=initialising module=querier
level=info ts=2019-09-27T17:27:12.113425154Z caller=loki.go:125 msg=initialising module=ingester
level=info ts=2019-09-27T17:27:12.114008975Z caller=loki.go:125 msg=initialising module=distributor
level=info ts=2019-09-27T17:27:12.114169835Z caller=loki.go:125 msg=initialising module=all
level=info ts=2019-09-27T17:27:12.114199715Z caller=main.go:75 msg="Starting Loki" version="(version=v0.3.0, branch=HEAD, revision=c673380b)"
level=debug ts=2019-09-27T17:27:12.114617808Z caller=consul_client_mock.go:99 msg=Get key=ring wait_index=0
level=debug ts=2019-09-27T17:27:12.114657166Z caller=consul_client_mock.go:106 msg="Get - not found" key=ring
level=info ts=2019-09-27T17:27:12.114699234Z caller=lifecycler.go:386 msg="entry not found in ring, adding with no tokens"
level=debug ts=2019-09-27T17:27:12.114829803Z caller=consul_client_mock.go:72 msg=CAS key=ring modify_index=0 value="\"$\\x8c\\n\\\"\\n\\x06loki-0\\x12\\x18\\n\\x0e10.4.1.14:9095\\x10\\xf0\\x8c\\xb9\\xec\\x05\\x18\\x02\""
level=info ts=2019-09-27T17:27:12.115440158Z caller=table_manager.go:206 msg="synching tables" num_expected_tables=15 expected_tables=15
level=debug ts=2019-09-27T17:27:12.115737111Z caller=lifecycler.go:312 msg="JoinAfter expired"
level=info ts=2019-09-27T17:27:12.115782045Z caller=lifecycler.go:316 msg="auto-joining cluster after timeout"
[...]
level=debug ts=2019-09-27T17:41:38.125608772Z caller=logging.go:44 traceID=21e2fac5c19d8231 msg="GET /loki/api/v1/query_range?query=%7Bapp%3D%22test%22%2Cnamespace%3D%22staging%22%7D&limit=30&start=1569602498097123118&end=1569606098097123118&direction=FORWARD (404) 52.259µs"

Server config:

auth_enabled: false
ingester:
  chunk_idle_period: 5m
  chunk_retain_period: 3m
  chunk_block_size: 262144
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
limits_config:
  enforce_metric_name: false
schema_config:
  configs:
  - from: 2019-06-23
    store: boltdb
    object_store: filesystem
    schema: v9
    index:
      prefix: index_
      period: 168h
server:
  log_level: debug
  http_listen_port: 3100
storage_config:
  boltdb:
    directory: /mnt/data/loki/index
  filesystem:
    directory: /mnt/data/loki/chunks
chunk_store_config:
  max_look_back_period: 0
table_manager:
  retention_deletes_enabled: false
  retention_period: 0

[joe-elliott]

The /loki/api/v1/query_range is a new endpoint that is currently supported by the tip of master, but has not yet been cut in a release. It was added here:

#1022

Your original logcli should work fine with all of the endpoints exposed in 0.3.0.

/api/prom/query
/api/prom/label
/api/prom/label/{name}/values
/api/prom/tail

The next version of Loki will support the new endpoints and be compatible with the new logcli.

[atombender]

Thanks, it looks like I goofed when building logcli.

In my Dockerfile I was doing git clone --depth=1, then git reset --hard v0.3.0. But if you do --depth=1, you don't get the older commits, obviously, but also git reset doesn't complain about the non-existent commit!

So every time I built it, I was actually building logcli from the tip of the master. Too subtle. 😞