Loki should indicate if there is more logs

[cyriltovena]

When doing log selection using {app="foo"} or {app="foo"} |= "err" the limit parameters is used to avoid to send back big responses that would not be possible to render.

However it would be useful to know if we have been limited or if we there isn't more log for the given period/filter.

I think we can add a field (resultInfo ) in the API to indicate if there is more logs, may be even the total count of log returned ?

WDYT ? /cc @joe-elliott @slim-bean

{
  "status": "success",
  "data": {
    "resultType": "streams",
    "resultInfo": {
       "more": "true",
        "count": 2,
    },
    "result": [
      {
        "stream": {
          "filename": "/var/log/myproject.log",
          "job": "varlogs",
          "level": "info"
        },
        "values": [
          {
            "1569266497240578000",
            "foo"
          },
          {
            "1569266492548155000",
            "bar"
          }
        ]
      }
    ]
  }
}

[joe-elliott]

Generally think this is a great idea. +1 to making a resultInfo object for any kind of future metadata we would include in a query response.

[cyriltovena]

#1098 this one went stale but is related and suggest to also add line/bytes processed by Loki.

[adityacs]

@cyriltovena I started looking into this. It's trickier than I thought. Could you please give me some clue on which part of the code is downloading the chunk and iterating over the logs. I initially thought len(h) here https://github.com/grafana/loki/blob/master/pkg/iter/iterator.go#L73 should give the number of log lines to iterate. My understanding was wrong. Kindly give some clue to proceed on this.

[cyriltovena]

This is not an easy task and an urgent one. I'd suggest you look at another issue.

I'd suggest you look at those may be ?

#1348
#1397
#1604
#1524
#1346

[adityacs]

@cyriltovena Sure. Will look into other issues you mentioned. Still, indicate if there is more logs would be an interesting task to work on 😃

[slim-bean]

I think we can push onto clients that if the return result count is == to the specified limit it's safe to assume there are more logs. In the rare cases where the results are exactly == to the limit they only end up making one more query which returns an empty result.