Use of startTimeMillis for findTraces API , when es-aliases are used

[sivatarunp]

Requirement - what kind of business use case are you trying to solve?

Improve the query performance in Jaeger, for Elasticsearch storage, when --es.use-aliases is set to true

Problem - what in Jaeger blocks you from solving the requirement?

Currently, when we use aliases for Elasticsearch storage, the findTraces API queries all indices present under jaeger-span-read alias (irrespective of time range we give in UI). Due to this, when the data set is huge, significant amount of time is being used for querying unnecessary shards which are not in the given time range.

Proposal - what do you suggest to solve the problem or improve the existing situation?

The findTraces API use the startTime field for querying, which is a long field. Elasticsearch in built has a feature, to skip shards before querying, when the query is a range query on date field.

https://discuss.elastic.co/t/timeline-query-on-timestamped-indices/129328/2 is the related discussion for the same.

Hence modifying the findTraces API , to use startTimeMillis field(which is already present in the data we store) which is a date type field, can help in skipping unnecessary shards hence improving the query performance

[albertteoh]

@sivatarunp that's an interesting find; did you get a chance to test this out to see how much of an improvement this change would make to the FindTraces query? If so, could you share those numbers?

[sivatarunp]

@albertteoh . We have couple of things here.

1. Changing the timefield to startTimeMillis.
2. Adding a time range query with startTimeMillis field in the msearch query used to fetch individual trace data, in the findTraces page.
   We observed a huge difference, in query times. It's more predominantly seen, when we have the hot and warm nodes setup. Same query got timed out without the above changes

[albertteoh]

@sivatarunp are you able to quantify the query time improvement by providing some numbers from your tests?

[sivatarunp]

@albertteoh Here is a panel I could build from grafana community dashboards. The current query took more than 4m and even timed out, where as with above changes results were under 1 min, for the same data set

[albertteoh]

Thanks for that, @sivatarunp. Your proposal sounds reasonable to me. It's not entirely clear to me why the startTime field to hold the microsecond unix epoch was created, perhaps for higher precision for sorting ES results and internal use. @pavolloffay?

Are you able to provide a contribution from the change you have tested?

[jpkrohling]

Looks like this was fixed already.