Chart a divergence between two time fields in the same elasticsearch index with timelion

Question

So I have an elastic search index with lots of data and I have found an issue with some of the data that I would like to visualise. Some items in the index matched under the itm.description field as say FOO have two timestamp entries called itm.timestamp and itm.jmsTimestamp.

These two fields have started to diverge quite considerably when they were very close a few days ago. Some ActiveMQ processing is going on between the two so that seems like the likely cause but I would like to visualise when this started and what the drift is over the last few days in Kibana using timelion.

.es(index=myindex*,q='itm.description:FOO',timefield='@itm.timestamp'),.es(index=myindex*,q='itm.description:FOO',timefield='@itm.jmsTimestamp')

So clearly this query is not particularity helpful as it produces two flat lines. What do I need to produce a graph that displays the difference in the drift between the two timestamps using timelion? Is it possible to display the two timestamps overlayed or would graphing the drift be more useful?

Answer 1

Have you tried using the subtract-expression?

Something like this:

.es(index=myindex*, q='itm.description:FOO', timefield='@itm.timestamp', metric=max:'@itm.timestamp').subtract(.es(index=myindex*, q='itm.description:FOO', timefield='@itm.timestamp', metric=max'@itm.jmsTimestamp'))

You cannot use the direct values of a field, Timelion requires an aggregation, as it renders 1 value per bucket. In my example, I used the max-aggregation.

Another solution would be to introduce a new field that you populate at indexing time with the delta of the two timestamp fields. Again you would need to use an aggregation for value to be displayed on the y-axis.