How to write the Kibana Timelion expression to get an average time of Celery task execution

81 views Asked by At

I'm using ELK of version 7.17. Running different Celery tasks and logging "Task started", "Task finished" events to the Elasticsearch via Logstash. I'm adding a trace_id field as log_extra, so, I can recognize pairs related to single task. I want to get an average time spent per task_type and how did it change in specified period? I read it's possible with Timelion, but everything I tried fails... is it possible at all?

UPD 1:

Example documents:

{
  "@timestamp": "2023-05-23T06:01:01.111Z",
  "message": "Task started",
  "trace_id": "1",
  "task_type": "A"
}     
{
  "@timestamp": "2023-05-23T06:02:02.222Z",
  "message": "Task started",
  "trace_id": "2",
  "task_type": "A"
}     
{
  "@timestamp": "2023-05-23T06:03:03.333Z",
  "message": "Task finished",
  "trace_id": "2",
  "task_type": "A"
}
{
  "@timestamp": "2023-05-23T06:04:04.444Z",
  "message": "Task started",
  "trace_id": "3",
  "task_type": "B"
}     
{
  "@timestamp": "2023-05-23T06:05:05.555Z",
  "message": "Task finished",
  "trace_id": "1",
  "task_type": "A"
}     
{
  "@timestamp": "2023-05-23T06:06:06.666Z",
  "message": "Task finished",
  "trace_id": "3",
  "task_type": "B"
}

In this example we have 3 Celery tasks of 2 types - "A" and "B". First task "A" with trace_id=1 start and write log message "Task started", then another task of type "A" (trace_id=2) start and finish until the end of the first task. Then a third task of type "B"(trace_id=3) start and will be finished only after the first task will finish.

0

There are 0 answers