How to import actuator httptrace in actuator prometheus? (actuator, spring boot, grafana)

Question

Imagine this is my http://localhost:8080/actuator ouotput:

{
"_links": {
    "self": {
        "href": "http://localhost:8080/actuator",
        "templated": false
    },
    "health": {
        "href": "http://localhost:8080/actuator/health",
        "templated": false
    },
    "prometheus": {
        "href": "http://localhost:8080/actuator/prometheus",
        "templated": false
    },
    "httptrace": {
        "href": "http://localhost:8080/actuator/httptrace",
        "templated": false
    }
}
}

Now I've hooked up my prometheus environment to /actuator/prometheus and that works fine. I als want prometheus to read my httptrace so I also added /actuator/httptrace to my prometheus config. However this does not work. The formatting on the httptrace endpoint is in json and the formatting in prometheus is in yaml, I think I need the httptrace in the prometheus yaml. Prometheus eats the yaml just fine, the json not so much.

How can I pass my httptrace to actuator/prometheus from my spring boot project? In the end my goal is to get the timeTaken value for every request in grafana.

Answer 1

Spring's HttpTraceRepository exposes the recent traces on an endpoint but not in prometheus as percentiles. So I can suggest two paths:

1. You implement your own HttpTraceRepository that wraps the one your using (default is InMemory....) and then override the method to fire off some custom Timer from [io.micrometer.core] with the timing (https://micrometer.io/docs/concepts#_timers) which will get aggregated as percentiles if you also enable via https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.customizing.per-meter-properties

In the end my goal is to get the timeTaken value for every request in grafana.

2. Just use http_server_requests_seconds_* that are captured per endpoint (not request)

http_server_requests_seconds_count
is the total number of requests your application received at this endpoint

http_server_requests_seconds_sum
is the sum of the the duration of every request your application received at this endpoint

and this great post explains how to use it in prometheus, https://tomgregory.com/spring-boot-default-metrics/