How can I get metrics in Kubernetes for the write layer disk usage of pod containers?

Question

I'm looking for a way to gather container FS usage bytes in a way that's actually useful as a metric. container_fs_usage_bytes from cadvisor shows the disk space used on the partition used for /var/lib/docker, which is absolutely useless.

The value from docker ps --size (also visible in docker system df -v) would be far more useful Anyone know of other exporters that can do this?

For reference, here is the output from docker ps --size

$ docker ps -s
CONTAINER ID        IMAGE                                                                 COMMAND                  CREATED             STATUS                    PORTS                    NAMES               SIZE
5bcf6d4bff20        ubuntu:bionic                                                         "sleep 100000"           27 seconds ago      Up 25 seconds                                      ubuntu              34.6MB (virtual 97.8MB)
0df1749b5458        gcr.io/cadvisor/cadvisor:v0.36.0                                      "/usr/bin/cadvisor -…"   24 minutes ago      Up 24 minutes (healthy)   0.0.0.0:8080->8080/tcp   cadvisor            0B (virtual 184MB)

Answer 1

The only solution that I could come up with is to use BaseUsage definition instead of fs.Usage. This would unfortunately require to to compile that stuff in one kubelet binary compatible with your desired k8s version.

fs_usage_bytes reflect the amount of data stored in /var/lib/docker for particular container. It comes from Usage property of FsStats object and it is compatible with different CRI. HoweverFsStats has another property called BaseUsage which compatible only with Docker, and considering Docker builtin support future deprecation there might be a good reason why it's not used. According to description in the code, BaseUsage contains the value that comes from Docker and it's likely to be the same with the size shown in docker ps -s output.

So you want to use BaseUsage type definition instead of fs.Usage:

// Number of bytes that is consumed by the container on this filesystem. 
        Usage uint64 `json:"usage"` 

// Base Usage that is consumed by the container's writable layer. 
        // This field is only applicable for docker container's as of now. 
        BaseUsage uint64 `json:"base_usage"

Reference in code can be found here.

This is what is returned as container_fs_usage_bytes:

getValues: func(s *info.ContainerStats) metricValues {
        return fsValues(s.Filesystem, func(fs *info.FsStats) float64 {
                return float64(fs.Usage)

Reference in the code can be found here.

To avoid breaking any existing process with swapping the return value you might want to add whole section like this:

}, {  
        name: "container_fs_base_usage_bytes",  
        help: "Docker writable layer size",  
        valueType: prometheus.GaugeValue,  
        extraLabels: []string{"device"},  
        getValues: func(s *info.ContainerStats) metricValues {  
                return fsValues(s.Filesystem, func(fs *info.FsStats) float64 {
                  
                        return float64(fs.BaseUsage)  
                }, s.Timestamp)  
        },  
},