Detect motion blur of a cropped face with python i.e. opencv

Question

I'm detecting faces with haarcascade and tracking them with a webcam using OpenCV. I need to save each face that is tracked. But the problem is when people are moving. In which case the face becomes blurry.

I've tried to mitigate this problem with opencv's dnn face detector and Laplacian with the following code:

blob = cv2.dnn.blobFromImage(cropped_face, 1.0, (300, 300), (104.0, 177.0, 123.0))
net.setInput(blob)
detections = net.forward()
confidence = detections[0, 0, 0, 2]
blur = cv2.Laplacian(cropped_face, cv2.CV_64F).var()
if confidence >= confidence_threshold and blur >= blur_threshold:
    cv2.imwrite('less_blurry_image', cropped_face)

Here I tried to limit saving a face if it is not blurry due to motion by setting blur_threshold to 500 and confidence_threshold to 0.98 (i.e. 98%).

But the problem is if I change the camera I have to change the thresholds again manually. And in most of the cases setting a threshold omits most of the faces.

Plus, it is difficult to detect since the background is always clear compared to the blurred face.

So my question is how can I detect this motion blur on a face. I know I can train an ML model for motion blur detection of a face. But that would require heavy processing resources for a small task.

Moreover, I will be needing a huge amount of annotated data for training if I go that route. Which is not easy for a student like me.

Hence, I am trying to detect this with OpenCV which will be a lot less resource intensive compared to using an ML model for detection.

Is there any less resource intensive solution for this?

Answer 1

You can probably use a Fourier Transform (FFT) or a Discrete Cosine Transform (DCT) to figure out how blurred your faces are. Blur in images leads to high frequencies disappearing, and only low frequencies remaining.

So you'd take an image of your face, zero-pad it to a size that'll work well for FFT or DCT, and look how much spectral power you have at higher frequencies.

You probably don't need FFT - DCT will be enough. The advantage of DCT is that it produces a real-valued result (no imaginary part). Performance-wise, FFT and DCT are really fast for sizes that are powers of 2, as well as for sizes that have only factors 2, 3 and 5 in them (although if you also have 3's and 5's it'll be a bit slower).

Answer 2

As mentioned by @PlinyTheElder, DCT information can give you motion blur. I am attaching the code snippet from repo below:

The code is in C and i am not sure if there is python binding for libjpeg. Else you need to create one.

/* Fast blur detection using JPEG DCT coefficients
 *
 * Based on "Blur Determination in the Compressed Domain Using DCT
 * Information" by Xavier Marichal, Wei-Ying Ma, and Hong-Jiang Zhang.
 *
 * Tweak MIN_DCT_VALUE and MAX_HISTOGRAM_VALUE to adjust
 * effectiveness.  I reduced these values from those given in the
 * paper because I find the original to be less effective on large
 * JPEGs.
 *
 * Copyright 2010 Julian Squires <[email protected]>
 */

#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <jpeglib.h>

static int min_dct_value = 1;   /* -d= */
static float max_histogram_value = 0.005; /* -h= */

static float weights[] = {  /* diagonal weighting */
    8,7,6,5,4,3,2,1,
    1,8,7,6,5,4,3,2,
    2,1,8,7,6,5,4,3,
    3,2,1,8,7,6,5,4,
    4,3,2,1,8,7,6,5,
    5,4,3,2,1,8,7,6,
    6,5,4,3,2,1,8,7,
    7,6,5,4,3,2,1,8
};
static float total_weight = 344;

static inline void update_histogram(JCOEF *block, int *histogram)
{
    for(int k = 0; k < DCTSIZE2; k++, block++)
        if(abs(*block) > min_dct_value) histogram[k]++;
}

static float compute_blur(int *histogram)
{
    float blur = 0.0;
    for(int k = 0; k < DCTSIZE2; k++)
        if(histogram[k] < max_histogram_value*histogram[0])
            blur += weights[k];
    blur /= total_weight;
    return blur;
}


static int operate_on_image(char *path)
{
        struct jpeg_error_mgr jerr;
    struct jpeg_decompress_struct cinfo;
    jvirt_barray_ptr *coeffp;
    JBLOCKARRAY cs;
    FILE *in;
    int histogram[DCTSIZE2] = {0};

        cinfo.err = jpeg_std_error(&jerr);
        jpeg_create_decompress(&cinfo);
    if((in = fopen(path, "rb")) == NULL) {
        fprintf(stderr, "%s: Couldn't open.\n", path);
        jpeg_destroy_decompress(&cinfo);
        return 0;
    }
    jpeg_stdio_src(&cinfo, in);
    jpeg_read_header(&cinfo, TRUE);
    // XXX might be a little faster if we ask for grayscale
    coeffp = jpeg_read_coefficients(&cinfo);

    /* Note: only looking at the luma; assuming it's the first component. */
    for(int i = 0; i < cinfo.comp_info[0].height_in_blocks; i++) {
        cs = cinfo.mem->access_virt_barray((j_common_ptr)&cinfo, coeffp[0], i, 1, FALSE);
        for(int j = 0; j < cinfo.comp_info[0].width_in_blocks; j++)
            update_histogram(cs[0][j], histogram);
    }

    printf("%f\n", compute_blur(histogram));
    // output metadata XXX should be in IPTC etc

    // XXX also need to destroy coeffp?
    jpeg_destroy_decompress(&cinfo);
    return 0;
}

int main(int argc, char **argv)
{
    int status, i;

    for(status = 0, i = 1; i < argc; i++) {
        if(argv[i][0] == '-') {
            if(argv[i][1] == 'd')
                sscanf(argv[i], "-d=%d", &min_dct_value);
            else if(argv[i][1] == 'h')
                sscanf(argv[i], "-h=%f", &max_histogram_value);
            continue;
        }
        status |= operate_on_image(argv[i]);
    }

    return status;
}

Compile the code:

gcc -std=c99 blur_detection.c -l jpeg -o blur-detection

Run the code:

./blur-detection <image path>