How to improve real-time image processing for moving objects?

66 views Asked by At

enter image description here Sorry for long explanation.

I am trying to develop a image processing app for moving object. However i have quality (performance) problem and looking for a way to improve it.

What i want: There is moving object beneath the camera (right picture box in the attached picture), i use canny filter to find edges of the moving object in real-time (middle picture box). After that i pick a pixel with on the found edges for a moment with mouse click event and extract x and y coordinates to the text boxes.At this step timer starts working and i am getting plus-minus 10 or so (number is defined by user) the pixel y-coordinates which has RGB values bigger than 1 (sometimes just white ones R=255 G=255 and B=255). I do this because object is moving and i need edge position. Then to plot correctly, in the code, i choose the minimum y-coordinate in the pixels (depends on the range and object there can be more white pixel). As a last i am plotting the y-coordinates against the time. I find the pixel number every 30 ms (i choose this to be almost the same with camera fps number) for moving object.

What i expect: I am trying to plot the figure given in picture on the right side. But i am not able to do that. Object moves like 1 cm/s. It is not that much for my intended app. I am getting just a flat line. Sometimes there are variations but not even close to the object shape (in my case it is like sin wave).

How can i improve real-time image processing for moving objects?

I am adding the code below:

public partial class Form1 : Form
    {
        VideoCapture capture;
        Mat mat = new Mat();
        public Image<Bgr, byte> cannyOut;

        int CP_X = Cursor.Position.X;
        int CP_Y = Cursor.Position.Y;

        private bool isMessageBoxShown = false;

        public Form1()
        {
            InitializeComponent();
            serialPort1.Close();

            int BaudRatePorts = 250000;
            serialPort1.BaudRate = BaudRatePorts;
            PerdePort.BaudRate = BaudRatePorts;
            laserPort.BaudRate = BaudRatePorts;

        }



        private void btnStart_Click(object sender, EventArgs e)
        {
            if (capture == null)
            {
                capture = new VideoCapture();
            }

            capture.ImageGrabbed += Capture_ImageGrabbed;
            capture.Start();
        }

        private void Capture_ImageGrabbed(object sender, EventArgs e)
        {

            capture.Retrieve(mat);
            pictureBox1.Image = mat.ToImage<Bgr, byte>().ToBitmap();
            pictureBox1.SizeMode = PictureBoxSizeMode.StretchImage;

            ///// Canny Applied Here /////////////////////////

            double cannyFilter1;
            double cannyFilter2;

            if (double.TryParse(numericUpDown3.Text, out cannyFilter1))
            {
            }
            else
            {
                MessageBox.Show("Invalid input for cannyFilter1");
            }

            if (double.TryParse(numericUpDown4.Text, out cannyFilter2))
            {
            }
            else
            {
                MessageBox.Show("Invalid input for cannyFilter1");
            }

            var image2 = mat.ToImage<Bgr, byte>();
            var grayScaleImage = image2.Convert<Gray, byte>();
            var blurredImage = grayScaleImage.SmoothGaussian(5, 5, 0, 0);
            var cannyImage = new Mat();
            CvInvoke.Canny(blurredImage, cannyImage, cannyFilter1, cannyFilter2);
            cannyOut = cannyImage.ToImage<Bgr, byte>();

            pictureBox2.Image = cannyOut.ToBitmap();
            pictureBox2.SizeMode = PictureBoxSizeMode.Normal;

        }

        ////// This part is for getting the frame to pick a white pixel /////

        private async void btnGetFrame_Click(object sender, EventArgs e)
        {
            if (capture != null)
            {
                pictureBox3.Image = cannyOut.ToBitmap();
            }
        }
        

       ////////  I start timer here to read all frames  ///////////////////////
        private void timer1_Tick(object sender, EventArgs e)
        {
            ReadAllFrames();
            isMessageBoxShown = false;
        }

        int minJValue;
        int currentPix, previousPix;

        /////////////   I find the white pixel here ////////////////
        ////////////    Then i find minimum y value ////////////////
        ////////////    Finally i plot here   //////////////////////

        private async void ReadAllFrames()
        {
            pictureBox3.Image = cannyOut.ToBitmap();

            int totalPixel = Convert.ToInt32(numericUpDown2.Text);
            Bitmap bitmap = cannyOut.ToBitmap();
            List<int> list1 = new List<int>();

            previousPix = currentPix;
            for (int j = CP_Y - totalPixel; j < CP_Y + totalPixel; j++)
            {
                Color pixel = bitmap.GetPixel(CP_X, j);

                // Compare pixel color with white and add to list
                if (pixel.R >= 255 && pixel.G >= 255 && pixel.B >= 255)
                {
                    listBox1.Items.Add(j);
                    minJValue = Math.Min(j, j);
                    list1.Add(minJValue);

                }
                if (list1.Count > 0)
                {
                    currentPix = minJValue;

                }

                chart1.Series[0].Points.Add(minJValue);

                chart1.ChartAreas[0].AxisY.Minimum = minJValue - Convert.ToInt32(numericUpDown2.Text) - 5;
                chart1.ChartAreas[0].AxisY.Maximum = minJValue + Convert.ToInt32(numericUpDown2.Text) + 5;
                chart1.ChartAreas[0].AxisY.Title = "Nokta Sayısı";
                chart1.ChartAreas[0].AxisX.Title = "Zaman (ms)";
                chart1.ChartAreas[0].AxisX.Minimum = 0;
                chart1.ChartAreas[0].AxisX.Maximum = 150;

                if (chart1.Series[0].Points.Count > 150)
                {
                    chart1.Series[0].Points.RemoveAt(0);
                    chart1.ResetAutoValues();
                }

                if (listBox1.Items.Count > 20)
                {
                    listBox1.Items.RemoveAt(0);
                }

            }

            label2.Text = "Fark=" + (currentPix - previousPix).ToString();
            label3.Text = "P1=" + currentPix.ToString();
            label4.Text = "P2=" + previousPix.ToString();
        }

        private void pictureBox3_MouseClick(object sender, MouseEventArgs e)
        {

            textBox1.Text = string.Format("x={0:000}", CP_X);
            textBox2.Text = string.Format("y={0:000}", CP_Y);

            if (capture == null)
            {
                return;
            }

            // Start the timer
            timer1.Enabled = true;
            timer1.Interval = Convert.ToInt16(numericUpDown1.Text);
            timer1.Start();

        }

What i tried:

  1. I have changed the timer interval from 30 to 200 ms but plot was not better.

  2. To catch all pixel i have changed the canny filters, but plot did not improved.

  3. I decreased the speed of the object, did not work.

  4. I increased pixel range from 10 to 50 to catch all white pixels but plot did not improve.

0

There are 0 answers