实时模板匹配 - OpenCV、C++

Real-time template matching - OpenCV, C++

提问by learner

I am trying to implement real-time tracking using templates. I wish to update the template with every frame. The main modifications I have done are:

我正在尝试使用模板实现实时跟踪。我希望每帧都更新模板。我所做的主要修改是：

1) separated the template matching and minmaxLoc into separate modules namely, TplMatch()and minmax()functions, respectively.

1) 将模板匹配和 minmaxLoc 分离成单独的模块，即TplMatch()和minmax()函数，分别。

2) Inside the track()function, the select_flag is kept always true so that new template is copied to 'myTemplate' with every iteration.

2) 在track()函数内部， select_flag 始终保持为真，以便每次迭代时将新模板复制到“myTemplate”。

3) The last 3 lines of function track()are to update the template (roiImg).

3) 函数track()的最后3行是更新模板(roiImg)。

4) Also,I have removed any arguments to track()function, since, imgand roiImgare global variables and hence no need to pass them to functions.

4)另外，我已经删除了track()函数的任何参数，因为img和roiImg是全局变量，因此不需要将它们传递给函数。

Following is the code:

以下是代码：

#include <iostream>
#include "opencv2/opencv.hpp"
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/objdetect/objdetect.hpp>

#include <sstream>


using namespace cv;
using namespace std;

Point point1, point2; /* vertical points of the bounding box */
int drag = 0;
Rect rect; /* bounding box */
Mat img, roiImg; /* roiImg - the part of the image in the bounding box */
int select_flag = 0;
bool go_fast = false;

Mat mytemplate;


///------- template matching -----------------------------------------------------------------------------------------------

Mat TplMatch( Mat &img, Mat &mytemplate )
{
  Mat result;

  matchTemplate( img, mytemplate, result, CV_TM_SQDIFF_NORMED );
  normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );

  return result;
}


///------- Localizing the best match with minMaxLoc ------------------------------------------------------------------------

Point minmax( Mat &result )
{
  double minVal, maxVal;
  Point  minLoc, maxLoc, matchLoc;

  minMaxLoc( result, &minVal, &maxVal, &minLoc, &maxLoc, Mat() );
  matchLoc = minLoc;

  return matchLoc;
}


///------- tracking --------------------------------------------------------------------------------------------------------

void track()
{
    if (select_flag)
    {
        roiImg.copyTo(mytemplate);
//         select_flag = false;
        go_fast = true;
    }

//     imshow( "mytemplate", mytemplate ); waitKey(0);

    Mat result  =  TplMatch( img, mytemplate );
    Point match =  minmax( result ); 

    rectangle( img, match, Point( match.x + mytemplate.cols , match.y + mytemplate.rows ), CV_RGB(255, 255, 255), 0.5 );

    std::cout << "match: " << match << endl;

    /// latest match is the new template
    Rect ROI = cv::Rect( match.x, match.y, mytemplate.cols, mytemplate.rows );
    roiImg = img( ROI );
    imshow( "roiImg", roiImg ); //waitKey(0);
}


///------- MouseCallback function ------------------------------------------------------------------------------------------

void mouseHandler(int event, int x, int y, int flags, void *param)
{
    if (event == CV_EVENT_LBUTTONDOWN && !drag)
    {
        /// left button clicked. ROI selection begins
        point1 = Point(x, y);
        drag = 1;
    }

    if (event == CV_EVENT_MOUSEMOVE && drag)
    {
        /// mouse dragged. ROI being selected
        Mat img1 = img.clone();
        point2 = Point(x, y);
        rectangle(img1, point1, point2, CV_RGB(255, 0, 0), 3, 8, 0);
        imshow("image", img1);
    }

    if (event == CV_EVENT_LBUTTONUP && drag)
    {
        point2 = Point(x, y);
        rect = Rect(point1.x, point1.y, x - point1.x, y - point1.y);
        drag = 0;
        roiImg = img(rect);
//  imshow("MOUSE roiImg", roiImg); waitKey(0);
    }

    if (event == CV_EVENT_LBUTTONUP)
    {
        /// ROI selected
        select_flag = 1;
        drag = 0;
    }

}



///------- Main() ----------------------------------------------------------------------------------------------------------

int main()
{
    int k;
/*    
///open webcam
    VideoCapture cap(0);
    if (!cap.isOpened())
      return 1;*/

    ///open video file
    VideoCapture cap;
    cap.open( "Megamind.avi" );
    if ( !cap.isOpened() )
    {   cout << "Unable to open video file" << endl;    return -1;    }
/*    
    /// Set video to 320x240
     cap.set(CV_CAP_PROP_FRAME_WIDTH, 320);
     cap.set(CV_CAP_PROP_FRAME_HEIGHT, 240);*/

    cap >> img;
    GaussianBlur( img, img, Size(7,7), 3.0 );
    imshow( "image", img );

    while (1)
    {
        cap >> img;
        if ( img.empty() )
            break;

    // Flip the frame horizontally and add blur
    cv::flip( img, img, 1 );
    GaussianBlur( img, img, Size(7,7), 3.0 );

        if ( rect.width == 0 && rect.height == 0 )
            cvSetMouseCallback( "image", mouseHandler, NULL );
        else
            track();

        imshow("image", img);
//  waitKey(100);   k = waitKey(75);
    k = waitKey(go_fast ? 30 : 10000);
        if (k == 27)
            break;
    }

    return 0;
}

The updated template is not being tracked. I am not able to figure out why this is happening since I am updating my template (roiImg) with each iteration. The matchvalue from minmax()function is returning the same point (coordinates) every-time. Test video is availbale at: http://www.youtube.com/watch?v=vpnkk7N2E0Q&feature=youtu.bePlease look into it and guide ahead...thanks a lot!

未跟踪更新的模板。我无法弄清楚为什么会发生这种情况，因为我每次迭代都在更新我的模板 (roiImg)。minmax()函数的匹配值每次都返回相同的点（坐标）。测试视频可在以下网址获得：http: //www.youtube.com/watch?v=vpnkk7N2E0Q&feature=youtu.be 请仔细研究并提前指导...非常感谢！

回答by Alessandro Jacopson

I get your original code from this revision of your question: https://stackoverflow.com/revisions/20180073/3

我从你的问题的这个修订版中得到你的原始代码：https: //stackoverflow.com/revisions/20180073/3

I made the smallest change to your original code, my resulting code is the following:

我对您的原始代码进行了最小的更改，结果代码如下：

#include <iostream>
#include "opencv2/opencv.hpp"
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/objdetect/objdetect.hpp>

#include <sstream>


using namespace cv;
using namespace std;

Point point1, point2; /* vertical points of the bounding box */
int drag = 0;
Rect rect; /* bounding box */
Mat img, roiImg; /* roiImg - the part of the image in the bounding box */
int select_flag = 0;
bool go_fast = false;

Mat mytemplate;


///------- template matching -----------------------------------------------------------------------------------------------

Mat TplMatch( Mat &img, Mat &mytemplate )
{
  Mat result;

  matchTemplate( img, mytemplate, result, CV_TM_SQDIFF_NORMED );
  normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );

  return result;
}


///------- Localizing the best match with minMaxLoc ------------------------------------------------------------------------

Point minmax( Mat &result )
{
  double minVal, maxVal;
  Point  minLoc, maxLoc, matchLoc;

  minMaxLoc( result, &minVal, &maxVal, &minLoc, &maxLoc, Mat() );
  matchLoc = minLoc;

  return matchLoc;
}


///------- tracking --------------------------------------------------------------------------------------------------------

void track()
{
    if (select_flag)
    {
        //roiImg.copyTo(mytemplate);
//         select_flag = false;
        go_fast = true;
    }

//     imshow( "mytemplate", mytemplate ); waitKey(0);

    Mat result  =  TplMatch( img, mytemplate );
    Point match =  minmax( result ); 

    rectangle( img, match, Point( match.x + mytemplate.cols , match.y + mytemplate.rows ), CV_RGB(255, 255, 255), 0.5 );

    std::cout << "match: " << match << endl;

    /// latest match is the new template
    Rect ROI = cv::Rect( match.x, match.y, mytemplate.cols, mytemplate.rows );
    roiImg = img( ROI );
    roiImg.copyTo(mytemplate);
    imshow( "roiImg", roiImg ); //waitKey(0);
}


///------- MouseCallback function ------------------------------------------------------------------------------------------

void mouseHandler(int event, int x, int y, int flags, void *param)
{
    if (event == CV_EVENT_LBUTTONDOWN && !drag)
    {
        /// left button clicked. ROI selection begins
        point1 = Point(x, y);
        drag = 1;
    }

    if (event == CV_EVENT_MOUSEMOVE && drag)
    {
        /// mouse dragged. ROI being selected
        Mat img1 = img.clone();
        point2 = Point(x, y);
        rectangle(img1, point1, point2, CV_RGB(255, 0, 0), 3, 8, 0);
        imshow("image", img1);
    }

    if (event == CV_EVENT_LBUTTONUP && drag)
    {
        point2 = Point(x, y);
        rect = Rect(point1.x, point1.y, x - point1.x, y - point1.y);
        drag = 0;
        roiImg = img(rect);
        roiImg.copyTo(mytemplate);
//  imshow("MOUSE roiImg", roiImg); waitKey(0);
    }

    if (event == CV_EVENT_LBUTTONUP)
    {
        /// ROI selected
        select_flag = 1;
        drag = 0;
    }

}



///------- Main() ----------------------------------------------------------------------------------------------------------

int main()
{
    int k;
/*    
///open webcam
    VideoCapture cap(0);
    if (!cap.isOpened())
      return 1;*/

    ///open video file
    VideoCapture cap;
    cap.open( "Megamind.avi" );
    if ( !cap.isOpened() )
    {   cout << "Unable to open video file" << endl;    return -1;    }
/*    
    /// Set video to 320x240
     cap.set(CV_CAP_PROP_FRAME_WIDTH, 320);
     cap.set(CV_CAP_PROP_FRAME_HEIGHT, 240);*/

    cap >> img;
    GaussianBlur( img, img, Size(7,7), 3.0 );
    imshow( "image", img );

    while (1)
    {
        cap >> img;
        if ( img.empty() )
            break;

    // Flip the frame horizontally and add blur
    cv::flip( img, img, 1 );
    GaussianBlur( img, img, Size(7,7), 3.0 );

        if ( rect.width == 0 && rect.height == 0 )
            cvSetMouseCallback( "image", mouseHandler, NULL );
        else
            track();

        imshow("image", img);
//  waitKey(100);   k = waitKey(75);
    k = waitKey(go_fast ? 30 : 10000);
        if (k == 27)
            break;
    }

    return 0;
}

The video at https://www.youtube.com/watch?v=rBCopeneCosshows a test of the above program.

https://www.youtube.com/watch?v=rBCopeneCos 上的视频显示了对上述程序的测试。

I would avoid the use of global variable because I think they do not help in understanding where the problems lie; furthermore I also would pay attention to the shallow vs deep copy for OpenCV's Matclass, as 1''wrote in his answer:

我会避免使用全局变量，因为我认为它们无助于理解问题所在；此外，我还会关注 OpenCVMat类的浅拷贝与深拷贝，正如1''在他的回答中所写：

OpenCV's Matclass is simply a header for the actual image data, which it contains a pointer to. The operator=copies the pointer (and the other information in the header, like the image dimensions) so that both Mats share the same data. This means that modifying the data in one Mat also changes it in the other. This is called a "shallow" copy, since only the top layer (the header) is copied, not the lower layer (the data).

To make a copy of the underlying data (called a "deep copy"), use the clone()method. You can find information about it on the page that you linked to.

OpenCV 的Mat类只是实际图像数据的标题，它包含指向的指针。的operator=复制指针（以及在报头中的其他信息，如图像尺寸），使得两个垫共享同一数据。这意味着修改一个 Mat 中的数据也会更改另一个 Mat 中的数据。这称为“浅”复制，因为只复制顶层（标题），而不复制底层（数据）。

要制作底层数据的副本（称为“深度副本”），请使用该 clone()方法。您可以在链接到的页面上找到有关它的信息。

Edit about the drift:In comment Real-time template matching - OpenCV, C++, learnerasks about the tracking drift. Looking at the video https://www.youtube.com/watch?v=rBCopeneCoswe see that at the beginning of the video the program is tracking the girl's right eye while at 0:15 it starts to track the girl's eyebrows, at 0:19 it starts to track the boy's eyebrows and it never tracks anymore the girl's eye, for example at 0:27 it tracks the girl's right eyebrow while the girl's right eye is clearly visible in the image.

编辑关于漂移：在评论实时模板匹配 - OpenCV, C++ 中，学习者询问跟踪漂移。查看视频https://www.youtube.com/watch?v=rBCopeneCos我们看到，在视频的开头，程序正在跟踪女孩的右眼，而在 0:15 开始跟踪女孩的眉毛，在0:19 开始跟踪男孩的眉毛，不再跟踪女孩的眼睛，例如在 0:27 跟踪女孩的右眉毛，而女孩的右眼在图像中清晰可见。

This drift from tracking the eye to tracking the eyebrow is normal in a simple code as the one I posted and the explanation is quite simple: see the video at https://www.youtube.com/watch?v=sGHEu3u9XvI, the video starts with the tracking (contents of the black rectangle) of the playing card, then I remove the playing card from the scene and the tracking black rectangle "drifts" to the bottom left of the scene; after all we are continuosly updating the template and so the behavior is correct: the program stops to track the playing card and starts to track a white background and so you have the "drift"... in other words, your TplMatch()function will always return a valid resultimage and your current implementation of minmax()will always return a valid a minimum.

从跟踪眼睛到跟踪眉毛的这种漂移在我发布的一个简单代码中是正常的，并且解释非常简单：请参阅https://www.youtube.com/watch?v=sGHEu3u9XvI 上的视频，视频从扑克牌的跟踪（黑色矩形的内容）开始，然后我从场景中取出扑克牌，跟踪黑色矩形“漂移”到场景的左下角；毕竟我们不断更新模板，所以行为是正确的：程序停止跟踪扑克牌并开始跟踪白色背景，所以你有“漂移”......换句话说，你的TplMatch()函数将始终返回一个有效的result图像和您当前的实现minmax()将始终返回一个有效的最小值。

回答by Alessandro Jacopson

You can follow the OpenCV tutorial "Template Matching". Your trackfunction may contain the code to find the template in the current frame; a simple code is based on the matchTemplateand minMaxLocfunctions.

您可以按照 OpenCV 教程“模板匹配”进行操作。您的track函数可能包含在当前帧中查找模板的代码；一个简单的代码基于matchTemplate和minMaxLoc函数。

The interesting issue related to the "real-time" part of your question is to succeed in finding the match, if present, within the time between the current frame and the next one.

与问题的“实时”部分相关的有趣问题是在当前帧和下一帧之间的时间内成功找到匹配项（如果存在）。

Edit:

编辑：

The following quick-and-dirty code and the video at http://www.youtube.com/watch?v=vpnkk7N2E0Q&feature=youtu.beshows what I mean for tracking.

以下快速而肮脏的代码和http://www.youtube.com/watch?v=vpnkk7N2E0Q&feature=youtu.be 上的视频显示了我对跟踪的意思。

Since I do not have a webcam I slightly modified your code to just use a video, this one https://code.ros.org/trac/opencv/export/7237/trunk/opencv/samples/cpp/tutorial_code/HighGUI/video-input-psnr-ssim/video/Megamind.avi

由于我没有网络摄像头，我稍微修改了您的代码以仅使用视频，这个https://code.ros.org/trac/opencv/export/7237/trunk/opencv/samples/cpp/tutorial_code/HighGUI/视频输入-psnr-ssim/video/Megamind.avi

I then add trackfunction and some logic to slow down the video until I choose a ROI and after that playing the video at normal speed.

然后我添加track函数和一些逻辑来减慢视频速度，直到我选择一个 ROI，然后以正常速度播放视频。

#include <iostream>
#include "opencv2/opencv.hpp"
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/objdetect/objdetect.hpp>

#include <sstream>


using namespace cv;
using namespace std;

Point point1, point2; /* vertical points of the bounding box */
int drag = 0;
Rect rect; /* bounding box */
Mat img, roiImg; /* roiImg - the part of the image in the bounding box */
int select_flag = 0;
bool go_fast = false;

Mat mytemplate;

void track(cv::Mat &img, const cv::Mat &templ, const cv::Rect &r )
{
    static int n = 0;

    if (select_flag)
    {
        templ.copyTo(mytemplate);
        select_flag = false;
        go_fast = true;
    }


    cv::Mat result;
    /// Do the Matching and Normalize
    matchTemplate( img, mytemplate, result, CV_TM_SQDIFF_NORMED );
    normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );

    /// Localizing the best match with minMaxLoc
    double minVal; double maxVal; Point minLoc; Point maxLoc;
    Point matchLoc;

    minMaxLoc( result, &minVal, &maxVal, &minLoc, &maxLoc, Mat() );
    matchLoc = minLoc;

    rectangle( img, matchLoc, Point( matchLoc.x + mytemplate.cols , matchLoc.y + mytemplate.rows ), CV_RGB(255, 255, 255), 3 );

    std::cout << matchLoc << "\n";
}

///MouseCallback function

void mouseHandler(int event, int x, int y, int flags, void *param)
{
    if (event == CV_EVENT_LBUTTONDOWN && !drag)
    {
        /* left button clicked. ROI selection begins */
        point1 = Point(x, y);
        drag = 1;
    }

    if (event == CV_EVENT_MOUSEMOVE && drag)
    {
        /* mouse dragged. ROI being selected */
        Mat img1 = img.clone();
        point2 = Point(x, y);
        rectangle(img1, point1, point2, CV_RGB(255, 0, 0), 3, 8, 0);
        imshow("image", img1);
    }

    if (event == CV_EVENT_LBUTTONUP && drag)
    {
        point2 = Point(x, y);
        rect = Rect(point1.x, point1.y, x - point1.x, y - point1.y);
        drag = 0;
        roiImg = img(rect);
    }

    if (event == CV_EVENT_LBUTTONUP)
    {
        /* ROI selected */
        select_flag = 1;
        drag = 0;
    }

}


///Main function

int main()
{
    int k;
    /*
        VideoCapture cap(0);
        if (!cap.isOpened())
        return 1;
    */
    VideoCapture cap;
    //cap.open("~/Downloads/opencv-2.4.4/samples/cpp/tutorial_code/HighGUI/video-input-psnr-ssim/video/Megamind.avi");
    cap.open("./Megamind.avi");
    if (!cap.isOpened())
    {
        printf("Unable to open video file\n");
        return -1;
    }

    /*
        // Set video to 320x240
        cap.set(CV_CAP_PROP_FRAME_WIDTH, 320);
        cap.set(CV_CAP_PROP_FRAME_HEIGHT, 240);
        */

    cap >> img;
    imshow("image", img);

    while (1)
    {
        cap >> img;
        if (img.empty())
            break;

        if (rect.width == 0 && rect.height == 0)
            cvSetMouseCallback("image", mouseHandler, NULL);
        else
            track(img, roiImg, rect);

        if (select_flag == 1)
            imshow("Template", roiImg);

        imshow("image", img);
        k = waitKey(go_fast ? 30 : 10000);
        if (k == 27)
            break;

    }


    return 0;
}

回答by Alessandro Jacopson

You can also have a general introduction to the subject starting from this wikipedia page http://en.wikipedia.org/wiki/Video_tracking

您还可以从这个维基百科页面http://en.wikipedia.org/wiki/Video_tracking开始对该主题的一般介绍