C++ 引入旋转/缩放不变性后，OpenCV Orb 找不到匹配项
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/9539473/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
OpenCV Orb not finding matches once rotation/scale invariances are introduced
提问by KLowe
I am working on a project using the Orb feature detector in OpenCV 2.3.1 . I am finding matches between 8 different images, 6 of which are very similar (20 cm difference in camera position, along a linear slider so there is no scale or rotational variance), and then 2 images taken from about a 45 degree angle from either side. My code is finding plenty of accurate matches between the very similar images, but few to none for the images taken from a more different perspective. I've included what I think are the pertinent parts of my code, please let me know if you need more information.
我正在使用 OpenCV 2.3.1 中的 Orb 特征检测器进行一个项目。我找到了 8 张不同图像之间的匹配，其中 6 张非常相似（相机位置相差 20 厘米，沿着线性滑块，因此没有比例或旋转变化），然后从大约 45 度角拍摄的 2 张图像来自边。我的代码在非常相似的图像之间找到了大量准确的匹配，但对于从更不同的角度拍摄的图像，几乎没有。我已经包含了我认为代码的相关部分，如果您需要更多信息，请告诉我。
// set parameters
int numKeyPoints = 1500;
float distThreshold = 15.0;
//instantiate detector, extractor, matcher
detector = new cv::OrbFeatureDetector(numKeyPoints);
extractor = new cv::OrbDescriptorExtractor;
matcher = new cv::BruteForceMatcher<cv::HammingLUT>;
//Load input image detect keypoints
cv::Mat img1;
std::vector<cv::KeyPoint> img1_keypoints;
cv::Mat img1_descriptors;
cv::Mat img2;
std::vector<cv::KeyPoint> img2_keypoints
cv::Mat img2_descriptors;
img1 = cv::imread(fList[0].string(), CV_LOAD_IMAGE_GRAYSCALE);
img2 = cv::imread(fList[1].string(), CV_LOAD_IMAGE_GRAYSCALE);
detector->detect(img1, img1_keypoints);
detector->detect(img2, img2_keypoints);
extractor->compute(img1, img1_keypoints, img1_descriptors);
extractor->compute(img2, img2_keypoints, img2_descriptors);
//Match keypoints using knnMatch to find the single best match for each keypoint
//Then cull results that fall below given distance threshold
std::vector<std::vector<cv::DMatch> > matches;
matcher->knnMatch(img1_descriptors, img2_descriptors, matches, 1);
int matchCount=0;
for (int n=0; n<matches.size(); ++n) {
if (matches[n].size() > 0){
if (matches[n][0].distance > distThreshold){
matches[n].erase(matches[n].begin());
}else{
++matchCount;
}
}
}
回答by KLowe
I ended up getting enough useful matches by changing my process for filtering matches. My previous method was discarding a lot of good matches based solely on their distance value. This RobustMatcherclass that I found in the OpenCV2 Computer Vision Application Programming Cookbookended up working great. Now that all of my matches are accurate, I've been able to get good enough results by bumping up the number of keypoints that the ORB detector is looking. Using the RobustMatcherwith SIFT or SURF still gives much better results, but I'm getting usable data with ORB now.
通过更改过滤匹配的过程，我最终获得了足够多的有用匹配。我以前的方法是仅根据它们的距离值丢弃很多好的匹配项。这个RobustMatcher类，我在发现OpenCV2计算机视觉应用编程食谱结束了伟大的工作。现在我的所有匹配都准确无误，通过增加 ORB 检测器正在寻找的关键点的数量，我已经能够获得足够好的结果。使用RobustMatcherSIFT 或 SURF 仍然可以提供更好的结果，但我现在使用 ORB 获取可用数据。
//RobustMatcher class taken from OpenCV2 Computer Vision Application Programming Cookbook Ch 9
class RobustMatcher {
private:
// pointer to the feature point detector object
cv::Ptr<cv::FeatureDetector> detector;
// pointer to the feature descriptor extractor object
cv::Ptr<cv::DescriptorExtractor> extractor;
// pointer to the matcher object
cv::Ptr<cv::DescriptorMatcher > matcher;
float ratio; // max ratio between 1st and 2nd NN
bool refineF; // if true will refine the F matrix
double distance; // min distance to epipolar
double confidence; // confidence level (probability)
public:
RobustMatcher() : ratio(0.65f), refineF(true),
confidence(0.99), distance(3.0) {
// ORB is the default feature
detector= new cv::OrbFeatureDetector();
extractor= new cv::OrbDescriptorExtractor();
matcher= new cv::BruteForceMatcher<cv::HammingLUT>;
}
// Set the feature detector
void setFeatureDetector(
cv::Ptr<cv::FeatureDetector>& detect) {
detector= detect;
}
// Set the descriptor extractor
void setDescriptorExtractor(
cv::Ptr<cv::DescriptorExtractor>& desc) {
extractor= desc;
}
// Set the matcher
void setDescriptorMatcher(
cv::Ptr<cv::DescriptorMatcher>& match) {
matcher= match;
}
// Set confidence level
void setConfidenceLevel(
double conf) {
confidence= conf;
}
//Set MinDistanceToEpipolar
void setMinDistanceToEpipolar(
double dist) {
distance= dist;
}
//Set ratio
void setRatio(
float rat) {
ratio= rat;
}
// Clear matches for which NN ratio is > than threshold
// return the number of removed points
// (corresponding entries being cleared,
// i.e. size will be 0)
int ratioTest(std::vector<std::vector<cv::DMatch> >
&matches) {
int removed=0;
// for all matches
for (std::vector<std::vector<cv::DMatch> >::iterator
matchIterator= matches.begin();
matchIterator!= matches.end(); ++matchIterator) {
// if 2 NN has been identified
if (matchIterator->size() > 1) {
// check distance ratio
if ((*matchIterator)[0].distance/
(*matchIterator)[1].distance > ratio) {
matchIterator->clear(); // remove match
removed++;
}
} else { // does not have 2 neighbours
matchIterator->clear(); // remove match
removed++;
}
}
return removed;
}
// Insert symmetrical matches in symMatches vector
void symmetryTest(
const std::vector<std::vector<cv::DMatch> >& matches1,
const std::vector<std::vector<cv::DMatch> >& matches2,
std::vector<cv::DMatch>& symMatches) {
// for all matches image 1 -> image 2
for (std::vector<std::vector<cv::DMatch> >::
const_iterator matchIterator1= matches1.begin();
matchIterator1!= matches1.end(); ++matchIterator1) {
// ignore deleted matches
if (matchIterator1->size() < 2)
continue;
// for all matches image 2 -> image 1
for (std::vector<std::vector<cv::DMatch> >::
const_iterator matchIterator2= matches2.begin();
matchIterator2!= matches2.end();
++matchIterator2) {
// ignore deleted matches
if (matchIterator2->size() < 2)
continue;
// Match symmetry test
if ((*matchIterator1)[0].queryIdx ==
(*matchIterator2)[0].trainIdx &&
(*matchIterator2)[0].queryIdx ==
(*matchIterator1)[0].trainIdx) {
// add symmetrical match
symMatches.push_back(
cv::DMatch((*matchIterator1)[0].queryIdx,
(*matchIterator1)[0].trainIdx,
(*matchIterator1)[0].distance));
break; // next match in image 1 -> image 2
}
}
}
}
// Identify good matches using RANSAC
// Return fundemental matrix
cv::Mat ransacTest(
const std::vector<cv::DMatch>& matches,
const std::vector<cv::KeyPoint>& keypoints1,
const std::vector<cv::KeyPoint>& keypoints2,
std::vector<cv::DMatch>& outMatches) {
// Convert keypoints into Point2f
std::vector<cv::Point2f> points1, points2;
cv::Mat fundemental;
for (std::vector<cv::DMatch>::
const_iterator it= matches.begin();
it!= matches.end(); ++it) {
// Get the position of left keypoints
float x= keypoints1[it->queryIdx].pt.x;
float y= keypoints1[it->queryIdx].pt.y;
points1.push_back(cv::Point2f(x,y));
// Get the position of right keypoints
x= keypoints2[it->trainIdx].pt.x;
y= keypoints2[it->trainIdx].pt.y;
points2.push_back(cv::Point2f(x,y));
}
// Compute F matrix using RANSAC
std::vector<uchar> inliers(points1.size(),0);
if (points1.size()>0&&points2.size()>0){
cv::Mat fundemental= cv::findFundamentalMat(
cv::Mat(points1),cv::Mat(points2), // matching points
inliers, // match status (inlier or outlier)
CV_FM_RANSAC, // RANSAC method
distance, // distance to epipolar line
confidence); // confidence probability
// extract the surviving (inliers) matches
std::vector<uchar>::const_iterator
itIn= inliers.begin();
std::vector<cv::DMatch>::const_iterator
itM= matches.begin();
// for all matches
for ( ;itIn!= inliers.end(); ++itIn, ++itM) {
if (*itIn) { // it is a valid match
outMatches.push_back(*itM);
}
}
if (refineF) {
// The F matrix will be recomputed with
// all accepted matches
// Convert keypoints into Point2f
// for final F computation
points1.clear();
points2.clear();
for (std::vector<cv::DMatch>::
const_iterator it= outMatches.begin();
it!= outMatches.end(); ++it) {
// Get the position of left keypoints
float x= keypoints1[it->queryIdx].pt.x;
float y= keypoints1[it->queryIdx].pt.y;
points1.push_back(cv::Point2f(x,y));
// Get the position of right keypoints
x= keypoints2[it->trainIdx].pt.x;
y= keypoints2[it->trainIdx].pt.y;
points2.push_back(cv::Point2f(x,y));
}
// Compute 8-point F from all accepted matches
if (points1.size()>0&&points2.size()>0){
fundemental= cv::findFundamentalMat(
cv::Mat(points1),cv::Mat(points2), // matches
CV_FM_8POINT); // 8-point method
}
}
}
return fundemental;
}
// Match feature points using symmetry test and RANSAC
// returns fundemental matrix
cv::Mat match(cv::Mat& image1,
cv::Mat& image2, // input images
// output matches and keypoints
std::vector<cv::DMatch>& matches,
std::vector<cv::KeyPoint>& keypoints1,
std::vector<cv::KeyPoint>& keypoints2) {
// 1a. Detection of the SURF features
detector->detect(image1,keypoints1);
detector->detect(image2,keypoints2);
// 1b. Extraction of the SURF descriptors
cv::Mat descriptors1, descriptors2;
extractor->compute(image1,keypoints1,descriptors1);
extractor->compute(image2,keypoints2,descriptors2);
// 2. Match the two image descriptors
// Construction of the matcher
//cv::BruteForceMatcher<cv::L2<float>> matcher;
// from image 1 to image 2
// based on k nearest neighbours (with k=2)
std::vector<std::vector<cv::DMatch> > matches1;
matcher->knnMatch(descriptors1,descriptors2,
matches1, // vector of matches (up to 2 per entry)
2); // return 2 nearest neighbours
// from image 2 to image 1
// based on k nearest neighbours (with k=2)
std::vector<std::vector<cv::DMatch> > matches2;
matcher->knnMatch(descriptors2,descriptors1,
matches2, // vector of matches (up to 2 per entry)
2); // return 2 nearest neighbours
// 3. Remove matches for which NN ratio is
// > than threshold
// clean image 1 -> image 2 matches
int removed= ratioTest(matches1);
// clean image 2 -> image 1 matches
removed= ratioTest(matches2);
// 4. Remove non-symmetrical matches
std::vector<cv::DMatch> symMatches;
symmetryTest(matches1,matches2,symMatches);
// 5. Validate matches using RANSAC
cv::Mat fundemental= ransacTest(symMatches,
keypoints1, keypoints2, matches);
// return the found fundemental matrix
return fundemental;
}
};
// set parameters
int numKeyPoints = 1500;
//Instantiate robust matcher
RobustMatcher rmatcher;
//instantiate detector, extractor, matcher
detector = new cv::OrbFeatureDetector(numKeyPoints);
extractor = new cv::OrbDescriptorExtractor;
matcher = new cv::BruteForceMatcher<cv::HammingLUT>;
rmatcher.setFeatureDetector(detector);
rmatcher.setDescriptorExtractor(extractor);
rmatcher.setDescriptorMatcher(matcher);
//Load input image detect keypoints
cv::Mat img1;
std::vector<cv::KeyPoint> img1_keypoints;
cv::Mat img1_descriptors;
cv::Mat img2;
std::vector<cv::KeyPoint> img2_keypoints
cv::Mat img2_descriptors;
std::vector<std::vector<cv::DMatch> > matches;
img1 = cv::imread(fList[0].string(), CV_LOAD_IMAGE_GRAYSCALE);
img2 = cv::imread(fList[1].string(), CV_LOAD_IMAGE_GRAYSCALE);
rmatcher.match(img1, img2, matches, img1_keypoints, img2_keypoints);
回答by snalx
I had a similar problem with opencv python and came here via google.
我在使用 opencv python 时遇到了类似的问题，并通过谷歌来到这里。
To solve my problem I wrote python code for matching-filtering based on @KLowes solution. I will share it here in case someone else has the same problem:
为了解决我的问题，我根据@KLowes 解决方案编写了用于匹配过滤的 python 代码。我会在这里分享，以防其他人遇到同样的问题：
""" Clear matches for which NN ratio is > than threshold """
def filter_distance(matches):
dist = [m.distance for m in matches]
thres_dist = (sum(dist) / len(dist)) * ratio
sel_matches = [m for m in matches if m.distance < thres_dist]
#print '#selected matches:%d (out of %d)' % (len(sel_matches), len(matches))
return sel_matches
""" keep only symmetric matches """
def filter_asymmetric(matches, matches2, k_scene, k_ftr):
sel_matches = []
for match1 in matches:
for match2 in matches2:
if match1.queryIdx < len(k_ftr) and match2.queryIdx < len(k_scene) and \
match2.trainIdx < len(k_ftr) and match1.trainIdx < len(k_scene) and \
k_ftr[match1.queryIdx] == k_ftr[match2.trainIdx] and \
k_scene[match1.trainIdx] == k_scene[match2.queryIdx]:
sel_matches.append(match1)
break
return sel_matches
def filter_ransac(matches, kp_scene, kp_ftr, countIterations=2):
if countIterations < 1 or len(kp_scene) < minimalCountForHomography:
return matches
p_scene = []
p_ftr = []
for m in matches:
p_scene.append(kp_scene[m.queryIdx].pt)
p_ftr.append(kp_ftr[m.trainIdx].pt)
if len(p_scene) < minimalCountForHomography:
return None
F, mask = cv2.findFundamentalMat(np.float32(p_ftr), np.float32(p_scene), cv2.FM_RANSAC)
sel_matches = []
for m, status in zip(matches, mask):
if status:
sel_matches.append(m)
#print '#ransac selected matches:%d (out of %d)' % (len(sel_matches), len(matches))
return filter_ransac(sel_matches, kp_scene, kp_ftr, countIterations-1)
def filter_matches(matches, matches2, k_scene, k_ftr):
matches = filter_distance(matches)
matches2 = filter_distance(matches2)
matchesSym = filter_asymmetric(matches, matches2, k_scene, k_ftr)
if len(k_scene) >= minimalCountForHomography:
return filter_ransac(matchesSym, k_scene, k_ftr)
To filter matches filter_matches(matches, matches2, k_scene, k_ftr)has to be called where matches, matches2represent matches obtained by orb-matcher and k_scene, k_ftrare corresponding keypoints.
filter_matches(matches, matches2, k_scene, k_ftr)必须调用过滤匹配，其中matches, matches2表示由 orb-matcher 获得的匹配并且k_scene, k_ftr是相应的关键点。
回答by Rui Marques
I don't think there is anything very wrong with your code. From my experience opencv's ORB is sensitive to scale variations.
我认为您的代码没有任何问题。根据我的经验，opencv 的 ORB 对比例变化很敏感。
You can probably confirm this with a small test, make some images with rotation only and some with scale variations only. The rotation ones will probably match fine but the scale ones won't (i think decreasing scale is the worst).
您可以通过一个小测试来确认这一点，制作一些仅旋转的图像和一些仅具有比例变化的图像。旋转的可能会匹配得很好，但比例的不会（我认为减少比例是最糟糕的）。
I also advise you to try the opencv version from the trunk (see opencv's site for compile instructions), ORB as been updated since 2.3.1 and performs a little better but still has those scale problems.
我还建议您尝试使用主干中的 opencv 版本（有关编译说明，请参阅 opencv 的站点），ORB 自 2.3.1 以来已更新，性能稍好一些，但仍然存在规模问题。
