Python 多张图像的 OpenCV 特征匹配

OpenCV feature matching for multiple images

提问by Jafu

How can I optimise the SIFT feature matching for many pictures using FLANN?

如何使用FLANN优化多张图片的SIFT特征匹配？

I have a working example taken from the Python OpenCV docs. However this is comparing one image with another and it's slow. I need it to search for features matching in a series of images (a few thousands) and I need it to be faster.

我有一个来自 Python OpenCV 文档的工作示例。然而，这是将一张图像与另一张图像进行比较，而且速度很慢。我需要它来搜索一系列图像（几千张）中匹配的特征，我需要它更快。

My current idea:

我目前的想法：

1. Run through all the images and save the features. How?
2. Compare an image from a camera with this above base, and find the correct one. How?
3. Give me the result, matching image or something.

1. 运行所有图像并保存特征。如何？
2. 将来自相机的图像与上述基础进行比较，并找到正确的图像。如何？
3. 给我结果，匹配图像什么的。

http://docs.opencv.org/trunk/doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html

http://docs.opencv.org/trunk/doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html

import sys # For debugging only
import numpy as np
import cv2
from matplotlib import pyplot as plt

MIN_MATCH_COUNT = 10

img1 = cv2.imread('image.jpg',0) # queryImage
img2 = cv2.imread('target.jpg',0) # trainImage

# Initiate SIFT detector
sift = cv2.SIFT()

# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)

FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)

flann = cv2.FlannBasedMatcher(index_params, search_params)

matches = flann.knnMatch(des1,des2,k=2)

# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
    if m.distance MIN_MATCH_COUNT:
    src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
    dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)

    M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
    matchesMask = mask.ravel().tolist()

    h,w = img1.shape
    pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
    dst = cv2.perspectiveTransform(pts,M)

    img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)

else:
    print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT)
    matchesMask = None

draw_params = dict(matchColor = (0,255,0), # draw matches in green color
                   singlePointColor = None,
                   matchesMask = matchesMask, # draw only inliers
                   flags = 2)

img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)

plt.imshow(img3, 'gray'),plt.show()

---

---

UPDATE

更新

After trying out many things I might have come closer to the solution now. I hope it's possible to build the index and then search in it like this:

在尝试了很多事情之后，我现在可能更接近解决方案了。我希望可以建立索引，然后像这样搜索：

flann_params = dict(algorithm=1, trees=4)
flann = cv2.flann_Index(npArray, flann_params)
idx, dist = flann.knnSearch(queryDes, 1, params={})

However I still haven't managed to build an accepted npArray to the flann_Index parameter.

但是，我仍然没有设法为 flann_Index 参数构建一个可接受的 npArray。

loop through all images as image:
  npArray.append(sift.detectAndCompute(image, None))
npArray = np.array(npArray)

采纳答案by Jafu

I never solved this in Python, however I switched environment to C++ where you get more OpenCV examples and don't have to use a wrapper with less documentation.

我从未在 Python 中解决过这个问题，但是我将环境切换到 C++，在那里您可以获得更多 OpenCV 示例，而不必使用文档较少的包装器。

An example on the issue I had with matching in multiple files can be found here: https://github.com/Itseez/opencv/blob/2.4/samples/cpp/matching_to_many_images.cpp

关于我在多个文件中匹配的问题的一个例子可以在这里找到：https: //github.com/Itseez/opencv/blob/2.4/samples/cpp/matching_to_many_images.cpp

回答by stanleyxu2005

Here are several pieces of my advice:

以下是我的几条建议：

1. You should reduce the amount of point data by using proper techniques.
2. Calculate the reference image repeatedly is a waste. You should persistent all calculated reference.
3. Do not put the calculate on a mobile device. You'd better upload the calculated reference of a captured image to a powerful server and do the searching there.

1. 您应该使用适当的技术来减少点数据的数量。
2. 重复计算参考图像是一种浪费。您应该持久化所有计算参考。
3. 不要将计算放在移动设备上。您最好将所捕获图像的计算参考上传到强大的服务器并在那里进行搜索。

This is a very interesting topic. My ears are opening too.

这是一个非常有趣的话题。我的耳朵也开了。

回答by rbaleksandar

Along with the reply of @stanleyxu2005 I'd like to add some tips as to how to do the whole matching itself since I'm currently working of such a thing.

随着@stanleyxu2005 的回复，我想添加一些关于如何完成整个匹配本身的提示，因为我目前正在处理这样的事情。

1. I strongly recommend to create some custom class that wraps around the cv::Mat but also stores various other essential pieces of data. In my case I have an ImageContainer store the original image (that I will use for the final stitching), the processed one (grayscaled, undistorted etc.), its keypoints and the descriptors for those. By doing so you can access all the matching-relevant information in a pretty well organized well. You can either implement the keypoint extraction and descriptor generation in it or do that outside the class and just store the results in that container.
2. Store all image containers in some kind of a structure (vector is usually a good choice) for easy access.
3. I also created a class called ImageMultiMatchContainer, which stores a pointer to a given query image (all images are query images), a vector with pointers to all train images (for a single query image of the image set all others are train images) that were matched to it and also a vector of the match vectors for each of those matches. Here I stumbled across a storage issue namely - first you have to skip matching of an image with itself because it is pointless and second you have the problem of comparing two images two times and thus generating a considerable overhead if you have a lot of images. The second problem is due to the fact that we iterate through all images (query images) and compare them to the rest in the set (train images). At some point we have image X (query) matched with image Y (train), but later we also have image Y (now query) matched with image X (now train). As you can see this is also pointless since it's basically matching the same pair of images twice. This can be solved (currently working on this) by creating a class (MatchContainer) that stores a pointer to each of the two images in a matched pair and also the match vector. You store this in a central location (in my case this is my matcher class) and for each image as query image you check the list of matched images of the train image. If it's empty then you create a new MatchContainer and add it to the rest of the MatchContainers. If it's not then you look in it and see if the current query image is not present there (comparing pointers is a fast operation). If it is then you just pass the pointer to that MatchContainer's vector item that stores the matches for those two images. If that is not the case, you do as if it's empty and create a new MatchContainer etc. MatchingContainers should be stored in a data structure with a small access times since you will be looking at them a lot and iterating from start to end costs too much time. I'm considering using a map but maybe a tree of some sort can offer some advantages as well.
4. The homography estimation is a very tricky part. Here I recommend you look at bundle block adjustment. I saw that the stitcher class in OpenCV has a BundleBase-class but haven't tested it yet to see what's in it.

1. 我强烈建议创建一些自定义类，它环绕 cv::Mat 但也存储各种其他基本数据。在我的例子中，我有一个 ImageContainer 存储原始图像（我将用于最终拼接）、处理后的图像（灰度、未失真等）、它的关键点和这些的描述符。通过这样做，您可以以组织良好的方式访问所有匹配相关信息。您可以在其中实现关键点提取和描述符生成，也可以在类之外执行此操作并将结果存储在该容器中。
2. 将所有图像容器存储在某种结构中（矢量通常是一个不错的选择）以便于访问。
3. 我还创建了一个名为 ImageMultiMatchContainer 的类，它存储一个指向给定查询图像的指针（所有图像都是查询图像），一个指向所有训练图像的向量（对于图像集的单个查询图像，所有其他图像都是训练图像）与它匹配，以及每个匹配项的匹配向量的向量。在这里，我偶然发现了一个存储问题 - 首先，您必须跳过图像与其自身的匹配，因为它毫无意义，其次，您会遇到两次比较两个图像的问题，如果您有很多图像，则会产生相当大的开销。第二个问题是由于我们遍历所有图像（查询图像）并将它们与集合中的其余图像（训练图像）进行比较。在某些时候，我们将图像 X（查询）与图像 Y（火车）相匹配，但后来我们也有图像 Y（现在查询）与图像 X（现在训练）匹配。正如您所看到的，这也是毫无意义的，因为它基本上是将同一对图像匹配两次。这可以通过创建一个类 (MatchContainer) 来解决（目前正在解决这个问题），该类存储指向匹配对中两个图像中每一个的指针以及匹配向量。您将其存储在一个中心位置（在我的情况下，这是我的匹配器类），对于作为查询图像的每个图像，您检查训练图像的匹配图像列表。如果它为空，则创建一个新的 MatchContainer 并将其添加到其余的 MatchContainer。如果不是，则查看它并查看当前查询图像是否不存在（比较指针是一种快速操作）。如果是，那么您只需将指针传递给该 MatchContainer' s 向量项，用于存储这两个图像的匹配项。如果情况并非如此，您就好像它是空的一样，然后创建一个新的 MatchContainer 等。 MatchingContainers 应该存储在访问时间较短的数据结构中，因为您将大量查看它们并从开始到结束成本也进行迭代很多时间。我正在考虑使用地图，但也许某种树也可以提供一些优势。
4. 单应性估计是一个非常棘手的部分。在这里，我建议您查看捆绑块调整。我看到 OpenCV 中的拼接器类有一个 BundleBase 类，但尚未对其进行测试以查看其中的内容。

A general recommendation is to look at the stitching process in OpenCV and read the source code. The stitching pipeline is a straight forward set of processes and you just have to see how exactly you can implement the single steps.

一般建议是查看 OpenCV 中的拼接过程并阅读源代码。拼接管道是一组直接的过程，您只需要了解如何准确地实现单个步骤。