java 将 SIFT 用于增强现实

Using SIFT for Augmented Reality

提问by Leo Jweda

I've come across MANY AR libraries/SDKs/APIs, all of them are marker-based, until I found this video, from the description and the comments, it looks like he's using SIFT to detect the object and follow it around.

我遇到过许多 AR 库/SDK/API，它们都是基于标记的，直到我找到这个视频，从描述和评论中，看起来他正在使用 SIFT 来检测对象并跟踪它。

I need to do that for Android, so I'm gonna need a full implementation of SIFT in pure Java.

我需要为 Android 做到这一点，所以我需要在纯 Java 中完全实现 SIFT。

I'm willing to do that but I need to know how SIFT is used for augmented reality first.

我愿意这样做，但我首先需要知道 SIFT 是如何用于增强现实的。

I could make use of any information you give.

我可以利用你提供的任何信息。

回答by alkar

In my opinion, trying to implement SIFTfor a portable device is madness. SIFT is an image feature extraction algorithm, which includes complex math and certainly requires a lot of computing power. SIFT is also patented.

在我看来，尝试为便携式设备实施SIFT是疯狂的。SIFT是一种图像特征提取算法，其中包含复杂的数学运算，当然需要大量的计算能力。SIFT 也获得了专利。

Still, if you indeed want to go forth with this task, you should do quite some research at first. You need to check things like:

不过，如果您确实想继续执行此任务，则应首先进行一些研究。您需要检查以下内容：

• Any variants of SIFT that enhance performance, including different algorithms all around
• I would recommend looking into SURFwhich is very robust and much more faster (but still one of those scary algorithms)
• Android NDK(I'll explain later)
• Lots and lots of publications

• 任何提高性能的 SIFT 变体，包括不同的算法
• 我建议研究SURF，它非常强大且速度更快（但仍然是那些可怕的算法之一）
• Android NDK（我稍后会解释）
• 大量的出版物

Why Android NDK? Because you'll probably have a much more significant performance gain by implementing the algorithm in a C library that's being used by your Java application.

为什么选择 Android NDK？因为通过在 Java 应用程序正在使用的 C 库中实现算法，您可能会获得更显着的性能提升。

Before starting anything, make sure you do that research cause it would be a pity to realize halfway that the image feature extraction algorithms are just too much for an Android phone. It's a serious endeavor in itself implementing such an algorithm that provides good results and runs in an acceptable amount of time, let alone using it to create an AR application.

在开始任何事情之前，请确保您进行了研究，因为中途意识到图像特征提取算法对于 Android 手机来说太多了，这将是一种遗憾。实现这样一种算法本身就是一项认真的努力，该算法可提供良好的结果并在可接受的时间内运行，更不用说使用它来创建 AR 应用程序了。

As in how you would use that for AR, I guess that the descriptor you get from running the algorithm on an image would have to be matched against with data saved in a central database. Then the results can be displayed to the user. The features of an image gathered from SURF are supposed to describe it such as that it can be then identified using those. I'm not really experienced on doing that but there's always resources on the web. You'd probably wanna start with generic stuff such as Object Recognition.

至于如何将其用于 AR，我猜您在图像上运行算法所获得的描述符必须与保存在中央数据库中的数据相匹配。然后可以将结果显示给用户。从 SURF 收集的图像的特征应该描述它，然后可以使用这些特征来识别它。我在这方面并没有真正的经验，但网络上总是有资源。你可能想从通用的东西开始，比如Object Recognition。

Best of luck :)

祝你好运:)

回答by Leo Jweda

If I where you, I'd look into how (and why) the SIFT feature works (as was said, its wikipedia-page offers a good cochise explanation, and for more details check the science paper (which is linked to at wikipedia)), and then build your own variant that suits your taste; i.e. has the optimal balance between performance and cpu-load, needed for your application.

如果我在哪里，我会研究 SIFT 功能如何（以及为什么）工作（如前所述，其维基百科页面提供了一个很好的解释，有关更多详细信息，请查看科学论文（链接到维基百科） )，然后构建适合您口味的自己的变体；即在性能和 CPU 负载之间达到最佳平衡，这是您的应用程序所需的。

For instance, I think Gaussian smoothing might be replaced by some faster way of smoothing.

例如，我认为高斯平滑可能会被一些更快的平滑方式所取代。

Also, when you build your own variant, you don't have anything to do with patents (there already are lots of variants, like GLOH).

此外，当您构建自己的变体时，您与专利没有任何关系（已经有很多变体，例如 GLOH）。

回答by mirror2image

I have tried SURF for 330Mhz Symbian mobile and it was still too slow even with all optimizations and lookup tables. And SIFT should be even more slow. Everyone using FAST for mobiles now. Anyway feature extraction is not a biggest problem. Correspondence and clearing false positive in it is more difficult. FAST link http://svr-www.eng.cam.ac.uk/~er258/work/fast.html

我已经为 330Mhz Symbian mobile 尝试了 SURF，但即使进行了所有优化和查找表，它仍然太慢。SIFT 应该更慢。现在每个人都在使用 FAST 手机。无论如何，特征提取不是最大的问题。其中的对应和清除误报比较困难。快速链接 http://svr-www.eng.cam.ac.uk/~er258/work/fast.html

回答by Hugo

I would recommend you to start by looking at the features already implemented in the OpenCV library, which include SURF, MSER and others:

我建议您首先查看 OpenCV 库中已经实现的功能，其中包括 SURF、MSER 等：

http://opencv.willowgarage.com/documentation/cpp/feature_detection.html

http://opencv.willowgarage.com/documentation/cpp/feature_detection.html

This might be enough for your application and are faster than SIFT. And as mentioned above, SIFT is patented.

这对于您的应用程序来说可能已经足够了，并且比 SIFT 更快。如上所述，SIFT 已获得专利。

Also, start by making performance tests in your mobile platform, just by extracting the features at every frame, this way you'll have an idea which ones can run real-time or not.

此外，首先在您的移动平台上进行性能测试，只需提取每一帧的特征，这样您就可以知道哪些可以实时运行。

回答by peakxu

Have you tried OpenCV's FAST implementation in the Android port? I've tested it out and it runs blazingly fast.

您是否尝试过在 Android 端口中使用 OpenCV 的 FAST 实现？我已经测试过了，它运行得非常快。

You can also compute reduced histogram descriptors around the detected FAST keypoints. I've heard of 3x3 rather than standard 4x4 of SIFT. That has a decent chance of working in real time if you optimize it heavily with NEON instructions. Otherwise, I'd recommend something fast and simple like sum of squared or absolute differences for a patch around the keypoints which are very fast.

您还可以计算检测到的 FAST 关键点周围的简化直方图描述符。我听说过 3x3 而不是标准的 4x4 SIFT。如果您使用 NEON 指令对其进行大量优化，那么它就有很大的机会实时工作。否则，我会推荐一些快速而简单的方法，例如非常快的关键点周围补丁的平方和或绝对差异。

SIFT is not a panacea. For real time video applications, it's usually overkill.

SIFT 不是灵丹妙药。对于实时视频应用程序，它通常是矫枉过正的。

回答by r0u1i

As always, Wikipedia is a good place to start from : http://en.wikipedia.org/wiki/Scale-invariant_feature_transform, but note that SIFT is patented.

与往常一样，维基百科是一个很好的起点：http: //en.wikipedia.org/wiki/Scale-invariant_feature_transform，但请注意 SIFT 已获得专利。