C++ 执行 cv::warpPerspective 以在一组 cv::Point 上进行假纠偏

Executing cv::warpPerspective for a fake deskewing on a set of cv::Point

提问by karlphillip

I'm trying to do a perspective transformationof a set of points in order to achieve a deskewingeffect:

我正在尝试对一组点进行透视变换，以实现偏斜效果：

http://nuigroup.com/?ACT=28&fid=27&aid=1892_H6eNAaign4Mrnn30Au8d

http://nuigroup.com/?ACT=28&fid=27&aid=1892_H6eNAaign4Mrnn30Au8d

I'm using the image below for tests, and the greenrectangle display the area of interest.

我正在使用下面的图像进行测试，绿色矩形显示感兴趣的区域。

I was wondering if it's possible to achieve the effect I'm hoping for using a simple combination of cv::getPerspectiveTransformand cv::warpPerspective. I'm sharing the source code I've written so far, but it doesn't work. This is the resulting image:

我在想，如果有可能实现，我希望使用的简单组合的效果cv::getPerspectiveTransform和cv::warpPerspective。我正在分享我到目前为止编写的源代码，但它不起作用。这是生成的图像：

So there is a vector<cv::Point>that defines the region of interest, but the points are not stored in any particular orderinside the vector, and that's something I can't change in the detection procedure. Anyway, later, the points in the vector are used to define a RotatedRect, which in turn is used to assemble cv::Point2f src_vertices[4];, one of the variables required by cv::getPerspectiveTransform().

因此，有一个vector<cv::Point>是定义感兴趣的区域，但点不存储在任何特定的顺序的载体里面，这件事情我不能在检测过程中发生改变。无论如何，稍后，向量中的点用于定义 a RotatedRect，而后者又用于组装cv::Point2f src_vertices[4];， 是 所需的变量之一cv::getPerspectiveTransform()。

My understanding about verticesand how they are organized might be one of the issues. I also think that using a RotatedRectis not the best ideato store the original points of the ROI, since the coordinates will changea little bit to fit into the rotated rectangle, and that's not very cool.

我对顶点及其组织方式的理解可能是问题之一。我也认为使用 a来存储 ROI 的原始点RotatedRect并不是最好的主意，因为坐标会稍微改变以适应旋转的矩形，这不是很酷。

#include <cv.h>
#include <highgui.h>
#include <iostream>

using namespace std;
using namespace cv;

int main(int argc, char* argv[])
{
    cv::Mat src = cv::imread(argv[1], 1);

    // After some magical procedure, these are points detect that represent 
    // the corners of the paper in the picture: 
    // [408, 69] [72, 2186] [1584, 2426] [1912, 291]
    vector<Point> not_a_rect_shape;
    not_a_rect_shape.push_back(Point(408, 69));
    not_a_rect_shape.push_back(Point(72, 2186));
    not_a_rect_shape.push_back(Point(1584, 2426));
    not_a_rect_shape.push_back(Point(1912, 291));

    // For debugging purposes, draw green lines connecting those points 
    // and save it on disk
    const Point* point = &not_a_rect_shape[0];
    int n = (int)not_a_rect_shape.size();
    Mat draw = src.clone();
    polylines(draw, &point, &n, 1, true, Scalar(0, 255, 0), 3, CV_AA);
    imwrite("draw.jpg", draw);

    // Assemble a rotated rectangle out of that info
    RotatedRect box = minAreaRect(cv::Mat(not_a_rect_shape));
    std::cout << "Rotated box set to (" << box.boundingRect().x << "," << box.boundingRect().y << ") " << box.size.width << "x" << box.size.height << std::endl;

    // Does the order of the points matter? I assume they do NOT.
    // But if it does, is there an easy way to identify and order 
    // them as topLeft, topRight, bottomRight, bottomLeft?
    cv::Point2f src_vertices[4];
    src_vertices[0] = not_a_rect_shape[0];
    src_vertices[1] = not_a_rect_shape[1];
    src_vertices[2] = not_a_rect_shape[2];
    src_vertices[3] = not_a_rect_shape[3];

    Point2f dst_vertices[4];
    dst_vertices[0] = Point(0, 0);
    dst_vertices[1] = Point(0, box.boundingRect().width-1);
    dst_vertices[2] = Point(0, box.boundingRect().height-1);
    dst_vertices[3] = Point(box.boundingRect().width-1, box.boundingRect().height-1);

    Mat warpMatrix = getPerspectiveTransform(src_vertices, dst_vertices);

    cv::Mat rotated;
    warpPerspective(src, rotated, warpMatrix, rotated.size(), INTER_LINEAR, BORDER_CONSTANT);

    imwrite("rotated.jpg", rotated);

    return 0;
}

Can someone help me fix this problem?

有人可以帮我解决这个问题吗？

采纳答案by Sam

So, first problem is corner order. They must be in the same order in both vectors. So, if in the first vector your order is:(top-left, bottom-left, bottom-right, top-right) , they MUST be in the same order in the other vector.

所以，第一个问题是角顺序。它们在两个向量中的顺序必须相同。因此，如果在第一个向量中您的顺序是:(左上、左下、右下、右上)，则它们在另一个向量中的顺序必须相同。

Second, to have the resulting image contain only the object of interest, you must set its width and height to be the same as resulting rectangle width and height. Do not worry, the src and dst images in warpPerspective can be different sizes.

其次，要使生成的图像仅包含感兴趣的对象，您必须将其宽度和高度设置为与生成的矩形宽度和高度相同。别担心，warpPerspective 中的 src 和 dst 图像可以是不同的大小。

Third, a performance concern. While your method is absolutely accurate, because you are doing only affine transforms (rotate, resize, deskew), mathematically, you can use the affine corespondent of your functions. They are much faster.

第三，性能问题。虽然您的方法绝对准确，因为您只进行仿射变换（旋转、调整大小、去歪斜），但在数学上，您可以使用函数的仿射对应。他们要快得多。

• getAffineTransform()

• warpAffine().

• getAffineTransform()

• 经仿射（）。

Important note: getAffine transform needs and expects ONLY 3 points, and the result matrix is 2-by-3, instead of 3-by-3.

重要说明：getAffine 变换只需要并且期望只有 3 个点，并且结果矩阵是 2×3，而不是 3×3。

How to make the result image have a different size than the input:

如何使结果图像具有与输入不同的大小：

cv::warpPerspective(src, dst, dst.size(), ... );

use

用

cv::Mat rotated;
cv::Size size(box.boundingRect().width, box.boundingRect().height);
cv::warpPerspective(src, dst, size, ... );

So here you are, and your programming assignment is over.

所以你到了，你的编程任务就结束了。

void main()
{
    cv::Mat src = cv::imread("r8fmh.jpg", 1);


    // After some magical procedure, these are points detect that represent 
    // the corners of the paper in the picture: 
    // [408, 69] [72, 2186] [1584, 2426] [1912, 291]

    vector<Point> not_a_rect_shape;
    not_a_rect_shape.push_back(Point(408, 69));
    not_a_rect_shape.push_back(Point(72, 2186));
    not_a_rect_shape.push_back(Point(1584, 2426));
    not_a_rect_shape.push_back(Point(1912, 291));

    // For debugging purposes, draw green lines connecting those points 
    // and save it on disk
    const Point* point = &not_a_rect_shape[0];
    int n = (int)not_a_rect_shape.size();
    Mat draw = src.clone();
    polylines(draw, &point, &n, 1, true, Scalar(0, 255, 0), 3, CV_AA);
    imwrite("draw.jpg", draw);

    // Assemble a rotated rectangle out of that info
    RotatedRect box = minAreaRect(cv::Mat(not_a_rect_shape));
    std::cout << "Rotated box set to (" << box.boundingRect().x << "," << box.boundingRect().y << ") " << box.size.width << "x" << box.size.height << std::endl;

    Point2f pts[4];

    box.points(pts);

    // Does the order of the points matter? I assume they do NOT.
    // But if it does, is there an easy way to identify and order 
    // them as topLeft, topRight, bottomRight, bottomLeft?

    cv::Point2f src_vertices[3];
    src_vertices[0] = pts[0];
    src_vertices[1] = pts[1];
    src_vertices[2] = pts[3];
    //src_vertices[3] = not_a_rect_shape[3];

    Point2f dst_vertices[3];
    dst_vertices[0] = Point(0, 0);
    dst_vertices[1] = Point(box.boundingRect().width-1, 0); 
    dst_vertices[2] = Point(0, box.boundingRect().height-1);

   /* Mat warpMatrix = getPerspectiveTransform(src_vertices, dst_vertices);

    cv::Mat rotated;
    cv::Size size(box.boundingRect().width, box.boundingRect().height);
    warpPerspective(src, rotated, warpMatrix, size, INTER_LINEAR, BORDER_CONSTANT);*/
    Mat warpAffineMatrix = getAffineTransform(src_vertices, dst_vertices);

    cv::Mat rotated;
    cv::Size size(box.boundingRect().width, box.boundingRect().height);
    warpAffine(src, rotated, warpAffineMatrix, size, INTER_LINEAR, BORDER_CONSTANT);

    imwrite("rotated.jpg", rotated);
}

回答by karlphillip

The problem was the order in which the points were declared inside the vector, and then there was also another issue related to this on the definition of dst_vertices.

问题是在向量内声明点的顺序，然后还有另一个与此相关的问题关于 的定义dst_vertices。

The order of the points matterto getPerspectiveTransform()and must be specified in the following order:

该点的顺序关系到getPerspectiveTransform()，必须按以下顺序指定：

1st-------2nd
 |         |
 |         |
 |         |
3rd-------4th

Therefore, the points of origin needed to be re-ordered to this:

因此，需要将原点重新排序为：

vector<Point> not_a_rect_shape;
not_a_rect_shape.push_back(Point(408, 69));
not_a_rect_shape.push_back(Point(1912, 291));
not_a_rect_shape.push_back(Point(72, 2186));
not_a_rect_shape.push_back(Point(1584, 2426));

and the destination:

和目的地：

Point2f dst_vertices[4];
dst_vertices[0] = Point(0, 0);
dst_vertices[1] = Point(box.boundingRect().width-1, 0); // Bug was: had mistakenly switched these 2 parameters
dst_vertices[2] = Point(0, box.boundingRect().height-1);
dst_vertices[3] = Point(box.boundingRect().width-1, box.boundingRect().height-1);

After this, some cropping need to be donebecause the resulting image is not just the area within the green rectangle as I thought it would be:

在此之后，需要进行一些裁剪，因为生成的图像不仅仅是我认为的绿色矩形内的区域：

I don't know if this is a bug of OpenCV or if I'm missing something, but the main issue has been solved.

我不知道这是 OpenCV 的错误还是我遗漏了什么，但主要问题已经解决。

回答by Tim

When working with a quadrangle, OpenCV isn't really your friend. RotatedRectwill give you incorrect results. Also you will need a perspective projection instead of a affine projection like others mentioned here..

在处理四边形时，OpenCV 并不是您真正的朋友。RotatedRect会给你错误的结果。此外，您将需要透视投影而不是像这里提到的其他人那样的仿射投影。

Basicly what must been done is:

基本上必须做的是：

• Loop through all polygon segments and connect those which are almost equel.
• Sort them so you have the 4 most largest line segments.
• Intersect those lines and you have the 4 most likely corner points.
• Transform the matrix over the perspective gathered from the corner points and the aspect ratio of the known object.

• 循环遍历所有多边形段并连接那些几乎相等的多边形段。
• 对它们进行排序，以便获得 4 个最大的线段。
• 将这些线相交，您将获得 4 个最有可能的角点。
• 在从角点和已知对象的纵横比收集的透视图上变换矩阵。

I implemented a class Quadranglewhich takes care of contour to quadrangle conversion and will also transform it over the right perspective.

我实现了一个类Quadrangle，它负责将轮廓转换为四边形，并且还将在正确的视角上对其进行转换。

See a working implementation here: Java OpenCV deskewing a contour

在此处查看工作实现： Java OpenCV 校正轮廓

回答by VaporwareWolf

UPDATE: RESOLVED

更新：已解决

I almost have this working. So close to being usable. It deskews properly but I seem to have a scale or translate issue. I have set the anchor point to zero and also experimented with changing the scale mode (aspectFill, scale to fit, etc...).

我几乎有这个工作。如此接近可用。它校正正确，但我似乎有比例或翻译问题。我已将锚点设置为零，并尝试更改缩放模式（aspectFill、缩放以适应等）。

Setup the deskew points (red makes them hard to see):

设置纠偏点（红色使它们难以看到）：

Apply the transform calculated:

应用计算的变换：

Now it deskews. This looks pretty good except that its not centered on the screen. By adding a pan gesture to the image view I can drag it over and verify that it lines up:

现在它歪斜了。这看起来很不错，只是它不在屏幕上居中。通过向图像视图添加平移手势，我可以将其拖过并验证它是否对齐：

This is not as simple as translate by -0.5, -0.5 because the original image become a polygon that stretches out very very far (potentially), so it's bounding rect is much bigger than the screen frame.

这不像通过 -0.5, -0.5 转换那么简单，因为原始图像变成了一个延伸非常远（可能）的多边形，因此它的边界矩形比屏幕框架大得多。

Does anyone see what I can do to get this wrapped up? I'd like to get it committed and share it here. This is a popular topic but I haven't found a solution that's as simple as copy/paste.

有没有人看到我能做些什么来解决这个问题？我想把它提交并在这里分享。这是一个热门话题，但我还没有找到像复制/粘贴一样简单的解决方案。

Full source code is here:

完整的源代码在这里：

git clone https://github.com/zakkhoyt/Quadrilateral.git

git 克隆https://github.com/zakkhoyt/Quadrilateral.git

git checkout demo

git 结帐演示

However, I'll paste the relevant parts here. This first method is mine and is where I get the deskew points.

但是，我将在此处粘贴相关部分。第一种方法是我的，也是我获得纠偏点的地方。

- (IBAction)buttonAction:(id)sender {

    Quadrilateral quadFrom;
    float scale = 1.0;
    quadFrom.topLeft.x = self.topLeftView.center.x / scale;
    quadFrom.topLeft.y = self.topLeftView.center.y / scale;
    quadFrom.topRight.x = self.topRightView.center.x / scale;
    quadFrom.topRight.y = self.topRightView.center.y / scale;
    quadFrom.bottomLeft.x = self.bottomLeftView.center.x / scale;
    quadFrom.bottomLeft.y = self.bottomLeftView.center.y / scale;
    quadFrom.bottomRight.x = self.bottomRightView.center.x / scale;
    quadFrom.bottomRight.y = self.bottomRightView.center.y / scale;

    Quadrilateral quadTo;
    quadTo.topLeft.x = self.view.bounds.origin.x;
    quadTo.topLeft.y = self.view.bounds.origin.y;
    quadTo.topRight.x = self.view.bounds.origin.x + self.view.bounds.size.width;
    quadTo.topRight.y = self.view.bounds.origin.y;
    quadTo.bottomLeft.x = self.view.bounds.origin.x;
    quadTo.bottomLeft.y = self.view.bounds.origin.y + self.view.bounds.size.height;
    quadTo.bottomRight.x = self.view.bounds.origin.x + self.view.bounds.size.width;
    quadTo.bottomRight.y = self.view.bounds.origin.y + self.view.bounds.size.height;

    CATransform3D t = [self transformQuadrilateral:quadFrom toQuadrilateral:quadTo];
//    t = CATransform3DScale(t, 0.5, 0.5, 1.0);
    self.imageView.layer.anchorPoint = CGPointZero;
    [UIView animateWithDuration:1.0 animations:^{
        self.imageView.layer.transform = t;
    }];

}


#pragma mark OpenCV stuff...
-(CATransform3D)transformQuadrilateral:(Quadrilateral)origin toQuadrilateral:(Quadrilateral)destination {

    CvPoint2D32f *cvsrc = [self openCVMatrixWithQuadrilateral:origin];
    CvMat *src_mat = cvCreateMat( 4, 2, CV_32FC1 );
    cvSetData(src_mat, cvsrc, sizeof(CvPoint2D32f));


    CvPoint2D32f *cvdst = [self openCVMatrixWithQuadrilateral:destination];
    CvMat *dst_mat = cvCreateMat( 4, 2, CV_32FC1 );
    cvSetData(dst_mat, cvdst, sizeof(CvPoint2D32f));

    CvMat *H = cvCreateMat(3,3,CV_32FC1);
    cvFindHomography(src_mat, dst_mat, H);
    cvReleaseMat(&src_mat);
    cvReleaseMat(&dst_mat);

    CATransform3D transform = [self transform3DWithCMatrix:H->data.fl];
    cvReleaseMat(&H);

    return transform;
}

- (CvPoint2D32f*)openCVMatrixWithQuadrilateral:(Quadrilateral)origin {

    CvPoint2D32f *cvsrc = (CvPoint2D32f *)malloc(4*sizeof(CvPoint2D32f));
    cvsrc[0].x = origin.topLeft.x;
    cvsrc[0].y = origin.topLeft.y;
    cvsrc[1].x = origin.topRight.x;
    cvsrc[1].y = origin.topRight.y;
    cvsrc[2].x = origin.bottomRight.x;
    cvsrc[2].y = origin.bottomRight.y;
    cvsrc[3].x = origin.bottomLeft.x;
    cvsrc[3].y = origin.bottomLeft.y;

    return cvsrc;
}

-(CATransform3D)transform3DWithCMatrix:(float *)matrix {
    CATransform3D transform = CATransform3DIdentity;

    transform.m11 = matrix[0];
    transform.m21 = matrix[1];
    transform.m41 = matrix[2];

    transform.m12 = matrix[3];
    transform.m22 = matrix[4];
    transform.m42 = matrix[5];

    transform.m14 = matrix[6];
    transform.m24 = matrix[7];
    transform.m44 = matrix[8];

    return transform; 
}

Update: I got it working properly. The coordinates needed to be origin in the center, not the upperleft. I applied xOffset and yOffset and viola. Demo code at the location mentioned above ("demo" branch)

更新：我让它正常工作。坐标需要是中心的原点，而不是左上角。我应用了 xOffset 和 yOffset 以及中提琴。上面提到的位置的演示代码（“演示”分支）

回答by MonsieurDart

I got the same kind of issue and fixed it using OpenCV's homography extraction function.

我遇到了同样的问题并使用 OpenCV 的单应性提取功能修复了它。

You can see how I did in this question: Transforming a rectangle image into a quadrilateral using a CATransform3D

你可以看到我在这个问题中是怎么做的：Transforming a rectangle image into a quadrilateral using a CATransform3D

回答by Bj?rn Egil

Very much inspired by @VaporwareWolf's answer, implemented in C# using Xamarin MonoTouch for iOS. The main difference is that I'm using GetPerspectiveTransform instead of FindHomography and TopLeft rather than ScaleToFit for content mode:

非常受@VaporwareWolf 回答的启发，该回答是在 C# 中使用 Xamarin MonoTouch for iOS 实现的。主要区别在于我使用 GetPerspectiveTransform 而不是 FindHomography 和 TopLeft 而不是 ScaleToFit 用于内容模式：

    void SetupWarpedImage(UIImage sourceImage, Quad sourceQuad, RectangleF destRectangle)
    {
        var imageContainerView = new UIView(destRectangle)
        {
            ClipsToBounds = true,
            ContentMode = UIViewContentMode.TopLeft
        };

        InsertSubview(imageContainerView, 0);

        var imageView = new UIImageView(imageContainerView.Bounds)
        {
            ContentMode = UIViewContentMode.TopLeft,
            Image = sourceImage
        };

        var offset = new PointF(-imageView.Bounds.Width / 2, -imageView.Bounds.Height / 2);
        var dest = imageView.Bounds;
        dest.Offset(offset);
        var destQuad = dest.ToQuad();

        var transformMatrix = Quad.GeneratePerspectiveTransformMatrixFromQuad(sourceQuad, destQuad);
        CATransform3D transform = transformMatrix.ToCATransform3D();

        imageView.Layer.AnchorPoint = new PointF(0f, 0f);
        imageView.Layer.Transform = transform;

        imageContainerView.Add(imageView);
    }