java 在进行字符识别之前使用 OpenCV 进行图像预处理（tesseract）

Image preprocessing with OpenCV before doing character recognition (tesseract)

提问by DocC

I'm trying to develop simple PC application for license plate recognition (Java + OpenCV + Tess4j). Images aren't really good (in further they will be good). I want to preprocess image for tesseract, and I'm stuck on detection of license plate (rectangle detection).

我正在尝试开发用于车牌识别的简单 PC 应用程序（Java + OpenCV + Tess4j）。图像不是很好（进一步它们会很好）。我想为 tesseract 预处理图像，但我一直在检测车牌（矩形检测）。

My steps:

我的步骤：

1) Source Image

1) 源图像

Mat img = new Mat();
img = Imgcodecs.imread("sample_photo.jpg"); 
Imgcodecs.imwrite("preprocess/True_Image.png", img);

2) Gray Scale

2) 灰度

Mat imgGray = new Mat();
Imgproc.cvtColor(img, imgGray, Imgproc.COLOR_BGR2GRAY);
Imgcodecs.imwrite("preprocess/Gray.png", imgGray);

3) Gaussian Blur

3) 高斯模糊

Mat imgGaussianBlur = new Mat(); 
Imgproc.GaussianBlur(imgGray,imgGaussianBlur,new Size(3, 3),0);
Imgcodecs.imwrite("preprocess/gaussian_blur.png", imgGaussianBlur);  

4) Adaptive Threshold

4) 自适应阈值

Mat imgAdaptiveThreshold = new Mat();
Imgproc.adaptiveThreshold(imgGaussianBlur, imgAdaptiveThreshold, 255, CV_ADAPTIVE_THRESH_MEAN_C ,CV_THRESH_BINARY, 99, 4);
Imgcodecs.imwrite("preprocess/adaptive_threshold.png", imgAdaptiveThreshold);

Here should be 5th step, which is detection of plate region (probably even without deskewing for now).

这里应该是第 5 步，即检测板区域（现在可能甚至没有纠偏）。

I croped needed region from image (after 4th step) with Paint, and got:

我用 Paint 从图像（第 4 步之后）裁剪了所需的区域，并得到：

Then I did OCR (via tesseract, tess4j):

然后我做了OCR（通过tesseract，tess4j）：

File imageFile = new File("preprocess/adaptive_threshold_AFTER_PAINT.png");
ITesseract instance = new Tesseract();
instance.setLanguage("eng");
instance.setTessVariable("tessedit_char_whitelist", "acekopxyABCEHKMOPTXY0123456789");
String result = instance.doOCR(imageFile); 
System.out.println(result);

and got (good enough?) result - "Y841ox EH" (almost true)

并得到（足够好？）结果 - “Y841ox EH”（几乎是真的）

How can I detect and crop plate region after 4th step? Have I to make some changes (improvements) in 1-4 steps? Would like to see some example implemented via Java + OpenCV (not JavaCV).
Thanks in advance.

在第 4 步后如何检测和裁剪板区域？我是否需要分 1-4 个步骤进行一些更改（改进）？希望看到一些通过 Java + OpenCV（不是 JavaCV）实现的示例。
提前致谢。

EDIT(thanks to @Abdul Fatir's answer) Well, I provide working (for me atleast) code sample (Netbeans+Java+OpenCV+Tess4j) for those who interested in this question. Code is not the best, but I made it just for studying.
http://pastebin.com/H46wuXWn(do not forget to put tessdatafolder into your project folder)

编辑（感谢@Abdul Fatir 的回答）好吧，我为那些对这个问题感兴趣的人提供了工作（至少对我而言）代码示例（Netbeans+Java+OpenCV+Tess4j）。代码不是最好的，但我编写它只是为了学习。
http://pastebin.com/H46wuXWn（不要忘记将tessdata文件夹放入您的项目文件夹中）

采纳答案by Abdul Fatir

Here's how I suggest you should do this task.

以下是我建议您执行此任务的方式。

1. Convert to Grayscale.
2. Gaussian Blur with 3x3 or 5x5 filter.

3. Apply Sobel Filter to find vertical edges.

   Sobel(gray, dst, -1, 1, 0)

4. Threshold the resultant image to get a binary image.
5. Apply a morphological close operation using suitable structuring element.
6. Find contours of the resulting image.
7. Find minAreaRectof each contour. Select rectangles based on aspect ratio and minimum and maximum area.
8. For each selected contour, find edge density. Set a threshold for edge density and choose the rectangles breaching that threshold as possible plate regions.
9. Few rectangles will remain after this. You can filter them based on orientation or any criteria you deem suitable.
10. Clip these detected rectangular portions from the image after adaptiveThresholdand apply OCR.

1. 转换为灰度。
2. 带有 3x3 或 5x5 过滤器的高斯模糊。

3. 应用 Sobel 过滤器来查找垂直边缘。

   Sobel(gray, dst, -1, 1, 0)

4. 对结果图像设置阈值以获得二值图像。
5. 使用合适的结构元素应用形态关闭操作。
6. 查找结果图像的轮廓。
7. 查找minAreaRect每个轮廓。根据纵横比和最小和最大面积选择矩形。
8. 对于每个选定的轮廓，找到边缘密度。设置边缘密度的阈值，并选择违反该阈值的矩形作为可能的板块区域。
9. 在此之后，将保留很少的矩形。您可以根据方向或您认为合适的任何标准过滤它们。
10. 之后从图像中剪辑这些检测到的矩形部分adaptiveThreshold并应用 OCR。

a) Result after Step 5

a)步骤 5 后的结果

b) Result after Step 7. Green ones are all the minAreaRects and the Red ones are those which satisfy the following criteria: Aspect Ratio range (2,12) & Area range (300,10000)

b)步骤 7 后的结果。绿色是所有minAreaRects，红色是满足以下条件的那些：纵横比范围 (2,12) & 面积范围 (300,10000)

c) Result after Step 9. Selected rectangle. Criteria: Edge Density > 0.5

c)步骤 9 后的结果。选定的矩形。标准：边缘密度 > 0.5

EDIT

编辑

For edge-density, what I did in the above examples is the following.

对于边缘密度，我在上面的例子中所做的如下。

1. Apply Canny Edge detector directly to input image. Let the cannyED image be Ic.
2. Multiply results of Sobel filter and Ic. Basically, take an AND of Sobel and Canny images.
3. Gaussian Blur the resultant image with a large filter. I used 21x21.
4. Threshold the resulting image using OTSU's method. You'll get a binary image
5. For each red rectangle, rotate the portion inside this rectangle (in the binary image) to make it upright. Loop through the pixels of the rectangle and count white pixels. (How to rotate?)

1. 将 Canny Edge 检测器直接应用于输入图像。让 cannyED 图像为Ic。
2. 将 Sobel 滤波器的结果与Ic相乘。基本上，对 Sobel 和 Canny 图像进行 AND 运算。
3. 高斯 使用大过滤器模糊结果图像。我使用了 21x21。
4. 使用 OTSU 的方法对生成的图像进行阈值处理。你会得到一个二进制图像
5. 对于每个红色矩形，旋转该矩形内的部分（在二值图像中）使其直立。遍历矩形的像素并计算白色像素。（如何旋转？）

Edge Density = No. of White Pixels in the Rectangle/Total no. of Pixels in the rectangle

边缘密度 = 矩形中白色像素的数量/总数量。矩形中的像素数

1. Choose a threshold for edge density.

6. 选择边缘密度的阈值。

NOTE: Instead of going through steps 1 to 3, you can also use the binary image from step 5 for calculating edge density.

注意：您还可以使用步骤 5 中的二进制图像来计算边缘密度，而不是执行步骤 1 到 3。

回答by sibnick

Actually OpenCV has pre-trained model specially for Russian license plates: haarcascade_russian_plate_number

实际上 OpenCV 有专门针对俄罗斯车牌的预训练模型：haarcascade_russian_plate_number

Also there is open source ANPR project for Russian license plates: plate_recognition. It is not use tesseract, but it has quite good pre-trained neural network.

还有用于俄罗斯车牌的开源 ANPR 项目： plate_recognition。它没有使用tesseract，但它有很好的预训练神经网络。

回答by RobAu

• You find all connected components (the white areas) and determine their outline.
• If you filter them based on size (as part of the image), ratio (width-height) and white/black ratio to retrieve candidate-plates.
• Undo the transformation of the rectangle
• Remove the bolts
• Pass in image to the OCR engine.

• 您找到所有连接的组件（白色区域）并确定它们的轮廓。
• 如果您根据大小（作为图像的一部分）、比率（宽高）和白/黑比率过滤它们以检索候选板。
• 撤消矩形的变换
• 拆下螺栓
• 将图像传递给 OCR 引擎。