C++ 如何在 Qt 中有效地显示 OpenCV 视频？

How to efficiently display OpenCV video in Qt?

提问by Murat ?eker

I'm capturing multiple streams from ip cameras with the help of OpenCV. When i try to display these stream from an OpenCV window(cv::namedWindow(...)), it works without any problem (i have tried up to 4 streams so far).

我在 OpenCV 的帮助下从网络摄像机捕获多个流。当我尝试从 OpenCV 窗口 ( cv::namedWindow(...))显示这些流时，它可以正常工作（到目前为止我已经尝试了多达 4 个流）。

The problem arises when i try to show these streams inside a Qt widget. Since the capturing is done in another thread, i have to use the signal slot mechanism in order to update the QWidget(which is in main thread).

当我尝试在 Qt 小部件中显示这些流时出现问题。由于捕获是在另一个线程中完成的，我必须使用信号槽机制来更新 QWidget（在主线程中）。

Basically, i emit the newly captured frame from the capture thread and a slot in the GUI thread catches it. When i open 4 streams, i can not display the videos smoothly like before.

基本上，我从捕获线程发出新捕获的帧，GUI 线程中的一个插槽捕获它。当我打开 4 个流时，我无法像以前一样流畅地显示视频。

Here is the emitter :

这是发射器：

void capture::start_process() {
    m_enable = true;
    cv::Mat frame;

    while(m_enable) {
        if (!m_video_handle->read(frame)) {
            break;
        }
        cv::cvtColor(frame, frame,CV_BGR2RGB);

        qDebug() << "FRAME : " << frame.data;

        emit image_ready(QImage(frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888));
        cv::waitKey(30);
    }
}

This is my slot :

这是我的插槽：

void widget::set_image(QImage image) {
    img = image;
    qDebug() << "PARAMETER IMAGE: " << image.scanLine(0);
    qDebug() << "MEMBER IMAGE: " << img.scanLine(0);
}

The problem seems like the overhead of copying QImages continuously. Although QImage uses implicit sharing, when i compare the data pointers of images via qDebug()messages, i see different addresses.

问题似乎是连续复制 QImages 的开销。虽然 QImage 使用隐式共享，但是当我通过qDebug()消息比较图像的数据指针时，我看到了不同的地址。

1- Is there any way to embed OpenCV window directly into QWidget ?

1- 有没有办法将 OpenCV 窗口直接嵌入到 QWidget 中？

2- What is the most efficient way to handle displaying multiple videos? For example, how video management systems show up to 32 cameras in the same time ?

2- 处理显示多个视频的最有效方法是什么？例如，视频管理系统如何同时显示多达 32 个摄像机？

3- What must be the way to go ?

3-必须要走的路是什么？

回答by Reinstate Monica

Using QImage::scanLineforces a deep copy, so at the minimum, you should use constScanLine, or, better yet, change the slot's signature to:

使用QImage::scanLine强制深度复制，因此至少应该使用constScanLine，或者更好的是，将插槽的签名更改为：

void widget::set_image(const QImage & image);

Of course, your problem then becomes something else: the QImageinstance points to the data of a frame that lives in another thread, and can (and will) change at any moment.

当然，您的问题随后变成了其他问题：QImage实例指向位于另一个线程中的帧的数据，并且可以（并且将）随时更改。

There is a solution for that: one needs to use fresh frames allocated on the heap, and the frame needs to be captured within QImage. QScopedPointeris used to prevent memory leaks until the QImagetakes ownership of the frame.

有一个解决方案：需要使用堆上分配的新帧，并且需要在QImage. QScopedPointer用于防止内存泄漏，直到QImage获得帧的所有权。

static void matDeleter(void* mat) { delete static_cast<cv::Mat*>(mat); }

class capture {
   Q_OBJECT
   bool m_enable;
   ...
public:
   Q_SIGNAL void image_ready(const QImage &);
   ...
};

void capture::start_process() {
  m_enable = true;
  while(m_enable) {
    QScopedPointer<cv::Mat> frame(new cv::Mat);
    if (!m_video_handle->read(*frame)) {
      break;
    }
    cv::cvtColor(*frame, *frame, CV_BGR2RGB);

    // Here the image instance takes ownership of the frame.
    const QImage image(frame->data, frame->cols, frame->rows, frame->step,
                       QImage::Format_RGB888, matDeleter, frame.take());       
    emit image_ready(image);
    cv::waitKey(30);
  }
}

Of course, since Qt provides native message dispatch anda Qt event loop by default in a QThread, it's a simple matter to use QObjectfor the capture process. Below is a complete, tested example.

当然，由于 Qt在 a 中默认提供了本地消息调度和Qt 事件循环QThread，因此QObject用于捕获过程是一件很简单的事情。下面是一个完整的、经过测试的示例。

The capture, conversion and viewer all run in their own threads. Since cv::Matis an implicitly shared class with atomic, thread-safe access, it's used as such.

捕获、转换和查看器都在它们自己的线程中运行。由于cv::Mat是具有原子、线程安全访问的隐式共享类，因此它被如此使用。

The converter has an option of not processing stale frames - useful if conversion is only done for display purposes.

转换器具有不处理陈旧帧的选项 - 如果转换仅用于显示目的，则非常有用。

The viewer runs in the gui thread and correctly drops stale frames. There's never a reason for the viewer to deal with stale frames.

查看器在 gui 线程中运行并正确丢弃陈旧的帧。观众永远没有理由处理陈旧的帧。

If you were to collect data to save to disk, you should run the capture thread at high priority. You should also inspect OpenCV apis to see if there's a way of dumping the native camera data to disk.

如果要收集数据以保存到磁盘，则应以高优先级运行捕获线程。您还应该检查 OpenCV api 以查看是否有将本机相机数据转储到磁盘的方法。

To speed up conversion, you could use the gpu-accelerated classes in OpenCV.

为了加快转换速度，您可以使用 OpenCV 中的 GPU 加速类。

The example below makes sure that in none of the memory is reallocated unless necessary for a copy: the Captureclass maintains its own frame buffer that is reused for each subsequent frame, so does the Converter, and so does the ImageViewer.

下面的示例确保除非复制需要，否则不会重新分配任何内存：Capture该类维护自己的帧缓冲区，该缓冲区可在每个后续帧中重用Converter， 和ImageViewer.

There are two deep copies of image data made (besides whatever happens internally in cv::VideoCatprure::read):

制作了两个图像数据的深层副本（除了内部发生的任何事情cv::VideoCatprure::read）：

1. The copy to the Converter's QImage.

2. The copy to ImageViewer's QImage.

1. 复制到Converter的QImage.

2. 复制到ImageViewer的QImage。

Both copies are needed to assure decoupling between the threads and prevent data reallocation due to the need to detach a cv::Mator QImagethat has the reference count higher than 1. On modern architectures, memory copies are very fast.

需要两个副本，以保证线程之间的解耦，并防止数据再分配由于需要拆下一个cv::Mat或者QImage具有比1.在现代建筑较高的引用计数，内存复制速度非常快。

Since all image buffers stay in the same memory locations, their performance is optimal - they stay paged in and cached.

由于所有图像缓冲区都位于相同的内存位置，因此它们的性能是最佳的——它们保持分页和缓存。

The AddressTrackeris used to track memory reallocations for debugging purposes.

将AddressTracker用于跟踪内存重新分配进行调试。

// https://github.com/KubaO/stackoverflown/tree/master/questions/opencv-21246766
#include <QtWidgets>
#include <algorithm>
#include <opencv2/opencv.hpp>

Q_DECLARE_METATYPE(cv::Mat)

struct AddressTracker {
   const void *address = {};
   int reallocs = 0;
   void track(const cv::Mat &m) { track(m.data); }
   void track(const QImage &img) { track(img.bits()); }
   void track(const void *data) {
      if (data && data != address) {
         address = data;
         reallocs ++;
      }
   }
};

The Captureclass fills the internal frame buffer with the captured frame. It notifies of a frame change. The frame is the user property of the class.

所述Capture类填充与所捕获的帧的内部帧缓冲器。它通知帧更改。框架是类的用户属性。

class Capture : public QObject {
   Q_OBJECT
   Q_PROPERTY(cv::Mat frame READ frame NOTIFY frameReady USER true)
   cv::Mat m_frame;
   QBasicTimer m_timer;
   QScopedPointer<cv::VideoCapture> m_videoCapture;
   AddressTracker m_track;
public:
   Capture(QObject *parent = {}) : QObject(parent) {}
   ~Capture() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   Q_SIGNAL void started();
   Q_SLOT void start(int cam = {}) {
      if (!m_videoCapture)
         m_videoCapture.reset(new cv::VideoCapture(cam));
      if (m_videoCapture->isOpened()) {
         m_timer.start(0, this);
         emit started();
      }
   }
   Q_SLOT void stop() { m_timer.stop(); }
   Q_SIGNAL void frameReady(const cv::Mat &);
   cv::Mat frame() const { return m_frame; }
private:
   void timerEvent(QTimerEvent * ev) {
      if (ev->timerId() != m_timer.timerId()) return;
      if (!m_videoCapture->read(m_frame)) { // Blocks until a new frame is ready
         m_timer.stop();
         return;
      }
      m_track.track(m_frame);
      emit frameReady(m_frame);
   }
};

The Converterclass converts the incoming frame to a scaled-down QImageuser property. It notifies of the image update. The image is retained to prevent memory reallocations. The processAllproperty selects whether all frames will be converted, or only the most recent one should more than one get queued up.

的Converter类将输入的帧按比例缩小的QImage用户属性。它通知图像更新。保留图像以防止内存重新分配。该processAll属性选择是否所有帧都将被转换，或者只有最近的帧应该多于一个排队。

class Converter : public QObject {
   Q_OBJECT
   Q_PROPERTY(QImage image READ image NOTIFY imageReady USER true)
   Q_PROPERTY(bool processAll READ processAll WRITE setProcessAll)
   QBasicTimer m_timer;
   cv::Mat m_frame;
   QImage m_image;
   bool m_processAll = true;
   AddressTracker m_track;
   void queue(const cv::Mat &frame) {
      if (!m_frame.empty()) qDebug() << "Converter dropped frame!";
      m_frame = frame;
      if (! m_timer.isActive()) m_timer.start(0, this);
   }
   void process(const cv::Mat &frame) {
      Q_ASSERT(frame.type() == CV_8UC3);
      int w = frame.cols / 3.0, h = frame.rows / 3.0;
      if (m_image.size() != QSize{w,h})
         m_image = QImage(w, h, QImage::Format_RGB888);
      cv::Mat mat(h, w, CV_8UC3, m_image.bits(), m_image.bytesPerLine());
      cv::resize(frame, mat, mat.size(), 0, 0, cv::INTER_AREA);
      cv::cvtColor(mat, mat, CV_BGR2RGB);
      emit imageReady(m_image);
   }
   void timerEvent(QTimerEvent *ev) {
      if (ev->timerId() != m_timer.timerId()) return;
      process(m_frame);
      m_frame.release();
      m_track.track(m_frame);
      m_timer.stop();
   }
public:
   explicit Converter(QObject * parent = nullptr) : QObject(parent) {}
   ~Converter() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   bool processAll() const { return m_processAll; }
   void setProcessAll(bool all) { m_processAll = all; }
   Q_SIGNAL void imageReady(const QImage &);
   QImage image() const { return m_image; }
   Q_SLOT void processFrame(const cv::Mat &frame) {
      if (m_processAll) process(frame); else queue(frame);
   }
};

The ImageViewerwidget is the equivalent of a QLabelstoring a pixmap. The image is the user property of the viewer. The incoming image is deep-copied into the user property, to prevent memory reallocations.

该ImageViewer窗口小部件是等效的一个QLabel存储的pixmap。图像是查看器的用户属性。传入的图像被深度复制到用户属性中，以防止内存重新分配。

class ImageViewer : public QWidget {
   Q_OBJECT
   Q_PROPERTY(QImage image READ image WRITE setImage USER true)
   bool painted = true;
   QImage m_img;
   AddressTracker m_track;
   void paintEvent(QPaintEvent *) {
      QPainter p(this);
      if (!m_img.isNull()) {
         setAttribute(Qt::WA_OpaquePaintEvent);
         p.drawImage(0, 0, m_img);
         painted = true;
      }
   }
public:
   ImageViewer(QWidget * parent = nullptr) : QWidget(parent) {}
   ~ImageViewer() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   Q_SLOT void setImage(const QImage &img) {
      if (!painted) qDebug() << "Viewer dropped frame!";
      if (m_img.size() == img.size() && m_img.format() == img.format()
          && m_img.bytesPerLine() == img.bytesPerLine())
         std::copy_n(img.bits(), img.sizeInBytes(), m_img.bits());
      else
         m_img = img.copy();
      painted = false;
      if (m_img.size() != size()) setFixedSize(m_img.size());
      m_track.track(m_img);
      update();
   }
   QImage image() const { return m_img; }
};

The demonstration instantiates the classes described above and runs the capture and conversion in dedicated threads.

该演示实例化上述类并在专用线程中运行捕获和转换。

class Thread final : public QThread { public: ~Thread() { quit(); wait(); } };

int main(int argc, char *argv[])
{
   qRegisterMetaType<cv::Mat>();
   QApplication app(argc, argv);
   ImageViewer view;
   Capture capture;
   Converter converter;
   Thread captureThread, converterThread;
   // Everything runs at the same priority as the gui, so it won't supply useless frames.
   converter.setProcessAll(false);
   captureThread.start();
   converterThread.start();
   capture.moveToThread(&captureThread);
   converter.moveToThread(&converterThread);
   QObject::connect(&capture, &Capture::frameReady, &converter, &Converter::processFrame);
   QObject::connect(&converter, &Converter::imageReady, &view, &ImageViewer::setImage);
   view.show();
   QObject::connect(&capture, &Capture::started, [](){ qDebug() << "Capture started."; });
   QMetaObject::invokeMethod(&capture, "start");
   return app.exec();
}

#include "main.moc"

This concludes the complete example. Note: The previous revision of this answer unnecessarily reallocated the image buffers.

完整的例子到此结束。注意：此答案的先前修订版不必要地重新分配了图像缓冲区。