ios iPhone OpenGL ES 2.0 中 glReadPixels 的更快替代方案

Faster alternative to glReadPixels in iPhone OpenGL ES 2.0

提问by atisman

Is there any faster way to access the frame buffer than using glReadPixels? I would need read-only access to a small rectangular rendering area in the frame buffer to process the data further in CPU. Performance is important because I have to perform this operation repeatedly. I have searched the web and found some approach like using Pixel Buffer Object and glMapBuffer but it seems that OpenGL ES 2.0 does not support them.

有没有比使用 glReadPixels 访问帧缓冲区更快的方法？我需要对帧缓冲区中的一个小的矩形渲染区域进行只读访问，以便在 CPU 中进一步处理数据。性能很重要，因为我必须重复执行此操作。我在网上搜索并找到了一些方法，例如使用 Pixel Buffer Object 和 glMapBuffer 但似乎 OpenGL ES 2.0 不支持它们。

回答by Brad Larson

As of iOS 5.0, there is now a faster way to grab data from OpenGL ES. It isn't readily apparent, but it turns out that the texture cache support added in iOS 5.0 doesn't just work for fast upload of camera frames to OpenGL ES, but it can be used in reverse to get quick access to the raw pixels within an OpenGL ES texture.

从 iOS 5.0 开始，现在有一种更快的方法可以从 OpenGL ES 获取数据。这不是很明显，但事实证明，iOS 5.0 中添加的纹理缓存支持不仅适用于将相机帧快速上传到 OpenGL ES，而且可以反向使用以快速访问原始像素在 OpenGL ES 纹理中。

You can take advantage of this to grab the pixels for an OpenGL ES rendering by using a framebuffer object (FBO) with an attached texture, with that texture having been supplied from the texture cache. Once you render your scene into that FBO, the BGRA pixels for that scene will be contained within your CVPixelBufferRef, so there will be no need to pull them down using glReadPixels().

您可以利用这一点通过使用带有附加纹理的帧缓冲区对象 (FBO) 来获取 OpenGL ES 渲染的像素，该纹理已从纹理缓存提供。一旦您将场景渲染到该 FBO 中，该场景的 BGRA 像素将包含在您的 CVPixelBufferRef 中，因此无需使用glReadPixels().

This is much, much faster than using glReadPixels()in my benchmarks. I found that on my iPhone 4, glReadPixels()was the bottleneck in reading 720p video frames for encoding to disk. It limited the encoding from taking place at anything more than 8-9 FPS. Replacing this with the fast texture cache reads allows me to encode 720p video at 20 FPS now, and the bottleneck has moved from the pixel reading to the OpenGL ES processing and actual movie encoding parts of the pipeline. On an iPhone 4S, this allows you to write 1080p video at a full 30 FPS.

这比glReadPixels()在我的基准测试中使用要快得多。我发现在我的 iPhone 4 上，这glReadPixels()是读取 720p 视频帧以编码到磁盘的瓶颈。它限制了编码以超过 8-9 FPS 的速度发生。用快速纹理缓存读取代替它，现在我可以以 20 FPS 编码 720p 视频，瓶颈已经从像素读取转移到 OpenGL ES 处理和管道的实际电影编码部分。在 iPhone 4S 上，这允许您以完整的 30 FPS 写入 1080p 视频。

My implementation can be found within the GPUImageMovieWriter class within my open source GPUImageframework, but it was inspired by Dennis Muhlestein's article on the subjectand Apple's ChromaKey sample application (which was only made available at WWDC 2011).

我的实现可以在我的开源GPUImage框架的 GPUImageMovieWriter 类中找到，但它的灵感来自Dennis Muhlestein 关于该主题的文章和 Apple 的 ChromaKey 示例应用程序（仅在 WWDC 2011 上提供）。

I start by configuring my AVAssetWriter, adding an input, and configuring a pixel buffer input. The following code is used to set up the pixel buffer input:

我首先配置我的 AVAssetWriter，添加一个输入，并配置一个像素缓冲区输入。以下代码用于设置像素缓冲区输入：

NSDictionary *sourcePixelBufferAttributesDictionary = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
                                                       [NSNumber numberWithInt:videoSize.width], kCVPixelBufferWidthKey,
                                                       [NSNumber numberWithInt:videoSize.height], kCVPixelBufferHeightKey,
                                                       nil];

assetWriterPixelBufferInput = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterVideoInput sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];

Once I have that, I configure the FBO that I'll be rendering my video frames to, using the following code:

一旦我有了它，我就会使用以下代码配置我要将视频帧渲染到的 FBO：

if ([GPUImageOpenGLESContext supportsFastTextureUpload])
{
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, (__bridge void *)[[GPUImageOpenGLESContext sharedImageProcessingOpenGLESContext] context], NULL, &coreVideoTextureCache);
    if (err) 
    {
        NSAssert(NO, @"Error at CVOpenGLESTextureCacheCreate %d");
    }

    CVPixelBufferPoolCreatePixelBuffer (NULL, [assetWriterPixelBufferInput pixelBufferPool], &renderTarget);

    CVOpenGLESTextureRef renderTexture;
    CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, coreVideoTextureCache, renderTarget,
                                                  NULL, // texture attributes
                                                  GL_TEXTURE_2D,
                                                  GL_RGBA, // opengl format
                                                  (int)videoSize.width,
                                                  (int)videoSize.height,
                                                  GL_BGRA, // native iOS format
                                                  GL_UNSIGNED_BYTE,
                                                  0,
                                                  &renderTexture);

    glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);
}

This pulls a pixel buffer from the pool associated with my asset writer input, creates and associates a texture with it, and uses that texture as a target for my FBO.

这会从与我的资产编写器输入关联的池中拉出一个像素缓冲区，创建一个纹理并将其与其关联，然后将该纹理用作我的 FBO 的目标。

Once I've rendered a frame, I lock the base address of the pixel buffer:

渲染一帧后，我锁定像素缓冲区的基地址：

CVPixelBufferLockBaseAddress(pixel_buffer, 0);

and then simply feed it into my asset writer to be encoded:

然后简单地将它输入到我的资产编写器中进行编码：

CMTime currentTime = CMTimeMakeWithSeconds([[NSDate date] timeIntervalSinceDate:startTime],120);

if(![assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:currentTime]) 
{
    NSLog(@"Problem appending pixel buffer at time: %lld", currentTime.value);
} 
else 
{
//        NSLog(@"Recorded pixel buffer at time: %lld", currentTime.value);
}
CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);

if (![GPUImageOpenGLESContext supportsFastTextureUpload])
{
    CVPixelBufferRelease(pixel_buffer);
}

Note that at no point here am I reading anything manually. Also, the textures are natively in BGRA format, which is what AVAssetWriters are optimized to use when encoding video, so there's no need to do any color swizzling here. The raw BGRA pixels are just fed into the encoder to make the movie.

请注意，我从来没有手动阅读任何内容。此外，纹理本身采用 BGRA 格式，这是 AVAssetWriters 在编码视频时优化使用的格式，因此无需在此处进行任何颜色调整。原始 BGRA 像素仅被输入编码器以制作电影。

Aside from the use of this in an AVAssetWriter, I have some code in this answerthat I've used for raw pixel extraction. It also experiences a significant speedup in practice when compared to using glReadPixels(), although less than I see with the pixel buffer pool I use with AVAssetWriter.

除了在 AVAssetWriter 中使用它之外，我在这个答案中有一些代码，我已经用于原始像素提取。与使用 相比，它在实践中也经历了显着的加速glReadPixels()，尽管比我在 AVAssetWriter 中使用的像素缓冲池中看到的要少。

It's a shame that none of this is documented anywhere, because it provides a huge boost to video capture performance.

遗憾的是，这些都没有记录在任何地方，因为它极大地提高了视频捕获性能。

回答by user2979732

Regarding what atisman mentioned about the black screen, I had that issue as well. Do really make sure everything is fine with your texture and other settings. I was trying to capture AIR's OpenGL layer, which I did in the end, the problem was that when I didn't set "depthAndStencil" to true by accident in the apps manifest, my FBO texture was half in height(the screen was divided in half and mirrored, I guess because of the wrap texture param stuff). And my video was black.

关于 atisman 提到的关于黑屏的内容，我也有这个问题。确实确保您的纹理和其他设置一切正常。我试图捕捉 AIR 的 OpenGL 层，我最终做到了，问题是当我没有在应用程序清单中意外将“depthAndStencil”设置为 true 时，我的 FBO 纹理的高度是一半（屏幕被分割了）一半和镜像，我猜是因为包装纹理参数的东西）。我的视频是黑色的。

That was pretty frustrating, as based on what Brad is posting it should have just worked once I had some data in texture. Unfortunately, that's not the case, everything has to be "right" for it to work - data in texture is not a guarantee for seeing equal data in the video. Once I added depthAndStencil my texture fixed itself to full height and I started to get video recording straight from AIR's OpenGL layer, no glReadPixels or anything :)

这非常令人沮丧，因为根据 Brad 发布的内容，一旦我在纹理中获得了一些数据，它应该可以正常工作。不幸的是，事实并非如此，一切都必须“正确”才能工作 - 纹理中的数据并不能保证在视频中看到相同的数据。添加 depthAndStencil 后，我的纹理将自身固定为全高，然后我开始直接从 AIR 的 OpenGL 层进行视频录制，没有 glReadPixels 或任何东西:)

So yeah, what Brad describes really DOES work without the need to recreate the buffers on every frame, you just need to make sure your setup is right. If you're getting blackness, try playing with the video/texture sizes perhaps or some other settings (setup of your FBO?).

所以是的，Brad 所描述的确实有效，无需在每一帧上重新创建缓冲区，您只需要确保您的设置正确。如果您变黑了，请尝试使用视频/纹理大小或其他一些设置（您的 FBO 的设置？）。