通过 Python 从 .idx3-ubyte 文件或 GZIP 中提取图像

Extract images from .idx3-ubyte file or GZIP via Python

提问by Roman

I have created a simple function for facerecognition by using the facerecognizer from OpenCV. It works all fine with images from people.

我使用 OpenCV 的 facerecognizer 创建了一个简单的人脸识别函数。它适用于人的图像。

Now I would like to make a test by using handwritten characters instead of people. I came across MNIST dataset, but they store images in a weird file which I have never seen before.

现在我想用手写字符代替人来做一个测试。我遇到了 MNIST 数据集，但它们将图像存储在一个我以前从未见过的奇怪文件中。

I simply need to extract a few images from:

我只需要从中提取一些图像：

train-images.idx3-ubyte

and save them in a folder as .gif

并将它们保存在一个文件夹中 .gif

Or am I missunderstand this MNIST thing. If yes where could I get such a dataset?

或者我误解了这个 MNIST 的事情。如果是，我在哪里可以获得这样的数据集？

EDIT

编辑

I also have the gzip file:

我也有 gzip 文件：

train-images-idx3-ubyte.gz

I am trying to read the content, but show()does not work and if I read()I see random symbols.

我正在尝试阅读内容，但show()不起作用，如果read()我看到随机符号。

images = gzip.open("train-images-idx3-ubyte.gz", 'rb')
print images.read()

EDIT

编辑

Managed to get some usefull output by using:

通过使用管理获得一些有用的输出：

with gzip.open('train-images-idx3-ubyte.gz','r') as fin:
    for line in fin:
        print('got line', line)

Somehow I have to convert this now to an image, output:

不知何故，我现在必须将其转换为图像，输出：

回答by Laurent LAPORTE

Download the training/test images and labels:

下载训练/测试图像和标签：

• train-images-idx3-ubyte.gz: training set images
• train-labels-idx1-ubyte.gz: training set labels
• t10k-images-idx3-ubyte.gz: test set images
• t10k-labels-idx1-ubyte.gz: test set labels

• train-images-idx3-ubyte.gz：训练集图像
• train-labels-idx1-ubyte.gz：训练集标签
• t10k-images-idx3-ubyte.gz：测试集图像
• t10k-labels-idx1-ubyte.gz：测试集标签

And uncompress them in a workdir, say samples/.

并将它们解压缩到工作目录中，例如samples/.

Get the python-mnistpackage from PyPi:

从 PyPi获取python-mnist包：

pip install python-mnist

Import the mnistpackage and read the training/test images:

导入mnist包并读取训练/测试图像：

from mnist import MNIST

mndata = MNIST('samples')

images, labels = mndata.load_training()
# or
images, labels = mndata.load_testing()

To display an image to the console:

向控制台显示图像：

index = random.randrange(0, len(images))  # choose an index ;-)
print(mndata.display(images[index]))

You'll get something like this:

你会得到这样的东西：

............................
............................
............................
............................
............................
.................@@.........
..............@@@@@.........
............@@@@............
..........@@................
..........@.................
...........@................
...........@................
...........@...@............
...........@@@@@.@..........
...........@@@...@@.........
...........@@.....@.........
..................@.........
..................@@........
..................@@........
..................@.........
.................@@.........
...........@.....@..........
...........@....@@..........
............@@@@............
.............@..............
............................
............................
............................

Explanation:

解释：

• Each imageof the imageslist is a Python listof unsigned bytes.
• The labelsis an Python arrayof unsigned bytes.

• 每个图像的的图像列表是一个Pythonlist的无符号字节。
• 该标签是一个Python的array无符号字节。

回答by Punnerud

(Using only matplotlib, gzip and numpy)
Extract image data:

（仅使用 matplotlib、gzip 和 numpy）
提取图像数据：

import gzip
f = gzip.open('train-images-idx3-ubyte.gz','r')

image_size = 28
num_images = 5

import numpy as np
f.read(16)
buf = f.read(image_size * image_size * num_images)
data = np.frombuffer(buf, dtype=np.uint8).astype(np.float32)
data = data.reshape(num_images, image_size, image_size, 1)

Print images:

打印图像：

import matplotlib.pyplot as plt
image = np.asarray(data[2]).squeeze()
plt.imshow(image)
plt.show()

Print first 50 labels:

打印前 50 个标签：

f = gzip.open('train-labels-idx1-ubyte.gz','r')
f.read(8)
for i in range(0,50):   
    buf = f.read(1)
    labels = np.frombuffer(buf, dtype=np.uint8).astype(np.int64)
    print(labels)

回答by avneesh mishra

You could actually use the idx2numpypackage available at PyPI. It's extremelysimple to use and directly converts the data to numpy arrays. Here's what you have to do:

您实际上可以使用PyPI 提供的idx2numpy包。这是非常简单易用，并直接将数据转换成numpy的阵列。这是你必须做的：

Downloading the data

下载数据

Download the MNIST dataset from the official website.
If you're using Linux then you can use wgetto get it from command line itself. Just run:

从官网下载MNIST数据集。
如果您使用的是 Linux，那么您可以使用wget从命令行本身获取它。赶紧跑：

wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
wget http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz

Decompressing the data

解压数据

Unzip or decompress the data. On Linux, you could use gzip

解压缩或解压缩数据。在 Linux 上，您可以使用gzip

Ultimately, you should have the following files:

最终，您应该拥有以下文件：

data/train-images-idx3-ubyte
data/train-labels-idx1-ubyte
data/t10k-images-idx3-ubyte
data/t10k-labels-idx1-ubyte

The prefix data/is just because I've extracted them into a folder named data. Your question looks like you're well done till here, so keep reading.

前缀data/只是因为我已将它们解压缩到名为data. 你的问题看起来你在这里做得很好，所以继续阅读。

Using idx2numpy

使用 idx2numpy

Here's a simple python code to read everything from the decompressed files as numpy arrays.

这是一个简单的python代码，用于将解压缩文件中的所有内容作为numpy数组读取。

import idx2numpy
import numpy as np
file = 'data/train-images-idx3-ubyte'
arr = idx2numpy.convert_from_file(file)
# arr is now a np.ndarray type of object of shape 60000, 28, 28

You can now use it with OpenCV juts the same way how you display any other image, using something like

您现在可以使用 OpenCV 以与显示任何其他图像相同的方式使用它，使用类似

cv.imshow("Image", arr[4])

To install idx2numpy, you can use PyPI (pippackage manager). Simply run the command:

要安装 idx2numpy，您可以使用 PyPI（pip包管理器）。只需运行以下命令：

pip install idx2numpy

回答by Aqib Mumtaz

Use this to extract mnist database to images and csv labels in python :

使用它在 python 中将 mnist 数据库提取到图像和 csv 标签：

https://github.com/sorki/python-mnist

https://github.com/sorki/python-mnist

回答by ho_khalaf

install idx2numpy

安装 idx2numpy

pip install idx2numpy

Downloading the data

下载数据

Download the MNIST dataset from the official website.

从官网下载MNIST数据集。

Decompressing the data

解压数据

Ultimately, you should have the following files:

最终，您应该拥有以下文件：

train-images-idx3-ubyte
train-labels-idx1-ubyte
t10k-images-idx3-ubyte
t10k-labels-idx1-ubyte

Using idx2numpy

使用 idx2numpy

    import numpy as np
    import idx2numpy
    import matplotlib.pyplot as plt

    imagefile = 'train-images.idx3-ubyte'
    imagearray = idx2numpy.convert_from_file(imagefile)

    plt.imshow(imagearray[4], cmap=plt.cm.binary)

The minst picture

最简单的图片