Python 与我的 PC 相比，Google Colab 非常慢
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/49360888/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Google Colab is very slow compared to my PC
提问by charel-f
I've recently started to use Google Colab, and wanted to train my first Convolutional NN. I imported the images from my Google Drive thanks to the answer I got here.
我最近开始使用 Google Colab，并想训练我的第一个卷积神经网络。由于我在这里得到的答案，我从我的 Google Drive 导入了图像。
Then I pasted my code to create the CNN into Colab and started the process. Here is the complete code:
然后我将用于创建 CNN 的代码粘贴到 Colab 中并开始该过程。这是完整的代码：
Part 1: Setting up Colab to import picture from my Drive
第 1 部分：设置 Colab 以从我的云端硬盘导入图片
(part 1 is copied from hereas it worked as exptected for me
（第 1 部分是从这里复制的，因为它对我来说是预期的
Step 1:
第1步：
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
Step 2:
第2步：
from google.colab import auth
auth.authenticate_user()
Step 3:
第 3 步：
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
Step 4:
第四步：
!mkdir -p drive
!google-drive-ocamlfuse drive
Step 5:
第 5 步：
print('Files in Drive:')
!ls drive/
Part 2: Copy pasting my CNN
第 2 部分：复制粘贴我的 CNN
I created this CNN with tutorials from a Udemy Course. It uses keras with tensorflow as backend. For the sake of simplicity I uploaded a really simple version, which is plenty enough to show my problems
我用 Udemy 课程的教程创建了这个 CNN。它使用带有 tensorflow 的 keras 作为后端。为了简单起见，我上传了一个非常简单的版本，足以说明我的问题
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.layers import Dropout
from keras.optimizers import Adam
from keras.preprocessing.image import ImageDataGenerator
parameters
参数
imageSize=32
batchSize=64
epochAmount=50
CNN
美国有线电视新闻网
classifier=Sequential()
classifier.add(Conv2D(32, (3, 3), input_shape = (imageSize, imageSize, 3), activation = 'relu')) #convolutional layer
classifier.add(MaxPooling2D(pool_size = (2, 2))) #pooling layer
classifier.add(Flatten())
ANN
人工神经网络
classifier.add(Dense(units=64, activation='relu')) #hidden layer
classifier.add(Dense(units=1, activation='sigmoid')) #output layer
classifier.compile(optimizer = "adam", loss = 'binary_crossentropy', metrics = ['accuracy']) #training method
image preprocessing
图像预处理
train_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)
training_set = train_datagen.flow_from_directory('drive/School/sem-2-2018/BSP2/UdemyCourse/CNN/dataset/training_set',
target_size = (imageSize, imageSize),
batch_size = batchSize,
class_mode = 'binary')
test_set = test_datagen.flow_from_directory('drive/School/sem-2-2018/BSP2/UdemyCourse/CNN/dataset/test_set',
target_size = (imageSize, imageSize),
batch_size = batchSize,
class_mode = 'binary')
classifier.fit_generator(training_set,
steps_per_epoch = (8000//batchSize),
epochs = epochAmount,
validation_data = test_set,
validation_steps = (2000//batchSize))
Now comes my Problem
现在是我的问题
First of, the training set I used is a database with 10000 dog and cat pictures of various resolutions. (8000 training_set, 2000 test_set)
首先，我使用的训练集是一个包含 10000 张各种分辨率的狗和猫图片的数据库。（8000 个训练集，2000 个测试集）
I ran this CNN on Google Colab (with GPU support enabled) and on my PC (tensorflow-gpu on GTX 1060)
我在 Google Colab（启用 GPU 支持）和我的 PC（GTX 1060 上的 tensorflow-gpu）上运行了这个 CNN
This is an intermediate result from my PC:
这是我的 PC 的中间结果：
Epoch 2/50
63/125 [==============>...............] - ETA: 2s - loss: 0.6382 - acc: 0.6520
And this from Colab:
这是来自 Colab：
Epoch 1/50
13/125 [==>...........................] - ETA: 1:00:51 - loss: 0.7265 - acc: 0.4916
Why is Google Colab so slow in my case?
在我的情况下，为什么 Google Colab 这么慢？
Personally I suspect a bottleneck consisting of pulling and then reading the images from my Drive, but I don't know how to solve this other than choosing a different method to import the database.
我个人怀疑一个瓶颈包括从我的驱动器中拉取然后读取图像，但除了选择不同的方法导入数据库之外，我不知道如何解决这个问题。
采纳答案by MROB
As @Fenghas already noted, reading files from drive is very slow. Thistutorial suggests using some sort of a memory mapped file like hdf5 or lmdb in order to overcome this issue. This way the I\O Operations are much faster (for a complete explanation on the speed gain of hdf5 format see this).
正如@ Feng已经指出的那样，从驱动器读取文件非常慢。本教程建议使用某种内存映射文件（如 hdf5 或 lmdb）来解决此问题。这样 I\O 操作要快得多（有关 hdf5 格式的速度增益的完整说明，请参阅此）。
回答by Feng
It's very slow to read file from google drives.
从谷歌驱动器读取文件非常慢。
For example, I have one big file(39GB).
例如，我有一个大文件（39GB）。
It cost more than 10min when I exec '!cp drive/big.file /content/'.
当我执行 '!cp drive/big.file /content/' 时，它花费了超过 10 分钟。
After I shared my file, and got the url from google drive. It cost 5 min when I exec '! wget -c -O big.file http://share.url.from.drive'. Download speed can up to 130MB/s.
在我共享我的文件并从谷歌驱动器获取 url 之后。当我执行 '! wget -c -O big.file http://share.url.from.drive'。下载速度可达130MB/s。
回答by anilsathyan7
You can load your data as numpy array (.npy format) and use flow method instead of flow_from_directory. Colab provides 25GB RAM ,so even for big data-sets you can load your entire data into memory. The speed up was found to be aroud 2.5x, with the same data generation steps!!! (Even faster than data stored in colab local disk i.e '/content' or google drive.
您可以将数据加载为 numpy 数组（.npy 格式）并使用 flow 方法而不是 flow_from_directory。Colab 提供 25GB RAM，因此即使对于大数据集，您也可以将整个数据加载到内存中。发现速度提高了大约 2.5 倍，数据生成步骤相同！！！（甚至比存储在 colab 本地磁盘（即“/content”或谷歌驱动器）中的数据还要快。
Since colab provides only a single core CPU (2 threads per core), there seems to be a bottleneck with CPU-GPU data transfer (say K80 or T4 GPU), especially if you use data generator for heavy preprocessing or data augmentation. You can also try setting different values for parameters like 'workers', 'use_multiprocessing', 'max_queue_size ' in fit_generator method ...
由于 colab 仅提供单核 CPU（每核 2 个线程），因此 CPU-GPU 数据传输（例如 K80 或 T4 GPU）似乎存在瓶颈，尤其是当您使用数据生成器进行大量预处理或数据增强时。您还可以尝试在 fit_generator 方法中为“workers”、“use_multiprocessing”、“max_queue_size”等参数设置不同的值...
回答by u5729330
Reading files from google drive slow down your training process. The solution is to upload zip file to colab and unzip there. Hope it is clear for you.
从谷歌驱动器读取文件会减慢你的训练过程。解决方案是将 zip 文件上传到 colab 并在那里解压缩。希望对你来说很清楚。
回答by Nadimprodutions
I have the same question as to why the GPU on colab seems to be taking at least just as long as my local pc so I can't really be of help there. But with that being said, if you are trying to use your data locally, I have found the following process to be significantly faster than just using the upload function provided in colab.
我有同样的问题，为什么 colab 上的 GPU 似乎至少与我的本地电脑一样长，所以我真的帮不上忙。但话虽如此，如果您尝试在本地使用您的数据，我发现以下过程比仅使用 colab 中提供的上传功能要快得多。
1.) mount google drive
1.) 安装谷歌驱动器
# Run this cell to mount your Google Drive.
from google.colab import drive
drive.mount('/content/drive')
2.) create a folder outside of the google drive folder that you want your data to be stored in
2.) 在 google drive 文件夹之外创建一个文件夹，您希望将数据存储在该文件夹中
3.) use the following command to copy the contents from your desired folder in google drive to the folder you created
3.) 使用以下命令将谷歌驱动器中所需文件夹中的内容复制到您创建的文件夹中
!ln -s "/content/drive/My Drive/path_to_folder_desired" "/path/to/the_folder/you created"
(this is referenced from another stackoverflowresponse that I used to find a solution to a similar issue )
（这是从我用来寻找类似问题的解决方案的另一个 stackoverflow响应中引用的）
4.) Now you have your data available to you at the path, "/path/to/the_folder/you created"
4.) 现在您可以在路径“/path/to/the_folder/you created”中使用您的数据
