Python 如何在 keras 中获得可重现的结果

How to get reproducible results in keras

提问by Pavel Surmenok

I get different results (test accuracy) every time I run the imdb_lstm.pyexample from Keras framework (https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py) The code contains np.random.seed(1337)in the top, before any keras imports. It should prevent it from generating different numbers for every run. What am I missing?

每次我imdb_lstm.py从 Keras 框架（https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py）运行示例时，我都会得到不同的结果（测试准确性）代码包含np.random.seed(1337)在顶部，在任何 keras 之前进口。它应该防止它为每次运行生成不同的数字。我错过了什么？

UPDATE: How to repro:

更新：如何重现：

1. Install Keras (http://keras.io/)
2. Execute https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.pya few times. It will train the model and output test accuracy.
   Expected result: Test accuracy is the same on every run.
   Actual result: Test accuracy is different on every run.

1. 安装 Keras ( http://keras.io/)
2. 执行https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py几次。它将训练模型并输出测试精度。
   预期结果：每次运行的测试准确度都相同。
   实际结果：每次运行的测试精度都不同。

UPDATE2: I'm running it on Windows 8.1 with MinGW/msys, module versions:
theano 0.7.0
numpy 1.8.1
scipy 0.14.0c1

UPDATE2：我在 Windows 8.1 上使用 MinGW/msys 运行它，模块版本：
theano 0.7.0
numpy 1.8.1
scipy 0.14.0c1

UPDATE3: I narrowed the problem down a bit. If I run the example with GPU (set theano flag device=gpu0) then I get different test accuracy every time, but if I run it on CPU then everything works as expected. My graphics card: NVIDIA GeForce GT 635)

UPDATE3：我把问题缩小了一点。如果我使用 GPU 运行示例（设置 theano 标志 device=gpu0），那么我每次都会获得不同的测试准确度，但是如果我在 CPU 上运行它，那么一切都会按预期进行。我的显卡：NVIDIA GeForce GT 635）

回答by PabTorre

Theano's documentationtalks about the difficulties of seeding random variables and why they seed each graph instance with its own random number generator.

Theano 的文档讨论了播种随机变量的困难以及为什么他们用自己的随机数生成器播种每个图实例。

Sharing a random number generator between different {{{RandomOp}}} instances makes it difficult to producing the same stream regardless of other ops in graph, and to keep {{{RandomOps}}} isolated. Therefore, each {{{RandomOp}}} instance in a graph will have its very own random number generator. That random number generator is an input to the function. In typical usage, we will use the new features of function inputs ({{{value}}}, {{{update}}}) to pass and update the rng for each {{{RandomOp}}}. By passing RNGs as inputs, it is possible to use the normal methods of accessing function inputs to access each {{{RandomOp}}}'s rng. In this approach it there is no pre-existing mechanism to work with the combined random number state of an entire graph. So the proposal is to provide the missing functionality (the last three requirements) via auxiliary functions: {{{seed, getstate, setstate}}}.

在不同的 {{{RandomOp}}} 实例之间共享一个随机数生成器使得无论图中的其他操作如何都难以产生相同的流，并且很难保持 {{{RandomOps}}} 的隔离。因此，图中的每个 {{{RandomOp}}} 实例都有自己的随机数生成器。该随机数生成器是该函数的输入。在典型用法中，我们将使用函数输入的新特征（{{{value}}}，{{{update}}}）来传递和更新每个 {{{RandomOp}}} 的 rng。通过将 RNG 作为输入传递，可以使用访问函数输入的正常方法来访问每个 {{{RandomOp}}} 的 rng。在这种方法中，没有预先存在的机制来处理整个图的组合随机数状态。

They also provide exampleson how to seed all the random number generators.

他们还提供了有关如何为所有随机数生成器播种的示例。

You can also seed all of the random variables allocated by a RandomStreams object by that object's seed method. This seed will be used to seed a temporary random number generator, that will in turn generate seeds for each of the random variables.

您还可以通过该对象的种子方法为 RandomStreams 对象分配的所有随机变量做种子。该种子将用于为临时随机数生成器提供种子，该生成器将依次为每个随机变量生成种子。

>>> srng.seed(902340)  # seeds rv_u and rv_n with different seeds each

回答by Yelaman

I agree with the previous comment, but reproducible results sometimes needs the same environment(e.g. installed packages, machine characteristics and so on). So that, I recommend to copy your environment to other place in case to have reproducible results. Try to use one of the next technologies:

我同意之前的评论，但可重现的结果有时需要相同的环境（例如已安装的包、机器特性等）。因此，我建议将您的环境复制到其他地方，以防得到可重现的结果。尝试使用以下技术之一：

1. Docker. If you have a Linux this very easy to move your environment to other place. Also you can try to use DockerHub.
2. Binder. This is a cloud platform for reproducing scientific experiments.
3. Everware. This is yet another cloud platform for "reusable science". See the project repositoryon Github.

1. 码头工人。如果您有 Linux，这很容易将您的环境移动到其他地方。您也可以尝试使用DockerHub。
2. 粘合剂。这是一个复制科学实验的云平台。
3. 永远软件。这是“可重用科学”的又一个云平台。请参阅Github 上的项目存储库。

回答by tauseef_CuriousGuy

I have trained and tested Sequential()kind of neural networks using Keras. I performed non linear regression on noisy speech data. I used the following code to generate random seed :

我已经Sequential()使用 Keras训练和测试了某种神经网络。我对嘈杂的语音数据进行了非线性回归。我使用以下代码生成随机种子：

import numpy as np
seed = 7
np.random.seed(seed)

I get the exact same results of val_losseach time I train and test on the same data.

val_loss每次对相同数据进行训练和测试时，我都会得到完全相同的结果。

回答by Alex

I would like to add something to the previous answers. If you use python 3and you want to get reproducible results for every run, you have to

我想在以前的答案中添加一些内容。如果您使用python 3并且希望每次运行都获得可重现的结果，则必须

1. set numpy.random.seed in the beginning of your code
2. give PYTHONHASHSEED=0 as a parameter to the python interpreter

1. 在代码的开头设置 numpy.random.seed
2. 将 PYTHONHASHSEED=0 作为参数提供给 python 解释器

回答by kepler

I finally got reproducible results with my code. It's a combination of answers I saw around the web. The first thing is doing what @alex says:

我终于用我的代码得到了可重现的结果。这是我在网上看到的答案的组合。第一件事就是按照@alex 所说的去做：

1. Set numpy.random.seed;
2. Use PYTHONHASHSEED=0for Python 3.

1. 设置numpy.random.seed;
2. 使用PYTHONHASHSEED=0的Python 3。

Then you have to solve the issue noted by @user2805751 regarding cuDNN by calling your Keras code with the following additional THEANO_FLAGS:

然后，您必须通过使用以下附加内容调用您的 Keras 代码来解决 @user2805751 指出的有关 cuDNN 的问题THEANO_FLAGS：

1. dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic

3. dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic

And finally, you have to patch your Theano installation as per this comment, which basically consists in:

最后，你必须按照这个评论修补你的 Theano 安装，它基本上包括：

1. replacing all calls to *_dev20operator by its regular version in theano/sandbox/cuda/opt.py.

4. 将所有对*_dev20operator 的调用替换为theano/sandbox/cuda/opt.py.

This should get you the same results for the same seed.

对于相同的种子，这应该会得到相同的结果。

Note that there might be a slowdown. I saw a running time increase of about 10%.

请注意，可能会出现放缓。我看到运行时间增加了大约 10%。

回答by Outcast

You can find the answer at the Keras docs: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development.

您可以在 Keras 文档中找到答案：https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development 。

In short, to be absolutely sure that you will get reproducible results with your python script on one computer's/laptop's CPUthen you will have to do the following:

简而言之，要绝对确定您将在一台计算机/笔记本电脑的 CPU 上使用您的 python 脚本获得可重现的结果，那么您必须执行以下操作：

1. Set the PYTHONHASHSEEDenvironment variable at a fixed value
2. Set the pythonbuilt-in pseudo-random generator at a fixed value
3. Set the numpypseudo-random generator at a fixed value
4. Set the tensorflowpseudo-random generator at a fixed value
5. Configure a new global tensorflowsession

1. 将PYTHONHASHSEED环境变量设置为固定值
2. 将python内置的伪随机生成器设置为固定值
3. 将numpy伪随机生成器设置为固定值
4. 将tensorflow伪随机生成器设置为固定值
5. 配置新的全局tensorflow会话

Following the Keraslink at the top, the source code I am using is the following:

按照Keras顶部的链接，我使用的源代码如下：

# Seed value
# Apparently you may use different seed values at each stage
seed_value= 0

# 1. Set the `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)

# 2. Set the `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)

# 3. Set the `numpy` pseudo-random generator at a fixed value
import numpy as np
np.random.seed(seed_value)

# 4. Set the `tensorflow` pseudo-random generator at a fixed value
import tensorflow as tf
tf.random.set_seed(seed_value)
# for later versions: 
# tf.compat.v1.set_random_seed(seed_value)

# 5. Configure a new global `tensorflow` session
from keras import backend as K
session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)
# for later versions:
# session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
# sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
# tf.compat.v1.keras.backend.set_session(sess)

It is needless to say that you do not have to to specify any seedor random_stateat the numpy, scikit-learnor tensorflow/kerasfunctions that you are using in your python script exactly because with the source code above we set globally their pseudo-random generators at a fixed value.

不用说，您不必完全指定您在 Python 脚本中使用的任何seed或random_state在numpy,scikit-learn或tensorflow/keras函数，因为在上面的源代码中，我们将它们的伪随机生成器全局设置为固定值。

回答by Victor Villacorta

This works for me:

这对我有用：

SEED = 123456
import os
import random as rn
import numpy as np
from tensorflow import set_random_seed

os.environ['PYTHONHASHSEED']=str(SEED)
np.random.seed(SEED)
set_random_seed(SEED)
rn.seed(SEED)

回答by user2543623

The problem is now solved in Tensorflow 2.0 ! I had the same issue with TF 1.x (see If Keras results are not reproducible, what's the best practice for comparing models and choosing hyper parameters?) but

这个问题现在在 Tensorflow 2.0 中解决了！我在 TF 1.x 中遇到了同样的问题（请参阅 如果 Keras 结果不可重复，比较模型和选择超参数的最佳做法是什么？）但是

import os
####*IMPORANT*: Have to do this line *before* importing tensorflow
os.environ['PYTHONHASHSEED']=str(1)

import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers 
import random
import pandas as pd
import numpy as np

def reset_random_seeds():
   os.environ['PYTHONHASHSEED']=str(1)
   tf.random.set_seed(1)
   np.random.seed(1)
   random.seed(1)

#make some random data
reset_random_seeds()
NUM_ROWS = 1000
NUM_FEATURES = 10
random_data = np.random.normal(size=(NUM_ROWS, NUM_FEATURES))
df = pd.DataFrame(data=random_data, columns=['x_' + str(ii) for ii in range(NUM_FEATURES)])
y = df.sum(axis=1) + np.random.normal(size=(NUM_ROWS))

def run(x, y):
    reset_random_seeds()

    model = keras.Sequential([
            keras.layers.Dense(40, input_dim=df.shape[1], activation='relu'),
            keras.layers.Dense(20, activation='relu'),
            keras.layers.Dense(10, activation='relu'),
            keras.layers.Dense(1, activation='linear')
        ])
    NUM_EPOCHS = 500
    model.compile(optimizer='adam', loss='mean_squared_error')
    model.fit(x, y, epochs=NUM_EPOCHS, verbose=0)
    predictions = model.predict(x).flatten()
    loss = model.evaluate(x,  y) #This prints out the loss by side-effect

#With Tensorflow 2.0 this is now reproducible! 
run(df, y)
run(df, y)
run(df, y)

回答by Richard Rudd-Orthner

The Conference Paper: Non-Random Weight Initialisation in Deep Learning Networks for Repeatable Determinism, publication date Jun 5, 2019 presented at 10th IEEE International Conference Dependable Systems, Services and Technologies (DESSERT-19) at Leeds Beckett University (LBU), United Kingdom, UK, Ireland and the Ukrainian section of IEEE June 5-7, 2019

会议论文：用于可重复确定性的深度学习网络中的非随机权重初始化，发表日期 2019 年 6 月 5 日，发表于英国利兹贝克特大学 (LBU) 的第 10 届 IEEE 国际可靠系统、服务和技术会议 (DESSERT-19) 、英国、爱尔兰和 IEEE 乌克兰部分 2019 年 6 月 5-7 日

https://ieeexplore.ieee.org/document/8770007

https://ieeexplore.ieee.org/document/8770007

shows how to get repeatable results by enforcing critical regions of code.

展示了如何通过强制执行代码的关键区域来获得可重复的结果。

it has been extended to a Journal Paper: Repeatable Determinism using Non-Random Weight Initialisations in Smart City Applications of Deep Learning publication in The Journal of Reliable Intelligent Environments in a Smart Cities special edition, and uses glorot xavier limts and achieve the same accuracy with perceptron layers but grow the weight in to a linear order which may have an advantage for rule extraction in perceptron layers.

它已扩展为期刊论文：在智能城市应用的深度学习中使用非随机权重初始化的可重复确定性发表在智能城市特别版的可靠智能环境期刊中，并使用 glorot xavier 限制并达到与感知器层，但将权重增加到线性顺序，这可能有利于感知器层中的规则提取。