Python Tensorflow：使用 Adam 优化器

Tensorflow: Using Adam optimizer

提问by pythonic metaphor

I am experimenting with some simple models in tensorflow, including one that looks very similar to the first MNIST for ML Beginners example, but with a somewhat larger dimensionality. I am able to use the gradient descent optimizer with no problems, getting good enough convergence. When I try to use the ADAM optimizer, I get errors like this:

我正在 tensorflow 中试验一些简单的模型，包括一个看起来与第一个MNIST for ML Beginners example非常相似的模型，但维度稍大一些。我可以毫无问题地使用梯度下降优化器，获得足够好的收敛性。当我尝试使用 ADAM 优化器时，出现如下错误：

tensorflow.python.framework.errors.FailedPreconditionError: Attempting to use uninitialized value Variable_21/Adam
     [[Node: Adam_2/update_Variable_21/ApplyAdam = ApplyAdam[T=DT_FLOAT, use_locking=false, _device="/job:localhost/replica:0/task:0/cpu:0"](Variable_21, Variable_21/Adam, Variable_21/Adam_1, beta1_power_2, beta2_power_2, Adam_2/learning_rate, Adam_2/beta1, Adam_2/beta2, Adam_2/epsilon, gradients_11/add_10_grad/tuple/control_dependency_1)]]

where the specific variable that complains about being uninitialized changes depending on the run. What does this error mean? And what does it suggest is wrong? It seems to occur regardless of the learning rate I use.

其中抱怨未初始化的特定变量根据运行而变化。这个错误是什么意思？它表明什么是错误的？无论我使用什么学习率，它似乎都会发生。

采纳答案by Touts

The AdamOptimizer class creates additional variables, called "slots", to hold values for the "m" and "v" accumulators.

AdamOptimizer 类创建额外的变量，称为“槽”，以保存“m”和“v”累加器的值。

See the source here if you're curious, it's actually quite readable: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py#L39. Other optimizers, such as Momentum and Adagrad use slots too.

如果您好奇，请参阅此处的源代码，它实际上非常易读：https: //github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py#L39。其他优化器，例如 Momentum 和 Adagrad 也使用槽。

These variables must be initialized before you can train a model.

必须先初始化这些变量，然后才能训练模型。

The normal way to initialize variables is to call tf.initialize_all_variables()which adds ops to initialize the variables present in the graph when it is called.

初始化变量正常的方法是调用tf.initialize_all_variables()它增加了OPS初始化变量出现在图表当它被称为。

(Aside: unlike its name suggests, initialize_all_variables() does not initialize anything, it only add ops that will initialize the variables when run.)

（旁白：不像它的名字暗示的那样， initialize_all_variables() 不初始化任何东西，它只添加在运行时初始化变量的操作。）

What you must do is call initialize_all_variables() afteryou have added the optimizer:

您必须做的是在添加优化器后调用 initialize_all_variables() ：

...build your model...
# Add the optimizer
train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
# Add the ops to initialize variables.  These will include 
# the optimizer slots added by AdamOptimizer().
init_op = tf.initialize_all_variables()

# launch the graph in a session
sess = tf.Session()
# Actually intialize the variables
sess.run(init_op)
# now train your model
for ...:
  sess.run(train_op)

回答by Cameron Fraser

I was having a similar problem. (No problems training with GradientDescent optimizer, but error raised when using to Adam Optimizer, or any other optimizer with its own variables)

我遇到了类似的问题。（使用 GradientDescent 优化器进行训练没有问题，但使用 Adam Optimizer 或任何其他具有自己变量的优化器时会出现错误）

Changing to an interactive session solved this problem for me.

更改为交互式会话为我解决了这个问题。

sess = tf.Session()

into

进入

sess = tf.InteractiveSession()

回答by Salvador Dali

FailedPreconditionError: Attempting to use uninitialized valueis one of the most frequent errors related to tensorflow. From official documentation, FailedPreconditionError

FailedPreconditionError: Attempting to use uninitialized value是与 tensorflow 相关的最常见的错误之一。来自官方文档，FailedPreconditionError

This exception is most commonly raised when running an operation that reads a tf.Variable before it has been initialized.

当运行在初始化之前读取 tf.Variable 的操作时，最常引发此异常。

In your case the error even explains what variable was not initialized: Attempting to use uninitialized value Variable_1. One of the TF tutorials explains a lot about variables, their creation/initialization/saving/loading

在您的情况下，错误甚至解释了未初始化的变量：Attempting to use uninitialized value Variable_1。TF 教程之一解释了很多关于变量，它们的创建/初始化/保存/加载

Basically to initialize the variable you have 3 options:

基本上要初始化变量，您有 3 个选项：

• initialize all global variables with tf.global_variables_initializer()
• initialize variables you care about with tf.variables_initializer(list_of_vars). Notice that you can use this function to mimic global_variable_initializer: tf.variable_initializers(tf.global_variables())
• initialize only one variable with var_name.initializer

• 初始化所有全局变量 tf.global_variables_initializer()
• 初始化你关心的变量tf.variables_initializer(list_of_vars)。请注意，您可以使用此函数来模拟 global_variable_initializer：tf.variable_initializers(tf.global_variables())
• 只初始化一个变量 var_name.initializer

I almost always use the first approach. Remember you should put it inside a session run. So you will get something like this:

我几乎总是使用第一种方法。请记住，您应该将它放在会话运行中。所以你会得到这样的东西：

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

If your are curious about more information about variables, read this documentationto know how to report_uninitialized_variablesand check is_variable_initialized.

如果您对有关变量的更多信息感到好奇，请阅读此文档以了解如何report_uninitialized_variables检查is_variable_initialized.

回答by booyoungxu

run init after AdamOptimizer，and without define init before or run init

在 AdamOptimizer 之后运行 init，并且之前不定义 init 或运行 init

sess.run(tf.initialize_all_variables())

sess.run(tf.initialize_all_variables())

or

或者

sess.run(tf.global_variables_initializer())

sess.run(tf.global_variables_initializer())

回答by LYu

You need to call tf.global_variables_initializer()on you session, like

你需要打电话tf.global_variables_initializer()给你的会议，比如

init = tf.global_variables_initializer()
sess.run(init)

Full example is available in this great tutorial https://www.tensorflow.org/get_started/mnist/mechanics

这个很棒的教程提供了完整的示例 https://www.tensorflow.org/get_started/mnist/mechanics