Python 如何在 TensorFlow 中添加正则化？

How to add regularizations in TensorFlow?

提问by Lifu Huang

I found in many available neural network code implemented using TensorFlow that regularization terms are often implemented by manually adding an additional term to loss value.

我在许多使用 TensorFlow 实现的可用神经网络代码中发现，正则化项通常是通过手动向损失值添加附加项来实现的。

My questions are:

我的问题是：

1. Is there a more elegant or recommended way of regularization than doing it manually?

2. I also find that get_variablehas an argument regularizer. How should it be used? According to my observation, if we pass a regularizer to it (such as tf.contrib.layers.l2_regularizer, a tensor representing regularized term will be computed and added to a graph collection named tf.GraphKeys.REGULARIZATOIN_LOSSES. Will that collection be automatically used by TensorFlow (e.g. used by optimizers when training)? Or is it expected that I should use that collection by myself?

1. 有没有比手动更优雅或推荐的正则化方法？

2. 我也觉得get_variable有道理regularizer。应该如何使用？根据我的观察，如果我们向它传递一个正则化器（例如tf.contrib.layers.l2_regularizer，将计算表示正则化项的张量并将其添加到名为 的图集合中tf.GraphKeys.REGULARIZATOIN_LOSSES。TensorFlow 会自动使用该集合吗（例如，优化器在训练时使用）？或者是否希望我自己使用该集合？

采纳答案by Lukasz Kaiser

As you say in the second point, using the regularizerargument is the recommended way. You can use it in get_variable, or set it once in your variable_scopeand have all your variables regularized.

正如你在第二点所说，使用regularizer参数是推荐的方式。您可以在 中使用它get_variable，也可以在您的中设置一次，variable_scope然后对所有变量进行正则化。

The losses are collected in the graph, and you need to manually add them to your cost function like this.

损失收集在图中，您需要像这样手动将它们添加到成本函数中。

  reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
  reg_constant = 0.01  # Choose an appropriate one.
  loss = my_normal_loss + reg_constant * sum(reg_losses)

Hope that helps!

希望有帮助！

回答by bluenote10

A few aspects of the existing answer were not immediately clear to me, so here is a step-by-step guide:

现有答案的几个方面对我来说不是很清楚，所以这里是一个分步指南：

1. Define a regularizer. This is where the regularization constant can be set, e.g.:

   regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
   

2. Create variables via:

       weights = tf.get_variable(
           name="weights",
           regularizer=regularizer,
           ...
       )
   

   Equivalently, variables can be created via the regular weights = tf.Variable(...)constructor, followed by tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, weights).

3. Define some lossterm and add the regularization term:

   reg_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
   reg_term = tf.contrib.layers.apply_regularization(regularizer, reg_variables)
   loss += reg_term
   

   Note: It looks like tf.contrib.layers.apply_regularizationis implemented as an AddN, so more or less equivalent to sum(reg_variables).

1. 定义一个正则化器。这是可以设置正则化常数的地方，例如：

   regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
   

2. 通过以下方式创建变量：

       weights = tf.get_variable(
           name="weights",
           regularizer=regularizer,
           ...
       )
   

   等效地，可以通过常规weights = tf.Variable(...)构造函数创建变量，后跟tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, weights).

3. 定义一些loss术语并添加正则化术语：

   reg_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
   reg_term = tf.contrib.layers.apply_regularization(regularizer, reg_variables)
   loss += reg_term
   

   注意：它看起来像是tf.contrib.layers.apply_regularization实现为AddN，因此或多或少相当于sum(reg_variables)。

回答by alyaxey

I'll provide a simple correct answer since I didn't find one. You need two simple steps, the rest is done by tensorflow magic:

我将提供一个简单的正确答案，因为我没有找到。你只需要两个简单的步骤，剩下的由 tensorflow magic 完成：

1. Add regularizers when creating variables or layers:

   tf.layers.dense(x, kernel_regularizer=tf.contrib.layers.l2_regularizer(0.001))
   # or
   tf.get_variable('a', regularizer=tf.contrib.layers.l2_regularizer(0.001))
   

2. Add the regularization term when defining loss:

   loss = ordinary_loss + tf.losses.get_regularization_loss()
   

1. 创建变量或层时添加正则化器：

   tf.layers.dense(x, kernel_regularizer=tf.contrib.layers.l2_regularizer(0.001))
   # or
   tf.get_variable('a', regularizer=tf.contrib.layers.l2_regularizer(0.001))
   

2. 在定义损失时添加正则化项：

   loss = ordinary_loss + tf.losses.get_regularization_loss()
   

回答by ComputerScientist

Another option to do this with the contrib.learnlibrary is as follows, based on the Deep MNIST tutorialon the Tensorflow website. First, assuming you've imported the relevant libraries (such as import tensorflow.contrib.layers as layers), you can define a network in a separate method:

使用contrib.learn库执行此操作的另一种选择如下，基于Tensorflow 网站上的Deep MNIST 教程。首先，假设您已导入相关库（例如import tensorflow.contrib.layers as layers），您可以在单独的方法中定义网络：

def easier_network(x, reg):
    """ A network based on tf.contrib.learn, with input `x`. """
    with tf.variable_scope('EasyNet'):
        out = layers.flatten(x)
        out = layers.fully_connected(out, 
                num_outputs=200,
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = tf.nn.tanh)
        out = layers.fully_connected(out, 
                num_outputs=200,
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = tf.nn.tanh)
        out = layers.fully_connected(out, 
                num_outputs=10, # Because there are ten digits!
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = None)
        return out 

Then, in a main method, you can use the following code snippet:

然后，在 main 方法中，您可以使用以下代码片段：

def main(_):
    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)
    x = tf.placeholder(tf.float32, [None, 784])
    y_ = tf.placeholder(tf.float32, [None, 10])

    # Make a network with regularization
    y_conv = easier_network(x, FLAGS.regu)
    weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, 'EasyNet') 
    print("")
    for w in weights:
        shp = w.get_shape().as_list()
        print("- {} shape:{} size:{}".format(w.name, shp, np.prod(shp)))
    print("")
    reg_ws = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES, 'EasyNet')
    for w in reg_ws:
        shp = w.get_shape().as_list()
        print("- {} shape:{} size:{}".format(w.name, shp, np.prod(shp)))
    print("")

    # Make the loss function `loss_fn` with regularization.
    cross_entropy = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
    loss_fn = cross_entropy + tf.reduce_sum(reg_ws)
    train_step = tf.train.AdamOptimizer(1e-4).minimize(loss_fn)

To get this to work you need to follow the MNIST tutorial I linked to earlier and import the relevant libraries, but it's a nice exercise to learn TensorFlow and it's easy to see how the regularization affects the output. If you apply a regularization as an argument, you can see the following:

要使其发挥作用，您需要遵循我之前链接的 MNIST 教程并导入相关库，但这是学习 TensorFlow 的一个很好的练习，并且很容易看出正则化如何影响输出。如果您将正则化作为参数应用，您可以看到以下内容：

- EasyNet/fully_connected/weights:0 shape:[784, 200] size:156800
- EasyNet/fully_connected/biases:0 shape:[200] size:200
- EasyNet/fully_connected_1/weights:0 shape:[200, 200] size:40000
- EasyNet/fully_connected_1/biases:0 shape:[200] size:200
- EasyNet/fully_connected_2/weights:0 shape:[200, 10] size:2000
- EasyNet/fully_connected_2/biases:0 shape:[10] size:10

- EasyNet/fully_connected/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0
- EasyNet/fully_connected_1/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0
- EasyNet/fully_connected_2/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0

Notice that the regularization portion gives you three items, based on the items available.

请注意，正则化部分根据可用项目为您提供三个项目。

With regularizations of 0, 0.0001, 0.01, and 1.0, I get test accuracy values of 0.9468, 0.9476, 0.9183, and 0.1135, respectively, showing the dangers of high regularization terms.

使用 0、0.0001、0.01 和 1.0 的正则化，我分别得到 0.9468、0.9476、0.9183 和 0.1135 的测试准确度值，显示了高正则化项的危险。

回答by MoltenMuffins

If anyone's still looking, I'd just like to add on that in tf.keras you may add weight regularization by passing them as arguments in your layers. An example of adding L2 regularization taken wholesale from the Tensorflow Keras Tutorials site:

如果有人还在看，我只想在 tf.keras 中添加它，您可以通过将它们作为参数传递到图层中来添加权重正则化。从 Tensorflow Keras Tutorials 站点批发获取的添加 L2 正则化的示例：

model = keras.models.Sequential([
    keras.layers.Dense(16, kernel_regularizer=keras.regularizers.l2(0.001),
                       activation=tf.nn.relu, input_shape=(NUM_WORDS,)),
    keras.layers.Dense(16, kernel_regularizer=keras.regularizers.l2(0.001),
                       activation=tf.nn.relu),
    keras.layers.Dense(1, activation=tf.nn.sigmoid)
])

There's no need to manually add in the regularization losses with this method as far as I know.

据我所知，无需使用此方法手动添加正则化损失。

Reference: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit#add_weight_regularization

参考：https: //www.tensorflow.org/tutorials/keras/overfit_and_underfit#add_weight_regularization

回答by ocean

I tested tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)and tf.losses.get_regularization_loss()with one l2_regularizerin the graph, and found that they return the same value. By observing the value's quantity, I guess reg_constant has already make sense on the value by setting the parameter of tf.contrib.layers.l2_regularizer.

我测试tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)，并tf.losses.get_regularization_loss()与一个l2_regularizer在图中，发现他们返回相同的值。通过观察值的数量，我猜 reg_constant 通过设置tf.contrib.layers.l2_regularizer.

回答by tsveti_iko

If you have CNN you may do the following:

如果您有 CNN，您可以执行以下操作：

In your model function:

在您的模型函数中：

conv = tf.layers.conv2d(inputs=input_layer,
                        filters=32,
                        kernel_size=[3, 3],
                        kernel_initializer='xavier',
                        kernel_regularizer=tf.contrib.layers.l2_regularizer(1e-5),
                        padding="same",
                        activation=None) 
...

In your loss function:

在你的损失函数中：

onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=num_classes)
loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)
regularization_losses = tf.losses.get_regularization_losses()
loss = tf.add_n([loss] + regularization_losses)

回答by user3201329

Some answers make me more confused.Here I give two methods to make it clearly.

一些回答让我更糊涂了，这里我给出两种方法说清楚。

#1.adding all regs by hand
var1 = tf.get_variable(name='v1',shape=[1],dtype=tf.float32)
var2 = tf.Variable(name='v2',initial_value=1.0,dtype=tf.float32)
regularizer = tf.contrib.layers.l1_regularizer(0.1)
reg_term = tf.contrib.layers.apply_regularization(regularizer,[var1,var2])
#here reg_term is a scalar

#2.auto added and read,but using get_variable
with tf.variable_scope('x',
        regularizer=tf.contrib.layers.l2_regularizer(0.1)):
    var1 = tf.get_variable(name='v1',shape=[1],dtype=tf.float32)
    var2 = tf.get_variable(name='v2',shape=[1],dtype=tf.float32)
reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
#here reg_losses is a list,should be summed 

Then,it can be added into the total loss

然后，可以将其添加到总损失中

回答by Alex-zhai

cross_entropy = tf.losses.softmax_cross_entropy(
  logits=logits, onehot_labels=labels)

l2_loss = weight_decay * tf.add_n(
     [tf.nn.l2_loss(tf.cast(v, tf.float32)) for v in tf.trainable_variables()])

loss = cross_entropy + l2_loss

回答by Elias Hasle

tf.GraphKeys.REGULARIZATION_LOSSESwill not be added automatically, but there is a simple way to add them:

tf.GraphKeys.REGULARIZATION_LOSSES不会自动添加，但有一个简单的方法来添加它们：

reg_loss = tf.losses.get_regularization_loss()
total_loss = loss + reg_loss

tf.losses.get_regularization_loss()uses tf.add_nto sum the entries of tf.GraphKeys.REGULARIZATION_LOSSESelement-wise. tf.GraphKeys.REGULARIZATION_LOSSESwill typically be a list of scalars, calculated using regularizer functions. It gets entries from calls to tf.get_variablethat have the regularizerparameter specified. You can also add to that collection manually. That would be useful when using tf.Variableand also when specifying activity regularizers or other custom regularizers. For instance:

tf.losses.get_regularization_loss()用于tf.add_n按tf.GraphKeys.REGULARIZATION_LOSSES元素求和。tf.GraphKeys.REGULARIZATION_LOSSES通常是一个标量列表，使用正则化函数计算。它从tf.get_variable具有regularizer指定参数的调用中获取条目。您也可以手动添加到该集合。这在使用tf.Variable以及指定活动正则化器或其他自定义正则化器时会很有用。例如：

#This will add an activity regularizer on y to the regloss collection
regularizer = tf.contrib.layers.l2_regularizer(0.1)
y = tf.nn.sigmoid(x)
act_reg = regularizer(y)
tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, act_reg)

(In this example it would presumably be more effective to regularize x, as y really flattens out for large x.)

（在这个例子中，正则化 x 可能更有效，因为对于大 x，y 真的变平了。）