Python 如何在 keras 中实现自定义指标？

how to implement custom metric in keras?

提问by Philippe C

I get this error :

我收到此错误：

sum() got an unexpected keyword argument 'out'

sum() 得到了一个意外的关键字参数“out”

when I run this code:

当我运行此代码时：

import pandas as pd, numpy as np
import keras
from keras.layers.core import Dense, Activation
from keras.models import Sequential

def AUC(y_true,y_pred):
    not_y_pred=np.logical_not(y_pred)
    y_int1=y_true*y_pred
    y_int0=np.logical_not(y_true)*not_y_pred
    TP=np.sum(y_pred*y_int1)
    FP=np.sum(y_pred)-TP
    TN=np.sum(not_y_pred*y_int0)
    FN=np.sum(not_y_pred)-TN
    TPR=np.float(TP)/(TP+FN)
    FPR=np.float(FP)/(FP+TN)
    return((1+TPR-FPR)/2)

# Input datasets

train_df = pd.DataFrame(np.random.rand(91,1000))
train_df.iloc[:,-2]=(train_df.iloc[:,-2]>0.8)*1


model = Sequential()
model.add(Dense(output_dim=60, input_dim=91, init="glorot_uniform"))
model.add(Activation("sigmoid"))
model.add(Dense(output_dim=1, input_dim=60, init="glorot_uniform"))
model.add(Activation("sigmoid"))

model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=[AUC])


train_df.iloc[:,-1]=np.ones(train_df.shape[0]) #bias
X=train_df.iloc[:,:-1].values
Y=train_df.iloc[:,-1].values
print X.shape,Y.shape

model.fit(X, Y, batch_size=50,show_accuracy = False, verbose = 1)

Is it possible to implement a custom metric aside from doing a loop on batches and editing the source code?

除了对批处理进行循环和编辑源代码之外，是否可以实现自定义指标？

采纳答案by Marcin Mo?ejko

The problem is that y_predand y_trueare not NumPy arrays but either Theano or TensorFlow tensors. That's why you got this error.

问题是，y_pred并y_true没有与NumPy阵列但无论哪种Theano或TensorFlow张量。这就是您收到此错误的原因。

You can define your custom metrics but you have to remember that its arguments are those tensors – not NumPy arrays.

你可以定义你的自定义指标，但你必须记住它的参数是那些张量——而不是 NumPy 数组。

回答by vogdb

Here I'm answering to OP's topic question rather than his exact problem. I'm doing this as the question shows up in the top when I google the topic problem.

在这里，我正在回答 OP 的主题问题，而不是他的确切问题。我这样做是因为当我用谷歌搜索主题问题时，问题会出现在顶部。

You can implement a custom metric in two ways.

您可以通过两种方式实施自定义指标。

1. As mentioned in Keras docu.

   import keras.backend as K
   
   def mean_pred(y_true, y_pred):
       return K.mean(y_pred)
   
   model.compile(optimizer='sgd',
             loss='binary_crossentropy',
             metrics=['accuracy', mean_pred])
   

   But here you have to remember as mentioned in Marcin Mo?ejko's answer that y_trueand y_predare tensors. So in order to correctly calculate the metric you need to use keras.backendfunctionality. Please look at this SO question for details How to calculate F1 Macro in Keras?

2. Or you can implement it in a hacky way as mentioned in Keras GH issue. For that you need to use callbacksargument of model.fit.

   import keras as keras
   import numpy as np
   from keras.optimizers import SGD
   from sklearn.metrics import roc_auc_score
   
   model = keras.models.Sequential()
   # ...
   sgd = SGD(lr=0.001, momentum=0.9)
   model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])
   
   
   class Metrics(keras.callbacks.Callback):
       def on_train_begin(self, logs={}):
           self._data = []
   
       def on_epoch_end(self, batch, logs={}):
           X_val, y_val = self.validation_data[0], self.validation_data[1]
           y_predict = np.asarray(model.predict(X_val))
   
           y_val = np.argmax(y_val, axis=1)
           y_predict = np.argmax(y_predict, axis=1)
   
           self._data.append({
               'val_rocauc': roc_auc_score(y_val, y_predict),
           })
           return
   
       def get_data(self):
           return self._data
   
   metrics = Metrics()
   history = model.fit(X_train, y_train, epochs=100, validation_data=(X_val, y_val), callbacks=[metrics])
   metrics.get_data()
   

1. 正如Keras docu 中提到的那样。

   import keras.backend as K
   
   def mean_pred(y_true, y_pred):
       return K.mean(y_pred)
   
   model.compile(optimizer='sgd',
             loss='binary_crossentropy',
             metrics=['accuracy', mean_pred])
   

   但是在这里你必须记住，正如 Marcin Mo?ejko 的回答中提到的那样，y_true并且y_pred是张量。因此，为了正确计算指标，您需要使用keras.backend功能。请查看这个 SO 问题的详细信息如何在 Keras 中计算 F1 宏？

2. 或者你可以像Keras GH issue 中提到的那样以一种hacky 的方式实现它。对于您需要使用callbacks的参数model.fit。

   import keras as keras
   import numpy as np
   from keras.optimizers import SGD
   from sklearn.metrics import roc_auc_score
   
   model = keras.models.Sequential()
   # ...
   sgd = SGD(lr=0.001, momentum=0.9)
   model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])
   
   
   class Metrics(keras.callbacks.Callback):
       def on_train_begin(self, logs={}):
           self._data = []
   
       def on_epoch_end(self, batch, logs={}):
           X_val, y_val = self.validation_data[0], self.validation_data[1]
           y_predict = np.asarray(model.predict(X_val))
   
           y_val = np.argmax(y_val, axis=1)
           y_predict = np.argmax(y_predict, axis=1)
   
           self._data.append({
               'val_rocauc': roc_auc_score(y_val, y_predict),
           })
           return
   
       def get_data(self):
           return self._data
   
   metrics = Metrics()
   history = model.fit(X_train, y_train, epochs=100, validation_data=(X_val, y_val), callbacks=[metrics])
   metrics.get_data()
   

回答by ahmedhosny

you can pass a model.predict() in your AUC metric function. [this will iterate on bacthes so you might be better off using model.predict_on_batch(). Assuming you have something like a softmax layer as output (something that outputs probabilities), then you can use that together with sklearn.metric to get the AUC.

您可以在 AUC 指标函数中传递 model.predict() 。[这将迭代 bacthes，因此您最好使用 model.predict_on_batch()。假设你有一个类似 softmax 层的输出（输出概率的东西），那么你可以将它与 sklearn.metric 一起使用来获得 AUC。

from sklearn.metrics import roc_curve, auc

from here

从这里

def sklearnAUC(test_labels,test_prediction):
    n_classes = 2
    # Compute ROC curve and ROC area for each class
    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    for i in range(n_classes):
        # ( actual labels, predicted probabilities )
        fpr[i], tpr[i], _ = roc_curve(test_labels[:, i], test_prediction[:, i])
        roc_auc[i] = auc(fpr[i], tpr[i])

    return round(roc_auc[0],3) , round(roc_auc[1],3)

now make your metric

现在制定你的指标

# gives a numpy array like so [ [0.3,0.7] , [0.2,0.8] ....]    
Y_pred = model.predict_on_batch ( X_test  ) 
# Y_test looks something like [ [0,1] , [1,0] .... ]
# auc1 and auc2 should be equal
auc1 , auc2 = sklearnAUC(  Y_test ,  Y_pred )