Accuracy and metrics with Model

[tristandeleu]

In #286 I briefly talk about the idea of separating the metrics computation (like the accuracy) from Model. At the moment, you can keep track of the accuracy in the logs (both history and console logs) easily with the flag show_accuracy=True in Model.fit(). Unfortunately this is limited to the accuracy and does not handle any other metrics that could be valuable to the user.

We could have the computation of these metrics outside of Model and call them with callbacks if one wants to keep track of them during training. It may be valuable for the future but for it could also raise some issues short term

• It would be impossible to log the accuracy (or any other metrics) with the base logger as callbacks do not interact with each other. One solution would be to let the user create her own logger on a different level of verbosity (possibly by inheriting from the current BaseLogger).
• We would have to think about how to access the training and validation set with callbacks.

[fchollet]

Right now I think the best way to achieve custom metrics monitoring is by calling .predict() on some test data after each epoch (outside of the fit loop, eg. by training one epoch at a time with fit or batch by batch with train) then running your own metrics on the results.

It's not that much hassle (esp. for advanced users). But anyway, adding built-in support for custom metrics in fit can definitely be interested. The main potential obstacle is that it would require compiling the custom metrics as a Theano function (for efficiency...).

[rolandmaio]

A workaround is to write your custom metric in keras/metrics.py. A couple of disadvantages of this approach are that you are editing the code base itself, and you have to express the custom metric either in terms of the backend you use or the Keras backend module.

For example the following implements the Sorensen-Dice index on a binary classification problem. It is written in terms of the Keras backend, but I have only verified that it works with Theano.

# In keras/metrics.py
def  dice(y_true, y_pred):
    # Symbolically compute the intersection
    y_int = y_true*y_pred
    # Technically this is the negative of the Sorensen-Dice index. This is done for
    # minimization purposes
    return -(2*K.sum(y_int) / (K.sum(y_true) + K.sum(y_pred)))

We can then add the dice metric when we compile the model:

model.compile(optimizer=optimizer, loss="binary_crossentropy", metrics=["dice"])

Now with ProgbarLogger the log output will look like this:

Epoch 1/1
2300/9000 [======>.......................] - ETA 26s - loss: 4.1548 - dice: -0.7071

This works equally well for custom objectives. Add the custom objective to keras/objectives.py, and then set the loss argument to the name of the custom objective function when compiling the model.

The first issue with this approach mentioned has to be addressed in providing users with a mechanism for writing custom metrics/objectives. However, it appears that some mechanism for injecting the custom metrics into the namespace of the built-in objectives prior to compiling the model should suffice.

[fchollet]

You do not need to put your custom metrics in metrics.py. You can pass function to the metrics argument, and these functions can be defined wherever.

[khozzy]

Hi, I'm new to Keras. Please tell me how to properly implement a custom metric. My code looks like this (I'm using scikit-learn wrapper):

def custom_metric(act, pred):
    print("Ahh, I'm here")
    print(act)
    print(pred)
    return 0.2 # dummy

def create_baseline():
    model = Sequential()
    model.add(Dense(no_features, input_dim=no_features, init='normal', activation='relu'))
    model.add(Dense(1, init='normal', activation='sigmoid'))

    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[custom_metric])

    return model

estimator = KerasClassifier(build_fn=create_baseline, nb_epoch=epochs, batch_size=batch_size, verbose=0)

results = cross_val_score(estimator, X.as_matrix(), Y, cv=kfold, scoring=scoring)

Which leads to:

Ahh, I'm here
Tensor("dense_64_target:0", shape=(?, ?), dtype=float32)
Tensor("Sigmoid_56:0", shape=(?, 1), dtype=float32)

Can not convert a float into a Tensor or Operation.

How to write a general metric function that will work for all backends (or cast arguments act and pred to numpy arrays)? Do you guys have any examples?

Regards

[varun-bankiti]

@khozzy I also need to do the same, were you able to solve? If so can you explain how?

[Cogitans]

You need to represent the value to be returned as a Tensor. Without directly accessing the backend (if, for example, you wanted to keep Keras agnostic to Theano vs TensorFlow), I'm not quite sure how to do this.

If you're willing to break this abstraction barrier, you can directly access tensorflow methods to return a tensor. You can't return a numpy array, but should rather return a tensor that will later be evaluated to a numpy array.

[zillionare]

When we pass metrics = ['accuracy'] in compile stage, what happend actually under the hood? Which kind of accuracy is computed, since keras has binary_accuracy, categorical_accuracy, ... and others?

[morrigan]

@hbaaron I am not sure if it changes depending on the loss function (haven't looked in the source code) but it's best that you check yourself using all metrics metrics=['accuracy', metrics.binary_accuracy, metrics.categorical_accuracy, metrics.sparse_categorical_accuracy, metrics.top_k_categorical_accuracy ]. For me it was categorical_accuracy which returned the same result.

[alyato]

hi,@morrigan @hbaaron @rolandmaio Im also confusing the metrics=['accuracy']. when i check the metric.py, there are many functions. They always calcuate the (y_pred,y_true). Im check the loss_function,it also do the same work. So im confused what the metrics does.

[zeka0]

I have a problem dealing with Callbacks here. I want to use categorical_accuracy as metrics and binary_crossentropy as loss function. But in EarlyStopping I tried using 'val_acc' as criteria then it pops out an error that it can't find val_acc. Is there a way to solve this problem?

[hoangcuong2011]

@khozzy : I also tried to find a solution to this problem. After Googling for a while, I found doing something similar to this can help
http://jmbeaujour.com/ml/2017/03/20/CoeffDetermination_CustomMetric4Keras/
https://stackoverflow.com/questions/41458859/keras-custom-metric-for-single-class-accuracy
The key to the solution is to use keras backend engine
from keras import backend as K

However, a better solution for me is to have a loop:

1. train your model using a few epochs
2. save the model,
3. load the model
4. test the result with your own metric
5. continue the loop

This gives us more flexible to implement I think.

I am not knowledgeable about Keras by anymeans. I just post it here and hope it helps.