[Feature Request] Make streaming metrics resettable

[untom]

Hi!
Currently, the streaming metrics are (as far as I know) not resettable. I'd like to be able to e.g. reset the counter after each epoch. This way, having e.g. a very bad accuracy in the beginning of training will not still influence the accuracy value ten epochs later. It makes it easier to compare my results to runs obtained outside tensorflow.

The only workaround I found is to do sess.run(tf.initialize_local_variables()) after each epoch, but of course this can have bad side effects if I have other local variables that I don't want to reset.

Or is there a way to achieve what I want that I didn't think of?

[tatatodd]

@untom Thanks for filing the issue! The request sounds reasonable to me. We welcome contributions!

Maybe @nathansilberman @langmore might know whether there's some existing way to reset streaming metrics; I couldn't come up with anything either.

[AshishBora]

Streaming metrics add two local variables total and count. You can find and reset them to get the required behavior. Please see the example below. Hope this helps.

import tensorflow as tf

value = 0.1
with tf.name_scope('foo'):
    mean_value, update_op = tf.contrib.metrics.streaming_mean(value)

init_op = [tf.initialize_variables(tf.local_variables())]
stream_vars = [i for i in tf.local_variables() if i.name.split('/')[0] == 'foo']
reset_op = [tf.initialize_variables(stream_vars)]
with tf.Session() as sess:
    sess.run(init_op)
    for j in range(3):
        for i in range(9):
            _, total, count = sess.run([update_op] + stream_vars) 
            mean_val = sess.run([mean_value])
            print total, count, mean_val
        sess.run(reset_op)
        print ''

[untom]

Thanks, that is very useful. It seems to me that i would make sense if the metrics would all return a reset_op together with the other ops. But that would break existing code. A workaround could be to only return the reset_op if the user specifies an optional return_reset_op=True parameter when creating the OP. Is that a sensible approach for this?

[littleDing]

Agree with @untom .
When building a multi-target training graph, there might be tens of prediction to evaluate on tens of labels. It would be nightmare to generate this local_variable reset op manually.
Following @AshishBora 's way, I'am getting the last 4 variable from local_variable list. But this doesn't seems to be extensible.

[astorfi]

Thank you @AshishBora for your guidance. However I am encountering an issue using TF-Slim. The problem is that the TF-Slim use tf.train.Supervisor() and after that the graph is finalized and cannot be be modified. I get the following error:

raise RuntimeError("Graph is finalized and cannot be modified.") RuntimeError: Graph is finalized and cannot be modified.

So the above solution won't work.

Is there any other solution?
Thanks

[nathansilberman]

You can run the following to reset the local variables used by metric computation:
session.run(tf.local_variables_initializer())

To avoid the issue of graph finalization, just create a reference to this op BEFORE session creation:

reset_op = tf.local_variables_initializer()
... session created
session.run(reset_op)

[foxik]

Note that this is tightly coupled to #9498.

[astorfi]

Thank you @nathansilberman .
Unfortunately it didn't work. Looks like the supervisor is much smarter than that!
Moreover I the the problem with tf.local_variables_initializer() is that by defining the reset_op it may reset all variables.
I am not %100 sure though.
But thank you so much for the hint. I think I may find the solution from this.

[nathansilberman]

Alternatively, you can always define the metrics within a name scope and only reset the local variables within that name scope.

…

On Fri, Jun 2, 2017 at 11:01 AM, Amirsina Torfi ***@***.***> wrote: Thank you @nathansilberman <https://github.com/nathansilberman> . Unfortunately it didn't work. Looks like the supervisor is much smarter than that! Moreover I the the problem with tf.local_variables_initializer() is that by defining the reset_op it may reset all variables. I am not %100 sure though. But thank you so much for the hint. I think I may find the solution from this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4814 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUMnHAfTXrDFN2hDwEKY06TUPgPd57jNks5sACPjgaJpZM4KQz81> .

[astorfi]

@untom I made it. Thanks to @nathansilberman .

I made a pull request to tensorflow for changing slim.learning.train and I added a reset_op for metric.
Although different changes must be made to train_image_classifier.py provided by slim, but the first pull request is necessary to accept.

pull request:
#10400
#10400 (comment)