What-If Tool: thresholded inference problem (confusion matrix/ROC)

[reinhouthooft]

Version info:

• TensorBoard 1.12.0a0
• TensorFlow 1.8.0
• MacOS 10.13.6
• Python 2.7

Description:
Running 2-class classification with a custom estimator results in incorrect confusion matrix/ROC curve values. When dragging the threshold slider, the "actual yes/no" percentages change (see screenshots). Other than that, when using a vocab file to specify the labels ("False", "True"), the legend shows "False" and "undefined". The inference scores seem to return correctly.

I would assume, even if my model would be incorrect, that the "actual" samples are unrelated to the threshold set.

Context:
The classification API is used as

signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: tf.estimator.export.ClassificationOutput(scores=softmax, classes=None)

with softmax a (?,2)-shaped Tensor. This leads to the following signature:

The given SavedModel SignatureDef contains the following input(s):
  inputs['inputs'] tensor_info:
      dtype: DT_STRING
      shape: (-1)
      name: Placeholder:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['scores'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 2)
      name: softmax/Reshape_1:0
Method name is: tensorflow/serving/classify

The ground truth is specified via a numeric integer value in [0,1] (about 97% 0 and 3% 1).

Inference result as shown in the datapoint editor:

[nfelt]

cc @jameswex

[jameswex]

@reinhouthooft Thanks for the bug report. Would you be willing to provide the saved model and a tf record file of examples for me to reproduce the problem with? Or is the model and/or data not for sharing?

[reinhouthooft]

@jameswex Thanks, I will build a minimal model+data that can be shared which reproduces the problem.

[reinhouthooft]

@jameswex The mock data + model can be found here: tensorboard_issue1463.zip

I load it via

docker run -p 8500:8500 --mount type=bind,source=<path>/tensorboard_issue1463/analysis_models/,target=/models/analysis_model/ \
-e MODEL_NAME=analysis_model -t tensorflow/serving

and load into the what-if tool via inference address 0.0.0.0:8500, model name analysis_model, and model signature serving_analysis. Path to examples set to <path>/tensorboard_issue1463/example.tfrecord. The ground truth is set to the gt feature.

[jameswex]

@reinhouthooft The link https://file.io/OphIwR gives me a 404 error.

[jameswex]

Another additional way for me to debug would be if you provided the response to the "infer" request that the what-if tool webpage makes to the TensorBoard web server. You can find it in the network tab of the chrome developer console (open the developers console before clicking "accept" in the what-if tool setup dialog to ensure the "infer" network request and response are captured).

[reinhouthooft]

@jameswex I'll check out the request/response.

I updated my previous comment, I added the model+data as an attachment to it.

[jameswex]

@reinhouthooft thanks! i'm able to reproduce. at first glance i think the what-if tool can't handle the fact that the labels for the two classes in the inference results are both blank (and therefore identical). in all previous uses, i've seen the label field in the protocol buffer returned by serving to the tensorboard webserver (and then passed along to the plugin's webpage) populated with 0 for the negative class and 1 for the positive class.

i can adjust the logic to auto-fill the labels 0 and 1 when two classes come back unlabeled, but its worth investigating why the labels for the tensors of the scores for your two classes aren't being filled out correctly by tensorflow serving for your model.

[reinhouthooft]

That is indeed very strange, I'll have a look at it, thanks for the pointer!

[reinhouthooft]

@jameswex The tf classification API is specified as follows:

@tf_export('estimator.export.ClassificationOutput')
class ClassificationOutput(ExportOutput):
  """Represents the output of a classification head.

  Either classes or scores or both must be set.

  The classes `Tensor` must provide string labels, not integer class IDs.

  If only classes is set, it is interpreted as providing top-k results in
  descending order.

  If only scores is set, it is interpreted as providing a score for every class
  in order of class ID.

  If both classes and scores are set, they are interpreted as zipped, so each
  score corresponds to the class at the same index.  Clients should not depend
  on the order of the entries.
  """

  def __init__(self, scores=None, classes=None):
    """Constructor for `ClassificationOutput`.

    Args:
      scores: A float `Tensor` giving scores (sometimes but not always
          interpretable as probabilities) for each class.  May be `None`, but
          only if `classes` is set.  Interpretation varies-- see class doc.
      classes: A string `Tensor` giving predicted class labels.  May be `None`,
          but only if `scores` is set.  Interpretation varies-- see class doc.

    Raises:
      ValueError: if neither classes nor scores is set, or one of them is not a
          `Tensor` with the correct dtype.

which allows the user to specify classes=None. When this is done, tf serving will output label="" in its inference. So I assume that label="" is actually part of the API specification and should therefore be handled by the what if tool as a valid inference response.

[jameswex]

Thanks for the doc reference. I created #1471 to have the what-if tool handle empty labels by using indices for the class labels, which seems to do the right thing for your case.