Food vision milestone - scikit-learn accuracy doesn't match evaluate() accuracy

[donalhill]

Hi there,

I have been working on the food vision milestone project. Having trained my full Food101 model with fine-tuning and early stopping, I ran:

model.evaluate(test_data)

and I get an accuracy score close to DeepFood. When I try to evaluate the accuracy using scikit-learn instead, by doing:

pred_probs = model.predict(test_data, verbose=1)
pred_classes = pred_probs.argmax(axis=1)

y_labels = []
for images, labels in test_data.unbatch(): # unbatch the test data and get images and labels
  y_labels.append(labels.numpy().argmax())

sklearn_accuracy = accuracy_score(y_labels, pred_classes)

the accuracy score comes out at around 1% (so basically a random guess). It's as if the true labels and predictions aren't matching, but I don't think there should be any shuffling that differs between the two since I am using test_data when calculating both pred_classes and y_labels.

I have tried setting shuffle_files=False when loading the data with tfds.load. And I do the following:

# Map preprocessing function to training data (and paralellize)
train_data = train_data.map(map_func=preprocess_img, num_parallel_calls=tf.data.AUTOTUNE)
# Shuffle train_data and turn it into batches and prefetch it (load it faster)
train_data = train_data.shuffle(buffer_size=1000).batch(batch_size=32).prefetch(buffer_size=tf.data.AUTOTUNE)

# Map prepreprocessing function to test data
test_data = test_data.map(preprocess_img, num_parallel_calls=tf.data.AUTOTUNE)
# Turn test data into batches (don't need to shuffle)
test_data = test_data.batch(32).prefetch(tf.data.AUTOTUNE)

where there is no shuffling applied to the test data. Any help to understand this would be great!

Cheers,
Donal

[mhd-danish]

Same Issue!

I tried to use:

read_config = tfds.ReadConfig(shuffle_seed=42)
tfds.load(dataset, read_config=read_config)

but no positive result.

Also, I didn't shuffle method in the test_data

[HemachandranD]

@donalhill Since, the labels are not one-hot encoded(0-100) in the datasets. There is no point in finding the argmax() while getting the y_labels.

y_labels = [] for images, labels in test_data.unbatch(): y_labels.append(labels.numpy())

Try this, also make sure you don't shuffle_files=False while loading from TFDS. If that is true it will affect the order.

[juan-deutsch]

I did it this way and still my accuracy is still 1% while when evaluating the model its close to 78%. I don't know what to do.

[astroboy07]

Hi,

I was facing the same issue. I would suggest you to check the line where you are importing the TensorFlow datasets using tfds.load(). Just changing the argument shuffle_files to False worked for me.

Hope this works out for you.

[varmatilak22]

I was also same when using notebook 7 template notebook there was model.evaluate the accuray of 68% and and after unbatching test data and predicting the classes the accuracy score was just 0.005. It was huge difference spend many time to solving this issue.
Pls help