SciPy for image transformations

[beraekoklu]

I'm trying to train a model which works on kaggle, but won't work on my M1 macbook pro. It says:
ImportError: Image transformations require SciPy. Install SciPy
I just built a model to identify plants and diseases, but the problem is that it isn't working when it's fitting. Compiling is fine, but it stops and gives me errors when the model tries to fit.

[beraekoklu]

This is the line:
model.fit_generator(datagen.flow(x_train,y_train,batch_size=8, shuffle=True), epochs=30, shuffle=True, steps_per_epoch=x_train.shape[0]//8, validation_data=valgen.flow(x_val,y_val,batch_size=8, shuffle=True), verbose=2)

[anna-tikhonova]

Thank you very much for reporting this issue. Could you please provide a reproducible test case, so we can investigate this issue locally?

[beraekoklu]

@anna-tikhonova I created my x and y's in kaggle, since I thought it would be easier to do so and not try to install sklearn and other things. This generates a pickle file, and you can download it in the output:
https://www.kaggle.com/mintstudios/project-plantus-pickle-generator
The rest of my project is this:

import numpy as np
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.regularizers import l2
from tensorflow.keras.layers import Conv2D, Dropout, Dense, Flatten, BatchNormalization, AveragePooling2D
import pickle
import gc

data = []

data = pickle.load(open("pickle.dat", "rb"))

x_train = data[0]
x_val = data[1]
y_train = data[2]
y_val = data[3]


print('datagen')
datagen = ImageDataGenerator(
    rescale=1.0/255.0,
    zoom_range = 0.1,
    shear_range=0.1,
    fill_mode = "reflect",
    vertical_flip=True,
    width_shift_range = 0.1,
    height_shift_range = 0.1,
)

print('datagen_fit')
#datagen.fit(x_train) <-- # Commented out for some purposes
valgen = ImageDataGenerator(
    rescale=1.0/255.0
)

gc.collect()

print('model')
model = Sequential()
model.add(Conv2D(6, kernel_size=(5, 5), strides=(1, 1), activation='relu', input_shape=(150, 150, 3), padding='same'))
model.add(AveragePooling2D(pool_size=(2, 2), strides=(1, 1), padding='valid'))
model.add(BatchNormalization())
model.add(Dropout(0.27))
model.add(Conv2D(16, kernel_size=(5, 5), strides=(1, 1), activation='relu', padding='valid'))
model.add(AveragePooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))
model.add(BatchNormalization())
model.add(Dropout(0.27))
model.add(Conv2D(120, kernel_size=(5, 5), strides=(1, 1), activation='relu', padding='valid'))
model.add(Flatten())
model.add(Dense(84, activation='relu'))
model.add(Dense(38, activation='softmax'))

print('Model compile')
model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, nesterov=True), metrics=['accuracy'])

print('Model fit')
model.fit_generator(datagen.flow(x_train,y_train,batch_size=8, shuffle=True), epochs=30, shuffle=True, steps_per_epoch=x_train.shape[0]//8, validation_data=valgen.flow(x_val,y_val,batch_size=8, shuffle=True), verbose=2)

I'm new to ML and deep learning, so this is not a great model.

The error again is at fit_generator and it says: ImportError: Image transformations require SciPy. Install SciPy
I've also tried fit instead of the generator.

Note that I tried to download SciPy through brew, which was under the Rosetta 2 emulation, and it worked. But the thing is, tensorflow wasn't native, so that would be useless. And also, I tried installing SciPy through native brew, which threw me a bunch of errors.

[beraekoklu]

The README.md says that scipy isn't available, so I was wondering if I could avoid doing these things. (Image transformations)

[iRonJ]

I was able to compile scipy from source for python 3.8 to run native. I haven't tested it thoroughly but I confirmed it did load correctly.

[iRonJ]

Followup, I copied and pasted the code above, and it seems kinda slow but it does run:

datagen
datagen_fit
model
Model compile
Model fit
/Users/ronj/Documents/tfmac/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1844: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.
  warnings.warn('`Model.fit_generator` is deprecated and '
2021-01-02 23:43:23.364967: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-01-02 23:43:23.365621: W tensorflow/core/platform/profile_utils/cpu_utils.cc:126] Failed to get CPU frequency: 0 Hz
Epoch 1/30
2202/2202 - 332s - loss: nan - accuracy: 0.0248 - val_loss: nan - val_accuracy: 0.0256
Epoch 2/30```

[beraekoklu]

Yeah, how did you get it to run? For some reason it is slow, I wonder if it's just the model or ML isn't for macs.

[MaEvGoR]

Same problem

ImportError: Image transformations require SciPy. Install SciPy.

model.fit( train_generator, steps_per_epoch = train_generator.samples // 32, validation_data = validation_generator, validation_steps = validation_generator.samples // 32, epochs = 10)

[YarrDOpanas]

Have the same problem
ImportError: Image transformations require SciPy. Install SciPy.

[YelleV]

I'm running into the same problem with a code I wrote... Did anyone find a solution yet?

[rrrrli]

same problem!

[rrrrli]

But for mine, the model.fit would work in my other scripts. But not on this one script where I got data from kaggle. does it have anything to do with kaggle?

[alocin98]

Same problem here. I installed scipy via pip but didn't work. I think the problem lies inside ImageDataGenerator

[Durobert]

Same error!!!how to solve it?

[mmccarthy1]

I am having the same issue after using cx_freeze

[lohant]

Same error here. The first time I'm training a model though so there is definitely a possibility that I did something wrong.

[Cdaniel089]

Hi, well I could fix it. Normally I opened Visual Studio to run a python script since my desktop, but when I open Visual Studio since 'search' from windows this problem about 'scipy' does not exist.