Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error in my results #5

Open
fatemehkalantari1993 opened this issue Jul 27, 2023 · 5 comments
Open

error in my results #5

fatemehkalantari1993 opened this issue Jul 27, 2023 · 5 comments

Comments

@fatemehkalantari1993
Copy link

thanks a lot for your help.
as you know, I am working in the image reconstruction field.

first step:
I extracted image features with MatConvNet that didn't work well in reconstruction. then, I extracted Caffe image features and then I extracted brain features with fastl2lir and finally, I did reconstruction and my results were very close to your result.

second step:
I update my Ubuntu to the Ubuntu 22 version and I couldn't install Caffe on it. Therefore, I used Torch for image feature extraction. Then, I used Torch image features in Fasl2lir and extract brain features. recently. you uploaded new codes that did reconstruction with Torch. I ran this code with my torch features, but I received big errors. when I ran this code with your data, I received big errors, too. I report this problem to you and you modified your code and errors decreased with your features.

But, when I run this code with my features that extract in the same way (fastl2lir) I still get big errors. The only difference is that I extract image features with Torch. I am attached my code. Please look at them.

I need to use Torch because I want to use resnet and I don't have a strong system in Iran to train this network in Caffe and get resnet.caffemodel. please help me.

for example in this code I extract conv5_1 image features in Torch:

import numpy as np
from google.colab.patches import cv2_imshow
import pandas as pd
import cv2
import os
import torch
from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
from torch import optim, nn
from torchvision import models, transforms
import torchvision.io as io
import torchvision.transforms as transforms
from PIL import Image
import imghdr
from bdpy.dataform import Features, load_array, save_array

Load the VGG19 model

vgg19 = models.vgg19(pretrained=True)

Access the features module

features = vgg19.features

Define a new model that outputs features from the conv3_1 layer

#model = nn.Sequential(features[0:12])

Set the model to evaluation mode

vgg19.eval()

import numpy as np
from google.colab.patches import cv2_imshow
import pandas as pd
import cv2
import os
import torch
from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
from torch import optim, nn
from torchvision import models, transforms
import torchvision.io as io
import torchvision.transforms as transforms
from PIL import Image
import imghdr
from bdpy.dataform import Features, load_array, save_array

Load the VGG19 model

vgg19 = models.vgg19(pretrained=True)

Access the features module

features = vgg19.features

Define a new model that outputs features from the conv3_1 layer

#model = nn.Sequential(features[10])

conv3_1_features = nn.Sequential(*list(vgg19.features.children())[:29])

Set the model to evaluation mode

conv3_1_features.eval()

Load an image

data_dir='/content/drive/MyDrive/image_feature_python/resultes/pytorch_image_feat_training/pytorch/VGG19/conv5_1/'
img_dir='/content/drive/MyDrive/matconvnet/data/training'

Get image files

imagefiles = []
for root, dirs, files in os.walk(img_dir):
imagefiles = [os.path.join(root, f)
for f in files
if imghdr.what(os.path.join(root, f))]

imgs_path = np.array(imagefiles)

#image name
images_name=[]
images=imagefiles
for f in files:
name=f[:-5]
images_name.append(name)
print ('Image num: %d' % len(imagefiles))

for n in range(imgs_path.shape[0]):
print(n)
img = io.read_image(imgs_path[n])

# Preprocess the image for the VGG16 model

# Load an image using PIL
pil_image = Image.open(imgs_path[n]).convert('RGB')

preprocess = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
                     std=[0.229, 0.224, 0.225])
])

#input_tensor = preprocess(pil_image)
#input_batch = input_tensor.unsqueeze(0)

input_tensor = preprocess(pil_image).unsqueeze(0)
# Pass the input tensor through the model up to the conv3_1 layer
features = conv3_1_features(input_tensor)

# Convert to NumPy Array
features_numpy = features.detach().numpy()

features = np.array(features_numpy)
print(features.shape)
feat = np.reshape(features,(1,512,14,14))


savefile = os.path.join(data_dir, '%s.mat' % images_name[n])
 #Save
save_array(savefile, feat, key='feat', dtype=np.float32, sparse=False)
@ShuntaroAoki
Copy link
Member

Thank you. Let me confirm two points.

  1. Are you using the same pretrained model for the image feature extraction and the encoder in the reconstruction script? You extracted features with torchvision's pretrained VGG19 in your code. However, the weights of the VGG19 model used as the encoder in the reconstruction script we provided are different from torchvision's VGG19, and they are not compatible. If the model used to calculate image features and the one used as the encoder in the reconstruction script are not the same, the reconstruction will not work correctly.
  2. When calculating image features using torchvision's VGG-19, please make sure that you are not accidentally saving the ReLU layer's features. In torchvision's VGG-19, ReLU is processed inplace, meaning you might mistakenly extract ReLU's features instead of the features from convolutions or fully connected layers.

@fatemehkalantari1993
Copy link
Author

Thank you.
I have a question about these features that you used. did you extract image features with Caffe and brain features with Fasl2lir???

@ShuntaroAoki
Copy link
Member

did you extract image features with Caffe and brain features with Fasl2lir???

Yes, the shared image and decoded features were extracted using Caffe and decoded using FastL2LiR. Furthermore, the weights of VGG-19 used in our PyTorch reconstruction code were actually imported from Caffe, making them compatible with the image features extracted using the Caffe model.

@fatemehkalantari1993
Copy link
Author

Thank you very much. I ran your code with your data with the generator and without the generator and the resulting figures were the same as the paper. But errors in with generator code were bigger than without the generator. I think this problem is due to calculate errors in the construction function.

for n02190790_15121.mat

with generator results:
iteration = 193; err = 61842.234375
iteration = 194; err = 61800.95703125
iteration = 195; err = 61347.01171875
iteration = 196; err = 61385.69921875
iteration = 197; err = 61229.046875
iteration = 198; err = 61087.90625
iteration = 199; err = 60957.53515625
iteration = 200; err = 60826.23046875

without generator results and function:
recon_image, loss_list = reconstruct(feat,
encoder,
generator=None,
layer_mapping=layer_mapping,
optimizer=optim.SGD,
image_size=encoder_input_shape,
crop_generator_output=True,
preproc=image_preprocess,
postproc=image_deprocess,
output_dir=save_dir,
save_snapshot=True,
snapshot_dir=snapshots_dir,
snapshot_interval=10,
snapshot_ext='jpg',
snapshot_postprocess=normalize_image,
return_loss=True,
**opts)

iteration = 195; err = 47284.10546875
iteration = 196; err = 47321.4609375
iteration = 197; err = 47359.74609375
iteration = 198; err = 47416.3671875
iteration = 199; err = 47473.265625
iteration = 200; err = 47544.5234375

@fatemehkalantari1993
Copy link
Author

fatemehkalantari1993 commented Jul 31, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ShuntaroAoki @fatemehkalantari1993