How to use the pretrained JSTL+DGD model for person re-identification?

[benstaf]

I don't understand how to do person re-identification with the pretrained JSTL+DGD model found here: https://drive.google.com/open?id=0B67_d0rLRTQYZnB5ZUZpdTlxM0k

I have two problems, one related to input, one related to output :

1. In person re-identification, we input two different pictures, and we ask the model if they depict the same person or not.

But here, in the file 'jstl_dgd_deploy_inference.prototxt', the input data is (1,3,144,56) and not, for example, (2,3,144,56).

2. In the file 'jstl_dgd_deploy_inference.prototxt', I don't see the output layer, it should be a binary softmax, with output '1' if the two photos represent the same person, and '0' if the persons are different.

Moreover, when loading the caffemodel weights, I receive the warnings:

I1108 08:48:31.324759 1525 net.cpp:752] Ignoring source layer relu7
I1108 08:48:31.324795 1525 net.cpp:752] Ignoring source layer drop7
I1108 08:48:31.324802 1525 net.cpp:752] Ignoring source layer fc8_jstl

This suggests that something is missing in the prototxt file.

[Cysu]

Our model does not directly produces the binary verification result of a pair of people. During test stage, we first go through all the images, and extract their features using our net. Then we compute the pairwise Euclidean distances between query and gallery people. At last, for each query, we just rank the gallery samples based on their distances.

If you just wish to do the verification, you can choose a distance threshold that balances the true positive rate and false positive rate.

[benstaf]

I tried to follow your suggestions, but my result is not convincing. I made some experiments with the PRID dataset.

In the multi shot case, I choose 2 pictures of persons 4 and 9, taken with cameras A and B (8 pictures in total).

We should get a large distance between pictures of different persons, and a small distance between pictures of the same person, but this is not the case. Why?

Some results are here (for example, a4_1.png is picture number 1 of person 4 by camera A):

distance between a4_1.png and a9_1.png: 6.65493
distance between a4_1.png and a4_34.png: 6.5565
distance between a4_1.png and a9_28.png: 4.84618
distance between a4_1.png and b4_1.png: 7.06474
distance between a4_1.png and b9_1.png: 8.09637
distance between a4_1.png and b4_34.png: 5.71222
distance between a4_1.png and b9_28.png: 5.91796
distance between b9_1.png and a4_34.png: 9.21853
distance between b9_1.png and a9_28.png: 7.02944
distance between b9_1.png and b4_1.png: 4.23969
distance between b9_1.png and b9_28.png: 5.4921

Cropped images are here (cropped to shape (56,144) for input in the neural network):
https://drive.google.com/drive/folders/0B86WKpvkt66BeVp4UGgxUlhzZG8?usp=sharing

Code (additional code here:
https://drive.google.com/drive/folders/0B86WKpvkt66BcDhxUW14bUsxd1k?usp=sharing
)

`from jstl_inference import JSTL # jstl_inference.py is the TensorFlow version of the file jstl_dgd_deploy_inference.prototxt , made with Caffe-Tensorflow #https://github.com/ethereon/caffe-tensorflow
#see code here: https://drive.google.com/drive/folders/0B86WKpvkt66BcDhxUW14bUsxd1k?usp=sharing

import tensorflow as tf
from scipy.misc import imread
import numpy as np
from PIL import Image, ImageOps

#Preparation of the feature extractor

x = tf.placeholder(tf.float32, shape=[1, 144, 56, 3])
y = tf.placeholder(tf.float32, shape=[1, 256])

net = JSTL({'data': x})
sess = tf.InteractiveSession()
sess.run(tf.initialize_all_variables())
net.load('jstl_inference.npy', sess) #jstl_inference.npy is the Numpy version of jstl_dgd_inference.caffemodel , obtained with Caffe-Tensorflow https://github.com/ethereon/caffe-tensorflow
#file here: https://drive.google.com/drive/folders/0B86WKpvkt66BcDhxUW14bUsxd1k?usp=sharing

person_feature = sess.graph.get_tensor_by_name("fc7/fc7:0") #gets the output from the layer FC7

def extract_vector(image_data):
img = imread(image_data)

img=Image.fromarray(img)
img=ImageOps.fit(img, size=(56,144), method=Image.ANTIALIAS) # resize (by maintaining the aspect ratio) and crops the input image


img=np.asarray(img)
img = np.reshape(img, (1, 144, 56, 3))


feed= {x: img}
person_vector = sess.run(person_feature, feed_dict=feed)
return person_vector[0]

def distance_pics(photo1,photo2):
person1=extract_vector(photo1)
person2=extract_vector(photo2)
dist = np.linalg.norm(person1-person2)
print( 'distance between ' + photo1 + ' and ' + photo2 + ': '+ str(dist))

#Results

distance_pics('a4_1.png','a9_1.png')
distance_pics('a4_1.png','a4_34.png')
distance_pics('a4_1.png','a9_28.png')

distance_pics('a4_1.png','b4_1.png')
distance_pics('a4_1.png','b9_1.png')
distance_pics('a4_1.png','b4_34.png')
distance_pics('a4_1.png','b9_28.png')

distance_pics('b9_1.png','a4_34.png')
distance_pics('b9_1.png','a9_28.png')
distance_pics('b9_1.png','b4_1.png')
distance_pics('b9_1.png','b9_28.png')`

[Cysu]

I guess there might be some mismatch between the image preprocessing methods we used.

When training the model, we use opencv to read the image, and subtract the mean pixel values. The input data to the CNN should be a 1x3x144x56 image, whose color channels are in BGR order, and are demeaned by [102, 102, 101].

Thanks for providing the script. I will verify this after the cvpr deadline.

[benstaf]

I revised my image pre-processing, but the result does not improve. My result is:

distance between a4_1.png and a9_1.png: 6.59645
distance between a4_1.png and a4_34.png: 7.80466
distance between a4_1.png and a9_28.png: 6.67408
distance between a4_1.png and b4_1.png: 11.086
distance between a4_1.png and b9_1.png: 10.6859
distance between a4_1.png and b4_34.png: 12.731
distance between a4_1.png and b9_28.png: 13.6327
distance between b9_1.png and a4_34.png: 9.13998
distance between b9_1.png and a9_28.png: 12.1658
distance between b9_1.png and b4_1.png: 5.44103
distance between b9_1.png and b9_28.png: 7.77282

My code is:

`from jstl_inference import JSTL # the output python script of caffe2tensorflow
import tensorflow as tf

import numpy as np

import cv2

x = tf.placeholder(tf.float32, shape=[1,144, 56, 3])
y = tf.placeholder(tf.float32, shape=[1, 256])

net = JSTL({'data': x})
sess = tf.InteractiveSession()
sess.run(tf.initialize_all_variables())
net.load('jstl_inference.npy', sess)

person_feature = sess.graph.get_tensor_by_name("fc7/fc7:0")

def preprocess(image):
img=cv2.imread(image)
shape=img.shape
ratio=float(144)/float(shape[0])
dim=(int(shape[1]*ratio), 144)
resized = cv2.resize(img, dim, interpolation = cv2.INTER_AREA)

#Crop on both sides
margin=dim[0]-56
if margin % 2==0:
cropped=resized[:,margin/2:dim[0]-margin/2]
else:
cropped=resized[:,margin/2:dim[0]+1-margin/2]

cv2.imwrite('cropped_' + image, cropped)`

`# subtract the mean pixel values
centered_array=cropped-np.array([102,102,101]) #demean by [102, 102, 101].

return centered_array

def extract_vector(image):
centered_array=preprocess(image)

input_array = np.reshape(centered_array, (1,144, 56,3))

feed= {x: input_array}
person_vector = sess.run(person_feature, feed_dict=feed)
return person_vector[0]

def distance_pics(photo1,photo2):
person1=extract_vector(photo1)
person2=extract_vector(photo2)
dist = np.linalg.norm(person1-person2)
print( 'distance between ' + photo1 + ' and ' + photo2 + ': '+ str(dist))

#Results:

distance_pics('a4_1.png','a9_1.png')
distance_pics('a4_1.png','a4_34.png')
distance_pics('a4_1.png','a9_28.png')

distance_pics('a4_1.png','b4_1.png')
distance_pics('a4_1.png','b9_1.png')
distance_pics('a4_1.png','b4_34.png')
distance_pics('a4_1.png','b9_28.png')

distance_pics('b9_1.png','a4_34.png')
distance_pics('b9_1.png','a9_28.png')
distance_pics('b9_1.png','b4_1.png')
distance_pics('b9_1.png','b9_28.png')`

[kaidic]

I've encountered the same problem. It seems that the feature layer outputs I got from using tensorflow and caffe are different.

[soulslicer]

Which prototxt in the code was used to train model for jstl_dgd_inference.caffemodel? I can't seem to find it