- python3
- pytorch>=1.0
- pillow>=5.0
- dlib ver.19.17 only support python3.5 with shape_predictor_68_face_landmarks.dat_百度云盘提取码：4qjg or shape_predictor_68_face_landmarks.dat_Google drive
- scikit-image>=0.14
- models_百度云盘提取码：u46u or models_Google drive
Download and unzip models
$ python target_iteration.pyIf you only add noise to the face area, you need to leverage dlib to crop the face, which will be elaborated later.
To address the black-box face attack challenge, we integrate the common DNN model structure[1], including IR50, IR101, IR152 (model depth is different). The code for model construction is in model_irse.py. Considering that the online evaluation system may determine the category of the image by similarity, we employ the target attack. Cal_likehood.py calculates the similarity between the faces through multi-model ensembling. We select the second similar image as the attack target. At the same time, our loss function is made up of three components, the classic distance loss such as L2, cos loss. TV loss is to maintain the smoothness of the image, which will be elaborated later. The resulting noise will be convolved by gauss kernel and finally superimposed on the original image. The above process is iterated until the current picture is terminated with its own matrix similarity of more than 0.25.
In addition, our model still adopts multi-process multi-graphics acceleration. We utilize two GTX 1080Ti, and it takes less than one hour to generate 712 samples.
In the process of noise cancelling, the artificial noises may have a very enormous visual impact on the result images. At this time, we need to add some regularizaiton to the optimization problem to restrain the image smooth. TV loss is A commonly used regularizaiton in the computer vision. The integration of the continuous domain becomes the summation in the discrete region of the pixel. The specific calculation process is as follows:
Gaussian filtering combines image frequency domain processing with time domain processing under the image processing concept. As a low-pass filter, it can filter low-frequency energy (such as noise) to smooth the image.
Gaussian filtering is performed on the generated interference noise, so that the generated noise of each pixel has correlation with surrounding pixels, which reduces the difference between the interference noise generated by different models (because different models have similar classification boundaries), effectively improving fight against the success rate of sample attacks. At the same time, considering that the online test may have a defense mechanism such as Gaussian filtering, adding Gaussian filtering when the algorithm generates noise can also invalidate the defense mechanism to improve the sample attack rate. This can be done by convolution using a Gaussian kernel function. The Gaussian kernel is as follows:
The existing neural network model largely rely on critical regions(eyes, noses) to distingush from human faces. In the Face Attention Maps Visualization.ipynb code, we try to generate an attention map on the image, thus find colored face region is more prominent in face classification task.
Therefore, we restrict the adversarial noises on significant facial areas. In the implementation, we use dlib[4] to calibrate the 68 landmarks of the face, select 17 points to form a non-mask area, and finally we will save the generated image as attentional masks mask1. For a few pictures that cannot be used to calibrate the mapmark with dlib , we manually frame the face range.
The order of selecting 17 face landmarks is (48, 59-54, 26-17), reference code crop_image.py In the experiment, it took about 10 minutes to generate 712 non-mask areas using dlib.
Of course, we can also convert the attention map into a mask of [0,1] to complete the matrix multiplication of noise.
Integrating the momentum into the iterative process of the attack stabilizes the update direction and leaves the poor local maximum during the iteration, resulting in adversarial samples with strong generalization ability. In order to elevated the success rate of black box attacks, we integrate the momentum iteration algorithm into our pipeline. Experiments show that the black box attack is better after adding the momentum term. The formula for the calculation is as follows:
When training the lfw dataset, in addition to directly cropping the face portion of 112*112, we also employ a random padding similar to data augmentation, random resizing operation, to promote the diversity of the input mode. The algorithm computation process is as follows:
