Face Recognition Systems: Are you sure they only consider your face?

Face recognition has been one of the major biometric authentication procedures in smart devices that allows users to provide an additional layer of security for accessing their device. The accuracy of image similarity should depend on the face and its expression, as could be extracted from the whole image. Importantly, the background may have a substantial amount of additional information that can potentially pose a threat to the privacy of the user. In this paper, we report the impact of background on the recommended measure of similarity, Euclidean-L2, across different pictures that represent distinguishable emotions and image background. Additionally, we report that this impact of the background varies for different ethnic groups. Our findings are despite the fact that background should not matter for Face Recognition. For each facial image, we perform two preprocessings, gray-scaling and background whitening, and compute the similarity between the original image and the preprocessed image by using the DeepFace Face Recognition System. We have considered six data sets, i) containing 100 blurry images of one American man, ii) and iii) contained 100 images each of one American man in normal settings, iv) contained 50 each of East Asian men and women, v) contained 50 each of Indian men and women, and vi) contained 50 each of African or African-American men and women. We observe that gray scaling or background whitening images makes them dissimilar, often to the point of being unrecognisable. Overall, we report that the information contained in the background of a facial image can be significant and it can have different impacts across different skin complexions and facial structure. Importantly, our initial results bring up an important question of how to identify the images having a higher risk of exposing private information via the background of a facial image.