人臉識別具有廣泛的潛在應用，如信息安全，監控系統和身份管理。然而有幾種變化增加了可見光人臉識別的難度，包括照明、遮擋、面部表情和環境條件。隨著近年來感光元件(Charge-coupled Device)的發展，可見光影像不再是各種領域中唯一使用的影像。熱紅外線(Thermal infrared)相機傳感器能捕獲生物發射的熱能，而不捕獲反射的能量，因此在夜間也不需要提供照明。由於生物與周圍環境之間的溫差相對地大，使用熱成像也容易地將物體與背景區分開，熱成像人臉識別也受到越來越多的關注。然而人臉熱成像可能會受到諸如成像條件（距離，眼鏡），新陳代謝（生病，運動）等影響，因此，熱成像人臉識別的實用性還有許多挑戰。
我們提出了一種利用模型融合進行熱成像人臉識別的方法。所提出的模型融合有三種主要方法：線性支持向量機分類器(linear SVC)，卷積神經網絡(CNN)和普通最小二乘(OLS)。線性支持向量機分類器方法的訓練階段包括預處理，特徵擷取和分類。在開始時，可以使用貝氏框架(Bayesian framework)從熱成像的背景描繪人臉並將其標準化為均勻尺寸。擷取22個網格中的點作為特徵向量，這22個擷取點用於訓練線性支持向量機分類器，分類器計算支持向量(support vector)並使用它們來查找用於分類的超平面(hyperplane)，將測試圖像的特徵向量輸入到用於人臉識別的分類器。新穎之處在於我們首先在人臉辨識中應用溫度信息。在卷積神經網絡模型中，卷積層(Convolutional layer)用於擷取特徵，這將在輸出圖像中生成特徵圖，並將特徵圖提供給下一層，最後一個匯集層的輸出被壓平(Flatten)並送入全連接層(Fully Connected layer)。全連接層的目標是使用這些基於訓練數據集的特徵將輸入圖像分類。在訓練階段，將來自上述兩種方法的預測值提供給普通最小二乘以進行線性回歸。普通最小二乘為這兩種方法分配加權值，這可以有效地補償兩種方法的優缺點。除了與傳統熱成像人臉識別比較之外，還進行了偽裝條件下的實驗，經由實驗可得知提出的方法具有優異的辨識率。

Face recognition technology has a wide range of potential applications such as information security, surveillance systems, and identity management. There are several variations that add to the difficulty of visible face recognition, including illumination, occlusion, facial expressions, and environmental conditions. With the development of Charge-coupled Device (CCD) in recent years, visible light images are no longer the only images used in different fields. A thermal infrared camera sensors capture the emitted thermal energy from the living creature, without capturing the reflected energy. Since the temperature difference is quite significant between the living being and the surrounding environment, thermal imaging is easily used to distinguish objects from the background. Thermal face recognition is receiving more and more commercial attention. However, thermal face image may get affected such as imaging conditions (distance, glasses), metabolism (sick, sport). Therefore, there are still many problems to be overcome in the practicality of thermal face recognition.
A thermal face recognition based on physiological information using model fusion is proposed. The proposed model fusion has three main approaches: Linear Support Vector Machine Classifier (Linear SVC), Convolutional Neural Network (CNN) and Ordinary Least Square (OLS). The training phase of linear SVC approach includes preprocessing, feature extraction and classification.
In the beginning, the human face can be depicted from the background of thermal image using the Bayesian framework and normalized to uniform size. A grid of 22 thermal points is extracted as a feature vector. These 22 extracted points are used to train linear SVC. The classifier calculates the support vectors and uses them to find the hyperplane for classification. A feature vector of testing image is inputted to the classifier for face recognition. The novelty is that we firstly apply temperature information in face recognition. In CNN model, the convolutional layer is utilized to extract features. This produces a feature map in the output image and the feature maps are fed to the next layer. The output from the last pooling layer is flattened and fed into a fully connected layer (FC layer). The goal of FC layer is to employ these features for classifying the input image into various classes based on the training datasets.
In the training phase, the predicted values from above two approaches are provided to the OLS for linear regression. The OLS assigns weighting values to these two approaches. This can effectively compensate for the advantages and disadvantages of two approaches. In addition to the comparison with the traditional thermal face recognition, an experiment under disguised conditions was conducted. Experimental results of the proposed method outperform the existing methods.

Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Thesis Organization 3
Chapter 2 Background 5
2.1 Thermal Imaging 5
2.2 Feature Extraction Regions 6
2.3 Convolutional Neural Networks (CNN) 7
Chapter 3 Related Works 9
3.1 Face Recognition and Drunk Classification Using Infrared Face Images 9
3.2 Very Deep Convolutional Networks for Large-Scale Image Recognition 11
3.3 Thermal Face Recognition Under Temporal Variation Conditions 13
Chapter 4 The Proposed Thermal Face Recognition Based on Physiological Information 15
4.1 SVM (Linear SVC)-based Face Recognition 16
4.1.1 Feature Extraction 16
4.1.2 Classification 17
4.2 CNN-based Face Recognition 19
4.2.1 CNN Architecture 19
4.3 Model Fusion (OLS)-based Face Recognition 22
Chapter 5 Experimental Results 23
5.1 Database 23
5.1.1 PUCV Thermal Temporal Face Database 24
5.1.2 UCH Thermal Temporal Face Database 24
5.2 Estimation of Parameters 25
5.2.1 Comparing different number of parameters in Linear SVC approach 26
5.2.2 Comparing different structure in CNN approach 29
5.2.3 Comparing different algorithm in model fusion approach 33
5.3 Comparison with Other Methods 36
5.4 Other Testing Results 37
Chapter 6 Conclusions 41
References 43
Appendix 45

[1]I. Pavlidis and P. Symosek, “The Imaging Issue in an Automatic Face/Disguise Detection System,” Proc. IEEE Workshop Computer Vision Beyond the Visible Spectrum: Methods and Applications, pp. 15-24, June 2000.
[2]S. Zhao and R.R. Grigat, “An Automatic Face Recognition System in the Near Infrared Spectrum,” Proc. Fourth Int’l Conf. Machine Learning and Data Mining in Pattern Recognition, pp. 437-444, July 2005.
[3]X. Zou, J. Kittler, and K. Messer, “Face Recognition Using Active Near-IR Illumination,” Proc. 2005 British Machine Vision Conf., Sept. 2005.
[4]S. Iwasawa, K. Ebihara, J. Ohya, and S. Morishima, “Real-time estimation of human body posture from monocular thermal images,” Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 15-20, 1997.
[5]M. Z. Uddin and J. Torresen, “A Deep Learning-Based Human Activity Recognition in Darkness,” 2018 CVCS, pp. 1-5, 19-20 Sept., 2018.
[6]P. Saponaro, S. Sorensen, A. Kolagunda, and C. Kambhamettu, “Material classification with thermal imagery,” 2015 CVPR, pp. 4649-4656, 7-12 June, 2015.
[7]P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos, “Physiology-based face recognition in the thermal infrared spectrum,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 4, pp. 613-626, Apr. 2007.
[8]A. R. Butt, A. Baig, and S. Ahmed, “Comparison of thermal and visual facial imagery for use in sparse representation based facial recognition system,” ICET, pp. 1-4, 9-10 Dec. 2013.
[9]M. Kanti Bhowmik, B. K. De, D. Bhattacharjee, D. K. Basu, and M. Nasipuri, “Multisensor fusion of visual and thermal images for human face identification using different SVM kernels,” IEEE Long Island Systems, Applications and Technology Conference (LISAT), pp. 1-7, 4 May 2012.
[10]Z. Wu, M. Peng, and T. Chen, “Thermal face recognition using convolutional neural network,” ICOIP, pp. 1-4, 10-12 June 2016.
[11]A. Seal, S. Ganguly, D. Bhattacharjee, M. Nasipuri, D. Kr. Basu, “Automated Thermal Face recognition based on Minutiae Extraction,” International Journal of Computational Intelligence Studies 2.2, pp 133-156, Sep. 2013.
[12]G. Hermosilla, J. L. Verdugo, G. Farias, E. Vera, F. Pizarro, and M. Machuca, “Face Recognition and Drunk Classification Using Infrared Face Images,” Journal of Sensors, vol. 2018, pp. 1-8, Article ID 5813514, Jan. 2018.
[13]J. Rustemeyer, J. Radtke and A. Bremerich, “Thermography and thermoregulation of the face,” Head Face Med., vol. 3, pp. 17, Mar. 2007.
[14]M. O. Simon et al, “Improved RGB-D-T based face recognition,” IET Biometrics, vol. 5, no. 4, pp. 297-303, Dec. 2016.
[15]FLIR ONE Pro-Series Comparison Flyer, 2018, https://www.flir.com/globalassets/imported-assets/document/flir-one-pro-series-comparison-flyer.pdf
[16]G. Koukiou and V. Anastassopoulos, “Drunk person identification using thermal infrared images,” International Journal of Electronic Security and Digital Forensics, vol. 4, no. 4, pp. 229-243, 2012.
[17]K. Simonyan, A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” International Conference on Learning Representations, 10 Apr., 2015.
[18]G. H. Vigneau, J. L. Verdugo, G. F. Castro, F. Pizarro, and E. Vera, “Thermal Face Recognition Under Temporal Variation Conditions,” in IEEE Access, vol. 5, pp. 9663-9672, 2017. doi: 10.1109/ACCESS.2017.2704296