人臉辨識是圖型識別和電腦視覺領域重要的研究議題並具有廣泛應用，如信息安全，身份管理和監控系統。在現實環境中有幾種變化增加了可見光人臉識別的難度，包括遮擋、照明、和環境條件。近年來隨著感光元件(Charge-coupled Device)的高速發展，可見光影像不再是各種領域中唯一使用的影像。熱紅外線(Thermal infrared)相機傳感器能捕獲生物發射的熱能，而不捕獲反射的能量，因此在夜間無須提供照明，熱紅外線感應器應用於人臉辨識就變得深具吸引力。然而熱成像人臉辨識可能會受到諸如新陳代謝，成像條件等影響，其實用性還有許多挑戰。

本文擬開發一個熱成像溫度資訊應用於臉部辨識和酒醉識別系統，目標是在熱成像人臉一般情況、戴眼鏡、戴口罩、雜訊干擾等情境時仍有良好的辨識性。所提出的系統有兩個階段，第一階段是人臉辨識，第二階段是酒醉識別。首先通過強健的人臉辨識方法來識別對象身分，在酒醉識別階段，透過人臉靜脈血管和表面血液中傳遞的熱量，提取明顯的數個熱點的網格(Grid)作為特徵向量，這些特徵向量用於訓練傳統機器學習模型(Radom Forest，RF 或 Support Vector Machine，SVM)並判斷是否為酒醉人臉。我們使用PUCV-DTF數據庫測試卷積神經網路(Convolutional neural network，CNN)與傳統機器學習方法(RF和SVM)，實驗結果顯示，三個酒醉辨識模型都具有優異的識別能力。此外，我們透過圖型增強演算法預處理，改善低解析度影像問題，使識別模型可以對特徵做更準確的分群，辨識率也會相對提升，與現有方法比較，實驗結果證明提出的方法優於現有方法。

Face recognition is an important research topic belonging to the fields of pattern recognition and computer vision. The technology has a wide range of potential applications such as information security, identity management, and surveillance systems. In a real operational scenario, there are several variations that add to the difficulty of visible face recognition, including occlusion, illumination, and environmental conditions. With the rapid development of Charge-coupled Device in recent years, visible light image has no longer been the only images used in various fields. Thermal infrared camera sensors capture the thermal energy emitted by living things, without capturing the reflected energy, then do not require illumination at night. Therefore, thermal face recognition has been received more and more commercial attention. However, thermal face images may be affected such as metabolism (sport, sick), imaging conditions (distance, face mask, and glasses). In the practicality of thermal face recognition, there are still many problems to overcome.

This thesis proposes an infrared-based face recognition and drunk classification system using temperature information. The goal is to achieve a good recognition rate under different conditions, including normal, glasses, face mask, and noise. The proposed system has two stages, face recognition and drunk classification. In the face recognition stage, a robust face recognition algorithm is used to identify individuals. In the drunk classification, a grid of several representative thermal points is extracted as feature vectors. These chosen points are where capillaries and veins cross the face. The feature vectors are used to train traditional machine learning models (Radom Forest, RF or Support Vector Machine, SVM) and determine whether the person is drunk or not.
 
To compare the performance with the Convolutional Neural Network (CNN)-based technique, experimental results of the proposed method demonstrate its robustness against different challenges, and outperform the existing method of drunk classification.

Chapter 1 Introduction 1
Section 1.1 Motivation 1
Section 1.2 Thesis Organization 3
Chapter 2 Backgrounds 4
Section 2.1 Thermal Imaging 4
Section 2.2 Feature Regions 6
Section 2.3 Support Vector Machine 6
Section 2.4 Random Forest 7
Section 2.5 Gaussian Mixture Model 8
Section 2.6 Convolutional Neural Networks 8
Chapter 3 Related Work 10
Section 3.1 Physiology-Based Face Recognition in the Thermal Infrared Spectrum 10
Section 3.2 Thermal Face Recognition Under Temporal Variation Conditions 11
Section 3.3 Face Recognition and Drunk Classification Using Infrared Face Images 12
Chapter 4 The Proposed Method 14
Section 4.1 Face Recognition using Temperature Information 15
Section 4.1.1 Traditional Face Recognition 15
Section 4.1.2 CNN-based Face Recognition 17
Section 4.2 Drunk Classification using Temperature Information 18
Section 4.2.1 Traditional Drunk Classification 18
Section 4.2.2 CNN-based Drunk Classification 21
Section 4.3 Adaptive Architecture of Face Recognition and Drunk Classification 23
Section 4.3.1 Occlusion Detection 24
Section 4.3.2 Feature Extraction for Face Recognition 24
Section 4.3.3 Feature Extraction for Drunk Classification 25
Chapter 5 Experimental Results and Performance Analysis 26
Section 5.1 Experiment Databases 26
Section 5.1.1 UCH Thermal Temporal Face Database 27
Section 5.1.2 PUCV Thermal Temporal Face Database 27
Section 5.1.3 Experiment Images 28
Section 5.2 Face Recognition Experiments 29
Section 5.2.1 Three training sets 30
Section 5.2.2 Results with RF approach 31
Section 5.2.3 Results with SVM approach 34
Section 5.2.4 Results with CNN approach 37
Section 5.2.5 Results with adaptive architecture 38
Section 5.2.6 Comparison with other methods 41
Section 5.3 Drunk Classification Experiments 45
Section 5.3.1 Results with RF approach 46
Section 5.3.2 Results with SVM approach 47
Section 5.3.3 Results with CNN approach 48
Section 5.3.4 Results with adaptive approach 49
Section 5.3.5 Comparison with existing methods 50
Section 5.4 Others Testing Results 51
Section 5.4.1 Different structures in CNN approach 51
Section 5.4.2 Different samples in drunk classification 54
Section 5.4.3 Different training and test sets 55
Section 5.4.4 Different threshold values in drunk classification 61
Chapter 6 Conclusions 62
References 64

1.Lin, S. D., Liu, B. F., and Lin, J. H., 2012, Combining Speeded Up Roust Features with Principal Component Analysis in Face Recognition System, International Journal of Innovative Computing, Information and Control, Vol. 8, No. 12, pp. 8545-8556.

2.Adini, Y., Moses, Y., and Ullman, S., 1997, Face recognition: the problem of compensating for changes in illumination direction, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, pp. 721-732.

3.Kopaczka, M., Kolk, P., and Merhof, D., 2018, A fully annotated thermal face database and its application for thermal facial expression recognition, IEEE International Instrumentation and Measurement Technology Conference (I2MTC), pp 1-6.

4.Pavlović, M., Stojanović, B., Petrović, R., and Stanković, S., 2018, Fusion of Visual and Thermal Imagery for Illumination Invariant Face Recognition System, 14th Symposium on Neural Networks and Applications (NEUREL), pp. 1-5.

5.Ghiass, R. S., Arandjelović, O., Bendada, H., and Maldague, X., 2013, Illumination-invariant face recognition from a single image across extreme pose using a dual dimension AAM ensemble in the thermal infrared spectrum, The International Joint Conference on Neural Networks (IJCNN), pp. 1-10.

6.Zhu, Z., Tsiamyrtzis, P., and Pavlidis, I., 2007, Forehead Thermal Signature Extraction in Lie Detection, 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 243-246.

7.Uddin, M. Z. and Torresen, J., 2018, A Deep Learning-Based Human Activity Recognition in Darkness, Colour and Visual Computing Symposium (CVCS), pp. 1-5.
8. Kirimtat, A., Krejcar, O., Selamat, A. and Herrera-Viedma, H., 2020, FLIR vs SEEK thermal cameras in biomedicine: comparative diagnosis through infrared thermography, BMC Bioinformatics 21, pp. 88.

9.Socolinsky, D. A. and Selinger, A., 2004, Thermal face recognition in an operational scenario, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, pp. 22.

10.Chiang, L. W. and Guo, S. J., 2017, The Research of Infrared Thermal Image Diagnostic Model for Building External Wall Tiles, Advances in Technology Innovation, Vol. 2, No. 2, pp. 56-60.

11.Chen, X., Flynn, P. J., and Bowyer, K. W., 2005, Visible-light and infrared face recognition, Computer Vision and Image Understanding, Vol. 99, No. 3, pp. 332-358.

12.Hermosilla, G., Ruiz-del-Solar, J., Verschae, R., and Correa, M., 2012, A comparative study of thermal face recognition methods in unconstrained environments, Pattern Recognition, Vol. 45, No. 7, pp. 2445-2459.

13.Guzman, Ana M., Goryawala, M., and Adjouadi, M., 2012, Generating thermal facial signatures using thermal infrared images, IEEE International Conference on Emerging Signal Processing Applications, pp. 21-24.

14.Buddharaju, P. and Pavlidis, I., 2009, Physiological face recognition is coming of age, IEEE Conference on Computer Vision and Pattern Recognition,, pp. 128-135.

15.Song, H., AlAlkeem, E., Yun, J., Kim, T, Yoo, H., Heo, D., Chae, M., and Yeun, C, Y., 2020, Deep user identification model with multiple biometric data, BMC Bioinformatics 21, pp. 315.

16.Buddharaju, P., Pavlidis, I. T., Tsiamyrtzis, P., and Bazakos, M., 2007, Physiology-based face recognition in the thermal infrared spectrum, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 4, pp. 613-626.

17.Vigneau, G. H., Verdugo, J. L., Verdugo, L., Farias Castro, G., Pizarro, F., and Vera, E., 2017, Thermal Face Recognition Under Temporal Variation Conditions, IEEE Access, Vol. 5, pp. 9663-9672.

18.Hermosilla, G., Verdugo, J. L., Farias, G., Vera, E., Pizarro, F., and Machuca, M., 2018, Face Recognition and Drunk Classification Using Infrared Face Images, Journal of Sensors, pp. 1-8.

19.Rustemeyer, J., Radtke, J., and Bremerich, A., 2007, Thermography and thermoregulation of the face, Head Face Med, Vol. 3, pp. 17.

20.Breiman, L., 2001, Random Forests, Machine Learning 45, pp. 5-32.

21.Bhowmik, M. K., De, B. K., Bhattacharjee, D., Basu, D. K., and Nasipuri, M., 2012, Multisensor fusion of visual and thermal images for human face identification using different SVM kernels, IEEE Long Island Systems, Applications and Technology Conference (LISAT), pp. 1-7.

22.LeCun, Y. and Bengio, T., 1995, Convolutional networks for images speech and time series, Handbook of Brain Theory and Neural Networks, MIT Press, pp. 255-258.

23.FLIR ONE Pro-Series Comparison Flyer, 2018, https://www.flir.com/globalassets/imported-assets/document/flir-one-pro-series-comparison-flyer.pdf

24.Marin, J. M., Mengersen. K., and Pobert. C. P., 2005, Bayesian modelling and inference on mixtures of distributions, Handbook of Statistics, pp 459-507.

25.Yu, I., Kim, D., Park, J., Hou, J., Choi, S., and Lee, H., 2017, Identifying photorealistic computer graphics using convolutional neural networks, IEEE International Conference on Image Processing (ICIP), pp. 4093-4097.

26.Yang, J., Ren, P., Zhang, D., Chen, D., Wen, F., Li, H., and Hua, G., 2017, Neural Aggregation Network for Video Face Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5216-5225.

27.Simonyan, K. and Zisserman, A., 2015, Very Deep Convolutional Networks for Large-Scale Image Recognition, International Conference on Learning Representations, 10 Apr.

28.Koukiou, K. and Anastassopoulos, G., 2012, Drunk person identification using thermal infrared images, International Journal of Electronic Security and Digital Forensics, Vol. 4, pp 229-243.

29.Gonzalez, R. C. and Woods, R, E., 2017, Digital image processing, Pearson FT Press.

30.Siddique, M, A. B., Arif, R. B., and Khan. M. M. R., 2018, Digital Image Segmentation in Matlab: A Brief Study on OTSU’s Image Thresholding, International Conference on Innovation in Engineering and Technology, pp 1-5.

31.Jagatheeswari, P., Suresh Kumar S., and Rajaram. M., 2010, Contrast Stretching Recursively Separated Histogram Equalization for Brightness Preservation and Contrast Enhancement, International Conference on Advances in Computing, Control, and Telecommunication Technologies, pp 111-115.

32.Wang, T., Su, J., Huang, Y., and Zhu, Y., 2010, Study of the pseudo-color processing for infrared forest-fire image, IEEE International Conference on Power Electronics, Intelligent Control and Energy Systems, pp 415-478.

33.Petrović, R., Stojanović, B., Pavlović, S., and Stanković, S., 2019, Thermal Imagery Noise Influence on Face Recognition Performance, 18th International Symposium INFOTEH-JAHORINA (INFOTEH), pp. 1-5.

34.Ghiass, R. S., Arandjelović, O., Bendada, H., and Maldague, X., 2013, Illumination-invariant face recognition from a single image across extreme pose using a dual dimension AAM ensemble in the thermal infrared spectrum, The International Joint Conference on Neural Networks (IJCNN), pp. 1-10.

35.Lin, S. D. and Chen, K., 2019, Thermal Face Recognition under Disguised Conditions, International Conference on Machine Learning and Cybernetics (ICMLC), pp. 1-7.

36.Lin, S. D., Chen, L., and Chen, W., 2021, Thermal face recognition under different conditions, BMC Bioinformatics, Vol. 22, Supp. 5, No. 313.