序列資料具有連續性、順序性、關聯性，透過循環神經網路(Recurrent Neural Network, RNN)，將序列資料輸入至神經網路中，以進行預測及分析，但若序列太長，循環神經網路會難以捕捉整個序列資料之間的關聯性，以及梯度爆炸(exploding gradient)、梯度消失(vanishing gradient)等問題，而長短期記憶(Long Short-Term Memory, LSTM)及 GRU(Gated Recurrent Unit)針對循環神經網路的問題，在神經網路的結構上進行修改，過去研究也表明此二種模型在時間序列的分析上，表現較循環神經網路優。
SVM(Support Vector Machine)在過去相關研究中，是分類(classification)工具，而其於預測(prediction)的延伸 SVR(Support Vector Regression)在預測數值上也獲得良好的表現。本篇研究將結合循環神經網路及 SVM 與 SVR，並測試多種類型的序列資料，分別使用循環神經網路、長短期記憶、GRU，並加入 SVM 與 SVR 做最後的分類或預測，本研究期望透過循環神經網路結合 SVM 與 SVR，以提高對於序列資料進行分類或預測的性能。

Sequence data is continuity and sequential, we can analyze the sequence data by Recurrent Neural Network(RNN), but if sequence data is too long, RNN can’t catch all relationship between sequence data, and there are exploding gradient problem and vanishing gradient problem. Long Short-Term Memory(LSTM) and Gated Recurrent Unit(GRU) modified the structure of neural network, and we can also analyze the sequence data by LSTM and GRU. In addition, Support Vector Machine (SVM) and Support Vector Regression(SVR) are good at classification and prediction, respectively.
In the literature, most studies combine LSTM and SVM to classify sequence data. In this study, we combine RNN, LSTM, and GRU with SVM and SVR to classify and predict sequence data. The experiment results show that combined models outperform RNN, LSTM, and GRU.

摘要 I
Abstract II
目錄 III
圖目錄 VII
表目錄 VIII
第一章 緒論 1
1.1 研究背景與動機 1
1.2 研究目的 2
第二章 文獻探討 3
2.1 序列資料 3
2.1.1 生物、遺傳領域 3
2.1.2 天氣 3
2.1.3 網頁瀏覽量 4
2.1.4 網路入侵檢測 4
2.1.5 降雨量 4
2.1.6 地震預測 5
2.1.7 股價預測 5
2.1.8 醫療領域 5
2.1.9 文本情感分析 6
2.2 深度學習 8
2.2.1 RNN 9
2.2.2 LSTM 10
2.2.3 GRU 11
2.3 SVM/SVR 12
2.4 應用深度學習模型及 SVM 於序列資料 13
第三章 研究方法 16
3.1 研究架構 16
3.2 資料收集 16
3.3 資料前處理 18
3.3.1 分類 18
3.3.2 預測值 20
3.4 訓練模型 22
3.4.1 RNN 與 SVM/SVR 28
3.4.2 LSTM 與 SVM/SVR 29
3.4.3 GRU 與 SVM/SVR 29
第四章 實驗結果 31
4.1 分類 31
4.1.1 Delhi Weather 31
4.1.2 Istanbul Weather 32
4.1.3 Occupancy Detection 32
4.1.4 Air Properties 32
4.1.5 Air Quality 33
4.1.6 Rainfall Prediction 33
4.2 預測 34
4.2.1 Yahoo stock 34
4.2.2 Twitter stock 34
4.2.3 Netflix stock 35
4.2.4 Apple stock 35
4.2.5 natural gas 35
4.2.6 gold price 36
4.3 t 檢定 36
4.3.1 Delhi Weather 36
4.3.2 Istanbul Weather 38
4.3.3 Occupancy Detection 39
4.3.4 Air Properties 40
4.3.5 Air Quality 43
4.3.6 Rainfall Prediction 44
4.3.7 Yahoo stock 45
4.3.8 Twitter stock 47
4.3.9 Netflix stock 48
4.3.10 Apple stock 49
4.3.11 natural gas 50
4.3.12 gold price 52
第五章 結論 54
參考文獻 55

[1] R. Adhikari and R. K. Agrawal, “An introductory study on time series modeling and forecasting,” 2013, arXiv:1302.6613.
[2] J. Cervantes, F. Garcia-Lamont, L. Rodríguez-Mazahua, and A. Lopez, “A comprehensive survey on support vector machine classification: applications, challenges and trends,” Neurocomputing, vol. 408, pp. 189-215, Sep. 2020.
[3] A. Cimino and F. Dell’Orletta, “Tandem LSTM-SVM approach for sentiment analysis,” Final Workshop(EVALITA 2016), 2016.
[4] O. Alharbi, “A deep learning approach combining CNN and Bi-LSTM with SVM classifier for Arabic sentiment analysis,” International Journal of Advanced Computer Science and Applications(IJACSA), vol. 12, no. 6, 2021.
[5] S. Patidar, M. Tripathi, and S. K. Gupta, “Leveraging LSTM-RNN combined with SVM for network intrusion detection,” in Int. Conf. on Data Science, Machine Learning and Artificial Intelligence, 2021, pp. 26-31.
[6] Y. Hamdi, H. Boubaker, B. Rabhi, W. Ouarda, B. B. Chaudhuri, and A. M. Alimi, “Hybrid architecture based on RNN-SVM for multilingual online handwriting recognition using Beta-elliptic and CNN models,” 2021, TechRxiv.
[7] P. Khan, P. Ranjan, Y. Singh, and S. Kumar, “Warehouse LSTM-SVM-Based ECG data classification with mitigated device heterogeneity,” IEEE Transactions on Computational Social Systems, vol. 9, no. 5, 2022.
[8] V. Anandhi and R. Manicka Chezian, “Support vector regression to forecast the demand and supply of pulpwood,” International Journal of Future Computer and Communication, vol. 2, no. 3, pp. 266-269, 2013.
[9] D. Y. Li, W. Xu, H. Zhao, and R. Q. Chen, “A SVR based forecasting approach for real estate price prediction,” in 2009 Int. Conf. on Machine Learning and Cybernetics(ICMLC), 2009.
[10] D. Quang and X. Xie, “DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences,” Nucleic Acids Research, vol. 44, no. 11, 2016.
[11] N. Kumar and G. K. Jha, “A Time series ANN approach for weather forecasting,” International Journal of Control Theory and Computer Modeling(IJCTCM), vol.3, no. 1, Jan. 2013.
[12] Z. Karevan and J. A. K. Suykens, “Transductive LSTM for time-series prediction: an application to weather forecasting,” Neural Networks, vol. 125, pp. 1-9, 2020.
[13] A. G. Salman, Y. Heryadi, E. Abdurahman, and W. Suparta, “Single layer & multi-layer long short-term memory (LSTM) model with intermediate variables for weather forecasting,” Procedia Computer Science, vol. 135, pp. 89-98, 2018.
[14] R. Casado-Vara, A. M. del Rey, D. Pérez-Palau, L. de-la-Fuente-Valentín, and J. M. Corchado, “Web traffic time series forecasting using LSTM neural networks with distributed asynchronous training,” Mathematics, 2021.
[15] T. Shelatkar, S. Tondale, S. Yadav, and S. Ahir, “Web traffic time series forecasting using ARIMA and LSTM RNN,” in ITM Web of Conf., 2020.
[16] K. Saurabh et al., “LBDMIDS: LSTM based deep learning model for intrusion detection systems for IoT networks,” 2022 IEEE World AI IoT Congress(AIIoT), 2022.
[17] Y. Fu, Y. Du, Z. Cao, Q. Li, and W. Xiang, “A Deep Learning Model for Network Intrusion Detection with Imbalanced Data,” Electronics, vol. 11, 2022.
[18] I. Salehin, I. M. Talha, Md. Mehedi Hasan, S. T. Dip, Mohd. Saifuzzaman, and N.
N. Moon, “An artificial intelligence based rainfall prediction using LSTM and neural network,” in 2020 IEEE Int. Women in Engineering(WIE) Conf. on Electrical and Computer Engineering(WIECON-ECE), 2020.
[19] Z. Xiang, J. Yan and I. Demir, “A rainfall-runoff model with LSTM-based sequence-to-sequence learning,” Water Resources Research, 2020.
[20] S. Aswin, P. Geetha and R. Vinayakumar, “Deep learning models for the prediction of rainfall,” in 2018 Int. Conf. on Communication and Signal Processing(ICCSP), 2018.
[21] A. Berhich, Fatima-Zahra Belouadha, and M. I. Kabbaj, “LSTM-based models for earthquake prediction,” in 3rd Int. Conf. on Networking, Information Systems & Security, 2020, pp. 1-7.
[22] K. Vardaan, T. Bhandarkar, N. Satish, S. Sridhar, R. Sivakumar, and S. Ghosh, “Earthquake trend prediction using long short-term memory RNN,” International Journal of Electrical and Computer Engineering(IJECE), vol. 9, no. 2, pp. 1304-1312, 2019.
[23] G. Ding and L. Qin, “Study on the prediction of stock price based on the associated network model of LSTM,” International Journal of Machine Learning and Cybernetics, pp. 1307-1317, 2020.
[24] W. Bao, J. Yue, and Y. Rao, “A deep learning framework for financial time series using stacked autoencoders and long-short term memory,” PLoS One, 2017.
[25] H. Abrishami, C. Han, X. Zhou, M. Campbell, and R. Czosek, “Supervised ECG interval segmentation using LSTM neural network,” in Int’l Conf. Bioinformatics and Computational Biology, 2018.
[26] A. Patel and A. K. Tiwari, “Sentiment Analysis by using Recurrent Neural Network,” in 2nd Int. Conf. on Advanced Computing and Software Engineering(ICACSE) 2019, 2019.
[27] X. Wang, W. Jiang, and Z. Luo, “Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts,” in 26th Int. Conf. on Computational Linguistics: Technical Papers, 2016.
[28] T. Tesha and B. Kichonge, “Analysis of Tanzanian Biomass Consumption Using Artificial Neural Network,” Journal of Fundamentals of Renewable Energy and Applications, 2015.
[29] I. Zhukovyts’kyy, V. Pakhomova, H. Domanskay, and A. Nechaiev, “Distribution of information flows in the advanced network of MPLS of railway transport by means of a neural model,” in MATEC Web of Conf., 2019.
[30] L. Datta, “A Survey on Activation Functions and their relation with Xavier and He Normal Initialization,” 2020, arXiv:2004.06632.
[31] L. Kurniasari and A. Setyanto, “Sentiment analysis using recurrent neural network,” in Journal of Physics: Conf. Series, 2020.
[32] D. Datta, P. E. David, D. Mittal, and A. Jain, “Neural machine translation using recurrent neural network,” International Journal of Engineering and Advanced Technology(IJEAT), vol.9, pp. 1395-1400, 2020.
[33] W. Feng, N. Guan, Y. Li, X. Zhang, and Z. Luo, “Audio visual speech recognition with multimodal recurrent neural networks,” in 2017 Int. Joint Conf. on Neural Networks(IJCNN), 2017.
[34] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[35] S. A. Zargar, “Introduction to Sequence Learning Models: RNN, LSTM, GRU”, 2021, ResearchGate.
[36] X. Li et al., “Long short-term memory neural network for air pollutant concentration predictions: method development and evaluation,” Environmental Pollution, vol. 231, pp. 997-1004, 2017.
[37] K. Cho et al., “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” 2014, arXiv:1406.1078.
[38] X. Wang, J. Xu, W. Shi, and J. Liu, “OGRU: An Optimized Gated Recurrent Unit Neural Network,” in Journal of Physics: Conf. Series, 2019.
[39] C. Cortes and V. Vapnik, ”Support-vector networks,” Machine Learning, pp. 273-297, 1995.
[40] S. Schlag, M. Schmitt, and C. Schulz, “Faster support vector machines,” ACM Journal of Experimental Algorithmics, vol. 26, no. 15, pp. 1-21, 2021.
[41] G. Liu, S. Mao, and J. H. Kim, “A mature-tomato detection algorithm Using machine learning and color analysis,” Sensors, 2019.
[42] X. Wang et al., “A hybrid model for prediction in asphalt pavement performance based on support vector machine and grey relation analysis,” Journal of Advanced Transportation, 2020.
[43] J. Chen, H. Chen, Y. Huo, and W. Gao, “Application of SVR models in stock index forecast based on different parameter search methods,” Open Journal of Statistics, pp. 194-202, 2017.
[44] Q. Guo, Y. Feng, X. Sun, and L. Zhang, “Power demand forecasting and application based on SVR,” Procedia computer science, vol. 122, pp. 269-275, 2017.
[45] D. Güleryüz, “Predicting health spending in Turkey using the GPR, SVR, and DT models,” Acta Infologica, pp. 155-166, 2021.
[46] G. Gou, “SVR-based prediction of carbon emissions from energy consumption in Henan province,” in IOP Conf. Series: Earth and Environmental Science, 2018.
[47] M. Alamaniotis, and V. Agarwal, “Fuzzy Integration of Support Vector Regression Models for Anticipatory Control of Complex Energy Systems,” The International Journal of Monitoring and Surveillance Technologies Research(IJMSTR), 2014.
[48] A. F. M. Agarap, “A neural network architecture combining gated recurrent unit (GRU) and support vector machine (SVM) for intrusion detection in network traffic data,” in ICMLC 2018: 2018 10th Int. Conf. on Machine Learning and Computing, 2018.
[49] X. Yin, Q. Liu, Y. Pan, X. Huang, J. Wu, and X. Wang, “Strength of stacking technique of ensemble learning in rockburst prediction with imbalanced data: comparison of eight single and ensemble models,” Natural Resources Research, pp. 1795-1815, 2021.
[50] Y. Tang, “Deep learning using linear support vector machines,”
2013, arXiv:1306.0239.
[51] Ö . Köroğlu, “Weather Istanbul data 2009-2019.”
Kaggle.https://www.kaggle.com/datasets/vonline9/weather-istanbul-data-20092019 (accessed Jul. 11, 2022).
[52] S. Jafer, “Delhi weather dataset.” Kaggle.
https://www.kaggle.com/datasets/syedjaferk/delhi-weather-dataset (accessed Jul. 11, 2022).
[53] Möbius, “Time series forecasting with Yahoo stock price.”
Kaggle. https://www.kaggle.com/datasets/arashnic/time-series-forecasting-withyahoo-stock-price (accessed Apr. 14, 2022)
[54] M. Pandya, “Twitter stocks dataset.” Kaggle. https://www.kaggle.com/datasets/maharshipandya/twitter-stocks-dataset (accessed Jul. 20, 2022).
[55] J. Shah, “Netflix stock price prediction.”
Kaggle.https://www.kaggle.com/datasets/jainilcoder/netflix-stock-priceprediction (accessed Jun. 15, 2022)
[56] M. Nagadia, “Stock Price of Top 10 SmartPhone Company 2016-2021.” Kaggle. https://www.kaggle.com/datasets/meetnagadia/stock-price-of-top-10-smartphone-company-20162021 (accessed Apr. 28, 2023).
[57] B. Senkal, “Daily Natural Gas Historical Data 12.2022. ” Kaggle.
https://www.kaggle.com/datasets/psycon/historical-natural-gas-data-from-2000-to-202204 (accessed May 19, 2023).
[58] N. Chodvadiya, “Daily Gold Price (2015-2021) Time Series.” Kaggle.https://www.kaggle.com/datasets/nisargchodavadiya/daily-gold-price-20152021-time-series (accessed Jul. 11, 2022).
[59] S. Patki, “Occupancy Detection Dataset. ” Kaggle. https://www.kaggle.com/datasets/saumitgp/occupancy-detection-dataset (accessed May 4, 2023).
[60] S. Shahane, “Air Quality Dataset. ”
Kaggle.https://www.kaggle.com/datasets/saurabhshahane/adl-classification (accessed Jun. 13, 2023).
[61] V. Madane, “Air Properties Time Series Dataset. ”
Kaggle.https://www.kaggle.com/datasets/vitthalmadane/pharma-air-handlingunits-data (accessed Jun. 19, 2023).
[62] ROHIT_GURUNADH, “Rainfall prediction. ” Kaggle.
https://www.kaggle.com/datasets/abhishek1405/rainfall-prediction (accessed Jun.30, 2023).