日活躍用戶數量(Daily Active User，DAU)是一種用於網站、社群媒體或網路遊戲運營情況的統計指標，對於公司進行運營決策有著關鍵的作用。DAU作為一種時間序列資料，其升降趨勢反映了相關運營決策是否正確，因此對其進行預測是十分重要的工作。

而近年Transformer模型於電腦科學的各個領域皆有廣泛的應用，有別於常見的深度學習方法，Transformer藉由Self-attention追蹤序列資料間的關係，學習序列的相關特徵，且有更快速的運算速度。
因此本文基於Transformer模型對 Steamdb 上 9 款不同流量網路遊戲的DAU進行預測，並使用Scheduled sampling 的方法來增加推論(inference)的準確性。實驗資料使用9款網路遊戲 2020/3/1至2023/3/1 三年的每日遊玩人數，根據流量分為低流量(小於5萬)、中流量(5至30萬)、高流量(大於30萬)與統整四個資料集，其中 2020/3/1至 2022/12/1作為訓練資料，並以2022/12/2至2023/3/1 的資料進行實驗預測。
實驗結果表明Scheduled Sampling有助於推論的準確性，相較於單純Teacher Forcing有更低的inference loss。在四個資料集中，Transformer模型對於未來2、3、5、7日之每日遊玩人數預測有良好的效果。另外本研究發現輸入模型的訓練長度與資料集內數據分布兩個因素對於Transformer模型的預測結果有著顯著影響。

The Daily Active Users (DAU) is a statistical indicator used in website, social media, or online game operations, and plays a crucial role in making operational decisions for companies. As a time-series data, the rise and fall trend of DAU reflects the correctness of relevant operational decisions, making its prediction an important task.
In recent years, the Transformer model has been widely applied in various fields of computer science. Unlike conventional deep learning methods, the Transformer utilizes self-attention to track the relationships between sequential data and learns relevant features of the sequence, while achieving faster computational speed.
Therefore, this paper is based on the Transformer model to predict the DAU of 9 different online games on Steamdb. Scheduled sampling is used to enhance the accuracy of inference. The experimental data consists of daily player counts of the 9 games from March 1, 2020, to March 1, 2023, categorized into four datasets based on traffic: low traffic (less than 50,000), medium traffic (50,000 to 300,000), high traffic (greater than 300,000), and the overall dataset. The training data includes the period from March 1, 2020, to December 1, 2022, and the experiment prediction is conducted using the data from December 2, 2022, to March 1, 2023.
The experimental results demonstrate that Scheduled Sampling contributes to the accuracy of inference, with lower inference loss compared to pure Teacher Forcing. In the four datasets, the Transformer model performs well in predicting the daily player counts for the next 2, 3, 5, and 7 days. Furthermore, this study finds that the length of training data and the data distribution within the dataset significantly affect the prediction results of the Transformer model.

第一章、緒論 1
1.1 研究背景和動機 1
1.2 研究目的 2
1.3 論文架構 3
第二章、文獻回顧 4
2.1 時間序列預測 4
2.2 Transformer 模型的各種應用 7
2.3 Self-Attention 11
Query, Key and Value 11
計算 Self-Attention分數 12
進行編碼(Encoding) 12
2.4 Scheduled Sampling 12
第三章、研究方法 15
3.1 資料蒐集 15
3.2 研究設計 16
Positional Encoding 16
模型架構 17
Masked Self-Attention 18
使用Scheduled Sampling 18
第四章、實驗結果與討論 21
4.1 實驗資料集 21
4.2 評估方式 21
4.3 訓練長度對預測之影響 21
4.4 Scheduled Sampling不同遞減方式之比較 22
4.5 Scheduled Sampling參數調整結果 24
4.6 各資料集不同天數之預測結果 26
4.7 資料集中數據分布影響 32
4.8 Transformer與LSTM、ARIMA模型之比較 35
第五章、結論與未來展望 38
第六章、參考文獻 40

[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,Aidan N.Gomez, Lukasz Kaiser, Illia Polosukhin.”Attention is all you need.”arXiv preprint arXiv:1706.03762v5,2017.
[2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer.”Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.”arXiv preprint arXiv:1506.03099v3,2015.
[3] Ayodele A. Adebiyi, Aderemi O. Adewumi, Charles K. Ayo.”Stock Price Prediction Using the ARIMA Model.”In UKSim-AMSS 16th International Conference on Computer Modelling and Simulation,2014.
[4] Guokun Lai, Wei-Cheng Chang, Yiming Yang, Hanxiao Liu.”Modeling Long-and Short-Term Temporla Patterns with Deep Neural Networks.” arXiv preprint arXiv:1703.07015v3,2018.
[5] Neo Wu, Bradley Green, Xue Ben, Shawn O’Banion.”Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case.”arXiv preprint arXiv:2001.08317v1,2020.
[6] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayany Kalagnanam.”A Time Series Is Worth 64 Words : Long-Term Forecasting With Transformers.”arXiv preprint arXiv:2211.14730v2,2023.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.”arXiv preprint arXiv:1810.04805v2,2019.
[8] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever. “Improving Language Understanding by Generative Pre-Training.” OpenAI,June.2018.
[9] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever.”Language Models are Unsupervised Multitask Learners.”OpenAI,Feb.2019.
[10] OpenAI.”Language Models are Few-Shot Learners.”arXiv preprint arXiv: 2005.14165v4,2020.
[11] OpenAI.”GPT-4 Technical Report.”arXiv preprint arXiv:2303.08774v3,2023.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.”An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale.”In International Conference on Learning Representations (ICLR),2021.
[13] Natasha Klingenbrunn.”Transformers for Time-series Forecasting.”Feb.2021.
[14] Meta Platforms.”Quarterly Report.”Meta Platforms,Apr.2023
[15] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang.”Informer:Beyond Efficient Transformer for Long Sequence Time-Series Forecasting.”arXiv preprint arXiv:2012.07436v3,2021.
[16] Box G, Jenkins G.“Time series analysis: forecasting and control.”,1976.
[17] Michael Phi.”Illustrated Guide to LSTM’s and GRU’s :A step by step explanation.”Towards Data Science,https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21,Sep.2018(accessed July.2023).
[18] Jay Alammar.”The Illustrated Transformer.”Github,
http://jalammar.github.io/illustrated-transformer/,June.2018(accessed July.2023).