With the use of Augmented Reality (AR) glasses, a new system to teach and help people how to cook was created. Recognition of cooking ingredients and hand gestures were achieved by using Computer Vision techniques. The new system scans for food ingredients and recommends a number of recipes, based on the scan, for the user to select from. The user is then able to follow the recipe by the aid of a step-by-step interactive guide. The system is entirely hands-free, and is able to be navigated through a series of eight unique hand gestures. Twenty participants were gathered to test the system and provide feedback via questionnaires. Most participants were satisfied with the food they made and would recommend the system to other people.

Acknowledgement i
Abstract ii
Table of Contents iii
List of Figures v
List of Tables vii
Chapter 1. Introduction 1
1.1 Research Background and Motivation 1
1.2 Research Objectives 3
1.3 Research Contributions 4
Chapter 2. Literature Review 5
2.1 Related Works 5
2.1.1 Smart Kitchens 5
2.1.2 AR Cooking 6
2.1.3 AR and Smart Kitchens 7
2.2 Pros and Cons 8
Chapter 3. Proposed Method 10
3.1 Getting a Recipe 12
3.1.1 Scanning Ingredients 13
3.1.2 No Scan 13
3.2 Recipe 13
3.2.1 Recipe - Scan 13
3.2.2 Recipe – No Scan 14
3.3 Practicing the Recipe 17
3.4 Gesture Recognition 18
Chapter 4. Experimental Methods 21
4.1 Experimental Environment 21
4.2 Neural Network 22
4.3 Dataset 22
4.4 ResNet-9 VS ResNeXt-50 Results 24
4.5 Yolov5 Results 32
4.6 Participants 38
4.7 Questionnaire 39
4.8 Questionnaire Results 41
Chapter 5. Conclusion and Future Works 48
5.1 Future Research Direction 48
5.2 Challenges 48
5.3 Possible Future Applications 49
References 50

[1] IBM Cloud Education. (2020, August 17). What are Neural Networks? [Learning]. What Are Neural Networks?| IBM. https://www.ibm.com/cloud/learn/neural-networks
[2] PyTorch ResNet. (n.d.). [Article Guide]. PyTorch ResNet. https://www.run.ai/guides/deep-learning-for-computer-vision/pytorch-resnet
[3] ResNet9 | Fastestimator. (n.d.). [Documentation]. ResNet9 | Fastestimator.
https://www.fastestimator.org/api/r1.2/fe/architecture/pytorch/ResNet9
[4] Oltean, M. (2021, September 12). Fruits 360 [Dataset]. Fruits 360|Kaggle.
https://www.kaggle.com/datasets/moltean/fruits
[5] Horea Muresan, Mihai Oltean, Fruit recognition from images using deep learning, Acta Univ. Sapientiae, Informatica Vol. 10, Issue 1, pp. 26-42, 2018.
[6] F. Nugroho and A.B. Pantjawati. 2018. Automation and Monitoring Smart Kitchen Based on Internet of Things (IoT). IOP Conference Series: Materials Science and Engineering 384 (2018), 012007. DOI:http://dx.doi.org/10.1088/1757-899x/384/1/012007
[7] Adrian Iftene, Diana Trandabăț, Vlad Rădulescu, Eye and Voice Control for an Augmented Reality Cooking Experience, Procedia Computer Science, Volume 176, 2020, Pages 1469-1478, ISSN 1877-0509, https://doi.org/10.1016/j.procs.2020.09.157.
[8] Dormehl, L. (2020, January 8). Samsung’s new food A.I. can suggest recipes based on what’s in your fridge. [News]. Sam-sung’s Food A.I. Suggests Recipes Based on What’s in Your Fridge | Digital Trends. https://www.digitaltrends.com/home/samsung-fridge-ai-suggest-recipes-ces-2020/
[9] Marcus Stander, Aristotelis Hadjakos, Niklas Lochschmidt, Christian Klos, Bastian Renner, and Max Muhlhauser. 2012. A Smart Kitchen Infrastructure. 2012 IEEE International Symposium on Multimedia(2012). DOI:http://dx.doi.org/10.1109/ism.2012.27
[10] Watts, D. (2021, March 16). How smart kitchens are improving our lives [News]. How Smart Kitchens Are Improving Our Lives - The AI Journal. https://aijourn.com/how-smart-kitchens-are-improving-our-lives/
[11] Chia-Hsun Lee, Leonardo Bonanni, and Ted Selker. 2005. CounterIntelligence: Augmented Reality Kitchen. CHI 2239 (January 2005), 45
[12] 1.1 Unity Setup. (n.d.). [Documentation]. Get Started with Unity | Magic Leap. https://developer.magicleap.com/en-us/learn/guides/get-started-developing-in-unity
[13] Xie, Saining & Girshick, Ross & Dollar, Piotr & Tu, Z. & He, Kaiming. (2017). Aggregated Residual Transformations for Deep Neural Networks. 5987-5995. 10.1109/CVPR.2017.634.
[14] M.Sundarapandiyan, Dr.S.Karthik, Mr.J.Alfred Daniel. "IOT based Smart Kitchen". International Journal of Computer Science Trends and Technology (IJCST) V7(2): Page(13-16) Mar-Apr 2019. ISSN: 2347-8578. www.ijcstjournal.org.Published by Eighth Sense Research Group.
[15] Palandurkar, V. R., Mascarenhas, S. J., Nadaf, N. D., & Kunwar, R. A. (2020). SMART KITCHEN SYSTEM USING IOT. In International Journal of Engineering Applied Sciences and Technology (Vol. 04, Issue 11, pp. 378–383). IJEAST. https://doi.org/10.33564/ijeast.2020.v04i11.067
[16] Hashimoto, Atsushi & Mori, Naoyuki & Funatomi, Takuya & Yamakata, Yoko & Kakusho, Koh & Minoh, Michihiko. (2008). Smart kitchen: A user centric cooking support system.
[17] Q-100, ingredients-classification, (2021), GitHub repository, https://github.com/Q-100/ingredients-classification?fbclid=IwAR2_Qu5XRjKFV_FerUzu7Ubqm_GWLX3KoHLTSXOQHGkvgGCFNeYsXhDnIDc