Realization of Gesture Control Application on Openmv Board Using Optical Flow in Real-Time Video Images

Abstract

OpenMV Board is designed for purpose of non-complex image processing applications. It is an image processing sensor that has been a MicroPython embedded operating-system(OS). In the study, it is aimed to develop gesture control applications for electrical household appliances and small budget devices. Therefore, the hardware to be used should be cheap and the algorithm should be simple. Thus three gesture control applications have been developed by using OpenMv board for use in different electrical appliances. These are 1-level control, 2-multi-component simple system control and 3-page flip. The algorithmsused in the study are independent of the user because they are optical flow-based. Thus,the use of low-cost simple gesture control applications for industrial purposes (electricalappliances) can be realized. Algorithms developed for applications were written on the OpenMV IDE. These application results were monitored in real-time through the IDE. In addition, the algorithm developed for level control has been embedded and tested on an SD card on OpenMv independent of OpenMV IDE. During the test, output information was generated using OpenMV pins and the level indicator created using yellow, green and red LEDs connected to the pins was checked real-time. Thus, the algorithm was tested on a computer-independent embedded system.

Keywords

• Gesture control
• Low-cost control with image processing
• Simple gesture control applications
• Image processing sensors
• Open source SmartCam

References

1. 1. Turan B, Eskikurt Hİ, Can MS. An aplication based on artificial neural network for determining viewpoint coordinates on a screen. Elektronika Ir Elektrotechnika 22 2 (2016) 86-91. DOI: http://dx.doi.org/10.5755/j01.eie.22.2.7586
2. 2. Lee WPO, Kaoli C, Huang JY. A smart TV system with body-gesture control, tag-based rating and context-aware recommendation. Knowledge-Based Systems 56 (2014) 167-178. https://doi.org/10.1016/j.knosys.2013.11.007
3. 3. Bellmore C, Ptucha R, Savakis A. Interactıve dısplay using depth and RGB sensors for face and gesture control. Western New York Image Processing Workshop, 2011. https://doi. org/10.1109/WNYIPW.2011.6122883.
4. 4. Yavşan E, Uçar A. Gesture imitation and recognition using Kinect sensor and extreme learning machines. Measurement 94 (2016) 852–861.
5. 5. Rahman ASMM, Saboune J, Saddik AE. Motion-path based in car gesture control of the multimedia devices. DIVANet '11 Proceedings of the first ACM international symposium on Design and analysis of intelligent vehicular networks and applications 69-76, 2016. https://doi. org/10.1145/2069000.2069013.
6. 6. Bhuiyan M, Picking R. Gesture-controlled user interfaces, what have we done and what’s next?. 2009. http://citeseerx. ist.psu.edu/viewdoc/download?doi=10.1.1.562.6140&rep =rep1&type=pdf. (accessed 13 11 2018).
7. 7. Kela J, Korpipaa P, Mantyjarvi J, Kallio S, Savino G, Jozzo L, Marca S. Accelerometer-based gesture control for a desing environment. Personal and Ubiquitous Computing (2006) 285-299. https://doi.org/10.1007/s00779-005-0033-
8. 8. Mantyjarvi J, Kela J, Korpipaa P, Kallio S. Enabling fast and effortless customisation in accelerometer based gestureinteraction. MUM '04 Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia 25-31, 2004. https://doi.org/10.1145/1052380.1052385

9. 9. Hackenberg G, McCall R, Broll W. Lightweight Palm and Finger Tracking for Real-Time 3D Gesture Control. 2011 IEEE Virtual Reality Conference, 2011. https://doi. org/10.1109/VR.2011.5759431.
10. 10. Akyol S, Canzler U, Bengler K, Hahn W. Gesture control for use in automobiles. IAPR Workshop on Machine Vision Applications, Nov. 28-30, 2000. The University of Tokyo, Japan. https://pdfs.semanticscholar.org/fb51/6222c7c 87f42872a28ff8fc74139447b1280.pdf. (accessed 13 11 2018).
11. 11. Bizzotto N, Costanza A, Bizzotto L, Revis D, Sandri A, Mangan B. Leap motion gesture control with osirix in the operating room to control imaging. Surgical Innovation, 2014. https://doi.org/10.1177/1553350614528384.
12. 12. Cohen CJ, Beach G, Foulk G. A Basic Hand Gesture Control System for PC Applications. IEEE Xplore Digital Library, 2001. https://doi.org/10.1109/AIPR.2001.991206.
13. 13. Gallo L, Placitelli AP, Ciampi M. Controller-free exploration of medical image data: Experiencing the Kinect. 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), 2011. https://doi.org/10.1109/ CBMS.2011.5999138.
14. 14. Doğan RÖ, Doğan H, Köse C. Virtual Mouse Control with Hand Gesture Information Extraction and Tracking. 23nd Signal Processing and Communications Applications Conference (SIU) 2015. https://ieeexplore.ieee.org/stamp/stamp. jsp?arnumber=7130228. (accessed 10 12 2019).
15. 15. Kaura HK, Honrao V, Patil S, Shetty P. Gesture Controlled Robot using Image Processing. (IJARAI) International Journal of Advanced Research in Artificial Intelligence, Vol. 2, No. 5, 201, 2013 https://pdfs.semanticscholar.org/ ff0f/20e3dbbdf257ec3ca36be4ed251036b49e11.pdf. (accessed 13 11 2018).
16. 16. Chowdary PRV, Babu MN, Subbareddy TV, Reddy BM, Elamaran V. Image Processing Algorithms for Gesture Recognition using MATLAB. 2014 IEEE International Conference on Advanced Communication Control and Computing Technologies (ICACCCT), 2014. https://doi. org/10.1109/ICACCCT.2014.7019356.
17. 17. Kar S, Banerjee S, Jana A, Kundu D, Chatterjee D, Ghosh S, Mitra D, Gupta SD. Image Processing Based Customized Image Editor and Gesture Controlled Embedded Robot Coupled with Voice Control Features. (IJACSA) International Journal of Advanced Computer Science and Applications 6 (2015) 11. https://pdfs.semanticscholar.org/f000/ be3e91d69dc8ce1f87ae32ae7e5395b09b86.pdf. (accessed 13 11 2018).
18. 18. Osimani C, Piedra-Fernandez JA, Ojeda-Castelo JJ, Iribarne L. Hand Posture Recognition with Standard Webcam for Natural Interaction, WorldCIST 2017. Advances in Intelligent Systems and Computing, vol 570 (2017) Springer.
19. 19. Hsiang-Yueh L, Hao-Yuan K, Yu-Chun H. Real-time Hand Gesture Recognition System and Application. Sensors and Materials, Vol. 30, No. 4 (2018) 869–884.
20. 20. Schacher JC. Gesture control is sounds in 3D space. Proceedings of the 2007 Conference on New Interfaces for Musical Expression (NIME07), New York, NY, USA, 2007. http://www.nime.org/proceedings/2007/nime2007_358. pdf. (accessed 13 11 2018).
21. 21. Erden F. Hand gesture recognition using two differential PIR sensors and a camera. 2014 22nd Signal Processing and Communications Applications Conference (SIU), 2014. https://doi.org/10.1109/SIU.2014.6830237.
22. 22. Şahin A. Hacking the Gestures of Past for Future Interactions. M.Sc. THESIS, 2013 http://muep.mau.se/ bitstream/handle/2043/15700/Atilim%20Sahin%20-%20 Hacking%20the%20Gestures%20of%20Past%20for%20 Future%20Interactions.pdf?sequence=2&isAllowed=y. (accessed 13 11 2018).
23. 23. Rautaray S, Agrawal A. Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43 (2015) 1–54
24. 24. Munir O, Ali AN, Javaan C. Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. Journal of Imaging 6(8) 2020 73.
25. 25. OpenMv Cam M7 Specifications, (2017). ttps://openmv.io/ products/openmv-cam-m7 . (accessed 14 11 2018).
26. 26. Optical flow-based gesture motion direction recognition method, https://patents.google.com/patent/ CN104331151A/en
27. 27. Pathak B, Jalal AS. Motion Direction Code—A Novel Feature for Hand Gesture Recognition. Advances in Intelligent Systems and Computing, vol 798 2019. Springer