◆Overview
Our Work aims at developing a system that helps guitarist to improve thier fingering without any single physical constraint such as color marker or sensor used in previous works. Recently, we mainly focus on deep learning-based methods of (1) a fully convolutional based hand segmentation, which is an end-to end, pixel-wised segmentation algorithm of guitarist hand area; (2) a CNN-based hand pose estimation, which is used for locating the joints of guitarist frame by frame during his or her playing; (3) a RNN-based fingering assessment, which evaluates how well the guitarist performs. Our previous works include (1) fingertips tracking algorithm using temporal grouping and extended particle filter (2) guitar neck tracking algorithm based on RANSAC and perspective transform; (3) guitarist fingering assessment using Support Vector Regression
◆Published literature
〇Journal
[1] Zhao, WANG, Jun OHYA. A Fingertips Tracking Algorithm for Guitarist Based on CNN-Segmentation and Extended Particle Filter IIEEJ Journal Paper (Coming Soon)
〇International Conference
[1] Zhao, W., & Ohya, J. Detecting and Tracking the Guitar Neck Towards the Actualization of a Guitar Teaching-aid System. In The… international conference on advanced mechatronics: toward evolutionary fusion of IT and mechatronics: ICAM: abstracts Vol. 2015, No. 6, pp. 187-188. 2015
[2] WANG, Zhao, and Jun OHYA. Tracking the Guitarist’s Fingers as Well as Recognizing Pressed Chords from a Video Sequence. Electronic Imaging 2016.15 : 1-6. 2016
[3] Wang, Z. and Ohya, J., 2016, November. Fingertips Tracking Algorithm for Guitarist Based on Temporal Grouping and Pattern Analysis. In Asian Conference on Computer Vision (pp. 212-226). 2016
[4]Gao, S., Tatematsu, N., Ohya, J., & Wang, Z. (2015, December). Estimating Clean-up Robots’ Mechanical Operations of Objects Using a SLAM Based Method. In The… international conference on advanced mechatronics: toward evolutionary fusion of IT and mechatronics: ICAM: abstracts Vol. 2015, No. 6, pp. 249-250. 2015
[5] Keishi Nishikawa, Zhao Wang, Jun Ohya, Takashi Matsuzawa, Kenji Hashimoto, and Atsuo Takanishi ,“Automatic, Accurate Estimation of the position and pose of a Ladder in 3D Point Cloud”, The 5th IIEEJ International Workshop on Image Electronics and Visual Computing
〇Domestic Conference
[1] Zhao WANG, Ye LI, Jing YAN, Jun OHYA, Study of Detecting the Frets and Strings on the Neck of the Guitar from RGBD Images towards the Actualization of an Autonomous Guitar Teaching System, Section D7,Media Computing Conference 2014
[2] Zhao WANG, Jun OHYA, Study of a Vision Based Method for Checking the Position of Each Finger of Guitar Players – Towards the Actualization of an Autonomous Guitar Chord Teaching System -,Section D-11-7,IEICE Society Conference 2015.
[3] Zhao WANG, Jun OHYA, A Method for Tracking Guitar Neck and Fingertips: Necking Tracking Robust against Occlusions Based on Geometry Analysis and Fingertips Tracking Based on Temporal Probability Map ,FIT Information and Science Technology Forum 2015.