INSPIRATION - Our hack, Lucid Drums, was motivated by our dedication to promoting music education and early childhood development. Notable learning scientists and psychologists, such as Lev Vygotsky, believe that children learn by experiencing their environments and making observations from the world around them. Carl Orff, one of the major figures in childhood music education, promotes the idea that music education should combine with movement, playing and improvisation. We believe that with VR technology, we can create a fantastic virtual reality environment that enables children to explore the world of music rhythmically and freely; through fantasy, we create a unique learning environment.
RHYTHM AUTO DETECTION - As humans, we can catch certain rhythms easily, but an artificial system is challenged to do so. Beat tracking may sound like a straightforward concept, but it’s actually an unsolved problem until now. We learned audio can be complex and noisy that this can befuddle an algorithm and lead it to produce false positives while detecting beats. I used Recurrent Neural Network to identify beats, but it was not giving us the desired output and predicted a lot of false positives as well. Therefore, we used neighboring methods to pick up the prediction that matched with the true beats in a song. Improved detection gave us the best/appropriate beats that nicely incorporate with each song.
HOW WE BUILD IT - Quest + Unity for VR gaming experience; Deep learning (RNN from madmom) for rhythm auto-generatioN; User/learner friendly design according to UX/learning design principles check the project detail at : https://devpost.com/software/lucid-drums and interview by CMU HCII department at: https://hcii.cmu.edu/news/2020/hcii-students-game-wins-most-creative-treehacks
