Tactile Language Learning

Our primary input is an audio input captured through our laptop’s microphone. No pre-processing of data was performed before sending it to Wekinator. We used an open source sound classifier application (Source) that records the audio input and sends 1600 inputs at port 6448 using classification.

Our output type is determined by a classifier algorithm with 5 gesture types that will act as the five voice commands that Wekinator processes. The classifier model used 1600 inputs to produce 1 output, allowing for a more accurate analysis and classification of the sounds.

Our Processing code received the Wekinator output to move the blocks and form the Mandarin characters for 一 (yī，“1”) ，二 (èr，“2”), 三 (sān，“3” ), 口 (kǒu，“mouth”), or 日 (rì，“sun”) accordingly, serving as a three dimensional visualization of the characters. With further research and time to experiment, we would consider doing these alternative methods so our system can be simplified in terms of number of motors, and applied to languages other than just Mandarin Chinese.

In this project, we designed a tool to learn languages, specifically Mandarin Chinese. Learning Mandarin Chinese breaks each character down into its phoneme (sound pattern), intonation, and writing. Each character is pronounced in one of four tones, and stroke is written in a specific order and direction. Thus, we wanted to design an educational tool that helps users to get feedback to correctly pronounce words in Chinese, and see the correct order and direction of writing each stroke.

Users are able to speak a word into the laptop’s microphone, and if pronounced correctly, then see the blocks move to form the character. The order in which the blocks move will show the user how to write the character. Users can trace over the moving blocks to also learn the handwriting motion. Due to the scope of this project, we limited our examples to the Mandarin Chinese characters for 一 (yī，“1”)，二 (èr，“2”), 三 (sān，“3” ), 口 (kǒu，“mouth”), and 日 (rì，“sun”).

→ Fall 2022 (5 Weeks)

→ Cyber Physical Systems

→ Instructors: Dr. Rebecca Stewart, Charlotte Slingsby

→ Collaborators: Hanju Seo, Xinyi Zhou

→ Video & Cover Photo Credit: Marco Da Re & Kevin Lee

Every Chinese character consists of two components: pronunciation and meaning. How can we make language learning more tactile for students to enhance pronunciation and writing skills?