Author: Hui Su
Director, Cognitive and Immersive Systems Lab
This post originally appeared on the IBM Research blog.
A student learning to speak Mandarin wanders into a marketplace on the streets of China on a sunny summer afternoon. Before long, two vendors approach and begin hawking products, trying to outbid one another. The student must now grasp what’s being said and formulate an appropriate response using proper pronunciation to avoid being misunderstood. It’s a challenging, yet common, scenario for anyone trying to learn a new language by interacting with native speakers and immersing themselves in a foreign culture.
Fortunately, the student in this case is able to pause the unfolding scenario to check the accuracy and tone of her planned response. That’s because, even though the student appears to be surrounded by the sights and sounds of a typical marketplace in China, she’s actually standing in the AI-enhanced, state-of-the-art Cognitive Immersive Room (CIR) on Rensselaer Polytechnic Institute’s (RPI) campus in Troy, N.Y. and the vendors are actually avatars on a 360-degree panoramic display.
Conversational AI, narrative generation, spatial context awareness, and gesture and facial landmark recognition technologies form the foundation of RPI’s interactive new class, “AI-Assisted Immersive Chinese.” Working with IBM Research, the university has created a first-of-its-kind, six-week credit-bearing course in Mandarin taught in the school’s Cognitive and Immersive Systems Laboratory (CISL). CISL is an ongoing collaboration between the university and IBM Research, and part of our AI Horizons Network to accelerate the research and application of AI, machine learning, natural-language processing and related technologies.
Our goal is to leverage AI to provide students with a culture-rich learning environment that feels almost like the real thing and gives students meaningful tasks to perform. AI-Assisted Immersive Chinese has already begun to offer those opportunities for RPI students in its inaugural semester this summer.
Since piloting the technology last summer, IBM Research and RPI have added a number of more advanced features, including the ability for multiple students to interact with multiple avatars in order to complete a daily task, a more flexible dialogue system for language teaching, the ability for learning assessment and assistance, and more immersive language learning games. Students can practice Mandarin in a number of different situations, including marketplace negotiations, living and studying on the campus of a university, survival phrases at an airport, visiting a city and ordering a meal in a restaurant. CISL’s Mandarin Project developed the class’s cognitive and immersive language learning environment specifically for teaching non-native speakers. The idea is to get language students out of the classroom and put them in a more realistic learning environment.
IBM Watson’s speech recognition, natural language understanding and conversation technologies for English and Chinese enable the students to ask the simulation program for help with pronunciation and clarify how to perform a particular task. A technology called pitch-tone contour analysis evaluates pronunciation and provides immediate audio and visual feedback comparing students’ pronunciations against those of native speakers.
In each virtual scenario, students use head movement and position to indicate to whom they are talking. That movement is tracked by cameras and other sensors placed throughout the immersive environment. A spatial context system tracks a person’s proximity to different areas of the panoramic screen, initiating dialogue when it detects a person is nearby.
It’s often said that the best way to learn a foreign language is to immerse oneself in an environment where it’s spoken. If students are to improve their conversational skills, they must have many opportunities to practice speaking. That is especially true of a complex language like Mandarin, which has four different tones that must be mastered in order for a speaker to be properly understood.
Our new AI-assisted course can help overcome those challenges, delivering a rich learning environment to RPI students—no travel required.
