BioVisualSpeech - An Interactive Platform for Speech Therapy with Visual Bio-feedback
Apr 2016 - Mar 2020
This long term project is the continuation of the VisualSpeech project. This project aims to research natural and multimodal interaction mechanisms for providing bio-feedback in speech therapy through the use of serious (computer) games. This project proposes a platform for speech therapy that has two main goals: the first goal is to keep the patients engaged in the exercises in order to have more fruitful sessions. For this, the platform will include a gamified interactive environment that uses visual stimuli, and a reward system that motivates children to keep exercising. More specifically, the environment provides instantaneous bio-feedback, replacing other tools such as a mirror, and assist the therapist during the session with audio-visual recordings and annotations of the session. While in the earlier VisualSpeech project the serious game has the therapist judge the correctness of the speech pronunciation, here we will include the automatic speech recognition of the child’s productions. The second goal aims at providing the therapist with a toolset to plan the course of the ongoing therapy session. For this, the platform will include an application for post-session analysis where a patient’s audio-visual recordings can be examined through browsing and searching, and a future session can be planned.
The novelty of this platform is the combination of visual bio-feedback and the gamification of the speech therapy. The gamified interactive environment is proposed as a key instrument to keep the patient engaged and reactive to the instant bio-feedback provided by the game.
