Construction of a Bi-Modal Database for a Barrier-Free Teaching System

Introduction

According to a 2010 survey of disability in China, it is estimated that the number of people with deafness was approximately 20,540,000, which ranked second in all types of disability. It is therefore important part that China carries out hearing impaired education and teaching research in development of special education (Yan jiaosheng, 2014).

Computerized speech-to-text technology used in hearing-impaired education was first put forward by Stuckless in one of his papers in 1981 (Furui, 2005). With IBM's strong technical support, Saint Mary's University Canada, set up the Barrier-Free Teaching Union to make speech recognition more widely applied.

In China, the earliest study about lip-reading was started by Yao HongXun from Harbin Institute of Technology. She used an extraction method of lip color filter based on the features of the lip to Identify five vowels. The lip-reading recognition in this method can reach above 90%. The Institute of Acoustics of the Chinese Academy of Sciences established the first Chinese speech dual-mode database by analyzing the structure of similar databases at home and abroad. The database is combined with the characteristics of Chinese pronunciation using a selected corpus of all the Chinese initials and finals. To an extent, it provided help for later researchers in acquisition and solution of the visual information. Moreover, Wuyi University, Dalian University of Technology, Zhejiang University, Southeast University and other institutions have also committed to lip reading study (Wudi, 2015). However, there is no special study of the technology used in teaching, especially in teaching hearing-impaired students (Jing, 2007). This leads to the status quo that the recognition ability of speech recognition software used in teaching is good, but the recognition of professional vocabularies is more difficult and makes errors frequently.

In the Chinese language environment, the use of dual mode speech recognition technology with audio and video combined for the teaching system (shown in Figure 1) can improve the recognition rate and robustness of the system. What we should understand is that the training process of speech recognition is a statistical process. In order to ensure the accuracy of the model, we require a basis of substantial audio and video data. In the field of pure speech recognition, a lot of standard speech corpus provide the baseline information, which plays a very important role in the development of speech recognition. However, in the field of audio visual, due to its application in recording different requirements in different areas, there is no such comprehensive and systematic database. Therefore, this paper introduces the construction process of an audio and video database in such a system.

Figure 1.

Structure of barrier-free teaching system

Dual-Mode Speech Recognition

The dual-mode speech recognition system with audio and video is based on an auditory single modal speech recognition system, adding a visual subsystem. This subsystem captures the facial images of speakers, then positions faces and other main features to extract the visual features associated with pronunciation, and inputs them to a recognizer with acoustic features. A bimodal speech recognition system has better performance and higher robustness, especially when there are various kinds of noise in the application environment, so it is more suitable for being applied in the actual environment.