Article Preview
Top1. Introduction
In the field of pattern recognition, it is well known that the variance and diversity of input data poses great challenge to performance assessment, since a system may achieve quite different performances on different test data. This makes it difficult to assess the overall performance of a system considering all possible test data and to compare the performances between different systems. Recently, while many pattern wrecognition systems, e. g. speech recognition systems achieve real-world application, the evaluation of performance on all possible data in real-world application becomes a major challenge, and the key problem of the challenge is the design and collection of test data.
Currently, a popular approach for assessing the performance of a system is the evaluation scheme (Ajot and Fiscus, 2009; Vandecatseye et al., 2004; Deshmukh et al., 1996; Douglas and Janet, 1992), in which test database is collected and all systems are tested with the same data. Though this makes comparison possible, the performances obtained are still highly dependant on the test data. Most test data are collected randomly or arbitrarily, so they are not representative of all possible data and the performances are not representative, either.
Some researchers noted that data properties (e. g. speaker gender, age, or dialect for speech data) may affect system performance and tried to arrange some basic data properties. For example, Deshmukh et al. (1996) and Douglas and Janet (1992) added noises to the test data at SNRs of 10dB, 16dB and 22dB. Pearce and Hirsch (2000) and Nakamura et al. (2003) added 8 types of noise at 5 SNR levels. However, the data properties and their values are all decided arbitrarily, which cannot ensure the test data are representative of data in real-world application. Some researchers (Steeneken et al., 1989; Nagorski et al., 2002) automatically select test data from a large data set using acoustic features. Although the selected data is representative, they are highly dependent on the large data set used. If the large data set is biased, the selected data will be biased, too.
In this paper, the PIC-TDD (Performance Influencing Class based Test Data Designing) framework is presented. Under the framework, performance on the population of all possible data is estimated to analyze the system. By means of the analysis, difference in performance caused by different test data can be avoided, performance in real application can be predicted, and comparison between different systems tested with different data can be easily realized. The PIC-TDD framework is applied to the analysis of a broadcast news segmentation system. Experimental results were given, which verified the effectiveness of PIC-TDD.
The rest of the paper is organized as follows. In Section 2, statistical description of evaluation are presented, which is the basis of the theory of PIC-TDD. In Section 3, the theory and procedure of PIC-TDD are presented. In Section 4, we describe the application of PIC-TDD to test data designing of a broadcast news segmentation system. Experimental results and analysis for the application in broadcast news segmentation are also given in Section 4. Finally, in Section 5, conclusions are drawn and future work is discussed.