Feature Extraction of Dialogue Text Based on Big Data and Machine Learning

Introduction

With the continuous development of computer and internet technology, information and data are exploding. How to effectively use these huge and disorderly collections of information, classify it accurately and efficiently, and summarize valuable information derived from it has become an urgent problem (Kang & Youn, 2020). In response, research into text classification technology has come into being. Text classification is the sorting of each piece of text into a predefined category, so that users can quickly and conveniently obtain the required information by some means of searching while reading the text. (Barbantan et al., 2016). Today’s daily information exchange often hides some clues or evidence, and we often lack the means of extracting useful knowledge from this information often lacks means (Bharti & Singh, 2015). For example, content collected from social media can be used to extract information about a person’s interests, behavior, and habits; information about the internal communication of a company can be extracted from its e-mail; information about user behavior can be extracted from a website’s network traffic. These text information analyses are based on text mining, text analysis technology, and classification technology in machine learning. Text mining is a technology for extracting useful information from text. It can be used to find patterns and rules in text. Text analysis technology, which is also designed for obtaining information from text, can be used to analyze the semantics and syntax of text. Classification technology in machine learning is a technology for extract useful information from data. It discerns patterns and rules in data, so as to extract useful information.

In both document-based processing and text processing, it is necessary to analyze the text itself (Karthikeyan, 2019). The information content and format documents are diverse and complex. Many data in the database are structured, such as relational data and data warehouse data, are structured (Lee, 2019). But some of the data in the document are semi-structured, and more are unstructured. Unstructured data can’t be processed directly, so text data must be converted into data form that can be recognized by the computer by using the established corresponding data model (Oskouie, 2014). The quantity and form of online data have changed correspondingly; while people are faced with these rich resources, there is a contradiction between people’s ever-growing demand for information and the increasing difficulty of access to the required information (Jiang et al., 2022). It is a great challenge in the field of information science to organize and mine these data effectively in order to find the information users need more quickly, accurately, and comprehensively. In this paper, a dialogue text feature extraction model based on big data and machine learning is constructed, which transforms the high-dimensional space of text features into a low-dimensional space that is easy to process, so that the best feature words can be selected to represent the document set. In high-dimensional space, the distribution of data is relatively complex, and the computational and storage costs will also increase accordingly. On the contrary, in low dimensional spaces, the number of feature vectors is relatively small. At the same time, in low dimensional space, we are more likely to discover hidden relationships and information in text data, and low-dimensional space, is also easier to visualize and intuitively display.