Experimenting Language Identification for Sentiment Analysis of English Punjabi Code Mixed Social Media Text

Literature Review

Language identification is an essential prerequisite for automatic text processing. It is a preprocessing task for computational tasks for code switching and is considered as almost a solved problem for monolingual text in which n-gram approaches, character encoding detection or stop word lists can reach up to 100% accuracy. Researchers use simple dictionary method or machine learning techniques such as Naive Bayes, Support Vector Machines (SVM), and Conditional Random Forests (CRF), Convolutional Neural Networks (CNN) etc. Language systems fail due to style of writing and brevity of texts. Language detection is a difficult and unsolved problem due to Anglicism, code mixing, code switching, lexical borrowings (all terms being used interchangeably).

Beesley (1988) developed a prototype for language identifier of online text based on cryptanalysis. (Cavnar & Trenkle, 1994) used character n-gram frequency lists to determine the language of a new piece of text in the underlying algorithm of TextCat, an automatic LID system developed by vanNoord. The results were reported are 99.8% accuracy for language models of more than 300 n-grams.