Article Preview
Top1. Introduction
Evaluation of recommender systems reveal key issues that demand attention, especially when business enterprises target methods to stay relevant in business. Designers of recommenders have been mainly engaged in evaluating system-centric factors neglecting user-centric factors which keep customers around for more profit (Leino, 2014; Lin, Wang, Wang, & Lu, 2013; Knijnenburg, 2012). Inadequate evaluation of user-centric factors such as buying decisions, user experience and user interactions can lead into endorsement of inferior recommender systems. This can also create a routine of designing inferior recommenders which lacks the quality and focus required to increase revenue and user experience.
This paper investigates the adequacy, or lack of it, in recommender systems evaluation methods. Recommender systems evaluation methods mainly comprises of user-centric evaluation and system-centric evaluation. It is of paramount importance that these evaluation methods are adequately assessed when evaluating a recommender system. Failure to assess user-centric factors gives rise to neglecting important factors that cause rejection of a recommender system (Knijnenburg, 2012). Published work has shown that there is little correlation between algorithmic accuracy and user satisfaction. User-centric evaluation methods have not been employed to the same degree as system-centric approaches (Leino, 2014; Cremonesi, Garzotto, & Turrin, 2013; Konstan & Riedl, 2012; Knijnenburg, 2012). It is thus imperative to expand from recommender algorithms, to user interactions, decision-making processes and overall experience which are the essence of user-centricity (Tintarev et al., 2014). The main aim of the research presented is to use a proposed user-centric evaluation conceptual framework to prove through literature review that current evaluation methods of recommender systems lack the adequacy needed to fully assess user-centric factors, resulting in low levels of user interaction and experience. This will provide a basis for further research into the development of algorithms and models that can address this problem.
An integrative review approach was used which draws conclusions from related articles and report on current state of affairs on user-centricity through a characterization and a conceptual framework which makes it a primary research . Terms were elicited from the research problem a search was conducted from August 2014 to December 2015, specifically on articles published between January 2009 to December 2015. For inclusion criteria, all types of recommender systems ranging from eLearning, to news and social networking were included whereas for exclusion criteria, only evaluations of recommender systems were included. Results from this paper show that there are more system-centric than user-centric methods and the existing user-centric evaluation methods are not assessing buying decisions, user experience and user interactions. Even if new developed recommender algorithms have high precision in terms of prediction, they will face resistance if they do not address the above mentioned factors. User-centric methods lack practical approaches, which make them difficult to implement in a computational environment.