Touchless Selection Schemes for Intelligent Automotive User Interfaces With Predictive Mid-Air Touch

Introduction

Predictive touch is an emerging HMI technology that employs a probabilistic Bayesian framework and novel algorithms to predict the interface component the user intends to select, notably early in the pointing-selection task (Ahmad, Murphy, Godsill, Langdon & Hardy, 2017). It infers the user intent from the available freehand pointing movements in 3D, for example from gesture trackers which are increasingly becoming commonplace in vehicles (Zhang & Angell, 2014; Ohn-Bar & Trivedi, 2014), and potentially other available sensory data such as eye-gaze. The pointing-selection task is simplified and expedited by the system via applying a suitable selection facilitation scheme. This can significantly reduce the effort and distractions associated with using in-vehicle displays whilst driving (Jæger, Skov & Thomassen, 2008). Figure 1 depicts the system block diagram including the sensory data sources utilized by a Bayesian predictor to estimate, early in the pointing task, the probability of each of the selectable interface items being the intended on-screen destination. Predictive touch was originally developed to mitigate the effects of perturbations on the user input, for example vibrations and accelerations due to the road and driving conditions. They can have a detrimental impact on the performance of interactive displays, such as touch screens (Goode, Lenné & Salmon, 2012; Ahmad et al., 2015), which often act as the gateway to control in-vehicle infotainment systems and are an integrated part of modern vehicles (Harvey & Stanton, 2016).

With predictive touch, which is not a pointing/ray-casting or conventional “symbolic” gestures recognition solution; as will be discussed, the user does not need to physically touch a display to select an interface component. Therefore, this touchless technology can not only improve the usability and performance of in-vehicle interactive displays, but it also provides the means to interact via the intuitive free-hand pointing with new automotive display technologies that do not have a physical surface such as head-up displays and 3D projections (Bark et al., 2014; Broy et al., 2015). It also offers additional design flexibilities in terms of the display placement and size which is otherwise limited by the reach of the driver/passenger. This can promote inclusive design practices by tailoring the display operation to the user capabilities by adequately configuring the “software-based” intent prediction algorithms and pointing facilitation schemes.

Figure 1.

System block diagram showing an in-car touchscreen, partial 3D pointing-finger trajectory (black solid line) available at the current time instant t_k with tracked locations (crosses), future pointing trajectory (red dotted line) and intended on-screen destination (red circle)

In this paper, we address the problem of identifying the most suitable scheme for facilitating the selection of the predicted interface component whilst driving. This human factor aspect is crucial for the deployment of the predictive touch technology in automotive. The selection facilitation functionality belongs to the Facilitation Scheme module in Figure 1. It involves altering the interface, for instance highlighting the predicted item, and then triggering the selection action. The users receive visual feedback on their input as the interface typically changes and the Graphical User Interface (GUI) page updates with each selection action. Other feedback modalities can be explored, e.g. audible (Ahmad et al., 2016b) or mid-air haptic (Shakeri, Williamson & Brewster, 2018), however this is outside the scope of this article.