Magnet-Based Around Device Interaction for Playful Music Composition and Gaming

Introduction

The recent advent of Around Device Interaction (ADI) (Butler et al., 2008) has expanded the interaction space on mobile devices to allow 3D motion gesture interaction around the device, with opportunities for playful music composition and gaming only now taking shape. Using sensors embedded in mobile devices (e.g., (magnetic) compass (Ketabdar et al., 2010), IR distance sensors (Kratz & Rohs, 2009), users can now take advantage of the extra interaction space that their mobile device affords, for leisure and entertainment (Davenport et al., 1998).

ADI can be useful for small tangible/wearable mobile or controller devices (e.g., mobile phones or wrist watches) (Ketabdar et al., 2010). In such devices, it is extremely difficult to operate small buttons and touch screens. By expanding the interaction space around the device, ADI can aid the user in such cases, alongside situations when the device screen is not in line of the user’s sight. The ADI paradigm can allow coarse movement-based gestures made in the 3D space around the device to be used for sending different interaction commands such as controlling a portable music player (changing sound volume or music track), zooming, rotation, etc. For mobile phones, it can be also used for dealing with incoming calls (e.g., accepting or rejecting a call). However, ADI need not be limited to use-cases comprising user situational impairments (Ashbrook et al., 2011) or substituting for basic touchscreen tasks (Baudisch & Chu, 2009), but can complement touchscreen interactions with 3D gestures to allow natural, playful interactions in music composition (Ketabdar et al., 2012, 2011) and gaming.

Magnet-based ADI is a novel interaction technique for mobile devices allowing gestural interaction in the whole 3D space around the device¹. Here, moving a properly shaped magnetic material in hand (e.g. bar shaped, pen, ring) is used to influence the internally embedded compass (magnetometer) sensor in mobile devices by different 3D gestures, hence allowing for touchless interaction around the device. Since the interaction here is based on magnetic fields (which can pass through the hand or clothes, and not depending on users’ line of sight), the space at the back and side of device can also be efficiently used for interaction. This technique does not require extra sensors on current smartphones. For these smartphones, it is only necessary to have a properly shaped magnet as an extra accessory. While this can be seen as a limitation of such systems, as will be shown later the use of a magnet allows for a more natural interaction with music related apps.

In this paper, we look closely at how magnet-based ADI can be used in a playful context, to facilitate natural interaction for music composition and gaming amongst both musically-trained and non-musically trained users. Using three musical applications developed under the ADI paradigm (Air Disc-Jockey, Air Guitar, Air GuitaRhythm), we investigate the potential of ADI for playful interaction, in order to gain insight into the acceptability and naturalness of ADI by users who wish to casually engage in playful mobile music composition. Under this investigation, our primary goal is to explore novel methods using mobile technology to entertain users. The rest of the paper is structured as follows: first we provide a review of related work, followed by our research questions and our magnet-based ADI framework. We then present our study design and methods, give our results and discuss them, and finally hint at future work and conclude.