Refine
Document Type
- Conference Proceeding (2)
- Article (1)
- Part of a Book (1)
Language
- English (4)
Has Fulltext
- no (4)
Is part of the Bibliography
- yes (4)
Keywords
- Augmented Reality (2)
- AR based assistance (1)
- Annotation (1)
- Assistance (1)
- Authoring (1)
- Cognitive Impairments (1)
- Human–computer interaction (1)
- Industry 4.0 (1)
- Listener gaze (1)
- Mixed Reality (1)
Institute
Quantifying the interplay of gaze and gesture in deixis using an experimental-simulative approach
(2018)
Gaze and gestures have been studied qualitatively, e.g., by Kendon and others (Kendon, 1990; McNeill, 1992; Kendon, 2004; McNeill, 2006). A quantitative assessment of gaze and gestures in dialogue, in particular regarding precise orientations, positions and timings, however, has only been possible with the advent of advanced measuring technologies, such as motion capturing or eye tracking. Especially in dynamic natural environments, when interlocutors are concerned with their surrounding three-dimensional environment, a precise three-dimensional reconstruction of the set-up is required to analyze the produced multimodal utterances.
In this article we review several of our past projects with a focus on our experimental-simulative approach in which we combine state-of-the-art tracking technologies with 3D representations and computer simulations to test different hypotheses in the context of deixis in human-human interaction.
Referential success is crucial for collaborative task-solving in shared environments. In face-to-face interactions, humans, therefore, exploit speech, gesture, and gaze to identify a specific object. We investigate if and how the gaze behavior of a human interaction partner can be used by a gaze-aware assistance system to improve referential success. Specifically, our system describes objects in the real world to a human listener using on-the-fly speech generation. It continuously interprets listener gaze and implements alternative strategies to react to this implicit feedback. We used this system to investigate an optimal strategy for task performance: providing an unambiguous, longer instruction right from the beginning, or starting with a shorter, yet ambiguous instruction. Further, the system provides gaze-driven feedback, which could be either underspecified (“No, not that one!”) or contrastive (“Further left!”). As expected, our results show that ambiguous instructions followed by underspecified feedback are not beneficial for task performance, whereas contrastive feedback results in faster interactions. Interestingly, this approach even outperforms unambiguous instructions (manipulation between subjects). However, when the system alternates between underspecified and contrastive feedback to initially ambiguous descriptions in an interleaved manner (within subjects), task performance is similar for both approaches. This suggests that listeners engage more intensely with the system when they can expect it to be cooperative. This, rather than the actual informativity of the spoken feedback, may determine the efficiency of information uptake and performance.
Augmented Reality (AR) based assistance has a huge potential in the context of Industry 4.0: AR links digital information to physical objects and processes in a mobile and, in the case of AR glasses, hands-free way. In most companies, order-picking is still done using paper lists. With the rapid development of AR hardware during the last years, the interest in digitizing picking processes using AR rises. AR-based guiding for picking tasks can reduce the time needed for visual search and reduce errors, such as wrongly picked items or false placements.
Choosing the best guiding technique is a non-trivial task: Different environments bring their own inherent constraints and requirements. In the literature, many kinds of guiding techniques were proposed, but the majority of techniques were only compared to non-AR picking assistance.
To reveal advantages and disadvantages of AR-based guiding techniques, the contribution of this paper is three-fold: First, an analysis of tasks and environments reveals requirements and constraints on attention guiding techniques which are condensed to a taxonomy of attention guiding techniques. Second, guiding techniques covering a range of approaches from the literature are evaluated in a large-scale picking environment with a focus on task performance and on factors as the users' feeling of autonomy and ergonomics. Finally, a 3D path-based guiding technique supporting multiple goals simultaneously in complex environments is proposed.
Augmented Reality (AR) is a promising technology for assistance and training in work environments, as it can provide instructions and feedback contextualised. Not only, but especially impaired workers can benefit from this technology. While previous work mostly focused on using AR to assist or train specific predefined tasks, "general purpose" AR applications, that can be used to intuitively author new tasks at run-time, are widely missing.
The contribution of this work is twofold: First we develop an AR authoring tool on the Microsoft HoloLens in combination with a Smartphone as an additional controller following considerations based on related work, guidelines and focus group interviews. Then, we evaluate the usability of the authoring tool itself and the produced AR instructions on a qualitative level in realistic scenarios and gather feedback. As the results reveal a positive reception, we discuss authorable AR as a viable form of AR assistance or training in work environments.