An LLM-driven Transcription Task for Mobile Text Entry Studies

Publication Type:

Conference Paper

Source:

Proceedings of the International Conference on Mobile and Ubiquitous Multimedia, Association for Computing Machinery, New York, NY, USA (2024)

ISBN:

9798400712838

URL:

https://doi.org/10.1145/3701571.3701586

Keywords:

Evaluation Methodologies, Large Language Models, Mobile Text Entry, Transcription tasks

Abstract:

We explore a novel transcription task in mobile text entry research, presenting stimuli within LLM-generated conversational contexts to improve participant engagement and phrase memorability. We conducted two studies: an eye-tracking study examining participants’ attention when presented with conversational contexts alongside stimuli, and an experiment comparing LLM-generated and human-generated prompt-response pairs in transcription tasks, involving both high and low memorability stimuli. Key findings reveal that presenting conversational contexts improves recall for low memorability phrases and results in fewer uncorrected errors during transcription. No significant effects were observed on other basic text entry metrics, or participant subjective appraisals of engagement with the novel task, suggesting it can be used safely as an alternative to the traditional transcription task. We discuss the potential of LLMs in improving text entry evaluation methods, including generating diverse linguistic styles, emotionally loaded contexts, and even simulating entire evaluation processes. Our study highlights the need for systematic approaches to generate and evaluate LLM outputs for research purposes, and for proposing new metrics and evaluation methods.

Download full text: