An LLM-driven Transcription Task for Mobile Text Entry Studies
Publication Type:
Conference PaperSource:
Proceedings of the International Conference on Mobile and Ubiquitous Multimedia, Association for Computing Machinery, New York, NY, USA (2024)ISBN:
9798400712838URL:
https://doi.org/10.1145/3701571.3701586Keywords:
Evaluation Methodologies, Large Language Models, Mobile Text Entry, Transcription tasksAbstract:
We explore a novel transcription task in mobile text entry research, presenting stimuli within LLM-generated conversational contexts to improve participant engagement and phrase memorability. We conducted two studies: an eye-tracking study examining participants’ attention when presented with conversational contexts alongside stimuli, and an experiment comparing LLM-generated and human-generated prompt-response pairs in transcription tasks, involving both high and low memorability stimuli. Key findings reveal that presenting conversational contexts improves recall for low memorability phrases and results in fewer uncorrected errors during transcription. No significant effects were observed on other basic text entry metrics, or participant subjective appraisals of engagement with the novel task, suggesting it can be used safely as an alternative to the traditional transcription task. We discuss the potential of LLMs in improving text entry evaluation methods, including generating diverse linguistic styles, emotionally loaded contexts, and even simulating entire evaluation processes. Our study highlights the need for systematic approaches to generate and evaluate LLM outputs for research purposes, and for proposing new metrics and evaluation methods.

