CaptainA self-study mobile app for practising speaking : task completion assessment and feedback with generative AI

Abstract

We introduce the CaptainA mobile app, designed to meet the needs of second language (L2) learners engaged in self-study of Finnish, with potential applicability to other languages. Our app can provide automatic speaking assessment (ASA) of task completion in picture-based tasks, along with grading explanations and corrective feedback. It can also automatically generate pictures for visual tasks, providing users with unlimited practice opportunities. The mobile app is based on our framework that combines visual natural language generation (NLG), automatic speech recognition (ASR), and prompting large language model (LLM) for low-resource language. Our goal is to promote the development of next-generation speech-based computer-assisted language learning (CALL) systems capable of providing automatic scoring with feedback for learners, even when minimal speech data of L2 learners is available. While the mobile app demonstration is designed for Finnish, the app can also be tested in English.