Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment

Atsushi Mizumoto; Natsuko Shintani; Miyuki Sasaki; Mark Feng Teng

doi:10.1016/j.rmal.2024.100116

Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment

Atsushi Mizumoto, Natsuko Shintani, Miyuki Sasaki, Mark Feng Teng

Faculty of Languages and Translation

Research output: Contribution to journal › Article › peer-review

4 Citations (Scopus)

Abstract

This study explores the effectiveness of ChatGPT as a tool for evaluating linguistic accuracy in second language (L2) writing, situated within the complexity, accuracy, and fluency (CAF) framework. By using the Cambridge Learner Corpus First Certificate in English (CLC FCE) dataset, an error-tagged learner corpus, it compares ChatGPT's performance to human evaluators and Grammarly in assessing errors or accuracy rates across 232 writing samples. The findings indicate a strong correlation between ChatGPT's assessments and human accuracy ratings, demonstrating its precision in automated assessments. In comparison to Grammarly, ChatGPT shows a closer alignment with human judgments and students’ writing scores. Thus, ChatGPT can be a potential tool for enhancing efficiency in L2 research and L2 writing pedagogy.

Original language	English
Article number	100116
Journal	Research Methods in Applied Linguistics
Volume	3
Issue number	2
DOIs	https://doi.org/10.1016/j.rmal.2024.100116
Publication status	Published - Aug 2024

Keywords

ChatGPT
Grammarly
Learner corpora
Linguistic accuracy

Access to Document

10.1016/j.rmal.2024.100116

Cite this

@article{50e43c0312dc4c93827bd118be528aa5,

title = "Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment",

abstract = "This study explores the effectiveness of ChatGPT as a tool for evaluating linguistic accuracy in second language (L2) writing, situated within the complexity, accuracy, and fluency (CAF) framework. By using the Cambridge Learner Corpus First Certificate in English (CLC FCE) dataset, an error-tagged learner corpus, it compares ChatGPT's performance to human evaluators and Grammarly in assessing errors or accuracy rates across 232 writing samples. The findings indicate a strong correlation between ChatGPT's assessments and human accuracy ratings, demonstrating its precision in automated assessments. In comparison to Grammarly, ChatGPT shows a closer alignment with human judgments and students{\textquoteright} writing scores. Thus, ChatGPT can be a potential tool for enhancing efficiency in L2 research and L2 writing pedagogy.",

keywords = "ChatGPT, Grammarly, Learner corpora, Linguistic accuracy",

author = "Atsushi Mizumoto and Natsuko Shintani and Miyuki Sasaki and Teng, {Mark Feng}",

note = "Publisher Copyright: {\textcopyright} 2024 The Author(s)",

year = "2024",

month = aug,

doi = "10.1016/j.rmal.2024.100116",

language = "English",

volume = "3",

journal = "Research Methods in Applied Linguistics",

issn = "2772-7661",

publisher = "Elsevier B.V.",

number = "2",

}

TY - JOUR

T1 - Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment

AU - Mizumoto, Atsushi

AU - Shintani, Natsuko

AU - Sasaki, Miyuki

AU - Teng, Mark Feng

PY - 2024/8

Y1 - 2024/8

N2 - This study explores the effectiveness of ChatGPT as a tool for evaluating linguistic accuracy in second language (L2) writing, situated within the complexity, accuracy, and fluency (CAF) framework. By using the Cambridge Learner Corpus First Certificate in English (CLC FCE) dataset, an error-tagged learner corpus, it compares ChatGPT's performance to human evaluators and Grammarly in assessing errors or accuracy rates across 232 writing samples. The findings indicate a strong correlation between ChatGPT's assessments and human accuracy ratings, demonstrating its precision in automated assessments. In comparison to Grammarly, ChatGPT shows a closer alignment with human judgments and students’ writing scores. Thus, ChatGPT can be a potential tool for enhancing efficiency in L2 research and L2 writing pedagogy.

AB - This study explores the effectiveness of ChatGPT as a tool for evaluating linguistic accuracy in second language (L2) writing, situated within the complexity, accuracy, and fluency (CAF) framework. By using the Cambridge Learner Corpus First Certificate in English (CLC FCE) dataset, an error-tagged learner corpus, it compares ChatGPT's performance to human evaluators and Grammarly in assessing errors or accuracy rates across 232 writing samples. The findings indicate a strong correlation between ChatGPT's assessments and human accuracy ratings, demonstrating its precision in automated assessments. In comparison to Grammarly, ChatGPT shows a closer alignment with human judgments and students’ writing scores. Thus, ChatGPT can be a potential tool for enhancing efficiency in L2 research and L2 writing pedagogy.

KW - ChatGPT

KW - Grammarly

KW - Learner corpora

KW - Linguistic accuracy

UR - http://www.scopus.com/inward/record.url?scp=85193976426&partnerID=8YFLogxK

U2 - 10.1016/j.rmal.2024.100116

DO - 10.1016/j.rmal.2024.100116

M3 - Article

AN - SCOPUS:85193976426

SN - 2772-7661

VL - 3

JO - Research Methods in Applied Linguistics

JF - Research Methods in Applied Linguistics

IS - 2

M1 - 100116

ER -

Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this