How different judgment strategies affect accuracy in generative AI text discrimination: BERT-based clustering and hierarchical Bayesian logistic regression.

Authors

Wataru Zaitsu, Mingzhe Jin, Shunichi Ishihara, Satoru Tsuge, Mitsuyuki Inaba

Published in

Frontiers in artificial intelligence. Volume 9. Pages 1918939. Epub Oct 09, 2026.

Abstract

Large language models (LLMs)-generated texts become increasingly similar to human writing. The purpose of this study was to examine (1) how individuals distinguish between human-written and LLM-generated texts, focusing on their judgment strategies; (2) the relationship between judgment strategies and detection accuracy; and (3) the interaction effects of gender, age, and frequency of large language model (LLM) use.
Participants were asked to judge whether the given texts (e.g., public comments) were written by LLMs or humans and to explain their reasoning using open-ended questions. The public comments were written by humans and generated using seven LLMs (ChatGPT [4o and o1], Claude3.5, Gemini, Microsoft Copilot, Llama3.1, and Perplexity). First, we classified participants' explanations of reasoning related to judgments into types using BERT-based clustering (i.e., Sentence-BERT and hierarchical cluster analysis).
BERT-based clustering suggested four interpretable clusters. Based on their characteristic patterns, we interpreted these clusters as four hypothetical judgment strategy types and tentatively proposed that they form a conceptual continuum: (1) Unsubstantiated intuition, (2) Stylistic-Cue, (3) Syntactic-Cue, and (4) Higher-Order Pattern Recognition (HOPR). A hierarchical Bayesian logistic regression analysis revealed that the detection accuracy varied across judgment types, with the HOPR type achieving relatively higher performance than the other strategy types, although its detection accuracy remained near the chance level. Additionally, we identified interaction effects between strategy type and gender; only female participants who used the HOPR strategy showed lower detection accuracy than male participants, although the qualitative interpretation of this difference warrants further empirical investigation.
These findings emphasize the role of judgment strategies and individual differences in discriminating between human and LLM-generated content.

PMID:
42920589
Bibliographic data and abstract were imported from PubMed on 24 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.