Can We Trust Them? A Critical Evaluation of AI-Generated Content Detection Tools
Keywords:
AI-generated text, Content detection, Accuracy, Limitations, Academic integrity, BiasAbstract
This explosion of AI-written text raises the issues of credibility and academic integrity. AI content detectors can be seen as possible solutions, but the question of their effectiveness is still open. This research performs an in-depth analysis of AI content detector accuracy and flaws. We test an array of detectors against a variety of human-written and AI-generated samples that have been modified using various obfuscation methods and are of different text genres. Findings showed notable issues with the detectors concerning specific obfuscation techniques and diversity of styles. We talk about the ethical aspects of the detector, such as possible biases and the risk of incorrect allegations. Finally, we want to point out the necessity of continuous critical evaluation together with the development of other tools like promoting critical thinking and process-level assessment.
References
S. Debora Weber-Wulff, A. Anohina-Naumeca, S. Bjelobaba, T. Foltýnek, J. Guerrero-Dib, O. Popoola, P. Šigut, and L. Waddington, “Testing of detection tools for AI-generated text,” International Journal for Educational Integrity, Vol.19, Issue.26, pp.1-39, 2023. https://doi.org/10.1007/s40979-023-00146-z
A. M. Elkhatat, K. Elsaid, and S. Almeer, “Testing of detection tools for AI-generated text,” International Journal for Educational Integrity, Vol.19, No.26, pp.1-39, 2023. https://doi.org/10.1007/s40979-023-00146-z
J. Otterbacher, “Why technical solutions for detecting AI-generated content in research and education are insufficient,” National Library Of Medicine, Vol.4, No.7, pp.1-9, 2023. https://doi.org/10.1016/j.patter.2023.100796
R. Sable, V. Baviskar, S. Gupta, D. Pagare, E. Kasliwal, D. Bhosale, and P. Jade, “AI Content Detection,” International Advanced Computing Conference, IACC, pp.267-283, 2024. https://doi.org/10.1007/978-3-031-56700-1_22
A. Harada, D. Bollegala, and N. P. Chandrasiri, “Discrimination of human-written and human and machine written sentences using text consistency,” International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp.41-47, 2021. https://doi.org/10.1109/ICCCIS51004.2021.9397237
C. Bail, L. Pinheiro, and J. Royer, “Difficulty Of Detecting AI Content Poses Legal Challenges,” Law 360, pp.1-4, 2023.
F. Ufuk, H. Peker, E. Sagtas, and A. B. Yagci, “Distinguishing GPT-4-generated Radiology Abstracts from Original Abstracts: Performance of Blinded Human Observers and AI Content Detector,” pp.1-7, 2023. https://doi.org/10.1101/2023.04.28.23289283
A. Bhattacharjee and H. Liu, “Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?,” ACM SIGKDD Explorations Newsletter, Vol.25, No.2, pp.14-21, 2024. https://doi.org/10.1145/3655103.3655106
V. Bellini, F. Semeraro, J. Montomoli, M. Cascella, and E. Bignami, “Between human and AI: assessing the reliability of AI text detection tools,” Current Medical Research and Opinion, Vol.30, No.3, pp.353-358, 2023. https://doi.org/10.1080/03007995.2024.2310086
C. Chaka, “Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools,” Journal of Applied Learning & Teaching, Vol.6, No.2, 2023.
H. Pandiyakumari S. and J. R., “A Survey on the Applications of Machine Learning in Identifying Predominant Network Attacks,” International Journal of Computer Science and Engineering, Oct., Vol.11, Issue.5, pp.16-22, 2023.
K. A. Okewale, I. R. Idowu, B. S. Alobalorun, and F. A. Alabi, “Effective Machine Learning Classifiers for Intrusion Detection in Computer Network,” International Journal of Computer Science and Engineering, Apr., Vol.11, Issue.2, pp.14-22, 2023.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.
