همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

A Multi-Layer Comparative Analysis of Human and Machine Reviewing Using Large Language Models

نویسندگان :

Mostafa Karimi Manesh¹ Mehrnoush Shamsfard²

1- دانشگاه شهید بهشتی 2- دانشگاه شهید بهشتی

کلمات کلیدی :

Peer Review Analysis

چکیده :

This study conducts a comparative analysis of the behavior of human reviewers and large language models in the peer-reviewing of scientific articles, aiming to systematically examine the cognitive, analytical, and linguistic dimensions of these two approaches. In the first step, a collection of 83 papers in the field of natural language processing and 2,379 human review comments was gathered, and corresponding machine-generated reviews were produced using the GPT-4 model. Subsequently, a four-layer annotation framework—covering article sections, evaluation aspects, reviewer functions, and comment severity—was designed to investigate behavioral and content-level differences between the two reviewing systems. The results indicate that the language model places greater emphasis on structural and technical components of the paper, such as methodology and data, and demonstrates strong performance in quantitative analysis; whereas human reviewers focus more on conceptual aspects such as novelty, comparison with prior work, and scientific impact. From a linguistic perspective, the language model tends to rewrite criticisms as constructive suggestions, while human reviewers generally adopt a more direct and critique-oriented style. The conceptual overlap between the two types of reviewing reaches up to 79% in technical sections and approximately 71% overall, reflecting a notable convergence between human and machine judgment. Key contributions of this study include the development of a “section-wise specialized reviewing” approach and the creation of a Persian-language scientific peer-review dataset. The findings suggest that an intelligent integration of human and machine reviewing can enhance the accuracy, fairness, and transparency of scientific evaluation and pave the way for more advanced automated reviewing systems.