همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

Calibrating Large Language Models for High-Quality Automated Writing Feedback

نویسندگان :

Mohammad Sajjad Aghadadi¹ Heshaam Faili²

1- دانشگاه تهر 2- دانشگاه تهران

کلمات کلیدی :

Automatic Writing Feedback،Calibration،Pointwise Mutual Information،LLM

چکیده :

Automated assessment of IELTS Writing Task 2 remains a challenging problem due to the multidimensional nature of the scoring rubric and the difficulty of generating criterion-aligned feedback. This study proposes a PMI-based calibration framework for enhancing LLM-driven evaluation and feedback generation. Using a large dataset of 38,205 annotated IELTS responses, we compute feature-level PMI weights to adjust model outputs in two configurations: (1) a Static PMI model with a global calibration vector, and (2) a Contextual PMI model that dynamically selects calibration weights based on semantic clustering of input essays. Experimental results on a 1,000-sample test set show that PMI calibration consistently improves performance across all evaluation metrics, including Cosine similarity, BERTScore, rubric consistency, and LLM-as-a-judge ratings. Notably, the Contextual PMI model yields the strongest gains—for example, improving cosine similarity in the Coherence and Cohesion criterion from 0.462 (zero-shot) to 0.780. These findings highlight the effectiveness of PMI-driven calibration, particularly in context-adaptive settings, and demonstrate its potential as a lightweight, interpretable, and scalable approach for improving automated writing feedback.

لیست مقالات

لیست مقالات بایگانی شده

Resilience or Recession? A Longitudinal Analysis of NFT Market Dynamics and AI-Generated Value During the Crypto Downturn

Aria Shakoo - Amirhossein Ashrafian - Maedeh Mosharraf

A Multi-Dimensional Taxonomy and Practical Framework for MLOps Monitoring

Tina Hafezi - Sara Pour Faraj - Seyed Mohammad Amin Alemohammad - Niloofar Mirzaei Chahardeh

Development of 3D Neural Cellular Automata for Learning Spatio-Temporal Patterns

Yashar Rezazadeh Shahir - َُAkram Beigi

Reliability-Aware Prompting for Diabetic Retinopathy Classification using Vision-Language Models

Danial Zohourian - Dorsa Asgari - Sadegh Madadi - Hadi Farahani

Fibroglandular Tissue Classification in Breast MRI: A Comparative Study of Automated Decision Strategies

Meysam Khalaj - Arvin Arian - Ala Torabi - Nasrin Ahmadinejad - Masoumeh Gity - Seyedeh Nooshin Miratashi Yazdi - Mohammad Pooya Afshari - Melika Sadeghi Tabrizi - Hamid Soltanian-Zadeh

Enhancing Facial Emotion Recognition Using YOLO11 Classification on the AffectNet Dataset

Reza Nasiri - Seyed Enayat Alavi - Mohammad Javad Rashti

Soil Shear Strength Prediction Using Genetic-Optimized Boosting Machine Learning Models

Mohammadreza Ghadami - Ali Noorzad - Hamid Mohammadnezhad

AdaLayer-CAM: Adaptive Multi-Layer Method for Visualization and Explanation of CNNs

MohammadJavad GhaderiPoor - Latifeh PourMohammadBagher - Zahra Rahimi

Enhancing Single-Cell Transcriptomic Quality in Gastric Cancer: An Intersective Approach Combining Adaptive and Manual Quality Control

Fatemeh Khalaji-Pirbalouti - Modjtaba Emadi-Baygi

Examining the Role of Artificial Intelligence in Enhancing Educational Equity: A Systematic Review

Ali Rahmanipur - Shakila Mohammadi

بیشتر