همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

Claim Verification in Persian: A Hybrid Pipeline with LLMs and Rationale-Quality Classification

نویسندگان :

Sara Bourbour¹ MohammadAli SeifKashani² Fatemeh Aali³

1- Tarbiat Modares University 2- Sharif University of Technology 3- Tarbiat Modares University

کلمات کلیدی :

claim verification،fakenews،rationale quality،stance detection

چکیده :

We present a Persian claim-verification pipeline that combines a fast deep stance classifier with an LLM-based verifier and a rationale-quality filter. Our corpus mixes private and public news articles paired with social-media posts. Annotation proceeds in two passes: (i) human stance labels (support, refute, unrelated) on post–article pairs; (ii) for pairs initially labeled as negative (refute/unrelated), an instruction-following LLM produces short Persian rationales which are then audited by humans for validity. Invalid rationales expose systematic failure modes and are used to augment training via class-conditional oversampling and rationale-aware multi-task learning. We train LaBSE- and ParsBERT-based classifiers and calibrate probabilities by temperature scaling. On a confidential test set, oversampling the negative class yields the strongest evidence classification for LaBSE (Accuracy 0.71, F1 0.80, Precision 0.68, Recall 0.97), while GPT‑4o provides higher recall but lower precision among LLM baselines. The rationale-quality head improves the reliability of LLM-driven decisions by gating low-quality explanations and stabilizes paragraph aggregation. We discuss deployment-oriented choices—probability calibration, paragraph-level scoring, and human-in-the-loop validation—that increase trustworthiness without disclosing sensitive dataset details.