همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

Analyzing Stack Overflow Question Types and Answer Characteristics: Implications for Designing and Benchmarking Coding Assistant LLMs

نویسندگان :

Omid Mohammadi Kia¹ Mahmood Neshati²

1- Shahid Beheshti University 2- Shahid Beheshti University

کلمات کلیدی :

Stack Overflow analysis،Question classification،Answer characteristics،Coding assistants،Large Language Models (LLMs)،Benchmarking

چکیده :

Large language models (LLMs) like GitHub Copilot excel as coding assistants, yet struggle with diverse developer queries; Stack Overflow (SO), a key Q&A hub, features conceptual, debugging, and best practice questions, but optimal answer formats—length, code, images, references—remain underexplored, hindering LLM design and benchmarking. We analyze 50,000 high-engagement SO questions from 2020 to 2024, classifying them via hybrid annotation and fine-tuned RoBERTa with high accuracy and 92 percent inter-annotator agreement. Results show: debugging questions, 42 percent of the sample, favor short code-rich answers, 78 percent under 150 words with 91 percent including code and strong acceptance correlation; conceptual queries, 31 percent, prefer long explanations with references, 65 percent over 400 words and 82 percent referenced; best practices, 27 percent, balance multimodal responses with 72 percent code, 56 percent images, and 71 percent references, with associations statistically significant. We propose adaptive LLM prompting and stratified benchmarks, We release an annotated dataset on GitHub to enhance context-aware coding aids.