همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

A Methodological Framework for Assessing Security-Oriented Language Models

نویسندگان :

Mostafa Rahimi¹ Maghsoud Abbaspour²

1- دانشگاه شهید بهشتی 2- دانشگاه شهید بهشتی

کلمات کلیدی :

security task،secure code generation،large language models،evaluation framework

چکیده :

The absence of standardized and reliable benchmarks poses a major challenge in selecting the most suitable large language model (LLM) for various security-related tasks. This limitation arises from the diversity and specificity of such tasks, the mismatch between benchmark metrics and real-world infrastructure requirements, and the limited maturity and adoption of newly proposed benchmarks. Moreover, discovering newly vulnerabilities in source code remains a slow process, as it demands up-to-date security expertise, deep programming knowledge, and extensive code analysis across multiple execution paths. To address these constraints, large language models can be leveraged as intelligent assistants capable of accelerating vulnerability analysis and detection. These models can aid security professionals by (1) understanding vulnerabilities found in source code and mitigate it through retrieval-augmented architectures, (2) performing intelligent static code reviews to identify all possible data and control flows without execution, and (3) generating complex test cases to facilitate inter-function transitions. This study aims to assess the capabilities of LLMs in security-related tasks and develop an evaluation framework for identifying the most effective models to support security experts efficiently and securely.