A Comparative Analysis of ChatGPT 3.5 and Llama 3 in LLM-Based Code Completion Using LR Parsing

LR 파싱을 이용한 LLM 기반 코드 완성에서 ChatGPT 3.5 와 Llama 3 의 비교 분석

Modern integrated development environments (IDEs) rely on code completion as a key feature to enhance coding efficiency and streamline the developer workflow. Traditional approaches to code completion have often relied on rule-based techniques, static ranking, and prefix-based filtering, which pose challenges in terms of usability and efficiency. Recent research has introduced LR-parsing-based approaches that generate structural candidate suggestions by leveraging language syntax and open-source programs, but they often require manual refinement. Meanwhile, recent advancements in large language models (LLMs) have significantly amplified predictive performance in code completion tasks. However, despite these progress, the impact of LLM selection on LR-parsing based syntax-aware code generation remains underexplored. In this study, we conduct a comparative analysis of the performance and impact of two prominent LLMs, ChatGPT 3.5 and Llama 3, within an LR parsing-based code completion framework. Our experiments, evaluated using SacreBLEU and SequenceMatcher accuracy metrics, reveal that ChatGPT 3.5 achieves higher accuracy than Llama 3, underscoring the importance of selecting an appropriate LLM for enhanced code completion. These findings highlight the role of model selection in LLM-based code completion using LR parsing. Future research could extend this comparative analysis to a broader range of LLMs.