Enhancing Efficiency in Systematic Reviews: Automating Study Selection with GPT-4o Model - Mustalahti, Jaakko (2025)
The systematic literature review process is essential for synthesizing existing knowledge and identifying research gaps across various academic fields. However, traditional literature review processes are time-consuming and costly, often requiring substantial manual effort. Despite rapid advancements in artificial intelligence and automation technologies, academic research methodologies, particularly literature reviews, have not fully leveraged these innovations. This study investigates how large language models and automa-tion solutions can enhance the efficiency and reliability of systematic litera-ture reviews while maintaining rigorous academic standards. The research problem focuses on integrating artificial intelligence ena-bled automation into the literature review process. Specifically, the study ex-plores how the usage of large language models can improve the efficiency of selecting and evaluating studies, how a human-in-the-loop approach can miti-gate biases and enhance reliability, and the extent to which automation can improve the accuracy and reproducibility of literature reviews. Using the Design Science Research methodology, this study develops and evaluates the Auto Literature Review artifact, an artificial intelligence enabled tool designed to automate study selection while incorporating human over-sight. The artifact was evaluated against a benchmark systematic literature review in the field of Information Systems. Evaluation results indicate that the artifact achieves classification accuracy comparable or superior to manual re-views while significantly reducing the time and effort required. Additionally, the study highlights that process descriptions in manually compiled literature reviews suffer from inconsistencies and irreproducibility, issues that automa-tion can help mitigate. However, limitations such as full-text paywall re-strictions, potential of bias in operating the artifact, and lack of quantitative evaluation were identified. The findings suggest that automation, when coupled with human over-sight, can enhance the efficiency and reliability of systematic literature re-views. This study contributes to academic research by providing a prototype artifact that demonstrates the potential of large language models in literature review automation while underscoring the importance of maintaining human control over critical decision-making processes. Future research should ex-plore full-text analysis integration, bias mitigation strategies, and novel op-portunities to integrate large language models into research methodologies.