This article explores the use of large language models (LLMs), specifically GPT, for enhancing information extraction from unstructured text in political science research. By automating the retrieval of explicit details from sources including historical documents, meeting minutes, news articles, and unstructured search results, GPT significantly reduces the time and resources required for data collection. The study highlights how GPT complements human research assistants, combining automated efficiency with human oversight to improve the reliability and depth of research. This integration not only makes comprehensive data collection more accessible; it also increases the overall research efficiency and scope of research.
The article highlights GPT’s unique capabilities in information extraction and its potential to advance empirical research in the field. Additionally, we discuss ethical concerns related to student employment, privacy, bias, and environmental impact associated with the use of LLMs.
In the expanding landscape of political science research, the integration of advanced artificial intelligence (AI) tools has opened novel avenues for data collection, annotation, and analysis. Among these tools, large language models (LLMs), such as OpenAI’s Generative Pre-trained Transformer (GPT), have garnered attention for their potential to enhance research productivity and expand empirical research capabilities (Ziems et al. Reference Ziems, Held, Shaikh, Chen, Zhang and Yang2024).Footnote 1 This study specifically examined the use of GPT for information extraction from unstructured text an essential task that involves retrieving explicitly stated details that may be challenging to access manually. Unlike broader applications such as generating text labels for classification (Chiu, Collins, and Alexander Reference Chiu, Collins and Alexander2022; Wang Reference Wang2023), simulating survey responses (Argyle et al. Reference Argyle, Busby, Fulda, Gubler, Rytting and Wingate2023b), generating stimulus for survey experiments (Velez and Liu Reference Velez and Liu2024), and engaging in conversations with humans (Argyle et al. Reference Argyle, Bail, Busby and Wingate2023a) information extraction focuses on accurately identifying and retrieving explicit content within documents. Although GPT shows promise in various tasks, this study highlights their particular effectiveness in information extraction.
Our study is divided into detailed examinations of the utility of GPT for various data-collection tasks. In these examples, GPT’s applications demonstrate its versatility in handling increasingly complex information tasks across two languages: English and Italian. In the first example, GPT is used to clean Optical Character Recognition (OCR) errors from scans of historical documents, demonstrating its basic ability to process textual data. In the more complex applications described in the second and third examples, GPT helps to extract participant information from semi-structured administrative-meeting-minutes data and detailed source information from lengthy news articles. In the fourth example, we show GPT’s ability to perform an advanced task of synthesizing data from multiple Internet sources.
Each of these four applications demonstrates how GPT performs labor-intensive tasks not only with remarkable speed but also with accuracy that either matches or exceeds human efforts. Furthermore, the use of GPT in these contexts highlights its potential to manage large volumes of data—a capability that is particularly useful in political science when researchers often are faced with extensive but only partially structured datasets. The examples presented in this article highlight GPT’s strengths in natural-language processing while mitigating its weaknesses in complex reasoning and “hallucination” (i.e., false information) (Ji et al. Reference Ji, Lee, Frieske, Yu, Su, Xu, Ishii, Bang, Madotto and Fung2023; Wei et al. Reference Wei, Wang, Schuurmans, Bosma, Xia, Chi, Le and Zhou2022) along with the reliability and consistency of synthetic survey data produced by LLMs (Bisbee et al., Reference Bisbee, Clinton, Dorff, Kenkel and Larson2024).
By presenting a range of unique examples, this article expands thinking in the discipline about the potential uses of LLMs rather than providing a specific how-to guide. We discuss the importance of creatively engineering prompts tailored to different tasks, illustrating that the first prompt may not always suffice and that careful refinement is crucial for optimal results. Through this approach, we hope to inspire further exploration and creative problem-solving using LLMs in political science research.
GPT’s potential to reduce the gap in unequal research resources is another significant benefit of its inclusion in the political science toolbox. Traditionally, large-scale research projects often have been the purview of well-funded researchers who can afford large teams of research assistants (RAs) and expensive data-processing tools. However, GPT’s ability to automate and streamline data extraction and analysis tasks could level the playing field, allowing researchers with limited budgets to undertake more extensive research efforts. However, the use of LLMs in research raises ethical concerns, including the potential loss of jobs for student RAs, privacy risks, social bias in output, and significant environmental impacts. The various ethical concerns of using GPT are discussed in detail.
