Human intelligence versus artificial intelligence in classifying economics research articles : exploratory evidence
Heikkilä, J. T. (2024). Human intelligence versus artificial intelligence in classifying economics research articles : exploratory evidence. Journal of Documentation, 81, 18-30. https://doi.org/10.1108/JD-05-2024-0104
Published in
Journal of DocumentationAuthors
Date
2024Copyright
© Jussi T.S. Heikkilä. Published by Emerald Publishing Limited.
Purpose
We compare human intelligence to artificial intelligence (AI) in the choice of appropriate Journal of Economic Literature (JEL) codes for research papers in economics.
Design/methodology/approach
We compare the JEL code choices related to articles published in the recent issues of the Journal of Economic Literature and the American Economic Review and compare these to the original JEL code choices of the authors in earlier working paper versions and JEL codes recommended by various generative AI systems (OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini) based on the abstracts of the articles.
Findings
There are significant discrepancies and often limited overlap between authors’ choices of JEL codes, editors’ choices as well as the choices by contemporary widely used AI systems. However, the observations suggest that generative AI can augment human intelligence in the micro-task of choosing the JEL codes and, thus, save researchers time.
Research limitations/implications
Rapid development of AI systems makes the findings quickly obsolete.
Practical implications
AI systems may economize on classification costs and (semi-)automate the choice of JEL codes by recommending the most appropriate ones. Future studies may apply the presented approach to analyze whether the JEL code choices between authors, editors and AI systems converge and become more consistent as humans increasingly interact with AI systems.
Originality/value
We assume that the choice of JEL codes is a micro-task in which boundedly rational decision-makers rather satisfice than optimize. This exploratory experiment is among the first to compare human intelligence and generative AI in choosing and justifying the choice of optimal JEL codes.
...


Publisher
Emerald PublishingISSN Search the Publication Forum
0022-0418Keywords
Publication in research information system
https://converis.jyu.fi/converis/portal/detail/Publication/245090161
Metadata
Show full item recordCollections
- Kauppakorkeakoulu [1413]
Additional information about funding
Financial support from the Päijät-Häme Regional Fund of the Finnish Cultural Foundation is gratefully acknowledged.License
Related items
Showing items with similar title or keywords.
-
Artificial intelligence and other minds : the search for strong AI
Vesterinen, Oskari (2019)Tämän tutkielman tarkoituksena on tutkia tietoisen eli ns. ”vahvan” tekoälyn mahdollisuutta. Tutkielma lähestyy tutkimuskysymystään kolmen keskeisen näkökulman kautta: Alan Turingin esittämän Turingin testin, filosofi John ... -
Challenges and insights in semantic search using language models
Hajihashemi Varnousfaderani, Elahe (2023)Information Retrieval systems such as search engines, originally designed to assist users in finding information, have evolved to become more potent and have found utility in wider range of applications by incorporating ... -
Computational Rationality as a Theory of Interaction
Oulasvirta, Antti; Jokinen, Jussi P. P.; Howes, Andrew (ACM, 2022)How do people interact with computers? This fundamental question was asked by Card, Moran, and Newell in 1983 with a proposition to frame it as a question about human cognition – in other words, as a matter of how information ... -
ChatGPT for STEM Education : A Working Framework
Zeeshan, Khaula; Hämäläinen, Timo; Neittaanmäki, Pekka (EJournal Publishing, 2024)This paper sheds light on the possible use of most recent technology, ChatGPT in STEM (Science, Technology, Engineering, and Mathematics) education. Here we used the ChatGPT tool and explored how it can help teachers in ... -
Open Source Language Models Can Provide Feedback : Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge
Koutcheme, Charles; Dainese, Nicola; Sarsa, Sami; Hellas, Arto; Leinonen, Juho; Denny, Paul (ACM, 2024)Large language models (LLMs) have shown great potential for the automatic generation of feedback in a wide range of computing contexts. However, concerns have been voiced around the privacy and ethical implications of ...