llm-metadata-extraction

Experiment on metadata extraction using large language models.

The experiment itself is in the Jupyter notebook ExtractMetadataUsingFinetunedGPT3.ipynb which you can view directly on GitHub without installing anything. If you want to run it yourself, see the bottom of this page.

Data set

The aim of the experiment is to extract metadata from recent (2021-2022) doctoral theses obtain from four Finnish universities (Åbo Akademi, University of Turku, University of Vaasa and Lappeenranta-Lahti University of Technology), using only the raw text from the first few pages of the PDF.

The set of 192 documents (88% English, 7% Finnish, 5% Swedish language) will be split into two subsets (train: 149, test: 43). We will extract the text from around 5 pages of text, aiming for 500 to 700 words. The corresponding metadata, which has been exported from DSpace repositories of the universities, is represented in a simple textual "key: value" format, which should be easy enough for a language model to handle. The train set is used to create a data set which will then be used to fine-tune a GPT-3 Curie model. Subsequently the model can be used to generate similar metadata for unseen documents from the test set.

Example output

Here is diff output showing the difference between the original (manually created) metadata (-, red) and the output of the model (+, green) with some comments after each example. These are all documents from the test set that the model has never seen before.

University of Vaasa

 https://osuva.uwasa.fi/handle/10024/11207
 ---
 title: How to apply technology, knowledge and operations decision models for strategically sustainable resource allocation?
-title/alternative: Kuinka soveltaa teknologiaan, tietoon ja tuotantoon liittyvän päätöksenteon malleja strategisesti kestävään resurssien allokointiin?
+title/alternative: Kuinka soveltaa teknologiaan, tietoon ja tuotantoon liittyvää päätöksen teon malleja strategisesti kestävään resurssien allokointiin?
 contributor/faculty: fi=Tekniikan ja innovaatiojohtamisen yksikkö|en=School of Technology and Innovations|
 contributor/author: Tilabi, Sara
 contributor/organization: fi=Vaasan yliopisto|en=University of Vaasa|
 publisher: Vaasan yliopisto
 date/issued: 2020
 relation/issn: 2323-9123
 relation/issn: 0355-2667
 relation/isbn: 978-952-476-915-0
 relation/ispartofseries: Acta Wasaensia
 relation/numberinseries: 445
 ---

Comment: The generated metadata is perfect except for a small mistake in the title.

 https://osuva.uwasa.fi/handle/10024/10432
 ---
 title: Persoonallinen ajattelu päättelyssä ja päätöksenteossa
-title/alternative: Personal Thinking in reasoning and decision making
+title/alternative: Personal thinking in judgment and decision-making
 contributor/faculty: fi=Tekniikan ja innovaatiojohtamisen yksikkö|en=School of Technology and Innovations|
 contributor/author: Alho, Tapio
 contributor/organization: fi=Vaasan yliopisto|en=University of Vaasa|
 publisher: Vaasan yliopisto
 date/issued: 2020
 relation/issn: 2323-9123
 relation/issn: 0355-2667
 relation/isbn: 978-952-476-903-7
 relation/ispartofseries: Acta Wasaensia
 relation/numberinseries: 440
 ---

Comment: The language model seems to have translated the Finnish title into English on its own, instead of picking up the English language title from the publication as it was written by the author.

University of Turku

 https://www.utupub.fi/handle/10024/152860
 ---
-title: Essays on income inequality and financial incentives to work
+title: Esa: Income inequality and financial incentives to work
 contributor/faculty: fi=Turun kauppakorkeakoulu|en=Turku School of Economics|
 contributor/author: Ollonqvist, Joonas
 publisher: fi=Turun yliopisto. Turun kauppakorkeakoulu|en=University of Turku, Turku School of Economics|
-date/issued: 2021
+date/issued: 2022
 relation/issn: 2343-3167
 relation/ispartofseries: Turun yliopiston julkaisuja - Annales Universitatis Turkuensis, Ser E: Oeconomica
 relation/numberinseries: 82
 ---

Comment: The title is a bit off and the model thinks the publication is newer than it really is. Both are clearly indicated in the PDF.

 https://www.utupub.fi/handle/10024/152852
 ---
-title: Run-time management of many-core SoCs: A communication-centric approach
+title: Run-time management of many-core systems – A communication-centric approach
 contributor/faculty: fi=Teknillinen tiedekunta|en=Faculty of Technology|
 contributor/author: Fattah, Mohammad
 publisher: fi=Turun yliopisto|en=University of Turku|
 date/issued: 2021
 relation/issn: 2736-9684
-relation/ispartofseries: Turun yliopiston julkaisua - Annales Universitatis Turkuensis, Ser. F: Technica - Informatica
+relation/ispartofseries: Turun yliopiston julkaisuja - Annales Universitatis Turkuensis, Ser. F: Technica - Informatica
 relation/numberinseries: 7
 ---

Comment: The title is a bit off - the model has turned "SoCs" (meaning systems-on-chip) into "systems" with no good reason. The series title is different too, but in this case the original metadata had a typo and the model got it right!

Åbo Akademi University

 https://www.doria.fi/handle/10024/181280
 ---
-title: Production and testing of magnesium carbonate hydrates for thermal energy storage (TES) application
+title: Production and Testing of Magnesium Carbonate Hydrates for Thermal Energy Storage (TES) Application
 contributor/faculty: Faculty of Science and Engineering
 contributor/faculty: Fakulteten för naturvetenskaper och teknik
 contributor/faculty: Luonnontieteiden ja tekniikan tiedekunta
 contributor/author: Erlund, Rickard
-contributor/opponent: Professor Brian Elmegaard, Technical University of Denmark, Lyngby, Denmark
-contributor/supervisor: Professor Ron Zevenhoven, Åbo Akademi University, Turku
+contributor/opponent: Professor, Technical University of Denmark, Lyngby, Denmark
+contributor/supervisor: Professor, Åbo Akademi University, Turku
+contributor/supervisor: Professor, Åbo Akademi University, Turku
 publisher: Åbo Akademi University
 date/issued: 2021
 ---

Comment: The title differs in case only. The model couldn't extract the names of the opponent and supervisor.

 https://www.doria.fi/handle/10024/181139
 ---
-title: Coulometric Transduction Method for Solid-Contact Ion-Selective Electrodes
+title: Coulometric transduction method for solid-contact ion-selective electrodes
 contributor/faculty: Faculty of Science and Engineering
 contributor/faculty: Fakulteten för naturvetenskaper och teknik
 contributor/faculty: Luonnontieteiden ja tekniikan tiedekunta
 contributor/author: Han, Tingting
 contributor/opponent: Professor Agata Michalska, University of Warsaw, Warsaw, Poland
 contributor/supervisor: Professor Johan Bobacka, Åbo Akademi University, Åbo
 contributor/supervisor: Dr. Ulriika Mattinen, Åbo Akademi University, Åbo
 contributor/supervisor: Docent Zekra Mousavi, Åbo Akademi University, Åbo
 publisher: Åbo Akademi University
 date/issued: 2021
 ---

Comment: The title differs in case only.

Lappeenranta-Lahti University of Technology

 https://lutpub.lut.fi/handle/10024/163304
 ---
 title: Responsible business practices in internationalized SMEs
 contributor/faculty: fi=School of Business and Management|en=School of Business and Management|
 contributor/author: Uzhegova, Maria
 contributor/organization: Lappeenrannan-Lahden teknillinen yliopisto LUT
 contributor/organization: Lappeenranta-Lahti University of Technology LUT
 contributor/opponent: Rialp-Criado, Alex
 contributor/reviewer: Rialp-Criado, Alex
 contributor/reviewer: Andersson, Svante
+publisher: Lappeenranta-Lahti University of Technology LUT
 date/issued: 2021
 relation/issn: 1456-4491
 relation/ispartofseries: Acta Universitatis Lappeenrantaensis
 ---

Comment: The model extracted the publisher, which was missing from original metadata.

 https://lutpub.lut.fi/handle/10024/163258
 ---
-title: Life cycle cost-driven design for additive manufacturing : the frontier to sustainable manufacturing in laser-based powder bed fusion
+title: Life cycle cost-driven design for additive manufacturing: the frontier to sustainable manufacturing in laser-based powder bed fusion
 contributor/faculty: fi=School of Energy Systems|en=School of Energy Systems|
 contributor/author: Nyamekye, Patricia
 contributor/organization: Lappeenrannan-Lahden teknillinen yliopisto LUT
 contributor/organization: Lappeenranta-Lahti University of Technology LUT
 contributor/opponent: Hryha, Eduard
 contributor/reviewer: Hryha, Eduard
 contributor/reviewer: Mazzi, Anna
 publisher: Lappeenranta-Lahti University of Technology LUT
 date/issued: 2021
 relation/issn: 1456-4491
 relation/ispartofseries: Acta Universitatis Lappeenrantaensis
 ---

Comment: The only difference is the use of a colon with or without preceding space in the title.

Installation

This has been tested using Python 3.10 on Ubuntu 22.04. Other recent Python versions (3.8, 3.9) will probably work too.

Create a virtual environment and install the dependencies listed requirements.txt:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Running the notebook

You will need to register an OpenAI account (same that you can use for ChatGPT) and generate an API key in the OpenAI user interface. Store the key in an environment variable:

export OPENAI_API_KEY=<my-generated-api-key>

Then start Jupyter Notebook:

jupyter notebook