experiments/5.7. Benchmarking - Classifier Building & Evaluation (SaProt with Structure Masking).ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "id": "55c673ce",
            "metadata": {},
            "source": [
                "# PHIStruct: Improving phage-host interaction prediction at low sequence similarity settings using structure-aware protein embeddings\n",
                "\n",
                "<b>Mark Edward M. Gonzales<sup>1, 2</sup>, Jennifer C. Ureta<sup>1, 2, 3</sup> & Anish M.S. Shrestha<sup>1, 2</sup></b>\n",
                "\n",
                "<sup>1</sup> Bioinformatics Lab, Advanced Research Institute for Informatics, Computing and Networking, De La Salle University, Manila 1004, Philippines <br>\n",
                "<sup>2</sup> Department of Software Technology, College of Computer Studies, De La Salle University, Manila 1004, Philippines <br>\n",
"<sup>3</sup> Walter and Eliza Hall Institute of Medical Research, Melbourne, Victoria, 3052, Australia\n",
                "\n",
                "✉️ gonzales.markedward@gmail.com, jennifer.ureta@gmail.com, anish.shrestha@dlsu.edu.ph"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "e9619d49",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "1eefb76b",
            "metadata": {},
            "source": [
                "# 💡 Prerequisites\n",
                "\n",
                "### Option 1: Download the prerequisite files\n",
                "1. Download `consolidated.tar.gz` from this [link](https://drive.google.com/file/d/1yQSXwlb37dm2ZLXGJHdIM5vmrzwPAwvI/view?usp=sharing), and unzip it. This should result in a folder named `consolidated`. <br> Technically, this notebook only needs `consolidated/rbp_embeddings_saprot_struct_mask_relaxed_r3.csv` and `consolidated/rbp_embeddings_saprot_relaxed_r3.csv`.\n",
                "1. Create a folder named `inphared` inside `data`, and save the extracted `consolidated` folder inside `data/inphared`. \n",
                "1. Download `fasta.tar.gz` from this [link](https://drive.google.com/file/d/1NMFR3JrrrCHLoCMQp2nia4dgtcXs5x05/view?usp=sharing), and unzip it. This should result in a folder named `fasta`. <br> Technically, this notebook only needs the `.clstr` files inside `fasta`.\n",
                "1. Save the extracted `fasta` folder inside `data/inphared`.\n",
                "\n",
                "### Option 2: Generate the prerequisite files yourself\n",
                "1. If you have run `3.4. Data Consolidation (SaProt with Structure Masking).ipynb`, then `data/inphared/consolidated` should have already been populated with the prerequisite files.\n",
                "1. Consolidate the sequences of the proteins with predicted structures into a single FASTA file. <br>\n",
                "   For reproducibility, we provide our consolidated FASTA file [here](https://drive.google.com/file/d/1LTZte1f4lreQ5MXWeM-y2Mtp9z96pXS7/view?usp=sharing).\n",
                "1. Generate the protein clusters by running CD-HIT on this FASTA file at a sequence similarity threshold of 100%, following the instructions [here](https://github.com/weizhongli/cdhit). \n",
                "1. Rename the resulting `.clstr` file to `complete-struct-100.fasta.clstr` and the resulting FASTA file (containing only the representative sequences) to `complete-struct-100.fasta`. \n",
                "1. Generate `complete-struct-80.fasta.clstr`, `complete-struct-60.fasta.clstr`, and `complete-struct-40.fasta.clstr` by running CD-HIT on `complete-struct-100.fasta` at sequence similarity thresholds of 80%, 60%, and 40%, respectively.\n",
                "1. Create a folder named `fasta` inside `data/inphared`, and save the four `.clstr` files inside `data/inphared/fasta`.\n",
                "\n",
                "### Resulting folder structure\n",
                "\n",
                "`experiments` (parent folder of this notebook) <br> \n",
                "↳ `data` <br>\n",
                "&nbsp; &nbsp;↳ `inphared` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `consolidated` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_embeddings_saprot_relaxed_r3.csv` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_embeddings_saprot_struct_mask_relaxed_r3.csv` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `fasta` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-100.fasta.clstr` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-80.fasta.clstr` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-40.fasta.clstr` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-60.fasta.clstr` <br>\n",
                "↳ `5.7. Benchmarking - Classifier Building & Evaluation (SaProt with Structure Masking).ipynb` (this notebook) <br>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "f3015489",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "c059ce7f",
            "metadata": {},
            "source": [
                "# 📁 Output files\n",
                "\n",
                "The output files (i.e., the results of evaluating the model's performance) &mdash; which are saved in `temp/results` &mdash; are already included when the repository was cloned. <br>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "cecf5469",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "f274d083",
            "metadata": {},
            "source": [
                "# Part I: Preliminaries"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "fc7d25a1",
            "metadata": {},
            "source": [
                "Import the necessary libraries and modules."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "id": "0840061c",
            "metadata": {},
            "outputs": [],
            "source": [
                "import math\n",
                "import pickle\n",
                "import os\n",
                "import warnings\n",
                "\n",
                "import pandas as pd\n",
                "import numpy as np\n",
                "import sklearn\n",
                "\n",
                "import ConstantsUtil\n",
                "import ClassificationUtil\n",
                "\n",
                "%load_ext autoreload\n",
                "%autoreload 2"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "98fb65d4",
            "metadata": {},
            "outputs": [],
            "source": [
                "pd.set_option(\"display.max_rows\", None)\n",
                "pd.set_option(\"display.max_columns\", 50)\n",
                "\n",
                "pd.options.mode.chained_assignment = None\n",
                "\n",
                "with warnings.catch_warnings():\n",
                "    warnings.filterwarnings(\n",
                "        \"ignore\", category=sklearn.exceptions.UndefinedMetricWarning\n",
                "    )"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "e27723c6",
            "metadata": {},
            "outputs": [],
            "source": [
                "constants = ConstantsUtil.ConstantsUtil()\n",
                "util = ClassificationUtil.ClassificationUtil()"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "0ada3031",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "1f9f7cc0",
            "metadata": {},
            "source": [
                "# Part II: Classifier Building and Evaluation"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "53d0124d",
            "metadata": {},
            "source": [
                "Train a multilayer perceptron, and evaluate its performance at different train-versus-test similarity and confidence thresholds."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "id": "0faa8454",
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "*** saprot_struct_mask_relaxed_r3, similarity = 100% ***\n",
                        "Constructing training and test sets...\n",
                        "Training set shape: (16944, 1280)\n",
                        "Test set shape: (2340, 1280)\n",
                        "Training the model...\n",
                        "Saving evaluation results...\n",
                        "Confidence threshold k: 0.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8602    0.7207    0.7843       111\n",
                        "  enterobacter     0.3234    0.4655    0.3816       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.8889    0.8000    0.8421      1040\n",
                        "    klebsiella     0.7678    0.8894    0.8241       461\n",
                        "        others     0.0000    0.0000    0.0000        51\n",
                        "   pseudomonas     0.8299    0.9237    0.8743       354\n",
                        "staphylococcus     0.9264    0.9679    0.9467       156\n",
                        "\n",
                        "      accuracy                         0.8120      2340\n",
                        "     macro avg     0.6831    0.7087    0.6922      2340\n",
                        "  weighted avg     0.8094    0.8120    0.8074      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 10.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8681    0.7117    0.7822       111\n",
                        "  enterobacter     0.3354    0.4569    0.3869       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.8916    0.7827    0.8336      1040\n",
                        "    klebsiella     0.7797    0.8829    0.8281       461\n",
                        "        others     0.0196    0.0196    0.0196        51\n",
                        "   pseudomonas     0.8359    0.9209    0.8763       354\n",
                        "staphylococcus     0.9321    0.9679    0.9497       156\n",
                        "\n",
                        "      accuracy                         0.8021      2340\n",
                        "     macro avg     0.6913    0.7056    0.6951      2340\n",
                        "  weighted avg     0.8156    0.8021    0.8055      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 20.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8652    0.6937    0.7700       111\n",
                        "  enterobacter     0.3356    0.4310    0.3774       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.8923    0.7644    0.8234      1040\n",
                        "    klebsiella     0.7937    0.8764    0.8330       461\n",
                        "        others     0.0291    0.0588    0.0390        51\n",
                        "   pseudomonas     0.8442    0.9181    0.8796       354\n",
                        "staphylococcus     0.9379    0.9679    0.9527       156\n",
                        "\n",
                        "      accuracy                         0.7910      2340\n",
                        "     macro avg     0.6957    0.7015    0.6949      2340\n",
                        "  weighted avg     0.8204    0.7910    0.8020      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 30.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8851    0.6937    0.7778       111\n",
                        "  enterobacter     0.3453    0.4138    0.3765       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.8963    0.7481    0.8155      1040\n",
                        "    klebsiella     0.7992    0.8633    0.8300       461\n",
                        "        others     0.0452    0.1373    0.0680        51\n",
                        "   pseudomonas     0.8500    0.9124    0.8801       354\n",
                        "staphylococcus     0.9437    0.9679    0.9557       156\n",
                        "\n",
                        "      accuracy                         0.7812      2340\n",
                        "     macro avg     0.7041    0.7048    0.6985      2340\n",
                        "  weighted avg     0.8263    0.7812    0.7992      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 40.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8953    0.6937    0.7817       111\n",
                        "  enterobacter     0.3485    0.3966    0.3710       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.8999    0.7346    0.8089      1040\n",
                        "    klebsiella     0.8128    0.8568    0.8342       461\n",
                        "        others     0.0359    0.1373    0.0569        51\n",
                        "   pseudomonas     0.8522    0.9124    0.8813       354\n",
                        "staphylococcus     0.9437    0.9679    0.9557       156\n",
                        "\n",
                        "      accuracy                         0.7731      2340\n",
                        "     macro avg     0.7070    0.7002    0.6968      2340\n",
                        "  weighted avg     0.8314    0.7731    0.7969      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 50.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8941    0.6847    0.7755       111\n",
                        "  enterobacter     0.3520    0.3793    0.3651       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.9033    0.7183    0.8002      1040\n",
                        "    klebsiella     0.8270    0.8503    0.8385       461\n",
                        "        others     0.0287    0.1373    0.0475        51\n",
                        "   pseudomonas     0.8606    0.9068    0.8831       354\n",
                        "staphylococcus     0.9497    0.9679    0.9587       156\n",
                        "\n",
                        "      accuracy                         0.7624      2340\n",
                        "     macro avg     0.7104    0.6933    0.6942      2340\n",
                        "  weighted avg     0.8373    0.7624    0.7936      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 60.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.8916    0.6667    0.7629       111\n",
                        "  enterobacter     0.3839    0.3707    0.3772       116\n",
                        "  enterococcus     0.8679    0.9020    0.8846        51\n",
                        "   escherichia     0.9052    0.6981    0.7883      1040\n",
                        "    klebsiella     0.8380    0.8416    0.8398       461\n",
                        "        others     0.0330    0.1961    0.0565        51\n",
                        "   pseudomonas     0.8689    0.8983    0.8833       354\n",
                        "staphylococcus     0.9557    0.9679    0.9618       156\n",
                        "\n",
                        "      accuracy                         0.7504      2340\n",
                        "     macro avg     0.7180    0.6927    0.6943      2340\n",
                        "  weighted avg     0.8435    0.7504    0.7889      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 70.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.9103    0.6396    0.7513       111\n",
                        "  enterobacter     0.4239    0.3362    0.3750       116\n",
                        "  enterococcus     0.8846    0.9020    0.8932        51\n",
                        "   escherichia     0.9072    0.6673    0.7690      1040\n",
                        "    klebsiella     0.8575    0.8221    0.8394       461\n",
                        "        others     0.0379    0.2941    0.0671        51\n",
                        "   pseudomonas     0.8824    0.8898    0.8861       354\n",
                        "staphylococcus     0.9557    0.9679    0.9618       156\n",
                        "\n",
                        "      accuracy                         0.7308      2340\n",
                        "     macro avg     0.7324    0.6899    0.6929      2340\n",
                        "  weighted avg     0.8536    0.7308    0.7805      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 80.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.9067    0.6126    0.7312       111\n",
                        "  enterobacter     0.4730    0.3017    0.3684       116\n",
                        "  enterococcus     0.9000    0.8824    0.8911        51\n",
                        "   escherichia     0.9122    0.6192    0.7377      1040\n",
                        "    klebsiella     0.8786    0.8004    0.8377       461\n",
                        "        others     0.0449    0.4510    0.0817        51\n",
                        "   pseudomonas     0.8960    0.8757    0.8857       354\n",
                        "staphylococcus     0.9554    0.9615    0.9585       156\n",
                        "\n",
                        "      accuracy                         0.7026      2340\n",
                        "     macro avg     0.7458    0.6881    0.6865      2340\n",
                        "  weighted avg     0.8648    0.7026    0.7649      2340\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 90.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.9265    0.5676    0.7039       111\n",
                        "  enterobacter     0.5882    0.2586    0.3593       116\n",
                        "  enterococcus     0.9184    0.8824    0.9000        51\n",
                        "   escherichia     0.9188    0.5548    0.6918      1040\n",
                        "    klebsiella     0.9084    0.7527    0.8233       461\n",
                        "        others     0.0401    0.5294    0.0745        51\n",
                        "   pseudomonas     0.9187    0.8616    0.8892       354\n",
                        "staphylococcus     0.9551    0.9551    0.9551       156\n",
                        "\n",
                        "      accuracy                         0.6594      2340\n",
                        "     macro avg     0.7718    0.6703    0.6746      2340\n",
                        "  weighted avg     0.8840    0.6594    0.7403      2340\n",
                        "\n",
                        "===================\n",
                        "Finished\n",
                        "===================\n",
                        "*** saprot_struct_mask_relaxed_r3, similarity = 80% ***\n",
                        "Constructing training and test sets...\n",
                        "Training set shape: (14812, 1280)\n",
                        "Test set shape: (2822, 1280)\n",
                        "Training the model...\n",
                        "Saving evaluation results...\n",
                        "Confidence threshold k: 0.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.5246    0.5872    0.5541       109\n",
                        "  enterobacter     0.1786    0.2096    0.1928       167\n",
                        "  enterococcus     0.6000    0.5000    0.5455        60\n",
                        "   escherichia     0.7327    0.7240    0.7283      1344\n",
                        "    klebsiella     0.5801    0.5957    0.5878       559\n",
                        "        others     0.0000    0.0000    0.0000        60\n",
                        "   pseudomonas     0.6343    0.6524    0.6433       351\n",
                        "staphylococcus     0.8063    0.8953    0.8485       172\n",
                        "\n",
                        "      accuracy                         0.6442      2822\n",
                        "     macro avg     0.5071    0.5205    0.5125      2822\n",
                        "  weighted avg     0.6355    0.6442    0.6394      2822\n",
                        "\n",
                        "===================\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Confidence threshold k: 10.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.5250    0.5780    0.5502       109\n",
                        "  enterobacter     0.1848    0.2036    0.1937       167\n",
                        "  enterococcus     0.6304    0.4833    0.5472        60\n",
                        "   escherichia     0.7375    0.7106    0.7238      1344\n",
                        "    klebsiella     0.5917    0.5832    0.5874       559\n",
                        "        others     0.0106    0.0167    0.0130        60\n",
                        "   pseudomonas     0.6366    0.6239    0.6302       351\n",
                        "staphylococcus     0.8191    0.8953    0.8556       172\n",
                        "\n",
                        "      accuracy                         0.6311      2822\n",
                        "     macro avg     0.5170    0.5118    0.5126      2822\n",
                        "  weighted avg     0.6424    0.6311    0.6362      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 20.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.5398    0.5596    0.5495       109\n",
                        "  enterobacter     0.1921    0.2036    0.1977       167\n",
                        "  enterococcus     0.6444    0.4833    0.5524        60\n",
                        "   escherichia     0.7449    0.7016    0.7226      1344\n",
                        "    klebsiella     0.6007    0.5760    0.5881       559\n",
                        "        others     0.0241    0.0667    0.0354        60\n",
                        "   pseudomonas     0.6416    0.6068    0.6237       351\n",
                        "staphylococcus     0.8235    0.8953    0.8579       172\n",
                        "\n",
                        "      accuracy                         0.6237      2822\n",
                        "     macro avg     0.5264    0.5116    0.5159      2822\n",
                        "  weighted avg     0.6502    0.6237    0.6359      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 30.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.5701    0.5596    0.5648       109\n",
                        "  enterobacter     0.1916    0.1916    0.1916       167\n",
                        "  enterococcus     0.6667    0.4667    0.5490        60\n",
                        "   escherichia     0.7486    0.6845    0.7151      1344\n",
                        "    klebsiella     0.6062    0.5617    0.5831       559\n",
                        "        others     0.0203    0.0833    0.0327        60\n",
                        "   pseudomonas     0.6422    0.5983    0.6195       351\n",
                        "staphylococcus     0.8280    0.8953    0.8603       172\n",
                        "\n",
                        "      accuracy                         0.6109      2822\n",
                        "     macro avg     0.5342    0.5051    0.5145      2822\n",
                        "  weighted avg     0.6549    0.6109    0.6311      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 40.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.5900    0.5413    0.5646       109\n",
                        "  enterobacter     0.2038    0.1916    0.1975       167\n",
                        "  enterococcus     0.6750    0.4500    0.5400        60\n",
                        "   escherichia     0.7544    0.6652    0.7070      1344\n",
                        "    klebsiella     0.6245    0.5564    0.5885       559\n",
                        "        others     0.0238    0.1333    0.0404        60\n",
                        "   pseudomonas     0.6440    0.5926    0.6172       351\n",
                        "staphylococcus     0.8415    0.8953    0.8676       172\n",
                        "\n",
                        "      accuracy                         0.5999      2822\n",
                        "     macro avg     0.5446    0.5032    0.5154      2822\n",
                        "  weighted avg     0.6641    0.5999    0.6288      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 50.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.6042    0.5321    0.5659       109\n",
                        "  enterobacter     0.2098    0.1796    0.1935       167\n",
                        "  enterococcus     0.6923    0.4500    0.5455        60\n",
                        "   escherichia     0.7623    0.6466    0.6997      1344\n",
                        "    klebsiella     0.6398    0.5403    0.5858       559\n",
                        "        others     0.0203    0.1500    0.0357        60\n",
                        "   pseudomonas     0.6440    0.5670    0.6030       351\n",
                        "staphylococcus     0.8547    0.8895    0.8718       172\n",
                        "\n",
                        "      accuracy                         0.5836      2822\n",
                        "     macro avg     0.5534    0.4944    0.5126      2822\n",
                        "  weighted avg     0.6729    0.5836    0.6231      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 60.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.6000    0.5229    0.5588       109\n",
                        "  enterobacter     0.2132    0.1737    0.1914       167\n",
                        "  enterococcus     0.7941    0.4500    0.5745        60\n",
                        "   escherichia     0.7726    0.6243    0.6905      1344\n",
                        "    klebsiella     0.6527    0.5313    0.5858       559\n",
                        "        others     0.0203    0.1833    0.0365        60\n",
                        "   pseudomonas     0.6544    0.5556    0.6009       351\n",
                        "staphylococcus     0.8693    0.8895    0.8793       172\n",
                        "\n",
                        "      accuracy                         0.5698      2822\n",
                        "     macro avg     0.5721    0.4913    0.5147      2822\n",
                        "  weighted avg     0.6847    0.5698    0.6192      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 70.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.6477    0.5229    0.5787       109\n",
                        "  enterobacter     0.2016    0.1557    0.1757       167\n",
                        "  enterococcus     0.8182    0.4500    0.5806        60\n",
                        "   escherichia     0.7816    0.5885    0.6715      1344\n",
                        "    klebsiella     0.6644    0.5242    0.5860       559\n",
                        "        others     0.0195    0.2167    0.0357        60\n",
                        "   pseudomonas     0.6512    0.5214    0.5791       351\n",
                        "staphylococcus     0.8941    0.8837    0.8889       172\n",
                        "\n",
                        "      accuracy                         0.5464      2822\n",
                        "     macro avg     0.5848    0.4829    0.5120      2822\n",
                        "  weighted avg     0.6941    0.5464    0.6079      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 80.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.6627    0.5046    0.5729       109\n",
                        "  enterobacter     0.2150    0.1377    0.1679       167\n",
                        "  enterococcus     0.8438    0.4500    0.5870        60\n",
                        "   escherichia     0.7926    0.5432    0.6446      1344\n",
                        "    klebsiella     0.6934    0.5098    0.5876       559\n",
                        "        others     0.0214    0.3000    0.0399        60\n",
                        "   pseudomonas     0.6692    0.4957    0.5696       351\n",
                        "staphylococcus     0.9096    0.8779    0.8935       172\n",
                        "\n",
                        "      accuracy                         0.5184      2822\n",
                        "     macro avg     0.6010    0.4774    0.5079      2822\n",
                        "  weighted avg     0.7102    0.5184    0.5941      2822\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 90.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.6892    0.4679    0.5574       109\n",
                        "  enterobacter     0.2462    0.0958    0.1379       167\n",
                        "  enterococcus     0.9643    0.4500    0.6136        60\n",
                        "   escherichia     0.8096    0.4874    0.6085      1344\n",
                        "    klebsiella     0.7377    0.4830    0.5838       559\n",
                        "        others     0.0230    0.4167    0.0436        60\n",
                        "   pseudomonas     0.6738    0.4473    0.5377       351\n",
                        "staphylococcus     0.9371    0.8663    0.9003       172\n",
                        "\n",
                        "      accuracy                         0.4784      2822\n",
                        "     macro avg     0.6351    0.4643    0.4978      2822\n",
                        "  weighted avg     0.7348    0.4784    0.5708      2822\n",
                        "\n",
                        "===================\n",
                        "Finished\n",
                        "===================\n",
                        "*** saprot_struct_mask_relaxed_r3, similarity = 60% ***\n",
                        "Constructing training and test sets...\n",
                        "Training set shape: (13099, 1280)\n",
                        "Test set shape: (3153, 1280)\n",
                        "Training the model...\n",
                        "Saving evaluation results...\n",
                        "Confidence threshold k: 0.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3387    0.5207    0.4104       121\n",
                        "  enterobacter     0.1055    0.2041    0.1391       196\n",
                        "  enterococcus     0.5571    0.7959    0.6555        49\n",
                        "   escherichia     0.6807    0.5890    0.6316      1589\n",
                        "    klebsiella     0.4793    0.4524    0.4654       588\n",
                        "        others     0.0000    0.0000    0.0000        49\n",
                        "   pseudomonas     0.5679    0.6213    0.5934       404\n",
                        "staphylococcus     0.8356    0.7771    0.8053       157\n",
                        "\n",
                        "      accuracy                         0.5446      3153\n",
                        "     macro avg     0.4456    0.4951    0.4626      3153\n",
                        "  weighted avg     0.5750    0.5446    0.5558      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 10.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3425    0.5124    0.4106       121\n",
                        "  enterobacter     0.1017    0.1837    0.1309       196\n",
                        "  enterococcus     0.5507    0.7755    0.6441        49\n",
                        "   escherichia     0.6782    0.5651    0.6165      1589\n",
                        "    klebsiella     0.4888    0.4456    0.4662       588\n",
                        "        others     0.0083    0.0204    0.0118        49\n",
                        "   pseudomonas     0.5788    0.6089    0.5935       404\n",
                        "staphylococcus     0.8472    0.7771    0.8106       157\n",
                        "\n",
                        "      accuracy                         0.5281      3153\n",
                        "     macro avg     0.4495    0.4861    0.4605      3153\n",
                        "  weighted avg     0.5775    0.5281    0.5482      3153\n",
                        "\n",
                        "===================\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Confidence threshold k: 20.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3503    0.5124    0.4161       121\n",
                        "  enterobacter     0.1034    0.1684    0.1282       196\n",
                        "  enterococcus     0.5672    0.7755    0.6552        49\n",
                        "   escherichia     0.6816    0.5469    0.6068      1589\n",
                        "    klebsiella     0.5000    0.4405    0.4684       588\n",
                        "        others     0.0127    0.0612    0.0211        49\n",
                        "   pseudomonas     0.5813    0.6015    0.5912       404\n",
                        "staphylococcus     0.8531    0.7771    0.8133       157\n",
                        "\n",
                        "      accuracy                         0.5167      3153\n",
                        "     macro avg     0.4562    0.4854    0.4625      3153\n",
                        "  weighted avg     0.5826    0.5167    0.5439      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 30.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3554    0.4876    0.4111       121\n",
                        "  enterobacter     0.1077    0.1633    0.1298       196\n",
                        "  enterococcus     0.5672    0.7755    0.6552        49\n",
                        "   escherichia     0.6801    0.5205    0.5897      1589\n",
                        "    klebsiella     0.5050    0.4269    0.4627       588\n",
                        "        others     0.0163    0.1224    0.0288        49\n",
                        "   pseudomonas     0.5860    0.5817    0.5839       404\n",
                        "staphylococcus     0.8592    0.7771    0.8161       157\n",
                        "\n",
                        "      accuracy                         0.4979      3153\n",
                        "     macro avg     0.4596    0.4819    0.4597      3153\n",
                        "  weighted avg     0.5842    0.4979    0.5334      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 40.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3665    0.4876    0.4184       121\n",
                        "  enterobacter     0.1082    0.1480    0.1250       196\n",
                        "  enterococcus     0.5606    0.7551    0.6435        49\n",
                        "   escherichia     0.6835    0.4934    0.5731      1589\n",
                        "    klebsiella     0.5169    0.4150    0.4604       588\n",
                        "        others     0.0156    0.1633    0.0284        49\n",
                        "   pseudomonas     0.6068    0.5767    0.5914       404\n",
                        "staphylococcus     0.8652    0.7771    0.8188       157\n",
                        "\n",
                        "      accuracy                         0.4808      3153\n",
                        "     macro avg     0.4654    0.4770    0.4574      3153\n",
                        "  weighted avg     0.5915    0.4808    0.5255      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 50.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3841    0.4793    0.4265       121\n",
                        "  enterobacter     0.1130    0.1378    0.1241       196\n",
                        "  enterococcus     0.5714    0.7347    0.6429        49\n",
                        "   escherichia     0.6860    0.4688    0.5570      1589\n",
                        "    klebsiella     0.5298    0.4082    0.4611       588\n",
                        "        others     0.0169    0.2245    0.0315        49\n",
                        "   pseudomonas     0.6230    0.5767    0.5990       404\n",
                        "staphylococcus     0.8841    0.7771    0.8271       157\n",
                        "\n",
                        "      accuracy                         0.4669      3153\n",
                        "     macro avg     0.4760    0.4759    0.4586      3153\n",
                        "  weighted avg     0.5993    0.4669    0.5192      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 60.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3944    0.4628    0.4259       121\n",
                        "  enterobacter     0.1132    0.1224    0.1176       196\n",
                        "  enterococcus     0.5932    0.7143    0.6481        49\n",
                        "   escherichia     0.6917    0.4462    0.5425      1589\n",
                        "    klebsiella     0.5545    0.4065    0.4691       588\n",
                        "        others     0.0191    0.3061    0.0360        49\n",
                        "   pseudomonas     0.6319    0.5693    0.5990       404\n",
                        "staphylococcus     0.9037    0.7771    0.8356       157\n",
                        "\n",
                        "      accuracy                         0.4535      3153\n",
                        "     macro avg     0.4877    0.4756    0.4592      3153\n",
                        "  weighted avg     0.6097    0.4535    0.5135      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 70.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.4154    0.4463    0.4303       121\n",
                        "  enterobacter     0.1180    0.1071    0.1123       196\n",
                        "  enterococcus     0.6111    0.6735    0.6408        49\n",
                        "   escherichia     0.7012    0.4135    0.5202      1589\n",
                        "    klebsiella     0.5533    0.3707    0.4440       588\n",
                        "        others     0.0175    0.3469    0.0333        49\n",
                        "   pseudomonas     0.6449    0.5619    0.6005       404\n",
                        "staphylococcus     0.9037    0.7771    0.8356       157\n",
                        "\n",
                        "      accuracy                         0.4278      3153\n",
                        "     macro avg     0.4956    0.4621    0.4521      3153\n",
                        "  weighted avg     0.6172    0.4278    0.4975      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 80.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.4474    0.4215    0.4340       121\n",
                        "  enterobacter     0.1154    0.0765    0.0920       196\n",
                        "  enterococcus     0.6038    0.6531    0.6275        49\n",
                        "   escherichia     0.7208    0.3801    0.4977      1589\n",
                        "    klebsiella     0.6000    0.3520    0.4437       588\n",
                        "        others     0.0157    0.3878    0.0302        49\n",
                        "   pseudomonas     0.6646    0.5396    0.5956       404\n",
                        "staphylococcus     0.9104    0.7771    0.8385       157\n",
                        "\n",
                        "      accuracy                         0.4022      3153\n",
                        "     macro avg     0.5098    0.4485    0.4449      3153\n",
                        "  weighted avg     0.6396    0.4022    0.4843      3153\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 90.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.4646    0.3802    0.4182       121\n",
                        "  enterobacter     0.1235    0.0510    0.0722       196\n",
                        "  enterococcus     0.6809    0.6531    0.6667        49\n",
                        "   escherichia     0.7237    0.3247    0.4483      1589\n",
                        "    klebsiella     0.6441    0.3078    0.4166       588\n",
                        "        others     0.0158    0.4898    0.0307        49\n",
                        "   pseudomonas     0.7368    0.5198    0.6096       404\n",
                        "staphylococcus     0.9231    0.7643    0.8362       157\n",
                        "\n",
                        "      accuracy                         0.3612      3153\n",
                        "     macro avg     0.5391    0.4363    0.4373      3153\n",
                        "  weighted avg     0.6616    0.3612    0.4547      3153\n",
                        "\n",
                        "===================\n",
                        "Finished\n",
                        "===================\n",
                        "*** saprot_struct_mask_relaxed_r3, similarity = 40% ***\n",
                        "Constructing training and test sets...\n",
                        "Training set shape: (11658, 1280)\n",
                        "Test set shape: (3203, 1280)\n",
                        "Training the model...\n",
                        "Saving evaluation results...\n",
                        "Confidence threshold k: 0.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.1631    0.5955    0.2560        89\n",
                        "  enterobacter     0.1422    0.1435    0.1429       216\n",
                        "  enterococcus     0.5000    0.7317    0.5941        41\n",
                        "   escherichia     0.7495    0.5896    0.6600      1796\n",
                        "    klebsiella     0.4272    0.4175    0.4223       527\n",
                        "        others     0.0000    0.0000    0.0000        41\n",
                        "   pseudomonas     0.3877    0.6270    0.4791       311\n",
                        "staphylococcus     0.8166    0.7582    0.7863       182\n",
                        "\n",
                        "      accuracy                         0.5389      3203\n",
                        "     macro avg     0.3983    0.4829    0.4176      3203\n",
                        "  weighted avg     0.5951    0.5389    0.5551      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 10.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.1721    0.5955    0.2670        89\n",
                        "  enterobacter     0.1371    0.1250    0.1308       216\n",
                        "  enterococcus     0.5172    0.7317    0.6061        41\n",
                        "   escherichia     0.7539    0.5696    0.6489      1796\n",
                        "    klebsiella     0.4424    0.4080    0.4245       527\n",
                        "        others     0.0214    0.0732    0.0331        41\n",
                        "   pseudomonas     0.3926    0.6174    0.4800       311\n",
                        "staphylococcus     0.8155    0.7527    0.7829       182\n",
                        "\n",
                        "      accuracy                         0.5245      3203\n",
                        "     macro avg     0.4065    0.4841    0.4217      3203\n",
                        "  weighted avg     0.6009    0.5245    0.5492      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 20.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.1787    0.5843    0.2737        89\n",
                        "  enterobacter     0.1381    0.1157    0.1259       216\n",
                        "  enterococcus     0.5357    0.7317    0.6186        41\n",
                        "   escherichia     0.7624    0.5484    0.6380      1796\n",
                        "    klebsiella     0.4530    0.3928    0.4207       527\n",
                        "        others     0.0104    0.0732    0.0182        41\n",
                        "   pseudomonas     0.3979    0.6013    0.4789       311\n",
                        "staphylococcus     0.8204    0.7527    0.7851       182\n",
                        "\n",
                        "      accuracy                         0.5076      3203\n",
                        "     macro avg     0.4121    0.4750    0.4199      3203\n",
                        "  weighted avg     0.6085    0.5076    0.5423      3203\n",
                        "\n",
                        "===================\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Confidence threshold k: 30.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.1871    0.5843    0.2834        89\n",
                        "  enterobacter     0.1589    0.1111    0.1308       216\n",
                        "  enterococcus     0.5455    0.7317    0.6250        41\n",
                        "   escherichia     0.7704    0.5306    0.6284      1796\n",
                        "    klebsiella     0.4561    0.3643    0.4051       527\n",
                        "        others     0.0113    0.1220    0.0207        41\n",
                        "   pseudomonas     0.4071    0.5916    0.4823       311\n",
                        "staphylococcus     0.8193    0.7473    0.7816       182\n",
                        "\n",
                        "      accuracy                         0.4920      3203\n",
                        "     macro avg     0.4194    0.4729    0.4197      3203\n",
                        "  weighted avg     0.6161    0.4920    0.5352      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 40.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.1955    0.5843    0.2930        89\n",
                        "  enterobacter     0.1679    0.1065    0.1303       216\n",
                        "  enterococcus     0.5686    0.7073    0.6304        41\n",
                        "   escherichia     0.7763    0.5100    0.6156      1796\n",
                        "    klebsiella     0.4842    0.3491    0.4057       527\n",
                        "        others     0.0100    0.1463    0.0188        41\n",
                        "   pseudomonas     0.4118    0.5627    0.4755       311\n",
                        "staphylococcus     0.8193    0.7473    0.7816       182\n",
                        "\n",
                        "      accuracy                         0.4749      3203\n",
                        "     macro avg     0.4292    0.4642    0.4189      3203\n",
                        "  weighted avg     0.6256    0.4749    0.5278      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 50.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.2063    0.5843    0.3050        89\n",
                        "  enterobacter     0.1780    0.0972    0.1257       216\n",
                        "  enterococcus     0.5714    0.6829    0.6222        41\n",
                        "   escherichia     0.7848    0.4894    0.6029      1796\n",
                        "    klebsiella     0.5043    0.3302    0.3991       527\n",
                        "        others     0.0120    0.2195    0.0227        41\n",
                        "   pseudomonas     0.4275    0.5498    0.4810       311\n",
                        "staphylococcus     0.8193    0.7473    0.7816       182\n",
                        "\n",
                        "      accuracy                         0.4589      3203\n",
                        "     macro avg     0.4380    0.4626    0.4175      3203\n",
                        "  weighted avg     0.6363    0.4589    0.5200      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 60.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.2308    0.5730    0.3290        89\n",
                        "  enterobacter     0.1667    0.0833    0.1111       216\n",
                        "  enterococcus     0.5652    0.6341    0.5977        41\n",
                        "   escherichia     0.7930    0.4671    0.5879      1796\n",
                        "    klebsiella     0.5387    0.3169    0.3990       527\n",
                        "        others     0.0164    0.3659    0.0314        41\n",
                        "   pseudomonas     0.4436    0.5434    0.4884       311\n",
                        "staphylococcus     0.8232    0.7418    0.7803       182\n",
                        "\n",
                        "      accuracy                         0.4433      3203\n",
                        "     macro avg     0.4472    0.4657    0.4156      3203\n",
                        "  weighted avg     0.6482    0.4433    0.5118      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 70.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.2500    0.5393    0.3416        89\n",
                        "  enterobacter     0.1753    0.0787    0.1086       216\n",
                        "  enterococcus     0.5778    0.6341    0.6047        41\n",
                        "   escherichia     0.8052    0.4326    0.5628      1796\n",
                        "    klebsiella     0.5639    0.2846    0.3783       527\n",
                        "        others     0.0134    0.3659    0.0259        41\n",
                        "   pseudomonas     0.4609    0.5305    0.4933       311\n",
                        "staphylococcus     0.8323    0.7363    0.7813       182\n",
                        "\n",
                        "      accuracy                         0.4159      3203\n",
                        "     macro avg     0.4598    0.4503    0.4121      3203\n",
                        "  weighted avg     0.6626    0.4159    0.4950      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 80.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.2635    0.4944    0.3438        89\n",
                        "  enterobacter     0.1744    0.0694    0.0993       216\n",
                        "  enterococcus     0.5909    0.6341    0.6118        41\n",
                        "   escherichia     0.8268    0.3881    0.5282      1796\n",
                        "    klebsiella     0.6000    0.2562    0.3590       527\n",
                        "        others     0.0124    0.4146    0.0241        41\n",
                        "   pseudomonas     0.5064    0.5080    0.5072       311\n",
                        "staphylococcus     0.8354    0.7253    0.7765       182\n",
                        "\n",
                        "      accuracy                         0.3821      3203\n",
                        "     macro avg     0.4762    0.4363    0.4062      3203\n",
                        "  weighted avg     0.6858    0.3821    0.4730      3203\n",
                        "\n",
                        "===================\n",
                        "Confidence threshold k: 90.0%\n",
                        "                precision    recall  f1-score   support\n",
                        "\n",
                        " acinetobacter     0.3307    0.4719    0.3889        89\n",
                        "  enterobacter     0.1935    0.0556    0.0863       216\n",
                        "  enterococcus     0.6341    0.6341    0.6341        41\n",
                        "   escherichia     0.8486    0.3246    0.4696      1796\n",
                        "    klebsiella     0.6424    0.2011    0.3064       527\n",
                        "        others     0.0146    0.6098    0.0286        41\n",
                        "   pseudomonas     0.5759    0.4759    0.5211       311\n",
                        "staphylococcus     0.8344    0.7198    0.7729       182\n",
                        "\n",
                        "      accuracy                         0.3350      3203\n",
                        "     macro avg     0.5093    0.4366    0.4010      3203\n",
                        "  weighted avg     0.7154    0.3350    0.4333      3203\n",
                        "\n",
                        "===================\n",
                        "Finished\n",
                        "===================\n"
                    ]
                }
            ],
            "source": [
                "models = list(constants.SAPROT_STRUCT_MASK_PLM.keys())\n",
                "\n",
                "for similarity in range(100, 39, -20):\n",
                "    for model in models:\n",
                "        model = model.lower()\n",
                "        df, df_all, protein_clusters = util.filter_proteins_based_on_struct_and_seq_sim(\n",
                "            f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_{model}.csv\",\n",
                "            f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_saprot_relaxed_r3.csv\",\n",
                "            f\"{constants.INPHARED}/{constants.FASTA}/complete-struct-{similarity}.fasta.clstr\",\n",
                "        )\n",
                "\n",
                "        include_proteins_in_cluster = True\n",
                "        if similarity == 100:\n",
                "            include_proteins_in_cluster = False\n",
                "\n",
                "        print(f\"*** {model}, similarity = {similarity}% ***\")\n",
                "        util.classify(\n",
                "            df,\n",
                "            model + \"-mlp-eskapee-smotetomek\",\n",
                "            similarity,\n",
                "            genus=[\n",
                "                \"enterococcus\",\n",
                "                \"staphylococcus\",\n",
                "                \"klebsiella\",\n",
                "                \"acinetobacter\",\n",
                "                \"pseudomonas\",\n",
                "                \"enterobacter\",\n",
                "                \"escherichia\",\n",
                "            ],\n",
                "            feature_columns=[f\"s{i}\" for i in range(1, 1281)],\n",
                "            include_proteins_in_cluster=include_proteins_in_cluster,\n",
                "            rbp_embeddings_all=df_all,\n",
                "            protein_clusters=protein_clusters,\n",
                "            undersample_others=True,\n",
                "            oversample_technique=\"SMOTETomek\",\n",
                "            model=\"MLP\",\n",
                "        )"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3 (ipykernel)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.10.9"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 5
}