experiments/3.5. Data Consolidation (SaProt with Sequence Masking).ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "id": "c517e7b4",
            "metadata": {},
            "source": [
                "# PHIStruct: Improving phage-host interaction prediction at low sequence similarity settings using structure-aware protein embeddings\n",
                "\n",
                "<b>Mark Edward M. Gonzales<sup>1, 2</sup>, Jennifer C. Ureta<sup>1, 2, 3</sup> & Anish M.S. Shrestha<sup>1, 2</sup></b>\n",
                "\n",
                "<sup>1</sup> Bioinformatics Lab, Advanced Research Institute for Informatics, Computing and Networking, De La Salle University, Manila 1004, Philippines <br>\n",
                "<sup>2</sup> Department of Software Technology, College of Computer Studies, De La Salle University, Manila 1004, Philippines <br>\n",
"<sup>3</sup> Walter and Eliza Hall Institute of Medical Research, Melbourne, Victoria, 3052, Australia\n",
                "\n",
                "✉️ gonzales.markedward@gmail.com, jennifer.ureta@gmail.com, anish.shrestha@dlsu.edu.ph"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "54584e67",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "20795e5c",
            "metadata": {},
            "source": [
                "# 💡 Prerequisites\n",
                "\n",
                "### Option 1: Download the prerequisite files\n",
                "1. Download `rbp_saprot_seq_mask_embeddings.tar.gz` from this [link](https://drive.google.com/file/d/1__Rok7MoEbTJ3P8iO3Z-bA7_pFUX4CoO/view?usp=sharing), and unzip it. This should result in a folder named `rbp_saprot_seq_mask_embeddings`.\n",
                "1. Create a folder named `inphared` inside `data`. <br>\n",
                "   Create a folder named `structure`, this time inside `data/inphared`, and save the extracted `rbp_saprot_seq_mask_embeddings` folder inside `data/inphared/structure`.\n",
                "1. Download `consolidated.tar.gz` from this [link](https://drive.google.com/file/d/1yQSXwlb37dm2ZLXGJHdIM5vmrzwPAwvI/view?usp=sharing), and unzip it. This should result in a folder named `consolidated`. <br> Technically, this notebook only needs `consolidated/rbp.csv`.\n",
                "1. Save the extracted `consolidated` folder inside `data/inphared`. \n",
                "\n",
                "### Option 2: Generate the prerequisite files yourself (may take a couple of weeks!)\n",
                "1. Use ColabFold to predict the protein structures of the sequences in `data/inphared/fasta/complete`, following the instructions [here](https://github.com/YoshitakaMo/localcolabfold). <br>Refer to our paper for the parameters at which we ran ColabFold. <br>For reproducibility, we provide the results of running ColabFold [here](https://drive.google.com/file/d/1ZPRdaHwsFOPksLbOyQerREG0gY0p4-AT/view?usp=sharing).\n",
                "1. Encode the predicted structures using SaProt's structure-aware alphabet, following the instructions [here](https://github.com/westlake-repl/SaProt?tab=readme-ov-file#convert-protein-structure-into-structure-aware-sequence). <br>**Make sure to change all the residue (uppercase) tokens to `#`.** <br>For reproducibility, we provide the results of this encoding step [here](https://drive.google.com/file/d/1ziE8krlisUQ_M-jVNYaRSVjyax0Eeai2/view?usp=sharing).\n",
                "1. Feed the results of the encoding step to SaProt in order to generate the structure-aware embeddings, following the instructions [here](https://github.com/westlake-repl/SaProt/issues/14). \n",
                "1. Save each embedding following this naming convention: `<protein_id>_relaxed.r3.pdb.pt`, and consolidate all the embeddings inside a folder named `rbp_saprot_seq_mask_embeddings`.\n",
                "1. Create a folder named `inphared` inside `data`. <br>\n",
                "   Create a folder named `structure`, this time inside `data/inphared`, and save `rbp_saprot_seq_mask_embeddings` inside `data/inphared/structure`.\n",
                "1. Download `consolidated.tar.gz` from this [link](https://drive.google.com/file/d/1yQSXwlb37dm2ZLXGJHdIM5vmrzwPAwvI/view?usp=sharing), and unzip it. This should result in a folder named `consolidated`. <br> Technically, this notebook only needs `consolidated/rbp.csv`.\n",
                "1. Save the extracted `consolidated` folder inside `data/inphared`. \n",
                "\n",
                "### Resulting folder structure\n",
                "\n",
                "`experiments` (parent folder of this notebook) <br> \n",
                "↳ `data` <br>\n",
                "&nbsp; &nbsp;↳ `inphared` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `consolidated` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp.csv` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `structure` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_saprot_seq_mask_embeddings` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `AAA74324.1_relaxed.r3.pdb.pt` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ ... <br>\n",
                "↳ `3.5. Data Consolidation (SaProt with Sequence Masking).ipynb` (this notebook) <br>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "0e827621",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "1c2214db",
            "metadata": {},
            "source": [
                "# 📁 Output files\n",
                "\n",
                "1. If you would like to skip running this notebook, download `rbp_saprot_seq_mask_relaxed_r3.csv` from this [link](https://drive.google.com/file/d/1TTNlUVcaNaWHXMq4n962JTvFEfvGsbVj/view?usp=sharing). This CSV file consolidates the embeddings.\n",
                "\n",
                "1. Create a folder named `inphared` inside `data`. <br>\n",
                "   Create a folder named `structure`, this time inside `data/inphared`, and save `rbp_saprot_seq_mask_relaxed_r3.csv` inside `data/inphared/structure`.\n",
                "\n",
                "1. Download `consolidated.tar.gz` from this [link](https://drive.google.com/file/d/1yQSXwlb37dm2ZLXGJHdIM5vmrzwPAwvI/view?usp=sharing), and unzip it. This should result in a folder named `consolidated`. <br> Technically, this notebook generates only `consolidated/rbp_embeddings_saprot_seq_mask_relaxed_r3.csv`, which consolidates the phage-host information and the embeddings.\n",
                "\n",
                "1. Save the extracted `consolidated` folder inside `data/inphared`. \n",
                "\n",
                "\n",
                "### Resulting folder structure\n",
                "\n",
                "`experiments` (parent folder of this notebook) <br> \n",
                "↳ `data` <br>\n",
                "&nbsp; &nbsp;↳ `inphared` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `consolidated` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp.csv` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_embeddings_saprot_seq_mask_relaxed_r3.csv` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `structure` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_saprot_seq_mask_embeddings` <br>\n",
                "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_saprot_seq_mask_relaxed_r3.csv` <br>\n",
                "↳ `3.5. Data Consolidation (SaProt with Sequence Masking).ipynb` (this notebook) <br>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "6d3364fd",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "827178f8",
            "metadata": {},
            "source": [
                "# Part I: Preliminaries"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "6c907ccc",
            "metadata": {},
            "source": [
                "Import the necessary libraries and modules."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "id": "4d080e9b",
            "metadata": {},
            "outputs": [],
            "source": [
                "import pandas as pd\n",
                "\n",
                "import ConstantsUtil\n",
                "import StructureUtil\n",
                "\n",
                "%load_ext autoreload\n",
                "%autoreload 2"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "94773cb6",
            "metadata": {},
            "outputs": [],
            "source": [
                "constants = ConstantsUtil.ConstantsUtil()\n",
                "util = StructureUtil.StructureUtil()"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "2e0b8146",
            "metadata": {},
            "source": [
                "<hr>"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "7d1479c0",
            "metadata": {},
            "source": [
                "# Part II: Consolidation of the SaProt embeddings"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "7f41fb2b",
            "metadata": {},
            "source": [
                "Consolidate the embeddings into a single data frame."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "abdd565a",
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "100%|████████████████████████████████████████████████████████████████████████████| 28977/28977 [27:28<00:00, 17.57it/s]\n"
                    ]
                },
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>Protein ID</th>\n",
                            "      <th>s1</th>\n",
                            "      <th>s2</th>\n",
                            "      <th>s3</th>\n",
                            "      <th>s4</th>\n",
                            "      <th>s5</th>\n",
                            "      <th>s6</th>\n",
                            "      <th>s7</th>\n",
                            "      <th>s8</th>\n",
                            "      <th>s9</th>\n",
                            "      <th>...</th>\n",
                            "      <th>s1271</th>\n",
                            "      <th>s1272</th>\n",
                            "      <th>s1273</th>\n",
                            "      <th>s1274</th>\n",
                            "      <th>s1275</th>\n",
                            "      <th>s1276</th>\n",
                            "      <th>s1277</th>\n",
                            "      <th>s1278</th>\n",
                            "      <th>s1279</th>\n",
                            "      <th>s1280</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>AAA74324.1</td>\n",
                            "      <td>0.005713</td>\n",
                            "      <td>-0.002139</td>\n",
                            "      <td>0.013508</td>\n",
                            "      <td>-0.021627</td>\n",
                            "      <td>0.040859</td>\n",
                            "      <td>-0.055532</td>\n",
                            "      <td>-0.006189</td>\n",
                            "      <td>-0.006692</td>\n",
                            "      <td>0.004894</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.046600</td>\n",
                            "      <td>-0.008128</td>\n",
                            "      <td>-0.055793</td>\n",
                            "      <td>0.085335</td>\n",
                            "      <td>0.048325</td>\n",
                            "      <td>-0.007057</td>\n",
                            "      <td>0.015243</td>\n",
                            "      <td>0.036059</td>\n",
                            "      <td>0.018610</td>\n",
                            "      <td>-0.052620</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>AAA74331.1</td>\n",
                            "      <td>0.018215</td>\n",
                            "      <td>-0.001640</td>\n",
                            "      <td>0.017582</td>\n",
                            "      <td>-0.055588</td>\n",
                            "      <td>0.013366</td>\n",
                            "      <td>-0.038805</td>\n",
                            "      <td>0.002043</td>\n",
                            "      <td>0.054023</td>\n",
                            "      <td>0.013386</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.022242</td>\n",
                            "      <td>-0.040679</td>\n",
                            "      <td>-0.024889</td>\n",
                            "      <td>0.069150</td>\n",
                            "      <td>0.055251</td>\n",
                            "      <td>0.043178</td>\n",
                            "      <td>0.005399</td>\n",
                            "      <td>0.049476</td>\n",
                            "      <td>0.030549</td>\n",
                            "      <td>0.003810</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>AAA98578.2</td>\n",
                            "      <td>0.016032</td>\n",
                            "      <td>0.040192</td>\n",
                            "      <td>-0.002418</td>\n",
                            "      <td>-0.020064</td>\n",
                            "      <td>0.035645</td>\n",
                            "      <td>-0.026979</td>\n",
                            "      <td>-0.003795</td>\n",
                            "      <td>-0.004521</td>\n",
                            "      <td>0.015021</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.054726</td>\n",
                            "      <td>-0.017037</td>\n",
                            "      <td>-0.039280</td>\n",
                            "      <td>0.072694</td>\n",
                            "      <td>0.050315</td>\n",
                            "      <td>0.004953</td>\n",
                            "      <td>-0.011000</td>\n",
                            "      <td>0.052622</td>\n",
                            "      <td>0.029711</td>\n",
                            "      <td>-0.011149</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>AAB09218.1</td>\n",
                            "      <td>0.005787</td>\n",
                            "      <td>0.029028</td>\n",
                            "      <td>0.014474</td>\n",
                            "      <td>0.008848</td>\n",
                            "      <td>0.004100</td>\n",
                            "      <td>-0.016634</td>\n",
                            "      <td>-0.019308</td>\n",
                            "      <td>0.012828</td>\n",
                            "      <td>0.009144</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.029474</td>\n",
                            "      <td>0.008304</td>\n",
                            "      <td>-0.041756</td>\n",
                            "      <td>0.076812</td>\n",
                            "      <td>-0.010433</td>\n",
                            "      <td>0.021119</td>\n",
                            "      <td>-0.004414</td>\n",
                            "      <td>0.039085</td>\n",
                            "      <td>0.001066</td>\n",
                            "      <td>-0.032709</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>AAB70057.1</td>\n",
                            "      <td>-0.002437</td>\n",
                            "      <td>0.007826</td>\n",
                            "      <td>0.029049</td>\n",
                            "      <td>-0.011243</td>\n",
                            "      <td>-0.010300</td>\n",
                            "      <td>-0.054436</td>\n",
                            "      <td>-0.011496</td>\n",
                            "      <td>0.044409</td>\n",
                            "      <td>0.031361</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.036546</td>\n",
                            "      <td>-0.002583</td>\n",
                            "      <td>-0.039133</td>\n",
                            "      <td>0.082181</td>\n",
                            "      <td>0.032020</td>\n",
                            "      <td>-0.001363</td>\n",
                            "      <td>0.025395</td>\n",
                            "      <td>0.036071</td>\n",
                            "      <td>-0.005219</td>\n",
                            "      <td>-0.026583</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>5 rows × 1281 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   Protein ID        s1        s2        s3        s4        s5        s6  \\\n",
                            "0  AAA74324.1  0.005713 -0.002139  0.013508 -0.021627  0.040859 -0.055532   \n",
                            "1  AAA74331.1  0.018215 -0.001640  0.017582 -0.055588  0.013366 -0.038805   \n",
                            "2  AAA98578.2  0.016032  0.040192 -0.002418 -0.020064  0.035645 -0.026979   \n",
                            "3  AAB09218.1  0.005787  0.029028  0.014474  0.008848  0.004100 -0.016634   \n",
                            "4  AAB70057.1 -0.002437  0.007826  0.029049 -0.011243 -0.010300 -0.054436   \n",
                            "\n",
                            "         s7        s8        s9  ...     s1271     s1272     s1273     s1274  \\\n",
                            "0 -0.006189 -0.006692  0.004894  ... -0.046600 -0.008128 -0.055793  0.085335   \n",
                            "1  0.002043  0.054023  0.013386  ... -0.022242 -0.040679 -0.024889  0.069150   \n",
                            "2 -0.003795 -0.004521  0.015021  ... -0.054726 -0.017037 -0.039280  0.072694   \n",
                            "3 -0.019308  0.012828  0.009144  ... -0.029474  0.008304 -0.041756  0.076812   \n",
                            "4 -0.011496  0.044409  0.031361  ... -0.036546 -0.002583 -0.039133  0.082181   \n",
                            "\n",
                            "      s1275     s1276     s1277     s1278     s1279     s1280  \n",
                            "0  0.048325 -0.007057  0.015243  0.036059  0.018610 -0.052620  \n",
                            "1  0.055251  0.043178  0.005399  0.049476  0.030549  0.003810  \n",
                            "2  0.050315  0.004953 -0.011000  0.052622  0.029711 -0.011149  \n",
                            "3 -0.010433  0.021119 -0.004414  0.039085  0.001066 -0.032709  \n",
                            "4  0.032020 -0.001363  0.025395  0.036071 -0.005219 -0.026583  \n",
                            "\n",
                            "[5 rows x 1281 columns]"
                        ]
                    },
                    "execution_count": 3,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "rbp_saprot_seq_mask_relaxed_r3 = util.convert_saprot_pt_to_df(\n",
                "    f\"{constants.INPHARED}/{constants.STRUCTURE_SAPROT_SEQ_MASK}\", \"_relaxed.r3\"\n",
                ")\n",
                "rbp_saprot_seq_mask_relaxed_r3.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "id": "271f79d8",
            "metadata": {},
            "outputs": [],
            "source": [
                "rbp_saprot_seq_mask_relaxed_r3.to_csv(\n",
                "    f\"{constants.INPHARED}/{constants.CSV_SAPROT_SEQ_MASK}_relaxed_r3.csv\",\n",
                "    index=False,\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "f2fd5da1",
            "metadata": {},
            "source": [
                "Combine the embeddings data frame with the data frame containing the phage-host information."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "id": "2a3fbbae",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>Protein ID</th>\n",
                            "      <th>Accession</th>\n",
                            "      <th>Description</th>\n",
                            "      <th>Classification</th>\n",
                            "      <th>Genome Length (bp)</th>\n",
                            "      <th>Jumbophage</th>\n",
                            "      <th>molGC (%)</th>\n",
                            "      <th>Molecule</th>\n",
                            "      <th>Modification Date</th>\n",
                            "      <th>Number CDS</th>\n",
                            "      <th>...</th>\n",
                            "      <th>Genbank Division</th>\n",
                            "      <th>Isolation Host (beware inconsistent and nonsense values)</th>\n",
                            "      <th>Host Superkingdom</th>\n",
                            "      <th>Host Phylum</th>\n",
                            "      <th>Host Class</th>\n",
                            "      <th>Host Order</th>\n",
                            "      <th>Host Family</th>\n",
                            "      <th>Year-Month</th>\n",
                            "      <th>Protein Sequence</th>\n",
                            "      <th>Nucleotide Sequence</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>BAD16801.1</td>\n",
                            "      <td>AB161975</td>\n",
                            "      <td>Wolbachia phage WOcauB1</td>\n",
                            "      <td>Wolbachia phage WOcauB1 Viruses</td>\n",
                            "      <td>20484</td>\n",
                            "      <td>False</td>\n",
                            "      <td>36.878</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2016-07-26</td>\n",
                            "      <td>27</td>\n",
                            "      <td>...</td>\n",
                            "      <td>PHG</td>\n",
                            "      <td>Wolbachia sp. wCauB</td>\n",
                            "      <td>bacteria</td>\n",
                            "      <td>pseudomonadota</td>\n",
                            "      <td>alphaproteobacteria</td>\n",
                            "      <td>rickettsiales</td>\n",
                            "      <td>anaplasmataceae</td>\n",
                            "      <td>2016-07</td>\n",
                            "      <td>MKEAIYQRIKDLAANSTPDQLAYLAKSLELIADKKAISNVVQMTEV...</td>\n",
                            "      <td>ATGAAAGAAGCAATATACCAAAGGATAAAGGATTTAGCAGCAAATA...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>BAF36105.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>PHG</td>\n",
                            "      <td>Unspecified</td>\n",
                            "      <td>bacteria</td>\n",
                            "      <td>cyanobacteriota</td>\n",
                            "      <td>cyanophyceae</td>\n",
                            "      <td>chroococcales</td>\n",
                            "      <td>microcystaceae</td>\n",
                            "      <td>2023-08</td>\n",
                            "      <td>MHQNISKENRGNYNNGIRPRIFMITTIDFRDIQAACIKQLDDMSKD...</td>\n",
                            "      <td>GTGCATCAAAATATTTCAAAGGAGAATCGTGGAAACTATAACAACG...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>BAF36110.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>PHG</td>\n",
                            "      <td>Unspecified</td>\n",
                            "      <td>bacteria</td>\n",
                            "      <td>cyanobacteriota</td>\n",
                            "      <td>cyanophyceae</td>\n",
                            "      <td>chroococcales</td>\n",
                            "      <td>microcystaceae</td>\n",
                            "      <td>2023-08</td>\n",
                            "      <td>MRIFYIHHPFLATHRYLLSNAYSTPYTDSITKLTTSYSSMPIILSV...</td>\n",
                            "      <td>GTGAGGATTTTTTATATCCACCATCCATTCCTCGCTACTCACCGAT...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>BAF36131.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>PHG</td>\n",
                            "      <td>Unspecified</td>\n",
                            "      <td>bacteria</td>\n",
                            "      <td>cyanobacteriota</td>\n",
                            "      <td>cyanophyceae</td>\n",
                            "      <td>chroococcales</td>\n",
                            "      <td>microcystaceae</td>\n",
                            "      <td>2023-08</td>\n",
                            "      <td>MLTDVDIQALIDASISGLSGEMPIVANIAARNALSLTKNTQVLVLD...</td>\n",
                            "      <td>TTGCTGACAGATGTCGATATTCAGGCATTAATTGATGCCTCAATTT...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>BAF36132.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>PHG</td>\n",
                            "      <td>Unspecified</td>\n",
                            "      <td>bacteria</td>\n",
                            "      <td>cyanobacteriota</td>\n",
                            "      <td>cyanophyceae</td>\n",
                            "      <td>chroococcales</td>\n",
                            "      <td>microcystaceae</td>\n",
                            "      <td>2023-08</td>\n",
                            "      <td>MFGVFIVRREGGYIGTQPNWDAANRPGNWDILDVYNRQRRNLWIQS...</td>\n",
                            "      <td>TTGTTCGGAGTTTTTATCGTGAGGCGTGAAGGCGGCTATATCGGAA...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>5 rows × 36 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   Protein ID Accession              Description  \\\n",
                            "0  BAD16801.1  AB161975  Wolbachia phage WOcauB1   \n",
                            "1  BAF36105.1  AB231700  Microcystis phage LMM01   \n",
                            "2  BAF36110.1  AB231700  Microcystis phage LMM01   \n",
                            "3  BAF36131.1  AB231700  Microcystis phage LMM01   \n",
                            "4  BAF36132.1  AB231700  Microcystis phage LMM01   \n",
                            "\n",
                            "                                      Classification  Genome Length (bp)  \\\n",
                            "0                    Wolbachia phage WOcauB1 Viruses               20484   \n",
                            "1  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "2  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "3  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "4  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "\n",
                            "   Jumbophage  molGC (%) Molecule Modification Date  Number CDS  ...  \\\n",
                            "0       False     36.878      DNA        2016-07-26          27  ...   \n",
                            "1       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "2       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "3       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "4       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "\n",
                            "   Genbank Division  Isolation Host (beware inconsistent and nonsense values)  \\\n",
                            "0               PHG                                Wolbachia sp. wCauB          \n",
                            "1               PHG                                        Unspecified          \n",
                            "2               PHG                                        Unspecified          \n",
                            "3               PHG                                        Unspecified          \n",
                            "4               PHG                                        Unspecified          \n",
                            "\n",
                            "   Host Superkingdom      Host Phylum           Host Class     Host Order  \\\n",
                            "0           bacteria   pseudomonadota  alphaproteobacteria  rickettsiales   \n",
                            "1           bacteria  cyanobacteriota         cyanophyceae  chroococcales   \n",
                            "2           bacteria  cyanobacteriota         cyanophyceae  chroococcales   \n",
                            "3           bacteria  cyanobacteriota         cyanophyceae  chroococcales   \n",
                            "4           bacteria  cyanobacteriota         cyanophyceae  chroococcales   \n",
                            "\n",
                            "       Host Family Year-Month  \\\n",
                            "0  anaplasmataceae    2016-07   \n",
                            "1   microcystaceae    2023-08   \n",
                            "2   microcystaceae    2023-08   \n",
                            "3   microcystaceae    2023-08   \n",
                            "4   microcystaceae    2023-08   \n",
                            "\n",
                            "                                    Protein Sequence  \\\n",
                            "0  MKEAIYQRIKDLAANSTPDQLAYLAKSLELIADKKAISNVVQMTEV...   \n",
                            "1  MHQNISKENRGNYNNGIRPRIFMITTIDFRDIQAACIKQLDDMSKD...   \n",
                            "2  MRIFYIHHPFLATHRYLLSNAYSTPYTDSITKLTTSYSSMPIILSV...   \n",
                            "3  MLTDVDIQALIDASISGLSGEMPIVANIAARNALSLTKNTQVLVLD...   \n",
                            "4  MFGVFIVRREGGYIGTQPNWDAANRPGNWDILDVYNRQRRNLWIQS...   \n",
                            "\n",
                            "                                 Nucleotide Sequence  \n",
                            "0  ATGAAAGAAGCAATATACCAAAGGATAAAGGATTTAGCAGCAAATA...  \n",
                            "1  GTGCATCAAAATATTTCAAAGGAGAATCGTGGAAACTATAACAACG...  \n",
                            "2  GTGAGGATTTTTTATATCCACCATCCATTCCTCGCTACTCACCGAT...  \n",
                            "3  TTGCTGACAGATGTCGATATTCAGGCATTAATTGATGCCTCAATTT...  \n",
                            "4  TTGTTCGGAGTTTTTATCGTGAGGCGTGAAGGCGGCTATATCGGAA...  \n",
                            "\n",
                            "[5 rows x 36 columns]"
                        ]
                    },
                    "execution_count": 5,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "rbp_nucleotide_seq = pd.read_csv(\n",
                "    f\"{constants.INPHARED}/{constants.CONSOLIDATED}/{constants.INPHARED_RBP_DATA}\"\n",
                ")\n",
                "rbp_nucleotide_seq.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "id": "389c8f3d",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>Protein ID</th>\n",
                            "      <th>Accession</th>\n",
                            "      <th>Description</th>\n",
                            "      <th>Classification</th>\n",
                            "      <th>Genome Length (bp)</th>\n",
                            "      <th>Jumbophage</th>\n",
                            "      <th>molGC (%)</th>\n",
                            "      <th>Molecule</th>\n",
                            "      <th>Modification Date</th>\n",
                            "      <th>Number CDS</th>\n",
                            "      <th>...</th>\n",
                            "      <th>s1271</th>\n",
                            "      <th>s1272</th>\n",
                            "      <th>s1273</th>\n",
                            "      <th>s1274</th>\n",
                            "      <th>s1275</th>\n",
                            "      <th>s1276</th>\n",
                            "      <th>s1277</th>\n",
                            "      <th>s1278</th>\n",
                            "      <th>s1279</th>\n",
                            "      <th>s1280</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>BAD16801.1</td>\n",
                            "      <td>AB161975</td>\n",
                            "      <td>Wolbachia phage WOcauB1</td>\n",
                            "      <td>Wolbachia phage WOcauB1 Viruses</td>\n",
                            "      <td>20484</td>\n",
                            "      <td>False</td>\n",
                            "      <td>36.878</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2016-07-26</td>\n",
                            "      <td>27</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.051533</td>\n",
                            "      <td>-0.022794</td>\n",
                            "      <td>-0.064905</td>\n",
                            "      <td>0.076057</td>\n",
                            "      <td>0.044809</td>\n",
                            "      <td>0.028928</td>\n",
                            "      <td>-0.003874</td>\n",
                            "      <td>0.027649</td>\n",
                            "      <td>0.063658</td>\n",
                            "      <td>-0.040593</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>BAF36105.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.033253</td>\n",
                            "      <td>-0.027492</td>\n",
                            "      <td>-0.053185</td>\n",
                            "      <td>0.081067</td>\n",
                            "      <td>0.037140</td>\n",
                            "      <td>0.031542</td>\n",
                            "      <td>-0.060938</td>\n",
                            "      <td>0.044506</td>\n",
                            "      <td>0.060620</td>\n",
                            "      <td>0.005437</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>BAF36110.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.044021</td>\n",
                            "      <td>-0.022810</td>\n",
                            "      <td>-0.114099</td>\n",
                            "      <td>0.141799</td>\n",
                            "      <td>0.018311</td>\n",
                            "      <td>0.044526</td>\n",
                            "      <td>-0.018967</td>\n",
                            "      <td>0.014370</td>\n",
                            "      <td>0.043462</td>\n",
                            "      <td>-0.090030</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>BAF36131.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.022680</td>\n",
                            "      <td>0.022862</td>\n",
                            "      <td>-0.063694</td>\n",
                            "      <td>0.089508</td>\n",
                            "      <td>0.019966</td>\n",
                            "      <td>0.006555</td>\n",
                            "      <td>-0.004751</td>\n",
                            "      <td>0.026278</td>\n",
                            "      <td>-0.007740</td>\n",
                            "      <td>-0.044939</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>BAF36132.1</td>\n",
                            "      <td>AB231700</td>\n",
                            "      <td>Microcystis phage LMM01</td>\n",
                            "      <td>Microcystis phage LMM01 Fukuivirus LMM01 Fukui...</td>\n",
                            "      <td>162109</td>\n",
                            "      <td>False</td>\n",
                            "      <td>45.953</td>\n",
                            "      <td>DNA</td>\n",
                            "      <td>2023-08-22</td>\n",
                            "      <td>189</td>\n",
                            "      <td>...</td>\n",
                            "      <td>-0.034211</td>\n",
                            "      <td>0.007903</td>\n",
                            "      <td>-0.047379</td>\n",
                            "      <td>0.079803</td>\n",
                            "      <td>0.025154</td>\n",
                            "      <td>-0.020763</td>\n",
                            "      <td>0.035391</td>\n",
                            "      <td>0.039050</td>\n",
                            "      <td>0.029944</td>\n",
                            "      <td>-0.051824</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "<p>5 rows × 1316 columns</p>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   Protein ID Accession              Description  \\\n",
                            "0  BAD16801.1  AB161975  Wolbachia phage WOcauB1   \n",
                            "1  BAF36105.1  AB231700  Microcystis phage LMM01   \n",
                            "2  BAF36110.1  AB231700  Microcystis phage LMM01   \n",
                            "3  BAF36131.1  AB231700  Microcystis phage LMM01   \n",
                            "4  BAF36132.1  AB231700  Microcystis phage LMM01   \n",
                            "\n",
                            "                                      Classification  Genome Length (bp)  \\\n",
                            "0                    Wolbachia phage WOcauB1 Viruses               20484   \n",
                            "1  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "2  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "3  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "4  Microcystis phage LMM01 Fukuivirus LMM01 Fukui...              162109   \n",
                            "\n",
                            "   Jumbophage  molGC (%) Molecule Modification Date  Number CDS  ...  \\\n",
                            "0       False     36.878      DNA        2016-07-26          27  ...   \n",
                            "1       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "2       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "3       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "4       False     45.953      DNA        2023-08-22         189  ...   \n",
                            "\n",
                            "      s1271     s1272     s1273     s1274     s1275     s1276     s1277  \\\n",
                            "0 -0.051533 -0.022794 -0.064905  0.076057  0.044809  0.028928 -0.003874   \n",
                            "1 -0.033253 -0.027492 -0.053185  0.081067  0.037140  0.031542 -0.060938   \n",
                            "2 -0.044021 -0.022810 -0.114099  0.141799  0.018311  0.044526 -0.018967   \n",
                            "3 -0.022680  0.022862 -0.063694  0.089508  0.019966  0.006555 -0.004751   \n",
                            "4 -0.034211  0.007903 -0.047379  0.079803  0.025154 -0.020763  0.035391   \n",
                            "\n",
                            "      s1278     s1279     s1280  \n",
                            "0  0.027649  0.063658 -0.040593  \n",
                            "1  0.044506  0.060620  0.005437  \n",
                            "2  0.014370  0.043462 -0.090030  \n",
                            "3  0.026278 -0.007740 -0.044939  \n",
                            "4  0.039050  0.029944 -0.051824  \n",
                            "\n",
                            "[5 rows x 1316 columns]"
                        ]
                    },
                    "execution_count": 6,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "rbp_structure_embeddings_relaxed_r3 = pd.merge(\n",
                "    rbp_nucleotide_seq,\n",
                "    rbp_saprot_seq_mask_relaxed_r3,\n",
                "    how=\"inner\",\n",
                "    validate=\"one_to_one\",\n",
                "    on=\"Protein ID\",\n",
                ")\n",
                "rbp_structure_embeddings_relaxed_r3.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "id": "f14da9ca",
            "metadata": {},
            "outputs": [],
            "source": [
                "rbp_structure_embeddings_relaxed_r3.to_csv(\n",
                "    f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_saprot_seq_mask_relaxed_r3.csv\",\n",
                "    index=False,\n",
                ")"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3 (ipykernel)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.10.9"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 5
}