edreisMD · Mar 25, 2021 · Mar 27, 2021 · Mar 30, 2021 · Apr 6, 2021 · Apr 7, 2021
diff --git a/.DS_Store b/.DS_Store
diff --git a/LICENSE b/LICENSE
diff --git a/README.md b/README.md
@@ -1,11 +1,84 @@
-Contrastive Learning Representations of Images and Text pairs
+# Colab Implement: ConVIRT model
+### Contrastive VIsual Representation Learning from Text
 
-### Pytorch implementation of the architecture descibed in the ConVIRT paper: Contrastive Learning of Medical Visual Representations from Paired Images and Text
-Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, Curtis P. Langlotz
+Deep neural networks learn from a large amount of data to obtain the correct parameters to perform a speciﬁc task. However, in practice, we often encounter a problem: **insuﬃcient amount of labeled data**. However, if your data contains pairs of images and text, you can solve the problem with contrastive learning. 
 
-Non-official open source release by Eduardo Reis.
+Contrastive learning is a kind of self-supervised learning method. It does not require specialized labels, but rather a method to learn the correct parameters from the unlabeled data itself. It aims to learn an encoder that makes the encoding results of similar classes of data similar and makes the encoding results of diﬀerent classes of data as diﬀerent as possible. Typical contrast learning is done based on comparisons between two images. However, if we have paired image and text data, contrast learning can also be applied between images and text.
 
-Note: This repository was forked and modified from https://github.com/sthalles/SimCLR.
+Based on this repository, we can implement various paired-image-text Contrastive Learning tasks on [Google Colab](https://colab.research.google.com/notebooks/intro.ipynb), which enable you to train effective pre-training models for transfer learning with insufficient data volume. With this pre-trained model, you can train with less labeled data to get a good performing model.
+
+
+
+## Usage
+
+### 1. Data Preparation
+
+Before starting training, we need to download the training data and make them can be read in pairs. 
+
+There are two example of data preparation:
+- Local based: [**data_prepare_MIMIC.ipynb**](https://github.com/waittim/ConVIRT-Colab/blob/master/data_prepare_MIMIC.ipynb)
+- Colab based: [**data_prepare_openi.ipynb**](https://github.com/waittim/ConVIRT-Colab/blob/master/data_prepare_openi.ipynb)
+
+After preparation, there should be a CSV file which contains image path and text file path for each paired-image-text. (Or we can save the text content in the CSV file directly.)
+
+### 2. Define Configuration
+
+In **config.yaml**, we need to define the training hyperperemeter, the data path, and the base models. Here is an example:
+
+```
+batch_size: 32
+epochs: 1000
+eval_every_n_epochs: 5
+fine_tune_from: Jan16_02-27-36_edu-GPU-Linux
+log_every_n_steps: 2
+learning_rate: 1e-4
+weight_decay: 1e-6
+fp16_precision: True
+truncation: True
+
+model:
+  out_dim: 512
+  res_base_model: "resnet50"
+  bert_base_model: 'emilyalsentzer/Bio_ClinicalBERT'
+  freeze_layers: [0,1,2,3,4,5]
+  do_lower_case: False
+  
+dataset:
+  s: 1
+  input_shape: (224,224,3)
+  num_workers: 4
+  valid_size: 0.1
+  csv_file: 'path/for/CSV_containing_paths_for_images_and_text.csv' 
+  text_from_files: True # If 'True' the text input will be read from .txt files, if 'False' it will be loaded direct from the CSV File 
+  img_root_dir: '/your/root/images/directory'
+  text_root_dir: '/your/root/text/directory' # The root directory for the text files if "text_from_files" is True
+  img_path_col: 0 # index for the image path column in the CSV dataframe.
+  text_col: 1 # index for the text column in the CSV dataframe. If text_from_files is 'True' it should contain the relative path for the files from the 'text_root_dir', if text_from_files is 'False' this column should contain the respective input text in its own cells.
+
+loss:
+  temperature: 0.1
+  use_cosine_similarity: True
+  alpha_weight: 0.75
+```
+
+The models used [res_base_model, bert_base_model] refers to the models provided by [transformers](https://huggingface.co/transformers/).
+
+### 3. Training
+
+For training in the Colab, please open [**Setup.ipynb**](https://github.com/waittim/ConVIRT-Colab/blob/master/Setup.ipynb), then follow the introduction inside.
+
+After run the code `python run.py` in the notebook, you can open another notebook [**tensorboard.ipynb**](https://github.com/waittim/ConVIRT-Colab/blob/master/tensorboard.ipynb) to monitor the training process.
+
+### 4. After Training
+
+At the end of training, the final model and the corresponding config.yaml will be saved to `./runs/`. Please use this model for transfer learning.
+
+
+## Others
+
+The repository is a Colab implementation of the architecture descibed in the ConVIRT paper: [*Contrastive Learning of Medical Visual Representations from Paired Images and Text*](https://arxiv.org/abs/2010.00747). The authors of paper are Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, Curtis P. Langlotz.
+
+This repository was originally modified from https://github.com/sthalles/SimCLR.
 
 References: 
 - Yuhao Zhang et al. Contrastive Learning of Medical Visual Representations from Paired Images and Text. https://arxiv.org/pdf/2010.00747.pdf

diff --git a/Setup.ipynb b/Setup.ipynb
@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Setup.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"-0RfVLkySLVH"},"source":["# Setup\n","\n","This notebook will help you train the contrastive learning model. Before running this notebook, please make sure you have modified the **config.yaml**.\n","\n","## Set up Google Colab environment\n","\n","For running the model in Colab, we need to mount to the path which we cloned the repository."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"WkriP92AwoTS","executionInfo":{"status":"ok","timestamp":1619598644306,"user_tz":300,"elapsed":30755,"user":{"displayName":"Zekun Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh140SMb0c80XwwVdi_E7-Lh2XNkXKVWOakF5qL=s64","userId":"05904147758498762863"}},"outputId":"71159307-c101-4f6d-e379-8f63de8d91b2"},"source":["from google.colab import drive\n","drive.mount('/content/drive')"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Mounted at /content/drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cV6DD-KEw4Xd","executionInfo":{"status":"ok","timestamp":1619598644307,"user_tz":300,"elapsed":30738,"user":{"displayName":"Zekun Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh140SMb0c80XwwVdi_E7-Lh2XNkXKVWOakF5qL=s64","userId":"05904147758498762863"}},"outputId":"d3dcb90c-255c-41b8-9b70-c60fbac91b38"},"source":["import os\n","cur_path = \"/content/drive/MyDrive/ConVIRT-Colab/\"\n","os.chdir(cur_path)\n","!pwd"],"execution_count":2,"outputs":[{"output_type":"stream","text":["/content/drive/MyDrive/ConVIRT-Colab\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"koVGfbLoSLVO"},"source":["Install necessary modules."]},{"cell_type":"code","metadata":{"id":"9_8LsxJefS0a"},"source":["!pip install pyyaml==5.3.1\n","!pip install transformers"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FQ6-wS0xIp_h"},"source":["## Training\n","\n","After running the following code, the training will start. You will need a long time to wait for the result."]},{"cell_type":"code","metadata":{"id":"lHPFCPmrZA-n"},"source":["!python run.py"],"execution_count":null,"outputs":[]},{"source":["During the process, you can check **tensorboard.ipynb** to monitor the training."],"cell_type":"markdown","metadata":{"id":"7VlsYuCpZeJt"}},{"cell_type":"code","metadata":{"id":"mztA6yN08lNc"},"source":[],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Omz4wUq-8l1D"},"source":[],"execution_count":null,"outputs":[]}]}
diff --git a/config.yaml b/config.yaml
@@ -1,9 +1,9 @@
-batch_size: 32
+batch_size: 64 #32
 epochs: 1000
 eval_every_n_epochs: 5
 fine_tune_from: Jan16_02-27-36_edu-GPU-Linux
 log_every_n_steps: 2
-learning_rate: 1e-4
+learning_rate: 5e-4 #1e-4
 weight_decay: 1e-6
 fp16_precision: True
 truncation: True
@@ -19,13 +19,13 @@ dataset:
   s: 1
   input_shape: (224,224,3)
   num_workers: 4
-  valid_size: 0.1
-  csv_file: 'path/for/CSV_containing_MIMIC-CXR_paths_for_images_and_text.csv'
+  valid_size: 0.25
+  csv_file: '/content/drive/MyDrive/Dataset/open-i/data/image_txt.csv' # 'path/for/CSV_containing_MIMIC-CXR_paths_for_images_and_text.csv'
   text_from_files: True # If 'True' the text input will be read from .txt files, if 'False' it will be loaded direct from the CSV File 
-  img_root_dir: '/your/root/images/directory'
-  text_root_dir: '/your/root/text/directory' # The root directory for the text files if "text_from_files" is True
+  img_root_dir: '/content/drive/MyDrive/Dataset/open-i/data/NLMCXR_png/' # '/your/root/images/directory'
+  text_root_dir: '/content/drive/MyDrive/Dataset/open-i/data/ecgen-radiology-txt/' # '/your/root/text/directory' # The root directory for the text files if "text_from_files" is True
   img_path_col: 0 # index for the image path column in the CSV dataframe.
-  text_col: 5 # index for the text column in the CSV dataframe. If text_from_files is 'True' it should contain the relative path for the files from the 'text_root_dir', if text_from_files is 'False' this column should contain the respective input text in its own cells.
+  text_col: 1 # index for the text column in the CSV dataframe. If text_from_files is 'True' it should contain the relative path for the files from the 'text_root_dir', if text_from_files is 'False' this column should contain the respective input text in its own cells.
 
 loss:
   temperature: 0.1
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Setup.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"-0RfVLkySLVH"},"source":["# Setup\n","\n","This notebook will help you train the contrastive learning model. Before running this notebook, please make sure you have modified the config.yaml.\n","\n","## Set up Google Colab environment\n","\n","For running the model in Colab, we need to mount to the path which we cloned the repository."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"WkriP92AwoTS","executionInfo":{"status":"ok","timestamp":1619598644306,"user_tz":300,"elapsed":30755,"user":{"displayName":"Zekun Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh140SMb0c80XwwVdi_E7-Lh2XNkXKVWOakF5qL=s64","userId":"05904147758498762863"}},"outputId":"71159307-c101-4f6d-e379-8f63de8d91b2"},"source":["from google.colab import drive\n","drive.mount('/content/drive')"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Mounted at /content/drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cV6DD-KEw4Xd","executionInfo":{"status":"ok","timestamp":1619598644307,"user_tz":300,"elapsed":30738,"user":{"displayName":"Zekun Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh140SMb0c80XwwVdi_E7-Lh2XNkXKVWOakF5qL=s64","userId":"05904147758498762863"}},"outputId":"d3dcb90c-255c-41b8-9b70-c60fbac91b38"},"source":["import os\n","cur_path = \"/content/drive/MyDrive/ConVIRT-Colab/\"\n","os.chdir(cur_path)\n","!pwd"],"execution_count":2,"outputs":[{"output_type":"stream","text":["/content/drive/MyDrive/ConVIRT-Colab\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"koVGfbLoSLVO"},"source":["Install necessary modules."]},{"cell_type":"code","metadata":{"id":"9_8LsxJefS0a"},"source":["!pip install pyyaml==5.3.1\n","!pip install transformers"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FQ6-wS0xIp_h"},"source":["## Training\n","\n","After running the following code, the training will start. You will need a long time to wait for the result."]},{"cell_type":"code","metadata":{"id":"lHPFCPmrZA-n"},"source":["!python run.py"],"execution_count":null,"outputs":[]},{"source":["During the process, you can check tensorboard.ipynb to monitor the training."],"cell_type":"markdown","metadata":{"id":"7VlsYuCpZeJt"}},{"cell_type":"code","metadata":{"id":"mztA6yN08lNc"},"source":[],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Omz4wUq-8l1D"},"source":[],"execution_count":null,"outputs":[]}]}