diff --git a/README.md b/README.md index f447ce6e7..5958e0d34 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ and production. ## Install -Make sure you have Python 3.7+ and one of Pytorch, Keras or PaddlePaddle installed on Linux/MacOS. +Make sure you have Python 3.7+ and one of Pytorch (>=1.9), Tensorflow (>=2.5) or PaddlePaddle installed on Linux/MacOS. ```bash pip install finetuner diff --git a/docs/get-started/celeba-labeler.gif b/docs/get-started/celeba-labeler.gif new file mode 100644 index 000000000..dc0cdf3dc Binary files /dev/null and b/docs/get-started/celeba-labeler.gif differ diff --git a/docs/get-started/celeba.md b/docs/get-started/celeba.md index 3f7ebf9f0..627f63b43 100644 --- a/docs/get-started/celeba.md +++ b/docs/get-started/celeba.md @@ -1,4 +1,8 @@ -# Finetuning Pre-Trained ResNet on CelebA Dataset +# Finetuning Pretrained ResNet for Celebrity Face Search + +```{tip} +For this example, you will need a GPU machine to enable the best experience. +``` In this example, we want to "tune" the pre-trained [ResNet](https://arxiv.org/abs/1512.03385) on [CelebA dataset](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). Note that, the original weights of the ResNet model was trained on ImageNet. @@ -14,58 +18,31 @@ Hopefully the procedure converges after several rounds; and we get a tuned embed Let's first make sure you have downloaded all the images [`img_align_celeba.zip`](https://drive.google.com/file/d/0B7EVK8r0v71pZjFTYXZWM3FlRnM/view?usp=sharing&resourcekey=0-dYn9z10tMJOBAkviAcfdyQ) and [`IdentityCelebA.txt`](https://drive.google.com/file/d/1_ee_0u7vcNLOfNLegJRHmolfH5ICW-XS/view?usp=sharing) locally. ```{caution} - -Beware that the original CelebA dataset is 1.3GB. In this example, we do not need the full dataset. Here is a smaller version which contains 1000 images from the original dataset. You can [download it from here](). +Beware that the original CelebA dataset is 1.3GB. In this example, we do not need the full dataset. Here is a smaller version which contains 1000 images from the original dataset. You can [download it from here](https://static.jina.ai/celeba/celeba-img.zip). ``` -Note that Finetuner accepts Jina `DocumentArray`/`DocumentArrayMemmap`, so we first convert CelebA data into this format. - - - -Since each celebrity has multiple facial images, we first create a `defaultdict` and group these images by their identity: +Note that Finetuner accepts Jina `DocumentArray`/`DocumentArrayMemmap`, so we first load CelebA data into this format using generator: ```python -from collections import defaultdict +from jina.types.document.generators import from_files -DATA_PATH = '~/[YOUR-DIRECTORY]/img_align_celeba/' -IDENTITY_PATH = '~/[YOUR-DIRECTORY]/identity_CelebA.txt' - -def group_imgs_by_identity(): - grouped = defaultdict(list) - with open(IDENTITY_PATH, 'r') as f: - for line in f: - img_file_name, identity = line.split() - grouped[identity].append(img_file_name) - return grouped -``` - -Then we create a data generator that yields every image as a `Document` object: - -```python -from jina import Document - -def train_generator(): - for identity, imgs in group_imgs_by_identity().items(): - for img in imgs: - d = Document(uri=DATA_PATH + img) - d.convert_image_uri_to_blob(color_axis=0) - d.convert_uri_to_datauri() - yield d +def data_gen(): + for d in from_files('/Users/jina/Downloads/img_align_celeba/*.jpg', size=100, to_dataturi=True): + d.convert_image_datauri_to_blob(color_axis=0) # no need of tf + yield d ``` -## Download the pretrained model +## Load the pretrained model -Let's import pre-trained ResNet as our {ref}`embedding model` using any of the following frameworks. +Let's import pretrained ResNet50 as our base model. ResNet50 is implemented in Pytorch, Keras and Paddle. You can choose whatever framework you feel comfortable: ````{tab} PyTorch - ```python import torchvision model = torchvision.models.resnet50(pretrained=True) ``` - ```` ````{tab} Keras ```python @@ -83,36 +60,52 @@ model = paddle.vision.models.resnet50(pretrained=True) ```` - - ## Put together -Finally, let's feed the model and the data into the Finetuner: +Finally, let's start the Finetuner. Note that we freeze the weights of the original ResNet and tuned only for the last linear layer, which leverages Tailor underneath: -```python -rv = fit( +```{code-block} python +--- +emphasize-lines: 5, 8 +--- +import finetuner + +finetuner.fit( model=model, interactive=True, - train_data=train_generator, + train_data=data_gen, freeze=True, + to_embedding_model=True, input_size=(3, 224, 224), - output_dim=512, # Chop-off the last fc layer and add a trainable linear layer. + output_dim=100 ) ``` +Note that how we specify `interactive=True` and `to_embedding_model=True` in the above code to activate Labler and Tailor, respectively. + +`input_size` is not required when you using Keras as the backend. + ## Label interactively -You can now label the data by mouse/keyboard. The model will get trained and improved as you are labeling. +After running the script, the browser will open the Labeler UI. You can now label the data by mouse/keyboard. The model will get trained and improved as you are labeling. If you are running this example on a CPU machine, it can take up to 20 seconds for each labeling round. + +```{figure} celeba-labeler.gif +:align: center +``` -From the backend you will see model's training procedure: +On the backend, you should be able to see the training procedure in the terminal. -```bash - Flow@22900[I]:πŸŽ‰ Flow is ready to use! +```console + Flow@6620[I]:πŸŽ‰ Flow is ready to use! πŸ”— Protocol: HTTP - 🏠 Local access: 0.0.0.0:52621 - πŸ”’ Private network: 172.18.1.109:52621 - 🌐 Public address: 94.135.231.132:52621 - πŸ’¬ Swagger UI: http://localhost:52621/docs - πŸ“š Redoc: http://localhost:52621/redoc - JINA@22900[I]:Finetuner is available at http://localhost:52621/finetuner + 🏠 Local access: 0.0.0.0:61622 + πŸ”’ Private network: 172.18.1.109:61622 + 🌐 Public address: 94.135.231.132:61622 + πŸ’¬ Swagger UI: http://localhost:61622/docs + πŸ“š Redoc: http://localhost:61622/redoc +UserWarning: ignored unknown argument: ['thread']. (raised from /Users/hanxiao/Documents/jina/jina/helper.py:685) +β ΄ Working... ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 estimating... JINA@6620[I]:Finetuner is available at http://localhost:61622/finetuner +⠏ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 0.0 step/s UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:180.) (raised from /Users/hanxiao/Documents/trainer/finetuner/labeler/executor.py:53) +β ¦ DONE ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:06 1.4 step/s 11 steps done in 6 seconds +β ™ DONE ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:03 0.3 step/s T: Loss= 0.75 ``` \ No newline at end of file diff --git a/docs/get-started/covid-qa.md b/docs/get-started/covid-qa.md index 06eff69db..9bc452633 100644 --- a/docs/get-started/covid-qa.md +++ b/docs/get-started/covid-qa.md @@ -1,4 +1,9 @@ -# Finetuning Bi-LSTM on Text +# Finetuning Bi-LSTM for Question-Answering + +```{tip} +This example is inspired by [`jina hello chatbot`](https://docs.jina.ai/get-started/hello-world/covid-19-chatbot/). We stronly recommend you to checkout that demo first before go through this tutorial. +``` + In this example, we want to "tune" the 32-dim embedding vectors from a bidirectional LSTM on Covid19 QA data, the same dataset that we are using in `jina hello chatbot`. diff --git a/docs/get-started/fashion-mnist.md b/docs/get-started/fashion-mnist.md index c8a6f1117..f93777f81 100644 --- a/docs/get-started/fashion-mnist.md +++ b/docs/get-started/fashion-mnist.md @@ -1,4 +1,8 @@ -# Finetuning MLP on Image +# Finetuning MLP for Fashion Image Search + +```{tip} +This example is inspired by [`jina hello fashion`](https://docs.jina.ai/get-started/hello-world/fashion/). We stronly recommend you to checkout that demo first before go through this tutorial. +``` In this example, we want to "tune" the 32-dim embedding vectors from a 2-layer MLP on the Fashion-MNIST image data, the same dataset that we are using in `jina hello fashion`. diff --git a/finetuner/labeler/executor.py b/finetuner/labeler/executor.py index f75336444..ac79e4b65 100644 --- a/finetuner/labeler/executor.py +++ b/finetuner/labeler/executor.py @@ -51,8 +51,9 @@ def embed(self, docs: DocumentArray, parameters: Dict, **kwargs): self._embed_model.eval() da_input = torch.from_numpy(da.blobs) docs_input = torch.from_numpy(docs.blobs) - da.embeddings = self._embed_model(da_input).detach().numpy() - docs.embeddings = self._embed_model(docs_input).detach().numpy() + with torch.inference_mode(): + da.embeddings = self._embed_model(da_input).detach().numpy() + docs.embeddings = self._embed_model(docs_input).detach().numpy() elif f_type == 'paddle': import paddle diff --git a/finetuner/labeler/ui/js/main.js b/finetuner/labeler/ui/js/main.js index 39553aa75..2ba716fcc 100644 --- a/finetuner/labeler/ui/js/main.js +++ b/finetuner/labeler/ui/js/main.js @@ -45,7 +45,7 @@ const app = new Vue({ advanced_config: { pos_value: {text: 'Positive label', value: 1, type: 'number'}, neg_value: {text: 'Negative label', value: -1, type: 'number'}, - epochs: {text: 'Epochs', value: 10, type: 'number'}, + epochs: {text: 'Epochs', value: 1, type: 'number'}, sample_size: {text: 'Match pool', value: 1000, type: 'number'}, model_path: {text: 'Model save path', value: 'tuned-model', type: 'text'} }, diff --git a/finetuner/labeler/ui/main.css b/finetuner/labeler/ui/main.css index 3aec9a0e6..80f976d08 100644 --- a/finetuner/labeler/ui/main.css +++ b/finetuner/labeler/ui/main.css @@ -122,6 +122,7 @@ main { footer { align-self: center; margin-top: auto; + font-weight: lighter; } footer a {