From 31408a88f19776dc02ddeb98de2b53093e6c4ee8 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Mon, 29 Jul 2024 16:35:37 +0200 Subject: [PATCH 01/14] corrected broken links and broken references --- README.md | 14 +++++++------- conda/amadesuGPT.yml | 2 -- conda/install_cpu.sh | 6 +++--- conda/install_gpu.sh | 6 +++--- conda/install_minimal.sh | 6 +++--- 5 files changed, 16 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 9c4af75..317d61e 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ In our original work (NeurIPS 2023) we used GPT3.5 and GPT4 as part of our agent Conda is an easy-to-use Python interface that supports launching [Jupyter Notebooks](https://jupyter.org/). If you are completely new to this, we recommend checking out the [docs here for getting conda installed](https://deeplabcut.github.io/DeepLabCut/docs/beginner-guides/beginners-guide.html#beginner-user-guide). Otherwise, proceed to use one of [our supplied conda files](https://github.com/AdaptiveMotorControlLab/AmadeusGPT/tree/main/conda). As you will see we have minimal dependencies to get started, and [here is a simple step-by-step guide](https://deeplabcut.github.io/DeepLabCut/docs/installation.html#step-2-build-an-env-using-our-conda-file) you can reference for setting it up (or see [BONUS](README.md#bonus---customized-your-conda-env) below). Here is the quick start command: ```bash -conda env create -f amadeusGPT.yml +conda env create -f conda/amadeusGPT.yml ``` To note, some modules AmadeusGPT can use benefit from GPU support, therefore we recommend also having an NVIDIA GPU available and installing CUDA. @@ -72,12 +72,12 @@ You can git clone (or download) this repo to grab a copy and go. We provide exam ### Here are a few demos that could fuel your own work, so please check them out! -1) [Draw a region of interest (ROI) and ask, "when is the animal in the ROI?"](notebook/EPM_demo.ipynb) -2) [Use a DeepLabCut SuperAnimal pose model to do video inference](notebook/custom_mouse_demo.ipynb) - (make sure you use a GPU if you don't have corresponding DeepLabCut keypoint files already! -3) [Write you own integration modules and use them](notebook/Horse_demo.ipynb). Bonus: [source code](amadeusgpt/integration_modules). Make sure you delete the cached modules_embedding.pickle if you add new modules! -4) [Multi-Animal social interactions](notebook/MABe_demo.ipynb) -5) [Reuse the task program generated by LLM and run it on different videos](notebook/MABe_demo.ipynb) -7) You can ask one query across multiple videos. Put your keypoint files and video files (pairs) in the same folder and specify the `data_folder` as shown in this [Demo](notebook/custom_mouse_video.ipynb). Make sure your video file and keypoint file follows the normal DeepLabCut convention, i.e., `prefix.mp4` `prefix*.h5`. +1) [Draw a region of interest (ROI) and ask, "when is the animal in the ROI?"](notebooks/EPM_demo.ipynb) +2) [Use a DeepLabCut SuperAnimal pose model to do video inference](notebooks/custom_mouse_demo.ipynb) - (make sure you use a GPU if you don't have corresponding DeepLabCut keypoint files already! +3) [Write you own integration modules and use them](notebooks/Horse_demo.ipynb). Bonus: [source code](amadeusgpt/integration_modules). Make sure you delete the cached modules_embedding.pickle if you add new modules! +4) [Multi-Animal social interactions](notebooks/MABe_demo.ipynb) +5) [Reuse the task program generated by LLM and run it on different videos](notebooks/MABe_demo.ipynb) +7) You can ask one query across multiple videos. Put your keypoint files and video files (pairs) in the same folder and specify the `data_folder` as shown in this [Demo](notebooks/custom_mouse_video.ipynb). Make sure your video file and keypoint file follows the normal DeepLabCut convention, i.e., `prefix.mp4` `prefix*.h5`. ### Minimal example diff --git a/conda/amadesuGPT.yml b/conda/amadesuGPT.yml index d82a235..128c79d 100644 --- a/conda/amadesuGPT.yml +++ b/conda/amadesuGPT.yml @@ -6,6 +6,4 @@ dependencies: - python==3.10 - pytables==3.8.0 - hdf5 - - pip - jupyter - - amadeusGPT diff --git a/conda/install_cpu.sh b/conda/install_cpu.sh index 9188ed8..3dc3584 100644 --- a/conda/install_cpu.sh +++ b/conda/install_cpu.sh @@ -1,10 +1,10 @@ #!/bin/bash source /Users/shaokaiye/miniforge3/bin/activate -conda env create -f conda/amadesuGPT-cpu.yml -conda activate amadeusgpt-cpu +conda env create -f conda/amadesuGPT.yml +conda activate amadeusgpt conda install pytorch torchvision cpuonly -c pytorch pip install "git+https://github.com/DeepLabCut/DeepLabCut.git@pytorch_dlc#egg=deeplabcut" pip install pycocotools pip install -e .[streamlit] # install the python kernel -python -m ipykernel install --user --name amadeusgpt-cpu --display-name "amadeusgpt-cpu" +python -m ipykernel install --user --name amadeusgpt --display-name "amadeusgpt" diff --git a/conda/install_gpu.sh b/conda/install_gpu.sh index 8057933..b779c5c 100644 --- a/conda/install_gpu.sh +++ b/conda/install_gpu.sh @@ -1,11 +1,11 @@ #!/bin/bash source /mnt/md0/shaokai/miniconda3/bin/activate -conda env create -f conda/amadesuGPT-gpu.yml -conda activate amadeusgpt-gpu +conda env create -f conda/amadesuGPT.yml +conda activate amadeusgpt # adjust this line according to your cuda version conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia pip install "git+https://github.com/DeepLabCut/DeepLabCut.git@pytorch_dlc#egg=deeplabcut" pip install pycocotools pip install -e .[streamlit] # install the python kernel -python -m ipykernel install --user --name amadeusgpt-gpu --display-name "amadeusgpt-gpu" +python -m ipykernel install --user --name amadeusgpt --display-name "amadeusgpt" diff --git a/conda/install_minimal.sh b/conda/install_minimal.sh index 4941bc5..097d7f7 100644 --- a/conda/install_minimal.sh +++ b/conda/install_minimal.sh @@ -1,8 +1,8 @@ #!/bin/bash # change this to your own miniconda / miniforge path source /Users/shaokaiye/miniforge3/bin/activate -conda env create -f conda/amadesuGPT-minimal.yml -conda activate amadeusgpt-minimal +conda env create -f conda/amadesuGPT.yml +conda activate amadeusgpt pip install pycocotools pip install -e .[streamlit] -python -m ipykernel install --user --name amadeusgpt-minimal --display-name "amadeusgpt-minimal" \ No newline at end of file +python -m ipykernel install --user --name amadeusgpt --display-name "amadeusgpt" \ No newline at end of file From 38e7ce822233f139cd0b02cf8eb17c609cfecbaf Mon Sep 17 00:00:00 2001 From: Mackenzie Mathis Date: Tue, 30 Jul 2024 10:42:06 +0200 Subject: [PATCH 02/14] Update setup.cfg - bump to stable v1 and v0.1.1 --- setup.cfg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.cfg b/setup.cfg index 05c4c7f..c6fdf8f 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = amadeusgpt -version = 0.1.1rc2 +version = 0.1.1 author = Shaokai Ye, Jessy Lauer, Mu Zhou, Alexander Mathis, Mackenzie Mathis author_email = mackenzie@post.harvard.edu description = AmadeusGPT🎻: We turn natural language descriptions of behaviors into machine-executable code @@ -12,7 +12,7 @@ url = https://github.com/AdaptiveMotorControlLab/AmadeusGPT project_urls = Bug Tracker = https://github.com/AdaptiveMotorControlLab/AmadeusGPT/issues classifiers = - Development Status :: 3 - Alpha + Development Status :: 5 - Production/Stable Environment :: GPU :: NVIDIA CUDA Intended Audience :: Science/Research Operating System :: OS Independent From 591d056d18949fb8377c172b73c581fccfc1e5e3 Mon Sep 17 00:00:00 2001 From: Mackenzie Mathis Date: Tue, 30 Jul 2024 10:42:19 +0200 Subject: [PATCH 03/14] Update pyproject.toml --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 4f4cd90..72af39e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ name = "AmadeusGPT" readme = "README.md" requires-python = ">=3.10" dynamic = ["version"] -version = "0.1.1rc2" +version = "0.1.1" [tool.setuptools] packages = ["amadeusgpt"] From 84e01eefa293d6f401ecb797023760b4efb41171 Mon Sep 17 00:00:00 2001 From: Mackenzie Mathis Date: Tue, 30 Jul 2024 10:42:32 +0200 Subject: [PATCH 04/14] Update version.py --- amadeusgpt/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/amadeusgpt/version.py b/amadeusgpt/version.py index 1be166e..09e1a53 100644 --- a/amadeusgpt/version.py +++ b/amadeusgpt/version.py @@ -6,5 +6,5 @@ # # Licensed under Apache 2.0 -__version__ = "0.1.1rc2" +__version__ = "0.1.1" VERSION = __version__ From 5aea5a63b621f4d5724dc123c5b3a0a60ff87ab4 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Tue, 30 Jul 2024 11:42:41 +0200 Subject: [PATCH 05/14] Corrected typo. Added config yamls in setup --- conda/{amadesuGPT.yml => amadeusGPT.yml} | 0 setup.cfg | 4 ++++ 2 files changed, 4 insertions(+) rename conda/{amadesuGPT.yml => amadeusGPT.yml} (100%) diff --git a/conda/amadesuGPT.yml b/conda/amadeusGPT.yml similarity index 100% rename from conda/amadesuGPT.yml rename to conda/amadeusGPT.yml diff --git a/setup.cfg b/setup.cfg index c6fdf8f..97189c3 100644 --- a/setup.cfg +++ b/setup.cfg @@ -52,6 +52,10 @@ dev = static/*.* static/images/*.* static/styles/*.* + configs/Horse_template.yaml + configs/EPM_template.yaml + configs/MausHaus_template.yaml + configs/MABe_template.yaml [options.entry_points] console_scripts = From 4c4e3e75d31a352b439c8185a7f721dbb4a57133 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Tue, 30 Jul 2024 11:43:20 +0200 Subject: [PATCH 06/14] Removed config files that are no longer needed --- amadeusgpt/configs/Custom_template.yaml | 12 ------------ amadeusgpt/configs/template.yaml | 11 ----------- 2 files changed, 23 deletions(-) delete mode 100644 amadeusgpt/configs/Custom_template.yaml delete mode 100644 amadeusgpt/configs/template.yaml diff --git a/amadeusgpt/configs/Custom_template.yaml b/amadeusgpt/configs/Custom_template.yaml deleted file mode 100644 index a505854..0000000 --- a/amadeusgpt/configs/Custom_template.yaml +++ /dev/null @@ -1,12 +0,0 @@ -keypoint_info: -llm_info: - keep_last_n_messages: 2 -object_info: - load_objects_from_disk: false -video_info: - scene_frame_number: 100 - video_file_path: -data_info: - data_folder: 'examples/Custom' - video_suffix: '.mp4' - result_folder: 'results' \ No newline at end of file diff --git a/amadeusgpt/configs/template.yaml b/amadeusgpt/configs/template.yaml deleted file mode 100644 index 5422097..0000000 --- a/amadeusgpt/configs/template.yaml +++ /dev/null @@ -1,11 +0,0 @@ -keypoint_info: -object_info: - load_objects_from_disk: false -llm_info: - keep_last_n_messages: 2 -video_info: - scene_frame_number: 100 -data_info: - data_folder: '/Users/shaokaiye/maushaus_workspace' - video_suffix: '.mp4' - result_folder: 'results' \ No newline at end of file From 0077578249b822df49d5558a1831ccb8ff99c3f6 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Tue, 30 Jul 2024 15:03:22 +0200 Subject: [PATCH 07/14] changed work from to pull the repo from git --- .github/workflows/pytest.yml | 2 +- notebooks/custom_mouse_video.ipynb | 35 ++++-------------------------- 2 files changed, 5 insertions(+), 32 deletions(-) diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index bf58f0f..afd37bc 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -43,5 +43,5 @@ jobs: env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} run: | - pip install '.[streamlit]' + pip install git+https://github.com/${{ github.repository }}.git@${{ github.sha }} pytest tests --timeout=400 diff --git a/notebooks/custom_mouse_video.ipynb b/notebooks/custom_mouse_video.ipynb index e326a9f..486b1e3 100644 --- a/notebooks/custom_mouse_video.ipynb +++ b/notebooks/custom_mouse_video.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "id": "f01f49c5", "metadata": {}, "outputs": [], @@ -15,7 +15,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "id": "bceb3204-2a87-4671-8135-2533a7a51771", "metadata": {}, "outputs": [], @@ -41,37 +41,10 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "id": "be76dc87-fbe8-452f-b85c-2af3e95a03bf", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Project created at temp_result_folder. Results will be saved to temp_result_folder\n", - "The project will load video files (*..mp4) and optionally keypoint files from temp_data_folder\n", - "A copy of the project config file is saved at temp_result_folder/config.yaml\n", - "{'data_info': {'data_folder': 'temp_data_folder',\n", - " 'result_folder': 'temp_result_folder',\n", - " 'video_suffix': '.mp4'},\n", - " 'llm_info': {'max_tokens': 4096, 'temperature': 1.0}}\n", - "No video files found in the data folder temp_data_folder. Please check the data folder and the video suffix\n" - ] - }, - { - "ename": "AttributeError", - "evalue": "'AMADEUS' object has no attribute 'sandbox'", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)", - "Cell \u001b[0;32mIn[3], line 13\u001b[0m\n\u001b[1;32m 10\u001b[0m config[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mscene_frame_number\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m scene_frame_number\n\u001b[1;32m 12\u001b[0m amadeus \u001b[38;5;241m=\u001b[39m AMADEUS(config)\n\u001b[0;32m---> 13\u001b[0m video_file_paths \u001b[38;5;241m=\u001b[39m \u001b[43mamadeus\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_video_file_paths\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 14\u001b[0m \u001b[38;5;28mprint\u001b[39m (video_file_paths)\n", - "File \u001b[0;32m~/AmadeusGPT-dev/amadeusgpt/main.py:121\u001b[0m, in \u001b[0;36mAMADEUS.get_video_file_paths\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 120\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_video_file_paths\u001b[39m(\u001b[38;5;28mself\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28mlist\u001b[39m[\u001b[38;5;28mstr\u001b[39m]:\n\u001b[0;32m--> 121\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msandbox\u001b[49m\u001b[38;5;241m.\u001b[39mvideo_file_paths\n", - "\u001b[0;31mAttributeError\u001b[0m: 'AMADEUS' object has no attribute 'sandbox'" - ] - } - ], + "outputs": [], "source": [ "scene_frame_number = 400\n", "\n", From 8bd38845d9f192f2027d0559763c2d7b4b81dbe8 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Tue, 30 Jul 2024 15:05:32 +0200 Subject: [PATCH 08/14] Added comments to remind people to pay attentino to data folder in the demo notebooks --- notebooks/EPM_demo.ipynb | 2 ++ notebooks/Horse_demo.ipynb | 2 ++ notebooks/MABe_demo.ipynb | 2 ++ notebooks/MausHaus_demo.ipynb | 3 ++- 4 files changed, 8 insertions(+), 1 deletion(-) diff --git a/notebooks/EPM_demo.ipynb b/notebooks/EPM_demo.ipynb index b4b1b77..60a7b50 100644 --- a/notebooks/EPM_demo.ipynb +++ b/notebooks/EPM_demo.ipynb @@ -51,6 +51,8 @@ "amadeus_root = Path(amadeusgpt.__file__).parent.parent\n", "config = Config(amadeus_root / \"amadeusgpt/configs/EPM_template.yaml\")\n", "config['video_info']['scene_frame_number'] = scene_frame_number\n", + "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# repo_root/examples/EPM\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", "amadeus = AMADEUS(config)\n", diff --git a/notebooks/Horse_demo.ipynb b/notebooks/Horse_demo.ipynb index a3aacda..b2d9425 100644 --- a/notebooks/Horse_demo.ipynb +++ b/notebooks/Horse_demo.ipynb @@ -41,6 +41,8 @@ "config = Config(amadeus_root / \"amadeusgpt/configs/Horse_template.yaml\")\n", "\n", "config['video_info']['scene_frame_number'] = scene_frame_number\n", + "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# repo_root/examples/Horse\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder'] \n", "\n", "amadeus = AMADEUS(config)\n", diff --git a/notebooks/MABe_demo.ipynb b/notebooks/MABe_demo.ipynb index 93431e6..53f69bc 100644 --- a/notebooks/MABe_demo.ipynb +++ b/notebooks/MABe_demo.ipynb @@ -38,6 +38,8 @@ "source": [ "amadeus_root = Path(amadeusgpt.__file__).parent.parent\n", "config = Config(amadeus_root / \"amadeusgpt/configs/MABe_template.yaml\")\n", + "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# repo_root/examples/MABe\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", "amadeus = AMADEUS(config)\n", diff --git a/notebooks/MausHaus_demo.ipynb b/notebooks/MausHaus_demo.ipynb index 99bb727..1b1b905 100644 --- a/notebooks/MausHaus_demo.ipynb +++ b/notebooks/MausHaus_demo.ipynb @@ -43,7 +43,8 @@ "config = Config(amadeus_root / \"amadeusgpt/configs/MausHaus_template.yaml\")\n", "\n", "config['video_info']['scene_frame_number'] = scene_frame_number\n", - "\n", + "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# repo_root/examples/MausHaus\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", "amadeus = AMADEUS(config)\n", From b5f76291c98c5553755381daf8b0427c6d423022 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Tue, 30 Jul 2024 16:09:10 +0200 Subject: [PATCH 09/14] fixed pypi typo --- notebooks/EPM_demo.ipynb | 2 +- notebooks/Horse_demo.ipynb | 2 +- notebooks/MABe_demo.ipynb | 2 +- notebooks/MausHaus_demo.ipynb | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/notebooks/EPM_demo.ipynb b/notebooks/EPM_demo.ipynb index 60a7b50..d9e9d65 100644 --- a/notebooks/EPM_demo.ipynb +++ b/notebooks/EPM_demo.ipynb @@ -51,7 +51,7 @@ "amadeus_root = Path(amadeusgpt.__file__).parent.parent\n", "config = Config(amadeus_root / \"amadeusgpt/configs/EPM_template.yaml\")\n", "config['video_info']['scene_frame_number'] = scene_frame_number\n", - "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# if you install from pypi or you used the conda file, you have to adjust the path of data folder to\n", "# repo_root/examples/EPM\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", diff --git a/notebooks/Horse_demo.ipynb b/notebooks/Horse_demo.ipynb index b2d9425..164cbdd 100644 --- a/notebooks/Horse_demo.ipynb +++ b/notebooks/Horse_demo.ipynb @@ -41,7 +41,7 @@ "config = Config(amadeus_root / \"amadeusgpt/configs/Horse_template.yaml\")\n", "\n", "config['video_info']['scene_frame_number'] = scene_frame_number\n", - "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# if you install from pypi or you used the conda file, you have to adjust the path of data folder to\n", "# repo_root/examples/Horse\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder'] \n", "\n", diff --git a/notebooks/MABe_demo.ipynb b/notebooks/MABe_demo.ipynb index 53f69bc..4e80b16 100644 --- a/notebooks/MABe_demo.ipynb +++ b/notebooks/MABe_demo.ipynb @@ -38,7 +38,7 @@ "source": [ "amadeus_root = Path(amadeusgpt.__file__).parent.parent\n", "config = Config(amadeus_root / \"amadeusgpt/configs/MABe_template.yaml\")\n", - "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# if you install from pypi or you used the conda file, you have to adjust the path of data folder to\n", "# repo_root/examples/MABe\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", diff --git a/notebooks/MausHaus_demo.ipynb b/notebooks/MausHaus_demo.ipynb index 1b1b905..c717557 100644 --- a/notebooks/MausHaus_demo.ipynb +++ b/notebooks/MausHaus_demo.ipynb @@ -43,7 +43,7 @@ "config = Config(amadeus_root / \"amadeusgpt/configs/MausHaus_template.yaml\")\n", "\n", "config['video_info']['scene_frame_number'] = scene_frame_number\n", - "# if you install from pypl or you used the conda file, you have to adjust the path of data folder to\n", + "# if you install from pypi or you used the conda file, you have to adjust the path of data folder to\n", "# repo_root/examples/MausHaus\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", From e14c42f02e6ace3078ff1465f104f7a565b85187 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Wed, 31 Jul 2024 15:10:07 +0200 Subject: [PATCH 10/14] Fixed a bug in create_project. Changed default use_vlm to False. Updated demo notebooks --- amadeusgpt/analysis_objects/animal.py | 4 ++-- amadeusgpt/analysis_objects/relationship.py | 6 ++++-- amadeusgpt/main.py | 3 +-- amadeusgpt/managers/animal_manager.py | 13 ++++++++++++- amadeusgpt/managers/object_manager.py | 4 +++- amadeusgpt/managers/visual_manager.py | 2 +- amadeusgpt/project.py | 8 +++++++- notebooks/EPM_demo.ipynb | 2 +- notebooks/Horse_demo.ipynb | 2 +- notebooks/MABe_demo.ipynb | 2 +- notebooks/MausHaus_demo.ipynb | 2 +- notebooks/custom_mouse_video.ipynb | 2 +- tests/test_project_creation.py | 4 ++-- 13 files changed, 37 insertions(+), 17 deletions(-) diff --git a/amadeusgpt/analysis_objects/animal.py b/amadeusgpt/analysis_objects/animal.py index aa7ffca..529b143 100644 --- a/amadeusgpt/analysis_objects/animal.py +++ b/amadeusgpt/analysis_objects/animal.py @@ -27,8 +27,8 @@ class AnimalSeq(Animal): body center, left, right, above, top are relative to the subset of keypoints. Attributes ---------- - self._coords: arr potentially subset of keypoints - self.wholebody: full set of keypoints. This is important for overlap relationship + self.wholebody: np.ndarray of keypoints of all bodyparts + self.keypoint """ def __init__(self, animal_name: str, keypoints: ndarray, keypoint_names: List[str]): diff --git a/amadeusgpt/analysis_objects/relationship.py b/amadeusgpt/analysis_objects/relationship.py index c098f0b..0c58de1 100644 --- a/amadeusgpt/analysis_objects/relationship.py +++ b/amadeusgpt/analysis_objects/relationship.py @@ -48,13 +48,15 @@ def calc_angle_between_2d_coordinate_systems(cs1, cs2): return np.rad2deg(np.arccos(dot)) -def get_pairwise_distance(arr1, arr2): +def get_pairwise_distance(arr1:np.ndarray, + arr2:np.ndarray): # we want to make sure this uses a fast implementation - # (n_frame, n_kpts, 2) + # arr: (n_frame, n_kpts, 2) assert len(arr1.shape) == 3 and len(arr2.shape) == 3 # pariwise distance (n_frames, n_kpts, n_kpts) pairwise_distances = np.ones((arr1.shape[0], arr1.shape[1], arr2.shape[1])) * 100000 for frame_id in range(arr1.shape[0]): + # should we use the mean of all keypooints for the distance? pairwise_distances[frame_id] = cdist(arr1[frame_id], arr2[frame_id]) return pairwise_distances diff --git a/amadeusgpt/main.py b/amadeusgpt/main.py index 6fcaf5b..55ebddf 100644 --- a/amadeusgpt/main.py +++ b/amadeusgpt/main.py @@ -2,7 +2,6 @@ from amadeusgpt.config import Config from amadeusgpt.programs.sandbox import Sandbox -import yaml ########## # all these are providing the customized classes for the code execution ########## @@ -21,7 +20,7 @@ class AMADEUS: - def __init__(self, config: Config | dict, use_vlm=True): + def __init__(self, config: Config | dict, use_vlm=False): self.config = config ### fields that decide the behavior of the application self.use_self_debug = True diff --git a/amadeusgpt/managers/animal_manager.py b/amadeusgpt/managers/animal_manager.py index 96a8406..8a01904 100644 --- a/amadeusgpt/managers/animal_manager.py +++ b/amadeusgpt/managers/animal_manager.py @@ -101,7 +101,6 @@ def configure_animal_from_meta(self, meta_info): self.superanimal_name = None def init_pose(self): - if not os.path.exists(self.keypoint_file_path): # no need to initialize here return @@ -111,6 +110,9 @@ def init_pose(self): elif self.keypoint_file_path.endswith(".json"): # could be coco format all_keypoints = self._process_keypoint_file_from_json() + elif self.keypoint_file_path.endswith(".npy"): + # assuming it's for 3D keypoints + all_keypoints = self._process_keypoint_file_from_npy() for individual_id in range(self.n_individuals): animal_name = f"animal_{individual_id}" # by default, we initialize all animals with the same keypoints and all the keypoint names @@ -130,6 +132,15 @@ def init_pose(self): self.animals.append(animalseq) + def _process_keypoint_file_from_npy(self) -> ndarray: + + all_keypoints = np.load(self.keypoint_file_path) + # (n_frames, n_keypoints, 4 -> (x,y,z, confidence)) + all_keypoints = all_keypoints[..., :3] + + + return all_keypoints + def _process_keypoint_file_from_h5(self) -> ndarray: df = pd.read_hdf(self.keypoint_file_path) self.full_keypoint_names = list( diff --git a/amadeusgpt/managers/object_manager.py b/amadeusgpt/managers/object_manager.py index 12b90f7..cbd8edc 100644 --- a/amadeusgpt/managers/object_manager.py +++ b/amadeusgpt/managers/object_manager.py @@ -28,7 +28,9 @@ def __init__( self.animal_manager = animal_manager self.roi_objects = [] self.seg_objects = [] - self.load_from_disk = self.config["object_info"]["load_objects_from_disk"] + + self.load_from_disk = self.config["object_info"].get("load_objects_from_disk", False) + if self.load_from_disk: self.load_objects_from_disk() else: diff --git a/amadeusgpt/managers/visual_manager.py b/amadeusgpt/managers/visual_manager.py index 9724d7c..437cc7b 100644 --- a/amadeusgpt/managers/visual_manager.py +++ b/amadeusgpt/managers/visual_manager.py @@ -51,7 +51,7 @@ def __init__( self.object_manager = object_manager def get_scene_image(self): - scene_frame_index = self.config["video_info"]["scene_frame_number"] + scene_frame_index = self.config["video_info"].get("scene_frame_number", 1) cap = cv2.VideoCapture(self.video_file_path) cap.set(cv2.CAP_PROP_POS_FRAMES, scene_frame_index) ret, frame = cap.read() diff --git a/amadeusgpt/project.py b/amadeusgpt/project.py index 1eddd0d..9e20e21 100644 --- a/amadeusgpt/project.py +++ b/amadeusgpt/project.py @@ -20,6 +20,12 @@ def create_project(data_folder, "temperature": 0.0, "keep_last_n_messages": 2 }, + "object_info": { + "load_objects_from_disk": False, + "use_grid_objects": False + }, + "result_info" :{}, + "video_info": {} } # save the dictionary config to yaml @@ -31,7 +37,7 @@ def create_project(data_folder, yaml.dump(config, f) print (f"Project created at {result_folder}. Results will be saved to {result_folder}") - print (f"The project will load video files (*.{video_suffix}) and optionally keypoint files from {data_folder}") + print (f"The project will load video files (*{video_suffix}) and optionally keypoint files from {data_folder}") print (f"A copy of the project config file is saved at {file_path}") pprint.pprint(config) diff --git a/notebooks/EPM_demo.ipynb b/notebooks/EPM_demo.ipynb index d9e9d65..6680c20 100644 --- a/notebooks/EPM_demo.ipynb +++ b/notebooks/EPM_demo.ipynb @@ -55,7 +55,7 @@ "# repo_root/examples/EPM\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", - "amadeus = AMADEUS(config)\n", + "amadeus = AMADEUS(config, use_vlm=True)\n", "video_file_paths = amadeus.get_video_file_paths()\n", "print (video_file_paths) " ] diff --git a/notebooks/Horse_demo.ipynb b/notebooks/Horse_demo.ipynb index 164cbdd..c98f6c7 100644 --- a/notebooks/Horse_demo.ipynb +++ b/notebooks/Horse_demo.ipynb @@ -45,7 +45,7 @@ "# repo_root/examples/Horse\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder'] \n", "\n", - "amadeus = AMADEUS(config)\n", + "amadeus = AMADEUS(config, use_vlm = True)\n", "video_file_paths = amadeus.get_video_file_paths()\n", "print (video_file_paths) " ] diff --git a/notebooks/MABe_demo.ipynb b/notebooks/MABe_demo.ipynb index 4e80b16..51fd2b9 100644 --- a/notebooks/MABe_demo.ipynb +++ b/notebooks/MABe_demo.ipynb @@ -42,7 +42,7 @@ "# repo_root/examples/MABe\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", - "amadeus = AMADEUS(config)\n", + "amadeus = AMADEUS(config, use_vlm=True)\n", "video_file_paths = amadeus.get_video_file_paths()\n", "print (video_file_paths)" ] diff --git a/notebooks/MausHaus_demo.ipynb b/notebooks/MausHaus_demo.ipynb index c717557..9ebfe04 100644 --- a/notebooks/MausHaus_demo.ipynb +++ b/notebooks/MausHaus_demo.ipynb @@ -47,7 +47,7 @@ "# repo_root/examples/MausHaus\n", "config['data_info']['data_folder'] = amadeus_root / config['data_info']['data_folder']\n", "\n", - "amadeus = AMADEUS(config)\n", + "amadeus = AMADEUS(config, use_vlm = True)\n", "video_file_paths = amadeus.get_video_file_paths()\n", "print (video_file_paths) " ] diff --git a/notebooks/custom_mouse_video.ipynb b/notebooks/custom_mouse_video.ipynb index 486b1e3..bcbe720 100644 --- a/notebooks/custom_mouse_video.ipynb +++ b/notebooks/custom_mouse_video.ipynb @@ -57,7 +57,7 @@ "\n", "config[\"scene_frame_number\"] = scene_frame_number\n", "\n", - "amadeus = AMADEUS(config)\n", + "amadeus = AMADEUS(config, use_vlm = True)\n", "video_file_paths = amadeus.get_video_file_paths()\n", "print (video_file_paths)" ] diff --git a/tests/test_project_creation.py b/tests/test_project_creation.py index f3bf076..f42b98d 100644 --- a/tests/test_project_creation.py +++ b/tests/test_project_creation.py @@ -8,13 +8,13 @@ # Create a project -data_folder = "temp_data_folder" +data_folder = "examples/EPM/" result_folder = "temp_result_folder" config = create_project(data_folder, result_folder) # Create an AMADEUS instance -amadeus = AMADEUS(config) +amadeus = AMADEUS(config, use_vlm=True) # query = "Plot the trajectory of the animal using the animal center and color it by time" # qa_message = amadeus.step(query) From 9d81d5484fa6a714690f04796e2879eed15cffa1 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Wed, 31 Jul 2024 15:22:53 +0200 Subject: [PATCH 11/14] removed WIP 3d keypoints --- amadeusgpt/managers/animal_manager.py | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/amadeusgpt/managers/animal_manager.py b/amadeusgpt/managers/animal_manager.py index 8a01904..bda7e25 100644 --- a/amadeusgpt/managers/animal_manager.py +++ b/amadeusgpt/managers/animal_manager.py @@ -110,9 +110,7 @@ def init_pose(self): elif self.keypoint_file_path.endswith(".json"): # could be coco format all_keypoints = self._process_keypoint_file_from_json() - elif self.keypoint_file_path.endswith(".npy"): - # assuming it's for 3D keypoints - all_keypoints = self._process_keypoint_file_from_npy() + for individual_id in range(self.n_individuals): animal_name = f"animal_{individual_id}" # by default, we initialize all animals with the same keypoints and all the keypoint names @@ -130,16 +128,7 @@ def init_pose(self): self.config["keypoint_info"]["head_orientation_keypoints"] ) - self.animals.append(animalseq) - - def _process_keypoint_file_from_npy(self) -> ndarray: - - all_keypoints = np.load(self.keypoint_file_path) - # (n_frames, n_keypoints, 4 -> (x,y,z, confidence)) - all_keypoints = all_keypoints[..., :3] - - - return all_keypoints + self.animals.append(animalseq) def _process_keypoint_file_from_h5(self) -> ndarray: df = pd.read_hdf(self.keypoint_file_path) From dbcb6c57b2c4da50ca04cfdf4151fc4f13d742b3 Mon Sep 17 00:00:00 2001 From: shaokaiye Date: Wed, 31 Jul 2024 15:29:43 +0200 Subject: [PATCH 12/14] Fixed one more --- amadeusgpt/project.py | 1 + 1 file changed, 1 insertion(+) diff --git a/amadeusgpt/project.py b/amadeusgpt/project.py index 9e20e21..a63cdd1 100644 --- a/amadeusgpt/project.py +++ b/amadeusgpt/project.py @@ -24,6 +24,7 @@ def create_project(data_folder, "load_objects_from_disk": False, "use_grid_objects": False }, + "keypoint_info": {}, "result_info" :{}, "video_info": {} } From 2a8232214df242d8774565d1ac6c5376faa43b8a Mon Sep 17 00:00:00 2001 From: shaokai Date: Wed, 31 Jul 2024 15:55:47 +0200 Subject: [PATCH 13/14] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5cb03f7..bd3cd30 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ In our original work (NeurIPS 2023) we used GPT3.5 and GPT4 as part of our agent Conda is an easy-to-use Python interface that supports launching [Jupyter Notebooks](https://jupyter.org/). If you are completely new to this, we recommend checking out the [docs here for getting conda installed](https://deeplabcut.github.io/DeepLabCut/docs/beginner-guides/beginners-guide.html#beginner-user-guide). Otherwise, proceed to use one of [our supplied conda files](https://github.com/AdaptiveMotorControlLab/AmadeusGPT/tree/main/conda). As you will see we have minimal dependencies to get started, and [here is a simple step-by-step guide](https://deeplabcut.github.io/DeepLabCut/docs/installation.html#step-2-build-an-env-using-our-conda-file) you can reference for setting it up (or see [BONUS](README.md#bonus---customized-your-conda-env) below). Here is the quick start command: ```bash -conda env create -f conda/amadeusGPT.yml +conda env create -f amadeusGPT.yml ``` To note, some modules AmadeusGPT can use benefit from GPU support, therefore we recommend also having an NVIDIA GPU available and installing CUDA. From 255b434201748853984eb55736bbe34c1d5de167 Mon Sep 17 00:00:00 2001 From: shaokai Date: Wed, 31 Jul 2024 15:59:49 +0200 Subject: [PATCH 14/14] Update test_project_creation.py --- tests/test_project_creation.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_project_creation.py b/tests/test_project_creation.py index f42b98d..12b34f8 100644 --- a/tests/test_project_creation.py +++ b/tests/test_project_creation.py @@ -14,8 +14,8 @@ config = create_project(data_folder, result_folder) # Create an AMADEUS instance -amadeus = AMADEUS(config, use_vlm=True) +amadeus = AMADEUS(config) # query = "Plot the trajectory of the animal using the animal center and color it by time" # qa_message = amadeus.step(query) -# parse_result(amadeus, qa_message) \ No newline at end of file +# parse_result(amadeus, qa_message)