Improve Caiman Env Construction Problem

[itutu-tienday]

問題

optinist-for-server の dokerfile 版で、caiman の動作（環境構築）に問題が生じている。 調査と対処を行う。

• related
  • Changed the conda tool used in the dockerfile barebone-studio#497

判明している問題点

• Docker on Mac/Ubuntu

  • build docker image
    • Changed the conda tool used in the dockerfile barebone-studio#497 の対応により解消
  • run expdb_batch (on cron)
    • create caiman env
      • branch barebone-studio-porting-dev-dockerfile での対処により、解消
    • run caiman
      • branch barebone-studio-porting-dev-dockerfile での対処により、解消
  • run workflow (on gui)
    • create caiman env
      • branch barebone-studio-porting-dev-dockerfile での対処により、解消
    • run caiman
      • 問題発生継続中
        • workflow実行時、以下のエラーが発生中

          python /app/.snakemake/scripts/xxxxxxxxxxx.func.py\' died with <Signals.SIGKILL: 9>.
          

        • Note
          • snakemakeが作成するcaima envと、dockerfile内で作成済みの expdb_batch envの間で、環境差分（conda list結果）がある。
            • conda config --set channel_priority flexible の影響の可能性もある？
      • (解決)
        • 原因はdocker containerのメモリ不足
        • 環境によるが、docker container で 12～16GB 程度確保すれば、サンプルデータでの処理完了を確認できた
    • roi of caiman
      • roiが適切に抽出されていない？
        • caiman paramater (params/cnmf.yaml) に要因がある可能性も想定される（開発用設定のため？）

• Docker on Mac

  • run expdb_batch
    • Image plot処理で問題発生（処理が完了しない）
      • 以下をコメントアウトすることで、暫定的に処理を通過させることは可能
        • studio/app/optinist/core/expdb/batch_runner.py

            def __process_dataset_registration()
              ~~~~
              # Temporary comment out.
              # expdb_batch.generate_plots(stat_data=stat_data)
              # expdb_batch.generate_cellmasks()
              # expdb_batch.generate_pixelmaps()

Supplemental documents

• reports
  • https://docs.google.com/spreadsheets/d/1kbVbpBd6pqWQaKmng8bH5qwxE5t2VXC94E3RImCnR84/
  • https://drive.google.com/drive/folders/1uIaQyAOdm2kuOFuNrWV0rlqZka0iXIC3?usp=drive_link

動作確認手順

1. 環境構築
   • 対象ブランチ
     • optinist-for-server -> barebone-studio-porting-dev-dockerfile
       • https://github.com/arayabrain/optinist-for-server/tree/barebone-studio-porting-dev-dockerfile
   • 手順資料
     • https://drive.google.com/file/d/1UJ5zK49t_QKHO9PdnTGj5AnwatX63iyk/view?usp=drive_link
2. 解析バッチを実行（expdb_batchの動作確認）
   1. 処理対象データの処理フラグをON
      • データ投入手順・処理仕様 資料
        • https://docs.google.com/presentation/d/1aC9EzJcAbiA2oFNuEOfVCQkQo4MuPb0D/edit#slide=id.p6
        • https://drive.google.com/file/d/1IWFUUZrt4gFM27hmyVZBLuh7io5WUyyC/view?usp=drive_link
   2. 解析バッチ実行

      conda activate expdb_batch
      python run_expdb_batch.py -o 1
      

3. 解析バッチで生成されたデータに対し、WebUIよりWorkflowを実行（workflowの動作確認）
   • 作成Node
     • microscope -> preprocessing -> caiman_cnmf
       • 

[milesAraya]

Issue running python run_expdb_batch.py -o 1

(optinist_dev) milesd@Miless-MacBook-Pro optinist-for-server % docker compose -f docker-compose.dev.yml up -d
WARN[0000] /Users/milesd/GitRepos/optinist-for-server/docker-compose.dev.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
[+] Running 4/4
✔ Container optinist-for-server-db-1 Started 1.7s
✔ Container optinist-for-server-phpmyadmin-dev-1 Started 1.7s
✔ Container optinist-for-server-studio-dev-fe-1 Started 1.7s
✔ Container optinist-for-server-studio-dev-be-1 Started 2.9s
(optinist_dev) milesd@Miless-MacBook-Pro optinist-for-server % docker exec -it optinist-for-server-studio-dev-be-1 bash
(base) optinist@f5479da52439:/app$ python run_expdb_batch.py -o 1
Traceback (most recent call last):
File "/app/run_expdb_batch.py", line 17, in
from studio.app.optinist.microscopes.ND2Reader import ND2Reader
File "/app/studio/app/optinist/microscopes/ND2Reader.py", line 9, in
import numpy as np
ModuleNotFoundError: No module named 'numpy'

Fixed numpy error with:

sudo su
apt-get update
apt-get install -y python3-dev build-essential
pip install optinist

Then new error:

2024-12-13 15:49:53,043 INFO: process():97 - process start.
2024-12-13 15:49:53,141 INFO: execsingle_context():1892 - SELECT DATABASE()
2024-12-13 15:49:53,141 INFO: execsingle_context():1897 - [raw sql] {}
2024-12-13 15:49:53,144 INFO: execsingle_context():1892 - SELECT @@sql_mode
2024-12-13 15:49:53,144 INFO: execsingle_context():1897 - [raw sql] {}
2024-12-13 15:49:53,145 INFO: execsingle_context():1892 - SELECT @@lower_case_table_names
2024-12-13 15:49:53,146 INFO: execsingle_context():1897 - [raw sql] {}
2024-12-13 15:49:53,150 INFO: beginimpl():1089 - BEGIN (implicit)
2024-12-13 15:49:53,155 INFO: execsingle_context():1892 - SELECT organization.id AS organization_id, organization.name AS organization_name, organization.created_at AS organization_created_at
FROM organization
WHERE organization.id = %(id_1)s
LIMIT %(param_1)s
2024-12-13 15:49:53,155 INFO: execsingle_context():1897 - [generated in 0.00035s] {'id_1': 1, 'param_1': 1}
2024-12-13 15:49:53,186 INFO: rollbackimpl():1115 - ROLLBACK
2024-12-13 15:49:53,192 ERROR: process():116 - <class 'AssertionError'>: Organization not found [id: 1]
Traceback (most recent call last):
File "/app/studio/app/optinist/core/expdb/batch_runner.py", line 104, in process
self.process_preprocess()
File "/app/studio/app/optinist/core/expdb/batch_runner.py", line 161, in process_preprocess
get_organization(db, self.org_id)
File "/app/studio/app/common/core/users/crud_organizations.py", line 13, in get_organization
assert organization is not None, f"Organization not found [id: {organization_id}]"
AssertionError: Organization not found [id: 1]
2024-12-13 15:49:53,193 INFO: process():138 - process finish. [status: error (suspended)]
2024-12-13 15:49:53,193 INFO: __stopwatch_callback():89 - processing done. [process()][elapsed_time: 0.149930 sec]

Tried

root@f5479da52439:/app# alembic upgrade head
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
root@f5479da52439:/app# python run_expdb_batch.py -o 1
root@f5479da52439:/app# env | grep MYSQL
root@f5479da52439:/app# mysql -h mysql -u root -p
bash: mysql: command not found

It seems nothing was installed by docker. However, there didn't appear to be any errors in startup

used

docker exec -it optinist-for-server-studio-dev-be-1 bash
apt-get install -y default-mysql-client
mysql -h db -u MILES -p OPTINIST_FOR_SERVER
USE OPTINIST_FOR_SERVER;
INSERT INTO organization (name) VALUES ('YOUR_ORGANIZATION_NAME');
INSERT INTO roles (id, role) VALUES (1, 'admin'), (10, 'data manager'), (20, 'operator'), (30, 'guest operator');
INSERT INTO users (uid, organization_id, name, email, active) VALUES ('USER_UID', 1, 'admin@araya.org', 'YOUR_PASSWORD', 1);
INSERT INTO user_roles (user_id, role_id) VALUES (1, 1);

[milesAraya]

@tsuchiyama-araya suggested
conda info --envs
conda environments:
expdb_batch /home/optinist/.conda/envs/expdb_batch
base * /opt/miniforge

conda activate expdb_batch
python run_expdb_batch.py -o 1

[milesAraya]

Do we need to add so we can pass DB variables into studio-dev-be?

studio-dev-be:
env_file:
- studio/config/.env

db-1 | 2024-12-13 17:25:43 4 [Warning] Aborted connection 4 to db: 'OPTINIST_FOR_SERVER' user: 'MILES' host: '172.18.0.5' (Got an error reading communication packets)
db-1 | 2024-12-13 17:28:17 5 [Warning] Access denied for user 'MILES'@'172.18.0.5' (using password: YES)
db-1 | 2024-12-13 17:32:39 9 [Warning] Aborted connection 9 to db: 'OPTINIST_FOR_SERVER' user: 'MILES' host: '172.18.0.5' (Got an error reading communication packets)

[milesAraya]

The correct file structure may need to be

/app/experiments_datasets/
└── M000024/
└── M000024_ori017/
├── M000024_ori017.nd2
├── M000024_ori017_metadata.json
└── M000024_ori017_trialstructure.mat

Then
docker exec -it optinist-for-server-studio-dev-be-1 bash
echo "command: regist" > /app/experiments_datasets/M000024/M000024_ori017.proc