Cannot Upload (Text) Files for Annotation Using MinIO

[opallch]

Hi all,

I have set up diffgram with docker and I could not upload files for annotation (basically everything with importing/exporting does not work, it kept on loading but with no error messages popping up). Everything worked quite well as I played around on diffgram.com and I guess the issue was originated from the fact that two of the containers stopped running. My machine is on Ubuntu 22.04.1 LTS and everything in the requirement.txt was successfully installed.

Here are the parameters used for setting up the docker (almost all are default settings):

First We need to know what static storage provider you will use: 
1. Minio Local Storage [Default]
2. Amazon Web Services S3 (AWS S3)
3. Microsoft Azure Storage (Azure)
4. Google Cloud Storage (GCP)
Enter 1, 2, 3 or 4. Or write exit to quit the installation: 1
Would you like to provide a custom MinIO host? Default will use docker Y/N?
Please provide the Minio S3 Bucket Name [Default is diffgram-storage]: 
Enter diffgram version: [Or Press Enter to Get The Latest Version]: 
Do you want to use the local database? Y/N [Press Enter to use Local DB]: 
Do Add Z Flag to Postgres Mount? (Use only when using SELinux distros or similar) Y/N [Press Enter to Skip]: 
Do you want to add Mailgun to Diffgram?[Y/n] n
Do you want to set RabbitMQ Config? 
 (default config: localhost:5672 => user - password will be "admin" - "admin"? [Y/n] 

The log for the first stopped container diffgram-db_migration-1:

2023-01-23 17:57:37 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
2023-01-23 17:57:37 INFO  [alembic.runtime.migration] Will assume transactional DDL.
2023-01-23 17:57:39 Postgres is up - executing command
2023-01-23 17:58:56 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
2023-01-23 17:58:56 INFO  [alembic.runtime.migration] Will assume transactional DDL.
2023-01-23 17:58:57 Postgres is up - executing command
2023-01-23 17:59:52 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
2023-01-23 17:59:52 INFO  [alembic.runtime.migration] Will assume transactional DDL.
2023-01-23 17:57:38 Error loading secrets: 
2023-01-23 17:57:38 No module named 'shared.settings.secrets'
2023-01-23 17:57:38 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-23 17:57:38 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-23 17:57:38 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-23 17:57:38 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-23 17:57:38 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-23 17:57:38 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-23 17:57:38 DATABASE_URL DATABASE_URL  True
2023-01-23 17:57:38 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000
2023-01-23 17:58:57 Error loading secrets: 
2023-01-23 17:58:57 No module named 'shared.settings.secrets'
2023-01-23 17:58:57 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-23 17:58:57 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-23 17:58:57 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-23 17:58:57 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-23 17:58:57 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-23 17:58:57 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-23 17:58:57 DATABASE_URL DATABASE_URL  True
2023-01-23 17:58:57 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000
2023-01-23 17:59:53 Error loading secrets: 
2023-01-23 17:59:53 No module named 'shared.settings.secrets'
2023-01-23 17:59:53 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-23 17:59:53 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-23 17:59:53 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-23 17:59:53 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-23 17:59:53 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-23 17:59:53 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-23 17:59:53 DATABASE_URL DATABASE_URL  True
2023-01-23 17:59:53 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000
2023-01-23 17:59:53 Postgres is up - executing command

And the second stopped container diffgram-createbuckets-1:

2023-01-23 17:57:06 Added `myminio` successfully.
2023-01-23 17:57:07 mc: Please use 'mc anonymous'
2023-01-23 17:59:42 Added `myminio` successfully.
2023-01-23 17:59:42 mc: Please use 'mc anonymous'
2023-01-23 17:57:06 mc: <ERROR> Unable to make bucket `myminio/diffgram-storage`. Your previous request to create the named bucket succeeded and you already own it.
2023-01-23 17:59:42 mc: <ERROR> Unable to make bucket `myminio/diffgram-storage`. Your previous request to create the named bucket succeeded and you already own it.

Please let me know if some important information is missing, thanks in advance!

[anthony-chaudhary]

Hello @opallch
I believe both of those containers are expected to stop once complete.

Are you uploading through the UI or SDK/API?
If UI can you share a video of a sample upload showing the dev console open?

Best
Anthony

[opallch]

Hi @Anthony-Sarkis,

I did that via the UI and uploaded local files. Here is the video. I realized that this time even the diffgram-eventhandlers-1 container stopped and here is the log:

2023-01-24 08:15:41 [2023-01-24 07:15:41,819] ERROR __init__.py Line 30: Traceback (most recent call last):
2023-01-24 08:15:41   File "/app/action_runners/actions/__init__.py", line 27, in <module>
2023-01-24 08:15:41     from .DeepCheckImagePropertyOutliers import DeepcheckImagePropertyOutliers
2023-01-24 08:15:41   File "/app/action_runners/actions/DeepCheckImagePropertyOutliers.py", line 3, in <module>
2023-01-24 08:15:41     from deepchecks.vision.checks import ImagePropertyOutliers
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/__init__.py", line 12, in <module>
2023-01-24 08:15:41     from .base_checks import ModelOnlyCheck, SingleDatasetCheck, TrainTestCheck
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/base_checks.py", line 25, in <module>
2023-01-24 08:15:41     from deepchecks.vision.batch_wrapper import Batch
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/batch_wrapper.py", line 19, in <module>
2023-01-24 08:15:41     from deepchecks.vision.utils.image_functions import crop_image
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/utils/image_functions.py", line 15, in <module>
2023-01-24 08:15:41     import cv2
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 181, in <module>
2023-01-24 08:15:41     bootstrap()
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 153, in bootstrap
2023-01-24 08:15:41     native_module = importlib.import_module("cv2")
2023-01-24 08:15:41   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
2023-01-24 08:15:41     return _bootstrap._gcd_import(name[level:], package, level)
2023-01-24 08:15:41 ImportError: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/local/lib/python3.7/site-packages/cv2/../opencv_contrib_python.libs/libpng16-186fce2e.so.16.37.0)
2023-01-24 08:15:41 
2023-01-24 08:15:42 [2023-01-24 07:15:42,153] ERROR __init__.py Line 42: Traceback (most recent call last):
2023-01-24 08:15:42   File "/app/action_runners/actions/__init__.py", line 39, in <module>
2023-01-24 08:15:42     from .VertexTrainDatasetAction import VertexTrainDatasetAction
2023-01-24 08:15:42   File "/app/action_runners/actions/VertexTrainDatasetAction.py", line 1, in <module>
2023-01-24 08:15:42     from eventhandlers.action_runners.base.ActionRunner import ActionRunner
2023-01-24 08:15:42 ModuleNotFoundError: No module named 'eventhandlers'
2023-01-24 08:15:42 
2023-01-24 08:15:42 [2023-01-24 07:15:42,156] INFO system_startup_checker.py Line 18: [eventhandlers] Performing System Checks...
2023-01-24 08:15:42 [2023-01-24 07:15:42,793] INFO system_events.py Line 95: Checking for version upgrades [eventhandlers]
2023-01-24 08:15:32 Error loading secrets: 
2023-01-24 08:15:32 No module named 'shared.settings.secrets'
2023-01-24 08:15:32 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-24 08:15:32 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-24 08:15:32 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-24 08:15:32 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-24 08:15:32 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-24 08:15:32 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-24 08:15:32 DATABASE_URL DATABASE_URL  True
2023-01-24 08:15:32 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000
2023-01-24 08:16:27 Error loading secrets: 
2023-01-24 08:16:27 No module named 'shared.settings.secrets'
2023-01-24 08:16:27 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-24 08:16:27 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-24 08:16:27 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-24 08:16:27 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-24 08:16:27 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-24 08:16:27 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-24 08:16:27 DATABASE_URL DATABASE_URL  True
2023-01-24 08:16:27 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000
2023-01-24 08:18:28 Error loading secrets: 
2023-01-24 08:18:28 No module named 'shared.settings.secrets'
2023-01-24 08:18:28 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-24 08:18:28 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-24 08:18:28 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-24 08:18:28 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-24 08:18:28 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-24 08:18:28 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-24 08:18:28 DATABASE_URL DATABASE_URL  True
2023-01-24 08:18:28 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000
2023-01-24 08:16:39 [2023-01-24 07:16:39,864] ERROR __init__.py Line 30: Traceback (most recent call last):
2023-01-24 08:16:39   File "/app/action_runners/actions/__init__.py", line 27, in <module>
2023-01-24 08:16:39     from .DeepCheckImagePropertyOutliers import DeepcheckImagePropertyOutliers
2023-01-24 08:16:39   File "/app/action_runners/actions/DeepCheckImagePropertyOutliers.py", line 3, in <module>
2023-01-24 08:16:39     from deepchecks.vision.checks import ImagePropertyOutliers
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/__init__.py", line 12, in <module>
2023-01-24 08:16:39     from .base_checks import ModelOnlyCheck, SingleDatasetCheck, TrainTestCheck
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/base_checks.py", line 25, in <module>
2023-01-24 08:16:39     from deepchecks.vision.batch_wrapper import Batch
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/batch_wrapper.py", line 19, in <module>
2023-01-24 08:16:39     from deepchecks.vision.utils.image_functions import crop_image
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/utils/image_functions.py", line 15, in <module>
2023-01-24 08:16:39     import cv2
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 181, in <module>
2023-01-24 08:16:39     bootstrap()
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 153, in bootstrap
2023-01-24 08:16:39     native_module = importlib.import_module("cv2")
2023-01-24 08:16:39   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
2023-01-24 08:16:39     return _bootstrap._gcd_import(name[level:], package, level)
2023-01-24 08:16:39 ImportError: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/local/lib/python3.7/site-packages/cv2/../opencv_contrib_python.libs/libpng16-186fce2e.so.16.37.0)
2023-01-24 08:16:39 
2023-01-24 08:16:40 [2023-01-24 07:16:40,716] ERROR __init__.py Line 42: Traceback (most recent call last):
2023-01-24 08:16:40   File "/app/action_runners/actions/__init__.py", line 39, in <module>
2023-01-24 08:16:40     from .VertexTrainDatasetAction import VertexTrainDatasetAction
2023-01-24 08:16:40   File "/app/action_runners/actions/VertexTrainDatasetAction.py", line 1, in <module>
2023-01-24 08:16:40     from eventhandlers.action_runners.base.ActionRunner import ActionRunner
2023-01-24 08:16:40 ModuleNotFoundError: No module named 'eventhandlers'
2023-01-24 08:16:40 
2023-01-24 08:16:40 [2023-01-24 07:16:40,720] INFO system_startup_checker.py Line 18: [eventhandlers] Performing System Checks...
2023-01-24 08:16:42 [2023-01-24 07:16:42,130] INFO system_events.py Line 95: Checking for version upgrades [eventhandlers]
2023-01-24 08:18:51 [2023-01-24 07:18:51,740] ERROR __init__.py Line 30: Traceback (most recent call last):
2023-01-24 08:18:51   File "/app/action_runners/actions/__init__.py", line 27, in <module>
2023-01-24 08:18:51     from .DeepCheckImagePropertyOutliers import DeepcheckImagePropertyOutliers
2023-01-24 08:18:51   File "/app/action_runners/actions/DeepCheckImagePropertyOutliers.py", line 3, in <module>
2023-01-24 08:18:51     from deepchecks.vision.checks import ImagePropertyOutliers
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/__init__.py", line 12, in <module>
2023-01-24 08:18:51     from .base_checks import ModelOnlyCheck, SingleDatasetCheck, TrainTestCheck
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/base_checks.py", line 25, in <module>
2023-01-24 08:18:51     from deepchecks.vision.batch_wrapper import Batch
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/batch_wrapper.py", line 19, in <module>
2023-01-24 08:18:51     from deepchecks.vision.utils.image_functions import crop_image
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/utils/image_functions.py", line 15, in <module>
2023-01-24 08:18:51     import cv2
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 181, in <module>
2023-01-24 08:18:51     bootstrap()
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 153, in bootstrap
2023-01-24 08:18:51     native_module = importlib.import_module("cv2")
2023-01-24 08:18:51   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
2023-01-24 08:18:51     return _bootstrap._gcd_import(name[level:], package, level)
2023-01-24 08:18:51 ImportError: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/local/lib/python3.7/site-packages/cv2/../opencv_contrib_python.libs/libpng16-186fce2e.so.16.37.0)
2023-01-24 08:18:51 
2023-01-24 08:18:52 [2023-01-24 07:18:52,756] ERROR __init__.py Line 42: Traceback (most recent call last):
2023-01-24 08:18:52   File "/app/action_runners/actions/__init__.py", line 39, in <module>
2023-01-24 08:18:52     from .VertexTrainDatasetAction import VertexTrainDatasetAction
2023-01-24 08:18:52   File "/app/action_runners/actions/VertexTrainDatasetAction.py", line 1, in <module>
2023-01-24 08:18:52     from eventhandlers.action_runners.base.ActionRunner import ActionRunner
2023-01-24 08:18:52 ModuleNotFoundError: No module named 'eventhandlers'
2023-01-24 08:18:52 
2023-01-24 08:18:52 [2023-01-24 07:18:52,768] INFO system_startup_checker.py Line 18: [eventhandlers] Performing System Checks...
2023-01-24 08:18:53 [2023-01-24 07:18:53,870] INFO system_events.py Line 95: Checking for version upgrades [eventhandlers]
2023-01-24 08:19:36 [2023-01-24 07:19:36,101] ERROR __init__.py Line 30: Traceback (most recent call last):
2023-01-24 08:19:36   File "/app/action_runners/actions/__init__.py", line 27, in <module>
2023-01-24 08:19:36     from .DeepCheckImagePropertyOutliers import DeepcheckImagePropertyOutliers
2023-01-24 08:19:36   File "/app/action_runners/actions/DeepCheckImagePropertyOutliers.py", line 3, in <module>
2023-01-24 08:19:36     from deepchecks.vision.checks import ImagePropertyOutliers
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/__init__.py", line 12, in <module>
2023-01-24 08:19:36     from .base_checks import ModelOnlyCheck, SingleDatasetCheck, TrainTestCheck
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/base_checks.py", line 25, in <module>
2023-01-24 08:19:36     from deepchecks.vision.batch_wrapper import Batch
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/batch_wrapper.py", line 19, in <module>
2023-01-24 08:19:36     from deepchecks.vision.utils.image_functions import crop_image
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/site-packages/deepchecks/vision/utils/image_functions.py", line 15, in <module>
2023-01-24 08:19:36     import cv2
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 181, in <module>
2023-01-24 08:19:36     bootstrap()
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/site-packages/cv2/__init__.py", line 153, in bootstrap
2023-01-24 08:19:36     native_module = importlib.import_module("cv2")
2023-01-24 08:19:36   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
2023-01-24 08:19:36     return _bootstrap._gcd_import(name[level:], package, level)
2023-01-24 08:19:36 ImportError: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/local/lib/python3.7/site-packages/cv2/../opencv_contrib_python.libs/libpng16-186fce2e.so.16.37.0)
2023-01-24 08:19:36 
2023-01-24 08:19:37 [2023-01-24 07:19:37,158] ERROR __init__.py Line 42: Traceback (most recent call last):
2023-01-24 08:19:37   File "/app/action_runners/actions/__init__.py", line 39, in <module>
2023-01-24 08:19:37     from .VertexTrainDatasetAction import VertexTrainDatasetAction
2023-01-24 08:19:37   File "/app/action_runners/actions/VertexTrainDatasetAction.py", line 1, in <module>
2023-01-24 08:19:37     from eventhandlers.action_runners.base.ActionRunner import ActionRunner
2023-01-24 08:19:37 ModuleNotFoundError: No module named 'eventhandlers'
2023-01-24 08:19:37 
2023-01-24 08:19:37 [2023-01-24 07:19:37,163] INFO system_startup_checker.py Line 18: [eventhandlers] Performing System Checks...
2023-01-24 08:19:40 [2023-01-24 07:19:40,337] INFO system_events.py Line 95: Checking for version upgrades [eventhandlers]
2023-01-24 08:19:06 Error loading secrets: 
2023-01-24 08:19:06 No module named 'shared.settings.secrets'
2023-01-24 08:19:06 DIFFGRAM_ERROR_SEND_TRACES_IN_RESPONSE True
2023-01-24 08:19:06 PROCESS_MEDIA_TRY_BLOCK_ON True
2023-01-24 08:19:06 PROCESS_MEDIA_REMOTE_QUEUE_ON True
2023-01-24 08:19:06 PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
2023-01-24 08:19:06 DIFFGRAM_MINIO_DISABLED_SSL_VERIFY False
2023-01-24 08:19:06 DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  sandbox
2023-01-24 08:19:06 DATABASE_URL DATABASE_URL  True
2023-01-24 08:19:06 SIGNED_URL_CACHE_MINIMUM_SECONDS_VALID 18000

If I use SDK instead, replacing from_url with from_local in this tutorial will do the job?

Thanks so much!
Opal

[anthony-chaudhary]

It looks like an assumption with opencv opencv/opencv-python#772
( ZLIB
ImportError: /lib/x86_64-linux-gnu/libz.so.1: version ZLIB_1.2.9' not found (required by /usr/local/lib/python3.7/site`)
I will look at what version we have pinned

The event handler service crashing may be part of it

For future debugging videos this is the chrome dev console we need (not docker)
https://diffgram.readme.io/docs/optimize#instructions

[anthony-chaudhary]

#1296

[anthony-chaudhary]

@opallch
try upgrading to 1.16.5 and let us know what you see
https://diffgram.readme.io/docs/update-dev

[opallch]

@Anthony-Sarkis I got an internal server error:

[anthony-chaudhary]

So the error shows when trying to upload that the key doesn't exist.
This may be related to the storage configuration.

If you go to the docker logs for the walrus container (while uploading) what is the raw_data_blob_path?
If you use another storage provider, do you get the same issue?
Do you get the same issue with an image file type as you do text?

[opallch]

These are what I got from the docker logs:

2023-01-25 09:35:08 [2023-01-25 08:35:08,754] INFO upload.py Line 256: upload_large_api: raw_data_blob_path projects/raw_input/1/raw/1
2023-01-25 09:39:46 [2023-01-25 08:39:46,160] INFO upload.py Line 256: upload_large_api: raw_data_blob_path projects/raw_input/1/raw/2

The same issue came up also for image files.

I haven't tried another storage provider yet, but I will get back to you later. But that means I need to rebuild the image and choose another storage provider, right?

[anthony-chaudhary]

Re-installing is one option.
You can also edit the ENV file directly, by changing the provider name and required items.

Please note the docker install script is provided as a convenience to play with it, so we only have limited support for cases of re-installing etc. You or your IT dept would need to follow this guide for any production data https://diffgram.readme.io/docs/open-installation-production. You can also use minikube to play with it.

The system does some self-checks to see if it can read/write, so it's unexpected to be having this issue.
Can you try accessing MinIO on it's own and see if you can upload files etc.?

[anthony-chaudhary]

Thank you for that info on the key. It appears ok.
I would double check that diffgram-storage bucket exists in your storage provider,
and that name matches your ENV file configuration.

The total path includes the bucket. There was that error earlier on the bucket creation. So I would check in the MinIO UI if that bucket actually exists and any permissions etc. And double check there aren't multiple installations of MinIO etc.

[opallch]

I don't know if it helps, but everytime when I start the minio container I will get this error:

2023-01-26 10:28:04 API: SYSTEM()
2023-01-26 10:28:04 Time: 09:28:04 UTC 01/26/2023
2023-01-26 10:28:04 DeploymentID: 09de32a4-d5eb-44de-af96-77e5d751aacd
2023-01-26 10:28:04 Error: write /export/.minio.sys/tmp/4a2ea6da-14d6-4641-803a-bc9a9141fa7a/xl.meta: invalid argument (*fs.PathError)
2023-01-26 10:28:04        6: internal/logger/logger.go:258:logger.LogIf()
2023-01-26 10:28:04        5: cmd/storage-errors.go:165:cmd.osErrToFileErr()
2023-01-26 10:28:04        4: cmd/xl-storage.go:2396:cmd.(*xlStorage).RenameData()
2023-01-26 10:28:04        3: cmd/xl-storage-disk-id-check.go:378:cmd.(*xlStorageDiskIDCheck).RenameData()
2023-01-26 10:28:04        2: cmd/erasure-object.go:759:cmd.renameData.func1()
2023-01-26 10:28:04        1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
2023-01-26 10:28:04 Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to initialize config system: migrateConfigToMinioSys: Storage resources are insufficient for the write operation .minio.sys/config/config.json)

It seems that it could be an minio issue, and I can't even access the minio interface for uploading data when just accessing minio.
Edit: and since we are not going to use any cloud storage provider, I can't if the issue exists in the three providers suggested in the installation.

[anthony-chaudhary]

Right. `insufficient for the write operation` Check if your hard disk or virtual disc has any free space.

…

On Thu, Jan 26, 2023, 1:33 AM Opal Leung ***@***.***> wrote: I don't know if it helps, but everytime when I start the minio container I will get this error: 2023-01-26 10:28:04 API: SYSTEM() 2023-01-26 10:28:04 Time: 09:28:04 UTC 01/26/2023 2023-01-26 10:28:04 DeploymentID: 09de32a4-d5eb-44de-af96-77e5d751aacd 2023-01-26 10:28:04 Error: write /export/.minio.sys/tmp/4a2ea6da-14d6-4641-803a-bc9a9141fa7a/xl.meta: invalid argument (*fs.PathError) 2023-01-26 10:28:04 6: internal/logger/logger.go:258:logger.LogIf() 2023-01-26 10:28:04 5: cmd/storage-errors.go:165:cmd.osErrToFileErr() 2023-01-26 10:28:04 4: cmd/xl-storage.go:2396:cmd.(*xlStorage).RenameData() 2023-01-26 10:28:04 3: cmd/xl-storage-disk-id-check.go:378:cmd.(*xlStorageDiskIDCheck).RenameData() 2023-01-26 10:28:04 2: cmd/erasure-object.go:759:cmd.renameData.func1() 2023-01-26 10:28:04 1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() 2023-01-26 10:28:04 Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to initialize config system: migrateConfigToMinioSys: Storage resources are insufficient for the write operation .minio.sys/config/config.json) It seems that it could be an minio issue, and I can't even access the minio interface for uploading data. — Reply to this email directly, view it on GitHub <#1294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJ6DJC24T7VZ4VG4AEPKG3WUJAE7ANCNFSM6AAAAAAUEDPJ6E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[opallch]

I have changed the volume binding for minio in docker-compose.yaml and docker-compose up, and now it works:

  minio:
    image: minio/minio
    ports:
      - "9000:9000"
      - "9001:9001"
    expose: [9000, 9001]
    volumes:
      - minio_storage:/export
      - ./test/.minio/config:/root/.minio
    environment:
      - "MINIO_ROOT_USER=${DIFFGRAM_MINIO_ACCESS_KEY_ID}"
      - "MINIO_ROOT_PASSWORD=${DIFFGRAM_MINIO_ACCESS_KEY_SECRET}"
    command: server /export --console-address :9001

  createbuckets:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add myminio http://minio:9000 ${DIFFGRAM_MINIO_ACCESS_KEY_ID} ${DIFFGRAM_MINIO_ACCESS_KEY_SECRET};
      /usr/bin/mc mb myminio/${DIFFGRAM_S3_BUCKET_NAME};
      /usr/bin/mc policy set policy myminio/${DIFFGRAM_S3_BUCKET_NAME} ;
      exit 0;
      "
volumes:
  static: {}
  minio_storage: {}

[anthony-chaudhary]

Great!

…

On Thu, Jan 26, 2023 at 9:01 AM Opal Leung ***@***.***> wrote: I have changed the volume binding for minio in docker-compose.yaml and docker-compose up, and now it works: minio: image: minio/minio ports: - "9000:9000" - "9001:9001" expose: [9000, 9001] volumes: - minio_storage:/export - ./test/.minio/config:/root/.minio environment: - "MINIO_ROOT_USER=${DIFFGRAM_MINIO_ACCESS_KEY_ID}" - "MINIO_ROOT_PASSWORD=${DIFFGRAM_MINIO_ACCESS_KEY_SECRET}" command: server /export --console-address :9001 createbuckets: image: minio/mc depends_on: - minio entrypoint: > /bin/sh -c " /usr/bin/mc config host add myminio http://minio:9000 ${DIFFGRAM_MINIO_ACCESS_KEY_ID} ${DIFFGRAM_MINIO_ACCESS_KEY_SECRET}; /usr/bin/mc mb myminio/${DIFFGRAM_S3_BUCKET_NAME}; /usr/bin/mc policy set policy myminio/${DIFFGRAM_S3_BUCKET_NAME} ; exit 0; " volumes: static: {} minio_storage: {} — Reply to this email directly, view it on GitHub <#1294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJ6DJEECXUT6AXDBTXYNELWUKUV7ANCNFSM6AAAAAAUEDPJ6E> . You are receiving this because you were mentioned.Message ID: ***@***.***>