Question of the error message when trying to upload #85

chenguangsmy · 2022-12-01T20:36:26Z

chenguangsmy
Dec 1, 2022

Hi everybody, I'm trying to upload my data into dandi and I have finished the data configuration under the help of Dr. Baker. However, I encountered one problem when I tried to upload the data. My computer is MacBook 2021 with M1 chip, and I run the code in the terminal. I installed python3. Here is specific information of the error:

The error came when I execute:

dandi organize /Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338 -f dry

I pasted the entire message with an error like the following:

2022-12-01 12:45:55,518 [ INFO] Loading metadata from 1 files
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 10 concurrent workers.
/opt/homebrew/lib/python3.10/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.5.1 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/opt/homebrew/lib/python3.10/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.1.0 because version 0.2.0 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
[Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 1.0s
[Parallel(n_jobs=-1)]: Done 1 out of 1 | elapsed: 1.0s finished
2022-12-01 12:45:56,541 [ WARNING] subject_id is missing for /Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/multiRaggedExpandableTableTest_GPW1.nwb
2022-12-01 12:45:56,542 [ INFO] Logs saved in /Users/cleave/Library/Logs/dandi-cli/20221201174554Z-7586.log
Error: 1 out of 1 files were found not containing all necessary metadata: /Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/multiRaggedExpandableTableTest_GPW1.nwb

I don't know what its means that "not containing all necessary metadata". Can you help me deal with that?

I also tried to skip this step and execute the upload.

dandi upload

I got the following message with one error:

2022-12-01 15:15:21,871 [ ERROR] Error 401 while sending GET request to https://api.dandiarchive.org/api/auth/token: {"detail":"Invalid token."}

Can you help me understand what's happening? Thank you so much!

Answered by CodyCBakerPhD

Dec 1, 2022

The second issue of

2022-12-01 15:15:21,871 [ ERROR] Error 401 while sending GET request to https://api.dandiarchive.org/api/auth/token: {"detail":"Invalid token."}

indicates it couldn't find the API token during upload.

It should, as I understand, prompt you for that if it can't find it, rather than just returning an error, but in any case this ought to be resolved by following the guide for storing DANDI credentials here: https://www.dandiarchive.org/handbook/13_upload/#storing-access-credentials

View full answer

CodyCBakerPhD · 2022-12-01T20:39:52Z

CodyCBakerPhD
Dec 1, 2022

The 'missing metadata' is included (albeit wrapped around a lot of technical-looking output from the call to organize) - maybe that could be formatted better in the DANDI CLI:

2022-12-01 12:45:56,541 [ WARNING] subject_id is missing for: /Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/multiRaggedExpandableTableTest_GPW1.nwb

DANDI requires subject IDs to be assigned in each NWB file; out of curiosity, this should have also been reported in the earlier step of dandi validate path/to/nwbfiles - did calling dandi validate also report this missing metadata?

5 replies

chenguangsmy Dec 2, 2022
Author

Calling dandi validate also report that missing. I debugged with it but came with a new question.

When I was doing dandi organize
I came across the error message like this:

2022-12-01 23:35:52,114 [ INFO] Loading metadata from 2 files
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 10 concurrent workers.
/opt/homebrew/lib/python3.10/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.5.1 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/opt/homebrew/lib/python3.10/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.5.1 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/opt/homebrew/lib/python3.10/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.1.0 because version 0.2.0 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/opt/homebrew/lib/python3.10/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.1.0 because version 0.2.0 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
[Parallel(n_jobs=-1)]: Done 2 out of 2 | elapsed: 1.1s remaining: 0.0s
[Parallel(n_jobs=-1)]: Done 2 out of 2 | elapsed: 1.1s finished
2022-12-01 23:35:53,187 [ INFO] 1 out of 2 paths are not unique. We will try adding _obj- based on crc32 of object_id
2022-12-01 23:35:53,188 [ INFO] Logs saved in /Users/cleave/Library/Logs/dandi-cli/20221202043551Z-11714.log
Error: Two files ('/Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/000338/sub-001/sub-001.nwb' and '/Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/multiRaggedExpandableTableTest_GPW1.nwb') have the same object_id 33f5cdf4-9a9e-4217-8cb5-d6eac81cd9c1. Must not happen. Either files are duplicates (remove one) or were not saved correctly using NWB>=2.1.0 standard (supported by pynwb>=1.1.0).

What's this error means? How do I deal with it?

CodyCBakerPhD Dec 2, 2022

The only thing I can think of to check is - are the session_ids of each file unique? If so, what about the identifier of each file?

chenguangsmy Dec 2, 2022
Author

The file /Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/000338/sub-001/sub-001.nwb seems to be a generated file after I execute the validation.
The problem was solved after I deleted the folder and re-validated it.

Thank you for answering the question. I didn't anticipate I have to validate it before organizing.

yarikoptic Dec 2, 2022
Maintainer

The file /Users/cleave/Documents/projBNU/delaywalking_analysis/Output_figures/000338/000338/sub-001/sub-001.nwb seems to be a generated file after I execute the validation.

do you really mean that dandi validate generated that file? or do you mean that you dandi organize'ed data under the same path, thus leading to the 000338/000338 ?

PS I just want to make sure we do not have some issue in dandi cli...

chenguangsmy Dec 2, 2022
Author

Actually, I don't know which line generated the file sub-001.nwb. I did not create that using Matlab.
For the 000338/000338 folder, the father folder is I created using the command mkdir, and the son folder is created after I executed the download, or organized the data.

CodyCBakerPhD · 2022-12-01T20:44:02Z

CodyCBakerPhD
Dec 1, 2022

The second issue of

2022-12-01 15:15:21,871 [ ERROR] Error 401 while sending GET request to https://api.dandiarchive.org/api/auth/token: {"detail":"Invalid token."}

indicates it couldn't find the API token during upload.

It should, as I understand, prompt you for that if it can't find it, rather than just returning an error, but in any case this ought to be resolved by following the guide for storing DANDI credentials here: https://www.dandiarchive.org/handbook/13_upload/#storing-access-credentials

2 replies

yarikoptic Dec 2, 2022
Maintainer

yes -- it should have asked. And if not - at least logged at debug level where it got that invalid token since version 0.46.5 of dandi client. Which version of client do you have? (I failed to locate it here)

chenguangsmy Dec 5, 2022
Author

My dandi version si 0.46.6

kysu1313 · 2022-12-28T23:43:53Z

kysu1313
Dec 28, 2022

😂 llmao.. don't click that link. [image: image.png] its malware

…

On Wed, Dec 28, 2022 at 6:39 PM YaimSputnik5 ***@***.***> wrote: Really bad phishing attempt. — Reply to this email directly, view it on GitHub <#85 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJGA5IMUZK746ZC46NOJBSLWPTFT7ANCNFSM6AAAAAASRGFM4E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question of the error message when trying to upload #85

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Question of the error message when trying to upload #85

chenguangsmy Dec 1, 2022

Replies: 4 comments · 8 replies

CodyCBakerPhD Dec 1, 2022

chenguangsmy Dec 2, 2022 Author

CodyCBakerPhD Dec 2, 2022

chenguangsmy Dec 2, 2022 Author

yarikoptic Dec 2, 2022 Maintainer

chenguangsmy Dec 2, 2022 Author

CodyCBakerPhD Dec 1, 2022

yarikoptic Dec 2, 2022 Maintainer

chenguangsmy Dec 5, 2022 Author

kysu1313 Dec 28, 2022

chenguangsmy
Dec 1, 2022

Replies: 4 comments 8 replies

CodyCBakerPhD
Dec 1, 2022

chenguangsmy Dec 2, 2022
Author

chenguangsmy Dec 2, 2022
Author

yarikoptic Dec 2, 2022
Maintainer

chenguangsmy Dec 2, 2022
Author

CodyCBakerPhD
Dec 1, 2022

yarikoptic Dec 2, 2022
Maintainer

chenguangsmy Dec 5, 2022
Author

kysu1313
Dec 28, 2022