--mode verify detects lots of unexpected diffs in metadata without server timestamp boost

[yarikoptic]

After

• FOI: backups2datalad-update-cron --mode verify got stuck #48

I manually ran the --mode verify sweep and it errorred out quite loudly -- here is the trail pointing to the full log

    +---------------- 15 ----------------
    | Traceback (most recent call last):
    |   File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 257, in process_blob
    |     raise UnexpectedChangeError(
    | backups2datalad.util.UnexpectedChangeError: Dandiset 000966: Metadata for asset sub-M230804-1/sub-M230804-1_ses-20231229T155815_ecephys.nwb was changed/added but draft timestamp was not updated on server:
    |
    | Metadata diff:
    |
    | --- old-metadata
    | +++ new-metadata
    | @@ -13,7 +13,7 @@
    |    contentSize: 247230064
    |    contentUrl:
    |    - https://api.dandiarchive.org/api/assets/acf6172c-d85f-4a22-ae19-6ba011a53e31/download/
    | -  - https://dandiarchive-embargo.s3.amazonaws.com/000966/blobs/b53/94e/b5394ed4-e80f-4fdf-bbc8-5d82717cf42a
    | +  - https://dandiarchive.s3.amazonaws.com/blobs/b53/94e/b5394ed4-e80f-4fdf-bbc8-5d82717cf42a
    |    dateModified: '2024-04-21T18:07:38.991543-04:00'
    |    digest:
    |      dandi:dandi-etag: 8fa0a66dc8ae41e2f124bf036cfc6594-4
    | @@ -77,6 +77,6 @@
    |        schemaKey: Software
    |        url: https://github.com/dandi/dandi-cli
    |        version: 0.61.2
    | -modified: '2024-04-21T22:07:46.774560Z'
    | +modified: '2024-04-29T19:35:16.321230Z'
    |  path: sub-M230804-1/sub-M230804-1_ses-20231229T155815_ecephys.nwb
    |  size: 247230064
    |
    |
    +---------------- ... ----------------
    | and 3 more exceptions
    +------------------------------------
2024-06-06T20:28:53-0400 [ERROR   ] backups2datalad: An error occurred:
Traceback (most recent call last):
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 119, in wrapped
    await f(datasetter, *args, **kwargs)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 228, in update_from_backup
    await datasetter.update_from_backup(dandisets, exclude=exclude)
  File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 94, in update_from_backup
    raise RuntimeError(
RuntimeError: Backups for 162 Dandisets failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.06.07.00.22.30Z.log
action summary:
  publish (notneeded: 2)

from which it looks like potentially unemabrgoing forgetting to reset the modified may be?

[jwodder]

@yarikoptic The error message seems pretty clear to me:

backups2datalad.util.UnexpectedChangeError: Dandiset 000966: Metadata for asset sub-M230804-1/sub-M230804-1_ses-20231229T155815_ecephys.nwb was changed/added but draft timestamp was not updated on server

This is the Archive's fault for not updating the Dandiset's draft version's modified timestamp upon unembargoing. Running the backup command with --mode force should get rid of the error.

[yarikoptic]

But it is RuntimeError: Backups for 162 Dandisets failed -- is there already so many dandisets which were unembargoed??? (very unlikely)

[jwodder]

@yarikoptic Based on the below script, there are only 6 Dandisets that have been unembargoed (000253, 000408, 000773, 000774, 000897, and 000935).

Is the problem described in the original comment still an issue?

#!/bin/bash
set -eu -o pipefail

dandiset_root=/mnt/backup/dandi/dandisets

cd "$dandiset_root"
for ds in 0*
do
    embargo_status="$(git -C "$ds" config --file .datalad/config --default OPEN --get dandi.dandiset.embargo-status)"
    if [ "$embargo_status" = OPEN ] \
        && git -C "$ds" log -S EMBARGOED -n1 -- .datalad/config | grep -q .
    then echo "$ds"
    fi
done

[jwodder]

@yarikoptic Ping.

[yarikoptic]

blocked by #56 ATM. Please just rerun that command with --verify whenever we do not have ongoing backup process running

[jwodder]

@yarikoptic This problem is still occurring, but seeing as it's affecting Dandisets that are still embargoed, the problem seems to be solely with Dandi Archive. I have filed dandi/dandi-archive#2002.

[yarikoptic]

currently, after @jjnesbitt ping on dandi/dandi-archive#2002 we have

• 000719 - zarr update within dandiset draft timestamp update

    | backups2datalad.util.UnexpectedChangeError: Dandiset 000719: Zarr 7e1b3b36-a94a-427f-9793-b14e344a04f2: entry 'acquisition/TwoPhotonSeries/data/10006.5.7' added, but Dandiset draft timestamp was not updated on server

• 000565, 000714, 000776: might be just a warning to attend to (smells smth odd with timestamps thus relates) or ignore?

2024-10-09T11:33:23-0400 [WARNING ] backups2datalad: Dandiset 000565: While tagging version 0.241009.1504 of Dandiset 000565, found candidate commits both before and after 2024-10-09 15:04:38.861038+00:00 with matching asset metadata: 5f00d78f32db11018b2d0790c8f1843d7b673f2e, 48df17fa24f4e9d14cb0d77441bc0631e91d21e4
2024-10-09T11:34:06-0400 [WARNING ] backups2datalad: Dandiset 000714: While tagging version 0.241009.1516 of Dandiset 000714, found candidate commits both before and after 2024-10-09 15:16:35.573148+00:00 with matching asset metadata: 7cd97fc4015b72a2e1a87b12c7dc00530b8dc5d8, 77aae62001855a0c21d181e6eb7f143153997342
2024-10-09T11:35:03-0400 [WARNING ] backups2datalad: Dandiset 000776: While tagging version 0.241009.1509 of Dandiset 000776, found candidate commits both before and after 2024-10-09 15:09:37.408888+00:00 with matching asset metadata: fc640a26c5aabbb923ac97f815de8d3720692007, 17b1751554a0e2a573f46e9ca879fdb504065fa2

• 000874 - metadata diff without draft timestamp update for some assets, e.g.

  | backups2datalad.util.UnexpectedChangeError: Dandiset 000874: Metadata for asset derivatives/MRI-pipeline/sub-SP002/anat/sub-SP002_ses-MRI_flip-3_chunk-1_to-SP002_xfm.json was changed/added but draft timestamp was not updated on server:

detailed diff - so went from null to filled out

    | Metadata diff:
    |
    | --- old-metadata
    | +++ new-metadata
    | @@ -1,2 +1,51 @@
    | -null
    | -...
    | +asset_id: a2958ce9-a5c8-4137-9fd0-ace1ffd0655b
    | +blob: 8a0a157a-58d7-463a-885b-796202eefe7d
    | +created: '2024-07-15T15:29:35.791159Z'
    | +metadata:
    | +  '@context': https://raw.githubusercontent.com/dandi/schema/master/releases/0.6.7/context.json
    | +  access:
    | +  - schemaKey: AccessRequirements
    | +    status: dandi:OpenAccess
    | +  blobDateModified: '2024-02-06T13:30:46.886873-05:00'
    | +  contentSize: 173
    | +  contentUrl:
    | +  - https://api.dandiarchive.org/api/assets/a2958ce9-a5c8-4137-9fd0-ace1ffd0655b/download/
    | +  - https://dandiarchive.s3.amazonaws.com/blobs/8a0/a15/8a0a157a-58d7-463a-885b-796202eefe7d
    | +  dateModified: '2024-07-15T11:29:34.548579-04:00'
    | +  digest:
    | +    dandi:dandi-etag: be589a56cf11aa658c8a8a368c90d299-1
    | +    dandi:sha2-256: f3692617d26821b40426a41de2ddcbdfbf4bc29e59a9f413c9bd6d0bd9b43ca9
    | +  encodingFormat: application/json
    | +  id: dandiasset:a2958ce9-a5c8-4137-9fd0-ace1ffd0655b
    | +  identifier: a2958ce9-a5c8-4137-9fd0-ace1ffd0655b
    | +  path: derivatives/MRI-pipeline/sub-SP002/anat/sub-SP002_ses-MRI_flip-3_chunk-1_to-SP002_xfm.json
    | +  schemaKey: Asset
    | +  schemaVersion: 0.6.7
    | +  wasGeneratedBy:
    | +  - description: Metadata generated by DANDI cli
    | +    endDate: '2024-07-15T11:29:34.217129-04:00'
    | +    id: urn:uuid:7069a42d-fedc-4eb5-a275-c4d578a18093
    | +    name: Metadata generation
    | +    schemaKey: Activity
    | +    startDate: '2024-07-15T11:29:34.217129-04:00'
    | +    wasAssociatedWith:
    | +    - identifier: RRID:SCR_019009
    | +      name: DANDI Command Line Interface
    | +      schemaKey: Software
    | +      url: https://github.com/dandi/dandi-cli
    | +      version: 0.62.3
    | +  - description: Metadata generated by DANDI cli
    | +    endDate: '2024-07-15T11:29:34.548518-04:00'
    | +    id: urn:uuid:738f5381-7476-429a-9efc-c00a0f058c6e
    | +    name: Metadata generation
    | +    schemaKey: Activity
    | +    startDate: '2024-07-15T11:29:34.548518-04:00'
    | +    wasAssociatedWith:
    | +    - identifier: RRID:SCR_019009
    | +      name: DANDI Command Line Interface
    | +      schemaKey: Software
    | +      url: https://github.com/dandi/dandi-cli
    | +      version: 0.62.3
    | +modified: '2024-07-15T15:29:35.791174Z'
    | +path: derivatives/MRI-pipeline/sub-SP002/anat/sub-SP002_ses-MRI_flip-3_chunk-1_to-SP002_xfm.json
    | +size: 173

and the same for rawdata/dataset_description.json , but for participants.tsv we did get some diff

    | backups2datalad.util.UnexpectedChangeError: Dandiset 000874: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server:
    |
    | Metadata diff:
    |
    | --- old-metadata
    | +++ new-metadata
    | @@ -1,52 +1,52 @@
    | -asset_id: 076f4dfa-8516-46b4-9703-2b5d4138957d
    | -blob: cf1711fc-e358-451f-ab3e-8feb9c670f6f
    | -created: '2024-01-30T16:38:24.357454Z'
    | +asset_id: b4082afd-5343-4b27-914c-4e60aaf92815
    | +blob: e48d261f-2ccd-4597-9ef5-acf52a8945df
    | +created: '2024-07-15T15:44:32.932863Z'
    |  metadata:
    | -  '@context': https://raw.githubusercontent.com/dandi/schema/master/releases/0.6.4/context.json
    | +  '@context': https://raw.githubusercontent.com/dandi/schema/master/releases/0.6.7/context.json
    |    access:
    |    - schemaKey: AccessRequirements
    |      status: dandi:OpenAccess
    | -  blobDateModified: '2024-01-30T11:37:50.304943-05:00'
    | -  contentSize: 41
    | +  blobDateModified: '2024-02-06T13:04:48.704805-05:00'
    | +  contentSize: 48
    |    contentUrl:
    | -  - https://api.dandiarchive.org/api/assets/076f4dfa-8516-46b4-9703-2b5d4138957d/download/
    | -  - https://dandiarchive-embargo.s3.amazonaws.com/000874/blobs/cf1/711/cf1711fc-e358-451f-ab3e-8feb9c670f6f
    | -  dateModified: '2024-01-30T11:38:22.961912-05:00'
    | +  - https://api.dandiarchive.org/api/assets/b4082afd-5343-4b27-914c-4e60aaf92815/download/
    | +  - https://dandiarchive.s3.amazonaws.com/blobs/e48/d26/e48d261f-2ccd-4597-9ef5-acf52a8945df
    | +  dateModified: '2024-07-15T11:44:31.980089-04:00'
    |    digest:
    | -    dandi:dandi-etag: 8b3fbf4a58ce93626641cd1afef2f0a9-1
    | -    dandi:sha2-256: 07fe262186154bebb1a437caf7302b47b94d38475e0a29635c08ad18ef5fb138
    | +    dandi:dandi-etag: 11388fc2440da060752f392b73ec8a9d-1
    | +    dandi:sha2-256: 3314819d2baf964448ce305964bb3f975563b9a54288f25429c9af9a23ef067e
    |    encodingFormat: text/tab-separated-values
    | -  id: dandiasset:076f4dfa-8516-46b4-9703-2b5d4138957d
    | -  identifier: 076f4dfa-8516-46b4-9703-2b5d4138957d
    | +  id: dandiasset:b4082afd-5343-4b27-914c-4e60aaf92815
    | +  identifier: b4082afd-5343-4b27-914c-4e60aaf92815
    |    path: participants.tsv
    |    schemaKey: Asset
    | -  schemaVersion: 0.6.4
    | +  schemaVersion: 0.6.7
    |    wasAttributedTo: []
    |    wasGeneratedBy:
    |    - description: Metadata generated by DANDI cli
    | -    endDate: '2024-01-30T11:38:22.443962-05:00'
    | -    id: urn:uuid:285e914a-2c83-4d5e-b141-dc6355b5ef65
    | +    endDate: '2024-07-15T11:44:31.467804-04:00'
    | +    id: urn:uuid:5ad6c844-ee5c-4276-b736-a76b743528fe
    |      name: Metadata generation
    |      schemaKey: Activity
    | -    startDate: '2024-01-30T11:38:22.443962-05:00'
    | +    startDate: '2024-07-15T11:44:31.467804-04:00'
    |      wasAssociatedWith:
    |      - identifier: RRID:SCR_019009
    |        name: DANDI Command Line Interface
    |        schemaKey: Software
    |        url: https://github.com/dandi/dandi-cli
    | -      version: 0.59.0
    | +      version: 0.62.3
    |    - description: Metadata generated by DANDI cli
    | -    endDate: '2024-01-30T11:38:22.961882-05:00'
    | -    id: urn:uuid:d1fe1adc-42b0-406d-b142-221b1d1094e9
    | +    endDate: '2024-07-15T11:44:31.980028-04:00'
    | +    id: urn:uuid:fddae54a-b1e6-4b83-92de-3ea8d96b8c80
    |      name: Metadata generation
    |      schemaKey: Activity
    | -    startDate: '2024-01-30T11:38:22.961882-05:00'
    | +    startDate: '2024-07-15T11:44:31.980028-04:00'
    |      wasAssociatedWith:
    |      - identifier: RRID:SCR_019009
    |        name: DANDI Command Line Interface
    |        schemaKey: Software
    |        url: https://github.com/dandi/dandi-cli
    | -      version: 0.59.0
    | -modified: '2024-01-30T16:38:24.376045Z'
    | +      version: 0.62.3
    | +modified: '2024-07-15T15:44:32.946532Z'
    |  path: participants.tsv
    | -size: 41
    | +size: 48

similarly a diff on derivatives/OCT-pipeline/sub-SP002/micr/sub-SP002_ses-OCT_sample-01_res-20um_OCT.json

    | --- old-metadata
    | +++ new-metadata
    | @@ -1,52 +1,51 @@
    | -asset_id: fc7f85dd-abd5-488e-857f-efc600e6dd0e
    | -blob: 035e3880-07c5-4fca-85d7-31162988506e
    | -created: '2024-01-30T16:57:58.459166Z'
    | +asset_id: 3e98c412-b4be-4e3d-8709-662e721cba30
    | +blob: d7aab918-d0a0-4abd-b116-edd53b379591
    | +created: '2024-07-15T15:43:57.648669Z'
    |  metadata:
    | -  '@context': https://raw.githubusercontent.com/dandi/schema/master/releases/0.6.4/context.json
    | +  '@context': https://raw.githubusercontent.com/dandi/schema/master/releases/0.6.7/context.json
    |    access:
    |    - schemaKey: AccessRequirements
    |      status: dandi:OpenAccess
    | -  blobDateModified: '2024-01-23T15:00:15.383186-05:00'
    | -  contentSize: 164
    | +  blobDateModified: '2024-02-06T13:23:09.788738-05:00'
    | +  contentSize: 165
    |    contentUrl:
    | -  - https://api.dandiarchive.org/api/assets/fc7f85dd-abd5-488e-857f-efc600e6dd0e/download/
    | -  - https://dandiarchive-embargo.s3.amazonaws.com/000874/blobs/035/e38/035e3880-07c5-4fca-85d7-31162988506e
    | -  dateModified: '2024-01-30T11:57:57.799582-05:00'
    | +  - https://api.dandiarchive.org/api/assets/3e98c412-b4be-4e3d-8709-662e721cba30/download/
    | +  - https://dandiarchive.s3.amazonaws.com/blobs/d7a/ab9/d7aab918-d0a0-4abd-b116-edd53b379591
    | +  dateModified: '2024-07-15T11:43:55.907615-04:00'
    |    digest:
    | -    dandi:dandi-etag: 97fb6db3d084dd8d1e73ea1cef3ec2ca-1
    | -    dandi:sha2-256: a2b6f12545fca55730d5295967fac3f6c3b28c24bd2b56047043057a5ecbbe08
    | +    dandi:dandi-etag: bc5150cdd5c99e1981297ab02c89cbff-1
    | +    dandi:sha2-256: e4affd47e0e521bfb16e05146816b85c58ac9e50ef7db0158b86fe0abeb7689c
    |    encodingFormat: application/json
    | -  id: dandiasset:fc7f85dd-abd5-488e-857f-efc600e6dd0e
    | -  identifier: fc7f85dd-abd5-488e-857f-efc600e6dd0e
    | +  id: dandiasset:3e98c412-b4be-4e3d-8709-662e721cba30
    | +  identifier: 3e98c412-b4be-4e3d-8709-662e721cba30
    |    path: derivatives/OCT-pipeline/sub-SP002/micr/sub-SP002_ses-OCT_sample-01_res-20um_OCT.json
    |    schemaKey: Asset
    | -  schemaVersion: 0.6.4
    | -  wasAttributedTo: []
    | +  schemaVersion: 0.6.7
    |    wasGeneratedBy:
    |    - description: Metadata generated by DANDI cli
    | -    endDate: '2024-01-30T11:57:57.019465-05:00'
    | -    id: urn:uuid:03a8ccfc-6407-41f0-9eab-f5a40c670f11
    | +    endDate: '2024-07-15T11:43:54.388406-04:00'
    | +    id: urn:uuid:83534e7b-f35c-41f4-ae5c-6f3cde372951
    |      name: Metadata generation
    |      schemaKey: Activity
    | -    startDate: '2024-01-30T11:57:57.019465-05:00'
    | +    startDate: '2024-07-15T11:43:54.388406-04:00'
    |      wasAssociatedWith:
    |      - identifier: RRID:SCR_019009
    |        name: DANDI Command Line Interface
    |        schemaKey: Software
    |        url: https://github.com/dandi/dandi-cli
    | -      version: 0.59.0
    | +      version: 0.62.3
    |    - description: Metadata generated by DANDI cli
    | -    endDate: '2024-01-30T11:57:57.799471-05:00'
    | -    id: urn:uuid:5f2bf9bb-dcfd-41e3-b418-4195ce154b24
    | +    endDate: '2024-07-15T11:43:55.907564-04:00'
    | +    id: urn:uuid:fa95ee7b-a5e2-42f1-8fcb-cf0d016d89a9
    |      name: Metadata generation
    |      schemaKey: Activity
    | -    startDate: '2024-01-30T11:57:57.799471-05:00'
    | +    startDate: '2024-07-15T11:43:55.907564-04:00'
    |      wasAssociatedWith:
    |      - identifier: RRID:SCR_019009
    |        name: DANDI Command Line Interface
    |        schemaKey: Software
    |        url: https://github.com/dandi/dandi-cli
    | -      version: 0.59.0
    | -modified: '2024-01-30T16:57:58.459185Z'
    | +      version: 0.62.3
    | +modified: '2024-07-15T15:43:57.700913Z'
    |  path: derivatives/OCT-pipeline/sub-SP002/micr/sub-SP002_ses-OCT_sample-01_res-20um_OCT.json
    | -size: 164
    | +size: 165

so on 2024-07-15 we seems had an upload of few updated files to 000874 (note: embargoed) but we still have metadata for dandiset saying

{
  "identifier": "000874",
  "created": "2024-01-23T18:59:36.596718Z",
  "modified": "2024-02-14T14:36:22.666095Z",

[yarikoptic]

on 000719 zarr issue getting the bits together

{
  "identifier": "000719",
  "created": "2023-11-30T17:09:30.037534Z",
  "modified": "2024-08-22T18:28:38.706169Z",

• zarr is in "status": "Pending",
• that path was added (or is that mtime from filesystem???) on 2024-04-02 13:03

dandi@drogon:~$ s3cmd -c ~/.s3cfg-dandi-backup ls -l s3://dandiarchive/zarr/7e1b3b36-a94a-427f-9793-b14e344a04f2/acquisition/TwoPhotonSeries/data/10006.5.7
2024-04-02 13:03        24908  5feda0eb2702d2422f1451d873caf677     STANDARD     s3://dandiarchive/zarr/7e1b3b36-a94a-427f-9793-b14e344a04f2/acquisition/TwoPhotonSeries/data/10006.5.7

our local zarr git has clean state and

dandi@drogon:/mnt/backup/dandi/dandizarrs/7e1b3b36-a94a-427f-9793-b14e344a04f2$ git show | head -n 20
commit e0f66f4592
Author: DANDI User <info@dandiarchive.org>
Date:   2024 Apr 19 13:25:56 -0400

    [backups2datalad] 510 files added, checksum updated

diff --git a/.dandi/s3sync.json b/.dandi/s3sync.json
index 7a9cc99208..fee70574e1 100644
--- a/.dandi/s3sync.json
+++ b/.dandi/s3sync.json
@@ -1,5 +1,5 @@
 {
     "bucket": "dandiarchive",
     "prefix": "zarr/7e1b3b36-a94a-427f-9793-b14e344a04f2/",
-    "last_modified": "2024-04-02T17:07:01Z"
+    "last_modified": "2024-04-19T17:25:56Z"
 }

and no file acquisition/TwoPhotonSeries/data/10006.5.7 , and git on dandiset itself has commit in Oct

dandi@drogon:/mnt/backup/dandi/dandisets/000719$ git status
On branch draft
Your branch is up to date with 'github/draft'.

nothing to commit, working tree clean
dandi@drogon:/mnt/backup/dandi/dandisets/000719$ git show | head 
commit 4a67e06
Author: DANDI User <info@dandiarchive.org>
Date:   2024 Oct 02 22:12:48 -0400

    [backups2datalad] 1 file added

I wish we had audit readily available to get records on this zarr to check exactly what/when was happening on that zarr ...

[yarikoptic]

ok, no time for chasing this bug in dandi-archive somewhere ATM. For now

• moved updates of 000719 into "zarrs" cron job on which we do not do --verify
• 000874 -- just "cheated" and adjusted the title (interesting if audit has that I did that "evil") causing modified to get adjusted. then verify call completed fine

now rerunning for all

[jjnesbitt]

@yarikoptic What is the concise bug report that we could move forward on solving in the archive once we're able? Is it mainly that the metadata of dandiset draft versions are updated without the modified field being updated as well?

[yarikoptic]

for non-zarr assets -- I have not troubleshooted to the form beyond the fact that changes to assets (not dandiset level) metadata did not result in the change of the dandiset modified field. It might as well be the same issue as just filed

• Do update dandisets' "modified" timestamp whenever assets metadata "gets fixed" dandi-archive#2050

may be you reextracted some metadata for some assets (listed in above report) also modifying them inplace and thus not triggering modified time stamp for the dandiset to change?

for zarr -- filed

• Finalize "Pending" zarr if all upload URLs must have timed out by now dandi-archive#2051