Add command to check ocis backup consistency #9238

kobergj · 2024-05-22T14:35:07Z

Adds consistency check command

Fixes #9004
E2E testing needs additional ticket: #9286

update-docs · 2024-05-22T14:35:11Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

kobergj · 2024-05-23T12:13:10Z

Example output inconsistent case:

🚨 Inconsistent Nodes:
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/32/b1/37/f6/-f621-4a1d-9faf-244ab044c068
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/a0/28/b0/fa/-a5bb-453d-baea-1090df55faae
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/f2/86/2b/0b/-919d-4cd9-a35e-5e5a0b387587
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/2d/08/8d/24/-96b5-478e-9602-425d1f084618
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/13/f6/fc/25/-0d6d-407b-9cbc-d4e4b8d46030
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/28/57/9b/25/-1ceb-46b8-b140-7ff536ef7b3d
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/14/db/c9/b8/-6382-41f8-b2d7-5e0d7bbdbd98
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/58/35/d7/ba/-e8d0-40c1-ae84-6282058a9023
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/87/a6/12/3f/-f05a-4d20-913c-a0bda5784fae
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/c5/ed/4e/9d/-4bd1-4788-b65f-e238b48b5191
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/ca/ab/4e/5c/-6403-4909-bac1-edc702be27d9
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/d7/e9/ff/cf/-77d6-4d3e-a402-af3605aece6d
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/e1/54/2b/ea/-cfa3-4475-b848-25199d1a54a5
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/38/ae/a7/38/-375a-428a-a578-b52d10b10992
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/62/e0/eb/91/-91da-4466-81df-153a500e8c0b
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/8f/71/a6/96/-7ffd-411c-8eba-dd99715b3860

🚨 Inconsistent Links:
        👉️ [node missing]       path: spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/a8/e5/d9/81/-41e4-4468-b532-258d5fb457d3/avatar(10)(12).png
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/a8/e5/d9/81/-41e4-4468-b532-258d5fb457d3
        👉️ [node missing]       path: spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/b9/66/72/90/-9baa-44ce-a321-82c2055dd772/readme.md
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/b9/66/72/90/-9baa-44ce-a321-82c2055dd772
        👉️ [node missing]       path: spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/be/d8/96/24/-5f34-4354-88c9-5935e13b81cd/burning_elmo.gif
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/be/d8/96/24/-5f34-4354-88c9-5935e13b81cd
        👉️ [node missing]       path: spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/fd/7e/15/95/-8946-4b6b-b78d-1fc715119386/oCIS activities-space history and eventlog.drawio(2).svg
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/fd/7e/15/95/-8946-4b6b-b78d-1fc715119386

🚨 Inconsistent Blobs:
        👉️ [blob orphaned]      blob: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/blobs/8f/28/64/7f/-3e67-4b22-8574-a4fda80faa9c

🚨 Inconsistent BlobReferences:
        👉️ [blob missing]       blob: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/blobs/07/f9/4c/67/-b51d-4a9f-8a90-d896d26f46cc
                                referencing node:/home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/14/db/c9/b8/-6382-41f8-b2d7-5e0d7bbdbd98

Example output when clean:

💚 No inconsistency found. The backup seems to be valid.

mmattel · 2024-06-05T13:42:57Z

Some documentation is needed. Dont go the route without...
In owncloud.dev, create a subsection like Backup and add all backup related content underneath, starting with the existing Backup Considerations documentation. See the image how it currently looks:

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com> Signed-off-by: jkoberg <jkoberg@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

Signed-off-by: jkoberg <jkoberg@owncloud.com>

docs/ocis/backup.md

mmattel · 2024-06-06T14:52:51Z

docs/ocis/backup.md

+ocis backup consistency -p "<path-to-base-folder>"
+```
+
+`path-to-base-folder` needs to be replaced with the path to the storage providers base path. For example: "/var/lib/ocis/storage/users" or "$HOME/storage/users".


...the path to the storage providers base path. This sentence makes the path origin not clear, you should add the relevant envvar for reference so one can have a look how it was defined.

There are multiple envvars for it. Could be BASE_DATA_BASE or STORAGE_USERS_OCIS_ROOT or even STORAGE_SYSTEM_OCIS_ROOT. That is why put two examples instead.

The docs say

STORAGE_USERS_OCIS_ROOT

The directory where the filesystem storage will store blobs and metadata. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/storage/users.

Referencing an envvar is imho much clearer as we have very good docs about...

docs/ocis/backup.md

Co-authored-by: Martin <github@diemattels.at>

wkloucek · 2024-06-11T10:59:58Z

@kobergj I gave this tool a try today. I was executing it in a environment that is configured for the storage-users service only (storageusers pod in the oCIS Helm Chart). To be actually able to run this tool, I needed to configure following additional environment variables:

OCIS_JWT_SECRET
OCIS_MACHINE_AUTH_API_KEY
OCIS_SYSTEM_USER_ID
OCIS_SYSTEM_USER_API_KEY

Is there a reasoning why those secrets are needed for the backup check tool? From what I understand, the tool is concepted to be run against a file directory and blobstore without any running oCIS instance, so we probably don't need all those configuration options to be checked, right?

wkloucek · 2024-06-11T11:24:42Z

@kobergj how hard would it be to have a flag to skip blob related checks? We could use this for validating a NFS snapshot only (without connecting to the S3 at all)

EDIT: it also would kinda act as decomposed fs only consistency check.

kobergj · 2024-06-11T14:36:32Z

From what I understand, the tool is concepted to be run against a file directory and blobstore without any running oCIS instance, so we probably don't need all those configuration options to be checked, right?

Yes that is how it is concepted. I'll check what's wrong.

EDIT: Fixed. Needed to change a bool from true to false

@kobergj how hard would it be to have a flag to skip blob related checks? We could use this for validating a NFS snapshot only (without connecting to the S3 at all)

No problem. We can skip blob checking. I'll add a commandline parameter to do so 👍

EDIT: Implemented. Use -b none flag to avoid blob checking. (Neither orphaned nor missing blobs will be shown.)

@wkloucek implemented as suggested 👍

Signed-off-by: jkoberg <jkoberg@owncloud.com>

sonarcloud · 2024-06-11T15:26:13Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
18.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

2403905

Looks good for me

kobergj · 2024-06-12T13:16:52Z

Merging this. Will deliver additional features in subsequent PRs. Just ping me.

Add command to check ocis backup consistency

kobergj force-pushed the OcisConsistencyCheck branch from 4369d56 to cf37c86 Compare May 23, 2024 10:04

kobergj force-pushed the OcisConsistencyCheck branch 13 times, most recently from 8b149ec to ba0aad4 Compare May 29, 2024 09:29

kobergj mentioned this pull request May 29, 2024

Export Path method in blobstores cs3org/reva#4699

Merged

kobergj force-pushed the OcisConsistencyCheck branch 2 times, most recently from c7b0b74 to 84bf2b7 Compare May 31, 2024 08:39

kobergj mentioned this pull request May 31, 2024

Test backup consistency command in ci #9286

Closed

kobergj force-pushed the OcisConsistencyCheck branch 3 times, most recently from b918ae7 to 90868e2 Compare June 3, 2024 13:22

kobergj and others added 6 commits June 6, 2024 15:55

feat(ocis): Add backup consistency command

8a08c9f

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

feat(ocis): improve consistency command

ca12c2f

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com> Signed-off-by: jkoberg <jkoberg@owncloud.com>

feat(ci): test consistency command

ada83f9

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

feat(ocis): show full blob path when inconsistent

6703dab

Signed-off-by: jkoberg <jkoberg@owncloud.com>

feat(ci): test consistency command in ci II

5086f55

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

feat(ocis): list link location on inconsistency

2479a30

Signed-off-by: jkoberg <jkoberg@owncloud.com>

kobergj and others added 6 commits June 6, 2024 15:55

feat(ocis): improve output of backup command

e4a4652

Signed-off-by: jkoberg <jkoberg@owncloud.com> Co-authored-by: dragonchaser <crichter@owncloud.com>

fest(ocis): fix backup command

969daba

Signed-off-by: jkoberg <jkoberg@owncloud.com>

feat(ocis): concurrency for consistency command

e969217

Signed-off-by: jkoberg <jkoberg@owncloud.com>

feat(ocis): finally unit tests for backup consistency

d655c81

Signed-off-by: jkoberg <jkoberg@owncloud.com>

feat(ocis): use only one channel for backup command

1525f7e

Signed-off-by: jkoberg <jkoberg@owncloud.com>

feat(docs): extend backup docu to mention command

e046768

Signed-off-by: jkoberg <jkoberg@owncloud.com>

kobergj force-pushed the OcisConsistencyCheck branch from 00fe2ac to e046768 Compare June 6, 2024 13:55

mmattel reviewed Jun 6, 2024

View reviewed changes

docs/ocis/backup.md Outdated Show resolved Hide resolved

mmattel reviewed Jun 6, 2024

View reviewed changes

docs/ocis/backup.md Outdated Show resolved Hide resolved

mmattel reviewed Jun 6, 2024

View reviewed changes

docs/ocis/backup.md Outdated Show resolved Hide resolved

mmattel reviewed Jun 6, 2024

View reviewed changes

kobergj commented Jun 7, 2024

View reviewed changes

docs/ocis/backup.md Outdated Show resolved Hide resolved

feat(ocis): Better documentation for backup command

222ead4

Co-authored-by: Martin <github@diemattels.at>

kobergj marked this pull request as ready for review June 11, 2024 07:52

feat(ocis): allow skiping blobcheck in consistency command

9a84284

Signed-off-by: jkoberg <jkoberg@owncloud.com>

2403905 approved these changes Jun 12, 2024

View reviewed changes

kobergj merged commit adbc45e into owncloud:master Jun 12, 2024
4 checks passed

kobergj deleted the OcisConsistencyCheck branch June 12, 2024 13:17

ownclouders pushed a commit that referenced this pull request Jun 12, 2024

Merge pull request #9238 from kobergj/OcisConsistencyCheck

ac45ad4

Add command to check ocis backup consistency

micbar mentioned this pull request Jun 19, 2024

6.0.0 Rolling Release #9391

Closed

24 tasks

mmattel mentioned this pull request Jun 24, 2024

Backup Consistency Command owncloud/docs-ocis#877

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add command to check ocis backup consistency #9238

Add command to check ocis backup consistency #9238

kobergj commented May 22, 2024 •

edited

Loading

update-docs bot commented May 22, 2024

kobergj commented May 23, 2024 •

edited

Loading

mmattel commented Jun 5, 2024 •

edited

Loading

mmattel Jun 6, 2024

kobergj Jun 7, 2024

mmattel Jun 7, 2024

wkloucek commented Jun 11, 2024 •

edited

Loading

wkloucek commented Jun 11, 2024 •

edited

Loading

kobergj commented Jun 11, 2024 •

edited

Loading

sonarcloud bot commented Jun 11, 2024

2403905 left a comment

kobergj commented Jun 12, 2024

Add command to check ocis backup consistency #9238

Add command to check ocis backup consistency #9238

Conversation

kobergj commented May 22, 2024 • edited Loading

update-docs bot commented May 22, 2024

kobergj commented May 23, 2024 • edited Loading

mmattel commented Jun 5, 2024 • edited Loading

mmattel Jun 6, 2024

Choose a reason for hiding this comment

kobergj Jun 7, 2024

Choose a reason for hiding this comment

mmattel Jun 7, 2024

Choose a reason for hiding this comment

wkloucek commented Jun 11, 2024 • edited Loading

wkloucek commented Jun 11, 2024 • edited Loading

kobergj commented Jun 11, 2024 • edited Loading

sonarcloud bot commented Jun 11, 2024

Quality Gate passed

2403905 left a comment

Choose a reason for hiding this comment

kobergj commented Jun 12, 2024

kobergj commented May 22, 2024 •

edited

Loading

kobergj commented May 23, 2024 •

edited

Loading

mmattel commented Jun 5, 2024 •

edited

Loading

wkloucek commented Jun 11, 2024 •

edited

Loading

wkloucek commented Jun 11, 2024 •

edited

Loading

kobergj commented Jun 11, 2024 •

edited

Loading