[ocis] Add S3NG consistency check

[tbsbdr]

Description

Assumptions

• S3ng
• snapshots

User Stories

• As an Admin I want an option to check if my backup is consistent so that I'm reassured that its complete and restorable.

• As an Admin I want an option to check if my data are consistent so that I'm reassured that there are not erroneous data.

Value

Acceptance Criteria

• add ocis command to check for consistency
  • the command checks the consistency of the decomposed fs, including the blob store
  • consistency check should not rely on a running ocis
  • symlinks/references can be followed / are not broken
  • referenced blob exists
  • no blobs without reference exists
  • check if file metadata are complete (xattr, msgpack...?)
  • if there are inconsistencies, return a list of the inconsistencies
• Note: search does not need to be checked; NATS does not need to be checked within the check

Definition of ready

• Everybody needs to understand the value written in the user story
• Acceptance criteria have to be defined
• All dependencies of the user story need to be identified
• Feature should be seen from an end user perspective
• Story has to be estimated
• Story points need to be less than 20

Definition of done

• Functional requirements
  • Functionality described in the user story works
  • Acceptance criteria are fulfilled
• Quality
  • Code review happened
  • CI is green (that includes new and existing automated tests)
  • Critical code received unit tests by the developer
• Non-functional requirements
  • No sonar cloud issues
• Configuration changes
  • The next branch of the ocis charts is compatible

[kobergj]

@wkloucek @tbsbdr

This is the proposed output:

Example output inconsistent case:

🚨 Inconsistent Nodes:
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/23/ebf113-76d4-43c0-8594-df974b02cd74/nodes/91/13/a7/18/-8285-4b32-9042-f930f1a58ac2
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/23/ebf113-76d4-43c0-8594-df974b02cd74/nodes/99/bd/9c/46/-0c9d-4e94-939b-0f0ab1d39238
🚨 Inconsistent Links:
        👉️ [node missing]       path: /home/jkoberg/.ocis/storage/users/spaces/6b/15688a-f0d8-47a2-b345-470181c4f016/nodes/d7/78/fc/16/-d3c0-490c-8866-9ab05e153a94
🚨 Inconsistent Blobs:
        👉️ [blob orphaned]      blob: d4174a97-3d9b-4529-aa25-cf5fd2029b1b
        👉️ [blob orphaned]      blob: 5f463b1e-5820-4240-bdea-2f3174f573a1
🚨 Inconsistent BlobReferences:
        👉️ [blob missing]       blob: 87b2a98f-9b27-4c9e-9bcd-7bcc55fed0aa

Example output when clean:

💚 No inconsistency found. The backup seems to be valid.

Any remarks?

[wkloucek]

It would be nice to have the full blob path in the S3 bucket (it includes the space id and the blob id itself is shared). This could help looking at the actual blob in the S3 bucket.

[wkloucek]

@RaniaMidaoui FYI

[kobergj]

Updated Output after suggestions:

🚨 Inconsistent Nodes:
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/32/b1/37/f6/-f621-4a1d-9faf-244ab044c068
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/a0/28/b0/fa/-a5bb-453d-baea-1090df55faae
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/f2/86/2b/0b/-919d-4cd9-a35e-5e5a0b387587
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/2d/08/8d/24/-96b5-478e-9602-425d1f084618
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/13/f6/fc/25/-0d6d-407b-9cbc-d4e4b8d46030
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/28/57/9b/25/-1ceb-46b8-b140-7ff536ef7b3d
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/14/db/c9/b8/-6382-41f8-b2d7-5e0d7bbdbd98
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/58/35/d7/ba/-e8d0-40c1-ae84-6282058a9023
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/87/a6/12/3f/-f05a-4d20-913c-a0bda5784fae
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/c5/ed/4e/9d/-4bd1-4788-b65f-e238b48b5191
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/ca/ab/4e/5c/-6403-4909-bac1-edc702be27d9
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/d7/e9/ff/cf/-77d6-4d3e-a402-af3605aece6d
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/e1/54/2b/ea/-cfa3-4475-b848-25199d1a54a5
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/38/ae/a7/38/-375a-428a-a578-b52d10b10992
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/62/e0/eb/91/-91da-4466-81df-153a500e8c0b
        👉️ [symlink missing]    path: /home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/8f/71/a6/96/-7ffd-411c-8eba-dd99715b3860

🚨 Inconsistent Links:
        👉️ [node missing]       path: spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/a8/e5/d9/81/-41e4-4468-b532-258d5fb457d3/avatar(10)(12).png
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/a8/e5/d9/81/-41e4-4468-b532-258d5fb457d3
        👉️ [node missing]       path: spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/b9/66/72/90/-9baa-44ce-a321-82c2055dd772/readme.md
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/b9/66/72/90/-9baa-44ce-a321-82c2055dd772
        👉️ [node missing]       path: spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/be/d8/96/24/-5f34-4354-88c9-5935e13b81cd/burning_elmo.gif
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/be/d8/96/24/-5f34-4354-88c9-5935e13b81cd
        👉️ [node missing]       path: spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/fd/7e/15/95/-8946-4b6b-b78d-1fc715119386/oCIS activities-space history and eventlog.drawio(2).svg
                                missing node:/home/jkoberg/.ocis/storage/users/spaces/be/d89624-5f34-4354-88c9-5935e13b81cd/nodes/fd/7e/15/95/-8946-4b6b-b78d-1fc715119386

🚨 Inconsistent Blobs:
        👉️ [blob orphaned]      blob: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/blobs/8f/28/64/7f/-3e67-4b22-8574-a4fda80faa9c

🚨 Inconsistent BlobReferences:
        👉️ [blob missing]       blob: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/blobs/07/f9/4c/67/-b51d-4a9f-8a90-d896d26f46cc
                                referencing node:/home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/nodes/14/db/c9/b8/-6382-41f8-b2d7-5e0d7bbdbd98

@RaniaMidaoui @wkloucek any other suggestions how we can make this more usable?

[kobergj]

Also, we would like to test this with a proper backup. We are unsure if we need to optimize memory consumption or execution speed. It's hard to extrapolate these values from our local dev machines.

Can we get our hands on a backup of some sort (maybe from a testing instance)?
If not, can you run the command for us and give us feedback?

[wkloucek]

blob: /home/jkoberg/.ocis/storage/users/spaces/a8/e5d981-41e4-4468-b532-258d5fb457d3/blobs/8f/28/64/7f/-3e67-4b22-8574-a4fda80faa9c

This looks weird for a blob that is stored in S3. I'd expect something like a8e5d981-41e4-4468-b532-258d5fb457d3/blobs/8f/28/64/7f/-3e67-4b22-8574-a4fda80faa9c. But that might be just a detail that's out of scope for that mock

[wkloucek]

If not, can you run the command for us and give us feedback?

I guess this is the way to include infra variances. We can eg. do it on an instance that we used for loadtesting, that should have some data.

[kobergj]

This looks weird for a blob that is stored in S3. I'd expect something like a8e5d981-41e4-4468-b532-258d5fb457d3/blobs/8f/28/64/7f/-3e67-4b22-8574-a4fda80faa9c. But that might be just a detail that's out of scope for that mock

Correct. This is coming from my local ocis blobstore. In s3ng it would look like you expect 👍

🚨 Inconsistent BlobReferences:
        👉️ [blob missing]       blob: e069c534-2903-4845-a389-8b9d36847dfa/c7/75/78/7d/-5cf6-4ff6-859e-9f72695fc6f3
                                referencing node:storage/users/spaces/e0/69c534-2903-4845-a389-8b9d36847dfa/nodes/c5/1e/e0/68/-e1e0-4a08-9c51-6cdf3d1958ee
🚨 Inconsistent Blobs:
        👉️ [blob orphaned]      blob: c94c787d-59df-4832-b357-27f28e498637/4c/8d/d7/e7/-d999-4792-9b81-b6d71fe2a003

I guess this is the way to include infra variances. We can eg. do it on an instance that we used for loadtesting, that should have some data.

Yes great. 👍 It would be much clearer to us which issues to address if we can test it on that.

[kobergj]

@wkloucek should we test this before merging? This could speed up development process.

Or we merge and delay the test if you don't find time at the moment.

[wkloucek]

@RaniaMidaoui I build a DEV build based on 2384d47

We can use it for testing. The image is docker.io/wkloucek/ocis:s3ng-consistency-2384d472ed5650b3c8bb5f6a65afef290427ae68

EDIT: updated build of the latest commit docker.io/wkloucek/ocis:s3ng-consistency-222ead44e6401f87e018f01624274a15818eff8f, which is built from 222ead4

EDIT2: update build of the latest commit docker.io/wkloucek/ocis:s3ng-consistency-9a84284372f004fd8b673c4e79def72cce81a02a, which is built from 9a84284