Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation to create an empty wal file #411

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

EdColeman
Copy link
Contributor

Updated the documentation for creating an empty wal file to use version 4 header, a uuid and other information.

_docs-2/troubleshooting/advanced.md Outdated Show resolved Hide resolved
_docs-2/troubleshooting/advanced.md Outdated Show resolved Hide resolved
_docs-2/troubleshooting/advanced.md Outdated Show resolved Hide resolved
_docs-2/troubleshooting/advanced.md Show resolved Hide resolved
_docs-2/troubleshooting/advanced.md Show resolved Hide resolved
backup policy is used which could lead to a long wait before Accumulo will
try to re-load the WAL file.
```sh
UUID=$(uuidgen); echo -n -e '--- Log File Header (v4) ---U+1F47B$'"$UUID"'\x00\x00\x00\x00' >"$UUID".wal
Copy link
Member

@ctubbsii ctubbsii Dec 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I was tracking down where the U+1F47B comes from. I think the original intent is to have the actual ghost emoji character stored there, but instead, it's a human-readable representation of the unicode code point for the ghost emoji that's stored there. It doesn't really matter what it's value is, though. It's purpose is to be a fixed value that represents the decryption parameters for a "not encrypted" file.

Specifically, this value is an indicator that it's "not encrypted" using our built-in non-encrypting CryptoService that we use in our built-in reference implementations for a CryptoServiceFactory. A user could provide their own factory that supports a "not encrypted" option using a different token to identify files it supports. But, this string is to denote ours.

I checked the serialization of our WALs in the code, and it looks like this is wrong as written. There should be an integer that is written before the "U+1F47B" part. After that, I'm not sure what the WAL is supposed to contain, because I only got so far tracing the code to see how it writes the WAL header.

In Java, this is what I saw as a header, when tracing the steps to write out the size of the crypto params, then writing out the string for the "no crypto" params:

byte[] header = "--- Log File Header (v4) ---\000\000\000\007U+1F47B".getBytes(UTF_8)
// { 45, 45, 45, 32, 76, 111, 103, 32, 70, 105, 108, 101, 32, 72, 101, 97, 100, 101, 114, 32, 40, 118, 52, 41, 32, 45, 45, 45, 0, 0, 0, 7, 85, 43, 49, 70, 52, 55, 66 }

That's equivalent in bash to:

echo -n -e '--- Log File Header (v4) ---\x00\x00\x00\x07U+1F47B' >empty.wal

I don't know where the UUID comes in, or that dollar sign in the above example, or the zeroes from the v2 format. There may be more needed after this header to result in a valid WAL file... I'm not sure, but I do think what's here now in this PR is wrong.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I'm creating a utility similar to the one to create an empty rfile. When complete these instructions will be updated.

Copy link
Contributor Author

@EdColeman EdColeman May 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation has been updated to remove the previous changes and add the create-empty WAL / RFILE options. Because this had drifted so far from main I ended up doing a merge and force push. Sorry.

Ed Coleman added 2 commits May 28, 2024 19:34
 - Update empty rfile to keyword executable command format
 - Update to include creating an empty wal file.
@EdColeman EdColeman force-pushed the update_empty_wal_cmds branch from 6da76d5 to 23a1898 Compare May 28, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants