-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update documentation to create an empty wal file #411
base: main
Are you sure you want to change the base?
Conversation
_docs-2/troubleshooting/advanced.md
Outdated
backup policy is used which could lead to a long wait before Accumulo will | ||
try to re-load the WAL file. | ||
```sh | ||
UUID=$(uuidgen); echo -n -e '--- Log File Header (v4) ---U+1F47B$'"$UUID"'\x00\x00\x00\x00' >"$UUID".wal |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, I was tracking down where the U+1F47B
comes from. I think the original intent is to have the actual ghost emoji character stored there, but instead, it's a human-readable representation of the unicode code point for the ghost emoji that's stored there. It doesn't really matter what it's value is, though. It's purpose is to be a fixed value that represents the decryption parameters for a "not encrypted" file.
Specifically, this value is an indicator that it's "not encrypted" using our built-in non-encrypting CryptoService that we use in our built-in reference implementations for a CryptoServiceFactory. A user could provide their own factory that supports a "not encrypted" option using a different token to identify files it supports. But, this string is to denote ours.
I checked the serialization of our WALs in the code, and it looks like this is wrong as written. There should be an integer that is written before the "U+1F47B" part. After that, I'm not sure what the WAL is supposed to contain, because I only got so far tracing the code to see how it writes the WAL header.
In Java, this is what I saw as a header, when tracing the steps to write out the size of the crypto params, then writing out the string for the "no crypto" params:
byte[] header = "--- Log File Header (v4) ---\000\000\000\007U+1F47B".getBytes(UTF_8)
// { 45, 45, 45, 32, 76, 111, 103, 32, 70, 105, 108, 101, 32, 72, 101, 97, 100, 101, 114, 32, 40, 118, 52, 41, 32, 45, 45, 45, 0, 0, 0, 7, 85, 43, 49, 70, 52, 55, 66 }
That's equivalent in bash to:
echo -n -e '--- Log File Header (v4) ---\x00\x00\x00\x07U+1F47B' >empty.wal
I don't know where the UUID comes in, or that dollar sign in the above example, or the zeroes from the v2 format. There may be more needed after this header to result in a valid WAL file... I'm not sure, but I do think what's here now in this PR is wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed, I'm creating a utility similar to the one to create an empty rfile. When complete these instructions will be updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The documentation has been updated to remove the previous changes and add the create-empty WAL / RFILE options. Because this had drifted so far from main I ended up doing a merge and force push. Sorry.
- Update empty rfile to keyword executable command format - Update to include creating an empty wal file.
6da76d5
to
23a1898
Compare
Updated the documentation for creating an empty wal file to use version 4 header, a uuid and other information.