Turning rawSecretKey() back into a Keypair?

[MikeFair]

How do you use the 64 byte array retrieved via kp.rawSecreteKey() to populate a new Keypair?

Thanks,
Mike

[nullstyle]

You'd need to look at the ed25519 specs for that, I don't think this is something we want to expose in our own API. IMO, stellar shouldn't encourage people to work directly with the underlying secret key... stellar is designed for upgradeable cryptography and if you want to ease your own software maintenance in the event we have to replace ed25519 (or choose to extend support for a new signature scheme) you should pass mostly try to interact with:

• high-level key objects like Keypair that eschew the underlying cryptographic protocol for in memory use
• strkey encoded seeds for a human-readable expression of the key or for any place you persist the secret (ideally never plain text)

As another note: rawSecretKey was really only added for debugging and testing support.

[MikeFair]

I'm devising a way to encrypt it and store a signer's secret in the manged data of the account.
When using the 64-byte raw secret, there's no information about the plainText to help an attacker. If first encoded as printable characters, then encrypted, there's information about the plainText there.

Specifically for encrypted storage purposes having access to a raw binary value the software can encrypt, store, and recover is better than storing encrypted human-readable characters...

I'd even be fine with a Keypair.encryptedSecret(); Keypair.fromEncryptedSecret(cipherText); pair of calls.
Then Stellar could control for the upgradeable encryption options.

[nullstyle]

When using the 64-byte raw secret, there's no information about the plainText to help an attacker. If first encoded as printable characters, then encrypted, there's information about the plainText there.

The plain text version of a secret (specifically, a strkey encoded ed25519 seed) is a fixed version byte, 32 bytes of random data, and then a checksum of the preceding 33 bytes. What specific information are you concerned about?

Specifically for encrypted storage purposes having access to a raw binary value the software can encrypt, store, and recover is better than storing encrypted human-readable characters...

Could you please explain? What attacks are you concerned with or what threat model are you working against?

[MikeFair]

What specific information are you concerned about?

The conversion into the ascii character set eliminates the randomness of the original value.

I came up with treating the string as a Base64 encoded value; getting the binary value of it; and then encrypting/decrypting that binary value instead.

I'm also double encrypting the value (there's two 64 byte keys used) so even if the plainText doesn't have a random distribution, the encrypted cipherText still will.

What attacks are you concerned

Any that can be automated ;) (see the threat model)

One version of an attack on plain ascii/unicode encrypted text is statistical, requiring a large number of samples to pull off. The plainText values are all in a relatively closely packed range which means the difference in the bits between neighbors is significantly less than random distribution. You use this information to construct probable keys.

I don't have a link to the original articles I used, however here's an example from the crypto exchange forums describing a way to go after XOR style encryption schemes:
http://crypto.stackexchange.com/questions/8115/repeating-key-xor-and-hamming-distance

what threat model are you working against?

I'm placing encrypted secret signing key values for signers on the account, and KYC values; Full Name, Address, SSN#, bank account#s, possibly DL#, email addresses, credit card #s, cell phone #s, and other values that are likely to be known or discoverable by would be attackers.

All that cipherText is then placed in a public repository, along with exact explanations of the process used to encrypt them, where anyone with a mind to try can snag the info off as many accounts as they like and execute any kind of offline attacks against them they wish.

I assume the attackers know every value on the account except the signing secret, which they have the cipherText for (and the technique used to encrypt it).

Increasing the statistical appearance of random keys encrypting random noise helps a lot.

This was actually one of those blessings in disguise moments as treating the ascii text as base64 encoded strings to create a binary value before encrypting them likely wouldn't have come to me if I wasn't staring at the stringified key wondering what I could do. :)
That trick helps with the other values as well.

Ideas and recommendations are welcomed.

Thanks!

[nullstyle]

The conversion into the ascii character set eliminates the randomness of the original value.

I think you're mistaken. The encoding does nothing to change the amount of entropy that is in the secret key. It's still 32 bytes of random data underlying any strkey encoded seed. Yes, the alphabet of any given byte is constrained, but the resulting string ends up commensurately longer.

I'm also double encrypting the value (there's two 64 byte keys used) so even if the plainText doesn't have a random distribution, the encrypted cipherText still will.

From what I understand, you're better off just using a longer key and a solid crypto system (when in doubt, use nacl). @stanford-scs can you confirm?

I assume the attackers know every value on the account except the signing secret, which they have the cipherText for (and the technique used to encrypt it).

It seems you can invalidate this assumption by simply including a nonce in your plain text.

Increasing the statistical appearance of random keys encrypting random noise helps a lot.

Could you link to a whitepaper with specific discussion? I'm still not sure what specific attack techniques you're trying to protect against.

This was actually one of those blessings in disguise moments as treating the ascii text as base64 encoded strings to create a binary value before encrypting them likely wouldn't have come to me if I wasn't staring at the stringified key wondering what I could do. :)
That trick helps with the other values as well.

I don't know man, this seems awfully misguided to me. In my opinion, you should probably be using an existing encryption scheme. From what I understand, most modern crypto systems are not known to be vulnerable to a known plain text attack. Without specific knowledge of the statistical methods you are concerned about, I can't really comment beyond that.

If you're concerned about offline dictionary attacks and need to expose your cipher texts on the internet, my understanding is that you'll want to use a key-stretching scheme to up the CPU/Memory cost for any attempt. We use scrypt in our systems to provide that mechanism.

[MikeFair]

Yes, the alphabet of any given byte is constrained, but the resulting string ends up commensurately longer.

The plainText being 0-31 instead of 0-255 is the information. The similarity requirement between all the plainText bytes is where the "attack guidance" came from.

At the time encryption schemes are employed; it's just hard to foresee how these constraints might be usable, then someone figures it out.

I'm also double encrypting the value (there's two 64 byte keys used) so even if the plainText doesn't have a random distribution, the encrypted cipherText still will.

From what I understand, you're better off just using a longer key and a solid crypto system (when in doubt, use nacl). @stanford-scs can you confirm?

1. Managed Data on an account is limited to 64 bytes atm; the keys are 64 bytes to match the message sizes. The limit is 512 bits until a system to partition up the resulting message over multiple data fields is in place (or the field size extended). The choices then, are block and stream ciphers. Which is fine, but it also eliminates otherwise excellent choices.

2. The principle though is an arbitrary input string of seemingly random noise that is then encrypted with an equally random key is always going to be harder to "crack" using automated analysis tools; over a system full of structured data and implementation restrictions that constrain the possible input values and formats of both the keys and datum. Folks are simply resourceful at finding ways to exploit seemingly insignificant artifacts. I'm looking to eliminate the attack profile as much as is reasonably possible, ideally making it impossible to tell even if/when they guess the right first key.

I assume the attackers know every value on the account except the signing secret, which they have the cipherText for (and the technique used to encrypt it).

It seems you can invalidate this assumption by simply including a nonce in your plain text.

The truth is the double encryption also handles this and the first encryption layer effectively turns the entire message into one big nonce. I completely spaced on the fact that this is one of the reasons why I was doing that in the first place. Forget I said it.

This came up when I was first considering what information an attacker might have access to, and the first thing that hit me was the entire history of cipherText "samples" would always be there for analysis.

Increasing the statistical appearance of random keys encrypting random noise helps a lot.

Could you link to a whitepaper with specific discussion? I'm still not sure what specific attack techniques you're trying to protect against.

I can't provide a link. The closest thing to a link would be the Wikipedia entry on the One Time Pad and why/how it gets its "perfect secrecy" property. An encrypted random string of numbers, has nothing "useful" to hang the crack attack on as all valid answers have an equal potential.

The most successful exploits come from information about the relatedness of values within the cipherText, constraining the input values used to create them in the first place, or exploiting some "angle" on the actual implementations of the algorithms.

Being able to quickly test when you've successfully decrypted a message; like recognizable words; or where knowing what a portion of the answer should be also tells you if the rest of the message was decrypted properly; gives you an automation "hook" to pursue.

I'm envisioning a world in 3-5 years where "the cloud" is potentially full of distributed GPU type services happy to take your distributed problem and crank on it, possibly in exchange for some cryptocoin (movie/image rendering, biomedical simulations, complex engineering optimizations, financial and environmental modelling, and a whole host of other "use cases" make such services viable). Leaving out the possibility of any quantum based things.

My theoretical "attacker" has effectively infinite parallel computing resources, incentive to put effort into cracking the codes, full access to every version of cipherText ever applied to every account, a direct feed to be informed in real time when a cipherText is about to change, to what, and by whom, an internet full of languages and scripts to aid in any analysis, and is a darn smart and creative group of people. :)

I really have no idea what they can actually do with all that; but if there was an exploit available to be found; my expectation is they'd be able to find it.

The fewer pieces of constraining information they have about both the keys and values, the better.

[nullstyle]

The plainText being 0-31 instead of 0-255 is the information. The similarity requirement between all the plainText bytes is where the "attack guidance" came from.

That's wrong. The entropy of the information contained, regardless of the encoding, is the same. The 56-byte strkey encoded seed has the same 32 bytes of entropy as the decoded, raw, 32 byte seed. The only thing that would influence the bits of entropy in a seed would be a poorly chosen PRNG that can be predicted.

The transformation from raw 32-byte seed to 56-byte encoded seed is deterministic and reversible. They are equivalent from an information perspective. The encoded form neither strips nor adds any information to the raw value.

The choices then, are block and stream ciphers. Which is fine, but it also eliminates otherwise excellent choices.

I'd prefer to talk specifics rather than generalities. I am explicitly arguing that you should be using an existing block or stream cipher, for example: http://nacl.cr.yp.to/stream.html.

It's not clear to me why you would be rolling your own scheme. Perhaps you can talk about what additional benefits you will be getting over and above nacl's secretbox, for example.

The principle though is an arbitrary input string of seemingly random noise that is then encrypted with an equally random key is always going to be harder to "crack" using automated analysis tools;

A relative comparison isn't really useful without establishing a baseline. A well constructed AES-256 scheme seems to be pretty strong against brute force attacks, regardless of the plain text: https://www.reddit.com/r/theydidthemath/comments/1x50xl/time_and_energy_required_to_bruteforce_a_aes256/

The truth is the double encryption also handles this and the first encryption layer effectively turns the entire message into one big nonce.

Isn't it just way simpler to just use a nonce and a larger key? It'd be a much more familiar construction, at least in my experience. Why is a scheme that uses two rounds of encryption with 64-byte keys any different than a scheme that uses one round of encryption with a 128-byte key? It's 128 bytes of entropy in the key material in either case. Instead of 2 times the cost of a bruteforce attack, it's closer to 2^n times the cost (where n is the number of additional bits in the key)

I'm no cryptographer, so this may be nonsensical, but couldn't you just make some extended round version of Salsa20, that uses say 40 rounds, to get the same effect? Why is your method of doing two back to back encryptions any different in outcome than the the rounds used.

[MikeFair]

The similarity requirement between all the plainText bytes is where the "attack guidance" came from

The encoded form neither strips nor adds any information to the raw value.

The key ingredient for this attack class is exploiting the high data redundancy in the plainText:

Since English has 1.3 bits of real information per byte (see Section 11.1), there is plenty of redundancy for determining a unique decryption.

• http://stackoverflow.com/questions/22255080/breaking-simple-xor-encryption

Starting with 16,000 random bits and then encoding them to ascii '0' (0x30) and '1' (0x31) (128,000 bits), the total entropy of the message remains the same, but the information stored in each plainText byte goes from 8 bits per byte with no redundancy -> 1 bit per byte almost entirely redundant. And consequently goes from being immune to this attack class to trivially vulnerable.

This particular attack is representative of a whole class of successful attacks.
They exploit the plainText and key data representations; not cracking through the information entropy.

---

If the folks using this XOR style encryption had first transformed their English plainText into something that made greater utilization of the available data space, say they gzip'd it; then exploiting English's restricted 1.3 bits of information per byte isn't accessible.

An alternative would be to apply an "Initialization Vector" to "noisify" the data before encrypting it. By first spreading the bits around with a bit of random data, it increases the information bits per byte.

---

Setting the encryption algorithm is going to be in the hands of those who control the Account. I rest easier by knowing the the plainText is being adjusting before applying the encryption because even if the algorithms chosen end up having these anomalies; these messages won't be vulnerable. Given how trivial it is to transcode the buffer into something guaranteed to make use of all 8-bits, I find it hard to justify not doing it; regardless of encryption algorithm used.

These cipherText messages will remain in the permanent archive. If an attack is discovered or becomes viable 10 years after we first encrypt the data, the cipherText will still be there, and some of that data is likely to still be useful.

---

I am explicitly arguing that you should be using an existing block or stream cipher
Perhaps you can talk about what additional benefits you will be getting over and above nacl's secretbox, for example.

My focus is on the mechanics of making shared encrypted data fields work with Stellar's Set Data operation. My JavaScript foo was making it difficult for me to provide the Crypto libraries a 64-byte buffer and get back an ecrypted 64-byte buffer. I wasn't picking an algorithm here, I'm designing a framework for instructing the machines which algorithms to use for a given Account.

I hadn't seen NaCLs secretbox before today; by default it looks like it wants to increase the size of the message so I'll have to look into the implications of not doing that; it requires a nonce that will have to be communicated; at least the TweetNaCL.js library Google showed me works directly in Uint8 arrays so that's a plus.

This issue was really only about keeping the data redundancy of the secret key at 0. If you're interested in the bigger picture of the things to be solved here; I posted on Stellar Protocol about related topics (it's more "conventions" than "protocol"):
stellar/stellar-protocol#35
stellar/stellar-protocol#26

I'm not inventing a better encryption algorithm. I'm "noisifying" the user's plainText. I'm being one of those people that gzip'd their english plainText before encypting it because it eliminates, or at least significantly reduces, exposure to the above attack class.

The SHA-512 gave me trivial access to a mechanism for doing that. Doing it a second time gave me something akin to an ECB mode AES-512 encryption; at which point I could stop fighting with making the encryption libraries encrypt a 64-byte buffer without making it bigger and focus on more of the "unsolved" parts knowing that this will get revisited when the hooks are there to define the encryption method via the Account's metadata.

---

Isn't it just way simpler to just use a nonce and a larger key?

A data value on the account currently stores 64-bytes. Whatever cipherText gets produced has to pack into the space available.

The user currently has access to 64-bytes of unencrypted data; the more of those bytes "encryption" takes up, the fewer that will be available to applications. At the moment I'm at 63-bytes of application data because I used one byte to store the original message length. This might change.

Key length is somewhat meaningless without knowing how the algorithm uses the key. In some AES modes keylength > 64-bytes will make no difference at all on a 64-byte message; in a stream cipher it's hard for me to say exactly what the difference between a 128, 256, 512, or 1024 bit key really even is, especially once combined with an initialization vector...

...

If you take on using a nonce, depending on how its used, it must be communicated to the reader. It's the moral equivalent of the "Initialization Vector" or "salt". Unlike the secret, this one changes on every write so it needs to be communicated much more frequently. I'm looking for a "trivially good" answer to communicate this to the reader but haven't seen it yet. Ideally, it should just be there when a wallet executes "LoadAccount" without having to fetch more data; and especially without having to retrieve more data on a per encrypted field level.

Another thing; just to be clear, I'm not talking about "keys" as the same thing as "secrets". I'm talking about the "key" as the block of bytes that is being mixed with the plainText to produce the cipherText. You use a "secret" to make a "key"; and "secrets" can be as large or small as you like.
[Not sure if that makes a difference at all.]

Why is a scheme that uses two rounds of encryption with 64-byte keys any different than a scheme that uses one round of encryption with a 128-byte key?

I think this quote sums up "the goal" quite nicely:

A perfect cipher means that an attacker has the same chance to guess the plaintext if he has the ciphertext or he has no ciphertext at all, i.e. the ciphertext gives no information to someone who doesn't have the key (even with infinite computation resources).

I'll show how a basic random Shift Cipher gets more secure by doing two rounds.

We start with our plainText: "S9876543210"
We use a "secret" to make the "key" that will shift the values: [0, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
We subtract the key from the plainText to get our cipherText: ['S', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0']

We stop at round one and see what the attacker can determine:

• They knew the plainText started with an 'S'; the first key position is 0. (plainText knowledge leaking key)
• They know the rest of the plainText is 10 ascii digits and the algorithm used unsigned 8-bit subtraction, therefore the remaining key values are each between 0x00 and 0x09 (inclusive).
• By building up enough samples of this kind of information they keep adding bounds.

e.g. if another message, encrypting a different plainText with this same key, gave you: ['S', ')', ... ]
The attacker now knows the second key position is a 9 because that is the only value that can produce both a '0' and ')' cipherText that is also lower than 0x0A. The plainText values are '9' in the first message and '0' in the second message (because only '0' - 9 can produce ')' as cipherText).

Now let's add a second round using another random number: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
We subtract, and get a round two cipherText: ['S', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0']

All the attacker can tell from this is that the original plainText is within 510 places % 256 of those values as it results from the subtraction of two random 8-bit values mod 256. Or said differently, each position has 256 possible starting points (the full 8-bits) that could have produced the given cipherText. That's not true with the Round 1 only cipherText.

If they get the second message ['S', ')', ... ] all they can tell is that both '0' and ')' are accessible within 510 places of the original plainText starting point (which is still all possible values).

Every cipherText output has an equal opportunity to have started from any plainText message.

There's lots of ways to describe what is happening.

I think of it as the first round is increasing the information per byte to 8; in other words, adjusting the plainText to contain all the values between 0x00 and 0xFF and the more evenly distributed (random) the values are the better. The encryption algorithm itself may have been the thing used; but this first round isn't really an encryption step (in my mind's eye anyway), it's an encoding scheme to transform any given plainText to look like random noise. Any mechanism that transforms the plainText this way qualifies for this "round 1". It doesn't need to be secret (though that might help), but it does need to be reversible. Using the encryption algorithm itself (with a different key) is usually a convenient choice.

The second round is then actually encrypting what is now arbitrary binary plainText. Any structural information the original message may have contained got eliminated, so there's no anomalous "footprints" or "relationships" a would-be attacker might be able to exploit.

Doing more than two rounds keeps reducing back to two rounds from the attacker's POV.
The first round maps your structured message -> arbitrary binary; regardless of the number of rounds, once you're done, there exists a mapping to go directly from the plainText -> that arbitrary binary.
Then you encrypted it.

Thanks!