Cannot decrypt binary data

[rogozind]

Hi,
I made the library work for standard plain English data. But my real use case is RSA decrypting a binary data chunk and it does not seem to produce expected result. I have validated that both OpenSSL and C# RSA provider produce identical bytes. But JSEncryptor behaves strange. It produce strange set of characters which sometimes match the expected results and sometimes do not. Please review details below.
Private Key and Encypted text below. I am banging my head against the wall on this one so I really appreciate any pointers...

Expected result:
$ base64 -D test.enc | openssl rsautl -decrypt -inkey test.priv -hexdump
0000 - 57 27 19 73 ba 3c f4 48-04 1c 8e 5f f8 1c e0 d1 W'.s.<.H..._....
0010 - d7 97 7e 73 cc 53 77 d1-88 3f 0d 79 42 13 65 87 ..~s.Sw..?.yB.e.

My results. See how small numbers match but large numbers cannot be mapped to the expected bytes. It feels like it is UTF-8 but not quite. And there are strange misses and unexpected character at the end:
C# (Bytes) JS (characters)
087 0x57 87
039 0x27 39
025 0x19 25
115 0x73 115
186 0xBA 44852
060 0x3C
244 0xF4
072 0x48 72
004 0x04 4
028 0x1C 28
142 0x8E 59384
095 0x5F
248 0xF8
028 0x1C 28
224 0xE0 ????? <--- this byte is unclear
209 0xD1 1111
215 0xD7
151 0x97 32691
126 0x7E
115 0x73
204 0xCC 787
083 0x53
119 0x77 119
209 0xD1 1096
136 0x88
063 0x3F 63
013 0x0D 13
121 0x79 121
066 0x42 66
019 0x13 19
101 0x65 101
135 0x87 28672 <-- This one is unexpected....

$ cat test.priv
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA7js4RCPP81/hVlmwlgz2BHvJwL4OqsB5f55Zbnewvxxxd/Wm
BE2tM9M4qmnV3Ub/B4YAdmG8CcJofyNApQvKLYatRJnnedyD/FYFj2e9O3iqs6UM
6MbS39Ce5rTuQEc5DXVUbsJKgR0J6PhtlF6QoU3GsluystuTX8NW2HTrodJa+2dO
4LLUTtE3nYahjFddqpxlbM0X5wp123Soviw5xotIrQFsMjn2xY5sY+d80qS9dwM5
OhmDWI+g2MB/ki1muBoaedxnj8Y3kehP1YWFOElBv5W/6E3jHqUKLk9E/M6udmKd
u5ASVwkVNAvBPW4xGt7I0CF0IXo7L6xpNQaNdwIDAQABAoIBACTyYkOPGk0wbY4q
0swyrmT7ncqar0OkHjPApzYOsgaxrpdgLaM1OMt00rD7QQPUYvETwzaY6cTJtdMh
8ICoV27AGFcvV0r6/C1CWhJnkF2M1UyPKHVhgIrdnAdDwD+PzDVeIP4ce9mSW1sa
TERadivOGCbJOemrTCvp66oeV/z8ii1Jww90CzIYO3M4Lag4QtSdI3SiCwTs0b1O
gZzk+RvoHIpTvYl+XbaLkzr64q9/ee26UArQ4BEfEjEKFbs+Pe0z/JNy33ROViH7
NohqbXaXtbFAPTikh1M8m1x/XsK2u+Bi8XeQYwUzENQs1vRv3iOoBVACGck1Aaw0
Zir4wyECgYEA+OWwIRinsREK95q+ce4gg0EMhXakLk/g9wvWorIENbMWEqwvrGfc
FFWNLuDZzSD5pYFyf4TngojeAdiPgrCaVjAaO7N/UK6CjZH8yw/Rzucsb6PFnp3a
yBs8JdlMD9MDpWS6tffhKtmHjAsjBr6Sxi/V8PEmKHvTETiUM+PjJiUCgYEA9Qec
xi6glKEgR5Ssjx3qAa6TMqF+dL/EeqB2DdH1sT08x+zdPONayATG3NSbGBwA5Q0i
XLBv4/J5ODw65F2yRFReU6gNbPBunzYwasnowke2p8U42G2RB0XN4GAGptret3xG
1qzETGZIDvjPdnVqnbT6ttM1e4UKh4eDv0GmbGsCgYEAnOYl5mclZsBomneI8Eb9
ZeA+pW+JxcwC85kZ34u6jaNsGa9DijoaiRL5IgEKK35LAWMdq3c7fZUL5jv/E5KC
aCyAjPv1GQY5NEdABT3es8AUyfeCXyABR72rZ+yYPy11EfDypj0xSxK/mK94kArY
eWi0Q+Owezq7bhKtkhxpF6UCgYBiMxfQkTVH+78tENeLT+jNncJeYzyD26sJni9R
4Jh/3NP0J+iOk2t5h/iO3iH+KpLtOTNRRGDDURTlxbWweiCMo+ddzZb71uvuoNAM
rDi9S/JXEIJ0soySBpa42qMv0b26eXMGbB0LpV4xqeOzTOFarQJhrQndI/eRU20o
hE4HWwKBgQCX5TEkoeBnO2DDNj5vPN0LCOzcG4c4QqpkhfmAEpRNBBNiEKwuWsHq
S1LLlsR1Z/ycs21KOeaG5mXS4WWZjVQILUtOsgAg48ulkPfzUQU0c3w8oON4acP/
0gn/pSDyeUb+I9I2+ht8ekbhD9tXBEJGsY31tALqDmS7VqKt9Folvg==
-----END RSA PRIVATE KEY-----

$ cat test.enc
T+KOs8MMBNDmEiwiSqBIJWMucotbAGFhW1/kBZoRBT9uM3yOQs10xuqCm2Jo
EG5YU70lLTEQmuLgHJkW4P6fkX2E/63aFlSpy8A4Nli1ou1VKI2BLmGYH6N9
S/DrP7qySg/o9cH4wiBAbIuNoQmpTxeOyC9/r3+XbFO24+guNK+S3ytxB8rT
ADtLDK6PhL/BzagWfxydt6kQAIHGiCnVnunmYUwLataqDNdyZI0vBebiwaZ9
oW206JEk/S2WznKw6BReeTqB5CNwDxWdrMLSx/j5yk7W4faM8DdPwXbfT1RI
jKdrOlwFyWUbmlLGbZgJjVT9RoUzlMCezaWupJVHmQ==

[zoloft]

@travist can you look into it?

[rogozind]

I actually figured it out today. JSEncyptor converts text it gets to encrypt into UTF-8 set of bytes and then it runs encryption. So on decryption it assumes it is UTF-8 and tries to convert it back into characters. This is done in the pad and unpad functions.

Basically it can only safely decrypt the text it is previously encrypted. Would be nice to have an option to remove UTF-8 conversion for interoperability with other encryptors and decryptors.

For now I am using patched code with UTF-8 conversion removed.

From: Antonio <notifications@github.commailto:notifications@github.com>
Reply-To: travist/jsencrypt <reply@reply.github.commailto:reply@reply.github.com>
Date: Monday, February 23, 2015 at 11:45 AM
To: travist/jsencrypt <jsencrypt@noreply.github.commailto:jsencrypt@noreply.github.com>
Cc: Dima Rogozin <dima.rogozin@everbridge.commailto:dima.rogozin@everbridge.com>
Subject: Re: [jsencrypt] Cannot decrypt binary data (#45)

@travisthttps://github.com/travist can you look into it?

Reply to this email directly or view it on GitHubhttps://github.com//issues/45#issuecomment-75580169.

[apps4av]

By the way, this is what I did and it worked. Dont know if this is the right way. But still.:

// Undo PKCS#1 (type 2, random) padding and, if valid, return the plaintext
function pkcs1unpad2(d,n) {
var b = d.toByteArray();
var i = 0;
while(i < b.length && b[i] == 0) ++i;
if(b.length-i != n-1 || b[i] != 2)
return null;
++i;
while(b[i] != 0)
if(++i >= b.length) return null;
var ret = "";
while(++i < b.length) {
var c = b[i] & 255;
//ZKZK if(c < 128) { // utf-8 decode
ret += String.fromCharCode(c);
//ZKZK }
/ZKZK
else if((c > 191) && (c < 224)) {
ret += String.fromCharCode(((c & 31) << 6) | (b[i+1] & 63));
++i;
}
else {
ret += String.fromCharCode(((c & 15) << 12) | ((b[i+1] & 63) << 6) | (b[i+2] & 63));
i += 2;
}
ZKZK/
}
return ret;
}

[Rondom]

It seems a bit strange to have an API that only supports strings, seems a bit strange for an encryption library. Ideally, the API should take Uint8Array or something to handle binary data.

Having the convenience of passing a string and having it decoded to UTF-8 is nice, if only it was working correctly. Fixing this might raise interesting questions about compatibility, but that is probably better than continuing to ship a broken implementation.