EVP_cast5_cfb master/1.1.1.f encryption output discrepancy

[guidovranken]

This is a reproducer for OSS-Fuzz bug https://oss-fuzz.com/testcase-detail/5088449485078528

#include <vector>
#include <openssl/evp.h>

#define CF_CHECK_EQ(expr, res) if ( (expr) != (res) ) { goto end; }
#define CF_CHECK_NE(expr, res) if ( (expr) == (res) ) { goto end; }

std::vector<unsigned char> openssl(const std::vector<unsigned char>& cleartext, const std::vector<unsigned char>& key, const std::vector<unsigned char>& iv) {
    std::vector<unsigned char> out(16);

    const EVP_CIPHER* cipher = nullptr;
    EVP_CIPHER_CTX* ctx = nullptr;

    CF_CHECK_NE(cipher = EVP_cast5_cfb(), nullptr);
    CF_CHECK_NE(ctx = EVP_CIPHER_CTX_new(), nullptr);
    CF_CHECK_EQ(EVP_EncryptInit_ex(ctx, cipher, nullptr, nullptr, nullptr), 1);
    CF_CHECK_EQ(EVP_EncryptInit_ex(ctx, nullptr, nullptr, key.data(), iv.data()), 1);
    {
        size_t outIdx = 0;
        int len = -1;

        CF_CHECK_EQ(EVP_EncryptUpdate(ctx, out.data() + outIdx, &len, cleartext.data(), cleartext.size()), 1);
        outIdx += len;

        CF_CHECK_EQ(EVP_EncryptFinal_ex(ctx, out.data() + outIdx, &len), 1);
        outIdx += len;
    }

end:
    EVP_CIPHER_CTX_free(ctx);
    
    return out;
}

int main(void)
{
    const std::vector<unsigned char> cleartext = {0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0xff, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20};
    const std::vector<unsigned char> key = {0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0xff, 0x20, 0x20, 0x20, 0x20};
    const std::vector<unsigned char> iv = {0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0xff};

    const auto openssl_out = openssl(cleartext, key, iv);

    for (size_t i = 0; i < openssl_out.size(); i++) {
        printf("%02X ", openssl_out[i]);
    }
    printf("\n");

    return 0;
}

Output when compiled against master branch:

CF 90 97 8A 85 CD 20 7D 6F 73 49 2F 30 EF 7D A9

Output when compiled against 1.1.1f:

62 F5 FE 05 67 8E 82 C6 76 42 E1 A7 54 89 53 10

The latter is probably the correct one as this is consistent with Crypto++.

[slontis]

Will have a look - there are other KAT tests for cast so this is a bit strange.

[slontis]

Hmm this works as a test vector in evp_test.. Must be the sequence of calls.

[slontis]

The default keylength is wrong for 2 of the modes.. Fixing it up now.