Add 12 XMSS and 16 XMSSMT parameters.

[ducnguyen-sb]

1. All XMSS parameters are separated by namespace.
2. After discussion with Andreas Huelsing, he recommends using the fast approach. This approach improved the performance of signing at the cost of larger sk.
3. The _fast parameters is incompatible with the general version. At first, I couldn't believe it, I went back and generate original KATs file from original Github repo. And I confirm that the _fast and general version, given the same message and same randomness, it will generate different signatures. Given the signature of _fast, the general version cannot verify using _fast parameter public key.

Similar issue in xmss-reference: XMSS/xmss-reference#17

4. All parameters are tested with pre-generated KAT files and SHA256 hashes of the KAT.

All the parameters added to XMSS. https://github.com/ducnguyen-sb/xmss-reference/blob/master/external/xmss_liboqs_params.md

Algorithms	oid	sk	pk	sig	n
XMSS-SHA2_10_256	0x01	1373	64	2500	32
XMSS-SHA2_16_256	0x02	2093	64	2692	32
XMSS-SHA2_20_256	0x03	2573	64	2820	32
XMSS-SHAKE_10_256	0x07	1373	64	2500	32
XMSS-SHAKE_16_256	0x08	2093	64	2692	32
XMSS-SHAKE_20_256	0x09	2573	64	2820	32
XMSS-SHA2_10_512	0x04	2653	128	9092	64
XMSS-SHA2_16_512	0x05	4045	128	9476	64
XMSS-SHA2_20_512	0x06	4973	128	9732	64
XMSS-SHAKE_10_512	0x0a	2653	128	9092	64
XMSS-SHAKE_16_512	0x0b	4045	128	9476	64
XMSS-SHAKE_20_512	0x0c	4973	128	9732	64
XMSSMT-SHA2_20/2_256	0x01	5998	64	4963	32
XMSSMT-SHA2_20/4_256	0x02	10938	64	9251	32
XMSSMT-SHA2_40/2_256	0x03	9600	64	5605	32
XMSSMT-SHA2_40/4_256	0x04	15252	64	9893	32
XMSSMT-SHA2_40/8_256	0x05	24516	64	18469	32
XMSSMT-SHA2_60/3_256	0x06	16629	64	8392	32
XMSSMT-SHA2_60/6_256	0x07	24507	64	14824	32
XMSSMT-SHA2_60/12_256	0x08	38095	64	27688	32
XMSSMT-SHAKE_20/2_256	0x11	5998	64	4963	32
XMSSMT-SHAKE_20/4_256	0x12	10938	64	9251	32
XMSSMT-SHAKE_40/2_256	0x13	9600	64	5605	32
XMSSMT-SHAKE_40/4_256	0x14	15252	64	9893	32
XMSSMT-SHAKE_40/8_256	0x15	24516	64	18469	32
XMSSMT-SHAKE_60/3_256	0x16	16629	64	8392	32
XMSSMT-SHAKE_60/6_256	0x17	24507	64	14824	32
XMSSMT-SHAKE_60/12_256	0x18	38095	64	27688	32

7. The upstream of the current code: https://github.com/ducnguyen-sb/xmss-reference

• Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
• Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in oqs-provider, OQS-OpenSSL, OQS-BoringSSL, and OQS-OpenSSH will also need to be ready for review and merge by the time this is merged.)

[ducnguyen-sb]

All test pass in local. CI/CD are time out in many platforms.

[ducnguyen-sb]

@dstebila I got CI timeout, what can I do to extend the test time longer?

[baentsch]

@dstebila I got CI timeout, what can I do to extend the test time longer?

https://stackoverflow.com/questions/36173553/how-to-extend-timeout-for-tests-in-circleci#36209650

HOWEVER, may I question the practical value of code that apparently requires more than 10 minutes for a single signature operation? Or is the test too big?

[ducnguyen-sb]

Hi @baentsch ,

I think the test is too big (if the test is all schemes in liboqs, XMSS test only 1 iteration per parameter), when it reaches to XMSS it's almost out of time.
On my local machine: Apple M1, all test for XMSS took 1.13 second.

I assume CI is slower, if I multiply my local number by 100x, then it's about ~100 seconds. Still below the 600 seconds (10 minutes).

Yes, I think the choice of parameters Sign/Verify (we don't test keygen here, because it's very very long) is reasonable. (again, 1 seconds for 28 Sign/Verify test in my local).

Should I go ahead and modify the timeout?
Or, How can I just test XMSS to iterate my code quicker?

[baentsch]

Should I go ahead and modify the timeout?

That should only be the last resort. But it seems you found a way without doing so: Great! Or are the tests now doing less?

[ducnguyen-sb]

@baentsch I realize that the test_cmdline.py actually run the key pair generation.
So I write a helper so at certain parameters, it should read pre-generated keypair instead of generating one.

You can see the idea in this section of the code:

liboqs/tests/test_sig_stfl.c

Lines 179 to 245 in d797221

	OQS_STATUS sig_stfl_KATs_keygen(OQS_SIG_STFL *sig, uint8_t *public_key, uint8_t *secret_key, const char *katfile) {
	printf("%s", sig->method_name);
	if (0) {
	#ifdef OQS_ENABLE_SIG_STFL_xmss_sha256_h16
	} else if (strcmp(sig->method_name, OQS_SIG_STFL_alg_xmss_sha256_h16) == 0) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_sha256_h20
	} else if (strcmp(sig->method_name, OQS_SIG_STFL_alg_xmss_sha256_h20) == 0) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_shake128_h16
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmss_shake128_h16)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_shake128_h20
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmss_shake128_h20)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_sha512_h16
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmss_sha512_h16)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_sha512_h20
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmss_sha512_h20)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_shake256_h16
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmss_shake256_h16)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmss_shake256_h20
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmss_shake256_h20)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmssmt_sha256_h40_2
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmssmt_sha256_h40_2)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmssmt_sha256_h60_3
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmssmt_sha256_h60_3)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmssmt_shake128_h40_2
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmssmt_shake128_h40_2)) {
	goto from_kats;
	#endif
	#ifdef OQS_ENABLE_SIG_STFL_xmssmt_shake128_h60_3
	} else if (0 == strcasecmp(sig->method_name, OQS_SIG_STFL_alg_xmssmt_shake128_h60_3)) {
	goto from_kats;
	#endif
	} else {
	goto from_keygen;
	}
	from_kats:
	return sig_stfl_keypair_from_KATs(sig, public_key, secret_key, katfile);
	from_keygen:
	return sig_stfl_keypair_from_keygen(sig, public_key, secret_key);

Now I don't know why my test on arm_emulated return different hash output. That's strange.

[ducnguyen-sb]

Oh, I don't know why the test on arm_emulated is off by one.
Instead of checking the 1st KAT hash of 1st parameter, it compares output KAT with the 2nd KATs hash.

No it's not off by one. But somehow I don't know why the hash is incorrect.

[ducnguyen-sb]

The reason the output hash is incorrect because the comparison with sigs_total and sigs_remaining.

In KAT file, when hash tree is at 2^40, it's overflow size_t type in 32-bit.
Change to uint64_t resolve the problem.

Next, scan_build give me errors in the reference code of XMSS.

Am I allowed to modify the reference code (as long as same KAT output) to fix the scan_build?
Anyway, all warnings are false positive in execution, all allocated memories are initialized and has data.

@baentsch can you recommend what should I do to clear CI?

[baentsch]

Am I allowed to modify the reference code (as long as same KAT output) to fix the scan_build?

I'd personally say "Yes, of course" -- if that corrects code otherwise containing flaws. Now, on second consideration the question is this: How did you import this reference code? If it's a fully automated process, please consider doing something similar to what we already do in scripts with copy_from_upstream and copy_from_xkcp: In both cases there's the concept of "copy-then-patch". In this case, the patch of course should contain the changes to fix "scan_build" (and you may want to consider contributing those changes upstream such as to eventually completely get rid of them here -- and improve the reference code). If you did the import manually, well, please consider automation: It'll save all of us a lot of problems in the future (that upstream changes or we harden our own tests).

[ducnguyen-sb]

How did you import this reference code?

Unfortunately, I import this code manually. The original reference does not:

1. Separate message and signature
2. Type confusion (unsigned long)

And it was not updated for years, due to this I think automation from upstream is a bit overkill. I think my code is a little far ahead of reference implementation.

I will look into the code to fix scan_build.

[baentsch]

I think my code is a little far ahead of reference implementation

Good: That then IMO gives you a clear answer to

Am I allowed to modify the reference code

Yes -- by all means: Make this code base the new (better) code base of reference.

[ducnguyen-sb]

Scan build is happy. This branch is ready to be merged.

[baentsch]

LGTM to merge to "stateful-sigs". Thanks for the latest code enhancements, @ducnguyen-sb ! For a "main" branch merge, I'm personally missing quite a bit of documentation -- but that's arguably another separate task/PR.