[RELEASE] Monero Cryptonight variants, and add one for v7

[moneromooo-monero]

This is the first variant of many, with the intent to improve
Monero's resistance to ASICs and encourage mining decentralization.

[SChernykh]

A few critical thoughts on VARIANT1_1 macro:

It depends on nonce which is controlled by miner,
so the obvious first step for a smart miner
would be to optimize it and use only even nonce values,
and nonce_flag would be always 0 as a result.

After a few transformations and with nonce_flag == 0,
VARIANT1_1 macro becomes this:

uint8_t tmp = ((const uint8_t*)p)[11];
((uint8_t*)p)[11] = (tmp ^ 16) | ((tmp & 16) << 1);

Almost all the original bit manipulations and branches are gone.
And I spent only 20 minutes to get to this point. Maybe it can be optimized even further...
I'm sure all optimized mining software will end up with a code similar to this.
Nonce values will be effectively reduced to 31 bits only.

What's even worse is the truth table of a function defined by VARIANT1_1 (bits 4 and 5):

nonce_flag == 0:
00 -> 01
01 -> 10
10 -> 11
11 -> 10

4 different input values become only 3 different output values.

It reduces entropy in the scratchpad on every iteration.
I have a feeling it makes the hashing function worse,
from cryptographical point of view.

Did any professional cryptographers review this?

P.S. There is one more post_aes() macro on line 729 (for aarch64 architecture). It looks like it doesn't have VARIANT1_1 macro. Was it tested on ARMv8 processors?

[moneromooo-monero]

Interesting comment about decreasing entropy. I'll pass it along.
The fact that some ops can be removed doesn't matter much for this iteration I think, as long as it is different. That is something we can remove/replace in 6 months' time if it is indeed a weakening factor.
We do not have cryptographers familiar with this kind of thing, sadly.
I do not have any ARM to test with. I'll fix the missing call, thanks.

[vtnerd]

I think nonce_flag should be determined by a byte/bit in the first keccak hash. User input cannot influence the selection of the nonce_flag flag (without breaking keccack pre-image resistance), and the length < 43 check is no longer needed. Or is the goal to "drag" something across the hash ?

[vtnerd]

@SChernykh -> The logic table:

tmp (bits 5,4) / nonce_flag	tmp3	tmp0	p[11] (bits 5, 4)
00 / 0	0	00	00
00 / 1	0	00	00
01 / 0	1	10	11
01 / 1	1	01	01
10 / 0	1	10	10
10 / 1	1	01	11
11 / 0	0	10	11
11 / 1	0	00	11

There is a bias towards 11 for bits 5,4 now. Based on this table, this algorithm is likely better off it does (01 -> 10) when !nonce_flag and (10->01) when nonce_flag (everything else is doing nothing anyway).

uint8_t tmp = p[11];
uint8_t tmp1 = ((tmp & 0x18) == 0x08) ? 0x10 : tmp);
uint8_t tmp2 = ((tmp & 0x18) == 0x10) ? 0x08 : tmp);
p[11] = nonce_flag ? tmp1 : tmp2;

Its practically the same thing, but much easier to read. Again, the only functional difference is that the mapping flips the bytes in the only two scenarios where the values are changing in this patch.

Also, why the tmp & 0xef ? This simply drops the high bit, what about tmp ^ (tmp & 0x80) instead? The high bit is still modified, but isn't fixed to zero. All of these changes have less math operations, and two additional cmov ops (or branching, in worst case).

EDIT: typo in last bitmask.
EDIT: corrected block of code, sorry for those reading via email.

[SChernykh]

I think your logic table and the resulting code is incorrect. For example, 01 / 0 should result in p[11] (bits 5, 4) = 10, not 11. I tested mine by running through all 256 values and comparing results with the original macro.

P.S. I've read your comment again. So you're suggesting a slightly different macro, not the same mathematically?

P.P.S. You new suggested macro replaces half of the input values with 0x10 or 0x08, depending on the nonce_flag. Did you forget to add some XOR manipulations?

[vtnerd]

I did get my masks wrong, in that they were masking one bit over from this patch (the bitshifting is zero based, dummy me), but otherwise the table is correct AFAIK. The 01 / 0 case you mentioned:

tmp = 0y00010000
tmp1 = 0y00000001
tmp2 = 0y00000000
tmp3 = 0y00000001
tmp0 = 0y00000010
p[11] = 0y00110000

The difference in our observations are that I was referring to (5,4) if bits indexes are (7,0). The code I wrote probably confused things because the constants needed to be 0x30, 0x20, and 0x10 (i.e. I made the same mixup when writing the constants that you likely did when analyzing the values).

P.S. -> yes I was suggesting a variant to remove the bias in the output. This probably doesn't matter much because the remainder of the algorithm will "whiten" this bias anyway. The primary reason to reject my change is that it might be easier to modify an existing ASIC. I think it would be the same - with either change a hardware designer might consider the equivalent of cmovs with constants instead of arithmetic operations. Although this getting a bit far out of my area of expertise (specifically the difficulty/expense at designing the types of circuits I am referencing).

P.P.S. -> No.

[vtnerd]

Whats the purpose of the alias? Why not just call the other one?

[SChernykh]

I do not have any ARM to test with. I'll fix the missing call, thanks.

Any modern Android phone has 64-bit ARM processor. Raspberry Pi 3 model B with 64-bit Linux would do as well.

[psychocrypt]

@moneromooo-monero Could you please explain the effect of this change on the cryptonight algorithm. IMO this change has a big potential to break the save hash algorithm. I am no crypto expert but the first simple rule is to use only transformations which not reduce the entropy. And the new POW extension is such a forbidden transformation.

PLEASE PLEASE do not mainline this POW change before the algorithm is changed to keep the full entropy.

[moneromooo-monero]

See above for discussion.

[moneromooo-monero]

A proposed modification from othe:

--- slow-hash.c	2018-02-04 21:02:32.000000000 +0400
+++ slow-hash_0213.c	2018-02-13 13:59:32.000000000 +0400
@@ -49,13 +49,17 @@
 extern int aesb_single_round(const uint8_t *in, uint8_t*out, const uint8_t *expandedKey);
 extern int aesb_pseudo_round(const uint8_t *in, uint8_t *out, const uint8_t *expandedKey);
 
+
+//old: 00->01  01 ->10  10 -> 11   11-> 10(repeat)
+//11->00(modify)
 #define VARIANT1_1(p) \
   do if (variant > 0) \
   { \
     uint8_t tmp = ((const uint8_t*)p)[11]; \
     uint8_t tmp1 = (tmp>>4)&1, tmp2 = (tmp>>5)&1, tmp3 = tmp1^tmp2; \
     uint8_t tmp0 = nonce_flag ? tmp3 : tmp1 + 1; \
-    ((uint8_t*)p)[11] = (tmp & 0xef) | (tmp0<<4); \
+    tmp0 = ((nonce_flag==0)&&(tmp1==1)&&(tmp2==1)) ? 0 : tmp0;\
+    ((uint8_t*)p)[11] =  nonce_flag ? ((tmp & 0xef) | (tmp0<<4)):((tmp & 0xcf) | (tmp0<<4)); \
   } while(0)
 
 #define VARIANT1_2(p) VARIANT1_1(p)

(and we can replace the nonce flag as per vtnerd's suggestion, or just remove it).

[psychocrypt]

Than please do not release the previous POW change. We know that the current POW change introduce a potential issue. Please provide directly the save change from @othe if the entropy in his POW change stay equal.

[moneromooo-monero]

Obviously, the modified version would be used only, assuming no problems are found with it.

[SChernykh]

A proposed modification from othe

Still has the same flaw. If nonce_flag = 0, it transforms 00 and 10 into 01. nonce_flag = 1 looks ok.

[moneromooo-monero]

Yes, but that's taken care of by using nonce_flag from keccak as vtnerd mentions, does it not ?

[SChernykh]

I mean, it still reduces entropy if nonce_flag is 0.

[moneromooo-monero]

tbh I'm thinking we might just xor some bytes with a constant. That's simple, and I expect it cannot modify the properties. What do you think ?

[SChernykh]

xor is certainly safe in this regard.

[psychocrypt]

Could you please provide the code snipped you thinking about (wirh the magic constants) To be sure that the hash function not breaks something obviously please create a histogram over all combinations only if all values has the same hight the extension should be used.

[moneromooo-monero]

I have no actual code yet, I will post a suggested one when I do.

[psychocrypt]

I will provide also a suggestion for the hash extension. Should the extension have some special operations in? Bitoperations, control flow branches, ...

[moneromooo-monero]

You'd have to chat with othe on IRC, as he has some knowledge about what kind of changes are how hard for ASIC design. I'd keep it simple myself, but then I have no such knowledge :)

[vtnerd]

@psychocrypt The values are still being modified by AES based on the scratchpad, then hashed again at the end. So having a bias in the intermediate step is like having a bias in the block itself. The block is not random data. My suggestion about removing the bias was purely on the observation that a hardware designer might implement it differently than the code anyway, and therefore the code should mirror that difference and remove the bias at the same time.

@SChernykh I think entropy is not the correct word to use here. This variant is not going to be used to for password stretching/whitening nor generating many secrets based on a single seed. The block that is the source of the hash data, has no entropy.

[vtnerd]

@SChernykh as a follow-up, I was critical of the word entropy, but I'm not sure exactly what to call this. A bias in the output is going to make it increase or decrease the probability of finding a valid PoW depending on the specifics of the bias. In this case, the "bias" is occurring on the larger AES scratchpad so its not the same as operating on the output of the last hash function in the process. Messing with the output of the first keccack stage is probably somewhere between the two, but I'd have to think about that some more.

[psychocrypt]

@vtnerd But all in all the cryptonight algorithm is AES. This mean that you will create an output (encrypted) where you can not run a frequency attack because all used symbols in the result will not correlate with the input symbols. You break this by using a manipulation which reduces the possibility of combinations (symbols order). This will not be an first order effect but it is an higher order effect. Such a change can (not is) lead to an correlation between the nonce and the resulting hash.
IMO the word entropy is fine because we are using a transformation with a bias which can also result in the worst case that the resulting hash will always be zero (not in this case but this can be an effect of the transformation with a bias)

Therefore we should not use any transformation which has a bias. Also a bias in the input data can not be compared with a bias during the hashing (AES). Keep in mind that each round added the bias to the temporary scratchpad.

[psychocrypt]

@moneromooo-monero If we really would to prevent ASICs we should increase the scratchpad size and the number of rounds. This will be the most secure change we can do without an crypto expert everything else has the potential side effect that we break the hash algorithm and adding a few bit shifts will not avoid ASICS or FPGAs.
I agree that the fast (6 month) change of the pow will avoid ASICs but no FPGAs (first step to ASICs)

[otheATgh]

#define VARIANT1_1(p) \
  do if (variant > 0) \
  { \
      uint8_t tmp = ((const uint8_t*)p)[11]; \
      uint8_t tmp1 = (tmp>>4)&1, tmp2 = (tmp>>5)&1, tmp3 = tmp1^tmp2; \
      uint8_t tmp0 = nonce_flag ? tmp3 : ((((tmp2<<1)|tmp1) + 1)&3); \
      ((uint8_t*)p)[11] =  nonce_flag ? ((tmp & 0xef) | (tmp0<<4)):((tmp & 0xcf) | (tmp0<<4)); \
  } while(0)

tmp(bit5,4)/nonce_flag	tmp3(xor)	tmp0	p11
00/0	0	01(used)	01
00/1	0(used)	0	00
01/0	1	10(used)	10
01/1	1(used)	1	01
10/0	1	11(used)	11
10/1	1(used)	1	11
11/0	0	00(used)	00
11/1	0(used)	0	10

With this there are no complains about "entropy" issues. Even the first version tho had no issues according to my simulation and using these 20k hashes: https://ufile.io/646co , converting to bits and coutning the distribution.

PS: The nonce complaining is irrelevant, as we use the orig nonce for keccak round but nonce plus 1 for the mainloop. No way to abuse using the same nonce.

[vtnerd]

@psychocrypt I do not follow your argument entirely. One thing to consider is that symbol reduction is always occurring after the AES scratchpad stage - a 1536-bit internal state is being reduced to 256-bit value. So having an internal reduction isn't necessarily bad, although it could be. This wasn't obviously bad (like mapping half of the values to one specific 1536-bit state), but could skew the difficulty - although I think that is unlikely. You had another comment about the nonce and the hash, which is why I recommended using the output of keccak for that value. To selectively influence the internal state, someone would to need to break keccak pre-image resistance.

Changing the rounds is probably a bad idea, it could help any ASIC instead of hurt. For example, increasing the number of AES rounds could help CPUs in comparison to GPUs since CPUs have instructions / circuits for one AES round and not a specific number of iterations. If instead, the AES round itself was tweaked, this would make the CPU instructions meaningless and benefit GPUs. This is the same situation, increasing the rounds could benefit ASICs that implement the entire round in a circuit so that it could be tweaked. Making the scratchpad bigger would be interesting, but probably only if the algorithm were tweaked. And it would mess with CPU miners big time. So in other words, Monero is probably better off doing no changes than these two in particular (because even bigger scratchpad with algo tweak will alter some relative CPU/GPU performance).

[otheATgh]

You had another comment about the nonce and the hash, which is why I recommended using the output of keccak for that value.

Last update is unbiased, we use different nonces in the mainloop than in the keccak loop, using the nonce is completly fine.

[SChernykh]

Can we see all the latest changes together somewhere? This pull request has never seen any commits but the original one.

[moneromooo-monero]

I do not understand "use the orig nonce for keccak round but nonce plus 1 for the mainloop", can you explain that please ?

[moneromooo-monero]

For the record, the following program goes through all the p[11] and checks all output values are reached for both version of nonce_flag.

#include <stdint.h>
#include <stdio.h>
#include <string.h>

#define VARIANT(p) \
  do { \
      uint8_t tmp = ((const uint8_t*)p)[11]; \
      uint8_t tmp1 = (tmp>>4)&1, tmp2 = (tmp>>5)&1, tmp3 = tmp1^tmp2; \
      uint8_t tmp0 = nonce_flag ? tmp3 : ((((tmp2<<1)|tmp1) + 1)&3); \
      ((uint8_t*)p)[11] =  nonce_flag ? ((tmp & 0xef) | (tmp0<<4)):((tmp & 0xcf) | (tmp0<<4)); \
  } while(0)

int main()
{
  uint8_t p[12];
  int count0[256], count1[256];
  memset(count0, 0, sizeof(count0));
  memset(count1, 0, sizeof(count1));
  for (int v = 0; v < 256; ++v)
  {
    int nonce_flag = 0;
    p[11] = v;
    VARIANT(p);
    ++count0[p[11]];
    nonce_flag = 1;
    p[11] = v;
    VARIANT(p);
    ++count1[p[11]];
  }
  for (int n = 0; n < 256; ++n)
  {
    if (count0[n] != 1) printf("count0[%d] = %d\n", n, count0[n]);
    if (count1[n] != 1) printf("count1[%d] = %d\n", n, count1[n]);
  }
  return 0;
}

[SChernykh]

@miki-bgd-011 They're not deleted, GitHub is glitching now. It doesn't show hidden comments in the middle when you try to expand them.

[SRCoughlin]

@SChernykh Yes, also confirmed. :) I'm just going to bow out of this conversation now.

[MGTOWMoneroXMR]

Hey guys i've uploaded some snips of those "missing" comments @dave-andersen @rufus210 , odd that I can't see them here on either mobile or desktop?

https://imgur.com/a/emq1Y

I think we should be consulting further with the community regarding this matter if it is infact a large enough "risk". please be vocal and step forward if you have something to share on the matter, thank you

I also wish that the "leadership" would actually refute or alleviate these concerns professionally instead of swatting away any form of debate.

[sammy007]

@fluffypony

hashrate follows the community and economically sensitive nodes

Hashrate follows profitable coin to mine, care about market depth (to dump revenue) and volume and nothing else.

[Jh0nW1cK]

Please gentlemen, transparency and flexibility. We can all collaborate for the common good. There are many people concerned about this change and I think it is the duty of those who lead it to inform and provide detailed information about the whole process. Personally I think that the possible threats are out. But with a certain attitude on our part we can also hurt. With MoneroV the price should be rising or holding, even with the general bearish trend driven by regulation. In short, I think we should approve solid projects and spread the benefits of these so that the value of XMR rises strongly and does not depend so much on BTC.

[el00ruobuob]

I can't understand all those angry posts.
I am looking forward for this PoW Change, hopping it will kick out ASICs and even botnets (at least for a while). And i'm sure many legit miners (as i am) are.
If ASICs are unavailable to centralize mining power, where is 1GH/s benefit over 500MH/s?
The only option i see: angry mens owned ASICs or non-upgradable botnets.

[Jh0nW1cK]

I would really like the hashrate to fall below 300Mh / s in the short and medium term. That would demonstrate the effectiveness of this change and improve the profitability of mining

[ghost]

They seem to be too focused on resisting ASIC and secondly, leave behind the bad and unfairness of botnet mining and cryptojacking. Yes, ASIC centralizes in industries and is not good, but Botnets, involuntary webmining, also centralize millions of dollars in profits for thieves and swindlers.

Today, thousands of honest miners compete against resource thieves and mafia groups, who control botnets and cryptojacking. Ignoring this reality, we fall into the hypocrisy of condemning the Chip Industry, but turn indirectly favouring thieves and criminals.

We need to make a change in the network protocol that really technically prevents these unjust behaviors from proliferating. I believe that the ethical and fairness issue is more essential than the problem of making changes.

How will this change stop theft and injustice?
They will simply upgrade miners in the botnet and injustice will continue to gain.

[fluffypony]

@cryptonixie What you're asking for is literally impossible - how can you honestly expect a protocol to be able to tell a botnet miner from an honest CPU miner? Surely you can logically see how that is impossible to do?

We can't prevent botnets except by embracing ASICs, which would destroy the profits of all "honest" CPU and GPU miners and completely centralise Monero's mining. This is a matter of choosing the lesser of two evils.

[fluffypony]

@el00ruobuob I tend to agree - existing CPU and GPU miners are embracing this change as their profits can only go up from kicking surreptitious ASICs off the network; anger seems to be stemming from people who either have access to ASICs or have pre-ordered them or similar.

[ghost]

@fluffypony If you can, it's software. But it is by no means easy to achieve consensus on such a big change. That's why I don't think it'll happen.

I understand that the issue is complex as far as the necessary changes are concerned, but it seems to me that this was the case because the creators of the Crypto coins never took into account the theft of resources.

That's why he thought we should influence to go down the road of changing the protocol by incorporating an anonymous mining log and a verification by device or similar that actually changes the scenario.

It could take control of the hashrate. a private, anonymous mining log, and a single human verification step by device that mines for the network.

[fluffypony]

@cryptonixie LOL - you think a human should be in charge of who is allowed to mine? I can't tell if you're joking or being serious. On the off chance you're being serious and are just naive, consider this: how do we prevent that human (or group of humans) from being paid off by a botnet to allow them to mine? Or what happens when a country decides that they don't want Monero mining at all in their country, and then apply legal pressure to that human to turn miners off in that country?

[ghost]

@fluffypony Of course, machines must always be governed by humans.

Unless you're on the machine side?

[fluffypony]

@cryptonixie on the contrary; Monero is permissionless to use, and permissionless to mine. Do you think a human should also check the transactions and also allow only the legitimate ones through?

[ghost]

@fluffypony The fault lies in mining, in allowing evil to steal resources from under our noses.

It's true, thieves take advantage of Monero's innocence. Why is it impossible to stop ignoring it and be aware of it?

[fluffypony]

@cryptonixie Sure, so then what happens when thieves take advantage of Monero transactions? Should we also "be aware of that" and have a human checking transactions?

[rufus210]

CryptoNight ASIC announced: https://www.baikalminer.com/product12.php

The ASIC is advertising 30x perf/W of a Vega, which sounds credible for off-chip vs on-chip memory accesses.

If they're announcing it today they're probably confident that this change is't going to disable their ASIC.

[Jh0nW1cK]

This massive increase of hash in the network monero was caused mainly by big botnets like coinhive, javascript miners, botnets of compromised servants / instances of cpu and other things. Nearly 70% of the Monero network are cotnols. Every month we see news about new big botnets. Only FYI: 10K units of Baikal Giant N are only 200Mh in cryptonight. Asic is only a small part of the problem because of its high price.

[miki-bgd-011]

I was against PoW change because I believed ASICs were at least a year away. I was wrong. Change the motherfucker. give them hell.

[SChernykh]

If they're announcing it today they're probably confident that this change is't going to disable their ASIC.

Or they're confident that they need to dump their ASICs before they become useless. They've already edited their bitcointalk and twitter posts and excluded Monero from the list of supported coins.

[ProofOfKeags]

Just got a marketing email from bitmain about the X3. Reddit link found here. Do we know if this change is being communicated to the makers of sgminer, or will they just be expected to catch up?

[Jh0nW1cK]

If that is true (considering the short time that the network has increased its hash) after the change of POW the network would have to go down from 1000 Mh / s from now to enter 100 and 300 Mh / s

[Gingeropolous]

@CaptJakk , ohgodacompany has been communicated with, and they have patches for sgminer

[nerdralph]

@fluffypony ASIC resistance and botnet resistance are not exclusive. Ethash is an example of this. I would agree that making a CPU-friendly and botnet-resistant algorithm is very difficult. I think making the algo just GPU-friendly (and keeping ASIC resistance as a goal) is reasonable.

[ghost]

@fluffypony That doesn't justify ignoring it. You are mixing two different issues, one thing is to control the cryptojacking, and another is a utopia of making a coin infallible from evil, they are different issues in terms of tackling the problem, one is realistic, and the last is impossible.

If you are a programmer, you should know that it is possible to create an authorized network while maintaining anonymity. How also to use an algorithm that is more robust against theft of botnet resources and hidden webminning.

However, it is clear that these changes in the protocol are not in the interest of the Core of the Monero project. And it's predictable that you guys they put forward a lot of arguments and philosophical foundations cyberpunk, which have nothing to do with the main reason for the issue, maintain and develop a technology that is infected with unethical cryptojacking. And it smells like rot.

[LordMajestros]

It has been some time after the fork. The current network hashrate shows most of the lost hashrate was from ASICs. The botnet problem is not as large as previously feared.

[Jh0nW1cK]

Undoubtedly it has been a great success, not only because it has lowered the hash of the network, but more good because of how much it would have gone up.
Thank you