-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Corrupted validators and untraceable tx ordering causes PoSA abuse & network spam #911
Comments
Excellent work. BSC indeed appears to be a rotten cesspool of corruption. I've been pounding the table about the current MEV situation (one that generates huge amounts of spam in the form of excess network traffic, fake nodes and massive tx bloat) for a while now, @'ing the devs, only to get not a single reply from any of them. Why is the whale botnet that is currently sucking up 95%+ of the MEV being allowed to continue ruining the network for their own financial gain? Not holding my breath for an answer... |
I would be willing to refactor the transaction sorting by Searching the validator rules I didn't find any information if they are forced to use the latest official Geth version. With the current implementation validators can sort transactions as they like without anyone being able to prove to them that this order has been manipulated or not. |
Sadly this has been a long standing issue and the devs could not be more indifferent about it. I'd be amazed if they merged your pull request. The spam under the current system is especially bad since each of the hundreds of txs being spammed per block are each being spammed by hundreds of different bots, each with different tx hash but same nonce, so really there are as many as tens of thousands of full txs flooding the entire network every block, all for the profit of a single individual or syndicate. Changing the tx sorting away from arrival time will immediately stop the need for these hundreds of fake spammer nodes and massively ease the network bandwidth demands currently being experienced by full nodes. Currently, the devs seem more interested in putting large amounts of work into reengineering geth to accommodate and benefit this single profiteer, rather than just disposing of them by changing a few lines of code back to what they used to be before the 1.10 fork.
From what I've seen, they generally seem to update to the latest version reasonably promptly. |
Before 1.10 fork, transactions with the same gas price were randomly sorted which also led to tx spam. I can't think of any other way than to use tx hash POW for this kind of tx sorting to make the sorting traceable. However, corrupt validators could then delay individual txs from the competition that have a better hash for this MEV opportunity and only include them in the next block. I would prefer that instead of the network being spammed with txs. |
You and many others have pointed out that validators can manually order txs in any way they see fit to their advantage, which is certainly plausible. However, this would not explain the need of the current MEV monopolist to spam hundreds of txs per block. If they were colluding with the validators (you'd need a backroom deal with 21 separate parties), or just were the validators in disguise, there would be no need for spam - a single tx would do the trick if you had god mode powers to reorder txs within a block. In fact, this is something I did witness for several months after the 1.10 fork. A single contract was successfully winning 50%+ of the arbs with a single tx. After several months of domination, this success rate dropped and dropped until they were no longer even breaking even, and they cut their losses and shut down. The competition now is so intense that you have to run hundreds of VPS just to have a hope of making a profit.
This is true, it did, it bloated blocks with failed arb txs. This is why the Eth devs changed the sortation to arrival time, the end result of which (on BSC at least), is that not only do we have tx spam of unprecedented levels, but also the network itself is bloated with fake nodes. So the system we have now is the worst of both worlds. I agree that sorting by tx hash would be the best approach. It would dramatically reduce spam since each bot owner knows not to send another tx until the current best hash has been undercut. It solves the problem of block bloat, and massively eases the burden on the full nodes. It's win/win for everyone except Mr. 0xaf25d6494a341675186f975c4cbc081fdac05dbc. |
I agree. I'll try to discuss with the maintainers via the Discord server if they would agree to such a change. Hopefully Ethereum POS will think about this, so that validators can not exploit their position. Would be a win for all but the corrupt validators ;) |
When nodereal introduced their DirectRoute, that ruined my previously profitable trader bot, because now malicious actors have a way of rug pulling without going through public mempool (thus no way for my bot to see rug pull tx and try to frontrun it).
I think you're exaggerating. I'm logging how many txs reach my nodes between blocks, and numbers are fairly stable 200-300 txs. There are occasions when I see several thousands, especially during "panic events" when everyone wants to buy/sell something and there are lots of arb opportunities, but those are rare and usually only last for several blocks. In general that's how decentralised network works - there is no "police", everyone is free to choose whether they're running a vanilla software or if they modify it for their own needs (i.e. different tx sorting, blocking, delaying, whatever). I think tx hash idea is good for overall network health. MEV searchooors will go from having many fake nodes to having few large nodes that will be constantly calculating a "better hash". But how do you stop CREAM from making exceptions for themselves or their friends, or how do you stop nodereal from offering DirectRoute? And IMO it's very unlikely that Binance will assume this "police" role and will start actively monitoring network and punishing validators for "not following sorting rules". Just my $0.02. |
To clarify, these arent "txs" in the sense that you're talking about, i.e. the final txs that are mined into the block by the validator. What this guy is doing is sending a different version of each of those couple hundred txs from each of a few hundred bots. So in terms of network bandwidth, these two values are multiplied. This is what I witnessed a few months ago when I was looking in real time at the actual contents of the messages I was getting from his nodes. Presumably this has something to do with ensuring the best arrival time by calculating the arb from many different fake nodes and sending txs from the same accounts with the same nonce from each. Whichever of these, per nonce, arrive first at the validator wins and the rest are stripped out as duplicates by any nodes they encounter. As far as network bandwidth is concerned, it could well be tens of thousands of tx messages, although obviously any given node will only forward one of them per nonce. |
I think that 200-300/s transactions arriving at your end is because the Full Nodes do not forward duplicate transactions (same Nonce & GasPrice) because they are invalid. But in the whole network there are suddenly thousands of transactions with the same nonce/gasPrice where only the first arrived tx is taken by the validator.
That the blockchain is uncontrollable and there is no police is correct. Nevertheless, it is centralized at 21 points (the validators), because it is not otherwise possible with PoSA. I have also rewritten Geth to my benefit, but there must still be clear rules for the validators, because they "control" the network. |
@ktrout1111 I'm sorry, I probably wasn't clear in my original message. I'm logging how many different txs reaches my own mempool between blocks, so my numbers include replacement txs. I don't really care about finalized txs.
that might be true, but I have multiple nodes in multiple locations and numbers are consistent. But I haven't really dug deep into this and you might be right.
yeah, but... anyone can become a validator if they pull together enough of BNB to become one. Community previously pushed nodereal into changing how their DirectRoute works, but I have little hope that same would work with 21 validators. Fingers crossed though! |
This is what I meant. It's extreme spam on the P2P network but has no extra spam effect on the actual blocks than simply sending a single tx per account per nonce. |
DirectRoute is a big problem. If everyone used that, then BSC would be completely centralized. To the example I wrote at the very beginning: Because millions of gas tokens are minted all the time in all CREAM validated blocks, millions of contracts are also created, which in turn generates a lot of garbage. When the gas tokens are then released, the block rewards become smaller at the expense of the other validators. So validators can piss on each other's legs. These 21 validators should not be unregulated and punished for manipulations. BSC is not decentralized! |
It would be interesting if txs w/ same gas price were sorted by hash as apposed to by arrival time. This would offer on-chain proof that txs (w/ same gas price) are being sorted in an entirely transparent and/or fair manner, as of right now, there is no way to know if validators are sorting txs fairly. |
Is there a possibility that CREAM validator is just so stupid that it sets gas floor as low as 1 gwei ? I tend not to explain phenomenon with evil. |
@48ClubSirIan No matter what the reason is: CREAM should be punished for something like this. Such things destroy the blockchain as I explained in this comment. Also, I don't think the validator changed any configs out of "stupidity". With so much money, they will think a thousand times which config is edited. |
I agree with this part that CREAM is harming other validators' profit, and really want it to stop doing so. However I am not sure it should be punished if it's only setting an extremely low gasfloor without selling gas token to specified people. It's easy to test whether an ordinary address can also mint gas tokens, right? To be clear I didn't do the test yet. It's not smart for validators to decide to accept a lower gas price, but not a crime either. |
If the validator has set the GasPrice to 1 GWEI, the "smart" attacker should have a direct connection to the validator node, since full nodes do not forward the txs if the gas price is too low, am I right? |
|
Yes you can send the transaction out. But the other full nodes drop transactions with low gas price. |
Well, the fact is not all peers drop it... |
Do we really want to discuss how many full node operators set the gas price at 1 GWEI? Because I can assure you that nobody does that, because it brings them NOTHING. |
I am not discussing how many, but whether it is possible. If so, we can't simply say CREAM is selling GAS token, which is a serious accusation, solid proof is needed. BTW once confirmed, me too vote for a punishment. |
Of course you can't confirm it. you will never be able to confirm who is doing what transactions on the blockchain. But you know that for sure :) The only thing I know is:
That's exactly how I wrote it in the issue ticket. I have written out my thought processes completely and how I personally come to the suspicion that CREAM is exploiting their position. And even if CREAM didn't know about it, they should be excluded as validators, because then they seem to have no idea what they are doing. I'm sorry if CREAM really didn't know about it and just incompetently changes settings. But the way the Binance Smart Chain is currently structured, I don't trust any validator, because nothing is traceable and validators can do what they want. How is it even possible to store transactions with 1GWEI? Such a thing should be recognized by other validators and then be punished. |
I may have another question: Can a validator legitly set gas floor to 1 gwei with fully aware of it? 5 gwei comes from a factly price alliance (default config), I didn't see why a validator must do so. It's reasonable to accept a lower price in order to raise short-term income. Although not a good idea from bigger view. The whole GAS token thing isn't good at least on BSC but it doesn't deprive validators' right of editing its own config file, I believe. Maybe we'd better raise a PR to remove the gas returning mechanism. |
Okey. Assuming that every validator would adjust the price, it would end up being a "battle" between the validators to see who has the best price. This would lead to thousands of txs being spammed on the network to end up only in the blocks with the cheapest transaction costs. The task of a validator should not be to change prices or to be god! But to give you a little help. It should be: decentralized and not controllable by any party. Not even by the validators, who, as the name suggests, should only validate blocks. Also, I find it a bit rude that someone who works for a validator (from your profile) writes such a comment. This shows that validators have completely missed their target. |
Why dev team doesnt respond to this topic ? |
This is just wrong. If you use the standard config.toml you will find GasPrice = 1000000000 / 1gwei. This means every fullnode running the stanard config will forward them. If you check your txpool you find find many tx bellow 5gwei. If you don't want to forward them just change the parameter.
Validators sell blockspace. It's their decision to set the minimum price for inclusion. There is a rpc method to change this price: https://geth.ethereum.org/docs/rpc/ns-miner#miner_setgasprice. 5gwei seems to be a constant the validators agreed on. If they don't like someone not obeying it, they should take action against it. But I can not find any offical rule stating that they can not make the gasPrice whatever they like.
I don't see how this is corruption. During this 1gwei phase they accepted any tx priced above 1gwei not just the chi mints. My last point is that chi minting at 1gwei is helping the chain. Some protocols like 1inch are relying on it to make transactions for the user cheaper. Since the end user will do that at 5gwei it needs to be minted bellow that. Someone has to do that but no one will do so at a loss. So dropping the floor to 1gwei is necessary for this token to be usefull. I'm pretty confident anyone could mint them at this gasPrice if they see that the validator is mining transaction at this level. Disclosure: I don't run any chi minting bots. There is no conflict of interest for me here. |
the sense that SELFDESTRUCT gas returned was not made with the intention that you buy cheap gas and use it when the gas price is high.
so if validators decide for themselves how much gas they validate blocks for, i no longer understand the point of a blockchain. |
The gastoken was created for this reason. Mint when the gas is cheap and burn it when its expensive to save tx costs. It's their usecase. Why should anyone mint it if they would be losing money? The miners on eth could do the same thing before EIP-1559. Now the basefee is calculated by some formula. But merging this will lead to gasPrice explosions like we see them on other chains and would not really improve something for the users. What is your proposal to change this? Hardcode it to 5gwei, change config.toml and remove the rpc method? |
yes that's why there are gas tokens, i know that.. but this feature (behind gas tokens) was NOT meant for buying gas cheap and selling it expensive. but gas tokens exploited this loophole. There is a reason why gas is getting more expensive and I don't think it's right to circumvent that with gas tokens. Especially if these tokens are minted only at a single validator for almost free. |
It's not a single validator if you look at past blocks. There might be more, but these were the ones I could find with a quick search. |
Our team talked with
The Before geth 1.10 , transactions with the same gas price were randomly sorted will also lead to tx spam since the arb robot will do every effort if there is an opportunity. Sorting by |
thanks for the clarification. however, the PR was created primarily because of other points. i answered you there. |
Except this isn't true. The spam is as bad now, or worse, than it was when the sorting was random. At least when the sorting was random there wasn't this extra problem with fake nodes. https://bscscan.com/txs?a=0x911c6f41f62ae50277dcd2254753bc832aab904b&p=3 See above and the surrounding pages. Dozens of spam txs per block, and that is just one of 4 contracts owned by the same entity. So the spam from a single entity is 4x worse than you can see in the link above. |
It's very unlikely that the BSC team will take any concrete actions like the previous thread. |
Just made a research on the affiliation between flash arbitrageurs and validators. Scanned all the history of BSC to identify all arbitrage transactions, contract and contract owners. Then I've measured the correlation between profitable transactions and block validators. I could not find any strong evidence of affiliation with validators - at least for the top-earning validators. Their profits seem to be fairly distributed (except for some natural skewness because of better peering with some of the validators). Of course, that doesn't rule out the possibility of affiliation completely. You can explore the data yourself in our public dashboard: https://datamint.ai/research/#arbitragemonitor More about the methodology of the research: https://medium.com/datamint-on-chain-data-analytics/can-you-earn-a-lambo-by-being-a-flash-arbitrageur-on-bnb-smart-chain-b119701329fe?source=friends_link&sk=12f2b772d39cf52315c35f514cf5d36f |
closing due to inactivity. |
I will try to explain my view on the current PoSA at BSC.
It's best if I start with an example:
In all blocks validated by the "CREAM Validator", an address performs transactions for ~1 GWEI to get gas tokens.
https://bscscan.com/address/0x111111102ba661adbc11daf8a88d6d54ad3e9ac2
(this is only one of many wallets that performs such transactions in all blocks validated by CREAM)
Then all gas tokens are sent to this wallet and sold there. Here you can trace back how all addresses minted Gas Tokens ONLY in CREAM Validated blocks.
Why are validators not punished for such behavior?
Transaction ordering
In addition, MEV opportunities should be shared fairly for all participants in the blockchain.
Currently transactions are sorted by
Nonce->GasPrice->ArrivalTime
. However, validators can sort the transactions with the same GasPrice as they like, because they can change the ArrivalTime themselves. A suggestion from a previous ticket was to sort the transactions byNonce->GasPrice->Hash
, which I think is fairer because the sorting of the transactions is traceable. Then corrupt validators could only slow down the transactions of the "competition" and include them in the next block. I can't think of any way to fix that.Also, the current method leads to network spam, as transactions with the same nonce/gasPrice but different hash are sent to many nodes to get a desired tx slot. The way tx broadcasting is set up, only a part of the full transactions are sent to other nodes, which leads to this kind of spam.
The blockchain should be decentralized, freely accessible and traceable. Why hasn't this problem, which I see as a major one, been solved earlier?
The text was updated successfully, but these errors were encountered: