-
Notifications
You must be signed in to change notification settings - Fork 215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Under load, brands do not match #5571
Comments
The cursed delivery was a [
{
"@qclass": "slot",
"iface": "Alleged: InvitationHandle",
"index": 0
},
{
"@qclass": "slot",
"iface": "Alleged: zoeSeatAdmin",
"index": 1
},
{
"initialAllocation": {
"Central": {
"brand": {
"@qclass": "slot",
"iface": "Alleged: RUN brand",
"index": 2
},
"value": {
"@qclass": "bigint",
"digits": "1500000000"
}
},
"Liquidity": {
"brand": {
"@qclass": "slot",
"iface": "Alleged: BLDPoolLiquidity brand",
"index": 3
},
"value": {
"@qclass": "bigint",
"digits": "0"
}
},
"Secondary": {
"brand": {
"@qclass": "slot",
"iface": "Alleged: BLD brand",
"index": 4
},
"value": {
"@qclass": "bigint",
"digits": "300000000"
}
}
},
"notifier": {
"@qclass": "slot",
"iface": "Alleged: notifier",
"index": 5
},
"offerArgs": {
"@qclass": "undefined"
},
"proposal": {
"exit": {
"onDemand": null
},
"give": {
"Central": {
"brand": {
"@qclass": "slot",
"index": 2
},
"value": {
"@qclass": "bigint",
"digits": "1500000000"
}
},
"Secondary": {
"brand": {
"@qclass": "slot",
"index": 4
},
"value": {
"@qclass": "bigint",
"digits": "300000000"
}
}
},
"want": {
"Liquidity": {
"brand": {
"@qclass": "slot",
"index": 3
},
"value": {
"@qclass": "bigint",
"digits": "1000"
}
}
}
},
"seatHandle": {
"@qclass": "slot",
"iface": "Alleged: SeatHandleHandle",
"index": 6
}
}
] The
Now, what's interesting is that the error message ("Brands in left Object [Alleged: BLDPoolLiquidity brand] {} and right Object [Alleged: BLDPoolLiquidity brand] {} should match but do not") is showing the same I'm looking at the deliveries to that vat, and the only time
The only vref that arrives with I think the next step is to talk with someone who knows the contract well, and figure out which deliveries are providing the two sides of that comparison. Also to try to reproduce it in a single vat, from just the slogfile trace. |
For reference, Ari said this run was using 9d670a . |
All three validators match on that delivery, so at least it's not a nondeterminism in the JS engine (the delivery->results function). |
In the 679 vpool-xyk-amm deliveries leading up to the "brands do not match" error, I see 78 errors emitted in one of three categories:
I think this is the army of clients all trying to be the first to register something unique, which is at least the right conditions for a race bug. Interestingly, the "already registered" error appears on delivery 303, in the middle of the other two types of errors. It seems surprising that it would continue to complain that BLD isn't found even after it complained that BLD was already registered, but I don't really know what the registration is about. |
@Chris-Hibbert and I narrowed this down to a race condition in the AMM's agoric-sdk/packages/run-protocol/src/vpool-xyk-amm/addPool.js Lines 17 to 59 in 8ad7422
Chris will write up the details properly, but my (weak) understanding is:
|
All the
which will always succeed, replacing any existing assignment to that keyword. Anything else is a TOCTOU confusion. |
I think Zoe is safe and consistent. the unsafe
That still leaves an issue in the AMM's addIssuer, which safely creates a unique liquidity token per pool, but unsafely hands out incorrect doppelgangers to anyone who asks before the canonical secondary pool is finished being created. |
It's not "immediately" because there is an intervening const saveIssuer = async (issuerP, keyword) => {
const issuerRecord = await issuerStorage.storeIssuer(issuerP);
await escrowStorage.createPurse(issuerRecord.issuer, issuerRecord.brand);
instanceRecordManager.addIssuerToInstanceRecord(keyword, issuerRecord);
return issuerRecord;
}; So multiple issuers could be successfully stored as if they were the issuer for the keyword, but only the last one would win and be legit. That seems pretty unquestionably a race bug. |
Proposed solution:
Alternatively, we can disclaim the problem and just say "contracts should make sure to not register multiple issuers with the same keyword". Any other approaches we should consider? |
@erights please respond to the above proposal. |
I am reluctant to add more internal state, especially state that would need to be durable. Aside from that I'm still chewing on this. We should talk. |
* fix: handle parallel requests for the same issuer in AMM's addIssuer refs: #5571 There are two bugs in #5571, and this fixes only one of them. * chore: clarify a comment * refactor: revert to multiple await style; handle race at .init() Added a test for multiple attempts to add the same issuer with the same keyword. * refactor: rearrange error handling drop try-catch for brand-to-promise add catch for saving issuer and notifying Reserve * chore: delete from brandToLiquidityMint on failure name cleanup * chore: cleanups: types, variable name, logging * chore: move DISPLAY_INFO to top level scope * chore: delete entries from brandToLiquidityIssuer when done This is probably what was intended instead of the code fixed in #5611 * chore: convert a .then to E.when(). * refactor: return addIssuer as named (rather than anonymous) function
#5692 is directed at the remaining issues. |
Describe the bug
slog at:
https://drive.google.com/file/d/1FIg_y0Tp1PsE438LtKBxhp_ki1yqfjEF/view?usp=sharing
The text was updated successfully, but these errors were encountered: