fix:splitstore:revert introduced bug #10366
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
magik6k
approved these changes
Feb 28, 2023
7 tasks
12 tasks
dumikau
pushed a commit
to protofire/lotus
that referenced
this pull request
Jun 8, 2023
* Fix the overzealous fix (filecoin-project#10366) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * chore: deps: update to actors v10.0.0 * build: v1.20.0: release prep (filecoin-project#10368) * add upgrade epoch and bump version * Update changelog --------- Co-authored-by: Aayush <arajasek94@gmail.com> * feat: stmgr: cache migrated stateroots * feat: vm: switch to the new exec trace format (filecoin-project#10372) This is now "FVM" native. Changes include: 1. Don't treat "trace" messages like off-chain messages. E.g., don't include CIDs, versions, etc. 2. Include IPLD codecs where applicable. 3. Remove fields that aren't filled by the FVM (timing, some errors, code locations, etc.). * fix: EthAPI: Make newEthBlockFromFilecoinTipSet faster and correct * feat: update renew-sectors with FIP-0045 logic * Hide `lotus-worker set` command Hide the `lotus-worker set` command and print that this command will be deprecated. * chain prune hot -- hotstore online gc * remove accidental add * fix: EthAPI: Correctly get parent hash * make debugging windowPoSt-failures human readable * make gen * indent output * post worker sched: Filter out disabled workers correctly * itests: Test PoSt worker RPC error handling * post worker sched: Retry on alternative worker on RPC errors * make gen * itests: Wait for both workers in TestWindowPostWorkerDisconnected * Unbump api version as its not necessary * deps: Update go-jsonrpc to v0.2.2 * Add command to trigger moving GC of hotstore manually * Timing info and updated docs * fix: EthAPI: Make newEthBlockFromFilecoinTipSet faster and correct * fix: EthAPI: Correctly get parent hash * v1.20.1 prep * fix: state: short-circuit genesis state computation * fix: state: short-circuit genesis state computation * update changelog * Begin account for size during walks * Track size of dags relevant to compaction * Plan out moving GC limiting * GC respects target for max hotstore space * lint * rpcenc: More reliably failing TestReaderRedirectDrop * rpcenc: Propagate closeOnce in beginPost * fix: eth API: correct gateway restrictions, drop unimplemented methods * refactor: EthAPI: Drop unnecessary param from newEthTxReceipt * feat(deps): update to go-data-transfer v2, still wip * chore(deps): update go-libipfs * chore(deps): update to get to compile * update minimum Go version to 1.19.7 * use go-libipfs/files instead of ipfs/go-ipfs-files * fix drand test * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * un-un-update filecoin-ffi * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * fix logging * deprecate MarketListRetrievalDeals, make gen * Fix up approximation and logging * run make gen * small doc patch on how to make gen after api changes * fix TestPrintGroupInfo * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * register rcmgr metrics * replace make all with make deps according to review * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * cid key size * computing szPurge leads to deadlock and unneeded, remove * Review Response * Lint * lint * docsgen-cli * only register metrics once * lint * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * fmt. * lint * feat:splitstore:Splitstore enabled by default (filecoin-project#10429) Discard mode --------- Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Review Response * fix lint. * Configur buffer and threshold * lint * make gen * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * review comments * simplify test. * feat: chain: make fetching tipset by height 1000x faster * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * run make gen * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * fmt. * fix lint. * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * simplify test. * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * make gen * change log * fiix bakcport error filecoin-project#10427 * retract v1.20.2 * go mod tidy * make gen * feat: cli: Add an EVM command to fetch a contract's bytecode * feat: eth API: reject masked ID addresses embedded in f410f payloads We'll never get an actor/account deployed to one of these addresses (although we might get a placeholder). However, converting such an address to an f4 address is definitely wrong. * chore: update the FFI for release (filecoin-project#10435) Specifically, this updates the proofs libraries and enables CUDA by default. * feat: stmgr: skip tipset execution when possible * feat: eth: optimize receipt reading This optimizes the eth APIs (except the fee history one) to lookup the tipset state/receipts instead of computing the state. * feat: api: optimize ChainGetParentReceipts Read the receipts all at once instead of fetching them one-by-one. * test: eth: reduce chances of chain-reorgs affecting the test Now that this API is _much_ faster, we're more likely to "catch up" to the head faster than it can stabilize. I'm pretty sure the test was intended to be written this way anyways. * gateway: fix: drop overzealous guard on MsigGetVested * eth_feeHistory: migrate to using TipSetState. * Eth API: drop support for 'pending' block parameter. After transitioning from using StateCompute to loading receipts, we can no longer handle the 'pending' block without forcing computation. Eth Core Devs are evaluating a proposal to remove support on their end too. * Eth API: make net_version return the chain ID. * fix imports. * remove superfluous filter. * drop irrelevant test. * Eth API: make eth_getTransactionByBlock* ops return ErrUnsupported. * fix: itest: avoid failing the test when we race the miner I've been seeing quite a few tests failing here, so I'm hoping this will deflake them a bit. * feat: eth: Avoid StateCompute in EthTxnReceipt lookup (filecoin-project#10460) Instead: 1. Use the receipt from the message search. 2. Re-compute the gas fees that would have been charged. fixes filecoin-project#10418 Co-authored-by: raulk <raul.kripalani@gmail.com> * refactor: update cache to the new generic version (filecoin-project#10463) - Adds type safety. - Reduces allocations. - Fixes the drand cache (was storing by value, but retrieving by pointer) * fix: gateway: correctly apply the fee history lookback max This fix ensures that the full fee-history range falls into the allowable lookback rang. * Plumb through a proper Flush() method on all blockstores * feat: consensus: log ApplyBlock timing/gas stats * fix: stmgr: don't attempt to lookup genesis state (filecoin-project#10472) * fix: stmgr: don't attempt to lookup genesis state * stmgr: More correct fix for genesis state compute * Revert "Eth API: drop support for 'pending' block parameter." This reverts commit 9412753. * fix: build: drop drand incentinet servers * feat: gateway: export StateVerifierStatus * chore: update go-libp2p-pubsub to v0.9.2 * chore: update ffi to increase execution parallelism (filecoin-project#10480) This changes the default validation parallelism from 1 to 4, and makes it configurable with a LOTUS_FVM_CONCURRENCY environment variable. * fix connection gater integration test * chore: update go-libp2p to v0.26.2 * make mod-tidy ci happy * update go-libp2p-pubsub to v0.9.3 * update go-jsonrpc to v0.2.3 * ethrpc: Buffer sub messages if subscriber is slow * make lint happy * feat: apply gateway lookback limit to eth API lookback This change: 1. Introduces new "limited" API endpoints for EthGetTransactionByHash and EthGetTransactionReceipt that accept lookback-limits. 2. Implements the gateway version of these API endpoints by calling the limited variants with the default message search lookback limit. fixes filecoin-project#10412 * fix: actually apply limit * feat: expose limited apis on the gateway * fix: eth: handle a potential divide by zero in receipt handling This isn't really possible to hit on-chain at this piont (message won't be accepted) but we might as well be extra careful. fixes filecoin-project#10471 * Don't enforce walking receipt tree during compaction * Demote now common logs (filecoin-project#10516) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * feat:config:force existing users to opt into new defaults (filecoin-project#10488) * Config default does not comment out EnableSplitstore * Loadability check * Remove test used for debugging * regexp for properly safe check that config is set * regexp for safely matching the EnableSpitstore field in the config * Add instructions for undeleting config and remind users to set splitstore false for full archive * UpdateConfig small docs and functional opts * make gen * Lint * Fix * nil pointer check on validate * Unit testing of EnableSplitstore cases * Address Review --------- Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * release v1.21.0-rc1 prep * chore: build: Update Changelog for v1.21.0 chore: build: Update Changelog for v1.21.0 * Add RPC API highlight Add RPC API highlight * Add callout to exchanges Add callout to exchanges about the execution traces. Also moving the `execution trace format changes` section to above the contributors section. * fix: miner: correctly count sector extensions * fix: miner: call ExtendSectorExpiration2 * chore: all: bump go-libipfs (cherry picked from commit f48c626) * Initialize with same length as partition Initialize the postParam.Partitions slice with the same length as i.Partitions before iterating over it in the loop. * build: docker: Update GO-version build: docker: Update GO-version * build: release: v1.21.0-rc2 build: release: v1.21.0-rc2 * Update go.mod Update go.mod * Update changelog for v1.21.0-rc2 Update changelog for v1.21.0-rc2 * perf: eth: gas estimate set applyTsMessages false (filecoin-project#10546) * have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization * set applyTsMessages = true in CallWithGas call in shed * update test with new callwithgas api optimization for eth call * Update chain/stmgr/call.go Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation --------- Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal> * feat: supply: drop genesis market locked funds * feat: supply: only grab genesis msig locks for writes * Update Changelog with backports Update Changelog with backports * Bump version in changelog Bump version in changelog * Add longer validation time cause to changelog Add longer validation time cause to changelog * refactor: stop using deprecated io/ioutil * Fix error when injecting StateManager * new actor bundle, fix ci, upgrade fx * fix fx version * upgrade go version circle ci * make linter happy :) * more go version fixing * replace deprecated mac build * removing support for macos build * Fix ctrl-c for GOOD --------- Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Jennifer Wang <jiayingw703@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Travis Person <travisperson@users.noreply.github.com> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Phi <orjan.roren@gmail.com> Co-authored-by: beck <1504068285@qq.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: hannahhoward <hannah@hannahhoward.net> Co-authored-by: Marten Seemann <martenseemann@gmail.com> Co-authored-by: Arsenii Petrovich <arsenii@protofire.io> Co-authored-by: Arsenii Petrovich <23522179+ArseniiPetrovich@users.noreply.github.com> Co-authored-by: Raúl Kripalani <raul@protocol.ai> Co-authored-by: ychiao <ychiaoli18@users.noreply.github.com> Co-authored-by: Travis Person <165274+travisperson@users.noreply.github.com> Co-authored-by: raulk <raul.kripalani@gmail.com> Co-authored-by: Peter Rabbitson <ribasushi@leporine.io> Co-authored-by: vyzo <vyzo@hackzen.org> Co-authored-by: Jorropo <jorropo.pgm@gmail.com> Co-authored-by: Mikers <snissn@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal> Co-authored-by: Henrique Moniz <1785239+hmoniz@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes
In an overzealous attempt to quash splitstore problems I introduced another bug in #10332. I was concerned that doing walk object incomplete on the ts refs would lead to uncontrolled graph traversal which would be a huge waste of resources and overall problem.
However after opening that PR I learned that this is not actually a problem because ts refs are not serialized as ipld (cbor tags with cid tag number 42), just cbor bytes. However by pinning these cids directly instead of using the incomplete walk I introduced a bug where warmup fails because lotus doesn't necessarily keep tsrefs around for all blocks.
This made it onto master because I went ahead and merged with red CI 😬 which was my bad. But note to our future selves this negligence was facilitated by the state flaky tests making CI untrustworthy, this is a great example of the cost of flaky tests.
Additional Info
Checklist
Before you mark the PR ready for review, please make sure that:
<PR type>: <area>: <change being made>fix: mempool: Introduce a cache for valid signaturesPR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, testarea, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps