Skip to content

Latest commit

 

History

History
1501 lines (1171 loc) · 98.9 KB

CHANGELOG.md

File metadata and controls

1501 lines (1171 loc) · 98.9 KB
Raw

Lotus changelog

1.5.2 / 2021-03-11

This is an hotfix release of Lotus that fixes a critical bug introduced in v1.5.1 in the miner windowPoSt logic. This upgrade is only affecting miner nodes.

Changes

1.5.1 / 2021-03-10

This is an optional release of Lotus that introduces an important fix to the WindowPoSt computation process. The change is to wait for some confidence before drawing beacon randomness for the proof. Without this, invalid proofs might be generated as the result of a null tipset.

Splitstore

This release also introduces the splitstore, a new optional blockstore that segregates the monolithic blockstore into cold and hot regions. The hot region contains objects from the last 4-5 finalities plus all reachable objects from two finalities away. All other objects are moved to the cold region using a compaction process that executes every finality, once 5 finalities have elapsed. The splitstore allows us to separate the two regions quite effectively, using two separate badger blockstores. The separation means that the live working set is much smaller, which results in potentially significant performance improvements. In addition, it means that the coldstore can be moved to a separate (bigger, slower, cheaper) disk without loss of performance. The design also allows us to use different implementations for the two blockstores; for example, an append-only blockstore could be used for coldstore and a faster memory mapped blockstore could be used for the hotstore (eg LMDB). We plan to experiment with these options in the future. Once the splitstore has been enabled, the existing monolithic blockstore becomes the coldstore. On the first head change notification, the splitstore will warm up the hotstore by copying all reachable objects from the current tipset into the hotstore. All new writes go into the hotstore, with the splitstore tracking the write epoch. Once 5 finalities have elapsed, and every finality thereafter, the splitstore compacts by moving cold objects into the coldstore. There is also experimental support for garbage collection, whereby unreachable objects are simply discarded. To enable the splitstore, add the following to config.toml:

[Chainstore]
  EnableSplitstore = true

Highlights

Other highlights include:

  • Improved deal data handling - now multiple deals can be adding to sectors in parallel
  • Rewriten sector pledging - it now actually cares about max sealing sector limits
  • Better handling for sectors stuck in the RecoverDealIDs state
  • lotus-miner sectors extend command
  • Optional configurable storage path size limit
  • Config to disable owner/worker fallback from control addresses (useful when owner is a key on a hardware wallet)
  • A write log for node metadata, which can be restored as a backup when the metadata leveldb becomes corrupted (e.g. when you run out of disk space / system crashes in some bad way)

Changes

1.5.0 / 2021-02-23

This is a mandatory release of Lotus that introduces the fifth upgrade to the Filecoin network. The network upgrade occurs at height 550321, before which time all nodes must have updated to this release (or later). At this height, v3 specs-actors will take effect, which in turn implements the following two FIPs:

  • FIP-0007 h/amt-v3 which improves the performance of the Filecoin HAMT and AMT.
  • FIP-0010 off-chain Window PoSt Verification which reduces the gas consumption of SubmitWindowedPoSt messages significantly by optimistically accepting Window PoSt proofs without verification, and allowing them to be disputed later by off-chain verifiers.

Note that the integration of v3 actors was already completed in 1.4.2, this upgrade simply sets the epoch for the upgrade to occur.

Disputer

FIP-0010 introduces the ability to dispute bad Window PoSts. Node operators are encouraged to run the new Lotus disputer alongside their Lotus daemons. For more information, see the announcement here.

Changes

  • #5341 Add a LOTUS_DISABLE_V3_ACTOR_MIGRATION envvar
    • Setting this envvar to 1 disables the v3 actor migration, should only be used in the event of a failed migration

1.4.2 / 2021-02-17

This is a large, and highly recommended, optional release with new features and improvements for lotus miner and deal-making UX. The release also integrates v3 specs-actors, which implements two FIPs:

  • FIP-0007 h/amt-v3 which improves the performance of the Filecoin HAMT and AMT.
  • FIP-0010 off-chain Window PoSt Verification which reduces the gas consumption of SubmitWindowedPoSt messages significantly by optimistically accepting Window PoSt proofs without verification, and allowing them to be disputed later by off-chain verifiers.

Note that this release does NOT set an upgrade epoch for v3 actors to take effect. That will be done in the upcoming 1.5.0 release.

New Features

  • #5341 Added sector termination API and CLI
    • Run lotus-miner sectors terminate
  • #5342 Added CLI for using a multisig wallet as miner's owner address
    • See how to set it up here
  • #5363, #5418, #5476, #5459 Integrated spec-actor v3
    • #5472 Generate actor v3 methods for pond
  • #5379 Added WindowPoSt disputer
  • #5309 Batch multiple deals in one PublishStorageMessages
    • #5411 Handle batch PublishStorageDeals message in sealing recovery
    • #5505 Exclude expired deals from batching in PublishStorageDeals messages
    • Added PublishMsgPeriod and MaxDealsPerPublishMsg to miner Dealmaking configuration. See how they work here.
    • #5538, #5549 Added a command to list pending deals and force publish messages.
      • Run lotus-miner market pending-publish
    • #5428 Moved waiting for PublishStorageDeals messages' receipt from markets to lotus
  • #5510 Added nerpanet build option
    • To build nerpanet, run make nerpanet
  • #5433 Added AlwaysKeepUnsealedCopy option to the miner configuration
  • #5520 Added MsigGetPending to get pending transactions for multisig wallets
  • #5219 Added interactive mode for lotus-wallet
  • 5529 Added support for minder nodes in lotus-shed rpc util

Bug Fixes

  • #5210 Miner should not dial client on restart
  • #5403 When estimating GasLimit only apply prior messages up to the nonce
  • #5410 Fix the calibnet build option
  • #5492 Fixed has for ipfsbstore for non-existing blocks
  • #5361 Fixed retrieval hangs when using IpfsOnlineMode=true
  • #5493 Fixed retrieval failure when price-per-byte is zero
  • #5506 Fixed contexts in the storage adpater
  • #5515 Properly wire up StateReadState on gateway API
  • #5582 Fixed error logging format strings
  • #5614 Fixed websocket reconnecting handling

Improvements

  • #5389 Show verified indicator for ./lotus-miner storage-deals list
  • #5229 Show power for verified deals in ./lotus-miner setocr list
  • #5407 Added explicit check of the miner address protocol
  • #5399 watchdog: increase heapprof capture threshold to 90%
  • #5398 storageadapter: Look at precommits on-chain since deal publish msg
  • #5470 Added --no-timing option for ./lotus state compute-state --html
  • #5417 Storage Manager: Always unseal full sectors
  • #5393 Switched to filecoin-ffi bls api for bls signatures
  • #5380 Refactor deals API tests
  • #5397 Fixed a flake in the sync manager edge case test
  • #5406 Added a test to ensure a correct window post cannot be disputed
  • #5294 Added jobs to build Lotus docker image and push it to AWS ECR
  • #5387 Added network info(mainnet|calibnet) in version
  • #5497 Export metric for lotus-gateaway
  • #4950 Removed bench policy
  • #5047 Improved the UX for ./lotus-shed bitfield enc
  • #5282 Snake a context through the chian blockstore creation
  • #5350 Avoid using mp.cfg directrly to prevent race condition
  • #5449 Documented the block-header better
  • #5404 Added retrying proofs if an incorrect one is generated
  • #4545 Made state tipset usage consistent in the API
  • #5540 Removed unnecessary database reads in validation check
  • #5554 Fixed build lotus-soup CI job
  • #5552 Updated CircleCI to halt gracefully
  • #5555 Cleanup and add docstrings of node builder
  • #5564 Stopped depending on gocheck with gomod
  • #5574 Updated CLI UI
  • #5570 Added code CID to StateReadState return object
  • #5565 Added storageadapter.PublishMsgConfig to miner in testkit for lotus-soup testplan
  • #5571 Added lotus-seed gensis car to generate lotus block for devnets
  • #5613 Check format in client commP util
  • #5507 Refactored coalescing logic into its own function and take both cancellation sets into account
  • #5592 Verify FFI version before building

Dependency Updates

Network Version v10 Upgrade

  • #5473 Merged staging branch for v1.5.0
  • #5603 Set nerpanet's upgrade epochs up to v3 actors
  • #5471, #5456 Set calibration net actor v3 migration epochs for testing
  • #5434 Implemented pre-migration framework
  • #5476 Tune migration

1.4.1 / 2021-01-20

This is an optional Lotus release that introduces various improvements to the sealing, mining, and deal-making processes. In particular, #5341 introduces the ability for Lotus miners to terminate sectors.

Changes

Core Lotus

Sealing

Dealmaking

UX

Testing

Build & Networks

Management

1.4.0 / 2020-12-19

This is a MANDATORY hotfix release of Lotus that resolves a chain halt at height 336,459 caused by nondeterminism in specs-actors. The fix is to update actors to 2.3.3 in order to incorporate this fix filecoin-project/specs-actors#1334.

1.3.0 / 2020-12-16

This is a mandatory release of Lotus that introduces the third post-liftoff upgrade to the Filecoin network. The network upgrade occurs at height 343200, before which time all nodes must have updated to this release (or later). The change that breaks consensus is an implementation of FIP-0009(https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0009.md).

Changes

1.2.3 / 2020-12-15

This is an optional Lotus release that introduces many performance improvements, bugfixes, and UX improvements.

Changes

1.2.2 / 2020-12-03

This is an optional Lotus release that introduces various improvements to the mining logic and deal-making workflow, as well as several new UX features.

Changes

1.2.1 / 2020-11-20

This is a very small release of Lotus that fixes an issue users are experiencing when importing snapshots. There is no need to upgrade unless you experience an issue with creating a new datastore directory in the Lotus repo.

Changes

1.2.0 / 2020-11-18

This is a mandatory release of Lotus that introduces the second post-liftoff upgrade to the Filecoin network. The network upgrade occurs at height 265200, before which time all nodes must have updated to this release (or later). This release also bumps the required version of Go to 1.15.

The changes that break consensus are:

Other Changes

1.1.3 / 2020-11-13

This is an optional release of Epik that upgrades Epik dependencies, and includes many performance enhancements, bugfixes, and UX improvements.

Highlights

Changes

Core Epik

Mining

Markets and Data Transfer

UX

Testing and Tooling

Dependencies

1.1.2 / 2020-10-24

This is a patch release of Epik that builds on the fixes involving worker keys that was introduced in v1.1.1. Miners and node operators should update to this release as soon as possible in order to ensure their blocks are propagated and validated.

Changes

1.1.1 / 2020-10-24

This is a patch release of Epik that addresses some issues caused by when miners change their worker keys. Miners and node operators should update to this release as soon as possible, especially any miner who has changed their worker key recently.

Changes

1.1.0 / 2020-10-20

This is a mandatory release that introduces the first post-liftoff upgrade to the Filecoin network. The changes that break consensus are an upgrade to specs-actors v2.2.0 at epoch 170000.

Changes

1.0.0 / 2020-10-19

It's 1.0.0! This is an optional release of Epik that introduces some UX improvements to the 0.10 series.

This very small release is largely cosmetic, and intended to flag the code that the Filecoin mainnet was launched with.

API changes

  • StateMsgGasCost has been removed. The equivalent information can be gained by calling StateReplay.
  • A GasCost field has been added to the InvocResult type, meaning detailed gas costs will be returned when calling StateReplay, StateCompute, and StateCall.
  • The behaviour of StateReplay in response to an empty tipset key has been changed. Instead of simply using the heaviest tipset (which is almost guaranteed to be an unsuccessful replay), we search now search the chain for the tipset that included the message, and replay the message in that tipset (we fail if no such tipset is found).

Changes

0.10.2 / 2020-10-14

This is an optional release of Epik that updates markets to 0.9.1, which fixes an issue affecting deals that were mid-transfer when the node was upgraded to 0.9.0. This release also includes some tweaks to default gas values and minor performance improvements.

Changes

0.10.1 / 2020-10-14

This is an optional release of Epik that updates markets to 0.9.0, which adds the ability to restart data transfers. This release also introduces Ledger support, and various UX improvements.

Changes

0.10.0 / 2020-10-12

This is a consensus-breaking hotfix that addresses an issue in specs-actors v2.0.3 that made it impossible to pledge new 32GiB sectors. The change in Epik is to update to actors v2.1.0, behind the new network version 5.

Changes

0.9.1 / 2020-10-10

This release fixes an issue which may cause the actors v2 migration to compute the state incorrectly when more than one migration is running in parallel.

Changes

0.9.0 / 2020-10-07

This consensus-breaking release of Epik upgrades the actors version to v2.0.0. This requires migrating actor state from v0 to v2. The changes that break consensus are:

Changes

Dependencies

Core Epik

Mining

UX

Tooling and validation

Contributors

The following contributors had 5 or more commits go into this release. We are grateful for every contribution!

Contributor Commits Lines ±
Stebalien 84 +3425/-2287
magik6k 41 +2121/-506
arajasek 39 +2467/-424
Kubuxu 25 +2344/-775
raulk 21 +287/-196
whyrusleeping 13 +727/-71
hsanjuan 13 +5886/-7956
dirkmc 11 +2634/-576
travisperson 8 +923/-202
ribasushi 6 +188/-128
zgfzgf 5 +21/-17

0.8.1 / 2020-09-30

This optional release of Epik introduces a new version of markets which switches to CBOR-map encodings, and allows datastore migrations. The release also introduces several improvements to the mining process, a few performance optimizations, and a battery of UX additions and enhancements.

Changes

Dependencies

Core Epik

Performance

Mining

UX

Tooling

Bootstrap

0.8.0 / 2020-09-26

This consensus-breaking release of Epik introduces an upgrade to the network. The changes that break consensus are:

  • Upgrading to specs-actors v0.9.11, which reduces WindowPoSt faults per FIP 0002 to reduce cost for honest miners with occasional faults (see filecoin-project/specs-actors#1181)
  • Revisions to some cryptoeconomics and network params

This release also updates go-fil-markets to fix an incompatibility issue between v0.7.2 and earlier versions.

Changes

Dependencies

Core Epik

UX

Tooling and validation

Contributors

The following contributors had 5 or more commits go into this release. We are grateful for every contribution!

Contributor Commits Lines ±
arajasek 66 +3140/-1261
Stebalien 64 +3797/-3434
magik6k 48 +1892/-976
raulk 40 +2412/-1549
vyzo 22 +287/-196
alanshaw 15 +761/-146
whyrusleeping 15 +736/-52
hannahhoward 14 +1237/-837
anton 6 +32/-8
travisperson 5 +502/-6
Frank 5 +78/-39
Jennifer 5 +148/-41

0.7.2 / 2020-09-23

This optional release of Epik introduces a major refactor around how a Epik node interacts with code from the specs-actors repo. We now use interfaces to read the state of actors, which is required to be able to reason about different versions of actors code at the same time.

Additionally, this release introduces various improvements to the sync process, as well as changes to better the overall UX experience.

Changes

Core Epik

User Experience

Testing and validation

0.7.1 / 2020-09-17

This optional release of Epik introduces some critical fixes to the window PoSt process. It also upgrades some core dependencies, and introduces many improvements to the mining process, deal-making cycle, and overall User Experience.

Changes

Some notable improvements:

Dependencies

0.7.0 / 2020-09-10

This consensus-breaking release of Epik is designed to test a network upgrade on the space race testnet. The changes that break consensus are:

  • Upgrading the Drand network used from the test Drand network to the League of Entropy main drand network. This is the same Drand network that will be used in the Filecoin mainnet.
  • Upgrading to specs-actors v0.9.8, which adds a new method to the Multisig actor.

Changes

Core Epik

Storage Miner

Message pool

Maintenance

Contributors

The following contributors had commits go into this release. We are grateful for every contribution!

Contributor Commits Lines ±
arajasek 28 +1144/-239
Kubuxu 19 +452/-261
whyrusleeping 13 +456/-87
vyzo 11 +318/-20
raulk 10 +1289/-350
magik6k 6 +188/-55
dirkmc 3 +31/-8
alanshaw 3 +176/-37
Stebalien 2 +9/-12
lanzafame 1 +1/-1
frrist 1 +1/-1
mishmosh 1 +1/-1
nonsense 1 +1/-0

0.6.2 / 2020-09-09

This release introduces some critical fixes to message selection and gas estimation logic. It also adds the ability for nodes to mark a certain tipset as checkpointed, as well as various minor improvements and bugfixes.

Changes

Messagepool

Core Epik

Storage

Maintenance

0.6.1 / 2020-09-08

This optional release introduces a minor improvement to the sync process, ensuring nodes don't fall behind and then resync.

Changes

0.6.0 / 2020-09-07

This consensus-breaking release of Epik is designed to test a network upgrade on the space race testnet. The changes that break consensus are:

This release also introduces many improvements to Epik! Among them are a new version of go-fil-markets that supports non-blocking retrieval, various spam reduction measures in the messagepool and p2p logic, and UX improvements to payment channels, dealmaking, and state inspection.

Changes

Core Epik and dependencies

Messagepool

Payment channels

Peer-to-peer

Miscellaneous

UX

Testing & tooling

0.5.10 / 2020-09-03

This patch includes a crucial fix to the message pool selection logic, strongly disfavouring messages that might cause a miner penalty.

Changes

0.5.9 / 2020-09-03

This patch includes a hotfix to the GasEstimateFeeCap method, capping the estimated fee to a reasonable level by default.

Changes

0.5.8 / 2020-09-02

This patch includes some bugfixes to the sector sealing process, and updates go-fil-markets. It also improves the performance of blocksync, adds a method to export chain state trees, and improves chainwatch.

Changes

0.5.7 / 2020-08-31

This patch release includes some bugfixes and enhancements to the sector lifecycle and message pool logic.

Changes

0.5.6 / 2020-08-29

Hotfix release that fixes a panic in the sealing scheduler (https://github.com/EpiK-Protocol/go-epik/pull/3389).

0.5.5

This patch release introduces a large number of improvements to the sealing process. It also updates go-fil-markets to version 0.5.8, and go-libp2p-pubsub to v0.3.5.

Downstream upgrades

Sector sealing

  • The following improvements were introduced in https://github.com/EpiK-Protocol/go-epik/pull/3350.

    • Allow epik-miner sectors remove to remove a sector in any state.
    • Create a separate state in the storage FSM dedicated to submitting the Commit message.
    • Recovery for when the Deal IDs of deals in a sector get changed in a reorg.
    • Auto-retry sending Precommit and Commit messages if they run out of gas
    • Auto-retry sector remove tasks when they fail
    • Compact worker windows, and allow their tasks to be executed in any order

  • Don't simply skip PoSt for bad sectors (https://github.com/EpiK-Protocol/go-epik/pull/3323)

Message Pool

Chainwatch

0.5.4

A patch release, containing a few nice bugfixes and improvements:

  • Fix parsing of peer ID in epik-miner actor set-peer-id (@whyrusleeping)
  • Update dependencies, fixing several bugs (@Stebalien)
  • Fix remaining linter warnings (@Stebalien)
  • Use safe string truncation (@Ingar)
  • Allow tweaking of blocksync message window size (@whyrusleeping)
  • Add some additional gas stats to metrics (@Kubuxu)
  • Fix an edge case bug in message selection, add many tests (@vyzo)

0.5.3

Yet another hotfix release. A lesson for readers, having people who have been awake for 12+ hours review your hotfix PR is not a good idea. Find someone who has enough slept recently enough to give you good code review, otherwise you'll end up quickly bumping versions again.

  • Fixed a bug in the mempool that was introduced in v0.5.2

0.5.2 / 2020-08-24

This is a hotfix release.

  • Fix message selection to not include messages that are invalid for block inclusion.
  • Improve SelectMessage handling of the case where the message pools tipset differs from our mining base.

0.5.1 / 2020-08-24

The Space Race release! This release contains the genesis car file and bootstrap peers for the space race network.

Additionally, we included two small fixes to genesis creation:

  • Randomize ticket value in genesis generation
  • Correctly set t099 (burnt funds actor) to have valid account actor state

0.5.0 / 2020-08-20

This version of Epik will be used for the incentivized testnet Space Race competition, and can be considered mainnet-ready code. It includes some protocol changes, upgrades of core dependencies, and various bugfixes and UX/performance improvements.

Highlights

Among the highlights included in this release are:

  • Gas changes: We implemented EIP-1559 and introduced real gas values.
  • Deal-making: We now support "Committed Capacity" sectors, "fast-retrieval" deals, and the packing of multiple deals into a single sector.
  • Renamed features: We renamed some of the binaries, environment variables, and default paths associated with a Epik node.

Gas changes

We made some significant changes to the mechanics of gas in this release.

Network fee

We implemented something similar to Ethereum's EIP-1559. The Message structure had three changes:

  • The GasPrice field has been removed
  • A new GasFeeCap field has been added, which controls the maximum cost the sender incurs for the message
  • A new GasPremium field has been added, which controls the reward a miner earns for including the message

A sender will never be charged more than GasFeeCap * GasLimit. A miner will typically earn GasPremium * GasLimit as a reward.

The Blockheader structure has one new field, called ParentBaseFee. Informally speaking,the ParentBaseFee is increased when blocks are densely packed with messages, and decreased otherwise.

The ParentBaseFee is used when calculating how much a sender burns when executing a message. Burning simply refers to sending attoEPK to a dedicated, unreachable account. A message causes ParentBaseFee * GasUsed attoEPK to be burnt.

Real gas values

This release also includes our first "real" gas costs for primitive operations. The costs were designed to account for both the time that message execution takes, as well as the space a message adds to the state tree.

Deal-making changes

There are three key changes to the deal-making process.

Committed Capacity sectors

Miners can now pledge "Committed Capacity" (CC) sectors, which are explicitly stated as containing junk data, and must not include any deals. Miners can do this to increase their storage power, and win block rewards from this pledged storage.

They can mark these sectors as "upgradable" with epik-miner sectors mark-for-upgrade. If the miner receives and accepts one or more storage deals, the sector that includes those deals will replace the CC sector. This is intended to maximize the amount of useful storage on the Filecoin network.

Fast-retrieval deals

Clients can now include a fast-retrieval flag when proposing deals with storage miners. If set to true, the miner will include an extra copy of the deal data. This data can be quickly served in a retrieval deal, since it will not need to be unsealed.

Multiple deals per sector

Miners can now pack multiple deals into a single sector, so long as all the deals fit into the sector capacity. This should increase the packing efficiency of miners.

Renamed features

To improve the user experience, we updated several names to mainatin standard prefixing, and to better reflect the meaning of the features being referenced.

In particular, the Epik miner binary is now called epik-miner, the default path for miner data is now ~/.epikminer, and the environment variable that sets the path for miner data is now $EPIK_MINER_PATH. A full list of renamed features can be found here.

Changelog

Downstream upgrades

Core protocol

Deal-making lifecycle

Enhancements

UX

Contributors

The following contributors had 10 or more commits go into this release. We are grateful for every contribution!

Contributor Commits Lines ±
magik6k 361 +13197/-6136
Kubuxu 227 +5670/-2587
arajasek 120 +2916/-1264
whyrusleeping 112 +3979/-1089
vyzo 99 +3343/-1305
dirkmc 68 +8732/-3621
laser 45 +1489/-501
hannahhoward 43 +2654/-990
frrist 37 +6630/-4338
schomatis 28 +3016/-1368
placer14 27 +824/-350
raulk 25 +28718/-29849
mrsmkl 22 +560/-368
travisperson 18 +1354/-314
nonsense 16 +2956/-2842
ingar 13 +331/-123
daviddias 11 +311/-11
Stebalien 11 +1204/-980
RobQuistNL 10 +69/-74

0.1.0 / 2019-12-11

We are very excited to release epik 0.1.0. This is our testnet release. To install epik and join the testnet, please visit lotu.sh. Please file bug reports as issues.

A huge thank you to all contributors for this testnet release!