EXPERIMENTAL Tree States v4.2.990

Disclaimer

⚠️ You should not run this alpha release supporting mainnet validators ⚠️

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Summary

This is an alpha release of upcoming changes to Lighthouse which improve disk usage and state management.

For more information about this family of releases, please see the v3.4.0-tree.1 release notes.

Compared to the previous release the main changes are:

• Capella support
• Freezer database rewritten to use hierarchical diffs (even lower disk usage, around 100 GiB for an archive node)

⚠️ Backwards Compatibility ⚠️

This release is not backwards compatible with stable Lighthouse or prior tree-states releases. It uses a different database schema (v23) for which no automatic upgrade or downgrade is implemented. We intend to implement an automatic upgrade process once the new schema is finalized. A re-sync will be required to run future versions of tree-states.

Please only run this release if you are willing to re-sync now, and again in several weeks/months.

There are several breaking changes to CLI flags:

• The secret dummy flag --unsafe-and-dangerous-mode must be supplied. This is to prevent accidental upgrades to the release by users who don't know what they're getting themselves into. Unfortunately we found that far too many people accidentally ran previous tree-states releases on mainnet.
• The flags --slots-per-restore-point and --historic-state-cache-size are deprecated and no longer necessary due to the freezer database using hierarchical diffs.

Known Issues

Expect a few sharp edges. Some things you may run into:

• aarch64 (ARM) support is currently broken. As a result there are no binaries for ARM.
• The Docker image on DockerHub is for x86_64/modern CPUs only. It was built manually to work around the ARM issue.
• WARN Parent state is not advanced is logged excessively during sync. This is harmless, albeit annoying.
• Block processing is a bit slower than stable Lighthouse.

If you observe any panics, crashes or major performance regressions, please open an issue!

Building from source

Build the v4.2.990-exp tag.

Binaries

See pre-built binaries documentation.

A Docker image for x86_64 only is available on DockerHub as sigp/lighthouse:v4.2.990.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

Fast Aging Alien

Summary

This is a medium-priority release for Mainnet users. It contains important improvements to the resilience of Lighthouse, however there is no immediate urgency to upgrade.

Users who connect their VC to multiple BNs will have reduced risk of missed attestations and orphaned blocks when there is an offline EE (#4295).

Support for "proposer-only" BNs has been added to Lighthouse. Although we don't recommend this functionality for most stakers, it may be useful for enterprise operations or in the case of a large-scale attack on Ethereum. Read about this advanced feature in the documentation.

This release contains many notable features, optimisations and bug fixes:

• Add a lighthouse/logs SSE endpoint to the VC API to expose logging (#4074)
• Update Gnosis Capella preset (#4302)
• Update Sepolia bootnodes (#4297)
• Prioritise the head state in the shuffling cache (#4296)
• Improve VC fallback behaviour when an EE is offline (#4295)
• Add a flag for storing invalid blocks as SSZ files (#4194)
• Fix bug with attestation verification on fork boundaries (#4263)
• Reduce bandwidth over the VC<>BN API using dependant roots (#4170)
• Fix bug when computing withdrawals root (#4249)
• Backfill blocks only to the WSP by default (#4082)
• Massively improve historic state lookup performance via the API (#4228)
• Enable build profiles for Docker source builds (#4237)
• Prevent disconnection from trusted peers (#4159)
• Provide support for "proposer-only" BNs to mitigate targeted DoS attacks (#4182)
• Add attestation inclusion distance to the HTTP API (#4148)

Breaking Changes

Breaking Change: Database Schema V17

To reduce complexity in the fork choice algorithm, the database schema has been upgraded to v17 (#4265). The schema upgrade will be applied automatically upon upgrading and should not take more than a few seconds.

To downgrade, follow the instructions in the book for Database Migrations.

Users may downgrade at any time.

Breaking Change: Backfilling

After a checkpoint sync, Lighthouse will only backfill to the most recent weak subjectivity checkpoint unless the --genesis-backfill flag is provided (#4082). The previous behaviour was to always backfill to the genesis block.

The --genesis-backfill flag is not required for staking. Nodes performing "archive" duties may wish to add this flag so they may download and store the entire history of the chain.

Omitting the --genesis-backfill flag for new BN databases will result in reduced disk usage and reduced network bandwidth during sync. Nodes which are already synced to genesis will maintain their history, even if the --genesis-backfill flag is not present. In a future release we will enable existing nodes to prune old blocks from their database so that they can achieve reduced disk usage without a re-sync.

Breaking Change: Graceful Handling of Offline EEs

The Lighthouse VC will now detect when the BN indicates that its EE (e.g., Geth, Nethermind, Besu, Erigon, etc) is offline. The VC will then prefer a BN with an online EE (#4295).

This functionality is only possible when the VC has multiple (aka "fallback") BNs provided via the --beacon-nodes flag.

If there is no alternative BN with an online EE, the VC will go back to the previous behaviour of attempting to perform its duties regardless of the EE status.

Breaking Change: Updated LLVM Version

Windows users may fail to compile Lighthouse if their LLVM version is earlier than 15.0 (#4220). Upgrading can typically be achieved with choco upgrade llvm.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Medium	Medium
Non-Staking Users	Medium	---

See Update Priorities more information about this table.

The Beacon Node may be updated without also updating the Validator Client. However we recommend upgrading both components for consistency.

All Changes

• v4.2.0 (#4309)
• Logging via the HTTP API (#4074)
• Avoid excessive logging of BN online status (#4315)
• Update gnosis capella preset (#4302)
• add more sepolia bootnodes (#4297)
• Add more metrics for tracking sync messages (#4308)
• Enshrine head state shuffling in the shuffling_cache (#4296)
• Implement el_offline and use it in the VC (#4295)
• Fix PERSIST_ETH1_CACHE / PERSIST_OP_POOL Metrics (#4278)
• Post merge local testnets (#3807)
• Remove redundant gossipsub tests (#4294)
• Don't requeue already-known RPC blocks (#4214)
• Add a flag for storing invalid blocks (#4194)
• Replace ganache-cli with anvil (#3555)
• DB migration for fork choice cleanup (#4265)
• Attestation verification uses head state fork (#4263)
• Reduce bandwidth over the VC<>BN API using dependant roots (#4170)
• Update links in Lighthouse Book (#4279)
• Fix attestation withdrawals root mismatch (#4249)
• Fix log on initializing external block builder (#4267)
• Add branching instructions to the contribution guide (#4272)
• Do not attempt to resubscribe to core topics (#4271)
• Backfill blocks only to the WSP by default (#4082)
• improve api-bn.md (#4244)
• use state cache to optimise historical state lookup (#4228)
• Update Lighthouse book Sec 3-6 and FAQ (#4221)
• simplify calculate_committee_fraction (#4213)
• Update igd to v0.12.1 (#4257)
• Un-deprecate test utils functions such as extend_chain (#4255)
• Enable build profiles for Docker source builds (#4237)
• Maintain trusted peers (#4159)
• Prepare CI for merge queues (#4252)
• remove withdrawal warning (#4207)
• Logfile no restrict help text for windows (#4165)
• Feat: lighthouse book - ui authentication (#4232)
• Split common crates out into their own repos (#3890)
• Separate BN for block proposals (#4182)
• add attestation inclusion distance in http api (#4148)
• Fix Rust 1.69 lints (#4222)
• Update LLVM version to 15.0 in CI workflows (#4220)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.2.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.2.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.2.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.2.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.2.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.2.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.2.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.2.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.2.0	sigp/lighthouse

Uncle Steve

Summary

This is a high-priority release for Mainnet users. It fixes an issue which caused excessive CPU usage shortly after the Mainnet Capella upgrade (#4183).

Users on v4.0.1 or earlier should upgrade as soon as possible. This release is less important for users already running v4.0.2-rc.0; that release already contains an effective fix. However the fix in this release is more accurate so we recommend that v4.0.2-rc.0 users upgrade to this release at their next convenience.

We expect the conditions that caused excessive CPU usage are unlikely to return, however we recommend users upgrade to bolster the resilience of the Ethereum network.

This release also contains new improvements that were not present in v4.0.1 or v4.0.2-rc.0:

• The /eth/v1/debug/fork_choice HTTP endpoint was added (#4003)
• The PATCH lighthouse/validators/{pubkey} endpoint has been optimised (#4126)
• The HTTP API has been updated to include references to the finalized checkpoint (#3753)
• Backfilling has been rate-limited to mitigate validator performance issues after a recent checkpoint sync. The old behaviour can be obtained with the --disable-backfill-rate-limiting flag (#3936)
• The Lighthouse VC can now create signed voluntary exit messages via the HTTP API (#4119)
• The --disable-peer-scoring flag has been added for developer usage (#4135)
• The proposer boost re-orging strategy is now configurable and more cautious by default (#4151)
• Payload reconstruction via the HTTP API now uses the dedicated EE endpoint, addressing an issue with Erigon compatibility (#4102)
• Lighthouse now supplies a user-agent to builders (#4199)

With this release we are also encouraging users to re-enable mev-boost and re-orgs if they had previously disabled them. The latency issues with the MEV relays have resolved, and re-orgs should be more reliable now that improvements to gossip have been rolled out by other clients. If you experience a missed block or failed re-org running Lighthouse v4.1.0 please reach out.

Compiling from Source

Users who compile from source should ensure they update Rust and checkout and pull the stable branch before building:

• Most users can update Rust with rustup update
• Checkout and pull the stable branch with git checkout stable && git pull

Breaking Changes

Breaking Change: Minimum Supported Rust Version 1.68.2

The minimum supported Rust version has been set to 1.68.2 to address a serious bug in prior Rust releases (see #4204 for more information).

Users who compile from source (i.e., not Docker or pre-built binary users) will receive the following error if they are using an earlier version of Rust:

lighthouse v4.1.0 (/home/karlm/lighthouse/lighthouse)` cannot be built because it requires rustc 1.68.2 or newer

Users can typically obtain the latest version of Rust by running rustup update.

Breaking Change: New Fields Added to HTTP API Responses

The HTTP API has been updated to include references to the finalized checkpoint (#3753).

This aligns Lighthouse with the standard BeaconAPIs, which added these fields in ethereum/beacon-APIs#254.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	High	Low
Non-Staking Users	Medium	---

See Update Priorities more information about this table.

The Beacon Node may be updated without also updating the Validator Client. However we recommend upgrading both components for consistency.

All Changes

• Release v4.1.0 (#4191)
• Check lateness of block before requeuing it (#4208)
• Bump Rust version (MSRV) (#4204)
• Set user agent on requests to builder (#4199)
• Use efficient payload reconstruction for HTTP API (#4102)
• Address observed proposers behaviour (#4192)
• Use head state for exit verification (#4183)
• Update Lighthouse book and some FAQs (#4178)
• Avoid processing redundant RPC blocks (#4179)
• Make re-org strat more cautious and add more config (#4151)
• Remove Redundant Trait Bound (#4169)
• Remove the unused ExecutionOptimisticForkVersionedResponse type (#4160)
• Add a flag to disable peer scoring (#4135)
• CI fix: move download web3signer binary out of build script (#4163)
• Add beacon.watch (#3362)
• remove dup log (#4155)
• Ban peer race condition (#4140)
• Add new validator API for voluntary exit (#4119)
• Rate limiting backfill sync (#3936)
• Update database-migrations.md (#4149)
• update README of local_testnet (#4114)
• Test failing CI tests due to port conflicts (#4134)
• Add finalized to HTTP API responses (#3753)
• Correct log for ENR (#4133)
• Optimise update_validators by decrypting key cache only when necessary (#4126)
• Add debug fork choice api (#4003)
• improve error message (#4141)
• Update Rust version in lcli Dockerfile (#4121)
• Update arbitrary (#4139)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.1.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.1.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.1.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.1.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.1.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.1.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.1.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.1.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.1.0	sigp/lighthouse

Goddess Beth

Summary

This high-priority hot-fix address high CPU usage experienced after the Capella upgrade (April 12 UTC).

We recommend all mainnet users update to this release to prevent high CPU usage from interfering with normal node operations (e.g., staking).

This hot-fix only applies to the Beacon Node. There is no need to upgrade the Validator Client from v4.0.1 (or v4.0.1-rc.0) to this release.

Building from Source

This hot-fix is not yet applied to the unstable or stable branches. To build from source, use the hotfix-exit-verification branch (or checkout the v4.0.2-rc.0 tag).

Breaking Changes

There are no breaking changes in this release.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	High	Low
Non-Staking Users	High	---

See Update Priorities more information about this table.

All Changes

• Bump versions
• Empty commit
• Use head state for exit verification

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.2-rc.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.2-rc.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.2-rc.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.0.2-rc.0	sigp/lighthouse

Campaign Manager Morty

Summary

This release is mandatory for all mainnet users (except for those running v4.0.1-rc.0). It enables the Capella/Shanghai ("Shapella") upgrade (#4111), which will occur at April 12, 2023, 10:27:35pm UTC. Any node which is not updated to Lighthouse a v4.x.x release before 10:27pm on April 12th (UTC) will stop following the chain and will need to be resynced. For stakers, this would result in missed rewards and penalties.

For clarity, all mainnet Lighthouse users must be running a v4.x.x release on their BNs and VCs by April 12, 2023, 10:27:35pm UTC.

This release also contains various bug-fixes, optimisations and new features:

• Improve the resilience of the fork choice algorithm (#3962)
• Add a flag to speed up responses to the committees HTTP API (#4081)
• Improve payload reconstruction by utilising a new execution engine API endpoint (#4028)
• Reduce false-positive ERRO logs claiming that builder blocks were published late (#4073)
• Fix a bug that resulted in a harmless ERRO Dialing an already dialing peer log (#4056)
• Add support for IPv6 (#4046)

v4.0.0 Release Retracted

This release replaces the v4.0.0 release which was retracted due to the discovery of a bug. A v4.0.1-rc.0 hot-fix was published to patch that bug. That bug fix is also included in this release.

This release (v4.0.1) replaces both v4.0.0 and v4.0.1-rc.0.

We have the following advice for users:

• Any node running v4.0.0 should immediately upgrade to this release since v4.0.0 is not reliable.
• Any node running v4.0.1-rc.0 may choose to continue on that release or upgrade to this release. This release and the rc.0 release are functionally identical.
• Any node running v3.5.1 or earlier should upgrade to this release at their next convenience (certainly before the Capella upgrade).

These release notes have been written with respect to v3.5.1 rather than the previous retracted release of v4.0.0.

Mainnet Capella/Shanghai ("Shapella") Upgrade

The Capella/Shanghai ("Shapella") upgrade will occur on mainnet at:

• Epoch 194048
• April 12, 2023, 10:27:35pm UTC

All Lighthouse Beacon Nodes and Validator Clients must be upgraded to a reliable v4.x.x release to ensure they follow the correct chain. The following table demonstrates which releases are reliable, Shapella-ready releases:

Release	Is Reliable, Shapella-Ready v4.x.x Release
v3.5.1	❌ No
v4.0.0	❌ No
v4.0.1-rc.0	✅ Yes
v4.0.1	✅ Yes
Any release after v4.0.1	✅ Yes

Preparation for the Shapella upgrade is much simpler than the preparation required for "The Merge" (Bellatrix). To be Shapella ready, users just need to:

• Upgrade their Lighthouse BN(s) to a v4.x.x release.
• Upgrade their Lighthouse VC(s) to v4.x.x release.
• Upgrade their Execution Engine(s) to a Shanghai-ready release:
  • Besu: 23.1.2 or later
  • Geth: v1.11.5 or later
  • Nethermind: v1.17.3 or later
  • Erigon: v2.42.0 or later

If your execution engine does not yet have a Shanghai-ready release then it is safe to upgrade Lighthouse to v4.x.x without also upgrading the execution engine. An up-to-date execution engine will be required before April 12th, though.

There are no new flags to be added or removed for the Shapella upgrade, simply upgrade and wait.

Lighthouse will start periodically emitting the following logs two weeks before the Shapella upgrade (29th of March):

• Not ready for Capella if Lighthouse has detected that the execution engine is too outdated to support Shanghai.
• Ready for Capella if Lighthouse has detected a modern execution engine release.

Just because Lighthouse is logging Ready for Capella does not indicate that your execution engine is on the correct version. There is no way for Lighthouse to determine this exactly and users are responsible for ensuring that their execution engine is using the latest release.

🐛 Known Issues 🐛

Fixed as of Erigon v2.42.0

There was a bug in Erigon v2.41.0 and earlier which caused Erigon to time out when paired with Lighthouse v4.0.1. This was reflected in the Lighthouse logs as an error log containing the phrase TimedOut, like this:

ERRO Error during execution engine upcheck error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec

The issue is resolved in Erigon v2.42.0 and later, see this issue for details: erigontech/erigon#7172.

Breaking Changes

There are no breaking changes between this release and v4.0.0 and v4.0.1-rc.0. The following changes are with reference to v3.5.1.

Breaking Change: Database Schema V16

To support changes to the fork choice algorithm, the database schema has been upgraded to v16. The schema upgrade will be applied automatically upon upgrading and should not take more than a few seconds.

To downgrade, follow the instructions in the book for Database Migrations.

Users may downgrade anytime before the Capella upgrade. Prior Lighthouse releases are not compatible with Capella, so downgrading is fundamentally impossible after the upgrade occurs.

Breaking Change: Minimum Supported Rust Version 1.66

The minimum supported Rust version has been set to 1.66. Users who compile from source (i.e., not Docker or pre-built binary users) will receive the following error if they are using an earlier version of Rust:

lighthouse v4.0.0 (/home/karlm/lighthouse/lighthouse)` cannot be built because it requires rustc 1.66 or newer

Users can typically obtain the latest version of Rust by running rustup update.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	High	High
Non-Staking Users	High	---

See Update Priorities more information about this table.

All Changes

• Release v4.0.1 (#4125)
• Release Candidate v4.0.1-rc.0 (#4123)
• Fix fork choice error message (#4122)

• Release v4.0.0 (#4112)
• Fork choice modifications and cleanup (#3962)
• Set Capella fork epoch for Mainnet (#4111)
• Reduce verbosity of reprocess queue logs (#4101)
• Customisable shuffling cache size (#4081)
• Improve Lighthouse Connectivity Via ENR TCP Update (#4057)
• Ignore self as a bootnode (#4110)
• Reconstruct Payloads using Payload Bodies Methods (#4028)
• Clarify "Ready for Capella" (#4095)
• Reduce false positive logging for late builder blocks (#4073)
• Make more noise when the EL is broken (#3986)
• Siren Ui Lighthouse Version Requirments (#4093)
• Correct a race condition when dialing peers (#4056)
• Remove Router/Processor Code (#4002)
• Complete match for has_context_bytes (#3972)
• Add parent_block_number to payload SSE (#4053)
• Support for Ipv6 (#4046)
• Correct /lighthouse/nat implementation (#4069)
• Added warning when new jwt is generated (#4000)
• Appease Clippy 1.68 and refactor http_api (#4068)
• Fix order of arguments to log_count (#4060)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
<img src="https://...

Tophat Jones

🚨 Hot-Fix for v4.0.0 🚨

This release is a hot-fix to patch a bug introduced in v4.0.0, which was published Wednesday, March 22, 2023 11:56:00 PM UTC. The v4.0.0 release has been removed from the Github releases page.

The bug was found by our fuzzers and has not been triggered on any networks. If triggered, it would result in temporary periods of effective downtime of several slots.

It is unlikely that this bug will present itself on mainnet, however we recommend the following actions for all networks (Mainnet, Goerli, Prater, Sepolia, Gnosis, etc):

• If you have not upgraded to v4.0.0 and are on v3.5.1, then take no action. Do not upgrade to v4.0.0 or v4.0.1-rc.0.
• If you have already upgraded to v4.0.0, then upgrade to v4.0.1-rc.0 immediately.

We are not recommending v4.0.0 users downgrade to v3.5.1 since a manual database migration is required. Users familiar with database migrations are safe to downgrade.

If you are not sure if you've updated, you can run lighthouse --version to find out.

The Docker stable flag has been updated to point to the patched version. If you have updated your stable Docker containers after Wednesday, March 22, 2023 11:04:00 PM UTC, we recommend updating again to obtain the patch.

All Changes

• Release Candidate v4.0.1-rc.0 (#4123)
• Fix fork choice error message (#4122)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-rc.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v4.0.1-rc.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v4.0.1-rc.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v4.0.1-rc.0	sigp/lighthouse

Tumblorkians

Summary

This release is a low-priority release for all users except for those using the Goerli test network.

For Goerli (aka Prater) users, this release includes the Capella fork parameters (#4044). Goerli will undergo the Capella upgrade (a.k.a "hardfork") on 14/03/2023 at 10:25:36 pm UTC. Goerli users must update to this version or later before the Capella upgrade. Users that fail to upgrade will cease to follow the chain and will be required to resync.

Users of Ethereum Mainnet or other networks/testnets (e.g., Sepolia, Gnosis Chain) may choose to update to this release to take advantage of various optimisations and bug fixes.

Notable changes in this release include:

• Add a flag to always use payloads from builders (#4052)
• Set Capella fork epoch for Goerli (#4044)
• Optimise signing in the VC to prevent late blocks (#4033)
• Add a BN latency measurement to the VC to help diagnose late blocks (#4024, #4051)
• Fix a bug where Lighthouse rejects an invalid block message from the EE (#4037)
• Delete Kiln and Ropsten configs (#4038)
• Improve compilation flexibility by allowing building without MDBX (#3888)

Breaking Changes

There are no breaking changes that affect Mainnet users.

⚠️ Breaking Change: Removal of Kiln and Ropsten Configs ⚠️

The --network kiln and --network ropsten flags are no longer supported. Kiln was deprecated Q3 2022 and Ropsten was deprecated in Q4 2022. Removing these networks has reduced the size of the Lighthouse binary by approximately 30MiB.

Dedicated Ropsten and Kiln users can still use these networks by manually obtaining the testnet configuration directories and using the --testnet-dir flag.

Goerli Capella Hard Fork

Goerli is upgrading to Capella on 14/03/2023 at 10:25:36 pm UTC! :tada:

Goerli users must update both the Lighthouse beacon node and validator client before March 14. Failure to update will result in missed validator duties and a corrupt beacon node database (requiring a re-sync).

Goerli users must also ensure they are running a compatible execution engine. The Ethereum Foundation will publish a "Goerli Shapella Announcement" in the coming days which will contain more information and a list of compatible execution layer releases.

Lighthouse will emit INFO Ready for Capella logs when both itself and the execution engine are ready for the Capella upgrade. Conversely, Lighthouse will emit WARN Not ready for Capella logs when it detects a misconfiguration.

VC to BN Latency Measurements

In #4024 and #4051 we added new Prometheus metrics for monitoring the round-trip latency between a VC and the BN. The latency measurement is the time it takes the VC to send, receive and parse a call to the BN's eth/v1/node/version endpoint.

There are two new metrics exposed by the VC:

• vc_beacon_node_latency_primary_endpoint: shows the latency for the primary BN.
• vc_beacon_node_latency: shows the latency for all BNs, with the endpoint URL as the endpoint label.

Producing a block requires two HTTP calls between the VC and the BN. Therefore, latency on this connection can contribute significantly to block publishing delays. Blocks which are published late are more likely to be orphaned.

Those using Grafana+Prometheus can use the following query to view the 99th percentile latency for their primary BN:

histogram_quantile(0.99, rate(vc_beacon_node_latency_primary_endpoint_bucket[$__rate_interval]))

Known Issues

The optimised maxperf profile is currently not working on Windows due to a suspected regression in the Rust compiler. The pre-compiled Windows binaries have been built with the release profile instead and may exhibit slightly reduced performance compared to previous versions. See #3964 for details.

Update Priority

This table provides priorities for which classes of users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Low Priority	Low Priority
Non-Staking Users	Low Priority	---

The Beacon Node may be updated without the Validator Client, however we recommend updating both components for completeness.

Goerli users must update both the beacon node and the validator client.

See Update Priorities for more information about this table.

All Changes

• Release v3.5.1 (#4049)
• Add a flag to always use payloads from builders (#4052)
• Set Capella fork epoch for Goerli (#4044)
• Add VC metric for primary BN latency (#4051)
• Log a WARN in the VC for a mismatched Capella fork epoch (#4050)
• Update dependencies incl tempfile (#4048)
• Optimise attestation selection proof signing (#4033)
• Optimise payload attributes calculation and add SSE (#4027)
• Add latency measurement service to VC (#4024)
• Permit a null LVH from an INVALID response to newPayload (#4037)
• Log a debug message when a request fails for a beacon node candidate (#4036)
• Cleaner logic for gossip subscriptions for new forks (#4030)
• Delete Kiln and Ropsten configs (#4038)
• Clean capella (#4019)
• Add more logs in the BN HTTP API during block production (#4025)
• Docs for Siren (#4023)
• Use consensus-spec-tests v1.3.0-rc.3 (#4021)
• Add content-type header to metrics server response (#3970)
• Allow compilation with no slasher backend (#3888)
• Execution Integration Tests Correction (#4034)

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.1-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.1-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.5.1	sigp/lighthouse

Gazorpazorpfield

Summary

This release is a low-priority release for all users (including mainnet), except for those using the Sepolia test network

For Sepolia users, this release includes the Capella fork parameters. Sepolia will undergo the Capella upgrade (a.k.a "hardfork") on 2023/2/28 at 4:04:48 AM UTC. Sepolia users must update to this version or later before the Capella upgrade. Users that fail to upgrade will cease to follow the chain and will be required to resync.

Ethereum mainnet, Ethereum testnet (e.g., Goerli, Ropsten) and Gnosis users may choose to update to this release, but it is low-priority.

Alongside the Sepolia changes, this release includes:

• ⚠️ Database schema upgrade to v15 ⚠️
• HTTP API returns 404 rather than 405 for an unknown route (#3836)
• Add CLI flag to specify the format of logs written to the logfile (#3839)
• Switch allocator to jemalloc (#3697)
• Implement new rewards APIs (#3822, #3903, #3907)
• Add attestation duty slot metric (#2704)
• Reduce some EE and builder related ERRO logs to WARN (#3966)
• Use Local Payload if More Profitable than Builder (#3934)

Breaking Changes

⚠️ Breaking Change: Database Schema v15 ⚠️

To support Capella the database schema has been upgraded to v15. The schema upgrade will be applied automatically upon upgrading and should not take more than a few seconds.

To downgrade, follow the instructions in the book for Database Migrations.

Users may downgrade anytime before the Capella upgrade. Prior Lighthouse releases are not compatible with Capella, so downgrading is fundamentally impossible after the upgrade occurs.

⚠️ Breaking Change: Minimum Supported Rust Vesion ⚠️

Users building from source will have to update their Rust compiler version to at least 1.65.0 using a command like:

rustup update stable

Newer versions of the compiler including the latest (v1.67.1 at time of writing) will also work.

Older versions will log an error during compilation.

Breaking Change: HTTP API Unknown Routes

Previously, the Beacon Node HTTP would return 405 (method not allowed) for an unknown route (e.g. http://localhost:5052/not_a_real_route). Lighthouse will now return a 404 (not found) error.

Sepolia Capella Hard Fork

Sepolia is upgrading to Capella on 2023/2/28 at 4:04:48 AM UTC! :tada:

Sepolia users must update both the Lighthouse beacon node and validator client before February 28. Failure to update will result in missed validator duties and a corrupt beacon node database (requiring a re-sync).

Sepolia users must also ensure they are running a compatible execution engine. The Ethereum Foundation Sepolia Shapella Annoucement contains more information and a list of compatible execution layer releases.

Lighthouse will emit INFO Ready for Capella logs when both itself and the execution engine are ready for the Capella upgrade. Conversely, Lighthouse will emit WARN Not ready for Capella logs when it detects a misconfiguration.

Known Issues

The optimised maxperf profile is currently not working on Windows due to a suspected regression in the Rust compiler. The pre-compiled Windows binaries have been built with the release profile instead and may exhibit slightly reduced performance compared to previous versions. See #3964 for details.

Update Priority

This table provides priorities for which classes of mainnet users should update particular components.

User Class	Beacon Node	Validator Client
Staking Users	Low Priority	Low Priority
Non-Staking Users	Low Priority	---

Note: this update is high-priority for Sepolia users.

The Beacon Node may be updated without the Validator Client, however we recommend updating both components for completeness. Sepolia users must update both the beacon node and the validator client.

See Update Priorities for more information about this table.

All Changes

This release contains the Capella functionality (#3763), which has been the result of several months of work. To ensure rightful attribution to its contributors, we have elected to perform a merge commit rather than our usual squash-merge. This has created an unusually complex history on the stable branch, so we have omitted the full list of changes in this release.

The full list of commits between this version (v3.5.0) and the previous version (v3.4.0) can be viewed on Github.

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.0-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.5.0-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.5.0-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.5.0	sigp/lighthouse

Tree States v3.4.0 alpha.3

Disclaimer

⚠️ You should not run this alpha release supporting mainnet validators ⚠️

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Summary

This is an alpha release of upcoming changes to Lighthouse which improve disk usage and state management.

For more information about the -tree.X family of releases, please see the v3.4.0-tree.1 release notes.

Compared to the previous release the main changes are:

• Fixed a database corruption bug which could occur when reconstructing historic states (dbb0cf7, 2f6ffff).
• Improved interaction between state reconstruction and finalization (481e792).
• Latest changes from unstable, including new rewards APIs.

Note that there is no v3.4.0-tree.2 release due to a build failure.

⚠️ Backwards Compatibility ⚠️

This release is not backwards compatible with stable Lighthouse. It uses a different database schema (v20) for which no automatic upgrade or downgrade is implemented. We intend to implement an automatic upgrade process once the new schema is finalized. A re-sync will be required to run future versions of tree-states.

This release is backwards compatible with the prior tree-states release (v3.4.0-tree.1).

Please only run this release if you are willing to re-sync now, and again in several weeks/months.

Known Issues

Expect a few sharp edges. Some things you may run into:

• WARN Parent state is not advanced is logged excessively during sync. This is harmless, albeit annoying.
• Block processing is a bit slower than stable Lighthouse.
• The Windows binary will not compile with the maxperf profile (use release).

Building from source

Build the v3.4.0-tree.3 tag.

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.3-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.3-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.3-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.4.0-tree.3	sigp/lighthouse

Tree States v3.4.0 alpha.1

Disclaimer

⚠️ You should not run this alpha release supporting mainnet validators ⚠️

If you are looking for the latest Lighthouse release please go to https://github.com/sigp/lighthouse/releases/latest.

Summary

This is an alpha release of upcoming changes to Lighthouse which improve disk usage and state management.

We are making this alpha release so that expert users may help us test these improvements. It is not backwards-compatible and not recommended for mainnet validators.

For the adventurous, the main benefits are:

• Smaller disk footprint for archive nodes, <300GB total. Use checkpoint sync, and set the flags --slots-per-restore-point=32 and --reconstruct-historic-states.
• Smaller disk footprint for regular nodes, <70GB.
• Faster handling of re-orgs.
• Faster restarts.
• Less disk I/O.

We hope that this is useful for running block explorers and supporting other beacon chain analytics. We are using it internally at SigP to run some of our analytics.

Aside from the experimental tree-states changes this release is up-to-date with Lighthouse v3.4.0 and includes the same features.

⚠️ Backwards Compatibility ⚠️

This release is not backwards compatible with stable Lighthouse. It uses a different database schema (v20) for which no automatic upgrade or downgrade is implemented. We intend to implement an automatic upgrade process once the new schema is finalized. A re-sync will be required to run future versions of tree-states.

Please only run this release if you are willing to re-sync now, and again in several weeks/months.

Known Issues

Expect a few sharp edges. Some things you may run into:

• WARN Parent state is not advanced is logged excessively during sync. This is harmless, albeit annoying.
• The JSON serialisation of the BeaconState does not quote integers for some fields. This will be fixed shortly, but SSZ can be used as a workaround.
• Block processing is a bit slower than stable Lighthouse (see below).

Additional Background

The fundamental change supporting these improvements is a change to the in-memory representation of the BeaconState from flat vectors to copy-on-write trees. This enables many more BeaconStates to be stored in memory, but comes at the cost of increased iteration time. As a result, block and epoch processing times are slower. To mitigate this we are working on new caches and optimisations which bring block processing times back in line with stable Lighthouse. This puts the tree-states branch in a slightly awkward spot, because we cannot responsibly merge it until it is better than stable Lighthouse on every possible metric (a Pareto improvement).

Seeing as tree-states alters the database schema quite drastically, we also decided to roll in a few other improvements. The plan is to get the new schema as close to perfect as possible so that in a future release everyone can upgrade once with minimal fuss.

Building from source

Build the v3.4.0-tree.1 tag.

Binaries

See pre-built binaries documentation.

The binaries are signed with Sigma Prime's PGP key: 15E66D941F697E28F49381F426416DC3F30674B0

System	Architecture	Binary	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-apple-darwin.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-apple-darwin-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-unknown-linux-gnu.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.1-aarch64-unknown-linux-gnu.tar.gz	PGP Signature
	aarch64	lighthouse-v3.4.0-tree.1-aarch64-unknown-linux-gnu-portable.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-windows.tar.gz	PGP Signature
	x86_64	lighthouse-v3.4.0-tree.1-x86_64-windows-portable.tar.gz	PGP Signature
System	Option	-	Resource
	Docker	v3.4.0-tree.1	sigp/lighthouse