Ethereum All Core Developers Call #139 & Consensus Layer Call #88

As part of Galaxy Digital Research’s ongoing coverage of Ethereum, we will periodically provide updates and overviews of important developer activity, including recurring meetings and calls among developers.

Key Takeaways

Ethereum core developers concluded their 139th bi-weekly Zoom meeting on Friday, May 27th. Unlike the Consensus Layer calls, which occur every other Thursday at 14:00 UTC and are attended primarily by developers building the consensus layer software of Ethereum, the All Core Developers (ACD) calls are attended by developers building either the consensus or execution layer of Ethereum. On Friday, developers discussed:

• the status of Merge testing,

• the difficulty bomb schedule,

• and two Ethereum Improvement Proposals related to sharding and Boneh–Lynn–Shacham (BLS) signatures.

In addition, Ethereum core developers building the consensus layer of Ethereum concluded their 88th bi-weekly Consensus Layer call on Thursday, June 2. On Thursday, developers, discussed:

• the forthcoming Ropsten hard fork

• outcomes from the 6th mainnet Merge shadow fork

• the 7-block reorg incident on Wednesday, May 25

Merge Testing

Ethereum core developers continue to focus their attention primarily on testing the Merge upgrade, which is the network’s forthcoming transition to a proof-of-stake (PoS) consensus protocol. This highly anticipated hard fork is by far the most complex upgrade for developers to execute and arguably one of the most important in Ethereum’s history, as moving off a proof-of-work (PoW) consensus protocol has been the goal of Ethereum developers since as early as 2015. For more information about the Merge upgrade, this Galaxy Digital research report.

Over the past six months, core developers have been spinning up new dedicated testnets for the Merge upgrade, most of which have been short-lived, lasting only for a few days or weeks. Now, developers are finally ready to release the Merge upgrade on their very first pre-existing Ethereum testnet, known as the Ropsten testnet. To this end, Ethereum client teams must release dedicated, public-facing code specifications for Ropsten users to download and use to upgrade their nodes. The first agenda item of both calls was about the final preparations being made for the forthcoming Ropsten Merge hard fork.

Ropsten is the first of three Ethereum testnets that will undergo the Merge before the upgrade is released on mainnet. During the last ACD call, developers decided to schedule the activation of the Merge on Ropsten to hit around June 8, 2022. As background, the Merge is engineered to activate once the mining difficulty of the network reaches a specified threshold known as the total terminal difficulty (TTD). Normally, system-wide network upgrades on Ethereum are scheduled by a block height. However, in the case of the Merge, developers were concerned about the potential for malicious actors with a minority of hash power to mine a version of Ethereum’s proof-of-work (PoW) blockchain that satisfies the block height requirement and thereby trigger the upgrade prematurely on Ethereum’s proof-of-stake (PoS) blockchain, also called the Beacon Chain or the consensus layer. To avoid such an outcome, developers had agreed to use TTD as a threshold, which would require a majority of hash rate for bad actors to influence.

Because the hash rate of Ethereum’s Ropsten testnet is significantly smaller than the hashrate of Ethereum mainnet, it is much easier for bad actors to manipulate. This is exactly what happened on Wednesday, May 25. A malicious miner on Ropsten caused network hash rate to surge 30x overnight. This caused the mining difficulty of the network to skyrocket and TTD for the Merge to be hit prematurely. In fact, TTD was hit on Ropsten before developers had even spun up the Ropsten Beacon Chain. In hindsight, developers agreed that a TTD should not have been announced before the Ropsten Beacon Chain was launched.

Because TTD was hit prematurely on Ropsten, developers agreed on the call to issue a new release for software clients overriding the original TTD threshold on Ropsten and adjusting it to a significantly high value that is twice the total difficulty of the current Ethereum mainnet and would overshoot Merge activation timing on Ropsten by 250 years. This way malicious actors are disincentivized from messing with the hash rate of the network given that they would have to spin up twice as more computational energy than what is currently being devoted to Ethereum mainnet in order to prematurely trigger the Merge upgrade on Ropsten again.

Developers still plan on activating the Merge on the Ropsten testnet around June 8 but will communicate a new TTD value after the Ropsten Beacon Chain has been launched and the network has gone through the necessary upgrades to be ready for Merge activation. Users running miners and nodes on Ropsten will need to be especially attentive to communication channels from core developers as the new TTD value for Merge activation will only be shared a few days in advance.

As highlighted by Micah Zoltu on Friday’s call, the manipulation of hash rate on Ropsten seen during Wednesday’s incident is not feasible on Ethereum mainnet due to the comparatively large amount of hash rate devoted by miners to securing and earning rewards on mainnet. However, it is true that, as on testnets, estimating the precise mining difficulty threshold of Ethereum’s mainnet remains an imperfect science. Therefore, once a TTD is set for Ethereum’s mainnet, Merge activation may vary by a few days from developers’ expectations. “With the Merge, everybody should be prepared for testnet and mainnet, we may get the day wrong and for testnets we may get the week wrong, as we saw with Ropsten,” said Zoltu. “Users need to be aware these are best guesses for us.”

Mirroring testnet Merge activation on mainnet

Given the unexpected attack on Ropsten prematurely triggering the Merge upgrade, developers discussed on Friday’s call how to mirror the same processes for Merge activation on Ropsten for future testnets, Goerli and Sepolia, as well as Ethereum mainnet. Even though messing with the hash rate on Ethereum would be more difficult than on these testnets, it’s worth replicating the same activation procedure for the Merge on testnets through to mainnet to ensure all client teams, users, validators, and other node operators are on the same page and familiar with the procedure.

To this end, developers discussed potentially pushing an upgrade with a high TTD value for the Ethereum Beacon Chain around the time of the last Merge testnet activation on Sepolia. This would mirror how the Merge is being activated on Ropsten by first setting an artificially high TTD value and then overriding this value a short period before upgrade activation. This would require users to update their nodes twice for Merge activation. This would also mean that by the time of Merge activation on Sepolia, which is the last of the three Ethereum testnets to be upgraded, that all consensus layer clients should have near-finalized candidates for the Merge ready. The initial TTD for the Merge would be set through what is called the Bellatrix upgrade and a new TTD overriding the initial value would be released closer to the scheduled date for the Merge.

Sepolia Merge activation

On the topic of Sepolia Merge activation, developers have yet to launch a dedicated Beacon Chain for the Sepolia Merge but are expected to do so sometime next week after further coordination with Ethereum Foundation developer Parithosh Jayanthi. There was some discussion on Thursday’s CL call on whether the Sepolia Merge activation should happen before Goerli. Chair of the ACD calls Tim Beiko emphasized that the rationale for launching the Merge upgrade on Goerli first is to allow more time for users and decentralized application developers familiar with the Goerli testnet to test their software on the network. However, if client teams don’t feel prepared to release near-final code for Merge activation by the time of the Goerli hard fork, Beiko did emphasize it would make sense to perhaps keep testing the Merge on a testnet with less user-facing activity like Sepolia first. On Thursday’s call, developers agree to hammer out the order of Merge testnet activation post-Ropsten on the next ACD call on June 10.

Ropsten Beacon Chain launch

On Tuesday, May 30, core developers successfully launched the Ropsten Beacon Chain and reaffirmed on Thursday’s call that they are on track to activate the Bellatrix upgrade on the newly launched Ropsten Beacon Chain this Thursday, June 2. As explained above, once Bellatrix is activated, the plan is to override the existing TTD value on the Ropsten Beacon Chain with a more accurate number that will trigger the Merge upgrade on Ropsten around June 8. Beiko reiterated on the CL call that once Bellatrix has been successfully activated, a new TTD figure would be privately communicated to client teams and developers at the Ethereum Foundation first before being communicated to the wider public.

Head of Research Development at Status Jacek Sieka, whose team is building the Nimbus CL client, stressed the importance of learning and practicing how to override TTD values in lead up to Merge activation, calling it a “a good skill” to have for users. This is because overriding TTD values does not require client teams to release new code for the Merge upgrade. It simply requires users to manually enter a new value replacing the original TTD value specified in the Bellatrix code release. To this end, Beiko is preparing an additional blog post explaining how users can override their Ropsten Beacon Chain TTD values, which users can read a rough draft of here.

Core developers are also currently working out an issue with the Ropsten Beacon Chain related to the network’s ability to track validator deposits. Like Ethereum mainnet today, there is a smart contract on Ropsten that is used to deposit the required amount of 32 ETH for activating a new validator on the Ropsten Beacon Chain. However, due to the abnormally long block times on Ropsten which according to Lighthouse developer Paul Hauner reached up to 60 seconds at times, certain client teams struggled to accurately track validator deposits from the Ropsten proof-of-work chain. Chair of the CL calls Danny Ryan stressed that the issue was not core to Ethereum’s proof-of-stake consensus protocol design but rather the engineering implementation of the design.

Hauner elaborated on the validator deposit tracking issue explaining that the longer block times made it difficult for Ethereum CL clients to track the depth of the canonical chain. “[We’re] trying to avoid unnecessarily downloading blocks so you make some assumptions about block times so you don’t have to download all of [the blocks],” Hauner said on Thursday’s CL call. This tolerance factor which assumes a normal block time range was broken when Ropsten block times wildly exceeded the average of 12 to 13 seconds. Fixes have been released for both Prysm and Lighthouse clients to address this issue. Despite the releases, the Ropsten Beacon Chain is still struggling to onboard new validators. There appears to be 376 deposits into the Ropsten deposit contract that has not successfully been read by the Ropsten Beacon Chain to create new validators. Ryan called for more attention on this issue on Thursday’s CL call and for further coordination on solutions in the Ethereum Research and Development Discord channel.

Other Merge testing items

Afri Schoeden gave a short update on Friday’s ACD call about the status of the Georli testnet, which is likely to be the second testnet after Ropsten that will be upgraded for the Merge. Schoeden highlighted that hash rate is more stable than on Ropsten and hence the TTD should be easier to estimate. Ryan highlighted on Thursday’s CL call that the Kiln testnet which has already undergone Merge activation will be deprecated at the time of mainnet merge. Decentralized application developers testing software on Kiln should prepare to move off of that testnet to Goerli or Sepolia post-Merge.

Ethereum core developers launched their 6th mainnet shadow fork testing the Merge on Tuesday, May 31st. There were a few issues discovered in EL software clients Besu and Erigon. In short, the EL clients were failing to produce blocks with transactions validated in them and instead returning only empty blocks. Ryan noted that results from the shadow fork this past week are still being communicated and that further discussions would be had on the Ethereum R&D Discord channel. For more information about what shadow fork are, read this blog post written by Beiko.

Difficulty Bomb Scheduling

Another major topic of discussion from Friday’s ACD call was on the difficulty bomb schedule, which already appears to be delaying block times slightly on Ethereum. The difficulty bomb is a mechanism on Ethereum that once activated begins to exponentially increase mining difficulty such that producing blocks over time becomes prohibitively costly and unprofitable for miners to do. This mechanism was first coded into Ethereum by core developers in 2015 to ensure that the development roadmap of the network eventually progressed to becoming fully reliant on validators and a proof-of-stake (PoS) consensus protocol rather than on miners and a proof-of-work protocol. In addition to pressuring core developers to work towards a transition to PoS, the difficulty bomb also ensures that once Ethereum has moved to PoS, malicious miners that may persist on building the PoW version of Ethereum’s blockchain cannot sustainably keep the competing chain running without organizing a hard fork to disable the bomb mechanism. At several points in Ethereum’s history, the schedule for the bomb has had to be manually delayed through a dedicated hard fork due to delays in research, development, and testing for the Merge upgrade.

As identified by founder of TrueBlocks Thomas Jay Rush, the weekly block production of Ethereum has seen a noticeable dip because of the impacts of the bomb mechanism over the last few weeks. “Just using my eyeballs, my chart is hinting at 15-second blocks through mid-June and 16-second blocks into late June / early July,” Rush wrote in an Ethereum Magicians thread.

Developers discussed on Friday whether such activity justifies a delay to the Ethereum difficulty bomb for the 6th time in chain history. The responses from developers were mixed.

Tomasz K. Stańczak, founder of Ethereum client Nethermind, proposed a 3 to 4 month delay of the difficulty bomb, which Andrew Ashikhim of the Erigon client team and “Potuz” of the Prysmatic Labs team were in broad support of. Pushing back on delaying the bomb by 3 months were core developers Ben Edgington and Marius van der Wijden. “I don’t want to advocate for moving the bomb because I just want to keep the pressure up to get this thing done ... 3 to 4 months I think is giving ourselves too much of an out,” said Edgington. “But a couple of weeks at least to take the heat off the block times on mainnet [works.]”

A few outstanding items for Merge activation to happen on mainnet, as highlighted by Chair of the ACD meetings Tim Beiko, include the rollout of the Merge on Goerli and Sepolia. In addition, Ethereum execution layer clients have yet to pass hive tests. As background, Hive is a testing platform used to target the robustness of new Ethereum engine APIs created as a result of the Merge upgrade. Łukasz Rozmej from the Nethermind team noted that all client teams are currently failing multiple hive tests and that is an important criteria developers would want to rectify before activating the Merge on mainnet.

Marius van der Wijden of the Geth client team said that he sees the rationale for both sides but that in his view he doesn’t see a “technical merit at the moment” for scheduling another delay to the Ethereum difficulty bomb at present. “I really want to avoid us having to push back the bomb twice. That’s the worst thing that we can do. I think pushing back the bomb once already is really bad because we have to schedule another fork. And it’s not about the scheduling. It’s about the coordination of the community. I think we might lose some people if we schedule another delay, bomb pushback. … I think teams should really, really strive to hit their timelines. Yeah, it would be really good for us to hit the timelines we set ourselves.”

At this stage, assuming that the Merge activates in August as anticipated by developers and that all further testing goes smoothly, block times are still expected to near 25 seconds by the time of Merge activation in August due to the effects of the difficulty bomb. As such, Ashikhim argued it would be a “disservice to the users” to let network capacity contract to such an extent and do nothing to prevent the artificial rise of mining difficulty even with the most optimistic assumption about testing in the coming weeks.

To help with the optics around another bomb delay, Ashikhim suggested coupling the bomb delay release with something “useful” such as near-finalized code specifications for users that would almost fully prepare them for the Merge upgrade. However, Beiko mentioned that in this case, a mainnet release for execution layer clients, called Paris, and one for consensus layer clients, called Bellatrix, would likely not be finalized until after the three testnet activations, in which case doing nothing about the bomb until the last testnet Merge activation would still mean hitting 16 to 18 second block times in the meanwhile for users.

“I don’t think 20 second block times are bad, and in Amsterdam we had the overall view that a degraded user experience for two months is worth it if we can ship the Merge faster,” said van der Wijden on the call. Stańczak rebutted van der Wijden’s point saying, “I just see all engineers working at their limits now so it starts hurting productivity. I don’t think not delaying the bomb is speeding things up. [I don’t think] that people will magically generate additional fire power to keep those times.”

Péter Szilágyi of the Geth team offered a middle ground to the debate which was a two-month difficulty bomb delay instead of a three-month delay. Ultimately, no consensus was reached. The prevailing impression among developers by the end of Friday’s call was that a bomb delay of some length would indeed be needed on Ethereum. However, the main source of controversy and indecision was around how long exactly that delay should be. Beiko suggested strong supporters of a bomb delay prepare a formal EIP in preparation for the next ACD call and further discuss this subject matter then.

Other miscellaneous items

A few other discussion items raised on the two latest Ethereum developer calls included:

• Change to Ethereum’s Engine API: Mikhail Kalinin presented a change to Ethereum’s Engine API based on the discussion around deprecating the step parameter used to return data about finalized blocks on the Beacon Chain. (More information about this in our previous write up of the Consensus Layer call here.) As background, Ethereum’s Engine API is responsible for specifying how consensus layer clients interacts with execution layer clients.

Developers agreed on the call that while this change would be a “nice to have,” it is not a time sensitive matter for the Merge. Execution client teams are expected to provide feedback on this issue as time allows in the next few weeks. Consensus layer (CL) client teams such as Prysmatic Labs are already working on their implementation for the Engine API change, though that is not the case for all CL teams, some of which are still working on building their software releases in accordance with the latest finalized Merge specifications.

• EIP 4844: EIP author George Kadianakis shared a quick update on the latest validation schemes for validating “blobs” on Ethereum. As background, blobs refer to a new transaction format for Ethereum that would significantly reduce the cost of executing transaction rollups. Rollups are a scalability solution for Ethereum that relies on executing transactions on a separate, Layer-2 blockchain. Read more about the Layer 2 landscape on Ethereum in this Galaxy Digital research report.

Kadianakis explained that with the former approach to EIP 2844, verifying blobs took roughly 40 milliseconds to complete on Ethereum. Using KZG proofs, the verification time can allegedly be reduced to 2.5 milliseconds. More background on KZG proofs here. For now, EIP 4844 is undergoing further research and development and is expected to be implemented sometime after Merge activation.

• EIP 2537: Peter Chunchi Liu, Lead Scientist at Ernst & Young Greater China, gave an update on EIP 2537, which would effectively add a new precompile to Ethereum supporting a new cryptographic signature scheme known as Boneh–Lynn–Shacham (BLS) signatures. BLS signatures are argued to be more secure and efficient than what is most commonly used on public blockchains which is ECDSA signatures. Liu said on the call that adding this new BLS precompile could increase the security and privacy of Ethereum and enable new use cases for the network’s dapp ecosystem.

Liu also emphasized that this EIP does not require additional R&D and is ready for activation immediately on the network. However, as discussed by core developers during a prior ACD call, no new EIPs will be considered for inclusion in subsequent hard forks until the Merge is complete and progress has been made on the EIPs already green lit for Ethereum’s Shanghai hard fork.

• 7-Block Reorg Post-Mortem: Ethereum Foundation researcher Caspar Schwarz-Schilling gave an overview of what caused the 7-block reorg on Wednesday, May 25. In short, 2 blocks were proposed at the exact same time on the Ethereum Beacon Chain with the same amount of weight, meaning attestations from validators. Slightly more than half of active validators had not been running a new update to the fork choice rule of the Beacon Chain known as proposer boosting, while the other half was. This caused a minor split in consensus between validators that was then exasperated when the next six block proposers randomly selected to build on top of the chain were all validators that had updated their software to include the proposer boosting mechanism. Important to note here that on top of the split in fork choice rule implementation between validators, there was also a bug in the proposer boosting mechanism. Block proposers who were running the proposer boosting mechanism were failing to re-run the fork choice logic at the time of their block proposal.

Eventually, the chain built upon by the six block proposers that were not correctly implementing proposer boosting was replaced by a competing chain. The chain reorg happened when the seventh validator random selected to propose the next block after the six happened to not be running the proposer boosting mechanism at all and thus correctly chose to build upon the alternative chain that had accumulated more weight from validator attestations. Schwarz-Schilling highlighted that if the bug in the proposer boosting had been rectified, the block reorg could have been avoided. Ryan added that the incident highlights a need for developers to analyze more deeply the roll out for incremental fixes to consensus layer specifications like the ones that had been rolled out for proposer boosting. A more robust analysis against potential unknown issues arising from incremental fixes and tighter coordination around subsequent updates to specifications is needed, said Ryan.

• New Github Pull Requests for Review on Beacon Chain Light Client Specifications: Sieka from the Nimbus client team updated developers about a new batch of PRs for enabling light client functionality on the Beacon Chain. The main use case for light clients on Ethereum is to make verifying the state of the blockchain and its transaction history more easily accessible for users. Instead of running an individual validator or full node for the CL and EL of Ethereum, light client protocols will eventually enable users to query the blockchain using minimal amounts of computational resources. More information on Ethereum light clients here.