Ethereum All Core Developers Call #140 & Consensus Layer Call #89 Writeup
Key Takeaways
Ethereum core developers concluded their 140th bi-weekly Zoom meeting on Friday, June 10th. The All Core Developers (ACD) calls are attended by developers building either the consensus or execution layer of Ethereum. On Friday, developers agreed on organizing a push back to the difficulty bomb schedule and discussed next steps for Merge upgrade testing. In addition, Ethereum core developers primarily focused on building the consensus layer of Ethereum concluded their 89th bi-weekly Consensus Layer call on Thursday, June 16. On Thursday, developers reiterated decisions made on the ACD call and came to a rough consensus around potentially activating the Merge upgrade on Sepolia during the last week of June.
Difficulty Bomb Pushback
On Friday, June 10th, Ethereum core developers reached a rough consensus around pushing back the schedule for the difficulty bomb by 2.5 months. Developers agreed to push out the bomb delay through a hard fork by the end of this month, which gives Ethereum execution layer client teams just under three weeks to release and test new code for deployment on mainnet. As background, hard forks unlike soft forks on blockchains require all computers, also called nodes, connecting to the Ethereum blockchain, to upgrade their software to a newer version at the same time to prevent causing a split to the canonical chain of the network. Historically, developers have budgeted at least 4 weeks for organizing and activating a hard fork on Ethereum.
In accordance with the tight turnaround schedule agreed on by developers during Friday’s call, the Geth client team, which maintains the majority execution layer client run by over 80% of all Ethereum nodes, released v1.10.19 on Wednesday, June 15th. The release prepares nodes for activation of the network’s 6th bomb delay dubbed Gray Glacier at block height 15050000, which is expected to hit around Wednesday, June 29th. All three other execution layer client teams including Besu, Erigon, and Nethermind have also followed suit with new releases. A detailed blog post communicating the Gray Glacier hard fork and client team releases to Ethereum node operators was released on the Ethereum Foundation blog on Thursday, June 16th.
Consensus around a difficulty bomb pushback was reached on Friday’s ACD call after founder of TrueBlocks Thomas Jay Rush gave developers an update on the impacts of the current bomb schedule on current block times. (For more information about the difficulty bomb on Ethereum and prior conversations by developers on this topic, read our prior Ethereum developer call summaries here.) “We’re at 14.7 second blocks here,” said Rush. “It’s going to probably be down around 18 to 20 second blocks I think by the middle of July, maybe the end of July. I’m conservative so I’m going to say by the middle of July you’re going to see 17 or 18 second blocks.” Rush then suggested the most optimal solution for developers moving forward considering the difficulty bomb’s impact on the network would be to disable the bomb permanently. This could potentially strengthen the argument that developers no longer need the bomb to motivate and guarantee Merge upgrade activation. However, such a decision may also be interpreted by the public as developers simply giving up on sticking to a timeline for Merge activation this year.
The next best solution according to Rush is to leave the bomb untouched. While this would mean reaching 20 to 30 second block times by August, Rush noted that it could be a good lesson for the community about the importance of participating in off-chain governance discussions such as the ACD calls. “The core devs are a great group of people but the community has to pay attention to the fact that the core devs have the power to make this decision,” said Rush. To Rush’s point, leaving the decision of delaying the difficulty bomb, and any changes to the code implementation of Ethereum for that matter, up to the discretion of developers does reinforce a dangerous precedent for the network’s governance process.
Core developers have historically had an outsized voice over miners, users, and other stakeholders in shaping Ethereum’s development roadmap, mainly because of increasing complexity of the network’s codebase. However, a decision to refrain from changing the bomb schedule and letting block times exponentially increase could potentially act as the wakeup call users need to start galvanizing and participating more in developer discussions. Both suggestions by Rush, that is a complete removal of the bomb and doing nothing about the bomb at all, garnered little support from other developers on the ACD call.
Most developers supported a delay to the difficulty bomb, which founder of Ethereum execution layer client Nethermind Tomasz Stanczak proposed formally to the group through Ethereum Improvement Proposal (EIP) 5133. EIP 5133 proposes a delay to bomb activation by 700,000 blocks, that is 2.5 months. “As we know the bomb [has] already activated and I think it looks worse than some people thought it was,” Stanczak said on the call. Given the urgency of the situation and the potential for block times to reach up to 40 seconds by November, Stanczak was in strong favor of confirming dates and block heights sooner rather than later.
Also, most participants of the call agreed on refraining from announcing a target mainnet Merge activation date in conjunction with a bomb delay announcement, even though doing this could potentially keep the pressure high on developers to deliver the Merge upgrade by a more immediate deadline. Chair of the ACD Call Tim Beiko emphasized that the Ethereum client teams he’s spoken to already feel “pretty stressed and urgent” and that there is “a fine balance” between maintaining a healthy amount of pressure to ship the upgrade faster and too much pressure creating negative impacts to the quality of developers’ work.
Despite Beiko not wanting to “rush” a decision on a difficulty bomb delay, consensus was reached by the end of the call to aim for hard fork activation in three weeks’ time on June 29th. Developers affirmed this would require all execution layer client teams to push out new releases for their code over the next few days after the call. All four client teams have since followed through and released new code for Ethereum node operators to run in preparation for Gray Glacier. In addition, Beiko has published a blog post communicating the details around the Gray Glacier fork to Ethereum stakeholders as of Thursday, June 16, which leaves just under two weeks for testing hard fork code leading up to mainnet activation.
The tight turnaround for activating Gray Glacier is noteworthy because it highlights the urgency felt by developers to mitigate the impacts of the bomb on users, miners, and decentralized applications (dapps). Up until Friday, May 27th, during the prior ACD call, it was unclear whether developers agreed that 20 second block times would be unacceptable for the network, even for a short duration before the Merge upgrade. The latest data shared by Rush about the impacts of the bomb on the network appears to have finally swayed the minds of core developers. A consensus around a 2.5 month delay suggests that developers have a high degree of confidence that Merge testing will be complete and the upgrade ready to implement within a few months.
What’s Next for Merge Testing?
Having completed the first of three major testnet activations earlier this month on Wednesday, June 8 with the activation of the Merge on the Ethereum Ropsten network, developers discussed their strategies for moving on to the next two testnets. To move on to the next testnet, Marek Moraczyński of the Nethermind team suggested that all client teams should be passing hive tests, which are dedicated testing tools for assessing the robustness of Ethereum’s execution layer and engine APIs. Adding to this, Stanczak mentioned fixes for the bugs identified in client software during Merge activation on Ropsten. For more information about the Ropsten Merge, read this Galaxy Digital newsletter brief.
Lukasz Rozmej also from the Nethermind team said that ideally the client software releases for the next testnet should be close to what will be activated on mainnet, which from his point of view would ideally take another 4-5 weeks to push out. Andrew Ashikhmin from the Erigon team stressed that in general there was still a lot of work to be done. “We are failing 88 out of 110 [hive tests]. … It’s a lot of work to fix all the hive tests and also to improve the robustness of our block building code and just more testing and more code stabilization.” As such, Ashikhmin said his team would need more than a couple of weeks to feel ready for the next testnet activation. Gary Schulte from the Besu team echoed the sentiments of the other client teams.
Looking ahead to the next testnet activation, a handful of outstanding concerns related to Merge activation that were raised on Friday’s call included:
Syncing procedures for EL clients post-Merge activation. Schulte raised concerns around the syncing mechanisms and procedures of execution layers clients with consensus layer clients. One suggestion by Péter Szilágyi from the Geth team was to add more detailed language for users operating execution layer nodes to alert them of their sync status with the consensus layer. Beiko encouraged further discussion around this issue to continue after the call on the Ethereum Research and Development Discord Channel.
Timing around “get payload” requests. This is a longstanding bug that developers have been trying to fix for the past few months. However, patches to fix the issue have been implemented differently across execution layer client teams. At its core, the issue arises when a consensus layer client sends a get payload request to an execution layer client before the execution layer client is ready to create a block. Some execution layer clients such as Nethermind and Nimbus have implemented workarounds to the issue by implementing short delays to their responses. However, as mentioned by Micah Zoltu on Friday’s call this fix is not sufficient and should be dropped from the code in favor of a different fix implemented among consensus layer clients. There are other nuances to this discussion that developers agreed should be considered and will be reflected in changes to Ethereum’s Engine API post-Merge activation on mainnet.
Making the gas limit of blocks configurable by external block builders. Alex Stokes, researcher at the Ethereum Foundation, raised a potential change to Ethereum’s Engine API that would help lower the barrier to entry for third-party block builders to propose blocks and earn maximal extractable value (MEV). Read more about MEV in this Galaxy Digital Research report. According to Stokes, his proposed change would help democratize the earnings of MEV post-Merge across a higher number of participants by making it easier for block builders earning MEV through third-party software like Flashbots to adjust the gas limit preferences of blocks.
There was pushback from developers like Danny Ryan about accepting a change to Ethereum’s Engine API this late in the Merge testing process and general consensus by participants of Friday’s call was to push the change in the next release of the Engine API after Merge activation on mainnet. It was reaffirmed on Thursday’s call that third-party software for extracting MEV post-Merge by organizations such as Flashbots should be intentionally delayed for roughly an hour after the upgrade is triggered to prevent adding more complexity to network functions at the time of the Merge. Some consensus client teams will be testing out their implementation of the software to support third-party block building during the next testnet activation of the Merge. For more information about builder code specifications, click here.
Sepolia Testnet Activation
Given the outstanding issues around Merge-related code and the readiness of client teams to push another major release of their software, developers floated the idea of swapping the order of testnet activation for the remaining two testnets, Goerli and Sepolia. As highlighted during prior ACD calls, the Sepolia testnet is a permissioned network that does not experience as high a volume of transaction activity from dapps and users as the Goerli testnet. Originally, developers wanted to activate the Merge on Goerli before Sepolia to give the larger, more active testnet a longer runway for core developers and dapp developers to monitor and experiment with. However, given that client teams need more time to finalize releases for the Merge, the thinking has shifted to potentially activating the upgrade on Sepolia first instead and with code that may still change before mainnet activation.
A clear decision around testnet ordering was not reached during the ACD call on Friday. However, the discussion highlighted the lack of preparation by client teams in the next few weeks to release code that would be mainnet ready. At the least, developers have agreed to launch a dedicated testnet Beacon Chain for the Sepolia network and ensure the Sepolia Beacon Chain is ready for Merge activation by early next week. On Thursday’s Consensus Layer call, developers also agreed to aim for a Merge activation date for Sepolia in the week following the next ACD call. This timeline would be dependent on Ethereum client teams fixing all the issues identified from the Ropsten Merge activation and completing a smooth launch of Ethereum’s 7th mainnet shadow fork.
The 7th mainnet shadow fork is scheduled to activate around Wednesday, June 22. Developers will replicate the same issues that were identified on the Ropsten testnet on this shadow fork to make sure that fixes to known issues have been correctly implemented by all client teams. For a deeper overview of the issues identified from Merge activation on Ropsten, read this Twitter thread by Ethereum Foundation developer Danny Ryan. Developers are continuing to test client software on Ropsten and will give another update on the status of the Ropsten testnet during the next ACD call on Friday, June 24. Given that Sepolia is a controlled testnet with a permissioned set of validators, developers agreed on Thursday’s call that less coordination is needed for the Merge activation on Sepolia and one could be scheduled for as early as 5 days after June 24. However, given that Gray Glacier is expected to be activated during the last week of June, the next major testnet activation date for the Merge is likely to be pushed back to sometime in July after Gray Glacier is activated on mainnet.
Other Miscellaneous Updates
Discussions around EIP 4844 will be continuing through a breakout room session today at 10am (ET). For more information about this EIP, read our prior developer call summary.
Core developers have spun up 6 dedicated shadow forks of mainnet Ethereum to test the Merge upgrade. Developers continue to experiment on most of them, though as mentioned by Marius van der Wijden of the Geth client team the second mainnet shadow fork will soon become deprecated.
Mikhail Kalinin presented his research around optimizing the staked ETH deposit process on Ethereum after Merge activation. For a full overview of his proposal, click here.
Tim Beiko announced a limitation to chatroom functions in the Ethereum Research and Development Discord channel on Thursday. Given the influx of messages in the “#allcoredevs” channel over the past few weeks, core developers have decided to make the chatroom a “read-only” channel for all participants except core contributors. “Once the Merge has passed, we’ll consult the client teams again and reevaluate whether having the channel open to all is more appropriate,” wrote Beiko.
Legal Disclosure:
This document, and the information contained herein, has been provided to you by Galaxy Digital Holdings LP and its affiliates (“Galaxy Digital”) solely for informational purposes. This document may not be reproduced or redistributed in whole or in part, in any format, without the express written approval of Galaxy Digital. Neither the information, nor any opinion contained in this document, constitutes an offer to buy or sell, or a solicitation of an offer to buy or sell, any advisory services, securities, futures, options or other financial instruments or to participate in any advisory services or trading strategy. Nothing contained in this document constitutes investment, legal or tax advice or is an endorsement of any of the stablecoins mentioned herein. You should make your own investigations and evaluations of the information herein. Any decisions based on information contained in this document are the sole responsibility of the reader. Certain statements in this document reflect Galaxy Digital’s views, estimates, opinions or predictions (which may be based on proprietary models and assumptions, including, in particular, Galaxy Digital’s views on the current and future market for certain digital assets), and there is no guarantee that these views, estimates, opinions or predictions are currently accurate or that they will be ultimately realized. To the extent these assumptions or models are not correct or circumstances change, the actual performance may vary substantially from, and be less than, the estimates included herein. None of Galaxy Digital nor any of its affiliates, shareholders, partners, members, directors, officers, management, employees or representatives makes any representation or warranty, express or implied, as to the accuracy or completeness of any of the information or any other information (whether communicated in written or oral form) transmitted or made available to you. Each of the aforementioned parties expressly disclaims any and all liability relating to or resulting from the use of this information. Certain information contained herein (including financial information) has been obtained from published and non-published sources. Such information has not been independently verified by Galaxy Digital and, Galaxy Digital, does not assume responsibility for the accuracy of such information. Affiliates of Galaxy Digital may have owned or may own investments in some of the digital assets and protocols discussed in this document. Except where otherwise indicated, the information in this document is based on matters as they exist as of the date of preparation and not as of any future date, and will not be updated or otherwise revised to reflect information that subsequently becomes available, or circumstances existing or changes occurring after the date hereof. This document provides links to other Websites that we think might be of interest to you. Please note that when you click on one of these links, you may be moving to a provider’s website that is not associated with Galaxy Digital. These linked sites and their providers are not controlled by us, and we are not responsible for the contents or the proper operation of any linked site. The inclusion of any link does not imply our endorsement or our adoption of the statements therein. We encourage you to read the terms of use and privacy statements of these linked sites as their policies may differ from ours. The foregoing does not constitute a “research report” as defined by FINRA Rule 2241 or a “debt research report” as defined by FINRA Rule 2242 and was not prepared by Galaxy Digital Partners LLC. For all inquiries, please email [email protected]. ©Copyright Galaxy Digital Holdings LP 2022. All rights reserved.