Ethereum All Core Developers Execution Call #206 Writeup
On February 27, 2025, Ethereum protocol developers met over Zoom for All Core Developers Execution (ACDE) Call #206. This week, the call was chaired by Ethereum Foundation (EF) Protocol Support Lead Tim Beiko. The ACDE calls are a bi-weekly meeting series where developers discuss and coordinate changes to the execution layer (EL) of Ethereum.
On ACDE #206, developers coordinated an action plan to recover the Holesky testnet. The Holesky testnet broke following the activation of Pectra due to misconfigured client software in three execution layer (EL) clients, Geth, Nethermind, and Besu. Developers agreed to proceed with the Sepolia testnet upgrade on Wednesday, March 5 with updated client software for the three EL clients and the Lodestar CL client. Finally, they discussed potential changes to EL and CL protocol specifications to make Ethereum more resilient in light of the Holesky incident.
Holesky Chain Status
Beiko explained the root cause of the Holesky testnet failure. On February 24, developers activated Pectra on Holesky. However, shortly following Pectra activation, developers realized that three out of five EL client teams had not set the correct address for validator deposits in their Holesky releases. Beiko noted that the deposit contract address differs between all Ethereum testnets and mainnet.
A validator deposit transaction on Holesky caused a chain split between nodes that correctly processed the deposit and those that did not. Specifically, Geth, Nethermind, and Besu nodes started to validate a different version of Holesky from Erigon and Reth nodes. Because most nodes stopped operating correctly, it was difficult for developers to restore Holesky as the network did not have enough validators attesting and progressing the correct head sometimes called “tip” of the Holesky chain.
Kamil Chodoła, Senior QA Engineer at Nethermind, said the issue with the deposit contract address has been fixed on Nethermind and his team is running more than 100,000 Holesky validators on the corrected client version. Prysm developer “Potuz” reported that some Prysm validators have been able to join the correct Holesky chain, but not all. Besu developer Justin Florentine said all Besu validators are back online and attesting to the correct Holesky head.
For context, Beiko noted that even though new validators are coming online to support the correct version of Holesky, most validators on Holesky are still in the exiting process, having been slashed for their incorrect block proposals or suffering an inactivity leak for not proposing to the correct chain.
“For context, because the quote valid chain was a very small minority chain, there’s only about 10% of stake on it,” said Beiko, adding, “I personally learned this week that even if you’re a validator who attested to an invalid, finalized chain, you can still propose [blocks] on another chain. So, this is why we see a larger percentage of blocks being proposed than of attestations being collected per epoch.”
At the time of the ACD call, Rafael Matias, an engineer at the EF, reported block production has recuperated to a 44% success rate on Holesky.
Holesky Next Steps
Beiko asked about the next steps for recovering the Holesky testnet. Potuz recommended coordinating a mass slashing event for all validators attesting to the incorrect head of the chain such that developers can force Holesky into a state of finalization. The benefit of reaching finalization is that nodes when resyncing to Holesky have a closer checkpoint to base new blocks and attestations off than the Pectra upgrade checkpoint. This is particularly useful for networking so that nodes can easily connect with other nodes validating the correct version of Holesky.
A mass slashing event may give enough weight to minority validators to finalize the correct version of Holesky but even if it doesn’t, Potuz explained that the network has still sped up the process of recovery by expediting the slashing of bad actors on the network. Before developers trigger the mass slashing event, Potuz recommended ensuring that at least 75% of Holesky node operators are operating corrected client software. This is because validators are shuffled each epoch into different committees for attesting and proposing blocks. For the network to finalize after the slashing event, there will need to be two consecutive epochs justified by at least 66% of non-slashed validators each.
A developer by the screen name “Jim McDonald” noted that even if Potuz’s plan succeeds, it will take at minimum three weeks from network finalization to fully exit all bad validators from Holesky, meaning the testnet will remain unstable for users for a prolonged period. “If it’s the best we can pull off then that’s okay. I guess the other thing it does is it buys us time to look [at] clients. I know Lighthouse, for example, has released a new version or new patch recently that gives them more ability to handle longer-term lack of finality,” said McDonald, adding, “We’re buying the time to allow us to do that and … getting a reset to avoid more forks. That’s reasonable. I just think it’s probably important to make sure it’s clear to people though that it’s not like we get to finality, and we’re done. It’s still a long road from that.”
Prysm developer Terence Tsao asked about the coordination needed between client teams to disable slashing protections. There are a variety of protections for validators to avoid slashing and each have different protocols for disabling. Beiko asked if CL client teams could set an arbitrary block as a checkpoint for validators in their code without having to go through the hassle of trying to force network finalization through a mass slashing event.
To Tsao’s question, Reth developer Roman Krasiuk pointed to Nethermind developer Marek Moraczyński’s document for disabling slashing protections. Potuz said it would be ideal to coordinate the slashing today or tomorrow. Tsao reminded developers the first step is to ensure there is a sufficient percentage of remaining validators to correctly finalize the chain. Grandine developer Saulius Grigaitis expressed skepticism at the plan for slashing saying that it requires “too much coordination”. “The setups with slashing protection are quite different among clients and signers and so on. So, it’s really hard to get it right, especially by tomorrow,” said Grigaitis, adding, “In any case, we will have a longer nonfinalization [period] after that. I just don’t see that it’s going to help a lot, even assuming that it will finalize.”
Potuz responded saying that his plan for triggering network finalization would likely fail but at the very least his plan will ensure that bad validators are exited from the network on the fastest possible timeline.
A developer by the screen name “Radek” asked if developers are wasting their time recovering Holesky with a plan that would never be possible on Ethereum mainnet. Realistically, Ethereum stakers would not accept a plan that requires mass slashing of the majority of staked ETH. Further, Ethereum would not survive three weeks of non-finality before being declared a “dead” or unusable chain, said Radek. So, Radek suggested focusing efforts on changing the Ethereum protocol itself in such a way that recovery for Holesky is possible and the process could be replicated for Ethereum mainnet.
Beiko responded to Radek’s comments saying that developers should spend time discussing longer-term changes to the Ethereum protocol to ensure network resilience in similar situations as Holesky. However, the exercise of recovering Holesky with the given network constraints offers developers useful learnings about how Ethereum works and buys developers time to consider longer-term protocol changes.
A representative from the Lido team by the screen name “Sacha” shared thoughts on the utility of Holesky for testing purposes. “The thing about Holesky for us is it’s more like a staging QA environment, rather than a sandbox,” said Sacha. Sacha recommended that if Holesky cannot be saved in time for Pectra testing, then developers should wait until they have a Holesky-like testnet up and running for stakeholders to use before moving forward with Pectra mainnet activation. Sacha added that a shorter-lived testnet such as another Pectra devnet is “better than nothing” but these types of environments will not offer the same monitoring and infrastructure to fully battle test their code in preparation for the Pectra upgrade.
Beiko acknowledged Sacha’s concerns but said that these concerns should be addressed as a separate discussion topic from the discussion topic of the next immediate steps for the Holesky testnet. Mcdonald shared skepticism about recovering Holesky to the point where the network could be used again to test any future Ethereum upgrade.
Beiko asked developers if they could coordinate a disabling of slashing protection across all clients on a call tomorrow, February 28, at 15:00 UTC. There were no objections to this plan.
Sepolia Fork Timing
Beiko then asked developers about the plan for the Sepolia upgrade which is scheduled to activate on Wednesday, March 5, less than a week from ACDE #206. Beiko stressed that because developers bundled the client releases for both the Holesky and Sepolia upgrades, changing the activation of the Sepolia upgrade would require all Sepolia node operators to update their software. There were no objections to keeping the original Sepolia upgrade date, time, and epoch number. However, Beiko said that he will update the Pectra testnet announcement blog post with new client releases for the Geth, Nethermind, and Besu clients. All Sepolia node operators running any one of these three EL clients must update their software between now and March 5.
In addition, a representative of the Lodestar team said that there is a mandatory update to the Lodestar CL client that is required for the Sepolia upgrade. So, Sepolia node operators running Lodestar must also update their software before March 5.
Other Holesky Initiatives
Then, Beiko asked about other longer-term initiatives for addressing the issues from the Holesky incident. Potuz spoke on how to upgrade CL clients such that they can checkpoint sync from a socially coordinated epoch or block rather than a finalized one. A developer by the screen name “Felix” proposed agreeing on a single EL genesis/fork configuration format to avoid the issue of misconfigured system contract addresses. EF Researcher Ansgar Dietrichs expressed concerns in the chat about the shifting of attention on Holesky away from planning for Ethereum’s next upgrade, Fusaka, which contains an important blob scaling update, PeerDAS.
On the topic of socially agreed upon checkpoint syncs across CL clients, Potuz added that at the minimum creating a flag for disabling optimistic sync mode in CL clients would be extremely beneficial for developers to create. A representative from the Lighthouse team pointed to work already happening in this direction.
On the topic of standardizing fork configuration formats across ELs, Nethermind developer Lukasz Rozmej suggested that developers use a mechanism similar to a fork ID to coordinate client teams on system contract configurations between forks. McDonald was in favor of this idea but noted that developers should not underestimate the work involved in creating such a mechanism. Beiko asked if someone on the call could spearhead these efforts. Independent Ethereum developer Danno Ferrin said that he could. Beiko emphasized that these efforts would be for implementation in a future Ethereum fork such as Fusaka, not for immediate use for Pectra.
There were many topics on today’s call agenda that were not discussed due to limited time on the call. ACD calls are usually limited to an hour and a half. Therefore, Beiko ended the call having discussed the most urgent items related to the Holesky incident. Other topics that were not discussed will be raised again on future ACD calls. Before ending the call, Beiko asked if there were any other urgent matters for developers to discuss. There were none raised.
Legal Disclosure:
This document, and the information contained herein, has been provided to you by Galaxy Digital Holdings LP and its affiliates (“Galaxy Digital”) solely for informational purposes. This document may not be reproduced or redistributed in whole or in part, in any format, without the express written approval of Galaxy Digital. Neither the information, nor any opinion contained in this document, constitutes an offer to buy or sell, or a solicitation of an offer to buy or sell, any advisory services, securities, futures, options or other financial instruments or to participate in any advisory services or trading strategy. Nothing contained in this document constitutes investment, legal or tax advice or is an endorsement of any of the stablecoins mentioned herein. You should make your own investigations and evaluations of the information herein. Any decisions based on information contained in this document are the sole responsibility of the reader. Certain statements in this document reflect Galaxy Digital’s views, estimates, opinions or predictions (which may be based on proprietary models and assumptions, including, in particular, Galaxy Digital’s views on the current and future market for certain digital assets), and there is no guarantee that these views, estimates, opinions or predictions are currently accurate or that they will be ultimately realized. To the extent these assumptions or models are not correct or circumstances change, the actual performance may vary substantially from, and be less than, the estimates included herein. None of Galaxy Digital nor any of its affiliates, shareholders, partners, members, directors, officers, management, employees or representatives makes any representation or warranty, express or implied, as to the accuracy or completeness of any of the information or any other information (whether communicated in written or oral form) transmitted or made available to you. Each of the aforementioned parties expressly disclaims any and all liability relating to or resulting from the use of this information. Certain information contained herein (including financial information) has been obtained from published and non-published sources. Such information has not been independently verified by Galaxy Digital and, Galaxy Digital, does not assume responsibility for the accuracy of such information. Affiliates of Galaxy Digital may have owned, hedged and sold or may own, hedge and sell investments in some of the digital assets and protocols discussed in this document. Affiliates of Galaxy Digital may also lend to some of the protocols discussed in this document, the underlying collateral of which could be the native token subject to liquidation in the event of a margin call or closeout. The economic result of closing out the protocol loan could directly conflict with other Galaxy affiliates that hold investments in, and support, such token. Except where otherwise indicated, the information in this document is based on matters as they exist as of the date of preparation and not as of any future date, and will not be updated or otherwise revised to reflect information that subsequently becomes available, or circumstances existing or changes occurring after the date hereof. This document provides links to other Websites that we think might be of interest to you. Please note that when you click on one of these links, you may be moving to a provider’s website that is not associated with Galaxy Digital. These linked sites and their providers are not controlled by us, and we are not responsible for the contents or the proper operation of any linked site. The inclusion of any link does not imply our endorsement or our adoption of the statements therein. We encourage you to read the terms of use and privacy statements of these linked sites as their policies may differ from ours. The foregoing does not constitute a “research report” as defined by FINRA Rule 2241 or a “debt research report” as defined by FINRA Rule 2242 and was not prepared by Galaxy Digital Partners LLC. For all inquiries, please email contact@galaxydigital.io. ©Copyright Galaxy Digital Holdings LP 2025. All rights reserved.