Skip to main content

[Goerli Testnet] Incident Update: EigenPod Withdrawals Impacted due to Goerli Hard Fork

· 5 min read

Resolved [Jan 26, 2024 - 13:00 PT]​

  • This incident has been resolved.

  • Additional Support: If you are still encountering issues with EigenPod withdrawals (partial or full), please share via the EigenLayer Discord #support-restakers channel.

Monitoring [Jan 25, 2024 - 12:00 PT]​

  • Actions: We successfully deployed the fixes yesterday, and are continuing to monitor the results for an additional day. We don’t expect users to face any further issues, and can resume normal operations for partial and full withdrawals from EigenPods on Goerli testnet.

  • Additional Support: If you are still encountering issues with EigenPod withdrawals (partial or full), please raise your issue via the EigenLayer Discord #support-restakers channel.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

  • Next Update: Jan 26, 2024 - 11:00 PT

Monitoring [Jan 24, 2024 - 16:30 PT]​

  • Restoration of EigenPod Functionalities on Goerli: We identified and deployed a number of changes needed to the EigenPod contracts (see PR: Layr-Labs/eigenlayer-contracts/pull/395), and to the proof generation implementation (tip of this branch). At around 13:05 PT, we verified that this successfully restored both partial and full withdrawal functionalities on the Goerli network (successful Deneb withdrawal on Deneb state - Etherscan transaction (timestamp of the withdrawal at validatorIndex:668660, timestamp:1705595700, post deneb). We re-enabled the native staking (EigenPod) functionality on the Goerli web application at 13:30 PT.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

  • Actions: We have successfully deployed the fixes, and are monitoring the results. We don’t expect users to face any further issues, and can resume normal operations for partial and full withdrawals from EigenPods on Goerli testnet.

  • Next Update: Jan 25, 2024 - 12:00 PT

Identified [Jan 19, 2024 - 10:40 PT]​

  • Summary: The issue was identified as being triggered by the Goerli hard fork event on 01/17/2024 (geth release v1.13.9 block height 1705473120) wherein the consensus layer upgraded from Capella to Deneb consensus client specifications.

  • Root cause: For context, the Deneb consensus specs (subsequent to the fork) for ExecutionPayload container contains 17 fields compared to that for Capella consensus specs (prior to the fork) which contains 15 fields. The addition of these two new fields changes the Merkle tree height from 4 to 5 as the number of fields goes over the next power of two of (2^4 = 16 fields). We found that our implementation of the proofgen library was not compatible with the new set of consensus layer client specifications included as part of the Deneb fork due to the additional 2 fields.

    More specifically, the proofgen library assumed a hard-coded Merkle tree height of 4 for the ExecutionPayload container, meaning that it expected Merkle proofs of fields within the container to be of precisely length 4. However, the Deneb fork changed the specification in a way that modified these proofs to be of length 5, but only for epochs after the fork (i.e., proofs against old state still had to be of length 4). It is still necessary for some proofs to be made against the state in these past epochs, which occurred prior to this change. Due to this nuance, we were unable to resolve the issue by simply changing the hard-coded “4” to a hard-coded “5”. Fully resolving the issue ultimately required modifying the code to be more flexible, as it was necessary for it to be capable of flexibly handling both cases.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

  • Actions: We disabled native ETH restaking functions on the Goerli testnet web application to avoid further inconvenience to users. We are identifying the fixes necessary to restore the system, and will share an update once those changes are ready for implementation.

Investigating [Jan 18, 2024 - 16:25 PT]​

  • Incident Context: We became aware of an issue on the Goerli testnet related to the proof generation that supports native ETH restaking functionality. This incident was confined to the testnet, with no impact on our mainnet operations.
  • Impact: Staking users are not able to initiate EigenPod partial and full withdrawal on Goerli testnet.
  • Safety of User Assets: While we work on a fix, we assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident.
  • Actions: Our team declared this as an operational incident internally, and initiated a thorough investigation to understand the cause and impact of the issue.