Skip to main content

[Goerli Testnet] Incident Update: EigenDA Blobs Not Retrieving

· 2 min read

Resolved [Feb 19, 2024 - 15:00 PT]​

  • This incident has been resolved.

  • Additional Support: If you are still encountering issues relating to EigenDA blob retrieval, please share via the EigenLayer Discord #support-eigenda channel.

Monitoring [Feb 19, 2024 - 15:00 PT]​

  • Actions: We identified the root cause of the issue as a race condition in the disperser that caused blobs to be encoded twice, resulting in the blob's dispersal metadata changing between one call to GetBlobStatus() and the next. We have successfully deployed a fix to testnet, and we are continuing to monitor the service. We don’t expect rollup operators to face any further issues relating to this, but testnet rollup operators may need to redeploy their Goerli chains with a new genesis in order to continue producing blocks.

Investigating [Feb 16, 2024 - 15:00 PT]​

  • Incident Context: We became aware that the Goerli EigenDA network was reporting blobs as confirmed and then later failing to retrieve these blobs. The error messages on retrieval include the string "there is no metadata for batch".
  • Impact: Testnet rollup sequencers and nodes experience chain halts and are unable to recover without DA backup.
  • Actions: Our team has declared this as an operational incident internally, and is initiating a thorough investigation to understand the cause of this issue.

[Goerli Testnet] Incident Update: EigenDA Operator Opt-In Churner function impacted on Goerli testnet

· 2 min read

Resolved [Feb 16, 2024 - 18:15 PT]​

  • This incident has been resolved.

  • Additional Support: If you are still encountering issues with opting-in into EigenDA, please share via the EigenLayer Discord #support-operators channel.

Monitoring [Feb 16, 2024 - 15:30 PT]​

  • Actions: We identified the root cause that resulted in this incident with the Churner service. We have successfully deployed the fixes, and are continuing to monitor the service. We don’t expect operators to face any further issues related to opting-in of EigenDA, and can resume normal operations on Goerli testnet.

  • Additional Support: If you (EigenLayer operators) are still encountering issues with the EigenDA opt-in functionality, please raise your issue via the EigenLayer Discord #support-operators channel.

Investigating [Feb 15, 2024 - 15:00 PT]​

  • Incident Context: We became aware of an issue on the Goerli testnet related to the EigenDA Churner that is not available. This incident is confined to the Goerli testnet.
  • Impact: EigenLayer operators are unable to opt-in to EigenDA on Goerli testnet. Request to the Churner service are failing with a 404 error response. Existing nodes are not impacted.
  • Actions: Our team declared this as an operational incident internally, and initiated a thorough investigation to understand the cause and impact of the issue.

[Goerli Testnet] Incident Update: Native Restaking Withdrawals Impacted on Goerli testnet

· 2 min read

Resolved [Feb 13, 2024 - 16:30 PT]​

  • This incident has been resolved.

  • Additional Support: If you are still encountering issues with EigenPod withdrawals (partial or full), please share via the EigenLayer Discord #support-restakers channel.

Monitoring [Feb 07, 2024 - 16:30 PT]​

  • Actions: We identified some scalability bottlenecks that led to this incident. We deployed fixes this week, restored normal opertation, and are continuing to monitor our service. We don’t expect users to face any further issues, and can resume normal operations of partial and full withdrawals from EigenPods on Goerli testnet.

  • Additional Support: If you are still encountering issues with EigenPod withdrawals (partial or full), please raise your issue via the EigenLayer Discord #support-restakers channel.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

Investigating [Feb 01, 2024 - 11:50 PT]​

  • Incident Context: We became aware of an issue on the Goerli testnet related to withdrawal proof generation that supports native ETH restaking functionality. This incident was confined to the testnet, with no impact on our mainnet operations.
  • Impact: Staking users are not able to initiate EigenPod partial and full withdrawal on Goerli testnet.
  • Safety of User Assets: While we work on a fix, we assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident.
  • Actions: Our team declared this as an operational incident internally, and initiated a thorough investigation to understand the cause and impact of the issue.

[Goerli Testnet] Incident Update: EigenPod Withdrawals Impacted due to Goerli Hard Fork

· 5 min read

Resolved [Jan 26, 2024 - 13:00 PT]​

  • This incident has been resolved.

  • Additional Support: If you are still encountering issues with EigenPod withdrawals (partial or full), please share via the EigenLayer Discord #support-restakers channel.

Monitoring [Jan 25, 2024 - 12:00 PT]​

  • Actions: We successfully deployed the fixes yesterday, and are continuing to monitor the results for an additional day. We don’t expect users to face any further issues, and can resume normal operations for partial and full withdrawals from EigenPods on Goerli testnet.

  • Additional Support: If you are still encountering issues with EigenPod withdrawals (partial or full), please raise your issue via the EigenLayer Discord #support-restakers channel.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

  • Next Update: Jan 26, 2024 - 11:00 PT

Monitoring [Jan 24, 2024 - 16:30 PT]​

  • Restoration of EigenPod Functionalities on Goerli: We identified and deployed a number of changes needed to the EigenPod contracts (see PR: Layr-Labs/eigenlayer-contracts/pull/395), and to the proof generation implementation (tip of this branch). At around 13:05 PT, we verified that this successfully restored both partial and full withdrawal functionalities on the Goerli network (successful Deneb withdrawal on Deneb state - Etherscan transaction (timestamp of the withdrawal at validatorIndex:668660, timestamp:1705595700, post deneb). We re-enabled the native staking (EigenPod) functionality on the Goerli web application at 13:30 PT.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

  • Actions: We have successfully deployed the fixes, and are monitoring the results. We don’t expect users to face any further issues, and can resume normal operations for partial and full withdrawals from EigenPods on Goerli testnet.

  • Next Update: Jan 25, 2024 - 12:00 PT

Identified [Jan 19, 2024 - 10:40 PT]​

  • Summary: The issue was identified as being triggered by the Goerli hard fork event on 01/17/2024 (geth release v1.13.9 block height 1705473120) wherein the consensus layer upgraded from Capella to Deneb consensus client specifications.

  • Root cause: For context, the Deneb consensus specs (subsequent to the fork) for ExecutionPayload container contains 17 fields compared to that for Capella consensus specs (prior to the fork) which contains 15 fields. The addition of these two new fields changes the Merkle tree height from 4 to 5 as the number of fields goes over the next power of two of (2^4 = 16 fields). We found that our implementation of the proofgen library was not compatible with the new set of consensus layer client specifications included as part of the Deneb fork due to the additional 2 fields.

    More specifically, the proofgen library assumed a hard-coded Merkle tree height of 4 for the ExecutionPayload container, meaning that it expected Merkle proofs of fields within the container to be of precisely length 4. However, the Deneb fork changed the specification in a way that modified these proofs to be of length 5, but only for epochs after the fork (i.e., proofs against old state still had to be of length 4). It is still necessary for some proofs to be made against the state in these past epochs, which occurred prior to this change. Due to this nuance, we were unable to resolve the issue by simply changing the hard-coded “4” to a hard-coded “5”. Fully resolving the issue ultimately required modifying the code to be more flexible, as it was necessary for it to be capable of flexibly handling both cases.

  • Safety of User Assets: We assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident. There has been no impact on mainnet operations.

  • Actions: We disabled native ETH restaking functions on the Goerli testnet web application to avoid further inconvenience to users. We are identifying the fixes necessary to restore the system, and will share an update once those changes are ready for implementation.

Investigating [Jan 18, 2024 - 16:25 PT]​

  • Incident Context: We became aware of an issue on the Goerli testnet related to the proof generation that supports native ETH restaking functionality. This incident was confined to the testnet, with no impact on our mainnet operations.
  • Impact: Staking users are not able to initiate EigenPod partial and full withdrawal on Goerli testnet.
  • Safety of User Assets: While we work on a fix, we assure all users that all funds on Goerli and mainnet have been safe and secure throughout the incident.
  • Actions: Our team declared this as an operational incident internally, and initiated a thorough investigation to understand the cause and impact of the issue.