30th July – 6th August
Check out the latest from Humanode's development team:
🔺 Finality incident analysis
This month, we faced a finality stall, which was quickly resolved, and the chain is back to normal. Our incident analysis confirmed the root cause. The node responsible for the primary slot (A1) took unusually long to distribute its block. Consequently, the chain continued with the block produced by the secondary slot node (B1), which produced two more blocks (B2, B3). Nodes began voting to finalize B1 on the fork (B1, B2, B3). Meanwhile, A1 continued producing blocks on the best chain (A1, A2, A3, etc.), leading to a chain reorganization. Finalization on B1 remained in progress but couldn't be finished since B1 was not part of the best chain. The Parity team acknowledged the complexity of this issue.
Since then, we have improved our internal alert systems to promptly respond to potential finalization lags. The code related to Grandpa logic has been deeply studied, and we issued a fix for the Grandpa RPC endpoint in mainnet 119 runtime.
Last week, we started experiencing frequent finalization stops again, prompting the creation of a tracking issue, Humanode Issue #1104, to actively track all occurrences. The Parity team has actively engaged in resolving the problem, and our research has been instrumental, in leading to a proposed fix from their side. The regression test, which checks the logic for block finalization after chain reorganization, was used to validate the fix. The pull request containing the fix and our test was merged: Polkadot SDK PR #5153.
However, the fix was not rolled out immediately due to potential unknown impacts on certain modules. It was decided to first reproduce the problem and then observe network behavior with the fix. Last week, we successfully reproduced the block finalization stop within the dev environment.
According to our plan, if the fix shows specific results in the dev environment without any side effects, we will promptly roll it out to the mainnet.
🔺Gossip fix
Additionally, this week, we rolled back an update to improve the state of gossip. For more detailed information, visit the Humanode GitHub.
🔺 Exploring CVM attestation
We've collaborated with the Rustwasm team to address some Web WASM/Rust infrastructure issues. Stay tuned for the new version of the
wasm-bindgen crate, which will include fixes for API generation and the async fn bindgen definitions.🔺 Introducing a new flow for the web app
We continued integrating the new flow for the web app, including a transition to a new authorization signature format and an improved error display format for users, providing detailed text on why the error occurred.
🔺 Front-end updates
We have updated the active validators counter on the mainnet landing.