Game of Zones was one of the most anticipated adversarial testnets to be launched in 2020.
At Persistence, we are trying our best to keep the community updated about the challenges and learnings of GoZ. This blog covers updates from Phases 1b and 2 of the Game of Zones.
Phase 1b
Phase 1a of Game of Zones saw amazing participation and community involvement from Validators, Developers, and Projects worldwide.
After facing some teething issues with the stability of the Hub during Phase 1a, The Game of Zones team decided to restart the competition as Phase 1b with the focus being on liveness. One of the key focus points of this Phase was the stability of the GoZ Hub. To ensure stability, Phase 1b was launched with some fixed parameters that tested each teams’ preparation for GoZ.
- The trust period restrictions were removed on the chain code.
- Every team was allocated 1.25 million Doubloons
- Gas prices were fixed at 0.0025 doubloons/gas
This meant that every team had to come up with a strategy to make sure that they set the lowest trust period while maintaining a connection to the Hub for the longest duration.
A Spam Reflection attack in Phase 1b:
We saw a very clever and effective attack in Phase 1b which killed more than 50 client connections. A not so adaptive Tendermint Mempool behaviour was exploited by an attacker who sent zero-fee transactions to fill up the mempool. However, this attack could have been avoided if participants had configured their nodes properly by setting a minimum gas price. On the GoZ Hub, there are no Validators that accept zero-fee transactions, but that’s not the case with the Cosmos mainnet (which is why this attack cannot take place on the mainnet). Since the attacker only sent zero-fee transactions, the validators didn’t accept these transactions and they sat in the mempool without being delivered or getting expired. This led to the problem of a full mempool and thus no new transactions could be added to the mempool which meant that no “update client txns” went through to the Hub. This eventually led to chains with shorter trust periods losing connections to the Hub.
Both the Persistence zones, Persistence and Audit.one, won Game of Zones Liveness reward. You can check out the list of all the winners here.
We would also like to congratulate all the winners of Phase 1.
Phase 2
The focus of Phase 2 was throughput and the winning team is the team that relays the most packets on the GoZ Hub as well as on the other zones in the GoZ networks. You can read about the scoring criteria for Phase 2 here.
One of the ideas of this phase was to test the performance of IBC under Tendermint limitations.
Before the start of Phase 2, it was expected that the GoZ Hub will be heavily flooded with transactions and it would be difficult for chains to relay packets to the Hub after a certain period. This meant that the efficiency of doing transactions between a zone and the Hub would be very low. Hence, the right strategy to win Phase 2 would have been to find the right balance between relaying packets between your zone and the GoZ Hub and from your zone to other zones in the network.
This would have resulted in racking up more points. But with the highest points (1 point) allocated for relaying packets to the Hub, most participants focused on relaying packets to the GoZ Hub.
The network saw very high amounts of traffic and seemed congested most of the time, some zones still managed to get transactions through to the Hub while others were struggling to do so. One of the zones to achieve this was Sentinel. Sentinel seemed to have a very good strategy in place. The competition for Phase 2 was a two-horse race between Stake.fish and Sentinel and the results will be announced soon by the GoZ team.
What we witnessed in Phase 2:
The GoZ Hub halted at Block height “23400” after seeing very large transaction volumes (as was expected before the start of Phase 2). To resume the GoZ Hub, the system running the Hub validator and sentry nodes needed to be upgraded. With the memory containing the state tree, channels, packets, and lite clients from about 100+ zones, the upgrade mandated the node memory sizes to be expanded to 64GBs.
After being down for more than 5–6 hours, the Hub started committing blocks again. However, it started lagging soon after the upgrade on every 100th block. What started as a lag of about a minute or two, later turned into a lag of approximately 15 mins as the chain evolved. This issue was resolved by changing the history state pruning strategy to prune nothing/disable pruning.
We witnessed very significant activity on the GoZ Hub with about 6 transactions per block which led to congestion and thus an unstable Hub.
We would like to thank the GoZ team for continuous optimization of the network to make this highly unstable phase an interesting and exciting challenge for all the participants. One of our learnings from this phase was an understanding of how IBC operates under high load and where optimizations are required.
Learnings from Phase 2:
- The aim of Phase 2 was to try to break the Hub with all the participant chains spamming IBC transactions. The Hub held up pretty well under load processing around 100 messages per second.
- The relayer and the IBC module does not fail even when under load. The real stress testing was done on Hub chain which had to connect with 125+ chains’ light clients and process transactions emitted by them.
- The history state pruning, when enabled, was causing the Hub node to slow down every 100 blocks, which is the time it would engage in the pruning process. This means that the I/O heavy activities like pruning need to be delayed when the application is under load, and modifications to the pruning strategy have to be adaptively decided.
- Clock drifts or differences in the server times of the block proposing nodes on the Hub and zone nodes cause the validation on the block header timestamps to fail, requiring a time syncing strategy between the participating nodes and chains.
- Problems faced during phase 2 like transactions with a large number of messages increasing block processing time, zero-fee transactions filling up Mempools, Hub node being slowed down by too many peer sync nodes and I/O and memory limitations of the validator nodes, might be some of the issues that only manifest in TestNets and will never carry over to the Mainnet. GoZ has helped uncover these problems to be tackled in the future.
- One could relay many more packets to the participant chains, which were almost load free, than to the Hub, which was receiving transactions from all the participating chains. This would mean that the Hub communication and bandwidth is a scarce resource and that might get reflected in the gas price in a real-life scenario.
Community Contributions
We would like to take this opportunity to acknowledge the contributions made by the Cosmos community. We have already seen some amazing visualizers, dashboards, and explorers by the community in Phase 1a, and the community hasn’t stopped working and contributing to Game of Zones and IBC.
We saw an IBC Wallet developed by the Chainapsis team. To learn more about it, please follow this guide on How to install Keplr IBC for Game of Zones.
The P2P team has been working tirelessly to provide real-time leaderboards to the participants.
The Map of Zones team has done a tremendous job with their visualizer, explorer, and leaderboard.
Looking forward to Phase 3
We believe that Phase 3 of GoZ might perhaps be the most important Phase of the whole event:
- Phases 1 and 2 were more like load tests of the IBC implementation while Phase 3 will be a test of the protocol itself.
- The findings/results of phase 3 will progress the understanding of the protocol for the participants and will be a great learning experience for the observers who are not directly participating in the event too.
- Phase 3 will be the validation of the IBC protocol as a first of a kind blockchain interoperability protocol, which will be of significant importance to the DApp building community in general.
- The participants on top of trying to test the logical sanity of the protocol will also be trying to figure out the most novel use cases out of IBC which might positively lead to a Cambrian explosion of new ideas/projects in the Cosmos ecosystem.
The vision of Interoperable Blockchains doesn’t seem too far now, and as participants in GoZ, we are glad to be witnessing experience of a lifetime. We are in awe of what IBC can and will do for the world of Blockchains.
About Persistence
Persistence is a Tendermint-based, specialised Layer-1 network powering an ecosystem of DeFi applications focused on unlocking the liquidity of staked assets.
Persistence facilitates the issuance and deployment of liquid-staked stkASSETs, allowing users to earn staking rewards while participating in DeFi primitives, such as lending/borrowing and liquidity provisioning on DEXs.
Persistence aims to offer a seamless staking and DeFi experience for PoS (Proof-of-Stake) users and enable developers to build innovative applications around stkASSETs.
Join Our Movement
Twitter | LinkedIn | Telegram | YouTube | Reddit | [email protected]