Strengthening Solana: The Invalidator Team

Introduction

At Anza, our commitment to Solana goes beyond core validator development—we challenge the protocol's limits to fortify its defenses. As the creators behind the Agave client that powers Solana, we understand that building a resilient blockchain requires more than innovation and new feature development; it demands rigorous, adversarial testing. Enter the Invalidator Adversarial Team, an in-house "redhat" team dedicated to probing for vulnerabilities and reinforcing the network’s reliability.

Origins and Purpose

The Invalidator Adversarial Team was formalized after the February 2024 outage that underscored the need for deeper, more focused testing and proactive resilience measures. The team's mission is straightforward: keep the network up by simulating extreme, real-world attack scenarios to expose vulnerabilities before bad actors do.

Simulating Real-World Attacks

One of the team's primary focuses has been designing and deploying Denial of Service (DoS) attacks against the various network ports to stress-test backpressure handling. This area, which posed challenges to the Solana Labs validator in 2022, has now seen significant improvements. The network's ability to maintain high transaction throughput and low confirmation times even during times of memecoin mania is a testament to these advances.

Crafting the Attacks

The Invalidator team uses sophisticated methods to simulate malicious activities:

• Adversarial Simulations: By crafting test scenarios that mimic both malicious users and block producers, the team can rigorously test the network under potentially catastrophic conditions. It also enables a deeper understanding of the capital requirements (e.g. amount of SOL) required to carry out different attacks.

• Protocol-Level Flexibility: These simulations are integrated at the protocol level, so they can be applied to any validator client, including Agave and alternatives like Firedancer, without needing special hooks or custom modifications. This approach allows seamless testing across implementations and ensures that vulnerabilities can be identified without bias toward specific validator software. This is essential in ensuring new validator clients coming online inherit the lessons of the past.

Economic Stress Testing

Another critical aspect of the Invalidator Team's work is block curation. By designing blocks that stress-test the economic model of operation pricing, we can ensure the safe expansion of block space while maintaining stability when increasing Compute Units (CUs) per block. This helps validate the network's block level constraints, even as validator hardware and traffic patterns continue to evolve.

Network Robustness Under Packet Loss

The team also experiments with network packet loss and induced delays to monitor how these conditions impact tower heights and confirmation times. These tests provide insights into how well the network holds up under suboptimal conditions, revealing opportunities for further optimization and resilience improvements.

Load Generation: Current and Future Goals

One of the most exciting areas of growth is in load generation. Recent enhancements have made it possible to simulate more complex synthetic loads that go beyond simple token transfers. This includes transactions that involve loading, allocating, and writing account data, as well as making cross-program invocation (CPI) calls. This level of load generation helps in understanding how the system behaves when facing real-world transaction types and pushes the boundaries of the system's resilience.

Going forward, the team aims to continue improving load generators to pattern match real mainnet load. These synthetic representations can serve as a baseline from which to stress specific dimensions and expose new potential vulnerabilities. Properly capturing these aspects of traffic is essential to ensure that upcoming features and releases are robust and secure.

Field Testing at Scale

The team's work isn't limited to controlled environments. Adversarial tests are conducted on private clusters and also on Solana's testnet, which operates at scale with thousands of nodes. This real-world testing allows the team to evaluate how the network behaves under true at-scale conditions, making it an invaluable part of Anza's comprehensive validation strategy.

Highlight: Slashing Development

A notable example of the team's work is the progress being made on slashing that included a validity proof being generated on testnet (https://explorer.solana.com/tx/2A6cdwon8673NKfDE4oKFjovvyp613LZKkCPbDYchxUGWoj5RkLqpCfQpmv1tkGqKZx9YLKTtNBBrRxWKnSLpeo1?cluster=testnet). This demonstration detected a misbehaving node from our redhat nodes, proving that slashing can be effectively enforced as a deterrent and protection mechanism against intentionally malicious activity.

Conclusion

The Invalidator Adversarial Team's mission goes beyond breaking Agave — it’s about ensuring that Solana the protocol remains strong and resilient under all conditions. From economic stress tests to packet loss resilience, the team's comprehensive approach has already contributed to major strides in network security and performance. With a focus on continuously improving load generation and adapting to new challenges, Anza is poised to maintain Agave & Solana’s reliability and keep it prepared for future growth.

Stay tuned for further insights as we continue pushing the boundaries of adversarial testing and fortifying Solana's ecosystem.