Agave XDP Setup Guide

In this article, we show how to enable XDP (eXpress Data Path) in Agave and why it matters for upcoming network upgrades. As Solana raises block compute limits, the main bottleneck is Turbine, the block propagation protocol. XDP accelerates Turbine by bypassing parts of the Linux networking stack to cut copies and context switches. By following this setup guide, you can help improve your node’s performance and prepare the network for higher throughput.

Motivation

Core developers are working to activate SIMD-0286: Increase Block Limits to 100M by late 2025. Under current conditions, large validators can approach 150,000 outbound packets per second due to Turbine’s aggressive fanout (each node relays shreds to ~200 others). This load is especially heavy for highly-staked nodes which, by having  more leader slots, probabilistically send more packets than lower-staked nodes.

XDP: eXpress Data Path is a Linux kernel technology that allows developers to write high-performance networking code which bypasses the kernel’s usual packet handling path. This means fewer data copies and fewer context switches between user and kernel space. By handling packets directly with the network interface card in user space, XDP greatly reduces the overhead per packet.

Agave version 2.3.8 and later supports XDP for Turbine packet handling. XDP works most efficiently with a NIC that has XDP driver support. Early testing indicates that 100M-CU blocks are achievable if operators adopt XDP in validator operations, providing the headroom needed to scale block propagation.

Configuration

Before rolling out XDP on a production validator, you should test it on your setup and verify a few things:

• Driver Compatibility: No unexpected NIC driver or hardware issues when XDP is enabled on your system.

• Performance Gain: Confirm that performance is improved with the new configuration (e.g. lower CPU usage or higher throughput in Turbine’s retransmit stage).

• Metric Visibility: Verify that you can observe the retransmit-stage metrics, which show time spent sending shreds, to gauge the impact of XDP on network transmission.

To enable XDP in Agave, add the following command-line flags to your validator startup command (using Agave v2.3.8+):

--experimental-retransmit-xdp-cpu-cores 1
--experimental-retransmit-xdp-zero-copy
--experimental-poh-pinned-cpu-core 10

Note that --experimental-retransmit-xdp-zero-copy will avoid using socket buffers for data, but this is only possible when talking directly to the Network Interface Card (NIC). If using a bonded network interface, do not include the zero copy parameter.

Also note that when XDP is enabled, you must move the PoH thread to a different core on a separate numa node than the XDP core (the --experimental-poh-pinned-cpu-core 10 parameter handles moving the PoH thread to a different core). If PoH and XDP are on the same numa node performance will degrade due to contention.

Conclusion

Enabling XDP in Agave allows a validator to send data out more efficiently (fewer copies and syscalls), which translates to faster block propagation and more headroom for future growth. For now, we encourage a subset validator operators, especially nodes with XDP-capable NICs, to try it out and report any issues to #testnet-validators in the Solana Tech Discord. Thank you for contributing to Solana and helping the cluster prepare for 100M CUs.