Simulating decoupled consensus
The test network
- A Prysm implementation built to test the networking requirements. It sends every object on the wire at
its correct size to the other nodes.
- Primary focus: can finality (FFG) votes be aggregated in time?
Mapping the devnet to mainnet: one attestation subnet
Mainnet model
Super 196 · 44%Home 251 · 56%
Our devnet
Super 200 · 40%Home 300 · 60%
| Mainnet | Devnet | change |
|---|
| Nodes in a subnet | 447 | 500 |
+12% |
|---|
| Validators per subnet per slot | 1,968 | 2,518 |
+28% |
|---|
| Super node, down / up | 512 / 512 Mbps | |
|---|
| Home node, down / up | 50 / 25 Mbps | |
|---|
A note on gossipsub
- A node forwards a new message to its mesh peers at once (mesh degree D = 8).
- Every heartbeat (700 ms), the node picks 25% of its other topic peers and sends each an IHAVE with the IDs
of the messages it has.
- The peer replies with IWANT for the messages it does not have.
- This is naive today: a node asks many peers for the same attestation, and sends the same IHAVE to many
peers.
- The beacon node has no control over IHAVE and IWANT messages.
Subnet structure same as mainnet today
- 64 committees over 64 subnets.
- Subrounds align with the slots.
- Every beacon node subscribes to 2 subnets.
- This keeps the number of beacon nodes in the subnet the same and keeps the same stable backbone.
Scenario 1 · staggered voting
Scenario 1: arrival times
Scenario 1: data usage, super node
objectFFG votesGoldfish votesPayloadPTC votesData columnsSync committeeBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)IHAVEparts metadataIWANT
Scenario 1: data usage, home node
objectFFG votesGoldfish votesPayloadPTC votesData columnsSync committeeBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)IHAVEparts metadataIWANT
EIP-8334: bundled attestation propagation
Attestation data and committee index, 136 BSignature, 96 BValidator index, 8 B
4 single attestations960 B
Attestation dataSignatureAttestation dataSignatureAttestation dataSignatureAttestation dataSignature
Bundle of 4 votes552 B
Attestation dataSignature 1Signature 2Signature 3Signature 4
Bytes per vote
240 B → 138 B
Scenario 2 · staggered voting + EIP-8334
Caveat: these numbers are worse than they would be in practice. The queues in this implementation were too small, and some messages were dropped.
Scenario 2: arrival times
Scenario 2: data usage, super node
objectFFG votesGoldfish votesPayloadPTC votesData columnsSync committeeBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)IHAVEparts metadataIWANT
Scenario 2: data usage, home node
objectFFG votesGoldfish votesPayloadPTC votesData columnsSync committeeBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)IHAVEparts metadataIWANT
Home node: FFG vote delay, scenario 1 vs 2
Caveat: scenario 2 is worse than it would be in practice. The queues in this implementation were too small, and some messages were dropped.
Home node: Goldfish vote delay, scenario 1 vs 2
Caveat: scenario 2 is worse than it would be in practice. The queues in this implementation were too small, and some messages were dropped.
Home node: data usage, scenario 1 vs 2
FFG · scenario 1 16.9 MB sent · 17.7 MB receivedFFG · scenario 2 4.8 MB sent · 6.5 MB receivedEverything else · scenario 1 3.8 MB sent · 4.0 MB receivedEverything else · scenario 2 3.5 MB sent · 3.7 MB received
Scenario 3 · three committees per subnet per slot
Scenario 3: arrival times
Scenario 3: data usage, super node
objectFFG votesGoldfish votesPayloadPTC votesData columnsSync committeeBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)IHAVEparts metadataIWANT
Scenario 3: data usage, home node
objectFFG votesGoldfish votesPayloadPTC votesData columnsSync committeeBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)IHAVEparts metadataIWANT
Staggered publishing vs multiple committees
- Multiple committees have a nice symmetry: at any time in the slot, FFG votes and aggregates flow in
parallel.
- Multiple committees have more aggregate traffic.
Simulation for 4-slot round and 8-slot finality
eth-slot-sim on Shadow · 1,000 nodes (200 super, 800 home) · 1M validators · 64 FFG subnets · EIP-8334
Arrival times: super nodes
eth-slot-sim on Shadow · 1,000 nodes (200 super, 800 home) · 1M validators · 64 FFG subnets · EIP-8334
Arrival times: home nodes
eth-slot-sim on Shadow · 1,000 nodes (200 super, 800 home) · 1M validators · 64 FFG subnets · EIP-8334
Data usage: super node
objectFFG votesGoldfish votesPayloadPTC votesData columnsBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)parts metadataIHAVEIWANT
Data usage: home node
objectFFG votesGoldfish votesPayloadPTC votesData columnsBlockFFG aggregatesOther topics and control
kinddata (publish / partial message)parts metadataIHAVEIWANT