A block one Solana client killed and the other voted for

A single Solana slot leader (the validator whose turn it is to produce a block) can build one block that every Agave node marks dead and every Frankendancer node votes for. The network splits along client lines. Every Frankendancer node stops finalizing, and if neither client holds two-thirds of the stake, the whole chain does.

No stolen keys, no supermajority. Just its own turn as leader and a block it signs itself.

The program rated it High, confirmed it and paid it.

How the two clients should agree

Solana has two validator clients. Agave is the reference. Frankendancer is a hybrid: Firedancer’s fast C networking in front, Agave’s Rust runtime and storage behind.

Blocks travel as shreds, small signed pieces grouped into FEC sets (bundles with error-correction data, so lost pieces can be rebuilt). Each FEC set has a Merkle root, a single hash that stands for its contents.

With chained Merkle roots, each FEC set also commits to the root of the set before it, so a block’s sets form a chain.

When Agave spots a broken link inside a block, and the validate_chained_block_id feature is on (it is on mainnet), it marks the whole slot dead. That slot never reaches replay or voting. Agave’s own code comments say this step exists not for Agave, but to keep other clients consistent with it.

That last line is the whole game. Consensus only holds if every honest client gives the same yes-or-no for the same bytes.

Where it breaks

Frankendancer verifies and reassembles incoming FEC sets, then hands them to Agave’s storage through a small piece of glue code (the adapter that lets Firedancer’s C call into Agave’s Rust).

The glue knocks on the wrong door. It calls an Agave insert function marked test-only, which routes to the path a leader uses to store shreds it produced itself.

That path still runs the chained-root check and collects any conflict into a list. Then it returns success and drops the list on the floor.

The real network path walks that same list and kills the slot on a chained-root conflict. Frankendancer computes the same conflict and throws it away.

// what Frankendancer's path effectively does
let (summary, conflicts) = do_insert(shreds);
return summary;              // conflicts silently dropped

// what the real network path does
let (summary, conflicts) = do_insert(shreds);
for c in conflicts { handle(c); }   // this is where the slot is killed
return summary;

So this is not a missing check. The check runs; its side effect never fires.

Splitting the network

1. An attacker runs a staked validator and waits for its turn as leader in the normal rotation. Any stake that earns a slot is enough; the reproduction used a small one.
2. In its slot, the attacker builds a block with valid signatures and valid per-piece Merkle proofs, linked to the correct parent. Only one interior FEC set’s chained root is wrong.
3. It sends the block as ordinary shreds to every validator.
4. Agave sees the broken link at ingest, marks the slot dead, and builds a sibling fork that skips it.
5. Frankendancer’s own shred checks look inside one FEC set at a time, so a bad link between sets slips through. The insert path behind the glue spots it and drops it. Replay only checks the link to the parent, which is correct. So Frankendancer executes the block and votes for it.
6. No conflict proof is broadcast for this case, so nothing nudges Frankendancer to reconsider. Solana’s vote lockout (a rule that makes switching forks after voting increasingly costly) pins each client to its fork.

The damage

The two clients now sit on conflicting forks, split by client identity, not stake. Frankendancer validators are bound to a fork the Agave majority will never extend, so they cannot finalize. That is the minimum: every Frankendancer node stalls, from one block.

The worse case: whenever neither client holds two-thirds of the stake, neither fork can finalize and the whole chain stops finalizing new blocks.

I ran it live: one Agave and one Frankendancer validator at equal stake, plus a small attacker leader. Agave marked attack slot 132 dead and built past it into the 140s. Frankendancer voted for 132 and kept building on it.

Finalization on both sides froze at slot 99, below the attack, for the rest of the run.

Agave’s dead-slotting is permanent, so the split doesn’t have to heal on its own. Repeat it across the attacker’s recurring slots and the gap deepens until rejoining could take a coordinated restart at an agreed root. That is a hard fork.

Where the fix belongs

Any Frankendancer path that runs Agave’s conflict detection must also apply its side effect before replay or voting.

The report offered two routes: feed network shreds through Agave’s own network intake, dead-slot step included, or pass the dropped conflict list back across the boundary and act on it there.

Either way, the fix belongs in Firedancer’s glue code. Upstream Agave already handles network shreds correctly.

The team confirmed and paid. The record does not say whether a fix has shipped.

Lessons for multi-client chains

• A check with no side effect is decoration. Find a conflict, return success anyway, and you may as well not have looked.
• Reusing an internal function across a trust boundary is where these bugs live. The leader-only insert path was fine for leaders and became a consensus bug once network shreds went into it.
• In a multi-client network, the invariant is not “we agree on the bytes” but “we reach the same decision about those bytes.” One dropped result breaks it.