ZOOT RGS integrates with multiple external systems. Their delivery guarantees are not uniform.

Some providers retry aggressively. Some retry once. Some never retry. Some resend identical payloads with new identifiers. Some send the same identifier for different transitions.

If your system assumes each message arrives exactly once, sooner or later it will pay someone twice.

What matters is not that each request arrives once. It is that each payout happens once.

Delivery vs Effect

Network delivery is at-least-once. That is the default in distributed systems.

Trying to enforce exactly-once delivery across unreliable boundaries introduces more complexity than it removes. You cannot prevent retries. You cannot prevent timeouts. You cannot prevent duplicate sends.

What you can do is make sure the same transition never produces two ledger entries.

So instead of asking:

Was this request delivered once?

You ask:

Has this economic effect already been applied?

That requires modeling the transaction explicitly.

Modeling the Effect, Not the Request

Each transaction is modeled as an aggregate. The aggregate does not store request history. It stores applied transitions.

Minimal shape:

type RoundState = {
  status: "NEW" | "BET_REGISTERED" | "FINALIZED"
  appliedKeys: Set<string>
}

Every transition carries a deterministic idempotency key. The aggregate tracks which keys have already been applied.

Rehydration reconstructs applied transitions from the event stream.

If a provider resends the same result five times, the same key is evaluated five times. The aggregate state does not change after the first.

We don’t deduplicate requests. We stop the same transition from being applied twice.

Why Idempotency at One Layer Is Not Enough

Idempotency must exist in two places:

  • At the aggregate transition level.
  • At the ledger mutation level.

If only the aggregate enforces idempotency but the ledger does not, retries during timeouts can produce duplicate credits.

If only the ledger enforces idempotency but the aggregate does not, lifecycle violations can occur.

Both layers must use the same deterministic key.

Event Sourcing Was Not About Audit

Event sourcing is often justified for auditability. That was secondary.

The primary reason was deterministic rehydration.

Mutable state invites ambiguity. If a row says FINALIZED, you cannot infer how it arrived there. If reconciliation fails, you must reconstruct history from logs.

With an append-only stream, the sequence of transitions is explicit.

If a dispute arises, you replay the stream. If a ledger mismatch appears, you replay the stream. If a bug is fixed, you replay the stream.

The events are the source of truth, and the current state is computed from them. That made incidents much easier to handle.

Timeouts and Ambiguity

The most dangerous case is a timeout during a ledger credit.

Did the credit succeed? Did it not? The network does not tell you.

If you assume failure and retry without idempotency, you duplicate funds.
If you assume success and do nothing, you risk missing a credit.

The only stable position is:

  • Retry with the same deterministic key.
  • Let the ledger decide.

We can live with not knowing for a few seconds. We can’t live with paying twice.

Stability Under Integration Growth

As integration count increases, retry behavior and error semantics diverge.

If each integration introduces special-case handling in core logic, the transaction model becomes fragile.

The aggregate lifecycle must remain constant regardless of provider behavior.

Provider quirks stay in the adapters, and the core model stays the same. That is what let us keep adding providers without breaking the ledger.