Operator Runbook

How to deploy, observe, and recover the off-ledger operator services that run the reference DEX. The through-line for every recovery decision below: the ledger is the source of truth. Every fact an operator needs to explain a trade or rebuild a service lives on-ledger, replicated by the synchronizer; the operator backend’s own SQLite is a projection over it. That single property is why most recovery here is “rebuild a cache”, not “restore a database”.

Specific cluster topology (cantond / participants / synchronizer config) is a Canton operational concern, not a DEX one — see Out of scope.

Roles and party model

The reference uses four logical roles and can involve many trader, LP, and asset-admin parties. Keeping control roles separate is the recommended production posture; a local learning instance may intentionally share a party where the setup guide says so.

Party	Owns	Signs
operator	DexPair, Order, MatchedTrade, SettledTrade, Pool, PoolState, PoolSlice, PoolRules, OrderMatchExecution	All DEX-side market state
lpRegistrar	LPTokenPolicy, LP registry config (reference: InstrumentConfig)	Mint/burn supply and LP-token policy
admin	Base/quote registry config (reference: InstrumentConfig), AllocationFactory, SettlementFactory	Allocations, settlement batches, registry-side mint/burn/transfer
trader / lp	OrderFundingRequest, Rfq, and the deposit/receipt/burn allocations they author against a LiquidityAllocationRequest	Their own intents and allocation accepts

The traffic-cost split (called out in module headers) follows the role ownership: each role pays for the transactions it submits. The party wiring is read from env at boot (CANTON_OPERATOR, CANTON_LP_REGISTRAR, CANTON_ADMIN); see .env.example.

Deployment checklist

In rough order of dependency:

1. Allocate parties. operator, lpRegistrar, base-asset admin, quote-asset admin, and any traders / LPs you want to onboard.
2. Bring up real registries. Run the idempotent bootstrap-registry.ts path to create Registry.V2 plus each InstrumentConfig, or configure a conforming external Token Standard V2 registry. MockAllocationFactory and MockSettlementFactory are Daml-test fixtures only: they do not create or move holdings and must not be used as a deployment recipe. When asset admin and LP registrar differ, record both registry cids for the backend’s per-admin factory mapping.
3. List trading pairs. Operator creates a DexPair per pair with the chosen tradingMode and feeModel. These fields are listing metadata in this revision; they do not independently gate pool/order terminal choices.
4. Create LP infrastructure (per pool).
   • lpRegistrar creates the LP token’s registry-specific instrument definition. In the reference registry this is one InstrumentConfig per pool
   • lpRegistrar creates the LPTokenPolicy for the full { admin = lpRegistrar, id = lpInstrumentId } instrument identity
5. Create pools. Operator creates the immutable Pool, the hot PoolState in PS_Unfunded, and the operator-side PoolRules / co-controlled PoolLiquidityRules. The first LP uses the same add-liquidity DvP request/allocate/settle flow as later LPs; the settle creates the first PoolSlice contracts and transitions the state to PS_Active.
6. Open the order book / swap surface. Once registries and holdings are live, pools are funded, PoolRules is active, and the operator’s off-ledger routing policy allows the market, traders may submit OrderFundingRequest, liquidity adds/removes via the DvP /request flow, Rfq, etc.

The focused PoolWorkflowTests.daml, OrderWorkflowTests.daml, TradeWorkflowTests.daml, and ChoiceContextWorkflowTests.daml walk DEX choices against mock factories, but those fixtures do not hold value. They are not deployment validators. Use the testnet guide for bring-up and the Daml proof map plus the self-contained live AMM round trip for value movement.

Operator-driven cleanup (on-ledger)

Iterated allocations put settlement authority in the executor’s hands, so the DEX application layer must constrain every permitted use. The choices below are app-owned cleanup hooks on the ledger; an operator service drives them on a schedule. None of them fabricate state — each is a real contract choice, so the cleanup surface is auditable in one place.

Stale or expired orders

• Order_Cancel (operator-driven, Order.daml): cancels the bound allocation via Allocation_Cancel, releasing the trader’s locked holdings back to their authorizer account. The operator’s sweep uses it both for orders past expiry (checked off-ledger when scheduling the cancel) and for operator-initiated takedowns (compliance, fat-finger cancels, pair de-listing).
• This sweep is not the trader’s only custody exit. GTC order allocations are uncommitted and authorizer-withdrawable at any time; expiring order allocations become authorizer-withdrawable after their deadline. A withdraw may leave stale order state for the operator to clean, but settlement against the consumed allocation fails safely.

Stale swaps

• A swap allocation is terminal and uncommitted. If its bound pool-state or slice contract becomes stale before settlement, the trader can exercise Allocation_Withdraw; the operator must not retry with altered trader legs.

Stale RFQs and quotes

• An RFQ past expiresAt is inert: Rfq_Accept asserts currentTime < expiresAt, so nothing can settle against it. Quote contracts stay until their own expiresAt; the operator sweep exercises RfqQuote_Withdraw (dealer-driven) or lets quotes age out.
• Rfq_Cancel (trader-driven): the trader retracts before any quote acceptance.

Stuck matched trades

• MatchedTrade_Cancel (venue-driven, MatchedTrade.daml): archives outstanding TradeAllocationRequest contracts and exercises Allocation_Cancel on any allocations that have already been created. Use when one leg’s authorizer rejects or times out before settlement.

Pool maintenance

• PoolRules_Pause (operator, PoolRules.daml): halts new swaps and liquidity actions while leaving reserve allocations in place. Useful for upgrades and incident response.
• PoolRules_Resume (operator): exits Paused back to Active.
• Remove-liquidity is slice-local: the PoolLiquidityRules_SettleRemoveLiquidity settle sources a routine withdrawal from at most one boundary re-allocation per side. The architecture and workflows docs describe the invariant; PoolLiquidityRulesTests.daml exercises the multi-slice boundary case (testDvpMultiSliceRemove).
• LP redemption requires both operator and LP registrar availability. This reference has no holder-only emergency redemption path; see Liquidity and custody.

LP supply reconciliation

• PoolState_RecordLPSupply (lpRegistrar, PoolState.daml): pushes the registrar-owned LP supply ledger back into the pool’s pricing state. Run after each mint/burn accept so add-liquidity quoting stays accurate.

Observability

The contract surface deliberately puts the audit-relevant facts on-ledger so operators do not need a parallel database to explain a trade.

Question	Where to look on-ledger
Why did this RFQ accept go to this dealer?	MatchedTrade.policyReceipt, also folded into SettlementInfo.meta via dex.policy.* keys
What pool fee was executed?	The immutable Pool.feeBps used by PoolRules; DexPair.feeModel is listing metadata and is not consumed by that choice
Where did this pool’s reserves come from?	Each PoolSlice is an Allocation CID, each carrying its admin, authorizer, and committed funding
What’s the current head slice / boundary candidate?	each active PoolSlice for the pool (query the ACS by poolId); the aggregate is PoolState.reserves.baseAmount/quoteAmount
Did this trader’s funding accept?	The OrderAllocationRequest archive event plus the corresponding Allocation create event
Why is this PoolRules_Swap failing slippage?	Call the quote endpoint before swap; the on-ledger choice re-validates against current reserves and minOutputAmount
Did this LP mint actually run?	PoolLiquidityRules_SettleAddLiquidity mints against the LP receipt allocation and records the resulting supply on LPTokenPolicy

Off-ledger telemetry the operator should also collect:

• Latency per explicitly named workflow boundary (OrderFundingRequest_Bind → Order_Fund, Rfq_Accept → MatchedTrade_Settle, or request → settlement around PoolRules_Swap).
• Failure counts per choice, especially slippage rejections, allocation conservation failures, and registry choice-context rejections.
• Slice-count distributions per pool side, to flag when consolidation maintenance would help reduce the slice list’s length.
• Pending request age: how long OrderAllocationRequest, LiquidityAllocationRequest and registry instruction records have been open without a downstream accept.

Every HTTP request carries an X-Request-Id (echoed back and stamped on each log line) and emits a structured, one-JSON-object-per-line record via lib/logger.ts — set LOG_LEVEL to tune verbosity. Errors and warnings go to stderr, everything else to stdout.

Indexer-backed endpoints

The operator backend ships with a polling indexer (indexer/index.ts) that projects ledger state into a local SQLite database (data/operator.db by default). These endpoints exist only when the server was started with a db handle; without one they return 503 indexer disabled.

Endpoint	Returns
GET /v1/trades?trader=&pair=&limit=	Matched-trade history including archived contracts (unscoped view is admin-only)
GET /v1/swaps?pair=&kind=&limit=	Per-rotation base/quote deltas + price after; kind ∈ swap (default) / add_liquidity / remove_liquidity / state_change
GET /v1/rfq/history?trader=&limit=	RFQ lifecycle events (open / accepted / closed)
GET /v1/price-history?pair=&hours= · GET /v1/stats/24h?pair=	Price points and derived 24h volume / change from the swaps table
GET /v1/admin/config	Operator KV (read open by default)
PUT /v1/admin/config (Bearer auth)	Set a KV key

A swaps row is not necessarily a swap: five different choices rotate a PoolState. The indexer polls the ACS and never sees a choice name, so it uses totalLpSupply as the discriminator (only an add or a remove moves it), in exact scaled-integer arithmetic so an LP mint that float subtraction would collapse to zero is never misclassified as a swap. Proven in indexer-pool-kind.test.ts (“sees an LP mint that float subtraction would lose entirely”) and indexer-projection-exactness.test.ts (the served magnitudes are the stored strings, digit for digit).

The indexer is single-flight and tolerant of restarts. Its own header states the guarantee:

// Crash safety: state is reconciled from current ACS on every tick,
// so a crash just means a missed poll, not a corrupt DB.

Set INDEXER_INTERVAL_MS to tune polling cadence (default 5s). Read scoping is enforced at the route: an unfiltered /v1/trades or /v1/rfq/history sweep names both counterparties, so it requires the admin token — proven in read-exposure.test.ts (refuses an unscoped read without the admin token).

Idempotency cache

Every command submission is wrapped by IdempotentLedger (indexer/idempotency.ts), keyed by commandId and stored in:

CREATE TABLE IF NOT EXISTS command_submissions (
  commandId TEXT PRIMARY KEY,
  submittedAt INTEGER NOT NULL,
  completedAt INTEGER,
  status TEXT NOT NULL,         -- 'pending' | 'ok' | 'error'
  resultJson TEXT
);

A retry with the same commandId returns the cached result if status='ok', rejects if it is still pending and younger than PENDING_STALE_MS (60s), and overwrites if the row is stale-pending or error. A same-commandId submit carrying different args is a replay conflict and is rejected rather than served a stale result — proven in idempotency.test.ts (“rejects a replay: same commandId, different args”). The cache survives operator restarts and is the recommended defence against double-fire across crash/replay boundaries. testnet-server sweeps rows past the 24h TTL once an hour.

Recovery procedures

Recovery starts from one distinction: what is authoritative versus what is a rebuildable projection. The on-ledger ACS is authoritative and replicated by the synchronizer. The operator backend’s operator.db holds only projections of it — with one exception, operator_kv, which carries runtime knobs (dealer whitelist, RFQ policy) that were never written on-ledger and therefore cannot be rebuilt from it.

flowchart LR
  ACS[("On-ledger ACS source of truth, synchronizer-replicated")]
  subgraph db["operator.db · local SQLite (WAL)"]
    PROJ["projections: trades · swaps rfq_history · pool_states"]
    IDEM["command_submissions (idempotency cache)"]
    KV["operator_kv dealer whitelist · RFQ policy"]
  end
  ACS -->|"indexer reconciles every tick"| PROJ
  PROJ -.->|"rebuildable on delete"| ACS
  IDEM -.->|"rebuildable"| ACS
  KV ==>|"off-ledger only — the one thing to back up"| BK[["operator backup"]]

Ledger errors are classified once, in the JSON-API driver’s errorFor (json-api.ts), into the LedgerErrorKind the rest of the backend reacts to. Only contention is retryable:

if (lower.includes("contention") || lower.includes("inconsistent")) {
  kind = "contention";
  retryable = true;
} else if (lower.includes("authoriz") || res.status === 401 || res.status === 403) {
  kind = "authorization";
} else if (res.status === 400) {
  kind = "validation";
}

Every operator write runs inside retryOnContention (submit-with-retry.ts), which retries only that class, with exponential backoff, up to five attempts:

if (e instanceof LedgerError && e.kind === "contention") {
  await sleep(delay);
  delay = Math.min(maxDelay, Math.floor(delay * 2));
  continue;
}
throw e;

Failure modes and recovery

Operational (infrastructure- and process-level) failures. For contract-choice rejections a trader or LP hits, see Contract-level rejections.

Symptom	Likely cause	Action
Operator backend crashed / was restarted	Process died; WAL keeps operator.db intact	None required. Indexer.start() reconciles from the current ACS on the first tick; the idempotency cache absorbs the dApp’s retry-on-restart.
Indexer endpoints stall; logs show [indexer] tick failed	Participant / JSON LAPI unreachable	Transient: the tick retries next interval — no corruption. Persistent: check the participant and CANTON_LEDGER_URL / token.
Operator writes fail with a transport LedgerError	Participant or synchronizer outage	The idempotency row is marked error; retry from the dApp once the participant recovers.
A write fails with a contention error after retrying	Two commands raced the same input UTXO and the five backoff attempts were exhausted	Resubmit; the on-ledger choice is safe to re-run once the contending commit lands.
operator.db corrupted / unreadable	Disk fault or partial write	Delete it and restart; the next tick rebuilds projections from the ACS. See Reference — this also drops operator_kv.
NOT_VALID_UPGRADE_PACKAGE on DAR upload	Smart-upgrade lineage broken	Revert the incompatible change or rename the package. See Reference.
A liquidity settle aborts on its supply-sync guard	LPTokenPolicy.totalSupply drifted from PoolState.totalLpSupply	Re-run PoolState_RecordLPSupply with newSupply = policy.totalSupply. See Reference.
Trader reports a missing holding	V2 holding not visible to the party’s query	Replay the registry’s Registry_RegisterInstrument / Registry_Mint events for that party via the EventLog interface.

Reference: multi-step procedures

The table cells above are one-liners for the failures that resolve in a step or two. These three need more.

operator.db corruption — rebuild from the ledger. SQLite runs in WAL mode (db.ts), so an ordinary crash leaves the file intact and nothing is needed. If the file is genuinely corrupt, delete it and restart: the indexer reconciles projections from the live ACS on the next tick. Two things do not come back — trade history older than the ACS-archive cutoff (it lived only in the indexer DB) and, because it lives in the same file, operator_kv. Restore operator_kv from backup after the rebuild (see Backup). Schema migrations are append-only and tolerant of a hand-repaired database, proven in indexer-migrations.test.ts (“a hand-repaired database can still advance”).

Smart-upgrade lineage break. Symptom: NOT_VALID_UPGRADE_PACKAGE on DAR upload. Either:

• Revert the offending change — add removed choices back as deprecated stubs, make new fields Optional, move new fields to the end of the record.
• Rename the package (e.g. canton-dex-trading → canton-dex). All existing contracts from the old name remain queryable but cannot be upgraded.

See Upgrade discipline for the lineage rules and the CI gate that catches a break before upload.

LP supply drift. LPTokenPolicy.totalSupply and PoolState.totalLpSupply are kept in lock-step: the DvP liquidity settles (PoolLiquidityRules_SettleAddLiquidity / _SettleRemoveLiquidity) rewrite both inline and assert they match on entry, so a divergence aborts the settle rather than corrupting reserves. Recovery: query the policy supply and re-run PoolState_RecordLPSupply with newSupply = policy.totalSupply.

Backup

The on-ledger state is the source of truth and is replicated by the synchronizer. operator.db is rebuildable from the ledger and does not need to be backed up for correctness; back it up only if you care about historical query performance during a rebuild. The one thing worth backing up is the operator_kv table — it carries the dealer whitelist, RFQ policy parameters, and similar runtime knobs that are not encoded on-ledger and cannot be reconstructed from the ACS.

Contract-level rejections

Rejections a trader, LP, or dealer hits at a contract choice — business-logic guards, not infrastructure faults. Each surfaces to the caller as the assert message shown; the operator’s job is to route the fix, not to override the guard.

Symptom	Likely cause	Remediation
FinalizedAllocation extra leg-sides exceed funding budget	Operator tried to settle more than the authorizer pre-committed	Re-quote: the swap or match math drifted from the budget. Fix off-ledger quoting state
Pool has no base slices / ... no quote slices	Pool drained to empty by Remove without entering Unfunded state	Inspect the slice list and reserves; if mismatched, escalate (the contract should prevent this)
LP tokens below minimum	LP’s minLpTokens slippage bound too tight	LP resubmits with a looser bound or smaller deposit
Output below slippage minimum	Reserve drift between quote time and submit	Trader resubmits with a looser bound, or operator routes through a different pool
Head output slice cannot cover swap	Head slice on the output side is smaller than amountOut	Operator should run consolidation, or split the swap across multiple smaller swaps
Pool must be Active	Pool was Paused (planned) or Unfunded (last LP exited)	If Paused: PoolRules_Resume after maintenance. If Unfunded: a new LP needs to complete add-liquidity request/allocate/settle

Single-operator dev shortcut

For local exploration, the control roles operator / lpRegistrar / admin may share one party, and the in-memory dev server may use DEX_DEV_OPEN=1 to bypass the operator-token gate. Do not collapse a value-moving counterparty into that party: a real registry rejects the self-transfer created when the LP or swapper equals the operator. The portable sandbox proof therefore allocates distinct LP/trader and swapper parties even though its three control roles share the bootstrap party. Production should normally separate the control roles too so audit-trail and key-management responsibilities stay decoupled, and must set DEX_OPERATOR_API_TOKEN / OPERATOR_ADMIN_TOKEN — both gates fail closed otherwise, proven in auth.test.ts (“fails closed when no token and no devOpen”).

Out of scope for this document

• Cluster topology (cantond, participants, synchronizers, sequencers)
• Backup, key custody, and HSM policy
• KMS / secrets management for the operator submission key
• Network ingress and rate limiting

These are operational concerns inherited from the underlying Canton deployment and are not constrained by the DEX contract surface.

---

Where to read next: Operator Guide · Deployment · All docs