Appearance

Testnet reset runbook — 2026-06-14 ​

One-shot procedure to recover from the cascading failures of 2026-06-14. Brings the chain back up at sanect_76287-1 with primary as sole majority validator, all contracts redeployed at deterministic addresses.

Total time: ~45 minutes.

What happened (for the postmortem) ​

  1. 15:00 UTC — --force snapshot-upload triggered on primary node. tar+lz4+rclone competed with CometBFT for ZFS I/O. Primary fell behind on block production. Chain forked between primary and Vultr.
  2. 15:30–19:00 UTC — Recovery cascade. Multiple FORCE_SNAPSHOT_RESTORE cycles. R2 snapshots stale (height 658,918). Pivoted to state-sync.
  3. 18:30 UTC — Vultr's Docker container destroyed; volume sanect-data cleared at some point. New priv_validator_key.json generated. Original validator key for 99.82%-bonded "sanect-node" validator is intact on primary's volume, but the chain's full LevelDB state was lost across all 4 nodes during repeated wipes.

Conclusion: the canonical chain at height 1,060,755 is unrecoverable. Reset is the only path forward.

What survives the reset ​

  • ✅ All code (contracts, dApps, scripts, docs) — untouched
  • ✅ Your wallet 0x7cB61D4117AE31a12E393a1Cfa3BaC666481D02E with 990M SNCT (regenerate at genesis)
  • ✅ Primary's CometBFT priv_validator_key.json (DsS4h53...) — reuse so consensus pubkey stays the same
  • ✅ Chain ID sanect_76287-1 — reuse so MetaMask add-network snippets keep working
  • ✅ Contract addresses — same source + same deployer + clean nonces = identical addresses if redeployed in original order

Pre-flight checklist (5 min) ​

  • [ ] Locate your deployer private key (the wallet with 990M SNCT, address 0x7cB61D…). You'll need it for the contract redeploys.
  • [ ] Note primary's CometBFT consensus pubkey: DsS4h53SOQ7NIx2VGT8lvnHOoVOFqIPc+VE+uBPYBXw=
  • [ ] Have Railway dashboard + Vultr SSH access open
  • [ ] Have a terminal with forge (Foundry) installed for contract redeploys

Phase 1 — Stop everything (5 min) ​

1.1 Railway ​

On each of the 4 Railway services (sanect-node, 2nd-node, 3rd-node, explorer-backend, plus any DEX/dApp services that depend on chain), set the service to Stopped state, OR set Custom Start Command to sleep 60 if you can't fully stop.

1.2 Vultr ​

On the Vultr box:

bash
docker stop sanect-node || true
docker rm sanect-node || true

Phase 2 — Wipe chain volumes (5 min) ​

2.1 Railway primary, 2nd-node, 3rd-node ​

For each service:

  1. Railway → service → Settings → Volumes
  2. Detach + delete the existing volume
  3. Re-attach a fresh empty volume to /data
  4. Save

EXCEPTION for primary: before wiping, copy priv_validator_key.json out:

  • Set primary's Custom Start Command to sleep 36000, redeploy
  • Shell in, copy the key:
    bash
    base64 /data/.sanectd/config/priv_validator_key.json
  • Save the output to a local file (paste it somewhere safe — you'll restore it after wipe)
  • Now delete the volume

2.2 Vultr ​

bash
docker volume rm sanect-data
docker volume create sanect-data

Phase 3 — Rebuild genesis on primary (10 min) ​

3.1 Update primary's env vars ​

Remove all of these:

  • FORCE_SNAPSHOT_RESTORE
  • JOIN_NETWORK
  • SEED_NODE_URL, SEED_NODE_ID, SEED_PEER_HOST
  • SNAPSHOT_MANIFEST_URL

Keep:

env
CHAIN_ID=sanect_76287-1
EVM_CHAIN_ID=76287
HOMEDIR=/data/.sanectd
MONIKER=sanect-node
EXPLORER_HEARTBEAT_URL=https://scan.testnet.sanect.com/api/network/heartbeat
PUBLIC_RPC_URL=https://rpc.testnet.sanect.com
NETWORK=testnet
GENESIS_WALLET=0x7cB61D4117AE31a12E393a1Cfa3BaC666481D02E

The GENESIS_WALLET env tells localnet.sh to allocate the full 1B SNCT to your address.

3.2 Reset Custom Start Command back to default ​

Set Custom Start Command to: (empty) — falls back to Dockerfile CMD.

3.3 Restore primary's priv_validator_key.json ​

Set Custom Start Command back to sleep 60, redeploy. Shell in:

bash
mkdir -p /data/.sanectd/config
cat > /data/.sanectd/config/priv_validator_key.json <<'EOF'
<paste the JSON you saved in Phase 2.1 here>
EOF
chmod 600 /data/.sanectd/config/priv_validator_key.json

3.4 Redeploy primary fresh ​

Clear Custom Start Command, Redeploy primary. The entrypoint sees no genesis.json and no JOIN_NETWORK, so it runs localnet.sh SETUP_ONLY=true. localnet.sh builds:

  • Fresh genesis with chain_id sanect_76287-1
  • 1B SNCT allocated to GENESIS_WALLET
  • Primary's existing priv_validator_key as sole bonded validator with self-delegation
  • All Variant A genesis params (50 max validators, ~400ms blocks, etc.)

After ~60s, check:

bash
curl -s https://rpc.testnet.sanect.com/rpc/status | jq '.result.sync_info'

Should show latest_block_height: "1" and climbing.

Phase 4 — Add other nodes back (10 min) ​

4.1 2nd-node, 3rd-node ​

For each, set env vars:

env
JOIN_NETWORK=true
CHAIN_ID=sanect_76287-1
EVM_CHAIN_ID=76287
HOMEDIR=/data/.sanectd
SEED_NODE_URL=https://rpc.testnet.sanect.com
SEED_NODE_ID=<primary's node_id>
SEED_PEER_HOST=p2p.testnet.sanect.com:46430
MONIKER=2nd-node    # or 3rd-node
EXPLORER_HEARTBEAT_URL=https://scan.testnet.sanect.com/api/network/heartbeat
PUBLIC_RPC_URL=https://2nd-node-production.up.railway.app

Get primary's node_id:

bash
curl -s https://rpc.testnet.sanect.com/rpc/status | jq -r '.result.node_info.id'

Reset Custom Start Command. Redeploy.

4.2 Vultr ​

bash
docker run -d --name sanect-node \
  --restart unless-stopped \
  -p 8080:8080 \
  -p 26656:26656 \
  -v sanect-data:/data \
  -e JOIN_NETWORK=true \
  -e CHAIN_ID=sanect_76287-1 \
  -e EVM_CHAIN_ID=76287 \
  -e HOMEDIR=/data/.sanectd \
  -e MONIKER=sanect-val-4 \
  -e SEED_NODE_URL=https://rpc.testnet.sanect.com \
  -e SEED_NODE_ID=<primary node_id> \
  -e SEED_PEER_HOST=p2p.testnet.sanect.com:46430 \
  -e EXPLORER_HEARTBEAT_URL=https://scan.testnet.sanect.com/api/network/heartbeat \
  -e PUBLIC_RPC_URL=http://207.148.74.161:8080 \
  -e EXTERNAL_P2P_ADDRESS=207.148.74.161:26656 \
  sanect-node

All 3 join the chain via primary, become RPC nodes (small or zero validator stake — you can delegate later if you want them bonded).

Phase 5 — Redeploy contracts (15 min) ​

CRITICAL: Use the same deployer wallet (0x7cB61D…) and deploy in the same order as before. Same source + same deployer + clean nonces → identical contract addresses → no env-var changes needed downstream.

The genesis chain has zero nonce for the deployer. Each forge create increments nonce by 1, so order is what determines addresses.

bash
export RPC=https://rpc.testnet.sanect.com
export DEPLOYER_PRIVATE_KEY=0x...your-key...
export DEPLOYER=0x7cB61D4117AE31a12E393a1Cfa3BaC666481D02E
export OWNER=$DEPLOYER

# 1. DEX core (nonces 0–2)
cd ~/sanect-node/dex
forge script script/DeployDEX.s.sol --rpc-url $RPC --broadcast
# → WSNCT, Factory, Router

# 2. LP farm (nonces 3–4)
export SNCT_TOKEN=<WSNCT from step 1>
forge script script/DeployFarm.s.sol --rpc-url $RPC --broadcast
# → TreasuryEmitter, MasterChef

# 3. Privacy module — verifier + pool (nonces 5–6)
cd ../privacy
forge create circuits/verifier/Verifier.sol:HonkVerifier --rpc-url $RPC --private-key $DEPLOYER_PRIVATE_KEY --broadcast
# → HonkVerifier v2

forge create contracts/ShieldedPool.sol:ShieldedPool --constructor-args <HonkVerifier> $OWNER --rpc-url $RPC --private-key $DEPLOYER_PRIVATE_KEY --broadcast
# → ShieldedPool v1.2 (240-byte memo)

# 4. .snct domain contracts (nonces 7–14)
cd ../domainapp/contracts
forge script script/Deploy.s.sol --rpc-url $RPC --broadcast
# → Registry, PublicResolver, BaseRegistrar, ReverseRegistrar, PriceOracle, SealedBidAuction, Marketplace, Reserved

Save every printed address. Compare to the addresses in CLAUDE.md — if same deployer + same order, they should match exactly. If they don't, update CLAUDE.md with the new ones.

Phase 6 — Update dApp env vars (5 min) ​

If contract addresses changed (most likely they will because deploy script details may differ), update each Railway dApp service:

staking-app ​

env
NEXT_PUBLIC_SHIELDED_POOL_ADDRESS=<new ShieldedPool>

explorer-backend ​

env
SHIELDED_POOL_ADDRESS=<new ShieldedPool>

dex-app ​

env
NEXT_PUBLIC_DEX_FACTORY=<new Factory>
NEXT_PUBLIC_DEX_ROUTER=<new Router>
NEXT_PUBLIC_WSNCT=<new WSNCT>
NEXT_PUBLIC_DEX_INIT_CODE_HASH=<from DeployDEX output>
NEXT_PUBLIC_MASTERCHEF=<new MasterChef>
NEXT_PUBLIC_TREASURY_EMITTER=<new TreasuryEmitter>

Redeploy each so the NEXT_PUBLIC_* envs get baked into the build.

Phase 7 — Verify (5 min) ​

bash
# Chain producing blocks
curl -s https://rpc.testnet.sanect.com/rpc/status | jq '.result.sync_info'

# Dashboard shows all nodes
open https://scan.testnet.sanect.com/network/nodes

# Your balance is 1B
curl -s https://rpc.testnet.sanect.com/ -X POST -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x7cB61D4117AE31a12E393a1Cfa3BaC666481D02E","latest"],"id":1}'
# expect: "0x33b2e3c9fd0803ce8000000" (= 1e27 = 1B SNCT)

Lessons baked into the recovery ​

To prevent this cascade from repeating:

  1. Don't run snapshot-upload on a producing validator. Dedicated archive node only — see follow-up PR for sanect-snapshot-publisher service.
  2. Bind-mount priv_validator_key.json from host directory, not just Docker named volume. Survives volume recreation.
  3. Daily R2 backup of just the 345-byte priv_validator_key.json so we never lose a validator key again.
  4. FORCE_SNAPSHOT_RESTORE requires explicit confirmation token (e.g. typing the chain ID). Removes "oh I left that env on by accident" risk.
  5. Disable destructive entrypoint paths in production. Wipe-and-rejoin should be a separate explicit script, not a startup env var.

These get rolled into a follow-up "node hardening" PR after recovery.

After recovery — public communication ​

Suggested X / Telegram post (optional, your call):

Testnet had a halt today. A --force snapshot upload on the primary validator hit a storage I/O contention bug — chain forked, recovery cascade lost the canonical state. We've reset cleanly: same chain ID sanect_76287-1, same contracts, same wallet. We're hardening the snapshot/recovery flow before mainnet — postmortem coming. Genesis Validator program continues as planned.

This actually builds credibility — projects that publicly own bugs and ship fixes get respected. Aleo, Penumbra, Sei have all done this. Audiences notice the difference between "swept under rug" and "we hit X, fixed it, here's how."