Mainnet primary validator: Railway → NVMe migration

The mainnet primary launched on Railway for speed (no NVMe host needed on Day 0). This is the runbook for moving it to a dedicated NVMe box, which is the right long-term home: lower block-time floor, no shared ZFS contention, no Railway-host OOM risk.

When to do this: Week 1 post-launch, once Day-0 dust has settled and bridge/airdrop deploys are out of the way. The migration is zero-downtime if done correctly.

---

Why move at all

Railway's ZFS pool gave testnet a ~720ms block floor with 3 validators (see CLAUDE.md "Block-time floor on Railway"). Mainnet single-validator on Railway will hit ~420ms — fine for now, but:

• Block floor on Vultr/Hetzner NVMe: ~380ms (faster UX)
• No OOM-kill exit 137 risk from Railway host pressure
• No risk of Railway pricing/policy change disrupting consensus
• Real validators on mainnet expect bare-metal-grade hosts

---

Host requirements

Pick one of:

Provider	Plan	Specs	Cost
Vultr High Frequency (recommended)	4 vCPU / 16GB / 256GB NVMe	NVMe SSD, 3GHz+	~$48/mo
Hetzner Cloud CCX23	4 vCPU / 16GB / 160GB NVMe	NVMe, dedicated vCPU	~€31/mo
Hetzner SX bare-metal	6c/12t / 64GB / NVMe RAID	best perf	~€60/mo

OS: Ubuntu 24.04 LTS (what scripts/mainnet/bootstrap-primary.sh targets). Other distros work but you'll re-derive the steps.

---

Step 1 — Provision host + DNS

1. Spin up the host. Get its public IPv4.
2. Cloudflare DNS:
   • rpc.sanect.com — leave pointing at Railway for now (we cut over later)
   • rpc2.sanect.com — A record to the new NVMe IP, DNS-only (orange-cloud OK for HTTPS later via Caddy)
   • p2p.sanect.com — leave pointing at Railway's TCP proxy for now

This lets you bring the NVMe node up and test it before cutting traffic.

---

Step 2 — Bootstrap

ssh root@<nvme-ip>
curl -fsSL https://raw.githubusercontent.com/sanect/sanect-node/main/scripts/mainnet/bootstrap-primary.sh | bash

bootstrap-primary.sh does:

• Install Docker + foundry + UFW
• Open ports 80/443/26656 (P2P)/22 (SSH)
• Clone the repo to /opt/sanect
• Generate a Caddyfile that fronts the node at :8080
• Print the Caddy command to bring it up

It does not start the chain — that's the next step.

---

Step 3 — Restore from snapshot

The NVMe node joins as a regular peer, restoring from the archive node's R2 snapshot. Zero validator key yet — we're just bringing up a healthy node first.

cd /opt/sanect
cat > /opt/sanect/.env <<'EOF'
NETWORK=mainnet
CHAIN_ID=sanect_7628-1
EVM_CHAIN_ID=7628
HOMEDIR=/data/.sanectd
DENOM=asnct
DISPLAY=snct
MONIKER=sanect-mainnet-nvme

JOIN_NETWORK=true
SEED_NODE_URL=https://rpc.sanect.com
SEED_NODE_ID=<get from current primary via Railway shell: sanectd cometbft show-node-id>
SEED_PEER_HOST=p2p.sanect.com:<current primary's TCP proxy port>

SNAPSHOT_MANIFEST_URL=https://snapshots.sanect.com/LATEST.json

# Advertise itself for PEX so future joiners can reach us via this host
EXTERNAL_P2P_ADDRESS=p2p.sanect.com:26656

# Keep history; this node may serve as backup archive
PRUNING=nothing
EOF

docker compose --env-file .env -f docker/docker-compose.yml up -d

Watch logs until catching_up: false:

docker logs -f sanectd
# In another terminal:
watch -n 2 'curl -s localhost:8080/rpc/status | jq .result.sync_info'

Expected: snapshot restore takes 3–5 min, then block-sync catches the delta to tip in 5–15 min depending on how far behind.

---

Step 4 — Cutover (the critical 30 seconds)

This is where the validator key moves from Railway to NVMe. Both nodes must NEVER have the key at the same time — double-sign on mainnet = 5% stake slash.

4a. Pre-flight

• Confirm NVMe node catching_up: false
• Confirm NVMe node's /rpc/status reports a height within 1 of primary's /rpc/status
• Confirm archive.sanect.com is publishing snapshots (you have a fallback if something goes wrong)

4b. The swap

# 1. STOP the Railway primary (Railway dashboard → service → ⋯ → Stop)
#    Wait 30s. Confirm `https://rpc.sanect.com` no longer responds.

# 2. On Railway shell BEFORE stopping (or via Railway CLI exec), grab
#    the validator key file:
#    sanectd cometbft show-validator --home /data/.sanectd > /tmp/pub.json
#    cat /data/.sanectd/config/priv_validator_key.json
#    Copy the FULL JSON content (it's ~3 lines, includes private key)

# 3. Stop NVMe sanectd container so we can swap the key file:
docker compose -f /opt/sanect/docker/docker-compose.yml stop sanectd

# 4. Replace the auto-generated key with the Railway one:
docker run --rm -v sanect_data:/data alpine sh -c \
  "cat > /data/.sanectd/config/priv_validator_key.json" <<'EOF'
<paste the JSON from step 2 here>
EOF

# 5. Reset signing state on the NVMe node so it starts fresh:
docker run --rm -v sanect_data:/data alpine sh -c \
  'cat > /data/.sanectd/data/priv_validator_state.json <<JSON
{"height":"0","round":0,"step":0}
JSON'

# 6. Bring NVMe back up
docker compose -f /opt/sanect/docker/docker-compose.yml start sanectd

# 7. Watch for signing activity:
docker logs -f sanectd 2>&1 | grep -E "Signed|prevote|precommit" | head -20

Within ~30s you should see the NVMe validator signing blocks.

4c. DNS cutover

Once NVMe is signing:

1. Cloudflare DNS: change rpc.sanect.com to point at the NVMe IP (or its Caddy-fronted hostname). TTL was already low; propagation <1 min.
2. Update Railway's TCP-proxy P2P record: change p2p.sanect.com from Railway's ballast.proxy.rlwy.net:NNNNN to point at the NVMe IP directly (CNAME → A record). Set port back to 26656 on every joiner's SEED_PEER_HOST.

4d. Decommission Railway primary

After 24h of NVMe signing without issues:

1. Railway primary service → Settings → Delete service
2. Detach + delete the volume (snapshots are in R2; nothing to keep)

Keep the Railway service config (env vars, build args) saved to a local file in case you ever need to spin it back up emergency-style.

---

Rollback (if NVMe misbehaves within first 24h)

If NVMe stops signing or block time spikes:

1. Cloudflare DNS: revert rpc.sanect.com to Railway
2. NVMe shell: docker compose stop sanectd (silences it)
3. Railway: redeploy the primary service (volume intact, key intact)
4. Reset priv_validator_state.json on Railway before restart, so it doesn't double-sign against any blocks the NVMe produced

This is why we don't delete the Railway service for 24h.

---

Verify

# Block time at tip — should be ~380ms typical
curl -s https://rpc.sanect.com/rpc/status | jq '.result.sync_info.latest_block_time'

# Signing validator address matches the moved key
curl -s https://rpc.sanect.com/rpc/status | jq '.result.validator_info'

# Snapshot publisher still working from archive node
curl -s https://snapshots.sanect.com/LATEST.json | jq '.height'

---

What this doesn't cover

• Multi-validator topology (Variant A): this runbook only moves the one primary. Recruiting 50 globally distributed validators is a separate recruitment + onboarding effort (see operator runbooks in operators/).
• Hot-standby validator: this is single-validator until you have 3+ validators with stake balance per CLAUDE.md "validator topology". Don't spin up a "backup" with the same key — that's the double-sign trap.