Skip to main content

Launch ops checklist

Short checklist for release / high-traffic days. Pair with Monitoring for endpoint details.

Before send

Staging auto-deploys from main; production Google connect/reconnect smokes (WP-00, #3841) run at the next production release, not before staging send.

Staging (2026-09-22, Email-ready v1)

  • GET /api/health on staging returns 200 {"status":"ok"} (verified 2026-09-22)
  • Google connect smoke on staging (founder or agent with test account)
  • Google reconnect smoke on staging (revoke, banner, refresh)
  • Production Google connect smoke (deferred, #3841)
  • Production Google reconnect smoke (deferred, #3841)
  • sse_connection_degraded PostHog alert uses a trailing 60 minute window (see table below)

All environments

  • GET /api/health returns 200 {"status":"ok"}
  • Sync GET /health/live and GET /health/ready are healthy; logs show execution=active when Google sync is enabled
  • Deploy Discord health webhook is configured (see CI/CD workflows)
  • PostHog Error Tracking is receiving $exception from web (open a staging page and trigger a handled test if needed)
  • Confirm sync_health_snapshot events arrive every ~5 minutes in PostHog
  • Confirm the Meeting dashboard is readable: empty server tiles are no-data until booking_operation ships, not 100% success. Runbook: Meeting monitoring

Alerts to create in PostHog (or Discord)

Alert on sync_health_snapshot properties (low cardinality — safe to alert):

SignalSuggested threshold
connections.actionRequiredrising day over day, or absolute > 0 for 10+ minutes under load
connections.oldestImportingAgeMs> 3600000 (one hour) while any connection is still importing
jobs.failedrising vs baseline
jobs.oldestDueAgeMs> 5 minutes while execution=active
freshness.percentOver30ssustained spike vs quiet baseline. Sample is live connections only (healthy, delayed, catchingUp), not actionRequired or disconnected.

Also watch:

  • Web $exception rate (Error Tracking)

Alerts that already exist in PostHog

Created by hand in the PostHog UI or via the PostHog MCP; all evaluate hourly and email the founder's PostHog account. Check the alerts page before creating another one.

AlertInsightFires when
Sync job terminal failurehourly count of sync_job_terminal_failurecount above 0 in the current hour
Sync reconcile sweep starved (production)sync_reconcile_sweep completions, trailing 45 minutes, productioncount below 1
SSE connection degraded burst (production)sse_connection_degraded, trailing 60 minutes, productioncount above 3

The SSE alert samples a trailing 60-minute window once an hour, so a burst that straddles two checks can be under-counted. Widen the insight's window to 90 minutes if that bites; 15-minute evaluation needs a PostHog add-on. The web app offers a sidebar-footer Refresh control for the same condition (SidebarRefreshButton) after the stream has been down for 30 seconds.

Meeting launch checks (exhausted recovery, oldest pending, infrastructure failure rate, missing heartbeat) are defined in Meeting monitoring. They are not armed and must not notify a new recipient until the release owner confirms the channel. Hourly evaluation is coarser than the 5-minute pending SLO.

Welcome email launch checks (provider complaint rate above 0.1%, non-zero failed_count_24h on email_send_heartbeat) are defined in Welcome email staging verification. Same not armed rule until the release owner confirms the channel.

During launch

  • Watch Sync health snapshot + Error Tracking side by side
  • If calendars look stuck: check SSE (sse_connection_degraded), then Sync diagnostic routes (see Troubleshoot)
  • Both compass-backend and compass-sync logs are in PostHog Logs. Express errors carry method, path, status, userId, and stack — filter by service and status for triage

After

  • Resolve or suppress any new Error Tracking noise
  • Note any actionRequired / delayed cohorts for follow-up reconnect email