Why staging doesn’t match production—and how we fix it.

The product being released

• The same source revision across affected apps and backend.
• The same dependency lockfile and pinned build tools.
• The same production-shaped build and request paths.
• A known release identity that can be verified after deploy.

The environment-specific configuration

• Staging and production data.
• Convex deployment URLs and Clerk instances.
• Analytics projects, domains, and secrets.
• Scale and real-world user/device behavior.

Testing and shipping are separate actions

A feature or staging deployment can be tested, then the branch can change or a different checkout can be used for production. Nothing currently requires production to use the approved staging revision.

Deploy commands trust local state

The scripts build whatever is in the operator’s workspace and explicitly allow dirty files. A deployment can therefore differ from its recorded Git SHA or come from a stale checkout.

Each surface deploys independently

Player, dashboard, landing, and Convex have separate commands. Staging can pair a new frontend with an older backend—or vice versa—despite each individual deployment succeeding.

“Staging looks good” has no durable meaning

With several people deploying, the shared staging URL can change during testing. Approval does not currently identify the exact deployment that was reviewed.

Environment-specific bugs still need checks

Current CI does not build the frontends or test the deployed release. Browser and device differences—such as the landing animation’s Safari/low-power behavior—can still create production-only symptoms.

Feature previews are early validation

A temporary URL for one feature branch—like Hen’s dedicated landing-staging site—is useful for design review. It is not final release approval because that branch can continue changing.

Shared staging is the final rehearsal

Deploy the resulting main SHA and all affected components together. Approval identifies that SHA or immutable deployment URL. A newer merge creates candidate Y; it cannot silently replace approved candidate X.

Build in GitHub Actions

Use a clean checkout, pinned Bun version, and frozen lockfile. Local laptop deploys become an explicit emergency path rather than the normal release mechanism.

Coordinate affected components

Player, dashboard, landing, and Convex changes deploy from the same monorepo SHA, in a defined order, so frontend/backend versions cannot drift silently.

Production accepts only staged SHA X

A manual “Promote to production” action receives the successful staging candidate—not an arbitrary local checkout or whatever staging happens to show later.

Expose release identity

Record the SHA for every deployed component. A small version endpoint or deployment summary makes mismatches visible without Cloudflare and Git archaeology.

Prevent deploy races

Only one staging or production deployment runs at a time. New merges create new candidates instead of overwriting the meaning of an existing approval.

Add a small smoke-test set

Verify frontend builds, app availability, authentication handshake, one known tangle flow, component version alignment, and key landing behavior on Chromium and WebKit.

Clean history, weaker ancestry

The incident evidence shows the final feature head, the squash into staging, and the squash into main had identical repository trees. Squash did not alter the shipped code; it made the lineage harder to inspect.

Clearer ancestry, same release requirement

Future feature commits remain visible in main, making audits easier. Existing history is unchanged, and this still does not stop post-test commits, dirty laptop deploys, or component skew.