Push to Main, Ship to Prod: The Bug That Ate a Deploy

Every check was green. The site was still down.

I merged a pull request. Every check on it went green: the build step, the tests, the AgentGate review, all of it. GitHub’s little checkmark, the one that means “this is safe to ship,” sat there fully lit. I closed the laptop feeling good about it.

The blog was still down.

The promise GitOps makes

This site deploys the way most modern sites do: push to main, and DigitalOcean App Platform notices the commit, pulls it, builds it, and puts it into production. No button to click, no pipeline to babysit, no SSH session to open at 11 p.m. That’s the entire pitch of GitOps: the deploy isn’t a separate step you perform, it’s a side effect of the merge you already wanted to do.

What that pitch quietly leaves out is who’s actually keeping the promise. On this site, the answer is two systems that have never met.

Two systems that don’t talk

GitHub Actions runs AgentGate on every pull request: it inspects the diff, checks for secrets and scope creep, and posts a pass/fail verdict as a status check. Separately, and with no awareness that AgentGate exists, DigitalOcean’s native GitHub integration watches main and fires its own build the moment something lands there.

Those two systems are not stages in one pipeline. They’re two independent observers of the same repository, each convinced it’s the one that matters. A red AgentGate check doesn’t stop DigitalOcean from deploying. A green one doesn’t mean DigitalOcean’s build will succeed. The checkmark on your pull request is a claim about one of those systems, not a guarantee about both.

It gets more disorienting than that if you go looking for evidence. GitHub has a “Deployments” tab, and it stays permanently empty on this repo, because DigitalOcean deploys through its own native integration and never bothers writing a GitHub Deployment object back. So the one place a reasonable person would check for deploy status shows nothing at all, in either direction. Not a failure. Not a success. Just silence, because the tool that would report one was never asked to.

I found out the hard way, on the pull request that added this blog’s content collection and its first post.

The build that failed somewhere I wasn’t looking

The PR merged clean. Every GitHub check was green. I moved on. A few minutes later I refreshed the live site to see the new post, and it wasn’t there. Nothing about GitHub suggested anything was wrong, because from GitHub’s side, nothing was.

The failure was one layer down, in a build log on a server I hadn’t thought to check yet. doctl apps list-deployments showed the deployment sitting in ERROR. doctl apps logs --type build showed why:

npm error Missing: @emnapi/[email protected] from lock file
!  npm lockfile is not in sync

npm ci — the command DigitalOcean’s buildpack runs, on purpose, instead of npm install — had looked at my package-lock.json and refused it outright. That’s not a bug in npm ci. Refusing anything that doesn’t match exactly is the entire feature.

Two node versions, two different answers

Here’s the part that actually mattered: I’d regenerated that lockfile locally on node 24 with npm 11, because that’s what was on my machine that week. DigitalOcean’s buildpack reads the engines field in package.json, saw >=20.3.0, and resolved that to node 22 with npm 10.

Different npm majors don’t always resolve a dependency tree the same way. In this case, npm 10 and npm 11 disagreed about which optional platform binaries sharp needed, specifically its @emnapi packages, and wrote two different lockfiles for the same package.json. Mine worked perfectly, on my machine, under npm 11. DigitalOcean’s installer, running npm 10, looked at that same lockfile and saw something that didn’t add up.

Nothing about that showed up in a diff review. A package-lock.json change is three hundred lines nobody reads line by line, and even if you did, “these two npm versions resolve optional deps differently” isn’t something you’ll spot by eye. It only shows up when the other npm tries to use what the first one wrote.

The fix, and the fix that makes it not happen twice

The immediate fix was straightforward: install node 22 locally, regenerate package-lock.json under it, and run npm ci myself before pushing again, because that’s the literal command DigitalOcean runs. If it passed on my machine under the same node version DigitalOcean uses, it would pass there too. It did.

The fix that actually mattered was the boring one. I capped engines to >=20.3.0 <23, so the buildpack can’t quietly drift onto a different major the next time DigitalOcean updates its defaults, and added an .nvmrc pinning 22, so my own shell defaults to the same version DigitalOcean resolves to. Now “build under node 22” isn’t a fact I have to remember. It’s the default in both places, which means it’s not really a fact at all anymore. It’s just what happens.

The lesson under the incident

The specific bug here was sharp’s optional dependencies and two npm majors disagreeing. That detail won’t matter to you; your stack will find its own version of it eventually. The shape underneath it will.

GitOps automates the deploy. It does not automate agreement about what “build” means, and when your machine and the builder’s machine quietly disagree about that, the failure doesn’t happen where you’re watching. It happens in a log you have to go looking for, on infrastructure you were trying not to think about, which is the entire reason you wanted GitOps in the first place. Push-to-deploy is only as simple as it looks for as long as both ends of the push agree on the plumbing in between.

It’s the same shape as who gets to touch the guardrails: the system that checks your work and the system that decides what goes live are not automatically the same system, and assuming they are is exactly how a fully green pull request ships nothing. If you’re evaluating a CI/CD setup, that’s the actual question worth asking, not “do we have checks” but “do our checks and our deploy agree on what passing means, or are they just two systems that happen to watch the same branch.”

Where it landed

Now when I push to main, the checkmark still lights up. AgentGate still runs its checklist. But I know, in a way I didn’t before this happened, that green means “GitHub is satisfied,” not “the internet can see this yet.” The actual proof lives one layer down, in a build log I now only have to check when something’s actually wrong, because the thing that used to go wrong quietly can’t drift anymore.

That’s the unglamorous point of pinning a node version and committing a lockfile: not that it makes deploys exciting, but that it makes them boring again, the way a merge to main is supposed to be.

The checkmark was green. Now it’s also right.