Why "it works on my machine" tells me nothing anymore

One of the commits I'm least proud of says, literally, "fixear la mierda de dependencias, que no dan error en build pero sí al desplegar". I wrote it tired, but it describes the problem well: the build passed, the tests passed, and the deploy fell over anyway.

It wasn't a one-off. In a Python service for processing documents I have six commits in a row on the same day, all titled "try to deploy on railway", numbered 1 and 2 once I ran out of names. Each one was a blind attempt against an environment I couldn't reproduce. That's not bad luck: it's a failure category with a shape of its own.

Dependencies that resolve differently depending on who installs them

The six-commit case was PyTorch. Locally I already had it cached and it started fine. In the container, uv resolved the torch>=2.0.0 from PyPI, which drags in the entire CUDA stack: nvidia-cublas-cu12 is 594 MB in a single wheel, and behind it come cuda-cupti, cuda-nvrtc and friends. The build didn't even fail on an error of mine, it died on disk and time. On a machine that will never see a GPU.

What fixed it was to stop asking for the package and start asking for the index:

dependencies = ["torch==2.9.1+cpu", "torchvision==0.24.1+cpu"]

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[tool.uv.sources]
torch = { index = "pytorch-cpu" }

The detail that cost me two attempts: with torch>=2.0.0 pointing at the CPU index, the lockfile ends up with two entries, 2.9.1 for sys_platform == 'darwin' and 2.9.1+cpu for everything else, because on macOS that index doesn't publish a +cpu suffix. It works, but it means what you install on your Mac and what you install in the container aren't the same artifact. Pinning the exact version with the suffix closes that door.

The other variant, dumber and more common: a dependency you use in the code that isn't in package.json. There's puppeteer-core sneaking in under a commit titled "adds missing dependency", and another one called "add missing dependencies" three weeks earlier. Locally it worked because the package was in node_modules as a transitive dependency of something else. In a container with a clean install, no.

The environment you don't replicate: system libraries

The most expensive failure I've had in this family was in a desktop transcription app. The microphone worked perfectly with uv run, and didn't work in the installed .deb. The error was Invalid sample rate [PaErrorCode -9997].

The cause: PyInstaller was bundling the PortAudio that ships in the sounddevice wheel, compiled without a PulseAudio backend, and a libasound.so.2 without its alsa-lib modules. Inside the bundle the pulse, pipewire and default PCMs disappeared, and only the hw: ones in exclusive access were left. PipeWire already held the mic, so recording failed. With uv run you couldn't see it because there it links against the system PortAudio, which does have PulseAudio.

The fix was to exclude libportaudio, libasound and libjack from the bundle. And then came part two, which is the part that actually teaches you something: the next version wouldn't start on any modern distro.

OSError: cannot load library 'libportaudio.so.2': .../_internal/libstdc++.so.6:
version `GLIBCXX_3.4.32' not found (required by /usr/lib/x86_64-linux-gnu/libjack.so.0)

Loading the system PortAudio pulls in the system libjack, which on a new distro needs GLIBCXX_3.4.32. But PyInstaller was still bundling libstdc++.so.6, and the bundle comes first in the linker's search path, so the build runner's copy won, capped at GLIBCXX_3.4.30. The rule I took from that: excluding a library cuts in both directions. If a dependency resolves against the system, everything it drags in has to resolve against the system too.

Why a green build says almost nothing

A build checks that the code compiles with the dependencies the resolver picked on that machine, at that moment. It doesn't check that it starts, that it finds its native libraries, or that the process knows how to listen where the runtime is going to send it traffic.

I've paid for that last one too. In an internal tool I spent an afternoon on the Procfile and the connection string: next start without -H 0.0.0.0 doesn't accept traffic from outside the container, web: pnpm next start -p $PORT became web: pnpm start -- -p $PORT so the flag would actually land, and along the way we had to drop channel_binding from the DATABASE_URL because not every runtime supports it. Three fixes, same day, none of them visible locally: locally the server listens on localhost and you're on localhost too.

The test that would have caught the bug

The important thing about the libstdc++ case isn't the fix, it's why the smoke test missed it: it ran on the same runner where we built. There the system libjack is as old as the bundled libstdc++, so they match and the app starts. The failure only showed up on distros newer than the build machine. That is, on the users' machines.

What I put in CI today, ordered by how much it has saved me:

• Start the artifact, don't just build it. A docker run of the real container that hits /health. A green build and a live process are two different things.
• Start it in an environment different from the build one. In this case, an ubuntu:24.04 container installing only the declared Depends, which also checks that list is complete.
• A static ABI check. A script with objdump that makes sure the C++ runtime doesn't travel in the bundle and compares what the system libs ask for against what the bundle offers. It can't be flaky, and it fails on the old runner too, where the bug doesn't reproduce.
• Clean install, no cache. It's the only way the dependency you have by accident shows up.

"It works on my machine" means it works with your system libraries, your package cache, your distro version and your network interface. Four things production doesn't share with you.