Managing software dependencies is one of the most time-intensive tasks for most software developers. There are nine circles of dependency hell. It takes away from building new features or bringing down technical debt. However, it causes headaches at every step of the development cycle — from local development to CI/CD, production, and maintenance.
Some useful tips on wrangling dependencies from someone who worked on large-scale projects, CI/CD, build tools, and more.
Pin your dependencies. If your package manager or developer tool allows it, pin packages to specific versions. Failing to pin packages introduces a series of complicated heisenbugs to fix. Software that works locally but fails in CI. Builds that suddenly start failing in CI without any changes (cache expires). Builds that work on some CI machines but not others (different cached versions).
In more extreme cases (e.g., docker images or git repositories), you might even want to pin to exact checksums (e.g., digest SHA or commit SHA). Eliminates the tricky bug on reuploaded versions or tags. Not only does this make builds more reproducible, but for a lot of systems might even make things much faster (i.e., not redeploying a docker container if the tag changes but the content doesn’t change).
Separate package file updates in a separate pull request. This makes it easier to review changes. In many cases, you don’t want to vendor in (i.e., check-in) the actual packages but have a reproducible package file lock (e.g., package-lock.json, go.sum) checked in. This avoids the subtle attack vector of a malicious third party updating dependencies but modifying a package maliciously. If everything is checked in, many CI pipelines might not pick up the change. And reviewers are unlikely to catch a vendored-in code change that differs from upstream. But if you must vendor packages in, have a step in CI that checks that the packages can be reproducibly built via the lock file (e.g., download and diff).
Cautiously add new dependencies. A little copying is often better than a little dependency. Be mindful of package dependencies within a repository but also outside a repository.
Don’t overmodulize your code. If you find yourself constantly updating two different packages atomically, consider putting them in the same repository. Overmodulizing code early on creates a lot of broken releases and bugs (diamond dependency problem, cascading releases, etc.).
Keep packages updated. Amortize the cost of upgrades by upgrading often (at least non-major versions).
Package environment dependencies in a Dockerfile. If you can, put your dependencies in a Dockerfile. But be sure that you pin your dependencies inside the Dockerfile (even more important for catching cache bugs).