01 · Challenge
The problem
The engineering team relies on slow, inconsistent manual deployments. Release failures require developer intervention, and different applications use inconsistent pipeline configurations.
Symptoms the team sees
- Releases are batched into large, infrequent deployments because each one takes significant effort.
- Deployment steps live in a wiki page and in the heads of two senior engineers.
- Each repository has its own pipeline file with slightly different build, test and tagging logic.
- When a release fails, developers SSH into servers to diagnose and patch by hand.
- There is no reliable way to tell which commit is running in which environment.
Root causes
- Pipelines were copied and modified per project rather than shared, so improvements never propagated.
- Artifacts were rebuilt per environment, meaning staging and production could run different builds.
- No automated health checks or rollback — failure detection depended on someone noticing.
- Environment configuration was mixed into build steps instead of being injected at deploy time.
02 · Approach
Implementation steps
- 01
Map the current delivery path
Walk through a real release end-to-end with the team, documenting every manual step, wait and hand-off.
- 02
Design a reusable pipeline
Create shared GitHub Actions reusable workflows (or GitLab CI includes) covering lint, test, build and publish, parameterized per service.
- 03
Build once, promote everywhere
Build a single container image per commit, tag it with the commit SHA and promote the same image from staging to production.
- 04
Automate tests in the pipeline
Run unit and integration tests on every pull request; block merges on failure.
- 05
Controlled environments
Use protected environments with required approvals for production and environment-specific configuration injected at deploy time.
- 06
Health checks and rollback
Gate rollout on readiness checks and smoke tests; automate rollback to the previous image when they fail.
- 07
Document and hand over
Write release and rollback runbooks, then migrate remaining services alongside the team so they own the pattern.
03 · Architecture
Solution architecture
A single, shared pipeline definition moves every change through the same stages. The image built in CI is the exact artifact deployed to each environment, and health checks decide whether a rollout completes or rolls back.
Standardized delivery workflow
- Source
- Build
- Deploy
- Observe
- 01Git repositoryTrunk-based flow, protected main branch
- 02CI pipelineShared GitHub Actions / GitLab CI templates
- 03Automated testsFail fast on unit and integration tests
- 04Container registryImage tagged with commit SHA
- 05Deployment environmentStaging → production with approvals
- 06Health checks & monitoringAutomatic rollback on failed checks
Technology stack
- GitHub Actions or GitLab CI
- Docker
- Amazon ECR / GitHub Container Registry
- Kubernetes or ECS
- Helm
- Prometheus & Grafana
- Slack notifications
04 · Validation
How the result is validated
- Run the new pipeline in parallel with the old process for a pilot service before switching over.
- Deliberately deploy a failing build to staging to confirm the health check blocks promotion and rollback works.
- Verify that the image digest in production matches the one tested in staging.
- Track deployment frequency, lead time and change failure rate before and after so the team can judge the effect with its own data.
05 · Expected impact
Business impact
- More repeatable deployments, because every service follows the same tested path.
- Less manual effort per release, freeing senior engineers from deployment duty.
- Failures that are easier to diagnose, with logs, test results and the exact image tied to each run.
- Smaller, more frequent releases become practical, which typically lowers the risk of each one.
These are expected directions of improvement, not measured results. Actual impact depends on the starting point and is tracked with the client's own metrics.
06 · Lessons learned
What tends to matter most
- Start with one representative service; a working example persuades faster than a design document.
- Keep the shared template small and composable — teams resist templates that hide everything.
- Rollback is a feature that needs testing, not an afterthought.