Customer Challenge
The United States Immigration services at DHS support millions of people from around the world trying to apply for and be granted access to the USA for work permits and permanent resident status. A multitude of disparate software applications being written by hundreds of developers across numerous technology stacks and by a collection of different contractors has caused significant problems. So, trying to sustain Agile best practices, DevSecOps CI/CD platforms for the different products, and controlling the code base of these products to deliver solutions to production on a regular cadence is difficult at best.
Sevatec Solution
As the contractor for managing the DevSecOps pipelines and processes, we saw the need firsthand and worked to develop new best practices that supported better delivery of quality products to production while also producing multiple release candidate builds per day. Our answer was Trunk Based Development (TBD).
TBD provided government Product Owners the peace of mind that producing 10 candidate builds per day could be achieved. We used pull requests to trigger short lived, individually oriented feature branches that quickly merged new functionality into the master branch preventing big bang merge disasters that require considerable time to recover. Small, quick merges allowed for rapid response to issues and defects prior to releases, ensuring a quality product delivery.
For TBD to work effectively, we required developers to commit code everyday with code reviews and quality inspections including security scanning. This approach minimizes test failures. Pull requests by the developers trigger pre-integration checks. When the code passes, it is merged to the master branch and the CI pipeline kicks off orchestration of the automated build, tests and integration processes. The optimized CI process we employ ensures little to no regression creeps into the code leading to an Always Releasable build from the master branch. Even so, we support the government mandates of Product Owners making build deployment decisions supported by a fully automated process that makes their decision-making much less cumbersome.
One aspect of using TBD and requiring developers to commit code every day is that features will be in an unfinished state when merges to the master branch. We use feature flags to control how and when features are incorporated into production ready code as a part of the deployment. For some people in the industry, all of this sounds similar to GitLabs Git Flow branching process. In reality, Flow is a lightweight branching strategy that imposes a high degree of control on the process which impacts the speed at which releases can be deployed. Our implementation of TBD incorporates the level of control necessary to meet program demands, while still allowing developers the autonomy to be creative which supports better, more usable products.
Results Delivered
The key benefits of employing TBD in scaled modern software development environments is many-fold. The fact that a deployable release is always ready greatly accelerates feature deployment to production and allows the teams to adapt to flexible work requirements and changes to government plans. As for development teams, TBD supports flexible cadence among teams allowing more complex work to be managed along a release branch while fast change work can continually be made available on the master branch. But the most important result of TBD is consistent, high quality and secure code deployed to production with the least risk to the enterprise from errors and vulnerabilities that are introduced in latter stages of release branches when using more traditional branch strategies that cause significant rework, delays and increased costs.
