DevOps in a Safety-Critical Market

By Mike Johnson, Business Leader

Teams across the industry are turning to DevOps as a better way to organize their software efforts for productivity, quality, and ease-of-deployment. Because DevOps integrates traditional software development with operations activities that are oriented toward frequent small releases, customers receive quality software faster with lower development cost. Still, there are some challenges to implementing DevOps in a safety-critical market segment such as aviation or defense. In these industries, formal verification and certification processes play an important role in preventing defects from turning into dangerous situations in the real world. As a consequence, safety-critical software teams will spend a significant amount of time getting their software ready for official acceptance.  The time and cost involved in this process make the DevOps paradigm of continuous release challenging, especially in delivering to the end customer.  Despite this, aviation and defense teams have embraced the principles of DevOps, especially at the pre-certification stage, because of the potential for the higher overall quality it offers.

One-way DevOps helps safety-critical teams achieve higher quality is through heavy use of automation. DevOps teams try to “automate everything” in order to concentrate their efforts on creating new features and fixing bugs. Individual and nightly builds, code-coverage tests, regression tests, and deployment to operational systems are all candidates for automation. The DevOps mindset makes this happen for traditional cloud and serverless applications, but with a little extra effort, embedded and safety-critical development can also achieve high rates of automation. One strategy that is proven to improve quality is to integrate developer check-ins with the full set of functional and regression tests. Adding fault injection and testing at scale can also help to find intermittent and low-frequency failures that might not otherwise be detected until later when the cost of recertification or field remediation are high.

Virtualization is a critical enabler of higher productivity and quality for DevOps.  This is especially the case with test automation, where virtual machines can be used to scale up the frequency and volume of testing. Virtualization simultaneously allows for a highly controlled environment, but with potential infinite variations in configurations, patch levels, and pre-defined system states. In true DevOps fashion, individual developers and teams can maintain their own virtual environments, even while the target hardware is still under development. Virtualization testing is also high-fidelity: in many cases, virtual machines can execute software code intended for the target platform without alterations or modifications. Teams may even receive “run-for-score” credit for application code tested on virtual machines certified as test tools. This means developers can run the certification tests over and over again in response to each change in code, significantly improving the pass rate and reducing the possibility of defects late in the game.

DevOps team practices can also lead to higher quality. In a traditional waterfall development process, developers are assigned tasks according to a master schedule and may be reassigned or have a hand in different systems or subsystems as the release takes shape. Sometimes, engineering management is not even responsible for the tasks assigned or the code produced. DevOps treats teams differently. Each feature or component is supported by a small team which remains committed to it throughout the lifecycle. This team is the same one that has run all the regression and certification tests for the feature many times. When problems are found, they can be easily identified and fixed by the engineers owning the feature or component.

Continuous deployment is unlikely to take the place of certification any time soon in safety-critical markets like aviation. Still, DevOps practices can help move products into, and out of, the certification process faster with fewer code updates and regression test iterations.  What’s more, for teams adopting the DevOps-style of automation and virtualization, certification can be much easier and predictable.  When executed well, DevOps will lead to higher overall software quality at a lower cost, without compromising safety.

When your safety-critical project needs additional development resources, contact Performance Software to learn how we can support you.

devops journeyembedded security