DevOps QA benchmarking — why and what?

To understand the current state of maturity of the organization to adapt to DevOps practices is important in ensuring successful implementation of DevOps. A baseline assessment helps to understand the challenges (such as those stated in the preceding section) to be acknowledged and addressed before an organization can embark on the DevOps journey, for those who have already embarked on the journey, such assessment helps to progress further in achieving a higher state of DevOps maturity.

The DevOps QA benchmarking framework assesses the maturity of the testing processes in a DevOps environment and helps us to develop a roadmap for envisioned end state.

Key DevOps practices (as against traditional IT) include the following:

• Use of an integrated Quality approach
• Build quality engineering teams in DevOps
• Implement Behavior-Driven and Test-Driven Development
• Automate, Automate, Automate, to enable a Zero-touch continuous automated testing
• Virtualize test environment, and automate test data needs for continuous integration
• Implement quality metrics for DevOps
• Continuously monitor quality of Continuous Integration/Continuous Deployment (CI/CD)

Benchmarking of the 25 elements of DevOps QA across 6 areas—source control, build management, release management, deployment, test management, and monitoring is done as follows.

The 4 states in the level of maturity for DevOps benchmarking include the following:

• Initial: This involves agile development, limited automation, focus on code quality, and versioning traditional infrastructure
• Controlled: This involves continuous integration, automated build, QA Integration simulation, business process modeling, and continuous testing
• Efficient: This involves extended testing automation, process optimization, analytics, automated Provisioning, and cloud Lite
• Optimizing: This involves process productization, managed KPI trajectory, and hybrid cloud