The Transformation of Software Testing

Traditional development and testing cycles have been a limiting factor for increasing the speed of creating and releasing new functionality as well as improving the quality of final releases. When much of a development team’s time is taken up with the looping, iterative cycle of design-develop-test-debug, a lot of creativity gets squeezed out of processes and people. Longer development cycles prevent new features—especially those specifically requested by customers—from being released on a timely basis. When customers have limited insight into how those features are being designed and implemented, they can be reluctant to implement them without extensive and time-consuming testing.

A trickier legacy issue to address is that IT buyers have lost trust in existing software development processes to deliver high quality code in dot zero releases. Instead they wait by default for future point deliveries, expecting more acceptable quality before even considering testing a release. Resolving this trust issue is a root driver of the transformation of testing.

We discussed the necessary shift in mindset required to digitize software development by making every person a developer and democratizing the entire process. We also touched on the value of integrating testing developers into the early design and develop stages. In this second post, we will examine in more detail this shift in testing to understand how it transforms the entire development cycle to the benefit of customers as well as developers.

Our goal in the Cisco platform independent group, which provides routing and control plane protocols and DevOps tools to the XE, XR and NX software development teams, is to digitize and transform processes and skillsets to create a hyper-efficient development organization. In particular, we are integrating the development of unit, integration, feature, system, and solution tests into the early stages of the development cycle with real-world use cases based on diverse customer network hardware and software configurations and topologies. How do we capture this detailed customer information? We listen. We share. We communicate.

Bidirectional Communication with Customers Critical in Early Development Stage

We are engaging customers much earlier in the development lifecycle with a goal to build a bidirectional communications channel between Cisco development and customers. First, we listen to understand customer requirements, topologies, and traffic patterns and feed those parameters into our design documents. We request customers’ device configuration files so we can prepare test plans incorporating an appropriate mix of “live in the field” hardware and software environments. We then verify with customer IT teams our design specifications to ensure a mutual understanding of goals. By providing insights into feature functionality and sharing test plans, customers can better prepare for implementation before the final release. Customers can also share their proposed test plans with our teams so that special use cases can be incorporated into our test plans as well.

Cisco customers have been eager to participate in early engagement opportunities to provide real-time feedback on specific feature designs and implementations. A participating customer related to our teams that the recent collaboration with Cisco Engineering “…was fruitful as it ensured that Cisco’s implementation of a specific feature was matching our expectations. Early engagement helps us understand new features so we can create successful design documents as well as train our certification teams. This early collaborative process also helps our team avoid ‘working as designed’ surprises during our testing.”

These collaborations among Cisco development teams and customers result in a reimagining of test design and procedures that permeate the development lifecycle.

Reimagining Testing Throughout Development

As we’ve previously discussed, within our platform-independent teams, everyone is a developer—from solution architects and designers to coders and testers. Each role plays a hand in ensuring the solutions and tools we build meet our customers’ requirements—whether internal teams or external enterprise IT organizations.

One key method of transforming testing efficiency and completeness is to integrate developers into the process who have in-depth experience with customer implementations, configurations, and troubleshooting. They participate upfront in the design stage to ensure that new features will work in real-world brownfield as well as greenfield environments. This change makes it possible to evolve from thinking primarily in terms of individual features that are designed, developed, and tested in isolation, to a customer-oriented solution approach. While each feature is coded with specific functionality by design, each must also be implemented as part of a complete networking ecosystem. Applying this philosophy not only helps identify unintended feature interactions, but also moves defect discovery to much earlier in the development cycle, in effect flattening the curve of found defects throughout the development cycle—a primary goal of testing transformation.

New features are not the only testing points to emphasize during the design phase. Since the main “users” of networking software are highly-trained technical professionals, serviceability is key to keeping them productive. For example, interfaces providing data such as telemetry and error codes, as well as CLI formats, are designed from the technical users’ point of view. In design documents, we consider how to expose sufficient debug information to enable faster problem resolutions, but without overwhelming technicians with irrelevant details. Here we are applying machine reasoning to assist in triaging issues. Ease of configuration of network devices and Day 2 management are also critical considerations for testing usability and serviceability. Training and automated checklists ensure that developers are abiding by serviceability guidelines and applying serviceability measurement to code during development.

New software releases are also scrutinized to minimize any unexpected changes in default behaviors. From release to release, behavior testing ensures that:

◉ Software doesn’t consume more memory or processing capacity than in a previous release unless a new feature requires it and is thoroughly documented to prepare the customer.

◉ New releases are backward compatible with supported hardware and software.

◉ Scale and performance do not degrade but stay consistent or improve.

Ultimately our goal in reimagining testing is to build a lasting bridge to quality to ensure our customers have trust in each and every release. While we have always performed intensive feature testing to validate functionality, integration, scalability, and usability, we are emphasizing a significant focus on solution level testing to ensure high levels of performance, interoperability, reliability, security, and conformance. Combined, these layers of testing will provide greater assurance that releases will perform as expected in a multitude of customer environments. We are building this bridge to quality with a unified development infrastructure for testing.

Unified Development Infrastructure Increases Automation and Consistency

Software in the process of being coded is often tested in virtual testbeds that can be quickly modified. This usually works fine for unit and integration testing. However, the further along the development cycle, the more complex the testing and interactions with the environment. Virtualized testing may not uncover all the issues that will be discovered in real-world configurations.

To address this gap, we are building flexible testbeds based on real hardware—routers, switches, servers, access points and software—that mimic real network deployments and operations. Since testbeds are based on a common infrastructure and environment, they enable reuse, code sharing, and complimentary software testing. Unifying topologies and infrastructure in development and testing improves efficiency by uncovering issues earlier in the cycle.

The next phase, already in progress, is to create “topology on demand” testbeds that enable developers to design tests based on a variety of environments and have them automatically configured, based on network devices customers are actually using. We are also creating new tools to automate whole testing processes with reusable Test Blocks. These will enable developers to pick and choose from a library of pre-constructed tests. In turn, the tests are run with automation tools that perform the processing and recording of results. The testing process becomes more of an intellectual design exercise compared to manually assembling and running test after test with slight variations—a boon for developers working on tight timelines.

Transformation of Software Testing Benefits Developers and Customers

Reimagining and transforming the development testing cycle is paying off at Cisco in multiple ways. Internally, new tools for automating testing processes are making work more efficient and more engaging for developers at every stage of the software cycle. As we involve customer teams earlier in the development cycles, they are regaining trust in software release readiness and are willing to deploy new solutions sooner after release with more confidence.