Empowering the four IT personas using Cisco DNA Center with Rings of Power

There are many variations of the “Law of Constant Change”; while they all have their own spin on it, the common thread is that change is constant and that it needs to be harnessed. When looking at changes and disruptions in technology, it comes as no surprise that there are numerous transformations and trends which are reshaping the IT landscape. The megatrends and change drivers span a wide range of business changes and transformation agents such as:

To keep up with the rapidly changing IT landscape, many IT organizations have been able to ascend and transform into new operational paradigms with the xOps transformation. Conversations around agility, AIOps, NetOps, SecOps, and DevOps are an outcome of a combination of organizational behavior and tooling in the networking and infrastructure realms. Separately, Gartner has also identified four IT personas (NetOps, SecOps, AIOps, and DevOps) which Gartner defined as predominant roles in today’s network operations realm.

In looking at key challenges, organizations are struggling with:

◉ Reducing time recovery objectives due to the reactive nature of traditional network operations practices.

◉ Bridging the growing IT skill gap.

◉ Keeping up with changing business requirements.

◉ Delivery of secure services in the hybrid workplace.

◉ Having to deliver more with less.

With Cisco’s years of expertise in designing, operating, and supporting networks of all sizes across the globe. Cisco has been an instrumental part in helping IT organizations move forward to the next operational level with tools to embrace and enable the xOps personas and embark on the transformation journey. This boils down to providing tools with analytics capabilities from the infrastructure and cultivating staff skills to use them effectively.

Speaking of how tooling can enable the transition, Cisco DNA Center is at the center of the IT/OT transition into the four IT personas, providing the digital agility to drive network insight automation and security while promoting key capabilities and tools to help in skill cultivation and changed operational models.

Network Operations or “NetOps” is the front line of administrators in the IT organization. The term NetOps is a way to classify the common tasks and responsibilities, or “Jobs to be Done,” by these individuals. With Cisco DNA Center at the heart of the network infrastructure, the NetOps persona is enhanced with varying levels of automation to simplify the creation and maintenance of networks with agile flexibility to move from manual tasks to AI-assisted to selectively autonomous network management. For example, the SWIM (Software Image Management) and network profiles feature not only save time but allow for consistency and elimination of human error with routine tasks. The NetOps automation brought into DevOps provides agility and scalability to IT organizations to keep up with changing demands and integration into the larger IT ecosystem. Gartner has stated that the next generation of Netops, which Gartner coined as “Netops 2.0” is the evolution of network operations towards automation.

Network, application, and user security is a key requirement for any enterprise network, and no network can operate safely without security. The security team is responsible for providing a safe digital experience in today’s connect-from-anywhere hybrid work environment and networks with countless numbers of endpoint devices. Also, many IT organizations in different market segments have various network security and architecture recruitments. Cisco DNA Center empowers the SecOps persona by enabling the complete zero-trust workplace solution with AI-driven security to classify endpoints and automated enforcement of security policies. This is achieved with Cisco’s fully integrated platform, which incorporates hardware and software designed to provide contextual security insights and automation. Cisco DNA Center SecOps can help eliminate security vulnerabilities with proactive security scans, automated security advisory alerting Cisco’s Product Security Incident Response Team (PSIRT), and proactive bug scans powered by Cisco AI Network Analytics engine to ensure the network is always secure.

The DevOps persona brings integration, automation, and orchestration together. Traditionally, DevOps teams focused on very specialized, proprietary, and home-spun applications. Today, these individuals are tasked with taking these apps and integrating them into a connected universe of corporate solutions. DevOps depends on manufacturer-supplied software tool kits (STKs) and standards-based application programming interfaces (APIs) in order to share information and intelligence between applications. With Cisco DNA Center, IT organizations can quickly utilize pre-built integrations to Cisco products and 3rd party enterprise applications such as ServiceNow, Splunk, PagerDuty, and a growing selection of partner integrations. Cisco DNA Center’s mature APIs enable the extraction of data and network management, leveraging and harnessing the power of Cisco DNA Center’s NetOps, AIOps and SecOps via the API interface.

AIOps defines the technologies that implement AI/ML (Artificial Intelligence and Machine Learning) and the individuals that leverage these technologies. Evidently, AI/ML is being implemented in so many of our networking components that it has become imperative that a specialized team of experts manage and amplify the use of this intelligence. Cisco DNA Center provides a simplified view into the complexities of big data and machine learning so that your AIOps teams can make the most of this rich data.  Additionally, Cisco DNA Center provides best-in-class AI-driven visibility, observability, and insights, ensuring the health and experience of users, applications, and infrastructure. AI/ML is packaged within Cisco DNA Center in an easy consumption interface that can deliver value in minutes and allow IT teams to work smarter and elevate the level of service to the users and organization. Hence, with Cisco DNA Center AIOps, IT organizations can gain visibility and insights otherwise not attainable without AI/ML combined with Cisco’s deep networking knowledge. Simply put, this powerful combination makes the IT team more agile and smarter and helps bridge growing IT skills gaps.

The xOps Rings of power

While the four IT personas were explained as distinct roles, in many organizations, they are simply different hats that IT staff can wear at different times depending on the business need.  It is also essential to keep the perspective that each of the personas enables and provides services to other personas, yielding the “Rings of Power” for example, with AI centricity, Cisco DNA Center empowers, enables, and enhances the NetOps, SecOps, and DevOps personas by providing interactions with all personas in the ring. Similarly, NetOps persona-centricity enables and empowers DevOps, SecOps, and AIOps personas.

An example of the AIOps ring of power:

AIOps discovers security vulnerabilities and recommends an upgrade.

NetOps performs the SWIM process to upgrade the software.

DevOps connects to ServiceNow for the change management and ticket creation processes.

SecOps reports the new network security posture, eliminating the security vulnerability from the network.

Leveraging Cisco DNA Center to enable and empower the new IT personas model, IT organizations can quickly and easily gain visibility, observability, insights, and out-of-the-box automation. While organizations with more modern operational models are also able to yield zero trust, and programmability from the Cisco Network infrastructure. This enables IT organizations to be more agile and transform into the new xOps operational paradigm, allowing the IT organization to progress on the operational maturity journey, become proactive and leave the reactive persona behind.

Source: cisco.com

Continue reading

Quick automation wins with Cisco DNA Center

With the investment into today’s modern and agile networks, many IT organizations are searching for intelligent tools that can help simplify the complexity that comes with the advanced capabilities of today’s networks and keep up with the business demands. Topping off the complex challenges, many organizations are facing challenges on how to bridge the growing IT skill gap and automate various aspects of their network management.

In a recent Gartner article regarding the State of Network Automation, according to the article:

◉ 41% of network activities are less than 10% automated.

◉ 31% of network activities are 11% to 25% automated.

Essentially 72% of network activities are less than 25% automated. Separately, Gartner has also identified 4 IT personas (AIOps, NetOps, SecOps, and DevOps), stating that NetOps2.0 is the evolution of network operations towards automation.

Attributes of NetOps 2.0 include an Automation-first approach, embedded analytics, SecOps integrations, and Turn-key DevOps tools.  IT organizations that embrace this approach can achieve increased IT agility, Proactive network operations, and an increased level of collaboration between common silos in IT organizations. An additional outcome is minimized friction between the NetOps, SecOps, and DevOps personas.

When it comes to automation products, the Inventor’s paradox states, “It is easier to solve a more general problem that covers the specifics of the sought-after solution”.  Organizations who transitioning to AIOps, NetOps2.0, and automation platforms, are faced with common challenges and limitations such as:

◉ Automation products are often not bi-directional with network equipment

◉ Third-party products lack Cisco’s deep understanding of the network and platforms

◉ Lack of tight integration between the hardware and software platforms

◉ Lack of cross-domain visibility between the campus, data center, and the cloud

◉ Reliance on legacy SNMP protocol which provides limited visibility and control

◉ Limited AI capabilities due to lack of data quality and domain specialization

Out-of-the-box automation with Cisco DNA Center

While there are various barriers to network automation, there are some pragmatic methods by iterating on non-change and/or non-production automation activities, leading to some “quick automation wins.” Below are some “quick automation wins” examples available out of the box with Cisco DNA Center automation.

◉ Network Device Configuration Backup and archival of all network devices.

◉ Integration with ServiceNow, which automats auto-population of trouble tickets.

◉ Automated creation of network availability baselines and compliance reporting.

◉ Automated creation of user experience baselines and reporting.

◉ Maintenance mode to enable/disable monitoring during change windows.

◉ Automated network performance testing with MRE (Machine Reasoning Engine) and features such as Truetrace and path trace to automate and expedite troubleshooting.

◉ Automated packet capture for network anomalies.

◉ Redundant Link Monitoring.

◉ RMA Automation workflows.

◉ Automated creation of application health and reporting.

◉ Software Upgrade Cycle

Granular Automation Control

In looking at Cisco DNA Center’s automation suite, Cisco DNA Center not only provides automation features but also provides the granular control to enable workflows and actions from manual to AI-assisted to selectively autonomous change management. Let’s look at the three modalities of automation possible with Cisco DNA Center:

Manual (clickOps) is where many organizations are today; all administrative actions are performed by or initiated by an operator. Numerous automated workflows need manual initiation, but they still automate numerous repetitive steps such as SWIM for software updates. Additionally, some of these can be automated through templates and EEM (Embedded Event Manager) triggers.

Figure 1. Cisco DNA Center (SWIM) Software Image Management Cycle

AI-Assisted is where leveraging the depth of knowledge, streaming telemetry, and Cisco’s vast knowledge and experience in running networks; Cisco DNA Center can identify issues and use the MRE to suggest troubleshooting steps and possible remediation. MRE is a network automation engine that uses AI (artificial intelligence) and ML (machine learning) to automate complex network operation workflows. This feature encapsulates human knowledge and expertise into a fully automated inference engine to help you perform complex root cause analysis, detects issues and vulnerabilities, and either manually or automatically perform corrective actions.

Figure 2. Cisco DNA Center Compliance automation with configuration drift

Autonomous Change Management (ACM) provides for Cisco DNA Center to be enabled to perform and enforce automated actions on the network under predefined conditions and events. As today’s networks grow at incredible rates with new demands, manually managing all aspects of the network is no longer feasible for humans. Nor do most organizations have staff watching alerts every second of the day. The integration of AI/ML into the automation engine enables Cisco DNA Center to regularly tune the network based on predictions and models, which can greatly optimize the user experience and network performance.  Compare human intervention as the ax vs. AI-driven automation doing it with a scalpel.  This can be the difference between a system taking proactive measures vs. correcting an issue after it occurred.

Doing a left shift and taking automation to the next level, depending on the intents and architecture of the network, there are several highly automated deployment models, such as the Software-Defined Access (SDA), User Defined Networking (UDN), and AI-RRM, which are highly ACM deployments within the Cisco DNA Center solutions suite.

Focusing on automation outcomes and benefits

Focusing on outcomes, as organizations embark on network automation, there are various success metrics and business outcomes that can be tracked, such as:

Tangible Metrics	Intangibles
Faster moves adds and changes Consistent Configuration Quicker MTTR Reduction in network issues Improved security posture	Team Agility Ability to scale at speed Bridging the IT skill gap

Source: cisco.com

Continue reading

How To Do DevSecOps for Kubernetes

In this article, we’ll provide an overview of security concerns related to Kubernetes, looking at the built-in security capabilities that Kubernetes brings to the table.

Kubernetes at the center of cloud-native software

Since Docker popularized containers, most non-legacy large-scale systems use containers as their unit of deployment, in both the cloud and private data centers. When dealing with more than a few containers, you need an orchestration platform for them. For now, Kubernetes is winning the container orchestration wars. Kubernetes runs anywhere and on any device—cloud, bare metal, edge, locally on your laptop or Raspberry Pi. Kubernetes boasts a huge and thriving community and ecosystem. If you’re responsible for managing systems with lots of containers, you’re probably using Kubernetes.

The Kubernetes security model

When running an application on Kubernetes, you need to ensure your environment is secure. The Kubernetes security model embraces a defense in depth approach and is structured in four layers, known as the 4Cs of Cloud-Native Security:

Read More: 350-801: Implementing Cisco Collaboration Core Technologies (CLCOR)

1. Cloud (or co-located servers or the corporate datacenter)

2. Container

3. Cluster

4. Code

Security at outer layers establishes a base for protecting inner layers. The Kubernetes documentation reminds us that “You cannot safeguard against poor security standards in the base layers by addressing security at the Code level.”

At the Cloud layer, security best practices are expected of cloud providers and their infrastructure. Working inward to the Cluster layer, cluster components need to be properly secured, as do applications running in the cluster.

At the Container level, security involves vulnerability scanning and image signing, as well as establishing proper container user permissions.

Finally, at the innermost layer, application code needs to be designed and built with security in mind. This is true whether the application runs in Kubernetes or not.

In addition to the 4 C’s, there are the 3 A’s: authentication, authorization, and admission. These measures apply at the Cluster layer. Secure systems provide resource access to authenticated entities that are authorized to perform certain actions.

Authentication

Kubernetes supports two types of entities: users (human users) and service accounts (machine users, software agents). Entities can authenticate against the API server in various ways that fit different use cases:

◉ X509 client certificates

◉ Static tokens

◉ Bearer tokens

◉ Bootstrap tokens

◉ Service account tokens

◉ OpenID Connect tokens

You can even extend the authentication process with custom workflows via webhook authentication.

Authorization

Once a request is authenticated, it goes through an authorization workflow which decides if the request should be granted.

The main authorization mechanism is role-based access control (RBAC). Each authenticated request has an HTTP verb like GET, POST, or DELETE, and authenticated entities have a role that allows or denies the request. Other authorization mechanisms include attribute-based access control (ABAC), node authorization, and webhook mode.

Admission

Admission control is a security measure that sets Kubernetes apart from other systems. When a request is authorized, it still needs to go through another set of filters. For example, an authorized request may be rejected by an admission controller due to quotas or due to other requests at a higher priority. In addition to validation, admission webhooks can also mutate incoming requests as a way of processing request objects for use before reaching the Kubernetes API server.

In the context of security, pod security admission might add an audit notation or prevent the scheduling of a pod.

Secrets management

Secrets are an important part of secure systems. Kubernetes provides a full-fledged abstraction and robust implementation for secrets management. Secrets are stored in etcd—Kubernetes’ state store—which can store credentials, tokens, SSH keys, and any other sensitive data. It is recommended to store small, sensitive data only as Kubernetes Secrets.

Data encryption

When you want to store a large amount of data, consider using dedicated data stores like relational databases, graph databases, persistent queues, and key-value stores. From the vantage point of security, It’s important to keep your data encrypted both at rest (when it is simply sitting in storage) as well as in transit (when it is sent across the wire). While data encryption is not unique to Kubernetes, the concept must be applied when configuring storage volumes for Kubernetes.

Encryption at rest

There are two approaches to encryption at rest. The first approach uses a data store that encrypts the data for you transparently. The other approach makes the application responsible for encryption, then storing the already-encrypted data in any data store.

Encryption in transit

Eventually, you’ll need to send your data for processing. Because the data is often (necessarily) decrypted at this point, it should be sent over a secure channel. Using  HTTPS, STCP, or SFTP for secure transit of data is best practice.

Kubernetes services can be configured with specific ports like 443 for HTTPS.

Managing container images securely

Kubernetes orchestrates your containers. These containers are deployed as images. Many Kubernetes-based systems take advantage of third-party images from the rich Kubernetes ecosystem. If an image contains vulnerabilities, your system is at risk.

There are two primary measures to safeguard your system. First, use trusted image registries, such as Google Container Registry, AWS Elastic Container Registry, or Azure Container Registry. Alternatively, you may run your own image registry using an open-source project like Harbor and curate exactly which trusted images you allow.

The other measure is to frequently scan images for vulnerabilities as part of the CI/CD process.

Defining security policies

Kubernetes and its ecosystem provide several ways to define security policies to protect your systems. Note that the built-in Kubernetes PodSecurityPolicy resource is deprecated and will be removed in Kubernetes 1.25. At the time of this writing, the Kubernetes community is working on a lightweight replacement. However, the current recommendation is to use a robust third-party project—for example, Gatekeeper, Kyverno, or K-Rail—as a policy controller.

Policies can be used for auditing purposes, to reject pod creation, or to mutate the pod and limit what it can do. By default, pods can receive traffic from any source and send traffic to any destination. Network policies allow you to define the ingress and egress of your pods. The network policy typically translates to firewall rules.

Resource quotas are another type of policy, and they’re particularly useful when multiple teams share the same cluster using different namespaces. You can define a resource quota per namespace and ensure that teams don’t try to provision too many resources. This is also important for security purposes, such as if an attacker gains access to a namespace and tries to provision resources (to perform crypto mining, for example).

Monitoring, alerting, and auditing

We have mostly discussed preventative measures thus far. However, a crucial part of security operations is detecting and responding to security issues. Unusual activity could be a sign that an attack is in progress or that a service is experiencing degraded performance. Note that security issues often overlap with operational issues. For example, an attacker downloading large amounts of sensitive data can cause other legitimate queries to time out or be throttled.

You should monitor your system using standard observability mechanisms like logging, metrics, and tracing. Kubernetes provides built-in logging and metrics for its own components. Once a serious problem is discovered, alerts should be raised to the relevant stakeholders. Prometheus can provide metrics monitoring and alerting, while Grafana provides dashboards and visualizations for those metrics. These tools, along with AppDynamics or countless others, can serve as effective Kubernetes monitoring solutions.

When investigating an incident, you can use the Kubernetes audit logs to check who performed what action at a particular time.

Source: cisco.com

Continue reading

What DevSecOps Means for Your CI/CD Pipeline

The CI/CD (Continuous Integration/Continuous Deployment) pipeline is a major ingredient of the DevOps recipe. As a DevSecOps practitioner, you need to consider the security implications for this pipeline. In this article, we will examine key items to think about when it comes to DevSecOps and CI/CD.

The type of CI/CD pipeline you choose—whether it’s managed, open source, or a bespoke solution that you build in-house—will impact whether certain security features are available to you out of the box, or require focused attention to implement.

Let’s dive in

Secret management for your CI/CD pipeline

Your CI/CD pipeline has the keys to the kingdom: it can provision infrastructure and deploy workloads across your system. From a security perspective, the CI/CD pipeline should be the only way to perform these actions. To manage your infrastructure, the CI/CD pipeline needs the credentials to access cloud service APIs, databases, service accounts, and more—and these credentials need to be secure.

Managed or hosted CI/CD pipelines provide a secure way to store these secrets. If you build your CI/CD solution, then you’re in charge of ensuring secrets are stored securely. CI/CD secrets should be encrypted at rest and only decrypted in memory, when the CI/CD pipeline needs to use them.

You should tightly lock down access to the configuration of your CI/CD pipeline. If every engineer can access these secrets, then the potential for leaks is huge. Avoid the temptation to let engineers debug and troubleshoot issues by using CI/CD credentials.

Some secrets (for example, access tokens) need to be refreshed periodically. CI/CD pipelines often use static secrets—which have much longer lifetimes, and so don’t need regular refreshing—to avoid the complexities of refreshing tokens.

Injecting secrets into workloads

Cloud workloads themselves also use secrets and credentials to access other resources and services that their functionality depends on. These secrets can be provided in several ways. If you deploy your system as packages using VM images or containers, then you can bake the secrets directly into the image, making them available in a file when the workload runs.

Another approach is to encrypt the secrets and store them in source control. Then, inject the decryption key into the workload, which can subsequently fetch, decrypt, and use the secrets.

Kubernetes allows for secrets that are managed outside of the workload image but exposed as an environment variable or a file. One benefit of secrets as files is that secret rotation doesn’t require re-deploying the workload.

Infrastructure as code: a security perspective

Infrastructure as code is not only an operational best practice; it is also a security best practice. 

software systems = infrastructure + workloads

When ad hoc changes are made to infrastructure configurations, this drift can introduce security risks. When resources are provisioned without any auditing or governance, it becomes difficult to maintain proper security measures across all resources.

Manage your infrastructure just like you manage your code. Use declarative configurations (like those of  Terraform, AWS CloudFormation, or Kubernetes CRDs). Review and audit every change.

Bring your own security tools

CI/CD pipelines are flexible. Generally speaking, they let you execute a sequence of steps and manage artifacts. The steps themselves are up to you. As a security engineer, you should take advantage of the security tools that already exist in your environment (especially in the cloud). For example, GitHub and GitLab both scan your commits for the presence of secrets or credentials. Some managed CI/CD solutions build in API scanning or application security scans. However, you may also prefer to add tools and checks into the mix.

You could also add static code analysis (like SonarQube) to ensure that code adheres to conventions and best practices. As another example, you mayincorporate vulnerability scanning (like Trivy or Grype) to your CI/CD pipeline, checking container images or third-party dependencies for security flaws.

Comprehensive detection and response

Application observability, monitoring, and alerting are fundamental DevOps Day 2 concerns. Although your CI/CD pipeline is not directly involved in these activities, you should use your CI/CD pipeline to deploy the security tools you use for these purposes. From the point of view of the CI/CD pipeline, these are just additional workloads to be deployed and configured.

Your CI/CD pipeline should include early detection of security issues that trigger on every change that affects workloads or infrastructure. Once changes are deployed, you need to run periodic checks and respond to events that happen post-deployment.

In case of faulty CI/CD, break glass

The CI/CD pipeline is a critical part of your system. If your CI/CD is broken or compromised, your application may continue to run, but you lose the ability to make safe changes. Large scale applications require constant updates and changes. If a security breach occurs, you need to be able to shut down and isolate parts of your application safely.

To do so, your CI/CD pipeline must be highly available and deployed securely. Whenever you need to update, rollback, or redeploy your application, you depend on your CI/CD pipeline.

What should you do if your CI/CD pipeline is broken? Prepare in advance for such a case, determining how your team and system will keep operating (at reduced capacity most likely) until you can fix your CI/CD pipeline. For complicated systems, you should have runbooks. Test how you will operate when the CI/CD is down or compromised.

Source: cisco.com

Continue reading

A Framework for Continuous Security

Technology is at the core of business today. Maintaining the resiliency of critical data, assets, systems, and the network is mission-critical; crucial to meeting business goals. As a result, development operations (DevOps) professionals must continuously improve the overall resilience —along with the security posture — of workloads, software, and applications (Figure 1). To do this at scale and speed requires the integration of a suite of application security tools in the continuous integration/continuous delivery (CI/CD) pipelines that automate posture assessment and provide visibility to help manage security risks.

At Cisco, we learned early on that application security processes were inhibiting our business agility. We knew we had to embrace an Agile and DevOps culture as early adopters to deliver software products based on business demands rapidly and iteratively. Agile DevOps without application security automation leads to a “hurry up and wait” situation, where some processes move quickly only to be bogged down by others. With evolving technologies such as cloud, Docker, Kubernetes, open-source as well as daily and frequent release cycles, it is hard for application security teams to keep up with the threat landscape. In a typical modern application development and deployment technology stack, 80% of the code base is comprised of third-party software. Only 20% is custom code. Most of the security breaches we have seen in recent years were entirely preventable had there been necessary security measures taken, not only for the custom code but also for the third-party software.

We set out to create a DevSecOps culture that empowers the application teams to continuously build and deploy secure applications instead of being gated by a central security function. To do this, we integrated and orchestrated a suite of application security tools within CI/CD pipelines under a program called Continuous Security Buddy (CSB) for CI/CD pipeline edition. It enables the development teams to ramp up their application security program while making application security transparent and friction-free.

Figure 1: DevSecOps – Security Implementation as Code

We used the following basic principles in the design of the program:

◉ Co-design and co-develop the security automation solution so it can work for the DevOps teams

◉ Integrate the DevSecOps workflow and empower the developers by giving them the flexibility to choose their application security tools

◉ Propagate security compliance requirements and hence eliminate the security friction points between security and development teams that impact development velocity

Co-design and Co-development of the Solution

We initially co-designed and co-developed CSB for CI/CD re-usable automated security capabilities using joint scrum planning with teams from Cisco Webex. To encourage adoption across development teams, we created an innovative, configurable rollout of CSB for CI/CD shared libraries to simplify the process.

Shared libraries are a collection of pipeline code made for Jenkins that can be used by any pipeline to reference any available code quickly. With one line of code in Jenkins, developers can access all the security scans available in the shared library. The shared library framework simplifies the code contribution workflow via the inner-source process and reusable code configuration in the pipeline by any team using Jenkins.

We quickly learned that we needed to provide CI-agnostic solutions for teams that used other CI tools. We offered such a solution using containers that are published in a centralized repository for development teams to access via Docker.

Security Scan Flexibility

Users can choose what type of automated security scans they want to configure and run. For example, a production pipeline may consist of a binary image scan, static code analysis scans, and a way to view a consolidated report of scans. The final step in the automation process is to send the scan results aligned to Security Control Framework (SCF) to a centralized security platform to meet compliance requirements. These features are all available as part of the shared library and the user needs to add configuration parameters to run it. As part of the CI process, security scans are configured and triggered to run whenever there is a code change. Developers can then continuously monitor the scan results for any new security issues.

Automated Compliance Reporting

Using the CSB for CI/CD shared library, teams can view reports generated from each security scan on the Jenkins dashboard and identify any failing security issues. Teams can also send the security results data to a centralized interface to Jira to help in various assessment processes, such as reviews by security architects. A consolidated report is generated (as shown in Figure 2), which shows an overall compliance score that considers which scans were enabled in the job, (e.g., binary scans, static code analysis, and dynamic scans). Developers can then use this report to view any quick fixes to improve the security posture.

Figure 2. CSB for CI/CD Scan Report

Measuring Progress and Success

After initial development with our Webex team, we scaled the CSB CI/CD approach across several business units at Cisco. We measured the agility, reliability, efficiency, quality, and success of the CSB for CI/CD shared library to ensure the system was operating effectively.

With the program now in place for over a year, some of business value we were able to deliver is captured in Figure 3.

Figure 3. CSB for CI/CD Benefits

Continue reading

How do you gauge software quality before deployment?

Business leaders often question software development processes to identify their effectiveness, validate if release quality is maintained across all products and features, and to ensure smooth customer deployments. While providing data from multiple perspectives, I hear teams struggling to respond in a meaningful way. In particular, it’s hard for them to be succinct without communicating nitty-gritty details and dependencies. Is there a way in which we can objectively arrive at the release quality measurement to ensure an expected level of quality? Absolutely!

With more than two decades in the industry, I understand the software development life cycle thoroughly, its processes, metrics, and measurement. As a programmer, I have designed and developed numerous complex systems, led the software development strategy for CI/CD pipelines, modernized processes, and automated execution. I have answered the call for stellar customer journey analytics for varied software releases, allowing our business to grow to scale. Given my background, I would like to share my thoughts on our software development process, product and feature release quality, and strategies to prepare for successful product deployments. I look forward to sharing my opinions and collaboratively working with you on building customer confidence through high-quality software deployments.

But, before I begin, here are some terms the way I think of them:

◉ Product Software Release— Mix of new and enhanced features, internally or customer-found defect fixes, and may contain operational elements related to installations or upgrades.

◉ Software Release Quality— Elements like content classification, development and test milestones, quality of the code and test suites, and regressions or collateral to track release readiness prior to deployment.

◉ Release Content— Classified list of features and enhancements with effort estimations for development. For example, we may use T-shirt size classification for development efforts (including coding, unit testing, unit test found bug fixes, and unit test automation): Small (less than 4 weeks), Medium (4-8 weeks), Large (8-12 weeks), Extra-Large (12-16 weeks). Categorize feature testing similarly as well.

◉ Release Quality and Health— Criteria for pre-customer deployment quality, with emphasis on code and feature development processes, corresponding tests, and overall release readiness.

Through this lens, let’s view the journey of our Polaris release. Before we do, let me emphasize that quality can never be an afterthought, it has to be integral to the entire process from the very beginning. Every aspect of software development and release logistics require you to adopt a quality-conscious culture. I believe there are four distinct phases or checkpoints to achieve this goal:

◉ Release content, execution planning and approvals— During this phase we must get our act straight. Good planning will yield great results. Preempting issues and executing on a mitigation plan is critical. Adopt laser-sharp focus on planning for features that will be developed and tested. To be effective, we must allocate a 70:20:10 ratio for complex and large features. Seventy percent of the challenging features will have to be developed first and tested early in the release cycle, twenty percent of them will be addressed in the next cycle, and ten percent at the end of the cycle. Small, medium and test-only features should be distributed throughout the development cycle depending on resource availability. In this way, the majority of the completed code can be tested early in the process and in parallel! This will help us drive shift left best practices and make them integral to the culture of our organization.

◉ Phase containment, schedules, and quality tracking—This phase represents the core of execution. We need to build a framework for success to guarantee quality. The key is to develop fast, tackle the complex stuff early, and to allow ample time for soak testing. Build the metric and measurement around it. Phase containment is essential for success. During this phase, focus on development and design issues, automation, code coverage, code review, static analytics, code complexity, and code churn data analytics to help build quality. Build the metrics and measurement around these elements and adhere to development principles. If any features do not meet the schedule or quality checkpoints, we must be prepared to defer them and remove them from the release train. The quality metrics should include, the number of features that have met their development schedule, undergone the feature/functional tests with 100% execution, and can claim a 95% pass-rate! If we follow an agile development model, each developmental and validation task must be tracked per sprint. We must document the unit-test found defects at the end of the sprint cycle; especially, if they move from one sprint cycle to the next. Daily defect tracking and weekly review with executives will bring the required attention and visibility as well.

The following image illustrates one such scenario:

◉ Testing and defect management convergence— This phase can make or break a release. Since development is complete and a certain level of quality has been achieved (though not quite release-ready) it entails more rigorous testing. Tests, such as system integration testing, solutions and use case-driven testing, and performance and scale testing, provide greater insight into the quality of the release. Use time effectively in this phase to track the test completion percentages, the pass-rate percentages, and your metrics surrounding defect management. Defect escape analysis testing will highlight developmental gaps while making for a good learning opportunity. Another invaluable metric is to study the trend of incoming defects. If things are working as they should, you will notice a steady decline each week. The incoming defect rate must decline towards the end of testing cycle! This is a key metric as well.

◉ Ready, set, go— In this phase, embark on a stringent review of readiness indicators, carry out checks and balances, address operational issues, finalize documentation content, and prepare for final testing. Testing may entail alpha, canary, early field trials in a real customer environment, or tests in a production environment to uncover residual issues. This phase will provide an accurate insight into the quality of the release.

As you can see, there are many ways we can equip ourselves to measure the quality and the health of a release. Building a process around developing the quality code and discipline in managing the phase containment are the key ingredients. It is important to build a culture to track the progress of shift left initiatives, focus on code quality and schedule discipline. Best of all, data-driven analytics and metrics will empower us to answer all queries from executives with confidence!

Continue reading

Unify NetOps and SecOps with SD-WAN Cloud Management

CIOs know that ubiquitous connectivity across domains—campus, branch, cloud, and edge, wired or wireless—is a baseline requirement for building a digital enterprise. But, as CISOs know, as the network fabric spreads to encompass devices and location-agnostic data and compute resources, the need for end-to-end integrated security is equally paramount. Add in the necessity to continuously monitor and maintain application performance throughout campus and branch and edge locations and you create an enormous workload for NetOps and SecOps teams that are simultaneously dealing with static CapEx and OpEx budgets. Often the result is a tug-of-war between the teams: one striving to keep the network optimized for performance and availability, the other striving to keeping data, applications, and devices secure.

Conflict or Collaboration?

The problem of balancing the goals of NetOps with SecOps has a lot to do with how the network and all the connected devices and domains are being managed. Traditionally in NetOps, there have been separate consoles and Unified Computing Servers (UCS) to configure, monitor and analyze network domains – several for the data center, multiple for the campus wireless network, and still more for cloud, branch, and edge deployments.

Similarly, in order for SecOps to capture, log, and analyze traffic in all the various domains, special taps are installed where traffic is entering and leaving the domains. SecOps has an additional burden of storing all the traffic logs in case of a breach or successful malware attack in order to pinpoint the cause and prove appropriate steps are taken to remediate breaches and prevent future attacks.

That’s a lot of boxes to buy, install, and securely manage—a number that grows with each expansion of the enterprise network. Ironically, the extra compute devices needed by SecOps ultimately have to be managed by NetOps to ensure they do not affect overall network performance. Thus, more conflict.

Can NetOps and SecOps get to the point of collaboration instead of conflict? In fact, new cross-enterprise business initiatives make collaboration a necessity.

Digital Transformation Projects Benefit from Unified Operations and Security

As organizations seek new ways to connect with customers, suppliers, and service partners by making business processes personal and frictionless, they initiate application development efforts that span across operations. A unifying foundation for these development efforts are the NetOps and SecOps teams.

Deploying new multi-cloud applications or moving processes to the edge—retail outlets, branch offices, medical clinics—requires assurance that the network is responsive, always available, and secure. NetOps needs to work with Development teams to understand network SLAs and cloud usage requirements for the new apps. SecOps needs to ensure that the proper network permissions, segmentations, and polices are applied to the network at application launch time. NetSecOps collaboration is key to timely deployment of next-generation applications with security and the required levels of performance.

Collaboration is important too in the battle of the budgets. With IT budgets generally flat over the last few years, making sure NetOps and SecOps teams use both CapEx and OpEx funds judiciously is critical for maximum efficiency. There is an opportunity to combine NetOps and SecOps teams to generate the most value from the available budget, equipment, and knowledge of how an enterprise’s unique network responds to changes in applications and threats.

From these examples, you can see that unifying NetOps and SecOps has solid benefits for enterprise digital transformation efforts. Is there a technology platform that makes unification not only possible, but also makes the transition a natural evolution rather than a forced organizational change? By combining a software-defined network fabric with single-console cloud management, SD-WAN can play a significant role in the unification of NetSecOps.

SD-WAN Unified Network Cloud Management for NetSecOps

A primary benefit of Cisco SD-WAN powered by Viptela for NetSecOps is the ability to provide a single, role-based interface in Cisco vManage to control network performance, segmentation, and security. Through the lens of vManage, NetSecOps can:

◉ Install and configure branch SD-WAN routers remotely with Zero Touch Provisioning (ZTP)

◉ Automatically route traffic through the most efficient and cost-effective path (MPLS, broadband, direct internet, LTE/5G) using dynamic path selection.

◉ Manage performance, security, and access policies for cloud onramps to SaaS, IaaS, and colocations.

◉ Remotely configure and manage at the branch level the application-aware firewalls, URL-filtering, intrusion detection/prevention, DNS-layer security, and Advanced Malware Protection (AMP) to secure branch traffic that is using direct internet connections to SaaS applications.

◉ Drawing on policies set up in Cisco SD-Access and Identity Services Engine (ISE), NetSecOps can collaborate to configure segmentation rules that are uniformly applied across distributed locations to keep traffic separated—such as employee wireless access from payment system traffic—improving performance and security.

These are some of the benefits SD-WAN provides to a unified NetSecOps team. One console—vManage—to configure, monitor, and protect a distributed organization’s branches, remote workforce, and applications. Let’s double-click on two common yet difficult to manage situations—securing east-west branch traffic and accessing direct internet access SaaS/IaaS-hosted applications—to see how SD-WAN helps a unified NetSecOps team operate.

Managing and Protecting East-West Traffic Flow and Security in Branches

With the plethora of integrated security layers that comes with Cisco SD-WAN, traffic entering and leaving a branch is thoroughly inspected for application infiltration, intrusion by malware, and accessing known bad URLs. But there is still the tricky problem of when malware is introduced by a device or someone inside the branch network.

In the days of spoke and hub WANs, traffic from each device within a branch would be backhauled to the enterprise data center for inspection and verification, and then back to the branch. This has always been a troublesome scenario for NetOps as the traffic load for just backhauling and inspecting interfered with traffic that legitimately had to go the data center for additional processing. The alternative, of course, was to lock down all the endpoints in branches, limiting their flexibility and any options to BYOD for employees.

Securing Access to SaaS Applications via Direct Internet Connections

The workforce is quickly becoming more dependent on applications hosted in SaaS cloud platforms, such as Office 365, which require routing through direct internet access. With SD-WAN, NetSecOps can focus on not just fine-tuning application performance but also the defenses that secure the valuable corporate data being transmitted over the internet connections to and from branch sites. By using Cisco SD-WAN Cloud OnRamps to SaaS and IaaS clouds, the network selects the path that is the most effective to handle Azure, AWS, or Google Cloud workloads while the built-in layers of security provide protection with DNS URL filtering, advanced malware protection, and application-aware firewalls. Both application performance and security are managed by NetSecOps via the SD-WAN vManage cloud controller portal.

Fostering Collaboration Among NetOps and SecOps is Key to Network Agility

With Cisco SD-WAN’s ability to manage operations and security via the same cloud portal, it really is achievable to create a NetSecOps team that promotes collaboration, reduces CapEx and OpEx, and maximizes device and application QoE and security. Unifying these two critical functions helps create an agile network that makes digital transformation projects possible while keeping on top of advanced security threats. I’d like to hear your thoughts on the ways SD-WAN can provide better synergy between operations and security.

Continue reading

Tune in: “Demystifying Cisco Orchestration for Infrastructure as Code”

Automating the software development life-cycle

DevOps teams are becoming more agile, reducing costs, and delivering a superb customer experience by automating the software development life-cycle. Cisco Orchestration solutions extend the benefits of automation to the entire stack. Each layer of the underlying infrastructure is delivered as Code (IaC). Orchestrators reduce the complexity of programmability, operational state, and visibility. In this session, we decode the differences between domain-specific workflow automation versus cross-domain orchestration. Achieving the goal of ‘Automate everything’ requires the right tool for the right use case. With use cases in mind, we will cover several Cisco orchestration solutions in their respective domains and cross domain capabilities. A brief demo will showcase Open Source and Cisco Orchestration tools working together hand-in-hand.

Level Set

What is Infrastructure as Code (IaC)?

IaC means writing imperative or declaration code to automate programmable infrastructure deployments and manage configurations. Imperative is how you do something step-by-step, as opposed to declarative which is ‘what to do’ by abstracting the configuration and state. DevOps best practices such as source control, verification, and visibility are building blocks to support infrastructure types (compute, network, storage, etc) as code.

Why do we need IaC?

With the advent of Continuous Integration and Continuous Delivery (CI/CD), we are able to build pipelines to automate the entire software development life-cycle (SDLC). Continuous integration (CI) is a set of tools to develop applications. Continuous delivery (CD) is the process of delivering updated software releases to infrastructure environments such as test, stage, and production. Using IaC for these platforms and environments is paramount to enabling software agility and rapid time to value. One could say, IaC is the easy button to building infrastructure to deliver software or other IT services.

Orchestration

What is orchestration? Wikipedia defines orchestration as an automated arrangement, coordination, and management that defines the policies and service levels through automated workflows, provisioning, and change management. In this same vein, a coffee grinder is automation where a brewing machine is orchestration.

Why do we need orchestration in addition to scripting? Production grade IaC at scale requires orchestration versus scripting to deliver advanced features such as intent, policy, governance, and Service Level Agreements. By building IaC to include configuration management, CICD, and other advanced orchestration features, similar benefits to application development are now possible in large scale technology domains (mulit-cloud, containers, campus, WAN, Data Center).

With network automation in mind, we see many pit falls with multi-threaded tasks running exclusively from scripts. Step (1) gather facts, Step (2) set conditions, Step (3) loop through items in jinja2 templates and parse to TextFSM and save the data to YAML files. Step (4) Push changes to devices and validate.

This level of scripted multi-threaded workflow is difficult to manage at scale. The main concerns are slow changes, configuration drift, lack of operation state, out of band config overwrites, and disruptive rollbacks. In spite of the current gaps, the Ansible engine is one of my favorite tools for pushing network configurations. One could argue that Tower provides a workflow for the playbooks to manage the order of these tasks but no configuration state is possible. In order to remediate some of these gaps, Ansible engine is adding a new ‘facts’ resource module in a future 2.9 release.

Caution

Sometimes we automate ourselves into a corner with too many scripts. What happens when major platform changes are made to the scripting tools? We’ve experienced this before with python 2.x to 3.x.and the impact to many of our product SDK libraries. We are seeing it again with the major change to the Ansible engine coming in 2.9 to introduce the ‘facts’ resource module. This change requires users to rewrite their playbooks from scratch to use these features. As a caution, consider limiting scripting to single threaded (CRUD ) actions while shifting the complexities of operational state and rollback to the domain specific orchestration engine.

Domain Orchestration

What is a domain orchestration? A domain orchestration engine focuses on delivering automation targeted to a single technology domain. For instance, the Cisco Network Services Orchestator (NSO) is focused on model driven “network” automation with Netconf and YANG. NSO converts CLI to YANG with network element drivers (NEDS) to supports a multitude of uses cases ranging from stand alone network devices, network services, and multiple controller domains (Meraki, Viptela, and ACI).

In a nut shell, NSO can deploy greenfield or snyc-from brownfield devices to build a transaction based configuration database state. Tools like Ansible engine have modules to integrate with NSO’s northbound JSON API to harness these differentiated capabilities for operations. In the following example, we are using Ansible playbooks with the Ansible NSO/Json module to make CRUD changes to NSO’s configuration database as a means to configure and operate tenants running on a N9K EVPN/VXLAN Data Center network fabrics versus CLI to the stand alone NXOS. The Ansible playbooks are then version controlled as YAML files in a git repository.

Top-level Orchestration

What is Top-level orchestration? A top-level orchestration engine is used to stitch together collaboration, notifications, governance, and source control for other lower level scripting tools and device APIs. Top-level orchestration supports use cases ranging from CICD pipelines for application development to automated infrastructure build and testing. The Cisco Action Orchestrator (AO) is a powerful Top-level orchestrator that enables automated workflows across technology domains and ITSM (ie., ServiceNow). Integrations to ITSM are key for customers who need low or no code catalogs and templates to simplify the delivery of IT services.

Internally, Cisco relies on ITSM and AO to automate the rapid delivery of CiscoLive and Devnet Sandboxes during our customer events. In the below example, the open source tools such as Gitlab work hand-in-hand with AO to create a workflow pipeline to automate the build and test for a tenant configuration across EVPN/VXLAN fabric and SDWAN network domains.

Confusion

Are you confused by CICD pipelines and their relationship with IaC? My ‘Ah hah’ moment was a realization that many of these DevOps methodologies are not mutually exclusive but highly complementary between AppDev and IaC. As operators, we can support the automated development life-cycle with CICD pipelines. This same knowledge and tools are adaptable to automated infrastructures. Operations can adopt the tools (open source or vendor) that make sense for updating, configuring and management of IaC in many domains.

If you look at AppDev the CICD pipeline for software development must CODE, BUILD, TEST, and DEPLOY the software to an environment that includes infrastructure (compute, network, and storage). Do we build these infrastructure environments ahead of time manually or automated on demand?

If the developer is not willing to patiently wait several weeks for the infrastructure environment to test their CODE, then fully automated IaC is the only answer! A second CICD pipeline managing the configuration, versioning, and alignment of the software build to the environment (test, stage, prod) version allows us to move much quicker and rebuild the environment later if needed.

AppDev CI/CD pipeline to IaC CI/CD pipeline

In the following example, we are using Gitlab to manage an application development CICD pipeline. Upon completion the AppDev pipeline triggers Action Orchestrator to build a second pipeline with workflows to automate the test environment to ultimately test the application stack. The idea is to test the software release in a test environment prior to pushing the same software into production. The Action Orchestrator (AO) has many adapters to make IaC very easy to build and test infrastructure technology domains.

Router/switch software upgrades are another use case for a network specific CICD pipeline. With CICD we can automate the upgrade of specific IOS software versions to devices in a version controlled and tested environment prior to production.

Controllers

Are controllers and orchestration one in the same? NOPE…  Controllers are a single API touch point and management system for Software Defined systems (SD-Access, SD-Wan, and SD-Networks) to manage the configuration state of the underlay and overlay and underlying protocols. Controllers are similar to orchestration by providing access to configuration snapshots and rollbacks, but unable to compose top-level workflows with other tools. In most cases, Controllers are bound to their single technology domain (campus, data center, WAN, or cloud). Often times, IaC is configured adequately with only scripting and source control in a single controller domain. Suffice it to say, when expanding from a single domain to cross domain controllers (ie SDWAN, and SDN) this cross domain integration introduces a catalyst for orchestration.

The Automation Challenge

There is a broad set of technology domains, each with many use cases for IaC. In order to succeed with IaC, we first need to address our automation challenges. From there we can target each specific use case mapped to the appropriate technology domain.

Challenges:

To many touchpoints: Need to consolidate and coordinate tasks using common automation tools.

Complexity: Need to abstract automation as much as possible to make resources consumable for the end users.

Operational Instrumentation: Need to automate and operationalize the tools into workflows that include visual dashboards, role-based access control, and other security services.

Verification: Need to make changes and check changes. With automation, we can move really fast and break things. Hence, we need the proverbial looking over our shoulder versus traditional stare and compare configuration checks. Ideally, verification should start in a test environment through production.

Community and Collaboration: Need to share finished code and avoid recreating the wheel with every workflow.

The key take away for automated solutions is to strive for a sharing culture, agility, simplicity, intent, security, and lower costs.

Technology Domains and Use Cases

The following table depicts the taxonomy of several Cisco orchestration options. As depicted below, the Action Orchestrator is positioned as the glue to bind together the multiple technology domains into a unified workflow.

What’s Next?

Multi Domain Policy

As our customers continue to strive for end-to-end automation their orchestration workflows are now spanning across multiple technology domains. As these workflows evolve we need to consolidate and coordinate tasks using a common automation platform.

A major step in the “automate everywhere” strategy is to consolidate automation on a Multi-Domain Policy (MDP) platform. Conceptually this upcoming platform is targeted to unify the existing orchestration engines across domains with a consistent UI, catalog, united operations, common segmentation, consistent on-boarding, and delivered on-prem or cloud.

AI Ops

Logs, telemetry, and health monitoring are currently used to build reactive dashboards for visibility. With the advent of AI Ops, the trend is predictive and self healing operations. AI Ops platforms utilize big data, modern machine learning, and other advanced analytics technologies. This new technology, directly and indirectly, enhances IT operations functions with proactive, personal and dynamic insight. Cisco Intersight is a SaaS addition to the portfolio of domain orchestration engines, making actionable intelligence available from AI Ops in Hyperflex and server domains. AI Ops capabilities are road-mapped into many other orchestration engines as well.

Continue reading