Tune in: “Demystifying Cisco Orchestration for Infrastructure as Code”

Automating the software development life-cycle

DevOps teams are becoming more agile, reducing costs, and delivering a superb customer experience by automating the software development life-cycle. Cisco Orchestration solutions extend the benefits of automation to the entire stack. Each layer of the underlying infrastructure is delivered as Code (IaC). Orchestrators reduce the complexity of programmability, operational state, and visibility. In this session, we decode the differences between domain-specific workflow automation versus cross-domain orchestration. Achieving the goal of ‘Automate everything’ requires the right tool for the right use case. With use cases in mind, we will cover several Cisco orchestration solutions in their respective domains and cross domain capabilities. A brief demo will showcase Open Source and Cisco Orchestration tools working together hand-in-hand.

Level Set

What is Infrastructure as Code (IaC)?

IaC means writing imperative or declaration code to automate programmable infrastructure deployments and manage configurations. Imperative is how you do something step-by-step, as opposed to declarative which is ‘what to do’ by abstracting the configuration and state. DevOps best practices such as source control, verification, and visibility are building blocks to support infrastructure types (compute, network, storage, etc) as code.

Why do we need IaC?

With the advent of Continuous Integration and Continuous Delivery (CI/CD), we are able to build pipelines to automate the entire software development life-cycle (SDLC). Continuous integration (CI) is a set of tools to develop applications. Continuous delivery (CD) is the process of delivering updated software releases to infrastructure environments such as test, stage, and production. Using IaC for these platforms and environments is paramount to enabling software agility and rapid time to value. One could say, IaC is the easy button to building infrastructure to deliver software or other IT services.

Orchestration

What is orchestration? Wikipedia defines orchestration as an automated arrangement, coordination, and management that defines the policies and service levels through automated workflows, provisioning, and change management. In this same vein, a coffee grinder is automation where a brewing machine is orchestration.

Why do we need orchestration in addition to scripting? Production grade IaC at scale requires orchestration versus scripting to deliver advanced features such as intent, policy, governance, and Service Level Agreements. By building IaC to include configuration management, CICD, and other advanced orchestration features, similar benefits to application development are now possible in large scale technology domains (mulit-cloud, containers, campus, WAN, Data Center).

With network automation in mind, we see many pit falls with multi-threaded tasks running exclusively from scripts. Step (1) gather facts, Step (2) set conditions, Step (3) loop through items in jinja2 templates and parse to TextFSM and save the data to YAML files. Step (4) Push changes to devices and validate.

This level of scripted multi-threaded workflow is difficult to manage at scale. The main concerns are slow changes, configuration drift, lack of operation state, out of band config overwrites, and disruptive rollbacks. In spite of the current gaps, the Ansible engine is one of my favorite tools for pushing network configurations. One could argue that Tower provides a workflow for the playbooks to manage the order of these tasks but no configuration state is possible. In order to remediate some of these gaps, Ansible engine is adding a new ‘facts’ resource module in a future 2.9 release.

Caution

Sometimes we automate ourselves into a corner with too many scripts. What happens when major platform changes are made to the scripting tools? We’ve experienced this before with python 2.x to 3.x.and the impact to many of our product SDK libraries. We are seeing it again with the major change to the Ansible engine coming in 2.9 to introduce the ‘facts’ resource module. This change requires users to rewrite their playbooks from scratch to use these features. As a caution, consider limiting scripting to single threaded (CRUD ) actions while shifting the complexities of operational state and rollback to the domain specific orchestration engine.

Domain Orchestration

What is a domain orchestration? A domain orchestration engine focuses on delivering automation targeted to a single technology domain. For instance, the Cisco Network Services Orchestator (NSO) is focused on model driven “network” automation with Netconf and YANG. NSO converts CLI to YANG with network element drivers (NEDS) to supports a multitude of uses cases ranging from stand alone network devices, network services, and multiple controller domains (Meraki, Viptela, and ACI).

In a nut shell, NSO can deploy greenfield or snyc-from brownfield devices to build a transaction based configuration database state. Tools like Ansible engine have modules to integrate with NSO’s northbound JSON API to harness these differentiated capabilities for operations. In the following example, we are using Ansible playbooks with the Ansible NSO/Json module to make CRUD changes to NSO’s configuration database as a means to configure and operate tenants running on a N9K EVPN/VXLAN Data Center network fabrics versus CLI to the stand alone NXOS. The Ansible playbooks are then version controlled as YAML files in a git repository.

Top-level Orchestration

What is Top-level orchestration? A top-level orchestration engine is used to stitch together collaboration, notifications, governance, and source control for other lower level scripting tools and device APIs. Top-level orchestration supports use cases ranging from CICD pipelines for application development to automated infrastructure build and testing. The Cisco Action Orchestrator (AO) is a powerful Top-level orchestrator that enables automated workflows across technology domains and ITSM (ie., ServiceNow). Integrations to ITSM are key for customers who need low or no code catalogs and templates to simplify the delivery of IT services.

Internally, Cisco relies on ITSM and AO to automate the rapid delivery of CiscoLive and Devnet Sandboxes during our customer events. In the below example, the open source tools such as Gitlab work hand-in-hand with AO to create a workflow pipeline to automate the build and test for a tenant configuration across EVPN/VXLAN fabric and SDWAN network domains.

Confusion

Are you confused by CICD pipelines and their relationship with IaC? My ‘Ah hah’ moment was a realization that many of these DevOps methodologies are not mutually exclusive but highly complementary between AppDev and IaC. As operators, we can support the automated development life-cycle with CICD pipelines. This same knowledge and tools are adaptable to automated infrastructures. Operations can adopt the tools (open source or vendor) that make sense for updating, configuring and management of IaC in many domains.

If you look at AppDev the CICD pipeline for software development must CODE, BUILD, TEST, and DEPLOY the software to an environment that includes infrastructure (compute, network, and storage). Do we build these infrastructure environments ahead of time manually or automated on demand?

If the developer is not willing to patiently wait several weeks for the infrastructure environment to test their CODE, then fully automated IaC is the only answer! A second CICD pipeline managing the configuration, versioning, and alignment of the software build to the environment (test, stage, prod) version allows us to move much quicker and rebuild the environment later if needed.

AppDev CI/CD pipeline to IaC CI/CD pipeline

In the following example, we are using Gitlab to manage an application development CICD pipeline. Upon completion the AppDev pipeline triggers Action Orchestrator to build a second pipeline with workflows to automate the test environment to ultimately test the application stack. The idea is to test the software release in a test environment prior to pushing the same software into production. The Action Orchestrator (AO) has many adapters to make IaC very easy to build and test infrastructure technology domains.

Router/switch software upgrades are another use case for a network specific CICD pipeline. With CICD we can automate the upgrade of specific IOS software versions to devices in a version controlled and tested environment prior to production.

Controllers

Are controllers and orchestration one in the same? NOPE…  Controllers are a single API touch point and management system for Software Defined systems (SD-Access, SD-Wan, and SD-Networks) to manage the configuration state of the underlay and overlay and underlying protocols. Controllers are similar to orchestration by providing access to configuration snapshots and rollbacks, but unable to compose top-level workflows with other tools. In most cases, Controllers are bound to their single technology domain (campus, data center, WAN, or cloud). Often times, IaC is configured adequately with only scripting and source control in a single controller domain. Suffice it to say, when expanding from a single domain to cross domain controllers (ie SDWAN, and SDN) this cross domain integration introduces a catalyst for orchestration.

The Automation Challenge

There is a broad set of technology domains, each with many use cases for IaC. In order to succeed with IaC, we first need to address our automation challenges. From there we can target each specific use case mapped to the appropriate technology domain.

Challenges:

To many touchpoints: Need to consolidate and coordinate tasks using common automation tools.

Complexity: Need to abstract automation as much as possible to make resources consumable for the end users.

Operational Instrumentation: Need to automate and operationalize the tools into workflows that include visual dashboards, role-based access control, and other security services.

Verification: Need to make changes and check changes. With automation, we can move really fast and break things. Hence, we need the proverbial looking over our shoulder versus traditional stare and compare configuration checks. Ideally, verification should start in a test environment through production.

Community and Collaboration: Need to share finished code and avoid recreating the wheel with every workflow.

The key take away for automated solutions is to strive for a sharing culture, agility, simplicity, intent, security, and lower costs.

Technology Domains and Use Cases

The following table depicts the taxonomy of several Cisco orchestration options. As depicted below, the Action Orchestrator is positioned as the glue to bind together the multiple technology domains into a unified workflow.

What’s Next?

Multi Domain Policy

As our customers continue to strive for end-to-end automation their orchestration workflows are now spanning across multiple technology domains. As these workflows evolve we need to consolidate and coordinate tasks using a common automation platform.

A major step in the “automate everywhere” strategy is to consolidate automation on a Multi-Domain Policy (MDP) platform. Conceptually this upcoming platform is targeted to unify the existing orchestration engines across domains with a consistent UI, catalog, united operations, common segmentation, consistent on-boarding, and delivered on-prem or cloud.

AI Ops

Logs, telemetry, and health monitoring are currently used to build reactive dashboards for visibility. With the advent of AI Ops, the trend is predictive and self healing operations. AI Ops platforms utilize big data, modern machine learning, and other advanced analytics technologies. This new technology, directly and indirectly, enhances IT operations functions with proactive, personal and dynamic insight. Cisco Intersight is a SaaS addition to the portfolio of domain orchestration engines, making actionable intelligence available from AI Ops in Hyperflex and server domains. AI Ops capabilities are road-mapped into many other orchestration engines as well.

Continue reading

Automating Your Network Operations: Focus on Outcomes

When I started this blog series, I had intended to present a cohesive approach to automating your network operations, eventually leading to DevOps. The intent was to take you beyond simply automating ad-hoc tasks using contrived examples to a more systematic way of automating your network operations (hence the title). Unfortunately, I completely failed because I presented it in an ad-hoc way. So in this blog, I am going to go back to the beginning: What are we trying to achieve?

Before I do that, I want to introduce a new blogger, Jason King. Jason and I have a very similar background in operations and development. We’ve both spent a large part of our careers on the front lines designing, building, and operating large systems and networks and hope to leverage that experience in our blogs, repos, and other works. With his help, hopefully we can get the series moving more smoothly.

Automated Humans vs. Automated Business

Do we just want a human to be able to perform a single operation faster? If so, then we might not actually achieve that. Why is this? Automation is geared at deploying single or multiple changes across a large number of devices. However, many changes do not fall into this category.

Engineering vs. CRUD

There are basically two types of changes that are made to a network in steady state operations:

◈ Architectural/Engineering: These are changes to the architecture of the network (e.g. Routing, QoS, Multicast) that generally affect the entire network. It is also the architecture for how new services are deployed (e.g. tenants, remote sites, etc.)

◈ Create, Read, Update, Delete (CRUD): These are changes that deal with delivery of network services to a particular customer or application (e.g. putting a port in a VLAN, adding an ACE to an ACL, or adding a load balancing rule).

The rigor of DevOps is absolutely the way to make major architectural changes to a network because of the network-wide effect that these changes have and the relatively small number of changes that occur.

CRUD is often different, however. While making a single change (e.g. SNMP Community Strings) on thousands of devices is an operation for which the overhead of DevOps is justified, that same overhead may significantly slow down operations involving a single change on a single device (e.g. changing a port VLAN).

The DevOps overhead is not necessarily a bad thing, even for small changes. There are significant advantages in enforcing configuration management, revision control, code review, and testing on every change. It does not, however, always make network operations faster or easier. This increased friction can make it unpalatable to many network teams and hinder adoption.

Constraints-based IT

There is also the Theory of Constraints with states that “any improvements made anywhere besides the bottleneck are an illusion.” (Eliyahu M. Goldratt, 2014)

This means that if you are automating processes that are not slowing down your business, you are not having an effect on your business. The entire effort is potentially a waste of time and money. That is why automation should begin with a thoughtful process that identifies the most important outputs of your infrastructure and what the current bottlenecks are in producing those outputs.

In general, automation is going to provide your business two main improvements:

When your enterprise produces customer value faster (e.g. onboarding new customer and/or offering new services), the business generally brings in more revenue. Adding a faster time to remediation (e.g. less maintenance and quicker trouble resolution) reduces operating costs and increases customer satisfaction. When done right, it is a powerful combination proves the best outcomes for any automation project.

This is why we must focus on automating business processes and not just humans. In fact, the best way to automate a business is to remove the human from the process, at least from that value chain between the customer’s request and the delivery of that request.

“If it does not have an API, it does not exist”

– Mitchell Hashimoto

Automating Business Processes: API-Driven Automation

When we consider how to automate business processes, we must focus on reducing the time between the time a customer requests a new service and the time they receive that service. Humans, generally, are not the best way to reduce this time, which is where APIs come in. APIs allow each step of the process to be automated.

For example, when a user wants to add a firewall exception for a new server, they can go to a self service portal to make that request. That request can then go through the review process to make sure that it is aligned with business policies (hopefully automatically) and appropriately approved. Once it is approved, the ITSM pokes the automation framework through an API to begin the delivery of the service. The advantages of this approach is that:

1. It takes the network team out of the CRUD

2. It allows the network team to define and put checks around how changes to the network are performed

Hold the phone!

We are going to let a customer request a complex service with potentially harmful repercussions without having a human in the loop??? Well … yes. But even if you do need a human in the loop, you don’t need an entire team of them in the loop. And in either case, that is why DevOps is important. DevOps is not just automation, it is the development, testing, deployment, and validation of the artifacts that provide the service. If you properly develop and test these artifacts, you can be reasonably assured that the process does not go pear shaped. If you do the proper validation of deployed services, you can “fail fast” back to a previous version of the artifacts.

DevOps is the safety harness for automation, but that does not mean that you cannot start automating without it. In fact, very few organizations are able to implement a righteous DevOps pipeline in one go. The DevOps journey should be viewed as a stepwise approach to delivering business value. In this blog, we have provided some criteria for how to determine what to automate in order to achieve maximum value. You should not automate just to automate. Start with automation that addresses your critical business needs that fall into the categories outlined above, but always keep an eye on the end goal of building toward DevOps processes and, ultimately, business transformation.

What’s next?

Well, instead of writing blogs over the past 6 months, we’ve been working with a team of people to create a CI/CD pipeline for SD-WAN. Not a set of examples of what you could do, but a fully functional, operationally righteous framework that we and our customers use in their operations. To get there, we wrote Ansible modules for Viptela, VIRL, NFVIS, and PyATS that automate every step of the process. We’ll cover this in our next blog, followed by an in-depth treatment of each component and how to consume it individually on your stepwise path to DevOps.

Continue reading

Continuous Integration and Deployment for the Network

The requisite for Continuous Integration and Deployment (CI/CD) pipeline being used in the network is growing. Continuous Integration and deployment helps counteract inaccuracies in daily network deployments and changes, hence is critically required. The upshot for change release and automated network testing is that changes are simplified, done more quickly and more streamlined, with CI/CD.

Continue reading