Using “WhatsOp” for Network Change Management

I often find myself passionately talking with my customers about programmability and how it unlocks hidden potential to do anything we can think of – even make a cup of coffee (if you purchase an API enabled coffee machine that is :-).

Some get it right away. Others can’t think of ideas for what they would like to do differently, and that reminds me of Henry Ford’s famous quote:

The WhatsOp use case

I understood I needed to provide a few examples of the potential use-cases in order to make it real. So, my quest began. At some point I stumbled upon Gabi Zapodeanu’s DevNet Create 2018 workshop – that was a perfect example of the use-case I was looking for.

WhatsOp is a very “visible” use-case in terms of user experience. It leverages several different programmability tools available on Cisco’s platform — Embedded Event Manager, Guestshell, Python, REST APIs, Webex Teams.

How I demonstrate WhatsOp

Change management is a use-case my customer faces on a daily basis in their production networks. Many of them have a policy that defines what changes are allowed throughout the day and what changes should be done only during off-hours. For example, assigning an access port to a vlan does not require any approvals, while changes to the routing table do.

With WhatsOp, every change (or selected changes) will notify selected network administrators and ask for their approval. The network administrators will be able to approve the change, or roll-back the change – through their smartphone, thanks to Webex Teams.

Experience the potential of programmability

My intention in demonstrating WhatOp is to expose the potential of programmability. We could restrict the commands privileged users are allowed to enter using ISE as TACACS, we could leverage Prime Infrastructure to archive and compare configuration revisions. However, integrating the functionality together and making it accessible from any device (that can access Webex Teams) would be challenging. Getting the functionality to work even without connectivity to the central management systems would be very challenging.

Don’t take the WhatsOp use-case as-is. Think about modifying it to address a unique problem your organization might be facing right now – how cool would that be?

If you have an interesting use-case – let me know! I’m always looking for new challenges to solve. For now, let me encourage you to try WhatOp yourself to experience the potential of programmability. Check out these resources:

◉ Webex Teams SDK which makes life much easier and the code cleaner working with the Webex Teams API.

◉ pyATS, which is super powerful for manipulations, parsing, differences and verification of network devices. The only reason we’re NOT using pyATS for configuration differences, is because the program needs to run in Guestshell, and we wanted to make it as lean as possible.

◉ Check out the repo on GitHub.

◉ Watch my demonstration video

Breaking it down:

How did it work?

1. Detecting the change: every change we have on a Cisco device triggers a log message, such as:

Aug 16 2020 07:05:12.521 UTC: %SYS-5-CONFIG_I: Configured from console by obrigg on vty0 (10.56.56.143)

We use EEM (Embedded Event Manager) built-in IOS-XE/NX-OS to wait for this log entry and kick off the Python program.

!

event manager applet config_change

 event syslog pattern "SYS-5-CONFIG_I" maxrun 240

 action 0 cli command "enable"

 action 1 cli command "guestshell run python3 /bootflash/guest-

share/WhatsOp/config_change.py"

 action 2 cli command "exit"

 action 3 cli command "exit"

!

2. Running the Python program: We’re using Guestshell to run our program on the device itself. Guestshell is a virtualized Linux-based environment, designed to run custom Linux applications, including Python for automated control and management of Cisco devices. This container shell provides a secure environment, decoupled from the host device, in which users can install scripts or software packages and run them.

3. What’s happening in Python:

a. The program starts with gathering information from the device: The user has performed the latest change and The running configuration (a copy is saved on bootflash:).

def save_config():

    # save running configuration, use local time to create new config file

name

    run = cli('show run')

    output = run[run.find('!'):len(run)]

    timestr = time.strftime('%Y%m%d-%H%M%S')

    filename = '/bootflash/guest-share/CONFIG_FILES/' + timestr + '_shrun'

    f = open(filename, 'w')

    f.write(output)

    f.close

    f = open('/bootflash/guest-share/CONFIG_FILES/current_config_name', 'w')

    f.write(filename)

    f.close

    return filename

syslog_input = cli('show logging | in %SYS-5-CONFIG_I')

syslog_lines = syslog_input.split('\n')

lines_no = len(syslog_lines)-2

user_info = syslog_lines[lines_no]

user = user_info.split()[user_info.split().index('by')+1]

old_cfg_fn = '/bootflash/guest-share/CONFIG_FILES/base-config' 

new_cfg_fn = save_config()

f = open(old_cfg_fn)

old_cfg = f.readlines()

f.close

f = open(new_cfg_fn)

new_cfg = f.readlines()

f.close

b. Then the program compares the running config, with the last approved configuration (saved on bootflash: as well).

def compare_configs(cfg1, cfg2):

    # compare two config files

    d = difflib.unified_diff(cfg1, cfg2)

    diffstr = ''

    for line in d:

        if line.find('Current configuration') == -1:

            if line.find('Last configuration change') == -1:

                if (line.find('+++') == -1) and (line.find('---') == -1):

                    if (line.find('-!') == -1) and (line.find('+!') == -1):

                        if line.startswith('+'):

                            diffstr = diffstr + '\n' + line

                        elif line.startswith('-'):

                            diffstr = diffstr + '\n' + line

    return diffstr

c. In case there are no changes (perhaps the user changed a setting and changed it back right after) – nothing will happen. However, if the program finds a difference between the running configuration and the last approved configuration – things will become more interesting.

d. The program will use the Cisco Webex Teams API to create a new space and add the preconfigured network administrators in that space. The program will alert the space about the changes that were made, and ask the admins for an approval or decline.

if diff != '':

    # find the device hostname using RESTCONF

    device_name = cli('show run | inc hostname ').split()[1]

    print('Hostname: ' + device_name)

    # Initialize Webex Teams API

    api = WebexTeamsAPI(access_token=WEBEX_TEAMS_ACCESS_TOKEN, proxies=PROXY)

    bot = api.people.me()

    # Create a new space

    room = api.rooms.create(WEBEX_TEAMS_ROOM)

    # Add members to the space

    for WEBEX_TEAMS_MEMBER in WEBEX_TEAMS_MEMBERS:

        api.memberships.create(room.id,personEmail=WEBEX_TEAMS_MEMBER)

    # Send initial message

    api.messages.create(roomId=room.id, markdown=f"Hello <@all>,  \n# Configuration change detected!**  \nDevice hostname: **{device_name}**, detected the following changes made by user: **{user}**.")

    api.messages.create(roomId=room.id, text=diff)

    api.messages.create(roomId=room.id, markdown=f"Do you approve these changes?  \nMention me using **@{bot.nickName}** and enter '**y**' or '**n**'.")

    counter = 6  # wait for approval = 10 seconds * counter, in this case 10 sec x 6 = 60 seconds

    last_message = ""

    # start approval process

    while (last_message != "Bye Bye") and (last_message != f"{bot.nickName} n") and (last_message != f"{bot.nickName} y"):

        time.sleep(10)

        messages = api.messages.list(room.id, mentionedPeople=bot.id)

        for message in messages:

            last_message = message.text

            break

e. If the changes are approved – the running config will be marked as the last approved configuration. If the changes are not approved (declined or timed out) – the program will perform a roll-back to the last approved configuration.

        if last_message == f'{bot.nickName} n':

            cli('configure replace flash:/guest-share/CONFIG_FILES/base-config force')

            approval_result = '- - -  \n<@all>,  \nConfiguration changes **not approved**, Configuration rollback to baseline'

            print('Configuration roll back completed')

        elif last_message == f'{bot.nickName} y':

            print('Configuration change approved')

            # save new baseline running configuration

            output = cli('show run')

            filename = '/bootflash/guest-share/CONFIG_FILES/base-config'

            f = open(filename, "w")

            f.write(output)

            f.close

            print('Saved new baseline configuration')

            approval_result = '- - -  \n<@all>,  \nConfiguration changes **approved**, Saved new baseline configuration'

        else:

            print("No valid response")

            counter = counter -1

            api.messages.create(roomId=room.id, markdown=f'- - -  \n<@all>, I did not receive a valid responce. **{str(counter)} attempts left**.  \nDo you approve these changes?  \nMention me using **@{bot.nickName}** and enter "**y**" or "**n**".')

            if counter == 0:

                last_message = "Bye Bye"

                cli('configure replace flash:/guest-share/CONFIG_FILES/base-config force')

                approval_result = '<@all>,  \nApproval request **timeout**, Configuration rollback to baseline'

                print('Configuration roll back completed')

    api.messages.create(roomId=room.id, markdown=approval_result)

f. Once it’s done, the Webex Teams space will be deleted in order to prevent flooding the admins with spaces.

    api.messages.create(roomId=room.id, markdown='This room will be deleted in 30 seconds')

    time.sleep(30)

    api.rooms.delete(room.id)

Continue reading

Cisco’s Magic Glue Binds the Pieces of the OvRAN Puzzle

Open Virtualized Radio Access Network i.e. “O-vRAN” has been a buzzword in the industry for quite some time now. To date, many renowned mobile industry analysts and researchers have conducted surveys on O-vRAN and revealed about the strong momentum of Mobile Network Operators (MNO) towards Commercial Off the Shelf (COTS) hardware, Software Defined Networks (SDN), and Virtualization. More and more MNOs are adopting or planning to modernize their Radio Access Networks (RAN) with Open vRAN architecture.

The decision to choose O-vRAN path by MNOs is a no-brainer. An operator survey by Senza Fili in 2019 tells that the key drivers behind the Open vRAN adoption are a reduction in Total Cost of Ownership (TCO) due to the increased vendor competition and deployment flexibilities that open architecture offers. Also, by decomposing, disaggregating, and virtualizing RAN components, MNOs are transforming their networks to future proof 5G ready networks.

Cisco’s Journey in O-vRAN

Cisco’s Contribution to Open vRAN Standards

Cisco an early advocate of Open Virtualized RAN, had launched “Open vRAN Ecosystem” way back in February 2018 at MWC Barcelona. With this launch, Cisco started its Open vRAN journey with like-minded partners e.g. Altiostar, Mavenir, Phazr, Tech Mahindra to name a few. The focus was to create an ecosystem that is designed to accelerate the viability and adoption of Open vRAN solution. Since then Cisco is working on architecting and building solutions, demos and trials with MNOs and helping to define open interfaces and management.

Early 2018, xRAN Forum defined the use of IETF’s NETCONF / YANG standard for programmatically configuring and managing its lower layer split RAN architecture. Cisco took a lead role working with operators and ecosystem partners, in helping the xRAN Forum define the use of NETCONF/YANG by the 5G Radio Unit. This laid the foundation for cross-domain orchestration of the RAN with other domains that had already adopted NETCONF / YANG.

Cisco’s Value Proposition

Cisco has a significant role in the Open Virtualized RAN ecosystem in the areas of automation, virtualization, software integration, infrastructure and lead integrator of the solution components. Cisco becomes the technology glue to bring together these modules thereby arriving at an end to end solution.

◉ Choice of virtualization & NFV

◉ Automation & Orchestration

◉ Systems integration

◉ Industry leading infrastructure

◉ Multiple choice of fronthaul, midhaul & backhaul solutions

◉ Most widely deployed mobile core solutions

Industry Trials, Demonstrations and Pilots

The UK government, in March 2018 funded testbed for the prestigious project “5G RuralFirst” to help the country take a leading position in 5G. The challenge was to provide secure and reliable mobile connectivity to the rural area and support rural communities and industries like agriculture, broadcasting, and utilities. Cisco led this project and supported with its 5G Core Network and Cloud service while Parallel Wireless was the part of the ecosystem with its multi-technology Open RAN solution.

In September 2018 MWC Americas at LA, Cisco demonstrated industry-first 5G SA data call using Phazr’s virtualized RAN and Cisco’s Cloud Native 5G Core. The solution also included MEC and Synamedia. (Earlier Cisco’s Service Provider Video Software Solution). In the same event, Cisco showcased a pre-certified, pre-integrated, ready to deploy end to end open vRAN system. Tech Mahindra, Altiostar and Cisco led this solution and proved that the open multivendor vRAN solutions were possible.

Recently in June 2020, Cisco and Telenor joined hands to strengthen the focus on areas like 5G, Open vRAN, and distributed Cloud to support critical infrastructure transitions for Telcos. The companies are continuing the exploration of new 5G architectures and have started an Open vRAN trial at Telenor headquarters in Norway to further investigate using a virtualized, open infrastructure to improve cost efficiency for service rollouts. A new round of financing of the joint venture, Working Group 2 (WG2) was also announced. WG2 offers a carrier-grade, cloud-native mobile core solution that enables rapid service innovation MNOs, MVNOs, and enterprises. Using open APIs and cross-network interoperability, WG2 radically transforms the ability of mobile operators to innovate quickly.

Real-World Deployments and Recognitions

While we had many demonstrations and trials of the Open vRAN solution at prominent events across the globe, the most rewarding for Cisco was the commercial deployment for Rakuten Japan in February 2019. The first-ever network that is fully virtualized from RAN to Core, leveraging mobile edge computing with end to end automation for both network and services. Cisco played a very critical role in this exciting journey alongside Rakuten and was well appreciated by the MNO emphasizing the crucial role of common and distributed telco cloud platform used to host many virtualized applications including Altiostar’s virtualized RAN applications vCU and vDU. End to end automation solution with Zero Touch Provisioning (ZTP) enabled instantiation of vRAN to fully operationalize a cell site in a matter of minutes with no manual intervention, helping Rakuen to scale the network in the future at a very faster rate.

In November 2018, Cisco was recognized for its 5G leadership by winning the Fierce Innovations Award for “NextGen Deployment Wireless” for its fully virtualized deployment and E2E automation with Rakuten. This was the 3rd industry award for Cisco 5G software-defined network in 2019. The first two given to Rakuten were the Light Reading Leading Lights awards for “Most Innovative Telco Cloud Strategy by an operator” and “Most Innovative Automation Strategy by an Operator.”

The Open vRAN wave is certainly building now. More and more MNOs are thinking about getting rid of legacy architectures and embrace Open vRAN.

Bharti Airtel, India’s leading MNO decided to take this path and in April this year announced its Open vRAN deployment with Cisco and Altiostar. Bharti became the first Indian Mobile Network Operator to deploy vRAN based 4G Network that is open that is seamlessly upgradeable to 5G.

Etisalat, one of the world’s leading telecom groups, announced the successful launch of Open vRAN early this year to become the first operator to adopt the technology in the Middle East. The success of this deployment was possible with the collaboration of Etisalat, Cisco, Altiostar and NEC.

Similarly, for one of its kind projects in the aviation industry, Gogo, a supplier of broadband connectivity products and services for Aviation, last year announced Cisco, Airspan, and First RF as its partners to supply elements for its air-to-ground (ATG) 5G network for use on business aviation aircraft and commercial regional jets operating within the United States and Canada.

Collaboration with WWT and Altiostar

World Wide Technology (WWT) is a technology solution provider that provides innovative technology and supply chain solutions to large public and private organizations. WWT, from their Advanced Technology Center brings integration and deployment capabilities for operators. Recently, in April 2020, World Wide Technology (WWT) announced a collaboration with Cisco and Altiostar to develop an Open vRAN blueprint and validation lab. WWT will be taking a lead role to integrate, validate, and deliver multi-vendor solutions Thus, Service Providers will be able to see the Open vRAN solution in operation, get their hands on it, and test features that they require helping faster validation and deployment.

Source: cisco.com

Continue reading

Universal Release Criteria 2.0–A Disruptive Quality Management Framework

Quality cannot be an afterthought! It takes strategic planning for meticulous implementation. The more time we invest in proactive thinking, tooling, Shift Left approaches, quality governance, and a process-driven culture, the more money and time we can save across the life of the product. Cisco’s Universal Release Criteria (URC 2.0) represents one such disruptive quality management framework that has been developed and adopted. The following criteria will help define a comprehensive set of quality goals, metric standardization, and process governance across the full development lifecycle from product development requirements to releases. The IOS-XE product fully adopted, implemented, and proved the URC 2.0 development process very effective.

Having worked in the software industry for more than two decades, I have gained experience in all aspects of the software development life cycle, process, metrics, and measurement. As a programmer, I designed and developed complex systems. I led the software development strategy and process for CI/CD pipelines, modernized it, and automated it. Also, I led the quality journey for various types of software releases for businesses to grow to scale. Given my background, I feel qualified to share my thoughts on URC development processes that helped us transform traditional development processes to more modern, automated, and streamlined counterparts.

Limited vision in quality goals and unchecked negative behaviors directly impact product quality. Resistance to upgrade requests from customers, lack of scaled environments to test deployments, and delays in early adoption really plague projects. Short-sighted quality goals impact deployments in the following manner:

• There is excessive focus on backlog issues with inadequate focus and management of incoming defects.
• Critical defects are prioritized exclusively.
• Inadequate or negligible attention is paid to proactive defect prevention.

URC 2.0 is the brainchild of cross-functional quality specialists with representatives from operations, development, testing, quality, and supply-chain organizations. The main objective of this process innovation is to “address defects found in a release” that “will have to be fixed in the same release” by bringing “no technical debt” forward. Initially, this simple rule placed tremendous pressure on teams, but within a couple of years, everyone willingly embraced this cultural shift!  The results of URC 2.0 can be summarized as follows:

• Provides a failproof framework for release quality management.
• Outlines comprehensive release quality criteria for product development.
• Transforms the departmental culture within software and hardware teams.
• Prevents defects and reduces escape conditions during the product development lifecycle.
• Fosters trickle-down innovation, enhances development practices, and furthers tools automation.

Shift Left techniques enjoy the benefits that come from URC’s quality algorithm innovations and the processes such as the following:

• Manages the incoming defect process using escape analytics and Raleigh’s curve.
• Enhances the algorithm to help address incoming, backlog, and bug disposal trends.
• Prioritizes age-based bug fixes to address high-risk defects.
• Simplifies operational management of the bug backlog.
• Reduces last-minute code churn.
• Operationalizes escape reduction.
• Targets addressing of security vulnerabilities at the right level.

The URC 2.0 Framework

The URC framework is based on four basic tenets of quality management:

• Prevention
• Identification
• Evaluation
• Removal

A set of metrics is identified for each tenet to measure and drive quality. The objective of each phase and corresponding metrics are described in the diagram below.

URC 2.0 Execution Framework and Metrics

To measure the effectiveness of activities in each framework, use a set of metrics to achieve very specific objectives per category. From an execution viewpoint, all URC 2.0 metrics can be grouped into categories: defects, bug evaluation, late code churn, security, testing, and parity. This grouping with associated goals are published and evangelized throughout the release cycles. Executives are briefed to ask the right questions to prevent the release of inferior code quality to customers. This discipline raises the bar for teams, encourages them to work hard to meet goals, and helps them proactively address challenges. In this way, teams are able to confidently stand behind the quality of their release offerings. Here are some categories to help address code improvement goals:

URC 2.0 Framework Strategies for Success

Important URC Framework Criteria

To enhance tracking clarity, the URC 2.0 framework provides the following critical and grouped elements.

• Track Defects–Address incoming and legacy issues based on severity and resolve issues within the same
• Use Metrics–Focus on execution quality and pass rate. Adhere to strict goals and periodically measure
• Ensure Release Defect Parity–Ensure that all fixes from prior releases are incorporated into the current release.
• Security Vulnerabilities–Address all known issues before general release availability.
• Control Late Code Churn–Set aggressive mean time to repair goals for all internally and externally found

The URC framework defines some critical terms for all releases: URC window and URC backlog.

• URC Window–The time between the “URC Freeze” date of the prior release and the “URC Freeze” date of the current release is considered the window of opportunity to make a difference.
• URC Backlog–When implementing URC 2.0 for the first time, address defects that fall into these major categories:
• All defects that are applicable to the current release, irrespective of where or when they were found.
• All URC defect fixes that must be carried forward from the previous release to maintain functional parity amongst releases.
• Pick a start date from the previous window to begin your URC window for the current release. This will help serve as a reference point to address critical defects.
• URC Freeze–Select a date when you stop working on your URC backlog. Ideally, this date must be after the completion of feature test for the release. The frozen URC backlog must be addressed and brought down to zero within three weeks of the freeze. This date must coincide with the critical defect cut-off date for the release. No bug fixes, immaterial of their severity, should be added to the current release after this critical defect cut-off date.
• Final Code Validation–Conduct final testing and release readiness checks before the release is shipped to

Proof of URC 2.0 Success

Witness the “defect mountain Shift Left”. It illustrates the success of adopting the URC 2.0 framework. It also clearly demonstrates Cisco’s emphasis on software quality before general release and serves as the best representation of all the hard work.

URC 2.0 Benefits IOS-XE (EM)

The unique and rewarding digitization and release quality journey makes a significant difference for our developers, customers, and to our business. It ensures our ability to deliver features and functionality with speed and at scale. With the combination of our DevSecOps digital transformation (captured in my blog, “How do you gauge software quality before deployment?”) and the URC 2.0 Framework, we will be able to elevate the quality of our software offerings! Customers will enjoy receiving software that is error-free and impactful right off the bat!

Continue reading

3 ways Cisco DNA Center and ServiceNow integration makes IT more efficient

Today’s highly complex and dynamic networks create demands that often exceed the capacity of IT operations teams. Within Cisco IT, we are meeting these demands by creating integrations between Cisco DNA Center and ServiceNow.

We use Cisco DNA Center to control the Cisco campus and branch network, as well as to track upgrades and manage the operational states of all network elements, connections, and users.

We use ServiceNow as one of the IT service management platforms for providing helpdesk support to users and management capabilities to our IT service owners.

Customer Zero implements emerging technologies into Cisco’s IT production environments ahead of product launch. We are integrating these systems in multiple ways to make it easier to find the right information to solve problems, streamline tasks for network changes, and allow routine operational tasks to run autonomously in an end-to-end automated workflow. Furthermore, Customer Zero is providing an IT operator’s perspective as we develop integrated solutions, best practices, and accompanying value cases to drive accelerated adoption.

To develop these integrations, Cisco IT takes advantage of Cisco DNA Center platform API Bundles, Cisco DNA Center customizable app in the ServiceNow App store, and other ServiceNow offerings.

Integration #1: All the right information, accessible in one place

One of our first integrations synchronizes inventory information about network devices from Cisco DNA Center to the ServiceNow configuration management database (CMDB). This inventory sync benefits users of both systems. Cisco DNA Center provides up-to-date information on a device so when there’s an issue, an engineer can see it in the CMDB along with context information, such as who to contact about solving the problem.

In the future, the engineer working in the CMDB will be able to click on a link to manage that device in Cisco DNA Center without needing a separate login and subsequent searches for the device. This feature will help the engineer save time, especially when troubleshooting network issues.

Integration #2: Streamlining deployment of software images

Another integration we created supports automation for managing software image updates on our network devices. In the past, Cisco IT engineers have spent thousands of hours every quarter managing these routine updates. But when Cisco IT receives a high-priority security alert, the updates must be distributed and verified ASAP on thousands of affected devices.

With a manual process, this effort requires extensive time for engineers to manage the change activity and track its status on every device. And the network remains exposed to the threat until this process is completed.

In the coming months, we will automate much of the change-management process through the Cisco DNA Center Software Management Functionality and ServiceNow integration. For emergency changes, the engineer can create one change request that covers all devices, which dramatically simplifies approvals. Once the device has been upgraded, Cisco DNA Center updates the individual device record in our ServiceNow system. This automation allows us to maintain current information without needing to process identical change requests for individual devices.

We will also create an integration for “allow list” change requests, which cover routine update tasks that have a low risk of impact on the network state. For these requests, Cisco DNA Center automatically attaches all covered devices to the request and updates each device record in the CMDB. In the past, these changes weren’t always tracked at the device level because of the time and effort required for an engineer to update the device data.

Integration #3: Turning routine work into autonomous processes

Today we are also creating a foundation to automate more tasks between Cisco DNA Center and ServiceNow. For example, an event generated by Cisco DNA Assurance will start a workflow that opens a ServiceNow case, which in turn will send a workflow to Cisco DNA Center with detailed information on the problem and how to fix it. This automation will reduce the number of issues that require attention from a network engineer.

ServiceNow will also be able to request details from Cisco DNA Center when a user files a ticket for a new type of problem. In this case, Cisco DNA Center will provide network information and past events that enrich the ticket with useful information for faster, more efficient troubleshooting and problem resolution.

More integrations to come

We’re just at the beginning of our Customer Zero vision for integrating Cisco DNA Center and our ServiceNow platform. We’ll continue to develop integrations that make our IT operations more efficient. And we’ll continue to share what we’re learning along the way.

What types of IT operational activity are you looking to automate?

Continue reading

Introduction to Programmability – Part 2

Part 1 of this series defined and explained the terms Network Management, Automation, Orchestration, Data Modeling, Programmability, and APIs. It also introduced the Programmability Stack and explained how an application at the top layer of the stack, wishing to consume an API exposed by the device at the bottom of the stack, does that. The previous post covered data modeling in some detail due to the novelty of the concept for most network engineers. I’m sure that Part I, although quite lengthy, left you scratching your head. At least a little.

So, in this part of the series, I will try to clear some more of the ambiguity related to programmability. As discussed in the previous post, the API exposed by a device uses a specific protocol. For example, a device exposing a NETCONF API will use the NETCONF protocol. The same applies to RESTCONF, gRPC, or Native REST APIs. The choice of protocol also decides which data encoding to use, as well as the transport over which the application speaks with the device.

Where to start?

One of the problems with discussing programmability is where to start. If you start with a protocol, you will need to understand the encoding in order to decipher the contents of the protocol messages. But for you to appreciate the importance of encoding, you need to understand its application and use by the protocol. The chicken first, or the egg! Moreover, with respect to RESTful protocols, you will also need a pretty good understanding of the transport protocol, HTTP in this case, in order to put all the pieces together.

So in order to avoid unnecessary confusion, this part of the series will only cover NETCONF and XML. HTTP, REST, RESTCONF, and JSON will be covered in the next part. Finally, gRPC and GPB will be covered in one last part of this series.

Note: In this blog post we will make very good use of Cisco’s DevNet Sandboxes. In case you didn’t already know that, Cisco DevNet provides over 70 sandboxes that constitute devices in different technology areas, for you to experiment with during your studies. Some of those are always-on, available for immediate use, and others need a reservation. All the sandboxes can be found at: https://devnetsandbox.cisco.com/RM/Topology. For the purpose of this blog, the sandboxes that do not need a reservation will suffice. Any other excuses for not reading on?… I didn’t think so!

APIs: RPC vs REST

In the previous part of this series we looked at APIs and identified them as software running on a device. An API exposed by the device provides a particular function or service to other software that wish to consume this API. The internal workings of an API are usually hidden from the software that consumes it.

For example, Twitter exposes an API that a program can consume in order to tweet to an account automatically without human intervention. Similarly, Google exposes a Geolocation API that returns the location of a mobile device based on information about cell towers and WiFi nodes that the device detects and sends over to the API.

Similarly, an API exposed by, say, a router, is software running on the router that provides a number of functions that can be consumed by external software, such as a Python script.

APIs may be classified in a number of different ways. Several API types (and different classifications) exist today. For the purpose of this blog series, we will discuss two of the most commonly used types in the network programmability arena today: RPC-based APIs and RESTful APIs.

Remote Procedure Call (RPC)-based APIs

A Remote Procedure Call (RPC) is a programmatic method for a client to Call (execute) a Procedure (piece of code) on another device. Since the device requesting the execution of the procedure (the client) is different than the device actually executing that procedure (the server), it is labelled as Remote.

An RPC-based API opens a software channel on the server, exposing the API, to clients, wishing to consume that API, for those clients to request the remote execution of procedures on the server. Both NETCONF and gRPC are RPC-based protocols/APIs. This part of the series will cover NETCONF and describe its RPC-based operation.

Representational State Transfer (REST):

REST is a framework, specification or architectural style that was developed by Roy Fielding in his doctoral dissertations in 2000 on APIs. The REST framework specifies six constraints, five mandatory and one optional, on coding RESTful APIs. The REST framework requires that a RESTful API be:

◉ Client-Server based

◉ Stateless

◉ Cacheable

◉ Have a uniform interface

◉ Based on a layered system

◉ Utilize code-on-demand (Optional)

When an API is described as RESTful, then this API adheres to the constraints listed above.

To elaborate a little, a requirement such as “Stateless” mandates that the client send a request to the API exposed by the server. The server processes the request, sends back the response, and the transaction ends at this. The server does not maintain the state of this completed transaction. Of course, this is an over simplification of the process and a lot of corner cases exist. An API may also be fully RESTful, or just partially RESTful. It all depends on how much it adheres to the constrains listed here.

REST is an architectural style for programming APIs and uses HTTP as an application-layer protocol to implement this framework. Thus far, HTTP is the only protocol designed specifically to implement RESTful APIs. RETCONF is a RESTful protocol/API and will be the subject of an upcoming part of this series, along with HTTP.

Although gRPC is an RPC-based protocol/API, it still uses HTTP/2 at the transport layer (recall the programmability stack from Part I ?) You may find this a little confusing. While it is beyond the scope of this part of the series to describe the operation of gRPC and its encoding GBP, this will be covered in an upcoming part. Stay with me on this series, and I promise that you won’t regret it ! For the sake of accuracy, gRPC also supports JSON encoding.

NETCONF

In the year 2003 the IETF assembled the NETCONF working group to study the shortcomings of the network management protocols and practices that were in use then (such as SNMP), and to design a new protocol that would overcome those shortcomings. Their answer was the NETCONF protocol. The core NETCONF protocol is defined in RFC 6241 and the application of NETCONF to model-based programmability using YANG models is defined in RFC 6244. NETCONF over SSH is covered on its own in RFC 6242.

Figure 1 illustrates the lifecycle of a typical NETCONF session.

Figure 1 – The lifecycle of a typical NETCONF session

NETCONF is a client-server, session-based protocol. The client initiates a session to the server (the network device in this case) over a pre-configured TCP port (port 830 by default). The session is typically initiated using SSH, but it may use any other reliable transport protocol. Once established, the session remains so until it is torn down by either peer.

The requirement that the transport protocol be reliable means that only TCP-based protocols (such as SSH or TLS) are supported. UDP is not. The NETCONF RFC mandates that a NETCONF implementation support, at a minimum, NETCONF over SSH. The implementation may optionally support other transport protocols in addition to SSH.

The first thing that happens after a NETCONF session is up is an exchange of hello messages between the client and server (either peer may send their hello first). Hello messages provide information on which version of NETCONF is supported by each peer, as well as other device capabilities. Capabilities describe which components of NETCONF, as well as which data models, the device supports. Hello messages are exchanged only once per session, at the beginning of the session. Once hello messages are exchanged, the NETCONF session is in established state.

On an established NETCONF session, one or more remote procedure call messages (rpc for short) are sent by the client. Each of these rpc messages specify an operation for the server to carry out. The get-config operation, for example, is used to retrieve the configuration of the device and the edit-config operation is used to edit the configuration on the device.

The server executes the operation, as specified in the rpc message (or not) and responds with a remote procedure call reply (rpc-reply for short) back to the client. The rpc-reply message contents will depend on which operation was requested by the client, the parameters included in the message, and whether the operation execution was successful or not.

All NETCONF messages (hello, rpc and rpc-reply) must be a well-formed XML document encoded in UTF-8. However, the content of these messages will depend on the data model referenced by the message. You will see what this means shortly !

In a best-case scenario, the client gracefully terminates the session by sending an rpc message to the server explicitly requesting that the connection be closed, using a close-session operation. The server terminates the session and the transport connection is torn down. In a not-so-good scenario, the transport connection may be unexpectedly lost due to a transmission problem, and the server unilaterally kills the session.

The architectural components of NETCONF discussed thus far can be summarized by the 4-layer model in Figure 2. The 4 layers are Transport, Messages, Operations and Content.

Figure 2 – The NETCONF architectural 4-Layer model

Now roll up your sleeves and get ready. Open the command prompt on your Windows machine or the Terminal program on your Linux or MAC OS machine and SSH to Cisco’s always-on IOS-XE sandbox on port 10000 using the command:

[kabuelenain@server1 ~]$ ssh -p 10000 developer@ios-xe-mgmt-latest.cisco.com

When prompted for the password, enter C1sco12345. Once the SSH connection goes through, the router will spit out its hello message as you can see in Example 1.

[kabuelenain@server1 ~]$ ssh -p 10000 developer@ios-xe-mgmt-latest.cisco.com

developer@ios-xe-mgmt-latest.cisco.com's password:

<?xml version="1.0" encoding="UTF-8"?>

<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

    <capabilities>

        <capability>urn:ietf:params:netconf:base:1.0</capability>

        <capability>urn:ietf:params:netconf:base:1.1</capability>

        <capability>urn:ietf:params:netconf:capability:writable-running:1.0</capability>

        <capability>urn:ietf:params:netconf:capability:xpath:1.0</capability>

        <capability>urn:ietf:params:netconf:capability:validate:1.0</capability>

        <capability>urn:ietf:params:netconf:capability:validate:1.1</capability>

        <capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability>

        <capability>urn:ietf:params:netconf:capability:notification:1.0</capability>

        <capability>urn:ietf:params:netconf:capability:interleave:1.0</capability>

        <capability>urn:ietf:params:netconf:capability:with-defaults:1.0?basic-mode=explicit&amp;also-supported=report-all-tagged</capability>

        <capability>urn:ietf:params:netconf:capability:yang-library:1.0?revision=2016-06-21&amp;module-set-id=730825758336af65af9606c071685c05</capability>

        <capability>http://tail-f.com/ns/netconf/actions/1.0</capability>

        <capability>http://tail-f.com/ns/netconf/extensions</capability>

        <capability>http://cisco.com/ns/cisco-xe-ietf-ip-deviation?module=cisco-xe-ietf-ip-deviation&amp;revision=2016-08-10</capability>

        <capability>http://cisco.com/ns/cisco-xe-ietf-ipv4-unicast-routing-deviation?module=cisco-xe-ietf-ipv4-unicast-routing-deviation&amp;revision=2015-09-11</capability>

        <capability>http://cisco.com/ns/cisco-xe-ietf-ipv6-unicast-routing-deviation?module=cisco-xe-ietf-ipv6-unicast-routing-deviation&amp;revision=2015-09-11</capability>

        <capability>http://cisco.com/ns/cisco-xe-ietf-ospf-deviation?module=cisco-xe-ietf-ospf-deviation&amp;revision=2018-02-09</capability>

------ Output omitted for brevity ------

    </capabilities>

    <session-id>468</session-id>

</hello>]]>]]>

Example 1 – Hello message from the router (NETCONF server)

Before getting into XML, note that the hello message contains a list of capabilities. These capabilities list three things about the device sending the hello message:

◉ The version(s) of NETCONF supported by the device (1.0 or 1.1)

◉ The optional NETCONF capabilities supported by the device (such as rollback-on-error)

◉ The YANG data models supported by the device

To respond to the server hello, all you need to do is copy and paste the hello message in Example 2 into your terminal.

<?xml version="1.0" encoding="UTF-8"?>

<hello

    xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

    <capabilities>

        <capability>urn:ietf:params:netconf:base:1.0</capability>

    </capabilities>

</hello>]]>]]>

Example 2 – Hello message from the client (your machine) back to the server

We will break down these messages in a minute – hold your breath!

Example 3 shows an rpc message to retrieve the configuration of interface GigabitEthernet1.

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">

    <get-config>

        <source>

            <running />

        </source>

        <filter>

            <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">

                <interface>

                    <GigabitEthernet>

                        <name>1</name>

                    </GigabitEthernet>

                </interface>

            </native>

        </filter>

    </get-config>

</rpc>]]>]]>

Example 3 – rpc message to retrieve the configuration of interface GigabitEthernet1

When you copy and paste this message into your terminal (right after the hello), you will receive the rpc-reply message in Example 4.

<?xml version="1.0" encoding="UTF-8"?>

<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">

    <data>

        <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">

            <interface>

                <GigabitEthernet>

                    <name>1</name>

                    <description>MANAGEMENT INTERFACE - DON'T TOUCH ME</description>

                    <ip>

                        <address>

                            <primary>

                                <address>10.10.20.48</address>

                                <mask>255.255.255.0</mask>

                            </primary>

                        </address>

                        <nat

                            xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-nat">

                            <outside/>

                        </nat>

                    </ip>

                    <mop>

                        <enabled>false</enabled>

                        <sysid>false</sysid>

                    </mop>

                    <negotiation

                        xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-ethernet">

                        <auto>true</auto>

                    </negotiation>

                </GigabitEthernet>

            </interface>

        </native>

    </data>

</rpc-reply>]]>]]>

Example 4 – rpc-reply message containing the configuration of interface GigabitEthernet1

Note that that the rpc message in Example 3 contains the XML elements rpc and get-config (highlighted in the example). The first indicates the message type and the second is the operation.

The rpc-reply message in Example 4 contains the XML elements rpc-reply and data (highlighted in the example). Again, the first is the message type and the second is the element that will contain all the data retrieved in case the operation in the rpc message is get or get-config.

The above examples are intended to give you a taste of NETCONF. Now let’s get into XML so we can dissect and decipher the 3 types of NETCONF messages.

eXtensible Markup Language (XML) – an interlude

Markup is information that you include in a document in the form of annotations. This information is not part of the original document content and is included only to provide information describing the sections of the document. A packaging of sorts. This markup is done in XML using elements.

Elements in XML are sections of the document identified by start and end tags. Take for example the following element in Example 4:

<description>MANAGEMENT INTERFACE - DON'T TOUCH ME</description>

This element name is description and is identified by the start tag <description> and end tag </description>. Notice the front slash (/) at the beginning of the end tag identifying it as an end tag. The tags are the markup and the content or data is the text between the tags. Not to state the obvious, but the start and end tags must have identical names, including the case. Having different start and end tags defies the whole purpose of the tag.

Elements may be nested under other elements. As a matter of fact, one of the purposes of markup in general and XML in particular is to define hierarchy. Child elements nested under parent elements are included within the start and end tags of the parent element. The description element is included inside the start and end tags of the GigabitEthernet element, which in turn is included inside the tag pair of its parent element interface.

A start and end tag with nothing in-between is an empty element. So an empty description element would look like:

<description></description>

But an alternative, shorter, representation of an empty element uses a single tag with a slash at the end of the tag:

<description/>

The top-most element is called the document or root element. All other elements in the document are children to the root element. In the case of NETCONF messages, the root element is always one of three options: hello, rpc or rpc-reply.

You may have noticed the very first line above the root element:

<?xml version="1.0" encoding="UTF-8"?>

This line is called the XML declaration. Very simply put, it tells the program (parser) that will read the XML document what version of XML and encoding are used. In this case, we are using XML version 1.0 and UTF-8 encoding, which is the encoding mandated by the NETCONF RFC.

The final piece of the puzzle are the attributes. Notice the root element start tag in Examples 3 and 4:

Example 3: <rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">

Example 4: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">

The words xmlns and message-id are called attributes. Attributes are used to provide information related to the element in which they are defined.

In the case of the rpc and rpc-reply elements, the attribute xmlns defines the namespace in which this root element is defined. XML namespaces are like VLANs or VRFs: they define a logical space in which an element exists, more formally referred to in programming as the scope. The NETCONF standard mandates that the all NETCONF protocol elements be defined in the namespace urn:ietf:params:xml:ns:netconf:base:1.0. This is why you will find that the xmlns attribute is assigned this value in every single NETCONF message.

Sometimes the attribute is used for elements other than the root element. Take for example the native element in both Examples 3 and 4:

<native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">

The xmlns attribute, also referring to a namespace, takes the value of the YANG model referenced by this element and all child elements under it, in this case the YANG model named Cisco-IOS-XE-native.

The other attribute is the message-id. This is an arbitrary string that is sent in the rpc message and mirrored back in the rpc-reply message unchanged, so that the client can match each rpc-reply message to its corresponding rpc message. You will notice that in both Examples 3 and 4 the message-id is equal to 101.

An XML declaration along with a root element (along with all the child elements under the root element) comprise an XML document. When an XML document follows the rules discussed so far (and a few more), it is referred to as a well-formed XML document. Rules here refer to the simple syntax and semantics governing XML documents, such as:

◉ For every start tag there has to be a matching end tag

◉ Tags start with a left bracket (<) and end with a right bracket (>)

◉ End tags must start with a left bracket followed by a slash then the tag name. Alternatively, empty elements may have a single tag ending in a slash and right bracket

◉ Do not include reserved characters (<,>,&,”) as element data without escaping them

◉ Make sure nesting is done properly: when a child element is nested under a parent element, make sure to close the child element using its end tag before closing the parent element

All NETCONF messages must be well-formed XML documents.

I wish I could say that we just scratched the surface of XML, but we didn’t even get this far. XML is so extensive and has a phenomenal number of applications that I would need several pages to just list the number of books and publications written on XML. For now, the few pointers mentioned here will suffice for a very basic understanding of NETCONF.

NETCONF

Now that you have seen NETCONF in action and have an idea on what each component of the XML document means, let’s dig a little deeper into the rpc message.

The rpc message in Example 3 contains an rpc root element indicating the message type, followed by the get-config element, indicating the operation. NETCONF supports a number of operations that allow for the full lifecycle of device management, some of which are:

◉ Operations for retrieving state data and configuration: get, get-config

◉ Operations for changing configuration: edit-config, copy-config, delete-config

◉ Datastore operations: lock, unlock

◉ Session operations: close-session, terminate-session

◉ Candidate configuration operations: commit, discard-changes

The elements that will follow the operations element will depend on which operation you are calling. 

For example, you will almost always specify the source or destination datastore on which the operation is to take place.

Which brings us to a very important concept supported by NETCONF: datastores. NETCONF supports the idea of a device having multiple, separate, datastores, such as the running, startup and/or candidate configurations. Based on the capabilities announces by the device in the hello message, this device may or may not support a specific datastore. The only mandatory datastore to have on a device is the running-configuration datastore.

Capabilities not only advertise what datastores are supported by the device, but also whether some of these datastores (such as the startup configuration datastore) are directly writable, or the client needs to write to the candidate datastore, and then commit the changes so that the configuration changes are reflected to the running and/or the startup datastores. Engineers working on IOX-XR based routers will be familiar with this concept.

When working with a candidate datastore, the typical workflow will involve the client implementing the configuration changes on the candidate configuration first, and then either issuing a commit operation to copy the candidate configuration to the running-configuration, or a discard-changes operation to discard the changes.

And before working on a datastore, whether the candidate configuration or another, the client should use the lock operation before starting the changes and the unlock operation after the changes are completed (or discarded) since more than one session can have access to a datastore. Without locking the datastore for your changes, several sessions may apply changes simultaneously.

To actually change the configuration, the edit-config operation introduces changes to the configuration in a target datastore, using new configuration in the rpc message body, in addition to a (sub-)operation that specifies how to integrate this new configuration with the existing configuration in the datastore (merge, replace, create or delete). The copy-config operation is used to create or replace an entire configuration datastore. The delete-config operation is used to delete an entire datastore.

NETCONF also supports the segregation between configuration data and state data. The get operation will retrieve both types of data from the router, while the get-config operation will only retrieve the configuration on the device (in the datastore specified in the source element).

In order to limit the information retrieved from the device, whether state or configuration, NETCONF supports two types of filters: Subtree filters and XPath filters. The first type is the default and works exactly as you see in Example 3. You specify a filter element under the operation and include only the branches of the hierarchy in the referenced data model that you want to retrieve. XPath filters use XPath expressions for filtering. XPath filters are part of XML and existed before the advent of NETCONF.

NETCONF and Python

Up till this point we have been sending and receiving NETCONF messages “manually”, which is a necessary evil to observe and study the intricacies of the protocol. However, in a real-life scenario, copying and pasting a hello or rpc message into the terminal, and reading through the data in the rpc-reply message kinda defies the purpose. We are, after all, discussing network programmability for the ultimate purpose of automation ! And an API is a software-to-software interface and not really designed for human consumption. Right ?

So let’s discuss a very popular Python library that emulates a NETCONF client: ncclient. The ncclient library provides a good deal of abstraction by masking a lot of the details of NETCONF, so you, the programmer, would not have to deal directly with most of the protocol specifics. Ncclient supports all the functions of NETCONF covered in the older RFC 4741.

Assuming you are on a Linux machine, before installing the ncclient library, make sure to install the following list of dependencies (using yum if you are on a CentOS or RHEL box):

◉ setuptools 0.6+

◉ Paramiko 1.7+

◉ lxml 3.3.0+

◉ libxml2

◉ libxslt

◉ libxml2-dev

◉ libxslt1-dev

Then Download the Python Script setup.py from the GitHub repo https://github.com/ncclient/ncclient and run it:

[kabuelenain@localhost ~]$ sudo python setup.py install

Or just use pip:

[kabuelenain@localhost ~]$ sudo pip install ncclient

The ncclient library operates by defining a handler object called manager which represents the NETCONF server. The manager object has different methods defined to it, each performing a different protocol operation. Example 5 shows how to retrieve the configuration of interface GigabitEthernet1 using ncclient.

from ncclient import manager

     filter_loopback_Gig1='''

        <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">

            <interface>

                <GigabitEthernet>

                    <name>1</name>

                </GigabitEthernet>

            </interface>

        </native>

       '''

with manager.connect(host='ios-xe-mgmt-latest.cisco.com',

                     port=10000,

                     username='developer',

                     password='C1sco12345',

                     hostkey_verify=False

                     ) as m:

     rpc_reply = m.get_config(source="running",filter=("subtree",filter_loopback_Gig1))

     print(rpc_reply) print(rpc_reply)

Example 5 – An rpc message containing a <get-config> operation using ncclient to retrieve the running configuration of interface GigabitEthernet1

In the Python script in the example, the manager module is first imported from ncclient. A subtree filter is defined as a multiline string named filter_loopback_Gig1 to extract the configuration of interface GigabitEthernet1 from the router.

A connection to the router is then initiated using the manager.connect method. The parameters passed to the method in this particular example use values specific to the Cisco IOS-XE sandbox. The parameters are the host address (which may also be an ip address), the port configured for NETCONF access, the username and password and finally the hostkey_verify, which when set to False, the server SSH keys on the client are not verified.

Then the get_config method, using the defined subtree filter, and parameter source equal to running, retrieves the required configuration from the running configuration datastore.

Finally the rpc-reply message received from the router is assigned to string rpc_reply and printed out. The output resulting from running this Python program is identical to the output seen previously in Example 4.

The manager.connect and get_config methods have a few more parameters that may be used for more granular control of the functionality. Only the basic parameters are covered here.

Similarly, the edit_config method can be used to edit the configuration on the routers. In this next example, the edit_config method is used to change the ip address on interface GigabitEthernet1 to 10.0.0.1/24.

from ncclient import manager

     config_data='''

       <config>

         <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">

           <interface>

             <GigabitEthernet>

               <name>1</name>

               <ip>

                 <address>

                   <primary>

                     <address>10.0.0.1</address>

                     <mask>255.255.255.0</mask>

                   </primary>

                 </address>

               </ip>

             </GigabitEthernet >

           </interface>

         </native>

       </config>

       '''

with manager.connect(host='ios-xe-mgmt-latest.cisco.com',

                     port=10000,

                     username='developer',

                     password='C1sco12345',

                     hostkey_verify=False

                     ) as m:

     rpc_reply = m.edit_config(target="running",config=config_data)

     print(rpc_reply)

Example 6 – An rpc message containing an <edit-config> operation using ncclient to change the IP address on interface GigabitEthernet1

The difference between the get_config and edit_config methods is that the latter requires a config parameter instead of a filter, represented by the config_data string, and requires a target datastore instead of a source.

Example 7 shows the output after running the script in the previous example, which is basically an rpc-reply message with an ok element. The show run interface GigabitEthernet1 command output from the router shows the new interface configuration.

### Output from the NETCONF Session ###

<?xml version="1.0" encoding="UTF-8"?>

<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:7da14672-68c4-4d7e-9378-ad8c3957f6c1" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">

    <ok />

</rpc-reply>

### Output from the router via the CLI showing the new interface configuration ###

csr1000v-1#show run interface Gig1

Building configuration...

Current configuration : 99 bytes

!

interface GigabitEthernet1

 description Testing the ncclient library

 ip address 10.0.0.1 255.255.255.0

end

Example 7 – The rpc-reply message after running the program in Example 6 and the new interface configuration

NETCONF is much more involved than what has been briefly described in this post. I urge you to check out RFCs 6241, 6242, 6243 and 6244 and my book “Network Programmability and Automation Fundamentals” from Cisco Press for a more extensive discussion of the protocol.

Continue reading