Tuesday, 6 October 2020

Using “WhatsOp” for Network Change Management

Cisco Prep, Cisco Learning, Cisco Tutorial and Material, Cisco Guides, Cisco Certification, Cisco Exam Prep

I often find myself passionately talking with my customers about programmability and how it unlocks hidden potential to do anything we can think of – even make a cup of coffee (if you purchase an API enabled coffee machine that is :-).

Some get it right away. Others can’t think of ideas for what they would like to do differently, and that reminds me of Henry Ford’s famous quote:

Cisco Prep, Cisco Learning, Cisco Tutorial and Material, Cisco Guides, Cisco Certification, Cisco Exam Prep

The WhatsOp use case


I understood I needed to provide a few examples of the potential use-cases in order to make it real. So, my quest began. At some point I stumbled upon Gabi Zapodeanu’s DevNet Create 2018 workshop – that was a perfect example of the use-case I was looking for.

WhatsOp is a very “visible” use-case in terms of user experience. It leverages several different programmability tools available on Cisco’s platform — Embedded Event Manager, Guestshell, Python, REST APIs, Webex Teams.

Cisco Prep, Cisco Learning, Cisco Tutorial and Material, Cisco Guides, Cisco Certification, Cisco Exam Prep

How I demonstrate WhatsOp


Change management is a use-case my customer faces on a daily basis in their production networks. Many of them have a policy that defines what changes are allowed throughout the day and what changes should be done only during off-hours. For example, assigning an access port to a vlan does not require any approvals, while changes to the routing table do.

With WhatsOp, every change (or selected changes) will notify selected network administrators and ask for their approval. The network administrators will be able to approve the change, or roll-back the change – through their smartphone, thanks to Webex Teams.

Experience the potential of programmability


My intention in demonstrating WhatOp is to expose the potential of programmability. We could restrict the commands privileged users are allowed to enter using ISE as TACACS, we could leverage Prime Infrastructure to archive and compare configuration revisions. However, integrating the functionality together and making it accessible from any device (that can access Webex Teams) would be challenging. Getting the functionality to work even without connectivity to the central management systems would be very challenging.

Don’t take the WhatsOp use-case as-is. Think about modifying it to address a unique problem your organization might be facing right now – how cool would that be?

If you have an interesting use-case – let me know! I’m always looking for new challenges to solve. For now, let me encourage you to try WhatOp yourself to experience the potential of programmability. Check out these resources:

◉ Webex Teams SDK which makes life much easier and the code cleaner working with the Webex Teams API.

◉ pyATS, which is super powerful for manipulations, parsing, differences and verification of network devices. The only reason we’re NOT using pyATS for configuration differences, is because the program needs to run in Guestshell, and we wanted to make it as lean as possible.

◉ Check out the repo on GitHub.

◉ Watch my demonstration video


Breaking it down:


How did it work?

1. Detecting the change: every change we have on a Cisco device triggers a log message, such as:
Aug 16 2020 07:05:12.521 UTC: %SYS-5-CONFIG_I: Configured from console by obrigg on vty0 (10.56.56.143)

We use EEM (Embedded Event Manager) built-in IOS-XE/NX-OS to wait for this log entry and kick off the Python program.

!
event manager applet config_change
 event syslog pattern "SYS-5-CONFIG_I" maxrun 240
 action 0 cli command "enable"
 action 1 cli command "guestshell run python3 /bootflash/guest-

share/WhatsOp/config_change.py"
 action 2 cli command "exit"
 action 3 cli command "exit"
!

2. Running the Python program: We’re using Guestshell to run our program on the device itself. Guestshell is a virtualized Linux-based environment, designed to run custom Linux applications, including Python for automated control and management of Cisco devices. This container shell provides a secure environment, decoupled from the host device, in which users can install scripts or software packages and run them.

3. What’s happening in Python:

a. The program starts with gathering information from the device: The user has performed the latest change and The running configuration (a copy is saved on bootflash:).

def save_config():
    # save running configuration, use local time to create new config file
name
    run = cli('show run')
    output = run[run.find('!'):len(run)]
    timestr = time.strftime('%Y%m%d-%H%M%S')
    filename = '/bootflash/guest-share/CONFIG_FILES/' + timestr + '_shrun'
    f = open(filename, 'w')
    f.write(output)
    f.close
    f = open('/bootflash/guest-share/CONFIG_FILES/current_config_name', 'w')
    f.write(filename)
    f.close
    return filename

syslog_input = cli('show logging | in %SYS-5-CONFIG_I')
syslog_lines = syslog_input.split('\n')
lines_no = len(syslog_lines)-2
user_info = syslog_lines[lines_no]
user = user_info.split()[user_info.split().index('by')+1]

old_cfg_fn = '/bootflash/guest-share/CONFIG_FILES/base-config' 
new_cfg_fn = save_config()

f = open(old_cfg_fn)
old_cfg = f.readlines()
f.close

f = open(new_cfg_fn)
new_cfg = f.readlines()
f.close

b. Then the program compares the running config, with the last approved configuration (saved on bootflash: as well).

def compare_configs(cfg1, cfg2):
    # compare two config files
    d = difflib.unified_diff(cfg1, cfg2)
    diffstr = ''
    for line in d:
        if line.find('Current configuration') == -1:
            if line.find('Last configuration change') == -1:
                if (line.find('+++') == -1) and (line.find('---') == -1):
                    if (line.find('-!') == -1) and (line.find('+!') == -1):
                        if line.startswith('+'):
                            diffstr = diffstr + '\n' + line
                        elif line.startswith('-'):
                            diffstr = diffstr + '\n' + line
    return diffstr

c. In case there are no changes (perhaps the user changed a setting and changed it back right after) – nothing will happen. However, if the program finds a difference between the running configuration and the last approved configuration – things will become more interesting.

d. The program will use the Cisco Webex Teams API to create a new space and add the preconfigured network administrators in that space. The program will alert the space about the changes that were made, and ask the admins for an approval or decline.

if diff != '':
    # find the device hostname using RESTCONF
    device_name = cli('show run | inc hostname ').split()[1]
    print('Hostname: ' + device_name)
    # Initialize Webex Teams API
    api = WebexTeamsAPI(access_token=WEBEX_TEAMS_ACCESS_TOKEN, proxies=PROXY)
    bot = api.people.me()
    # Create a new space
    room = api.rooms.create(WEBEX_TEAMS_ROOM)
    # Add members to the space
    for WEBEX_TEAMS_MEMBER in WEBEX_TEAMS_MEMBERS:
        api.memberships.create(room.id,personEmail=WEBEX_TEAMS_MEMBER)
    # Send initial message
    api.messages.create(roomId=room.id, markdown=f"Hello <@all>,  \n# Configuration change detected!**  \nDevice hostname: **{device_name}**, detected the following changes made by user: **{user}**.")
    api.messages.create(roomId=room.id, text=diff)
    api.messages.create(roomId=room.id, markdown=f"Do you approve these changes?  \nMention me using **@{bot.nickName}** and enter '**y**' or '**n**'.")
    counter = 6  # wait for approval = 10 seconds * counter, in this case 10 sec x 6 = 60 seconds
    last_message = ""
    # start approval process
    while (last_message != "Bye Bye") and (last_message != f"{bot.nickName} n") and (last_message != f"{bot.nickName} y"):
        time.sleep(10)
        messages = api.messages.list(room.id, mentionedPeople=bot.id)
        for message in messages:
            last_message = message.text
            break

e. If the changes are approved – the running config will be marked as the last approved configuration. If the changes are not approved (declined or timed out) – the program will perform a roll-back to the last approved configuration.

        if last_message == f'{bot.nickName} n':
            cli('configure replace flash:/guest-share/CONFIG_FILES/base-config force')
            approval_result = '- - -  \n<@all>,  \nConfiguration changes **not approved**, Configuration rollback to baseline'
            print('Configuration roll back completed')

        elif last_message == f'{bot.nickName} y':
            print('Configuration change approved')
            # save new baseline running configuration
            output = cli('show run')
            filename = '/bootflash/guest-share/CONFIG_FILES/base-config'
            f = open(filename, "w")
            f.write(output)
            f.close
            print('Saved new baseline configuration')
            approval_result = '- - -  \n<@all>,  \nConfiguration changes **approved**, Saved new baseline configuration'

        else:
            print("No valid response")
            counter = counter -1
            api.messages.create(roomId=room.id, markdown=f'- - -  \n<@all>, I did not receive a valid responce. **{str(counter)} attempts left**.  \nDo you approve these changes?  \nMention me using **@{bot.nickName}** and enter "**y**" or "**n**".')
            if counter == 0:
                last_message = "Bye Bye"
                cli('configure replace flash:/guest-share/CONFIG_FILES/base-config force')
                approval_result = '<@all>,  \nApproval request **timeout**, Configuration rollback to baseline'
                print('Configuration roll back completed')

    api.messages.create(roomId=room.id, markdown=approval_result)

f. Once it’s done, the Webex Teams space will be deleted in order to prevent flooding the admins with spaces.

    api.messages.create(roomId=room.id, markdown='This room will be deleted in 30 seconds')
    time.sleep(30)
    api.rooms.delete(room.id)

Related Posts

0 comments:

Post a Comment