Change management is a use-case my customer faces on a daily basis in their production networks. Many of them have a policy that defines what changes are allowed throughout the day and what changes should be done only during off-hours. For example, assigning an access port to a vlan does not require any approvals, while changes to the routing table do.
With WhatsOp, every change (or selected changes) will notify selected network administrators and ask for their approval. The network administrators will be able to approve the change, or roll-back the change – through their smartphone, thanks to Webex Teams.
My intention in demonstrating WhatOp is to expose the potential of programmability. We could restrict the commands privileged users are allowed to enter using ISE as TACACS, we could leverage Prime Infrastructure to archive and compare configuration revisions. However, integrating the functionality together and making it accessible from any device (that can access Webex Teams) would be challenging. Getting the functionality to work even without connectivity to the central management systems would be very challenging.
Don’t take the WhatsOp use-case as-is. Think about modifying it to address a unique problem your organization might be facing right now – how cool would that be?
If you have an interesting use-case – let me know! I’m always looking for new challenges to solve. For now, let me encourage you to try WhatOp yourself to experience the potential of programmability. Check out these resources:
◉ Webex Teams SDK which makes life much easier and the code cleaner working with the Webex Teams API.
◉ pyATS, which is super powerful for manipulations, parsing, differences and verification of network devices. The only reason we’re NOT using pyATS for configuration differences, is because the program needs to run in Guestshell, and we wanted to make it as lean as possible.
Breaking it down:
How did it work?
1. Detecting the change: every change we have on a Cisco device triggers a log message, such as:
Aug 16 2020 07:05:12.521 UTC: %SYS-5-CONFIG_I: Configured from console by obrigg on vty0 (10.56.56.143)
We use EEM (Embedded Event Manager) built-in IOS-XE/NX-OS to wait for this log entry and kick off the Python program.
!
event manager applet config_change
event syslog pattern "SYS-5-CONFIG_I" maxrun 240
action 0 cli command "enable"
action 1 cli command "guestshell run python3 /bootflash/guest-
share/WhatsOp/config_change.py"
action 2 cli command "exit"
action 3 cli command "exit"
!
2. Running the Python program: We’re using Guestshell to run our program on the device itself. Guestshell is a virtualized Linux-based environment, designed to run custom Linux applications, including Python for automated control and management of Cisco devices. This container shell provides a secure environment, decoupled from the host device, in which users can install scripts or software packages and run them.
3. What’s happening in Python:
a. The program starts with gathering information from the device: The user has performed the latest change and The running configuration (a copy is saved on bootflash:).
def save_config():
# save running configuration, use local time to create new config file
name
run = cli('show run')
output = run[run.find('!'):len(run)]
timestr = time.strftime('%Y%m%d-%H%M%S')
filename = '/bootflash/guest-share/CONFIG_FILES/' + timestr + '_shrun'
f = open(filename, 'w')
f.write(output)
f.close
f = open('/bootflash/guest-share/CONFIG_FILES/current_config_name', 'w')
f.write(filename)
f.close
return filename
syslog_input = cli('show logging | in %SYS-5-CONFIG_I')
syslog_lines = syslog_input.split('\n')
lines_no = len(syslog_lines)-2
user_info = syslog_lines[lines_no]
user = user_info.split()[user_info.split().index('by')+1]
old_cfg_fn = '/bootflash/guest-share/CONFIG_FILES/base-config'
new_cfg_fn = save_config()
f = open(old_cfg_fn)
old_cfg = f.readlines()
f.close
f = open(new_cfg_fn)
new_cfg = f.readlines()
f.close
b. Then the program compares the running config, with the last approved configuration (saved on bootflash: as well).
def compare_configs(cfg1, cfg2):
# compare two config files
d = difflib.unified_diff(cfg1, cfg2)
diffstr = ''
for line in d:
if line.find('Current configuration') == -1:
if line.find('Last configuration change') == -1:
if (line.find('+++') == -1) and (line.find('---') == -1):
if (line.find('-!') == -1) and (line.find('+!') == -1):
if line.startswith('+'):
diffstr = diffstr + '\n' + line
elif line.startswith('-'):
diffstr = diffstr + '\n' + line
return diffstr
c. In case there are no changes (perhaps the user changed a setting and changed it back right after) – nothing will happen. However, if the program finds a difference between the running configuration and the last approved configuration – things will become more interesting.
d. The program will use the Cisco Webex Teams API to create a new space and add the preconfigured network administrators in that space. The program will alert the space about the changes that were made, and ask the admins for an approval or decline.
if diff != '':
# find the device hostname using RESTCONF
device_name = cli('show run | inc hostname ').split()[1]
print('Hostname: ' + device_name)
# Initialize Webex Teams API
api = WebexTeamsAPI(access_token=WEBEX_TEAMS_ACCESS_TOKEN, proxies=PROXY)
bot = api.people.me()
# Create a new space
room = api.rooms.create(WEBEX_TEAMS_ROOM)
# Add members to the space
for WEBEX_TEAMS_MEMBER in WEBEX_TEAMS_MEMBERS:
api.memberships.create(room.id,personEmail=WEBEX_TEAMS_MEMBER)
# Send initial message
api.messages.create(roomId=room.id, markdown=f"Hello <@all>, \n# Configuration change detected!** \nDevice hostname: **{device_name}**, detected the following changes made by user: **{user}**.")
api.messages.create(roomId=room.id, text=diff)
api.messages.create(roomId=room.id, markdown=f"Do you approve these changes? \nMention me using **@{bot.nickName}** and enter '**y**' or '**n**'.")
counter = 6 # wait for approval = 10 seconds * counter, in this case 10 sec x 6 = 60 seconds
last_message = ""
# start approval process
while (last_message != "Bye Bye") and (last_message != f"{bot.nickName} n") and (last_message != f"{bot.nickName} y"):
time.sleep(10)
messages = api.messages.list(room.id, mentionedPeople=bot.id)
for message in messages:
last_message = message.text
break
e. If the changes are approved – the running config will be marked as the last approved configuration. If the changes are not approved (declined or timed out) – the program will perform a roll-back to the last approved configuration.
if last_message == f'{bot.nickName} n':
cli('configure replace flash:/guest-share/CONFIG_FILES/base-config force')
approval_result = '- - - \n<@all>, \nConfiguration changes **not approved**, Configuration rollback to baseline'
print('Configuration roll back completed')
elif last_message == f'{bot.nickName} y':
print('Configuration change approved')
# save new baseline running configuration
output = cli('show run')
filename = '/bootflash/guest-share/CONFIG_FILES/base-config'
f = open(filename, "w")
f.write(output)
f.close
print('Saved new baseline configuration')
approval_result = '- - - \n<@all>, \nConfiguration changes **approved**, Saved new baseline configuration'
else:
print("No valid response")
counter = counter -1
api.messages.create(roomId=room.id, markdown=f'- - - \n<@all>, I did not receive a valid responce. **{str(counter)} attempts left**. \nDo you approve these changes? \nMention me using **@{bot.nickName}** and enter "**y**" or "**n**".')
if counter == 0:
last_message = "Bye Bye"
cli('configure replace flash:/guest-share/CONFIG_FILES/base-config force')
approval_result = '<@all>, \nApproval request **timeout**, Configuration rollback to baseline'
print('Configuration roll back completed')
api.messages.create(roomId=room.id, markdown=approval_result)
f. Once it’s done, the Webex Teams space will be deleted in order to prevent flooding the admins with spaces.
api.messages.create(roomId=room.id, markdown='This room will be deleted in 30 seconds')
time.sleep(30)
api.rooms.delete(room.id)
Posts
