Configuration Compliance in DCNM 11

We discussed Using DCNM 11 for Easy Provisioning of Networks and VRF’s. Today, we are continuing the discussion by featuring how DCNM empowers compliance of the configurations defined by a user.

Validation of configuration forms an integral part of any Network Controller. Configurations need to be pushed down from the controller to the respective switches as intended by the user. More importantly, configurations need to be in sync and in compliance with the expressed intent at all times. Any deviation from the intended configuration has to be recognized, reported, and remediated – this approach is often described as “closed loop.” In the DCNM LAN Fabric install mode, Configuration Compliance is supported for VXLAN EVPN networks (within Easy Fabrics) as well as traditionally built networks within an External Fabric.

Configuration Compliance is embedded and integrated within the DCNM Fabric builder for all configuration including underlay, overlay, interfaces and every other configuration that is driven through the DCNM policies.

The user typically builds intent for the fabric customizing the various fabric setting options as well a combination of best practice and custom templates. Once the intent is saved and pushed out by DCNM, it periodically monitors what is running in the switches and tracks if there was any Out-of-Band change made in any function of the switch using CLI or another method. If changes are made differing from the applied intent, DCNM will mark the switches as Out-of-Sync indicating a violation in compliance. This warning to the user provides information about the running configuration of the respective switch does not match the intent defined in DCNM. The Out-of-Sync state is indicated by a colour code in the topology view as well as tagged with Out-of-Sync in the tabular view which lists all the switches in a fabric.

Configuration Compliance status with color codes

While the general concept of Configuration Compliance provides a simple colored representation of the state across the nodes, DCNM also generates a side-by-side diff view of the running configuration and expected configuration for each switch.

This diff in configuration is intended to provide the user a full picture of why a particular switch was marked out of compliance aka OUT-OF-SYNC. While at it, Configuration Compliance function provides a set of pending configurations that once pushed to the switch using DCNM, will bring the switch back to compliance aka IN-SYNC. The set of pending configurations are intelligently derived using a model-based approach that is agnostic to commands configured using CLI.

Side-by-side diff generated on Out-of-SYNC

While Configuration Compliance runs periodically, DCNM also provides an on-demand option to “Re-sync” the entire fabric or individual switches to immediately trigger compliance check.

View the demo below to see a walk through of performing configuration compliance in DCNM 11

Continue reading

Using the Cisco DNA Center SDK CLI Tool

API flexibility, with Linux shell CLI convenience

Why CLI, I hear you ask?  CLI tools are really flexible and powerful. The whole concept of the Linux shell is based around a powerful set of tools that can be linked together to perform a task. Rather than needing to build a new tool from scratch (writing code), I can solve my problem linking smaller existing tools together.

The DNA Center CLI tools provide all the flexibility of the API with the convenience of  Linux shell tools.

Installing

Both the  Cisco DNA Center SDK and CLI are available via PyPI, so all that is required is “pip install”

I would recommend using a virtual environment. This is optional, but means you do not require root access and helps keep different versions of Python libraries separate.   Once created, it needs to be activated, using the “source” command.

If you logout and back in, activation needs to be repeated.

python3 -m venv env3
source env3/bin/activate

To install, you just need to install the cli as dnacentersdk is a dependency.

pip install dnacentercli

You are now able to use the CLI tool.

Getting Started

If you just run the cli tool without any arguments, you will get a help message.  I have truncated for brevity

$ dnacentercli
Usage: dnacentercli [OPTIONS] COMMAND [ARGS]...

  DNA Center API wrapper.

  DNACenterAPI wraps all of the individual DNA Center APIs and represents
  them in a simple hierarchical structure.

Options:
  --help  Show this message and exit.
<SNIP>

You will need to provide credentials and potentially the URL for your DNA Center. In this example I am going to use environment variables to simplify the username and password. I am also going to use the default DNAC, sandboxdnac2.cisco.com. You can change this to use your own DNA Center.

export DNA_CENTER_USERNAME='devnetuser'
export DNA_CENTER_PASSWORD='Cisco123!'
export DNA_CENTER_BASE_URL='https://sandboxdnac2.cisco.com'

# optional needs to be False if self signed certificate
export DNA_CENTER_VERIFY="True"

# optional.  This is the default
export DNA_CENTER_VERSION="1.3.0"

To get a count of the number of network devices, I could use the following:

$ dnacentercli devices get-device-count
{"response": 14,"version": "1.0"}

The structure of the CLI follows the DNA Center SDK. To find out the valid suboptions for devices, simply use –help or run without and suboption. (I have truncated for brevity).

$ dnacentercli devices
Usage: dnacentercli  devices [OPTIONS] COMMAND [ARGS]...

  DNA Center Devices API (version: 1.3.0).

  Wraps the DNA Center Devices API and exposes the API as native Python
  commands.

Options:
  --help  Show this message and exit.

Commands:
  add-device                      Adds the device with given...
  delete-device-by-id             Deletes the network device for...
  export-device-list              Exports the selected network...
  get-all-interfaces              Returns all available...
  get-device-by-id                Returns the network device...
  get-device-by-serial-number     Returns the network device with...
  get-device-config-by-id         Returns the device config by...
  get-device-config-count         Returns the count of device...
  get-device-config-for-all-devices
                                  Returns the config for all...
 <SNIP>

You can also format the JSON output with the -pp option. It requires an argument to indicate the level of indentation. For example (truncated for brevity):

$ dnacentercli devices get-device-list -pp 2
{
  "response": [
    {
      "apManagerInterfaceIp": "",
      "associatedWlcIp": "",
      "bootDateTime": "2019-01-19 02:33:05",
      "collectionInterval": "Global Default",

 <SNIP>

Advanced Use Cases

One common requirement is to get a csv file of the devices in the inventory. The JSON output can be processed with the jq command. jq is a really useful utility that can parse JSON fields. For example, getting a csv file of the DNAC inventory with specific fields.   In the example below, jq is looking at each of the list entries in the response, and extracting the hostname, managementIpAddress, softwareVersion, and id fields.  It then formats them as a csv.  The “-r” option for jq, provides raw output.

$ dnacentercli v1-3-0 devices get-device-list | jq -r '.response[] | [.hostname, .managementIpAddress, .softwareVersion, .id] |@csv'
"3504_WLC","10.10.20.51","8.5.140.0","50c96308-84b5-43dc-ad68-cda146d80290"
"leaf1.labb.local","10.10.20.81","16.6.4a","6a49c827-9b28-490b-8df0-8b6c3b582d8a"
"leaf2.labb.local","10.10.20.82","16.6.4a","d101ef07-b508-4cc9-bfe3-2acf7e8a1015"
"spine1.abc.in","10.10.20.80","16.3.5b","b558bdcc-6835-4420-bfe8-26efa3fcf0b9"
"T1-1","10.10.20.241","8.6.101.0","8cd186fc-f86e-4123-86ed-fe2b2a41e3fc"
"T1-10","10.10.20.250","8.6.101.0","a2168b2d-adef-4589-b3b5-2add5f37daeb"
"T1-2","10.10.20.242","8.6.101.0","0367820f-3aa4-434a-902f-9bd39a8bcd21"
"T1-3","10.10.20.243","8.6.101.0","8336ae01-e1a8-47ea-b0bf-68c83618de9e"
"T1-4","10.10.20.244","8.6.101.0","b65cea84-b0c2-4c44-a2e8-1668460bd876"
"T1-5","10.10.20.245","8.6.101.0","0aafed14-666b-4f9d-a172-6f169798631a"
"T1-6","10.10.20.246","8.6.101.0","e641ce97-dbba-4024-b64c-2f88620bcc23"
"T1-7","10.10.20.247","8.6.101.0","3aaffd4f-0638-4a54-b242-1533e87de9a7"
"T1-8","10.10.20.248","8.6.101.0","a4e0a3ab-de5f-4ee2-822d-a5437b3eaf49"
"T1-9","10.10.20.249","8.6.101.0","10cdbf6d-3672-4b4d-ae75-5b661fa0a5bc"

One big advantage of CLI tools is linking them together with standard Linux tools such as xargs. This provides a way to run a command based on a set of arguments. For example if I wanted to get a dump of all of the configuration files for devices, I would need to call the “get-device-config-by-id” API for each device, giving it an argument of a device UUID (as seen above in the id field). To start with I am just going to get the device ID.

I am choosing switches as the Access Points do not have a configuration (the configuration for AP is stored on the Wireless LAN controller).

$ dnacentercli devices get-device-list --family "Switches and Hubs"| jq -r '.response[] | .id | @text '
6a49c827-9b28-490b-8df0-8b6c3b582d8a
d101ef07-b508-4cc9-bfe3-2acf7e8a1015
b558bdcc-6835-4420-bfe8-26efa3fcf0b9

Next I use the xargs command to run request for the configuration with each of the id as an argument. The cli command is dnacentercli devices get-device-config-by-id –network_device_id

The xargs command creates an instance of get-device-config-by-id for each id.
The output is quite long so I have truncated it.

$ dnacentercli devices get-device-list --family "Switches and Hubs"| jq -r '.response[] | .id | @text ' | xargs -n1 dnacentercli devices get-device-config-by-id --network_device_id
{"response": "\nBuilding configuration...\n\nCurrent configuration : 18816 bytes\n!\n! Last configuration change at 02:02:05 UTC Sat Aug 31 2019 by admin\n!\nversion 16.6\nno service pad\nservice

This is an example of deleting a device with the ip address 10.10.15.200. Of course i could turn this into a script or a shell macro, and provide the IP address as an argument. I can either run a seperate command to get the status, or i could link it yet again. I am going to use a separate command in this case.

$ dnacentercli devices get-network-device-by-ip --ip_address 10.10.15.200 | jq -r '.response.id | @text ' | xargs dnacentercli devices delete-device-by-id --id
{"response": {"taskId": "cc8b8c29-98cd-4cd4-8c6c-eee96af31057","url": "/api/v1/task/cc8b8c29-98cd-4cd4-8c6c-eee96af31057"},"version": "1.0"}

$ dnacentercli task get-task-by-id --task_id cc8b8c29-98cd-4cd4-8c6c-eee96af31057 -pp 2
{
  "response": {
    "endTime": 1570147128225,
    "id": "cc8b8c29-98cd-4cd4-8c6c-eee96af31057",
    "instanceTenantId": "5d817bf369136f00c74cb23b",
    "isError": false,
    "lastUpdate": 1570147117840,
    "progress": "Network device deleted successfully",
    "rootId": "cc8b8c29-98cd-4cd4-8c6c-eee96af31057",
    "serviceType": "Inventory service",
    "startTime": 1570147117759,
    "version": 1570147117840
  },
  "version": "1.0"
}

The final example is using the CLI to create a site. –type and –area are required arguments.   As this is a POST, we can force the call to run synchronously by passing the __runsync header.  You can see the site was successfully created.

$ dnacentercli sites create-site --type "area" --site '{ "area" : { "name":"Adam","parentName":"Global"}}' --headers '{"__runsync" : true }'
{"result": {"result": {"endTime": 1570132102038,"progress": "Site Creation completed successfully","startTime": 1570132101935}},"siteId": "2ea3f4c2-04e2-4d01-8c12-4459c1e7a2c1","status": "True"}

Continue reading

Tune in: “Demystifying Cisco Orchestration for Infrastructure as Code”

Automating the software development life-cycle

DevOps teams are becoming more agile, reducing costs, and delivering a superb customer experience by automating the software development life-cycle. Cisco Orchestration solutions extend the benefits of automation to the entire stack. Each layer of the underlying infrastructure is delivered as Code (IaC). Orchestrators reduce the complexity of programmability, operational state, and visibility. In this session, we decode the differences between domain-specific workflow automation versus cross-domain orchestration. Achieving the goal of ‘Automate everything’ requires the right tool for the right use case. With use cases in mind, we will cover several Cisco orchestration solutions in their respective domains and cross domain capabilities. A brief demo will showcase Open Source and Cisco Orchestration tools working together hand-in-hand.

Level Set

What is Infrastructure as Code (IaC)?

IaC means writing imperative or declaration code to automate programmable infrastructure deployments and manage configurations. Imperative is how you do something step-by-step, as opposed to declarative which is ‘what to do’ by abstracting the configuration and state. DevOps best practices such as source control, verification, and visibility are building blocks to support infrastructure types (compute, network, storage, etc) as code.

Why do we need IaC?

With the advent of Continuous Integration and Continuous Delivery (CI/CD), we are able to build pipelines to automate the entire software development life-cycle (SDLC). Continuous integration (CI) is a set of tools to develop applications. Continuous delivery (CD) is the process of delivering updated software releases to infrastructure environments such as test, stage, and production. Using IaC for these platforms and environments is paramount to enabling software agility and rapid time to value. One could say, IaC is the easy button to building infrastructure to deliver software or other IT services.

Orchestration

What is orchestration? Wikipedia defines orchestration as an automated arrangement, coordination, and management that defines the policies and service levels through automated workflows, provisioning, and change management. In this same vein, a coffee grinder is automation where a brewing machine is orchestration.

Why do we need orchestration in addition to scripting? Production grade IaC at scale requires orchestration versus scripting to deliver advanced features such as intent, policy, governance, and Service Level Agreements. By building IaC to include configuration management, CICD, and other advanced orchestration features, similar benefits to application development are now possible in large scale technology domains (mulit-cloud, containers, campus, WAN, Data Center).

With network automation in mind, we see many pit falls with multi-threaded tasks running exclusively from scripts. Step (1) gather facts, Step (2) set conditions, Step (3) loop through items in jinja2 templates and parse to TextFSM and save the data to YAML files. Step (4) Push changes to devices and validate.

This level of scripted multi-threaded workflow is difficult to manage at scale. The main concerns are slow changes, configuration drift, lack of operation state, out of band config overwrites, and disruptive rollbacks. In spite of the current gaps, the Ansible engine is one of my favorite tools for pushing network configurations. One could argue that Tower provides a workflow for the playbooks to manage the order of these tasks but no configuration state is possible. In order to remediate some of these gaps, Ansible engine is adding a new ‘facts’ resource module in a future 2.9 release.

Caution

Sometimes we automate ourselves into a corner with too many scripts. What happens when major platform changes are made to the scripting tools? We’ve experienced this before with python 2.x to 3.x.and the impact to many of our product SDK libraries. We are seeing it again with the major change to the Ansible engine coming in 2.9 to introduce the ‘facts’ resource module. This change requires users to rewrite their playbooks from scratch to use these features. As a caution, consider limiting scripting to single threaded (CRUD ) actions while shifting the complexities of operational state and rollback to the domain specific orchestration engine.

Domain Orchestration

What is a domain orchestration? A domain orchestration engine focuses on delivering automation targeted to a single technology domain. For instance, the Cisco Network Services Orchestator (NSO) is focused on model driven “network” automation with Netconf and YANG. NSO converts CLI to YANG with network element drivers (NEDS) to supports a multitude of uses cases ranging from stand alone network devices, network services, and multiple controller domains (Meraki, Viptela, and ACI).

In a nut shell, NSO can deploy greenfield or snyc-from brownfield devices to build a transaction based configuration database state. Tools like Ansible engine have modules to integrate with NSO’s northbound JSON API to harness these differentiated capabilities for operations. In the following example, we are using Ansible playbooks with the Ansible NSO/Json module to make CRUD changes to NSO’s configuration database as a means to configure and operate tenants running on a N9K EVPN/VXLAN Data Center network fabrics versus CLI to the stand alone NXOS. The Ansible playbooks are then version controlled as YAML files in a git repository.

Top-level Orchestration

What is Top-level orchestration? A top-level orchestration engine is used to stitch together collaboration, notifications, governance, and source control for other lower level scripting tools and device APIs. Top-level orchestration supports use cases ranging from CICD pipelines for application development to automated infrastructure build and testing. The Cisco Action Orchestrator (AO) is a powerful Top-level orchestrator that enables automated workflows across technology domains and ITSM (ie., ServiceNow). Integrations to ITSM are key for customers who need low or no code catalogs and templates to simplify the delivery of IT services.

Internally, Cisco relies on ITSM and AO to automate the rapid delivery of CiscoLive and Devnet Sandboxes during our customer events. In the below example, the open source tools such as Gitlab work hand-in-hand with AO to create a workflow pipeline to automate the build and test for a tenant configuration across EVPN/VXLAN fabric and SDWAN network domains.

Confusion

Are you confused by CICD pipelines and their relationship with IaC? My ‘Ah hah’ moment was a realization that many of these DevOps methodologies are not mutually exclusive but highly complementary between AppDev and IaC. As operators, we can support the automated development life-cycle with CICD pipelines. This same knowledge and tools are adaptable to automated infrastructures. Operations can adopt the tools (open source or vendor) that make sense for updating, configuring and management of IaC in many domains.

If you look at AppDev the CICD pipeline for software development must CODE, BUILD, TEST, and DEPLOY the software to an environment that includes infrastructure (compute, network, and storage). Do we build these infrastructure environments ahead of time manually or automated on demand?

If the developer is not willing to patiently wait several weeks for the infrastructure environment to test their CODE, then fully automated IaC is the only answer! A second CICD pipeline managing the configuration, versioning, and alignment of the software build to the environment (test, stage, prod) version allows us to move much quicker and rebuild the environment later if needed.

AppDev CI/CD pipeline to IaC CI/CD pipeline

In the following example, we are using Gitlab to manage an application development CICD pipeline. Upon completion the AppDev pipeline triggers Action Orchestrator to build a second pipeline with workflows to automate the test environment to ultimately test the application stack. The idea is to test the software release in a test environment prior to pushing the same software into production. The Action Orchestrator (AO) has many adapters to make IaC very easy to build and test infrastructure technology domains.

Router/switch software upgrades are another use case for a network specific CICD pipeline. With CICD we can automate the upgrade of specific IOS software versions to devices in a version controlled and tested environment prior to production.

Controllers

Are controllers and orchestration one in the same? NOPE…  Controllers are a single API touch point and management system for Software Defined systems (SD-Access, SD-Wan, and SD-Networks) to manage the configuration state of the underlay and overlay and underlying protocols. Controllers are similar to orchestration by providing access to configuration snapshots and rollbacks, but unable to compose top-level workflows with other tools. In most cases, Controllers are bound to their single technology domain (campus, data center, WAN, or cloud). Often times, IaC is configured adequately with only scripting and source control in a single controller domain. Suffice it to say, when expanding from a single domain to cross domain controllers (ie SDWAN, and SDN) this cross domain integration introduces a catalyst for orchestration.

The Automation Challenge

There is a broad set of technology domains, each with many use cases for IaC. In order to succeed with IaC, we first need to address our automation challenges. From there we can target each specific use case mapped to the appropriate technology domain.

Challenges:

To many touchpoints: Need to consolidate and coordinate tasks using common automation tools.

Complexity: Need to abstract automation as much as possible to make resources consumable for the end users.

Operational Instrumentation: Need to automate and operationalize the tools into workflows that include visual dashboards, role-based access control, and other security services.

Verification: Need to make changes and check changes. With automation, we can move really fast and break things. Hence, we need the proverbial looking over our shoulder versus traditional stare and compare configuration checks. Ideally, verification should start in a test environment through production.

Community and Collaboration: Need to share finished code and avoid recreating the wheel with every workflow.

The key take away for automated solutions is to strive for a sharing culture, agility, simplicity, intent, security, and lower costs.

Technology Domains and Use Cases

The following table depicts the taxonomy of several Cisco orchestration options. As depicted below, the Action Orchestrator is positioned as the glue to bind together the multiple technology domains into a unified workflow.

What’s Next?

Multi Domain Policy

As our customers continue to strive for end-to-end automation their orchestration workflows are now spanning across multiple technology domains. As these workflows evolve we need to consolidate and coordinate tasks using a common automation platform.

A major step in the “automate everywhere” strategy is to consolidate automation on a Multi-Domain Policy (MDP) platform. Conceptually this upcoming platform is targeted to unify the existing orchestration engines across domains with a consistent UI, catalog, united operations, common segmentation, consistent on-boarding, and delivered on-prem or cloud.

AI Ops

Logs, telemetry, and health monitoring are currently used to build reactive dashboards for visibility. With the advent of AI Ops, the trend is predictive and self healing operations. AI Ops platforms utilize big data, modern machine learning, and other advanced analytics technologies. This new technology, directly and indirectly, enhances IT operations functions with proactive, personal and dynamic insight. Cisco Intersight is a SaaS addition to the portfolio of domain orchestration engines, making actionable intelligence available from AI Ops in Hyperflex and server domains. AI Ops capabilities are road-mapped into many other orchestration engines as well.

Continue reading

Service Providers: The Quest to Attain NFV Enlightenment

The promise of Network Function Virtualisation (NFV) was to lower Total Cost of Ownership (TCO) for the network and to improve service agility (time-to-market). However, like all new technologies, the hype of expectations has subsided over time, and disillusionment has set-in for network Service Providers with false starts on NFV investments.

NFV hype cycle

Fortunately, with experience comes ‘enlightenment’, and given that NFV has been deployed in live networks for a variety of functions and at various scales for years now, SPs are now able to ascertain the best practice approach and create true business value paired with ‘enhanced productivity’.

Looking forward, NFV platforms will be essential as they serve as the foundation for upcoming architectural shifts; to 5G core, edge computing (MEC) and cloud-native functions.

Best Practice for NFV Platforms

There are various approaches to building an NFV environment, from single-vendor vertical stack to a do-it-yourself (DIY) approach building a platform from various vendor and open-source software.

NFV platform stack approaches

Over time, Service Providers have discovered that the Vertical approach may not be as capital-efficient, due to its fixed configuration leaving stranded capacity assets. On the other hand, the DIY approach promises horizontal scaling but the complexities of integration and operations creates prohibitive additional costs, a struggle only the largest service providers are equipped to navigate.

Considering both options, the most efficient approach appears to be a combination of both, packaging the platform in modules for life-cycle management, but remaining open to supporting various virtual network functions (VNFs) to support horizontal scaling. This creates a platform that is Open and Modular with the right balance of multi-function scaling, with carrier-grade operational packaging and a single point of ownership.

Requirements of an Open Modular NFV Platform

An NFV platform must create business value; optimized to lower network TCO and increase service agility.

An open modular NFV platform achieves this with:

◈ A scalable architecture for large core to small edge locations, with common orchestration

◈ Network DC SDN that supports bare-metal, VM and container functions

◈ End to end instrumentation for carrier grade operations

◈ Modular life-cycle management for up-grades

◈ The ability to support a wide range of multi-tenanted VNFs with an eco-system of pre-validated VNFs and a replicable process for new VNF on-boarding

Benefits of an Open Modular NFV Platform

Based on keen observations and conversations with customers, the benefits of this approach have been proven. Personally, I have witnessed service agility (time-to-market) improvements of more than 10x, with new services pushed to launch in a matter of days instead of the several months the process used to require.

Replacing a separate vertical NFV stack deployment with a single open modular platform has shown TCO improvement of more than 30%, attributed to operations, power and rack space reductions.

These results are encouraging, with the tangible business benefits indicating that we have attained enlightenment on how NFV platforms should be built for the Service Providers of the future.

Continue reading

Threats in encrypted traffic

There was a time when the web was open. Quite literally—communications taking place on the early web were not masked in any significant fashion. This meant that it was fairly trivial for a bad actor to intercept and read the data being transmitted between networked devices.

This was especially troublesome when it came to sensitive data, such as password authentication or credit card transactions. To address the risks of transmitting such data over the web, traffic encryption was invented, ushering in an era of protected communication.

Today more than half of all websites use HTTPS. In fact, according to data obtained from Cisco Cognitive Intelligence, the cloud-based machine learning engine behind Stealthwatch—Cisco’s network traffic analysis solution—82 percent of HTTP/HTTPS traffic is now encrypted.

The adoption of encrypted traffic has been a boon for security and privacy. By leveraging it, users can trust that sensitive transactions and communications are more secure. The downside to this increase in encrypted traffic is that it’s harder to separate the good from the bad. As adoption of encrypted traffic has grown, masking what’s being sent back and forth, it’s become easier for bad actors to hide their malicious activity in such traffic.

A brief history of encrypted traffic

The concerns around security and privacy in web traffic originally led Netscape to introduce the Secure Sockets Layer (SSL) protocol in 1995. After a few releases, the Internet Engineering Task Force (EITF) took over the protocol, which released future updates under then name “Transport Layer Security” (TLS). While the term SSL is often used informally to refer to both today, the SSL protocol has been depreciated and replaced by TLS.

TLS protocol works directly with existing protocols and encrypts the traffic. This is where protocols like HTTPS come from— the hypertext transfer protocol (HTTP) is transmitted over SSL/TLS. While HTTPS is by far the most common protocol secured by TLS, other popular protocols, such as SFTP and SMTPS can take advantage of the protocol. Even lower-level protocols like TCP and UDP can use TLS.

Threat actors follow suit

Attackers go to great pains to get their threats onto systems and networks. The last thing they want after successfully penetrating an organization is to have their traffic picked up by network-monitoring tools. Many threats are now encrypting their traffic to prevent this from happening.

Where standard network monitoring tools might be able to quickly identify and block unencrypted traffic in the past, TLS provides a mask for the communication threats utilize to operate. In fact, according to data taken from Cognitive Intelligence, 63 percent of all threat incidents discovered by Stealthwatch were discovered in encrypted traffic.

In terms of malicious functionality, there are a number of ways that threats use encryption. From command-and-control (C2) communications, to backdoors, to exfiltrating data, attackers consistently use encryption to hide their malicious traffic.

Botnets

By definition, a botnet is a group of Internet-connected, compromised systems. Generally, the systems in a botnet are connected in a client-server or a peer-to-peer configuration. Either way, the malicious actors usually leverage a C2 system to facilitate the passing of instructions to the compromised systems.

Common botnets such as Sality, Necurs, and Gamarue/Andromeda have all leveraged encryption in their C2 communications to remain hidden. The malicious activity carried out by botnets include downloading additional malicious payloads, spread to other systems, perform distributed-denial-of-service (DDoS) attacks, send spam, and other malicious activities.

Botnets mask C2 traffic with encryption.

RATs

The core purpose of a RAT is to allow an attacker to monitor and control a system remotely. Once a RAT manages to implant itself into a system, it needs to phone home for further instructions. RATs require regular or semi-regular connections to the internet, and often use a C2 infrastructure to perform their malicious activities.

RATs often attempt take administrative control of a computer and/or steal information from it, ranging from passwords, to screenshots, to browser histories. It then sends the stolen data back to the attacker.

Most of today’s RATs use encryption in order to mask what is being sent back and forth. Some examples include Orcus RAT, RevengeRat, and some variants of Gh0st RAT.

RATs use encryption when controlling a computer.

Cryptomining

Cryptocurrency miners establish a TCP connection between the computer it’s running on and a server. In this connection, the computer is regularly receiving work from the server, processing it, then sending it back to the server. Maintaining these connections is critical for cryptomining. Without it the computer would not be able to verify its work.

Given the length of these connections, their importance, and the chance that they can be identified, malicious cryptomining operations often ensure these connections are encrypted.

It’s worth noting that encryption here can apply to any type of cryptomining, both deliberate and malicious in nature. As we covered in our previous Threat of the Month entry on malicious cryptomining, the real difference between these two types of mining is consent.

Miners transfer work back and forth to a server.

Banking trojans

In order for a banking trojan to operate, it has to monitor web traffic on a compromised computer. To do that, some banking trojans siphon web traffic through a malicious proxy or exfiltrate data to a C2 server.

To keep this traffic from being discovered, some banking trojans have taken to encrypting this traffic. For instance, the banking trojan IcedID uses SSL/TLS to send stolen data. Another banking trojan called Vawtrak masks its POST data traffic by using a special encoding scheme that makes it harder to decrypt and identify.

Banking trojans encrypt the data they’re exfiltrating.

Ransomware

The best-known use of encryption in ransomware is obviously when it takes personal files hostage by encrypting them. However, ransomware threats often use encryption in their network communication as well. In particular, some ransomware families encrypt the distribution of decryption keys.

How to spot malicious encrypted traffic

One way to catch malicious encrypted traffic is through a technique called traffic fingerprinting. To leverage this technique, monitor the encrypted packets traveling across your network and look for patterns that match known malicious activity. For instance, the connection to a well-known C2 server can have a distinct pattern, or fingerprint. The same applies to cryptomining traffic or well-known banking trojans.

However, this doesn’t catch all malicious encrypted traffic, since bad actors can simply insert random or dummy packets into their traffic to mask the expected fingerprint. To identify malicious traffic in these cases, other detection techniques are required to identify the traffic, such as machine learning algorithms that can identify more complicated malicious connections. Threats may still manage to evade some machine learning detection methods, so implementing a layered approach, covering a wide variety of techniques, is recommended.

In addition, consider the following:

◈ Stealthwatch includes Encrypted Traffic Analytics. This technology collects network traffic and uses machine learning and behavioral modeling to detect a wide range of malicious encrypted traffic, without any decryption.

◈ The DNS protection technologies included in Cisco Umbrella can prevent connections to malicious domains, stopping threats before they’re even able to establish an encrypted connection.

◈ An effective endpoint protection solution, such as AMP for Endpoints, can also go a long way towards stopping a threat before it starts.

Continue reading

16 Cable Industry Terms You Need to Know to Get Ready for Cable-Tec 2019

SCTE’s Cable-Tec Expo 2019 is just around the corner (September 30th-October 3rd in New Orleans). Plan on a busy week given the wide range of technology on display in the exhibit hall, the 115 papers being presented in the Fall Technical Forum, numerous panel discussions on the Innovation Stage, keynote presentations during the opening General Session and annual awards luncheon, and so much more. If you’re a newcomer to the industry (or new to Cable-Tec Expo), you may find some of the jargon at the conference a bit overwhelming.

I’ve defined 16 terms that you need to know before you go to Cable-Tec 2019:

1. Multiple System Operator (MSO) – A corporate entity such as Charter, Comcast, Cox, and others that owns and/or operates more than one cable system. “MSO” is not intended to be a generic abbreviation for all cable companies, even though the abbreviation is commonly misused that way. A local cable system is not an MSO, either – although it might be owned by one – it’s just a cable system. An important point: All MSOs are cable operators, but not all cable operators are MSOs.

2. Hybrid Fiber/Coax (HFC) – A cable network architecture developed in the 1980s that uses a combination of optical fiber and coaxial cable to transport signals to/from subscribers. Prior to what we now call HFC, the cable industry used all-coaxial cable “tree-and-branch” architectures.

3. Wireless – Any service that uses radio waves to transmit/receive video, voice, and/or data in the over-the-air spectrum. Examples of wireless telecommunications technology include cellular (mobile) telephones, two-way radio, and Wi-Fi. Over-the-air broadcast TV and AM & FM radio are forms of wireless communications, too.

4. Wi-Fi 6 – The next generation of Wi-Fi technology, based upon the Institute of Electrical and Electronics Engineers (IEEE) 802.11ax standard (the sixth 802.11 standard, hence the “6” in Wi-Fi 6), that is said to support maximum theoretical data speeds upwards of 10 Gbps.

5. Data-Over-Cable Service Interface Specifications (DOCSIS®) – A family of CableLabs specifications for standardized cable modem-based high-speed data service over cable networks. DOCSIS is intended to ensure interoperability among various manufacturers’ cable modems and related headend equipment. Over the years the industry has seen DOCSIS 1.0, 1.1, 2.0, 3.0 and 3.1 (the latest deployed version), with DOCSIS 4.0 in the works.

6. Gigabit Service – A class of high-speed data service in which the nominal data transmission rate is 1 gigabit per second (Gbps), or 1 billion bits per second. Gigabit service can be asymmetrical (for instance, 1 Gbps in the downstream and a slower speed in the upstream) or symmetrical (1 Gbps in both directions). Cable operators around the world have for the past couple years been deploying DOCSIS 3.1 cable modem technology to support gigabit data service over HFC networks.

7. Full Duplex (FDX) DOCSIS – An extension to the DOCSIS 3.1 specification that supports transmission of downstream and upstream signals on the same frequencies at the same time, targeting data speeds of up to 10 Gbps in the downstream and 5 Gbps in the upstream! The magic of echo cancellation and other technologies allows signals traveling in different directions through the coaxial cable to simultaneously occupy the same frequencies.

8. Extended Spectrum DOCSIS (ESD) – Existing DOCSIS specifications spell out technical parameters for equipment operation on HFC network frequencies from as low as 5 MHz to as high as 1218 MHz (also called 1.2 GHz). Operation on frequencies higher than 1218 MHz is called extended spectrum DOCSIS, with upper frequency limits as high as 1794 MHz (aka 1.8 GHz) to 3 GHz or more! CableLabs is working on DOCSIS 4.0, which will initially spell out metrics for operation up to at least 1.8 GHz.

9. Cable Modem Termination System (CMTS) – An electronic device installed in a cable operator’s headend or hub site that converts digital data to/from the Internet to radio frequency (RF) signals that can be carried on the cable network. A converged cable access platform (CCAP) can be thought of as similar to a CMTS. Examples include Cisco’s uBR-10012 and cBR-8.

10. 5G – According to Wikipedia, “5G is the fifth generation cellular network technology.” You probably already have a smart phone or tablet that is compatible with fourth generation (4G) cellular technology, the latter sometimes called long term evolution (LTE). Service providers are installing new towers in neighborhoods to support 5G, which will provide their subscribers with much faster data speeds. Those towers have to be closer together (which means more of them) because of plans to operate on much higher frequencies than earlier generation technology. So, what does 5G have to do with cable? Plenty! For one thing, the cable industry is well-positioned to partner with telcos to provide “backhaul” interconnections between the new 5G towers and the telcos’ facilities. Those backhauls can be done over some of our fiber, as well as over our HFC networks using DOCSIS.

11. 10G – Not to be confused with 5G, this term refers to the cable industry’s broadband technology platform of the future that will deliver at least 10 gigabits per second to and from the residential premises. 10G supports better security and lower latency, and will take advantage of a variety of technologies such as DOCSIS 3.1, full duplex DOCSIS, wireless, coherent optics, and more.

12. Internet of Things (IoT) – IoT is simply the point in time when more ‘things or objects’ were connected to the Internet than people. Think of interconnecting and managing billions of wired and wireless sensors, embedded systems, appliances, and more. Making it all work, while maintaining privacy and security, and keeping power consumption to a minimum are among the challenges of IoT.

13. Distributed Access Architecture (DAA) – An umbrella term that, according to CableLabs, describes relocating certain functions typically found in a cable network’s headends and hub sites closer to the subscriber. Two primary types of DAA are remote PHY and flexible MAC architecture, described below. Think of the MAC (media access control) as the circuitry where DOCSIS processing takes place, and the PHY (physical layer) as the circuitry where DOCSIS and other RF signals are generated and received.

14. Remote PHY (R-PHY) – A subset of DAA in which a CCAP’s PHY layer electronics are separated from the MAC layer electronics, typically with the PHY electronics located in a separate shelf or optical fiber node. A remote PHY device (RPD) module or circuit is installed in a shelf or node, and the RPD functions as the downstream RF signal transmitter and upstream RF signal receiver. The interconnection between the RPD and the core (such as Cisco’s cBR-8) is via digital fiber, typically 10 Gbps Ethernet.

15. Flexible MAC Architecture (FMA) – Originally called remote MAC/PHY (in which the MAC and PHY electronics are installed in a node), FMA provides more flexibility regarding where MAC layer electronics can be located: headend/hub site, node (with the PHY electronics), or somewhere else.

16. Cloud – I remember seeing a meme online that defined the cloud a bit tongue-in-cheek as “someone else’s computer.” When we say cloud computing, that often means the use of remote computer resources, located in a third-party server facility and accessed via the Internet. Sometimes the server(s) might be in or near one’s own facility. What is called the cloud is used for storing data, computer processing, and for emulating certain functionality in software that previously relied upon dedicated local hardware.

There are many more terms and phrases you’ll see and hear at Cable-Tec Expo than can be covered here. If you find something that has you stumped, stop by Cisco’s booth (Booth 1301) and ask one of our experts.

Continue reading

Cable Service Providers: You Have a New Game to Play. And You Have the Edge

Time to take gaming seriously

Video gaming is huge, by any measure you choose. By revenue, it’s expected to be more than $150 billion in 2019, making it bigger than movies, TV and digital music, with a strong 9% CAGR.

And it’s not just teenagers. Two-thirds of adult Americans — your paying customers — call themselves gamers.

This makes gaming one of the biggest opportunities you face as a cable provider today. But how can you win those gamers over and generate revenue from them?

New demands on the network

A person’s overall online gaming experience depends on lots of factors, from their client device’s performance through to the GPUs in the data center. But the network — your network — clearly plays a critical role. And gaming related traffic is already growing on your networks. Cisco’s VNI predicts that by 2022, gaming will make up 15% of all internet traffic.

When Gamers talk about “lag” affecting their play, saying they care about milliseconds of latency, they really mean overall performance – latency, jitter, drops. Notice how latency changes the gamer’s position (red vs blue) in the below screenshot:

Many would even pay a premium not just for a lower ping time, but also for a more stable one, with less jitter and no dropped packets. Deterministic SLAs become key.

But latency, jitter and drops aren’t the only factors here. Gamers also need tremendous bandwidth, especially for:

◈ Downloading games (and subsequent patches) after the purchase. Many games can exceed 100 GB in size!

◈ Watching others’ video gameplay on YouTube or Twitch. The most popular gamer on YouTube has nearly 35 million subscribers!

◈ Playing the games using cloud gaming services such as the upcoming Google Stadia. 4K cloud gaming could require around 15 GB per hour, twice that of a Netflix 4K movie.

In many cases, the upstream is as important as downstream bandwidth — an opportunity for cable ISPs to differentiate themselves on all those factors with end-to-end innovations.

Your chance to lead

As a cable ISP, you’re the first port of call for gamers looking for a better experience. You can earn their loyalty with enhanced Quality of Experience and even drive new premium service revenue from it.

There’s opportunity for you to be creative, forging new partnerships with gaming providers, hosting gaming servers in your facilities, and even providing premium SLAs for your gaming customers along with new service plans.

But there’s plenty to be done in the network to make these opportunities real.

At the SCTE Expo, we will be discussing specific recommendations for each network domain. To give a teaser, in access network domain, you need to take action to reduce congestion and increase data rate, setting up prioritized service flows for gaming to assure QoS. New technologies like Low Latency DOCSIS (LLD) will be critical for delivering the performance your customers want, optimizing traffic flows and potentially delivering sub-1ms latency without any need to overhaul your HFC network infrastructure itself. In the peering domain, you need to … OK, let’s save that for the live discussion. We will be happy to help on all those fronts.

The cable edge is your competitive edge

Gaming is not the only low-latency use case in town. For example, Mobile Xhaul (including CBRS) and IoT applications depend on ultra-responsive and reliable network connectivity between nodes. And there are plenty of other use cases beyond gaming that are putting new strains on pure capacity, including video and CDNs.

All of these use cases too will benefit from the traffic optimization that LLD enables, but it’s only part of the solution.

IT companies of all shapes and sizes are recognizing that for many of these use cases, putting application compute closer to the customers, at the edge, is the only way forward. 

After all, the best way to reduce latency (and offer better experience) is to cut the route length by hosting application workloads as geographically and topologically close to the customers as possible. This approach also reduces the need for high network bandwidth capacity to the centralized data centers for bandwidth-heavy applications like video and CDN.

Imagine a gaming server colocated in a hub giving local players less than 10ms latency/jitter. Or a machine-vision application that monitors surveillance camera footage for alerts right at the edge, eliminating the need to send the whole livestream back to a central data center. The possibilities are endless.

Expand your hub sites into application cloud edge

In the edge world, your real differentiator becomes the thousands of hub sites that you use to house your CMTS/CCAP, EQAM and other access equipment — sites that SaaS companies and IT startups simply can’t replicate. Far from being a liability to shed and consolidate, this distributed infrastructure is one of your critical advantages. 

By expanding the role of your hub sites into application cloud edge sites, you can increase utilization of your existing infrastructure (for example, cloud-native CCAP), and generate revenue (for example, hosting B2B applications), both by innovating new services of your own and by giving third-party service providers access to geographic proximity to their B2B and B2C users. 

If you’re also a mobile operator, this model also allows you to move many virtualized RAN functions for into your hub sites, leaving a streamlined set of functions on the cell site itself (this edge cloud model is one that Rakuten is using for its 5G-ready rollout, across 4,000 edge sites).

Making cable edge compute happen

We’ve introduced the concept of Cable Edge Compute, describing how you can turn your hubs into next generation application-centric cloud sites to capture this wide-ranging opportunity.

While edge compute architectures do present a number of challenges — from physical space, power & cooling constraints to extra management complexity and new investment in equipments — these are all solvable with the optimal innovations, right design and management approaches. It’s vital to approach an initiative like this with an end-to-end service mindset, looking at topics like assurance, orchestration and scalability from the start.

Four key ingredients for cable edge compute

Here are essentially four key ingredients for a cost-optimized cable edge compute architecture: 

1. Edge hardware: takes the form of standardized, common SKU modular pods with x86 compute nodes and top-of-rack switches, with optional GPUs and other specialized processing acceleration for specific applications. Modularity, Consistency and flexibility are key here, so as to be able to scale easily. 

2. Software stack: enables the Edge hardware to optimally host a wide range of virtualized applications in containers or VMs or both, whether managed through Kubernetes or Openstack or something else. What’s important is to minimize the x86 CPU cores usage by the software stack and provide deterministic performance. Cisco has made it possible by combining cloud controller nodes with the compute nodes at the edge, but moving storage nodes and management nodes to the remote clusters with specific optimization and security. This optimizes the usage of physical space and power in the Hub site.  

3. Network Fabric: provides ultra-fast connectivity for the application workloads to communicate with each other and consumers. A one- or two-tier programmable network fabric based on 10GE/40GE/100GE/400GE Ethernet transport with end-to-end IPv6 and BGP-based VPN. 

4. And finally, this infrastructure model depends totally on SDN with automation, orchestration and assurance. Configuration and provisioning must be possible remotely via intent files, for example. At this scale, with an architecture this distributed, tasks should be zero-touch across the lifecycle. Assurance is utterly foundational, both to assure appropriate per-application SLAs and the enforcement of policy around prioritization, security and privacy. 

Discover your opportunity with low-latency and edge compute

In the new world of low-latency apps delivered through the edge, cable SPs are in a great position.  

And there’s never been a better opportunity to learn more about what this future holds. Cisco CX is presenting at SCTE Cable-Tec Expo on the gaming opportunity and cable edge compute, and we’ve published two new white papers that you can consult as an SCTE member.

Continue reading