How Stealthwatch Cloud protects against the most critical Kubernetes vulnerability to-date, CVE-2018-1002105

The increasing popularity of traditional cloud computing technologies such as server-less, on-demand compute and containerized environments has made technologies like Kubernetes part of our daily vernacular as it relates to running our applications and workloads.

Kubernetes solves many of the problems with managing containers at-scale. Automation, orchestration, elasticity are a few of the major draws for organizations to leverage Kubernetes, either in the cloud, on premises, or hybrid. Kubernetes creates a network abstraction layer around the siloed containers that allows for this facilitation. Think of it as a wide open highway that allows you to route throughout the many containers that are actually performing your workloads.  From web servers to database servers, these containers are the flexible, scalable workhorses for an organization.

With great accessibility comes a drawback, however. Should an attacker gain access to a pod, a node or an internal Kubernetes service, then part or all of that cluster is at risk of compromise. Couple that with the fact that in many instances Docker containers are actually running scaled down Linux operating systems like CoreOS or Alpine Linux. Should one of those containers become exposed to the Internet (and many workloads require access to the Internet), you now have an exposed attack surface that expands along with the exposed workloads themselves.

Last week the most severe Kubernetes vulnerability discovered to-date was announced, CVE-2018-1002105. It scored a 9.8 out of a possible 10.0 on the CVSS severity score…which is unprecedented.  In a nutshell this vulnerability allows an attacker to send an unauthenticated API request to the Kubernetes API service. Despite being unauthenticated, the access request leaves a remaining TCP connection open for the API backend server. This connection then allows an attacker to exploit the connection to run commands that would grant them complete access to do anything they desire on the cluster.  Scary stuff!

This vulnerability underscores the fact that organizations need to have both the visibility to see such traffic and also the analytics to know if the traffic represents a risk or compromise. Suppose you unknowingly expose a group of Apache Kubernetes pods to the Internet to perform their intended web services and a new vulnerability is exploited on that pod, like Struts. The attacker would then have root access on the pod to perform recon, install necessary tools and pivot around the cluster. And, if they are aware of the API vulnerability, then it’s a walk in the park for them to take full control of your cluster in a matter of minutes.

Not a good day for an organization if – and more likely when – this occurs. Data theft, compute theft, skyrocketing bills….just to name a few, are immediate side effects to a takeover of this magnitude.  So how can Stealthwatch Cloud help in this scenario and similar potential exploits?

How Stealthwatch Cloud Protects Kubernetes Environments 

Stealthwatch Cloud deploys into a Kubernetes cluster via an agentless sensor that leverages Kubernetes itself to automatically deploy, expand and contract across a cluster. No user interaction is required. The solution deploys instantly to every node in a cluster and exposes every pod and the communication with those pods between internal nodes and clusters, as well as externally. This allows for an unprecedented level of visibility into everything a cluster is doing, from pods communicating to the internet to worker nodes communicating internally with the master node. We then add entity modeling which compares new behavior to previous behavior and machine learning based anomaly detection to alert on IOC’s throughout the Kill Chain to alert on over 60 indicators of suspicious activity across a cluster.

Hypothetically speaking, if one of your Kubernetes clusters were compromised, Stealthwatch Cloud would send alerts in real-time on various aforementioned activities. The tool would alert on the initial pod reconnaissance, and on connection activity once the pod was exploited. If the attacker moved towards the API server, Stealthwatch Cloud would alert on internal reconnaissance, suspicious connections to the API server itself, further data staging, data exfiltration and a variety of other alerts that would indicate a change from known good behavior across every component of a Kubernetes cluster…all in an agentless, automated, scalable solution.

Continue reading

Planning a Cloud Communications Migration: Navigating IT Priorities & LAN/WAN Management

In this post, we take on another common challenge in migrating business communications to the cloud – the increased demands on IP access connectivity – both from a technical and commercial perspective. From a technical standpoint, hosted voice and video communications are some of the most sensitive traffic to circuit quality and bandwidth limitations. From a commercial standpoint, IP access service contracts typically bring extended terms, multiple planning horizons, and a broad set of stakeholders and priorities.

The substance of this topic has led us to split the fourth installment of this series into three parts. These parts align to key steps common in IP access service procurement: 1) situation analysis; 2) requirements definition; and 3) options review and selection.

Part 1: Background and communications delivery model comparison
Part 2: Key methodologies for delivering service assurance
Part 3: Key elements in the “stack” of IP access service connectivity

Background and Communications Delivery Model Comparison

We start by looking at the breadth of challenges for IT managers and the CSPs when it comes to IP Access and WAN procurement. IT managers have a wide and growing list of priorities outside of communications. In fact, communications often sits at the bottom of the queue unless there is a problem that requires urgent attention. We see evidence of this through a review of the 2018 Tech Target Survey of IT Priorities which tracked the responses of more than 1,300 enterprise IT professionals.

The results of the survey (see figure 1 below) show an array of disparate priorities – from wireless networking and network automation to WAN optimization. Note that these priorities are tracked separately from “security,” which is covered in a separate survey question. Security cuts across all enterprise networking initiatives and typically sits at first or second overall. Back to the results shown in figure 1 – look at the position of UC applications and platforms. It actually rates at the bottom of the list at 14%. This is not surprising, especially as this equates roughly to the number of businesses at any one time that are in the midst of a communications networking project. This finding also suggests that most IT managers take a somewhat “reactive” stance to communications networking – which comes back to the entire point of this blog series – helping IT managers get ahead of communications issues by taking a more proactive and structured approach to their communications.

In addition, this survey result shows the number of competing priorities IT faces when the time comes to move communications to the cloud. They see 11 or more enterprise networking priorities, most of which are impact communications networking. Facing these obstacles, no wonder many businesses extend the operation of their PBXs five and 10 years beyond their planned end-of-life.

Figure 1: Tech Target Top Enterprise Network Initiatives Survey Results for 2018

Add to this mix of priorities the available options for access IP. For IT managers evaluating connectivity, they still see a “mixed bag” of circuit types – various types of fiber, copper, and wireless services. Consider the April 2018 report from Vertical Systems Group that only 54.8% of US businesses even have access to fiber-based connectivity. Yes, this is an advance over 39.3% availability in the US in 2013. With most countries still near or below 50% fiber availability, it means that IT planners need to work with a variety of non-fiber based connectivity options. This should not halt cloud communications migrations. In fact, there are plenty of excellent non-fiber based options in the market today with new solutions on the horizon, including new 5G point-to-point technologies.

To help IT planners, we will do a round-up of common types of IP access connectivity suitable for supporting cloud communications. We will also look at service assurance mechanisms and CSP managed services typically delivered as part of network connectivity. We will outline how these mechanisms and managed services support QoS and can help IT managers move forward with confidence.

PBX-Based Communications

In a PBX-based communications architecture, the PBX or private branch exchange is typically located at the businesses site location. The PBX serves as a local registration point for communications devices and manages inbound and outbound communications traffic. The PBX together with the site’s access device enable businesses to separate their network architecture, both physically and from a management standpoint, into a “local area” and a “wide area.”

The local area network (LAN) is often managed by the business’s site IT team or can be outsourced to a 3rd party IT firm. The LAN typically serves applications that vary from business systems (ERP, CRM), security systems (entry/exit, surveillance), communications (voice, video, paging), and IT device management (printers, copiers).

If the business still operates a TDM PBX or analog lines, then the LAN may represent multiple local networks – one dedicated physical plant for voice traffic and a separate network for the broader set of IT applications.

For calls and communications that originate or terminate outside the business’s site, the calls need an outside connection. Businesses typically procure access connectivity from their communications service provider (CSP). For larger businesses that might need to manage site-to-site services, the business would procure wide area network (WAN) services from their CSP to provide PSTN access, internet access, and specialized connectivity (e.g., private lines) across sites.

As the PBX came to dominate the business communications landscape, the importance of this distinction between LAN and WAN grew. The interconnect point between the LAN and WAN became known as a “demarcation” point which represented the physical, logical, and contractual hand-off point between CSP-managed WAN traffic and IT-managed LAN traffic. In the case of PSTN services, the CSP provides voice “trunking” which might be sold a set of simultaneous call paths, minutes of use (MOU), and regulatory services such as 911.

Figure 2. Common PBX deployment with Site 1 operating a TDM PBX and Site 2 an IP PBX

Cloud-Based Communications

In a cloud communications solution, PBX functionality such as device registration, call processing, and media mixing are moved into the cloud and delivered by the CSP. The preferred way to deliver cloud-based communications is through voice-over-IP protocols and requires an end-to-end IP connection between CSP systems and user endpoints, including handsets, soft clients, conference room video systems, and more. With the call processing moving to the cloud, the nature of the LAN to WAN boundary changes. Now, the CSP directly manages endpoints and services that sit within the customer’s LAN.

This seemingly subtle change in how cloud communications services are delivered creates a significant shake-up in telecom and communications services procurement – breaking the long-established demarcation point between CSP and IT department service responsibilities. Ideally, the CSP owns “end-to-end” responsibility for the service. In reality, the IT department owns some measure of responsibility given that LAN management is outside the control of the CSP.

This shake-up in responsibilities is compounded by the opportunities to consolidate more and more services and applications in the cloud and on IP networks. WAN links will carry a lot more traffic when ERP, CRM, HR, and other apps move to the cloud. And where a business might have procured a combination of IP access, TDM, and analog circuits for resilience and legacy application needs, businesses are increasingly able to procure a single, high-quality fiber-based IP connection that can handle all or most applications requirements. While more efficient and cost-effective, this “shared services” environment also sets the stage for more questions around CSP vs. IT responsibilities for issue-resolution, SLAs, and performance optimization.

To help CSPs and IT departments address these challenges, we return to the overall theme of this blog series – reducing migration risk, building a working partnership between CSP and IT departments, and taking a more staged and structured migration approach.

To these ends, one of the areas where CSPs and IT managers need to partner is in the assessment of access network connectivity options available for each site. A business’s preferred CSP for hosted voice services may not offer bundled access connectivity with service assurance at any or most of the sites targeted for migration. In some cases, the CSP may partner with 3rd party access connectivity providers who offer access solutions with service assurance options for communications traffic. In other cases, the only option will be to carry media traffic “over the top” (OTT) on unmanaged IP connectivity.

IT managers should understand the key characteristics of access network connectivity and how these characteristics impact communications application performance. These characteristics can be broken into two areas:

◈ Layer 1 and 2 connectivity type

◈ Network services to support access layer service quality

Figure 3.  Common Hosted UC Deployment Supporting Services Across Site 1, Site 2, and Mobile Users

Continue reading

Integrating Warehouses and Distribution Centers into the Enterprise Network

We’re witnessing digital transformation all around us, and it’s driving fierce competition, new challenges, and compelling opportunities in every industry. We talked about some of the ways business are using the Internet of Things (IoT) to extend operations beyond traditional carpeted spaces inside the office and out to environments of all kinds.

In this edition, we’ll focus on how organizations with complex enterprise supply chains are transforming their businesses. Global retailers, distributors, manufacturers, healthcare, pharmaceuticals, and container operators are all driving innovation in the Extended Enterprise.

These innovators are exploring new strategies for Digital Transformation in industries to increase profitability, fulfill orders faster, improve safety, reduce cost and complexity, and more.

Everything Starts with Connectivity

To achieve these outcomes, secure connectivity across your entire supply chain is key.

You may be hampered by siloed and independent enterprise operations in your warehouses and distribution facilities due to inadequate system integration, lack of real-time visibility, and inconsistent safety control. Manual and error-prone inventory and reporting processes cost time and money—and create dissatisfied customers. To overcome these challenges and increase productivity and safety of your employees, you need to adopt digitization of your entire enterprise.

This isn’t always easy in busy and open industrial environments with massive volumes of traffic and employees constantly on the move. You need smart cameras and sensors to monitor operations and physical security to protect yourself against asset loss and theft, but also sophisticated cybersecurity to keep sensitive company and customer data from being compromised.

By securely connecting various warehouse systems – sensors, automated conveyors and sorters, safety and security systems and smart mobile devices you can monitor all warehouse systems and operations closely and gain access to data from disparate data sources. You can then analyze the data and develop insights to improve safety and operational efficiency.

Finally, you need real-time visibility to track heavy equipment and other assets, so you can perform timely maintenance and keep resources available and working at their best.

What You’ll Need

What’s required to connect and digitize warehouses? Enterprise supply chain environments are often harsh, dynamic environments that may be exposed to the weather, and require ruggedized connectivity to warehouse systems. To support employees and equipment on the move, they also require wireless connectivity everywhere. Physical security and real-time surveillance systems are key to keeping materials, people, and assets safe.

Also in order to keep IT operational costs low and centrally manage the entire enterprise, the warehouse network must be fully manageable using the same familiar management and policy-based orchestration tools used in the core enterprise. The network should provide secure access between warehouse devices and the enterprise, with the ability to scale and replicate to more warehouses and distribution centers as the organization evolves and grows.

Putting the Extended Enterprise to Work

Let’s take a closer look at how Cisco’s IoT solutions can help retailers scale to hundreds of connected and fully digitized warehouses and fulfillment centers.

Cisco Industrial Ethernet Switches and Cisco Outdoor and Industrial Access Points are purpose-built ruggedized solutions that can operate in environments without air-conditioning and deliver high performance Gigabit Ethernet connectivity for warehouse systems – sensors, cameras, sorters, conveyors, mobile phones etc. You can use the Cisco DNA Center to design, provision and manage these switches and APs.

Retailers also require end-to-end management and smooth communication with upper layer systems such as Supervisory Control and Data Acquisition (SCADA), Warehouse Control Systems (WCS), and cloud services. Cisco Industrial Ethernet Switches and  Cisco Industrial Security Appliances (ISA) offer strong security mechanisms to enable secure integration and communication with these upper layer systems. Cisco Industrial Integrated Services Routers extend connectivity further to service fleet such as delivery trucks for efficient fleet management and tracking of shipments.

Cisco Extended Enterprise solutions improve tracking and management of assets and inventory, as well as machinery and shipping equipment. Robust cybersecurity and video surveillance help keep data and assets safe. Retailers also gain complete operational visibility into operations across the organization, by fully integrating their supply chain into business applications.

No matter what type of warehouse or distribution center environment you’re supporting, Cisco Extended Enterprise solutions can help you extend your organization to gain real competitive advantages in today’s IoT age.

Continue reading

Simplify Your Communications Migration Options: Private & Public Cloud Solutions

In this post, we tackle another common challenge in migration planning – the decision to deploy sites and users on either multi-tenant or “public” cloud solutions versus dedicated, virtualized systems in “private” cloud solutions. In the communications industry, PBX vendors and service providers tend to promote the specific solutions that they sell, taking an almost “black and white” perspective about the benefits of either public or private cloud options.

Public or private cloud. Where do you stand?

If you are a service provider or IT manager, you are probably bombarded by strong opinions on each side – especially for telecom and business communications services. This resulting cacophony of opinions only serves to confuse decision-makers and delay migration plans. To bring some clarity here, this blog post tackles the merits of both public and private cloud, and particularly where these two solutions can work together

The reality, of course, is that there are advantages of both private and public cloud. In fact, a 2018 survey of almost 1,000 IT professionals reveals that 81% of businesses have a “multi-cloud” strategy as shown below: On the right-hand side, you can see some added color to what “Multi-Cloud” means. For that 81% that are Multi-Cloud, 10% are entirely private cloud, 21% are entirely public cloud while 51% is a mix of public and private.

Figure 1: RightScale | 2018 State of the Cloud Report

These numbers present a difficult starting point for migration planning for both IT Managers and CSPs. While in the long term (10+ years), most applications are envisioned for public cloud, the data show that a variety of intermediate steps and near-term plans (over ~5 years) will need a mix of private cloud. To help business IT planners and CSPs, this post will cover:

◈ Description of private and public cloud relative to premises equipment systems

◈ Migration scenarios where a mix of private and public cloud solutions might make sense

◈ CSP opportunity to productize a mix of cloud solution offers

Premises vs. Private vs. Public Cloud

The differences between private and public cloud are subtle but important. From the perspective of end-user applications, there may be no significant difference between the receipt of services from private and public cloud infrastructure. For IT planners and CSPs, the differences can have significant implications. Before describing the attributes of these solutions, it is important to provide some context – the typical starting point for a communications migration, the premises-based PBX system. Below see key attributes of premises PBX, private and public cloud solutions.

Premises-based PBX:

◈ Financing: Systems infrastructure is purchased as a capital expense and depreciated over the life of the equipment.

◈ Management: Systems are managed by the business’s IT staff or in combination with an IT tech firm to provide maintenance and support.

◈ Location: Individual systems are located on a site – in either a wiring closet or, for bigger sites, in an IT rack with power, cooling, controlled access, etc.

◈ Telecom Services (PSTN): Telecom and PSTN services are delivered from a 3rd party (CSP typically) to a “demarcation” point and interconnected to the PBX to allow PSTN calling and emergency services.

◈ Integrations: Connections into 3rd party cloud applications (e.g., salesforce.com) or local systems (e.g., elevators, entry/exit systems, and alarm systems) are managed as “custom” integrations leveraging native PBX APIs and through a specialist vendor or through a preferred IT tech firm.

◈ Economics: Provide lower cost / seat / month for larger sites with simple telephony needs.

Private cloud or dedicated, virtualized cloud systems:

◈ Financing: Systems infrastructure can be owned by the CSP with services delivered for a monthly service charge or owned by the business and treated as a capital expense. There are a variety of financing models depending on the CSP or PBX vendor.

◈ Management: Systems infrastructure can be managed by the CSP, an IT services firm, or the PBX vendor directly. There are multiple options for who manages systems, provides upgrades, and support.

◈ Location: Systems can be located in CSP data centers, IT datacenter, or even the business’s own IT data centers.

◈ Telecom Services: Telecom and PSTN services can either be bundled communications applications or split off and managed as separate contracts.

◈ Integrations: Similar to PBX systems with most integrations managed as “custom” projects leveraging the PBX APIs. There may be some limitations based on location or if the virtualized system leverages a specialized provisioning and management system or “wrapper.”

◈ Economics: Provide lower cost / seat / month when deployed as part of a broader “private cloud” deployment and serve more basic communications needs.

Public cloud or multi-tenant deployments:

◈ Financing: Systems infrastructure is owned and managed by the CSP. Services are provided off this infrastructure to the business for a monthly fee, typically priced by the user, metered charge (e.g., minutes), or virtual service (e.g., IVRs, Meeting Bridges).

◈ Management: Systems are managed by the CSP, including upgrades, maintenance, and support. CSP systems serve multiple businesses.

◈ Location: Systems are typically managed by the CSP in a state of the art data center that leverages public cloud providers (e.g., Google, Amazon, etc.) with extensive peering to partner networks.

◈ Telecom Services: Telecom and PSTN services are typically bundled with user, group, and virtual services with limited to no options to split traffic across multiple wholesale carriers.

◈ Integrations: Connections to 3rd party cloud providers are typically provided as part of standard service packages. There may be limitations to what is actually supported via CSP APIs.

◈ Economics: Provide lower cost / seat / month for multi-site deployments, with more advanced services, and more 3rd party cloud integrations.

Hybrid Cloud Migration Options

From the above description of the attributes of private and public cloud deployments, you may begin to see some of the scenarios where a business may want to take advantage of a mix of cloud solutions. Some common scenarios are as follows:

Large HQ with many small branch offices: Many businesses and specific verticals combine a large headquarters or regional site with multiple smaller branch offices. Headquarter sites typically serve knowledge workers, office workers, and executives. Within these large sites or even campus locations, PBX systems may integrate to co-located business systems serving specific business applications or added communications needs. These sites are different from branch locations. Consider in particular this scenario for retail sites and branch banks. The smaller sites may only need basic telephony for lightweight customer service and mobility applications.

Recommendation: These particular businesses may prefer private cloud for the larger sites or campus environments and public cloud for the branch locations. Larger sites can take advantage of PBX economics and legacy co-located integrations. The branch sites take advantage of multi-site services, reduced management costs, and enhanced mobility offers from public cloud services.

Site(s) with integrated call/contact center(s): In some cases, the PBX that services a large site serves two roles – providing both local calling for business telephony applications and also providing inbound and outbound for contact center employees. There may be certain integrations and efficiencies already achieved through using the same call control platform to support both applications. Deploying a PBX via this approach could be part of a broader effort to integrate employee calling and contact center services.

Recommendation: Sites with integrated contact centers might be targeted for a private cloud deployment while all other sites are targeted for public cloud. The business may want to look at vendors that bring a combination of contact center and enterprise telephony on a public cloud – though such a migration should be planned on its own timeline.

Differing regional cloud migration strategies: There are still some considerable differences in the maturity of cloud communications offers based on region. These differences apply for CSPs, PBX vendors, and IT tech integrators. These differences may also apply to how partners support private cloud solutions.

Recommendation: Businesses should complete a thorough assessment of the maturity of both private and public cloud offers across regional sites. A site-specific recommendation might be necessary along with an interim solution that combines private and public cloud communications across regions.

Communications Service Provider – Hybrid Solution Productization

Considering all instances where businesses may prefer a mix of public and private cloud, CSPs should consider how to “productize” such an offer with a mix of solutions. Such an offer may need to include “managed PBX” options where PBXs remain on site for a period of time during interim migration phases. CSPs should include state-of-the-art SIP trunking and network connectivity to support managed premises equipment sites along with public and private cloud service offerings.

Current forecasts show that businesses currently consume a mix of public and private cloud services along with a mix of SaaS, PaaS, IaaS and other IT solutions. Increasingly CSPs benefit through partnerships to offer a more complete portfolio of offers, enabling them to drive sales engagements. Note the current forecast below from a recent BAIN BRIEF publication which brings forward data from IDC, Gartner, and Forrester:

Figure 2: Bain.com

For the CSP who brings such a mix of managed equipment, private, and public cloud offers, they open the possibility of winning coveted large enterprise opportunities and forming deep partnerships with these businesses as they help them plan a prudent and transparent cloud migration.

Though businesses pay a premium for these types of migration offers, they can ultimately achieve 1) operating and management efficiencies through the consolidation of vendors and systems, 2) productivity benefits as they chart the path to greater enterprise-wide services engagement (see incremental cut-over plans) and 3) reduced risk of disruption to mission-critical communications and data services.

One of the most important things for CSPs in productizing a mix of cloud offers is around pricing. Studies have shown that lack of pricing transparency is one of the critical concerns for businesses that can delay or deter cloud migration. For vendors, offering clear pricing is critical to engaging with businesses’ IT leaders. An August 2018 BCG study suggests that over the next five years, over 90% of revenue growth in the hardware and software sectors will be coming from either on-premises or off-premises offerings sold with “cloud-like” pricing models. BCG goes on to state that re-thinking industry pricing models is “no longer optional.” Thus, the CSP that can position a combination of public and private cloud solutions, all delivered with cloud-like pricing, will likely see strong receptivity in the marketplace.

CSPs should prepare to encounter a variety of existing cloud solutions and “in process” cloud migrations as a “starting point” for preparing hybrid migration offers. This starting point might include an existing mix of premises systems, managed virtualized systems, and some public cloud solutions. The CSP should try to use a small number of relatively simple starting points to enable them to scale their offer. They should consider the Pareto principle (80:20 rule) where the majority of prospects can be targeted with a smaller subset of solutions. Defining these productized solutions may require the CSP to consider geographic or industry vertical characteristics. See below for the wide variety of cloud migration statuses reported in a 2016 McKinsey survey:

Figure 3: McKinsey & Company

This McKinsey chart shows that a CSP may want to consider a different mix of public and private cloud solution offers if the CSP is planning to target financial services vs. healthcare vs. insurance. Most important of all, the CSP needs to understand that they will struggle if they try to productize and especially scale a hybrid offer migration strategy as a one-size-fits-all approach.

In conclusion, there is a tremendous opportunity for both businesses and CSPs to benefit from a migration strategy that includes a mix of both private and public cloud solutions. To the extent that private and public cloud offers can work together can benefit both businesses and CSPs. In the most basic case, businesses and CSPs should look for combined solutions that can emulate an integrated or single cloud solution. Private and public cloud solutions can emulate a single cloud from a functional, economic, and/or delivery point of view – providing benefits in the form of added productivity, cost transparency, and migration project risk reduction.

Continue reading

Automating Your Network Operations, Part 2 – Data Models

Keeping your IT infrastructure operational

Before I get into data models, I want to take a slight diversion to incorporate some of the feedback that I received from the first blog. It was pointed out that my use of the ios_config module was “naïve.” I contend that it is more accurate to say that my use of Ansible in general was “naïve,” since this was a pretty straight forward use of the module. In any case, it was by design. Why? Well, if you are like the majority of the network operators that I’ve worked with over the years, you are not a programmer (or you are a “naïve” programmer). You spend 110% of your time busting your chops making sure that the network that underpins your company’s IT infrastructure is operational. I certainly do not want to discourage network operators going to classes, workshops, etc. to learn new skills if they have the time and the motivation, but it should not be necessary to consume network automation. We can do better!

The NTP example that I used in the first blog was also chosen for specific reasons. First, NTP and AAA were the questions that I got most from customers on this issue. They were able to configure these things initially, but ran into problems when they moved to an operational posture. Second, it is representative of the “naïve” approach that you’ll see in the examples and workshops that are being delivered in the Ansible network automation genre. These “naive” examples and workshops perpetuate the first reason. Lastly, there is no way to use the ios_config module, the most used Ansible module for Cisco IOS, to fix this. Nor is the issue addressed by an NTP-specific Ansible network module. Yes, there might be some other module in some git repo written by someone that addresses your corner case, but do you want to use it? Is writing a module for every single corner case for network automation a scalable approach? I say no. Partially because it is intractable, partially because it is a support nightmare.

Where do you get help for the modules that underpin the automation?

Red Hat supports a specific set of modules (generally the ones they write), other vendors support their modules, other modules are completely unsupported. Where do you get help for the modules that underpin the automation that makes your network work? Incidentally, we’ll cover a technique to augment the ios_config module with a parser to address this problem in a later blog, but it is not intuitively obvious to the casual observer.

I believe that we need to focus on a smaller number of more capable modules and couple that with a more sophisticated, or more realistic, approach to network automation. Subsequent blogs will focus on different aspects of this more realistic approach, but this one will start with the biggest: Data Models.

So, what is a data model?

I know what you are thinking: “Wait a minute! I didn’t need data models for automating my servers!” Well, building a system is a well-defined procedure with relatively few permutations or interdependencies on other systems. Also, provisioning a system generally consists of configuring values like hostname, IPs, DNS, AAA, and packages. Each of these are key/value pairs (e.g. nameservers = 8.8.8.8, 8.8.4.4) that define the operation of that system and there are relatively few of them for a system.

This is not the case for a network element. If we take a standard 48-port Top-of-Rack Switch, each port could have a description, a state, a VLAN, an MTU, etc. A single ToR could have hundreds of key/value pairs that dictate its operation. Multiply that across hundreds or even thousands of switches, and the number of key/value pairs grows rapidly. Collectively, all of these key/value pairs make up the Source of Truth (SoT) of your network and there can be a lot of them. In fact, automating networks is really more of a data management and manipulation problem than it is a programming problem.

So, what is a data model? Generally speaking, a data model is a structure in which the meaning of a key/value pair is defined by its relative position in that structure. As an example, let’s start with the de-facto standard in the networking space: YANG. According to RFC 7950, “YANG is a data modeling language used to model configuration data, state data, Remote Procedure Calls, and notifications for network management protocols.”

If we look at the way we setup a simple BGP peering on Cisco IOS and Juniper JunOS, we basically have a bunch of values accompanied by a bunch of words using a particular grammar that describe those values:

But the values, two ASNs and an IP address, are the only things that really matter and they are the same in each.

In fact, the switch hardware does not care about the words that describe those values since they get stored in a config DB anyway. The words are what the engineers gave to the humans to communicate the meaning of those values to the hardware. After all, we can’t just specify 2 ASNs because we need to know which is the local and which is the remote. We could, however, communicate their meaning by order: e.g. <Local ASN>, <Peer ASN>, <Peer IP>. This is basically a small data model. Well, BGP gets A LOT more complicated, so we’ll need a more capable data model. Here we have an example of the same data in the OpenConfig data model rendered in YAML:

The data in the model on the left contains the information needed to deliver either of the syntax specific versions… just add words. Yes, we still have words as tags in the model, but it normalizes those tags across vendors and gets rid of the grammar needed to specify how values relate to each other. We do not want to add words back if we can avoid it, so the next step is to encode all of this data into XML and shove it into the device via NETCONF.

NETCONF/YANG gives us programmability, but we still need automation since the two are not the same. This is where Ansible enters back into the story. In my opinion, it is the best of the open-source IaC tools for delivering data models to devices.

Continue reading

Cisco Threat Response with Email Security Integration: Harmonizing Your Security Products

Those of us who have been in security for more than 20 years are very familiar with the assertion that security is a process. For me, security has always been a process like a melody that ties in all other parts of the song.  Staying on this musical analogy, if process is the melody, and you consider Cisco’s security portfolio as different instruments, then Cisco Threat Response is leading in this beautiful orchestration of investigation. Threat Response focuses on the process aspect of security. In this blog post, I want to introduce you to its value as an incident response tool and show you how to best utilize it with the integration of the Cisco Email Security product.

What is Cisco Threat Response?

Just like how harmony is essential to a great piece of music, it’s equally a critical aspect of modern security architecture. Today’s Security Operations Centers (SOCs) are inundated with alerts to potential threats, all scattered across a discordant array of security products that don’t always like to play nice with each other. More often than not, this results in a lot of wasted labor and time. Think of Cisco Threat Response as a conductor that harmonizes the various components of your security infrastructure. And with Cisco’s open architecture, the Security portfolio of products works together like an orchestra.

Threat Response integrates threat intelligence from Cisco Talos and the various third party sources that make up your SOC to automatically research indicators of compromise (IOCs) and confirm threats quickly. By channeling threat data into a single user-friendly interface, your SOC team analysts can quickly aggregate alerts, investigate, and remediate threats lurking about your systems, network and cloud. No matter how big or small your organization may be, Threat Response is designed to scale, ensuring your cyber security programs run efficiently, effectively, and most of all harmoniously.

Cisco Email Security

Email is easily the most pervasive technology across businesses. It’s hard to imagine living without it. However, as integral as it is to day-to-day business functions, it’s also one of the most commonly exploited attack vectors. With over 90% of breaches starting via email, having an email security solution in place is no longer a luxury, but a necessity.

Cisco Email Security provides best in class efficacy and has been recognized for the third year in a row as the Top vendor in The Radicati Group’s 2018  Secure Email Gateway report. In addition, the integration of Email Security with Cisco Threat Response provides you with a more robust approach to email security. This layered defense provides industry-leading protection against malware, phishing, spoofing, ransomware and business email compromise (BEC).

Avoid Phishy Melodies

Most threat narratives have a very similar beginning. An attacker has phished their way into a network via an email and successfully socially engineered a user into disclosing their credentials. Unfortunately, this is usually followed by organizations discovering far too late that they’ve been compromised. Remember, however, that in order for the above scenario to play out, the attacker has to complete a series of phases of their attack without being detected. Luckily, as the defender, all you need to do is detect them in one phase! Even if something does get through, if you can detect that threat and respond with a countermeasure before the attacker completes that phase, you have won.

Cisco Threat Response brings your threat detection capabilities into simple focus with:

Simpler Integration: Let’s say you have a situation where AMP for Endpoints has detected a malicious file represented by a file hash (SHA) or Stealthwatch has alerted you of a user connecting to a suspicious URL. You can use Threat Response to pivot directly to Cisco’s Email Security and ask for the email messages associated with this SHA or URL and this information can be then used to stop that attack’s campaign dead in its tracks.

Simpler Data Tracking: Want to know where that malicious email or potential threat came from? Threat Response allows you to track and isolate which user received that SHA or URL and lets you block the domain of origin without needing to switch programs. Additionally, if said malicious link was shared amongst your network, you can also see when and where it was sent and prevent it from spreading further.

Simpler Workflow: Like I mentioned earlier, Cisco Threat Response is all about simplifying threat response through harmonizing the various tools you already have in place into a single resource. Furthermore, not only does this allow security analysts to reduce the number of interfaces they’re using, but Threat Response also provides them with context based graphs, telemetry charts, and even response suggestions.

Simpler Threat Hunting: As your analysts respond to threats, they can also log their threat response processes and observations into the integrated Casebook function. Since Casebooks are built on cloud API and data storage, they can also follow you from product to product, across your entire Cisco Security portfolio. In turn, this allows for faster and effective threat response and remediation efforts across your enterprise thanks to the ability for analysts to easily build and share Casebooks.

Continue reading

Automating Your Network Operations, Part 1 – Ansible Basics

I’ve spent the last couple of years at Red Hat helping customers automate their networks with Ansible. If there is one thing that I’ve learned during that time, it is that network automation is not as easy as many would have you believe. That is not to say that tools like Ansible are not good tools for automation or that anyone is trying to sell you snake oil, but I believe that there is a fundamental impedance mismatch in translating the success Ansible has had with automating systems to automate networks.

Part of this disconnect stems from a fundamental mis-understanding of the capabilities that Ansible provides. According to Red Hat, Ansible is a “common language to describe your infrastructure.” In practice, however, Ansible is more of a framework that brings an inventory of things together with a set of modules, plugins, and Jinja2 capabilities that perform operations on those things. The language, rendered in YAML or JSON, just passes key/value pairs between the modules, plugins, and Jinja2 capabilities. (Yes, that’s a simple description of a complex tool, but one that is accurate to illustrate the point of this and subsequent blogs.)

That is not to say that Ansible is not a powerful framework, but it has no native linguistic ability to describe a network. When I say an “inventory of things,” it is because Ansible really does not care what that thing is. Because of its agentless approach, it can talk to many things: systems, network devices, clouds, lightbulbs, etc. This is a great capability and part of why Ansible is so popular, but Ansible truly does not know one thing from another. It has no innate prowess for automating networks. It is simply a tool for automating what an operator does task by task. You cannot “describe” what you want OSPF to look like on your network. You simply provide a bunch of key/value pairs that get passed to the devices on your network through modules in hopes of yielding the OSPF configuration that you want.

Configuring settings on an IOS device

To illustrate this, let’s look at configuring two simple settings on an IOS device: hostname and NTP servers. Using Ansible parlance, we’ll describe the desired end state of the hostname of a particular device. Hostname is a great use case because it is a scalar (i.e. a single value). To change the hostname, the Ansible ios_config module does a simple textual compare of the configuration. If ‘hostname newname’ is not present, it sends that line to the device. Since hostname is a scaler, the old hostname gets replaced by the desired hostname.

A list of NTP servers, however, is more difficult. Say you’ve set the NTP server to 1.1.1.1 with:

- ios_config:
   lines:
    - ntp server 1.1.1.1

Now you want to change your NTP server to 2.2.2.2, so you do:

- ios_config:
   lines:
    - ntp server 2.2.2.2

Simple, right? But the problem is that you would end up with 2 NTP servers in the configuration:

ntp server 1.1.1.1
ntp server 2.2.2.2

This is because the Ansible ios_config module does not see `ntp server 2.2.2.2` present in the configuration, so it sends the line. Since ntp server is a list, however, it adds a new NTP server instead of replacing the existing one, giving you 2 NTP servers (one that you do not want). To end up with just 2.2.2.2 as your NTP server, you would have to know that 1.1.1.1 was already defined as an NTP server and explicitly remove it… exactly what an operator would do. This is also the case with ACLs, IP prefix-lists, and any other list in IOS. Ansible does not have a native way to describe the desired end state of something simple like NTP servers on a network device, much less something more complex like OSPF, QoS, or Multicast.

Does that mean that Ansible is not a great tool for network automation? No, but like any tool, it needs to be used for the right task and can only complete a complex task when used in concert with other tools. As a framework, it is not a complete solution.

The intent of this blog series is to go beyond the hype and simple demonstrations prevalent in network automation conversations today and to dive more deeply into how and why to automate your network operations. In the next installment, I’ll talk about data models and why they are a critical piece of any automation framework.

Continue reading