Curated Code Repos Get Your Integration Project Done Faster, Better

Having a hard time getting started on your next big integration with Cisco products? Found the platform and API docs on DevNet but need help turning this into running code? Check out Code Exchange, one more way DevNet makes it easy for developers to be successful with Cisco products and platforms.

Code Exchange is an online, curated set of code repositories that help you develop applications with/on Cisco platforms and APIs. Inside Code Exchange, you will find hundreds of code repositories – code created and maintained by Cisco engineering teams, ecosystem partners, technology and open source communities, and individual developers. Anyone can use this code to jumpstart their app development with Cisco platforms, products, application programming interfaces (APIs), and software development kits (SDKs).

Curated for quality

There is a large and growing amount of sample code and applications, helpful tools and libraries, and open source projects related to Cisco technologies on GitHub. However, finding up-to-date content best suited for your immediate needs can be difficult. Code Exchange helps you tackle this challenge.

To get things started, our team of DevNet Developer Advocates identified candidate repositories using GitHub crawlers and an algorithm that scores repositories based on a number of criteria. We then reviewed top repos to make sure they are in good shape and of general interest to the DevNet community. While we do not actively maintain all of the code, we provide confirmation that the code is a worthwhile investment of your time before accepting it into Code Exchange.

Simple filters for technology space and programming language may be used independently or in combination with keywords you provide to zero in on the set of repos most relevant to your immediate needs. Want more guidance? Sorting by those most recommended by DevNet Developer Avocates, or the date the repo was last updated, presents you with the best and brightest projects.

Key Features:

1. Curated view of code repositories related to all Cisco platforms

2. Easy discoverability using filters and search features

3. Link to repository on GitHub for direct access to code and contributors

For example, let’s say you are looking for some sample code written in Python for automation of Cisco IOS XR platforms using APIs defined by native and standard YANG models. Simply enter that in the Code Exchange search field and filters and back comes a set of highly relevant resources.

Or perhaps you’re looking for Javascript for an integration with Cisco’s collaboration platforms? Let Code Exchange do the heavy lifting.

Community contributions provide even more options

At present, the majority of the code in Code Exchange comes from GitHub organizations managed by employees at Cisco. These include some obvious ones, such as Cisco DevNet, Cisco, Cisco Systems, and Cisco SE, as well as others that are less obvious at first glance, such as Talos, IOS-XR, and Meraki.

That said, we realize, and very much appreciate, that a huge amount of very useful code for working with Cisco technologies exists throughout the community at large. We encourage and welcome contributions to Code Exchange from the entire DevNet community, including code in your personal GitHub account.

Follow these base requirements to prepare your GitHub repositories related to Cisco technologies:

1. Include a LICENSE in the repository

2. Add a clear README

3. Ensure repository is publicly available

4. Show evidence of repository being maintained

Then fill out the form and DevNet Developer Advocates will take a look!

How to make sure your repo is accepted

What better way to get your application, your company, your name out in the community working with Cisco products and APIs than to have your repo featured in Code Exchange? In addition to meeting the requirements enforced by the submission form, there are several things you can do to help us realize how great your code is and gladly accept it into Code Exchange.

Your README should provide new users with all the information they need to understand what your repo contains, including a getting started section with step by step instructions for how to install, run, and/or use it, and where to turn to get answers or to provide feedback. Your README will show best in Code Exchange if written in Markdown (i.e. README.md). We are in the process of adding support for reStructured Text (i.e. README.rst).

It is also highly recommended to include a CONTRIBUTING.md file that outlines how best to contribute back to the project by reporting issues, fixing bugs, adding new functionality, etc. Is it best to fork the project and send a pull request? Should an issue be opened first? What if I simply want to ask a question? Make it clear and easy for others to not only use your code but also help make it better.

Tips for enhancing the discoverability of your repos

At the time of project submission, you can identify the set of technologies to which your code is related. Identifying all and only those that truly apply is very helpful. Equally important is to add a meaningful description and GitHub topics. The search functionality of Code Exchange relies on these as well as first and second level headings in your README.

Continue reading

Cisco Intersight: AI-Driven IT Operations Strategy

Cisco launched our cloud-based platform for AI-driven IT operations (a.k.a. AI Ops), Cisco Intersight, last September. It already delivers significant benefits, and we intend to take it to the next level with artificial intelligence and machine learning.

A New Era in Operations Management

“Work smarter, not harder” is critical to improving IT efficiency. Organizations are adopting a multicloud strategy, so you need scalable and consistent management across data centers, private clouds, edge, and branch environments. Cisco Intersight delivers this consistent management, automation and policy enforcement across a variety of servers and hyperconverged infrastructure (HCI). It helps you work smarter by delivering proactive support and actionable intelligence through artificial intelligence (AI) and machine learning (ML), so that you can proactively manage complex environments and reduce risk.

Traditional IT operations management tools are deployed on-premise, they are vendor and device focused, difficult to maintain, and have limited ability to scale. We introduced a new era of systems management with Cisco Intersight. It provides the simplicity of software as a service (SaaS) with unlimited scalability. Intersight is enhanced by AI and ML to provide users with actionable intelligence.

Requirements for AI Operations

Gartner introduced the concept of AI Operations (AI Ops) a few years ago. AI Ops should not be confused with AI systems, like the Cisco UCS C480 ML M5 we announced earlier this month. This definition:

AIOps platforms utilize big data, modern machine learning and other advanced analytics technologies to directly and indirectly enhance IT operations (monitoring, automation and service desk) functions with proactive, personal and dynamic insight. 

A recent IDC report also provides more information regarding AI-driven systems management.

Our product management teams looks at the requirements for AI Ops from a practical perspective. We try to offer functionality that will provide actionable insights and automation to simplify and enhance day to day operations. Something that we ask ourselves everyday is, how can Intersight make daily routines easier and faster across operations groups and teams? In order to achieve these objectives, we believe AI Ops needs to provide the following:

◈ Open automation to streamline routine processes

◈ Consistent management services across a wide variety of infrastructure

◈ Ability to proactively recognize problems and assist users to respond quickly

Open and Unified AI Ops

If your going to make it easier to work across teams, you have to provide an open framework for AI Ops. One of our competitors claims to deliver “AI for the data center”. However, the vendor’s platform currently supports only their storage products. Their strategy is vendor focused. When you read about their roadmap, the vendor only plans to support their systems. Customers use systems from a variety of vendors, so their marketing does not align with reality.

An open and unified framework is foundational. That’s why we designed Cisco Intersight to provide an open framework for AI Ops and infrastructure as a service. Intersight supports Cisco UCS and HyperFlex systems today, and we are working with partners to provide support for third party systems.

One of the reasons many IT processes are manual is because tools used by different teams have separate data stores and user interfaces. When you can aggregate data from a wide range of servers, fabric, storage and hyperconverged infrastructure, you have a common repository that is necessary for effective analysis. There are currently hundreds of thousands of devices connected to Intersight. The vast amount of data we are collecting is used with AI and ML algorithms to identify potential problems and provide users with actionable intelligence. We have integrated Intersight with Cisco Technical Assistance Center (TAC) cognitive support, so we have can leverage their AI and ML capabilities as well as best practices.

The Four Benefits of AI Operations

As we continue to enhance Intersight and the Cisco management portfolio, our primary goal is to increase the customer benefits we can deliver through AI Ops. We have defined four categories of benefits:

1. Improved user experience

2. Proactive support and maintenance

3. Predictive operational analytics

4. Self-optimization of resources

Cisco is executing on our strategy to consistently enhance the customer benefits we deliver through AI Ops. We will be posting a series of blogs to explain how we are currently delivering the benefits in each of the four categories and our plans for the future.

Continue reading

Improve Office 365 Connectivity with Cisco SD-WAN

As more applications move to the cloud, the traditional approach of backhauling traffic over expensive WAN circuits to the data center or a centralized Internet gateway via a hub-and-spoke architecture is no longer relevant. Traditional WAN infrastructure was not designed for accessing applications in the cloud. It is expensive and introduces unnecessary latency that degrades the user experience. The scale-up effect of the centralized network egress model coupled with perimeter stacks optimized to handle conventional Internet browsing often pose bottlenecks and capacity ceilings, which can hinder or bring to a stall customer transition to the SaaS cloud.

As enterprises aggressively adopt SaaS applications such as Office 365, the legacy network architecture poses major problems related to complexity and user experience. In many cases, network administrators have minimal visibility into the network performance characteristics between the end user and software-as-a-service (SaaS) applications. ‘One size fits all’ approach focusing on perimeter security without application awareness, which legacy network architectures often have, do not allow enterprises to differentiate and optimize sanctioned and more trusted cloud business applications from recreational Internet use, resulting the former to be subject to expensive and intrusive security scanning further slowing down user experience.

Massive transformations are occurring in enterprise networking as network architects are reevaluating the design of their WANs to support a cloud transition, reduce network costs, increase visibility and manageability of their cloud traffic, while ensuring an excellent user experience. These architects are turning to software-defined WAN (SD-WAN) to take advantage of inexpensive broadband Internet services and to find ways to intelligently route trusted SaaS cloud bound traffic directly from remote branches. Cisco SD-WAN fabric is an industry-leading platform that delivers an elegant and simplified secure, end-to-end hybrid WAN solution that can facilitate policy based, local and direct connectivity from users to your trusted, mission critical SaaS applications, such as Office 365, straight from your branch office. Enterprises can use this fabric to build large-scale SD-WAN networks that have advanced routing, segmentation, and security capabilities with zero-touch bring-up, centralized orchestration, visibility and policy control. The result is a SaaS cloud-ready network that is easy to manage and more cost-efficient to operationalize and that empowers enterprises to deliver on their business objectives.

A fundamental tenet of the Cisco SD-WAN fabric is connecting users at the branch to applications in the cloud in a seamless, secure, and reliable fashion. Cisco delivers this comprehensive capability for SaaS applications with the Cloud onRamp for SaaS solution in alignment with Microsoft’s connectivity principles for Office 365.

With Cloud OnRamp for SaaS, the SD-WAN fabric continuously measures the performance of a designated SaaS application through all permissible paths from a branch and assign a score. This score gives network administrators visibility into application performance that has never before been available. Most importantly, the fabric automatically makes real-time decisions to choose the best-performing path between the end users at a remote branch and the cloud SaaS application. Enterprises have the flexibility to deploy this capability in multiple ways, according to their business needs and security requirements.

In some deployments, enterprises connect remote branches to the SD-WAN fabric using inexpensive broadband Internet circuits, and they want to apply differentiated security policies depending on the type of services users are connecting to.  For example, instead of sending all branch traffic to a secure web gateway (SWG) or cloud access security broker (CASB), an enterprise may wish to enforce their IT security policies in a targeted manner – by routing regular Internet traffic through SWG, while allowing performance optimal direct connectivity for a limited set of sanctioned and trusted SaaS applications, such as Office 365. In such scenarios, Cloud onRamp for SaaS can be set up to dynamically choose the optimal path among multiple ISPs for both applications permitted to go directly and for applications routable per enterprise policy through SWG.

To learn more about Cloud onRamp for Office 365, read our white paper. For more information about Cisco SD-WAN, click here.

If you’re attending Microsoft Ignite in Orlando next week, make sure to visit Cisco at booth #418. I’d love to show you how to improve your Office 365 connectivity and user experience using Cisco SD-WAN.

Continue reading

Updated IOS-XR Programmability Learning Labs and Sandbox Expand Your Options

A few weeks back I shared this blog post New XR Programmability Learning Labs and Sandbox introducing the new IOS-XR Learning Lab and a dedicated sandbox environment for IOS-XR programmability. This IOS-XR Programmability sandbox and learning labs provide an environment where developers and network engineers can explore the programmability options available in this routing platform.

So, great news for all you IOS-XR programmability fans, we are pleased to bring you even more great XR Programmability learning content. Here is the full list of content, broken down by module and learning labs.

Module One: CLI automation: IOS-XR CLI automation.

Show commands, config-apply, config-replace, and more using on-box bash scripts or remote bash commands

Cisco IOS-XR offers a comprehensive portfolio of APIs at every layer of the network stack, allowing users to leverage automated techniques to provision and manage the lifecycle of a network device. In this module, we start with the basics: the Command Line Interface (CLI) has been the interaction point for expect-style scripters (TCL, expect, pexpect etc.) for ages.  But these techniques rely on send/receive buffers, thus are prone to errors and inefficient code. This is where the new onbox ZTP libraries come in handy. Use them for automated device bring-up automate Day1 and Day2 behavior of the device through deterministic APIs and return values in a rich Linux environment on the router.

◈ IOS-XR CLI automation – Bash
◈ IOS-XR CLI automation – Python

Setting up a Telemetry Client/Collector with “Pipeline” is a flexible, multi-function collection service that is written in Go.

Module Two: IOS-XR Streaming Telemetry changes networking monitoring for the better

SNMP is dead. It’s time to move away from slow, polling techniques employed by SNMP for monitoring that are unable to meet the cadence or scale requirements associated with modern networks. Further, Automation is often misunderstood to be a one-way street of imperative (or higher-layer declarative) commands that help bring a network to an intended state. However, a core aspect of automation is the ability to monitor real-time state of a system during and post the automation process to accomplish a feedback loop that helps make your automation framework more robust and accurate across varied circumstances. In this module, we learn how Streaming Telemetry capabilities in IOS-XR are all set to change network monitoring for the better – allowing tools to subscribe to structured data, contractually obliged to the YANG models representing operational state of the IOS-XR internal database (SYSDB) at a cadence and scale that are orders of magnitude higher than SNMP.

◈ IOS-XR Streaming Telemetry: Monitoring done the right way

◈ Creating your first python Telemetry Collector

◈ Creating your first c++ Telemetry Collector

◈ Deploying a Telemetry Collector on-box

On-Box agents and custom protocols that co-exist with standard protocols to influence routing. Facebook’s Open/R protocol that behaves like an IGP but runs as a third-party application on the router.

Module Three: IOS-XR Service-Layer APIs, programming is exposed through the service layer API

Cisco IOS-XR offers a comprehensive portfolio of APIs at every layer of the network stack. For most automation use cases, the manageability layer that provides the CLI, YANG models and Streaming Telemetry capabilities, is adequate. However, over the last few years, we have seen a growing reliance in web-scale and large-scale Service Provider networks on off-box Controllers or on-box agents.  These extract away the state machine of a traditional protocol, or feature and marry their operation to the requirements of a specific set of applications on the network. These agents/controllers require highly performant access to the lowest layer of the network stack called the Service Layer and the model-driven APIs built at this layer are called the Service-Layer APIs. With the ability to interact with RIB, the Label Switch Database (LSD), BFD events, and interface events. And with more capabilities coming in the future, now is the time to take your automation chops to the next level.

◈ Service-Layer APIs: Bring your own Protocol/Controller

◈ Your first python service-layer API client

◈ Your first c++ service-layer API client

◈ Deploying a Service-layer API client on-box

Continue reading

Automated Policy & Segmentation Violation Alerting with Stealthwatch Cloud

Stealthwatch Cloud is best known for network behavioral anomaly detection and entity modeling, but the level network visibility value it provides far exceeds these two capabilities. The underlying traffic dataset provides an incredibly accurate recording for every network conversation that has transpired throughout your global network.  This includes traffic at remote locations and deep into the access layer that is far more pervasive than sensor-based solutions could provide visibility into.

Stealthwatch Cloud can perform policy and segmentation auditing in an automated set-it and forget-it fashion. This allows security staff to detect policy violations across firewalls, hardened segments and applications forbidden on user endpoints. I like to call this setting virtual “tripwires” all over your network, unbeknownst to users, by leveraging your entire network infrastructure as a giant security sensor grid. You cannot hide from the network…therefore you cannot hide from Stealthwatch Cloud.

Here is how we set this framework up and put it into action!

1. Navigate to Alerts from the main dashboard under the gear icon:

2. Click Configure Watchlists on the Settings screen:

3. Click Internal Connection Blacklist on the Watchlist Config screen:

4. Here are your options:

5. From here you’ll want to fill out the above form as such:

Name:  Whatever you’d like to call this rule, for example “Prohibited outbound RDP” or “permitted internal RDP”Source IP:  Source IP address or CIDR rangeSource Block Size:  CIDR notation block size, for example (0, 8, 16, 24, etc.)Source Ports:  Typically this is left blank as the source ports are usually random ephemeral ports but you have the option if you require a specific source port to trigger the alert.

Destination IP:  Target IP Address or CIDR range

Destination Block Size:  CIDR notation block size, for example (0, 8, 16, 24, etc.)

Destination Ports:  The target port traffic you wish to allow or disallow, for example (21, 3389, etc)

Connections are Allowed checkbox:  Check this if this is the traffic you’re going to permit.  This is used in conjunction with a second rule to specify all other traffic that’s not allowed.

Reason:  Enter a user friendly description of the intent for this rule.

6. Click Add to make the rule active.

7. Here’s an example of a set of rules both permitting and denying traffic on Remote Desktop over TCP 3389:

1. Permit rule:

2. Deny Rule:

8. Resulting Alert set:

9. Now to test this new ruleset, I will attempt two RDP connections within my Lab.  The first will be a lateral connection to another host on the 10.0.0.0/8 subnet and the second to an external IP residing on the public Internet.

10. Here is the resulting observation that triggered:

11. And the resulting Alert:

12. You can also see the observed ALLOWED traffic from my lateral RDP testing. This traffic did not trigger any observation or alert:

This policy violation alerting framework allows you to be fully accountable for all network prohibited network traffic that will inevitably transit your network laterally or through an egress point.  Firewall rules, hardening standards and compliance policies should be adhered to but how can you be certain that they are?  Human error, lack of expertise and troubleshooting can and will easily lead to a gap in your posture and Stealthwatch Cloud is the second line of defense to catch any violation the moment that first packet traverses a segment using a prohibited protocol.  It’s not a matter of IF your posture will be compromised but WHEN.

Continue reading

Secure Multi-Tenancy Part 2: Going Multi-Instance

Requirements Overview

In the previous blog post, we went over the common requirements for partitioning a single physical security appliance into multiple virtual firewalls. We talked about how this logical separation brings a lot of complexity into environments where true data and management plane isolation between different tenants is not required. It was further established that even the full isolation requirements are not truly addressed by the existing virtual firewall solutions. A single tenant can easily consume a disproportionally large amount of shared CPU and memory resources, thus impacting everyone else on the same physical appliance. As such, there was a clear need for a better solution to this problem.

A multi-tenancy solution for Cisco Firepower Threat Defense (FTD) had to overcome these constraints. The goal was to address the management simplification and routing separation requirements through different features. We wanted to concentrate specifically on management and traffic separation in a multi-tenant environment. Our virtual firewall instances would be completely isolated from each other in terms of CPU and memory resources, such that no individual tenant could exceed its allocation and impact someone else on the same Firepower appliance. This approach would extend to management sessions, where each tenant could use a separate Firepower Management Center (FMC) instance and push configuration changes completely independently. Last but not least, we wanted to eliminate the disparity in feature support when running virtual firewalls. If we support something within a single application, the same support should extend to a multi-tenant deployment going forward. These were very ambitious goals, but we managed to come up with an elegant solution.

Sizing Expectations

Before diving deeper into our solution, I want to say a few words about virtual firewall scalability. Traditional stateful firewall platforms support up to several hundreds of virtual contexts. However, this scale obviously comes with a large degree of resource sharing. If a security appliance is capable of delivering 50Gpbs of basic stateful firewalling, dividing it into 200 security contexts yields about 250Mbps of average throughput per tenant. This may be suitable for some environments, but then one should also consider packet-per-second (PPS) rates. Assuming a relatively powerful stateful firewall that does around 20 million PPS in the best case scenario, it comes down to only about 100 thousand PPS per each of the 200 tenants – a level easily exceeded by a single server in a modern data center.

As we start looking at more advanced firewall features, such as Intrusion Prevention Services (IPS), URL filtering, file and malware inspection, cloud sandboxing, and especially encrypted traffic inspection, performance implications become even more pronounced. There is frequently an order of magnitude of difference when comparing a threat-centric security application to a basic stateful firewall running on the same hardware. Getting a little over 20Mbps of threat-protected throughput per tenant is rarely acceptable, especially when migrating from a classic firewall feature set. If a tenant required 250Mbps of protected throughput before transitioning to a threat-centric product, their needs would not change simply because the firewall has to spend more cycles on much deeper inspection after the migration. As such, the expectations for tenant scale will be significantly reduced when migrating from Cisco ASA (Adaptive Security Appliance) and similar classic stateful firewalls to FTD.

Firepower Multi-Instance Capability

Firepower 4100 and 9300 appliances were meant to deliver multi-service security capabilities by design. The currently support ASA, FTD, and Radware Virtual DefensePro applications. When we looked at all of the possible multi-tenancy solutions for FTD, I immediately thought of extending the physical platform capabilities to host multiple instances of security applications on a single security module — this is how the multi-instance term was coined. Leveraging a common hypervisor for this did not seem very exciting, so a Docker container was picked as a form factor of choice. This approach leverages a proven application virtualization framework and enables future portability beyond the hardware appliances. Container-based FTD instances on Firepower 4100 and 9300 appliances would become available first, but we envision building a similar ASA package with mix-and-match capabilities in the future.

Given our desire to provide complete data plane and management plane separation, each FTD instance would get a completely independent CPU, memory, and disk reservation. Unequally sized instances can be deployed, and the firewall administrator gets to decide a CPU core allocation for each instance – memory and disk are sized automatically based on this assignment. This is important to ensure that a single FTD instance cannot impact any other instances running on the same module or appliance. Given a finite number of available CPU cores, it obviously puts a constraint on the maximum total number of instances that can reside on a particular Firepower appliance. As we had established earlier, a total tenant count with a threat-centric security application is significantly lower than with a basic stateful firewall on the same hardware platform. As such, the full resource separation requirement is more important to most customers than scaling to hundreds of oversubscribed virtual firewalls.

Each FTD container behaves like a separate firewall with its own software image. This means that individual instances can be upgraded, downgraded, or rebooted completely independently. One would no longer have to stand up a separate physical appliance to test software upgrades on a single tenant. Furthermore, each FTD instance would have dedicated management CPU cores to ensure no contention between different tenants during configuration deployment, event generation, and monitoring. An administrator can even assign different FTD containers on a single blade to be managed by different FMC appliances. Most importantly, each instance would support the same complete feature set as a full-module FTD application – no more exceptions for multi-tenancy.

In order to support the new multi-instance capability, Firepower 4100 and 9300 platforms would introduce several new network interface assignment models. Physical and Etherchannel interfaces can be shared between two or more instances or assigned exclusively to a single FTD container. Furthermore, one would gain an ability to create VLAN subinterfaces directly on the chassis Supervisor and assign them to instances on the same shared or unique basis. Needless to say, the instances would be able to communicate to each other directly on the shared data interfaces or VLAN subinterfaces – this includes supporting inter-instance multicast connectivity for dynamic routing. A management interface can be shared across multiple FTD containers as well, but inter-instance communication would be blocked in order to support the fully isolated model.

The following figure illustrates a hypothetical deployment where a single Firepower module runs a set of unequally sized ASA and FTD instances with a combination of shared and unique interfaces:

Looking Forward

The Firepower multi-instance capability definitely represents a unique and novel approach to deploying secure multi-tenancy. I am obviously very excited about this idea, and there are many new directions that it opens for us and our customers. As we are finalizing this feature for the public release, feel free to leave a comment about what additional details you would like me to cover in the next post on this topic.

Continue reading

Security Multi-Tenancy Part 1: Defining the Problem

Pre-Virtual Virtual Firewalls

Nowadays, everyone likes to talk about network function virtualization. Most security vendors build their firewall products to run on a few popular hypervisors. However, the “virtual firewall” term predates this virtualization craze. Many firewall administrators use this nomenclature to describe an ability to create multiple virtual partitions or contexts within a single physical security appliance. Each of these virtual firewalls has its own configuration, stateful connection table, and management capabilities. However, they may not be as independent or isolated as one would assume – more on this later. Even though Cisco Adaptive Security Appliance (ASA) software supported virtual firewalls with multiple-context mode for quite some time, we deliberately delayed similar functionality in our threat-centric Firepower Threat Defense (FTD) product in order to get it right. As any decent engineer would tell you, getting the right solution starts with fully understanding the problem. Namely, why do our security customers deploy virtual firewalls?

Understanding Use Cases

As it turns out, not all customers deploy multiple security contexts specifically for multi-tenancy. Some look for routing table separation, where each virtual firewall represents a separate Virtual Routing and Forwarding (VRF) domain. This functionality comes in handy especially when trying to protect several internal organizations with overlapping IP spaces. Other firewall administrators leverage multiple-context mode to separate and simplify policy management across different domains. Instead of looking at a single flat policy, they break it up into smaller chunks based on individual network segments. This may also involve management separation, where administering individual security contexts is delegated to other organizations. A common example here is a big college where several departments manage their own networks and configure individual virtual firewalls on a shared physical appliance at the extranet edge. Other customers go even deeper and require complete traffic processing separation between different tenants or network segments. For instance, one typically does not want their production applications to be affected by some traffic from a lab environment. As these requirements add up, it becomes clear how most existing firewall multi-tenancy solutions come apart at the seams.

Reality Check

There are several operational considerations that need to be taken into account when deploying virtual firewalls.  All security contexts on a single appliance run the same software image, so you cannot test upgrades on a limited number of tenants. Similarly, they all live or die together – rebooting just one is not possible. When it comes to features, you need to keep track of which are not supported in the virtual firewall mode. Often enough, these subtle nuances come up when you are already so far down the implementation path that turning back is either expensive or completely impossible. But wait, there is more!

While virtual firewalls can certainly be used for routing or policy domain separation, it comes with a lot of unnecessary complexity. One needs to create firewall contexts, assign different interfaces, configure them all independently, and then keep switching back and forth in order to manage policies and other relevant configuration. If you need a single access policy across all of your contexts, it must be independently programmed into each virtual firewall. Luckily, features like VRF help in avoiding multiple-context mode by enabling routing domain separation only. When it comes to policy simplification, some of my customers found managing multiple virtual firewalls too cumbersome, converged back into a single security context, and leveraged Security Group Tags (SGTs) to significantly reduce the rule set. Unless you indeed require complete separation between tenants, it makes very little sense to deploy virtual firewalls.

When it comes to management separation, multiple-context mode seems like a perfect fit. After all, each tenant gets their own firewall to play with, all without impacting anyone else. Or is that really true? Even though each virtual firewall has its own independent configuration, they all run within a single security application on a shared physical device. In most implementations, it means that the management plane is shared across all of the virtual contexts. If one tenant is pushing a lot of policy changes or constantly polling for connection information, this will inevitably impact every other virtual firewall that runs on the same device. However, the real problem lies within the shared data plane.

Despite the perceived separation, all virtual firewalls ultimately run on shared CPU, memory, and internal backplane resources. Even when assigning different physical interfaces to different security contexts, all of the traffic typically converges at the ingress classification function in the CPU. While one sometimes can configure maximum routing or connection table sizes on per-context basis, it still does not limit the amount of network traffic or CPU resources that each particular tenant consumes. In order to classify packets to a particular virtual firewall, the system must spend CPU cycles on processing them first. If a particular tenant is getting a lot of traffic from the network, it can consume a disproportionally large amount of system CPU resources even if this traffic is later dropped by a rate-limiter. As such, there is never a guarantee that one virtual firewall does not grow too big and impact every other security context on the same box. I have seen many cases where firewall administrators were caught completely unaware by this simple caveat. Not being a problem with any specific vendor, this is just how most virtual firewalls are implemented today.

Thinking Outside the Contexts

After looking at the use cases and analyzing challenges with existing virtual firewall implementations, I knew that our approach to implementing multi-tenancy in FTD must fundamentally change. An ideal solution would provide complete management and traffic processing separation across all tenants, so one virtual firewall truly cannot impact anyone else on the same box. This separation should extend to independent software upgrades and reloads. At the same time, all of the available FTD features should always be supported when implementing virtual firewalls. Not only must it simplify the experience for an end user, but also significantly cut down on both development and testing times.

While these may have seemed like impossible requirements, I had a really cool idea on how we can get there for our customers. This novel approach builds on the multi-service capabilities of our Firepower platforms as well as such developing trends as application containerization.

Continue reading