Secure Multi-Tenancy Part 2: Going Multi-Instance

Requirements Overview

In the previous blog post, we went over the common requirements for partitioning a single physical security appliance into multiple virtual firewalls. We talked about how this logical separation brings a lot of complexity into environments where true data and management plane isolation between different tenants is not required. It was further established that even the full isolation requirements are not truly addressed by the existing virtual firewall solutions. A single tenant can easily consume a disproportionally large amount of shared CPU and memory resources, thus impacting everyone else on the same physical appliance. As such, there was a clear need for a better solution to this problem.

A multi-tenancy solution for Cisco Firepower Threat Defense (FTD) had to overcome these constraints. The goal was to address the management simplification and routing separation requirements through different features. We wanted to concentrate specifically on management and traffic separation in a multi-tenant environment. Our virtual firewall instances would be completely isolated from each other in terms of CPU and memory resources, such that no individual tenant could exceed its allocation and impact someone else on the same Firepower appliance. This approach would extend to management sessions, where each tenant could use a separate Firepower Management Center (FMC) instance and push configuration changes completely independently. Last but not least, we wanted to eliminate the disparity in feature support when running virtual firewalls. If we support something within a single application, the same support should extend to a multi-tenant deployment going forward. These were very ambitious goals, but we managed to come up with an elegant solution.

Sizing Expectations

Before diving deeper into our solution, I want to say a few words about virtual firewall scalability. Traditional stateful firewall platforms support up to several hundreds of virtual contexts. However, this scale obviously comes with a large degree of resource sharing. If a security appliance is capable of delivering 50Gpbs of basic stateful firewalling, dividing it into 200 security contexts yields about 250Mbps of average throughput per tenant. This may be suitable for some environments, but then one should also consider packet-per-second (PPS) rates. Assuming a relatively powerful stateful firewall that does around 20 million PPS in the best case scenario, it comes down to only about 100 thousand PPS per each of the 200 tenants – a level easily exceeded by a single server in a modern data center.

As we start looking at more advanced firewall features, such as Intrusion Prevention Services (IPS), URL filtering, file and malware inspection, cloud sandboxing, and especially encrypted traffic inspection, performance implications become even more pronounced. There is frequently an order of magnitude of difference when comparing a threat-centric security application to a basic stateful firewall running on the same hardware. Getting a little over 20Mbps of threat-protected throughput per tenant is rarely acceptable, especially when migrating from a classic firewall feature set. If a tenant required 250Mbps of protected throughput before transitioning to a threat-centric product, their needs would not change simply because the firewall has to spend more cycles on much deeper inspection after the migration. As such, the expectations for tenant scale will be significantly reduced when migrating from Cisco ASA (Adaptive Security Appliance) and similar classic stateful firewalls to FTD.

Firepower Multi-Instance Capability

Firepower 4100 and 9300 appliances were meant to deliver multi-service security capabilities by design. The currently support ASA, FTD, and Radware Virtual DefensePro applications. When we looked at all of the possible multi-tenancy solutions for FTD, I immediately thought of extending the physical platform capabilities to host multiple instances of security applications on a single security module — this is how the multi-instance term was coined. Leveraging a common hypervisor for this did not seem very exciting, so a Docker container was picked as a form factor of choice. This approach leverages a proven application virtualization framework and enables future portability beyond the hardware appliances. Container-based FTD instances on Firepower 4100 and 9300 appliances would become available first, but we envision building a similar ASA package with mix-and-match capabilities in the future.

Given our desire to provide complete data plane and management plane separation, each FTD instance would get a completely independent CPU, memory, and disk reservation. Unequally sized instances can be deployed, and the firewall administrator gets to decide a CPU core allocation for each instance – memory and disk are sized automatically based on this assignment. This is important to ensure that a single FTD instance cannot impact any other instances running on the same module or appliance. Given a finite number of available CPU cores, it obviously puts a constraint on the maximum total number of instances that can reside on a particular Firepower appliance. As we had established earlier, a total tenant count with a threat-centric security application is significantly lower than with a basic stateful firewall on the same hardware platform. As such, the full resource separation requirement is more important to most customers than scaling to hundreds of oversubscribed virtual firewalls.

Each FTD container behaves like a separate firewall with its own software image. This means that individual instances can be upgraded, downgraded, or rebooted completely independently. One would no longer have to stand up a separate physical appliance to test software upgrades on a single tenant. Furthermore, each FTD instance would have dedicated management CPU cores to ensure no contention between different tenants during configuration deployment, event generation, and monitoring. An administrator can even assign different FTD containers on a single blade to be managed by different FMC appliances. Most importantly, each instance would support the same complete feature set as a full-module FTD application – no more exceptions for multi-tenancy.

In order to support the new multi-instance capability, Firepower 4100 and 9300 platforms would introduce several new network interface assignment models. Physical and Etherchannel interfaces can be shared between two or more instances or assigned exclusively to a single FTD container. Furthermore, one would gain an ability to create VLAN subinterfaces directly on the chassis Supervisor and assign them to instances on the same shared or unique basis. Needless to say, the instances would be able to communicate to each other directly on the shared data interfaces or VLAN subinterfaces – this includes supporting inter-instance multicast connectivity for dynamic routing. A management interface can be shared across multiple FTD containers as well, but inter-instance communication would be blocked in order to support the fully isolated model.

The following figure illustrates a hypothetical deployment where a single Firepower module runs a set of unequally sized ASA and FTD instances with a combination of shared and unique interfaces:

Looking Forward

The Firepower multi-instance capability definitely represents a unique and novel approach to deploying secure multi-tenancy. I am obviously very excited about this idea, and there are many new directions that it opens for us and our customers. As we are finalizing this feature for the public release, feel free to leave a comment about what additional details you would like me to cover in the next post on this topic.

Continue reading

Security Multi-Tenancy Part 1: Defining the Problem

Pre-Virtual Virtual Firewalls

Nowadays, everyone likes to talk about network function virtualization. Most security vendors build their firewall products to run on a few popular hypervisors. However, the “virtual firewall” term predates this virtualization craze. Many firewall administrators use this nomenclature to describe an ability to create multiple virtual partitions or contexts within a single physical security appliance. Each of these virtual firewalls has its own configuration, stateful connection table, and management capabilities. However, they may not be as independent or isolated as one would assume – more on this later. Even though Cisco Adaptive Security Appliance (ASA) software supported virtual firewalls with multiple-context mode for quite some time, we deliberately delayed similar functionality in our threat-centric Firepower Threat Defense (FTD) product in order to get it right. As any decent engineer would tell you, getting the right solution starts with fully understanding the problem. Namely, why do our security customers deploy virtual firewalls?

Understanding Use Cases

As it turns out, not all customers deploy multiple security contexts specifically for multi-tenancy. Some look for routing table separation, where each virtual firewall represents a separate Virtual Routing and Forwarding (VRF) domain. This functionality comes in handy especially when trying to protect several internal organizations with overlapping IP spaces. Other firewall administrators leverage multiple-context mode to separate and simplify policy management across different domains. Instead of looking at a single flat policy, they break it up into smaller chunks based on individual network segments. This may also involve management separation, where administering individual security contexts is delegated to other organizations. A common example here is a big college where several departments manage their own networks and configure individual virtual firewalls on a shared physical appliance at the extranet edge. Other customers go even deeper and require complete traffic processing separation between different tenants or network segments. For instance, one typically does not want their production applications to be affected by some traffic from a lab environment. As these requirements add up, it becomes clear how most existing firewall multi-tenancy solutions come apart at the seams.

Reality Check

There are several operational considerations that need to be taken into account when deploying virtual firewalls.  All security contexts on a single appliance run the same software image, so you cannot test upgrades on a limited number of tenants. Similarly, they all live or die together – rebooting just one is not possible. When it comes to features, you need to keep track of which are not supported in the virtual firewall mode. Often enough, these subtle nuances come up when you are already so far down the implementation path that turning back is either expensive or completely impossible. But wait, there is more!

While virtual firewalls can certainly be used for routing or policy domain separation, it comes with a lot of unnecessary complexity. One needs to create firewall contexts, assign different interfaces, configure them all independently, and then keep switching back and forth in order to manage policies and other relevant configuration. If you need a single access policy across all of your contexts, it must be independently programmed into each virtual firewall. Luckily, features like VRF help in avoiding multiple-context mode by enabling routing domain separation only. When it comes to policy simplification, some of my customers found managing multiple virtual firewalls too cumbersome, converged back into a single security context, and leveraged Security Group Tags (SGTs) to significantly reduce the rule set. Unless you indeed require complete separation between tenants, it makes very little sense to deploy virtual firewalls.

When it comes to management separation, multiple-context mode seems like a perfect fit. After all, each tenant gets their own firewall to play with, all without impacting anyone else. Or is that really true? Even though each virtual firewall has its own independent configuration, they all run within a single security application on a shared physical device. In most implementations, it means that the management plane is shared across all of the virtual contexts. If one tenant is pushing a lot of policy changes or constantly polling for connection information, this will inevitably impact every other virtual firewall that runs on the same device. However, the real problem lies within the shared data plane.

Despite the perceived separation, all virtual firewalls ultimately run on shared CPU, memory, and internal backplane resources. Even when assigning different physical interfaces to different security contexts, all of the traffic typically converges at the ingress classification function in the CPU. While one sometimes can configure maximum routing or connection table sizes on per-context basis, it still does not limit the amount of network traffic or CPU resources that each particular tenant consumes. In order to classify packets to a particular virtual firewall, the system must spend CPU cycles on processing them first. If a particular tenant is getting a lot of traffic from the network, it can consume a disproportionally large amount of system CPU resources even if this traffic is later dropped by a rate-limiter. As such, there is never a guarantee that one virtual firewall does not grow too big and impact every other security context on the same box. I have seen many cases where firewall administrators were caught completely unaware by this simple caveat. Not being a problem with any specific vendor, this is just how most virtual firewalls are implemented today.

Thinking Outside the Contexts

After looking at the use cases and analyzing challenges with existing virtual firewall implementations, I knew that our approach to implementing multi-tenancy in FTD must fundamentally change. An ideal solution would provide complete management and traffic processing separation across all tenants, so one virtual firewall truly cannot impact anyone else on the same box. This separation should extend to independent software upgrades and reloads. At the same time, all of the available FTD features should always be supported when implementing virtual firewalls. Not only must it simplify the experience for an end user, but also significantly cut down on both development and testing times.

While these may have seemed like impossible requirements, I had a really cool idea on how we can get there for our customers. This novel approach builds on the multi-service capabilities of our Firepower platforms as well as such developing trends as application containerization.

Continue reading