Secure Multi-Tenancy Part 2: Going Multi-Instance

Requirements Overview

In the previous blog post, we went over the common requirements for partitioning a single physical security appliance into multiple virtual firewalls. We talked about how this logical separation brings a lot of complexity into environments where true data and management plane isolation between different tenants is not required. It was further established that even the full isolation requirements are not truly addressed by the existing virtual firewall solutions. A single tenant can easily consume a disproportionally large amount of shared CPU and memory resources, thus impacting everyone else on the same physical appliance. As such, there was a clear need for a better solution to this problem.

A multi-tenancy solution for Cisco Firepower Threat Defense (FTD) had to overcome these constraints. The goal was to address the management simplification and routing separation requirements through different features. We wanted to concentrate specifically on management and traffic separation in a multi-tenant environment. Our virtual firewall instances would be completely isolated from each other in terms of CPU and memory resources, such that no individual tenant could exceed its allocation and impact someone else on the same Firepower appliance. This approach would extend to management sessions, where each tenant could use a separate Firepower Management Center (FMC) instance and push configuration changes completely independently. Last but not least, we wanted to eliminate the disparity in feature support when running virtual firewalls. If we support something within a single application, the same support should extend to a multi-tenant deployment going forward. These were very ambitious goals, but we managed to come up with an elegant solution.

Sizing Expectations

Before diving deeper into our solution, I want to say a few words about virtual firewall scalability. Traditional stateful firewall platforms support up to several hundreds of virtual contexts. However, this scale obviously comes with a large degree of resource sharing. If a security appliance is capable of delivering 50Gpbs of basic stateful firewalling, dividing it into 200 security contexts yields about 250Mbps of average throughput per tenant. This may be suitable for some environments, but then one should also consider packet-per-second (PPS) rates. Assuming a relatively powerful stateful firewall that does around 20 million PPS in the best case scenario, it comes down to only about 100 thousand PPS per each of the 200 tenants – a level easily exceeded by a single server in a modern data center.

As we start looking at more advanced firewall features, such as Intrusion Prevention Services (IPS), URL filtering, file and malware inspection, cloud sandboxing, and especially encrypted traffic inspection, performance implications become even more pronounced. There is frequently an order of magnitude of difference when comparing a threat-centric security application to a basic stateful firewall running on the same hardware. Getting a little over 20Mbps of threat-protected throughput per tenant is rarely acceptable, especially when migrating from a classic firewall feature set. If a tenant required 250Mbps of protected throughput before transitioning to a threat-centric product, their needs would not change simply because the firewall has to spend more cycles on much deeper inspection after the migration. As such, the expectations for tenant scale will be significantly reduced when migrating from Cisco ASA (Adaptive Security Appliance) and similar classic stateful firewalls to FTD.

Firepower Multi-Instance Capability

Firepower 4100 and 9300 appliances were meant to deliver multi-service security capabilities by design. The currently support ASA, FTD, and Radware Virtual DefensePro applications. When we looked at all of the possible multi-tenancy solutions for FTD, I immediately thought of extending the physical platform capabilities to host multiple instances of security applications on a single security module — this is how the multi-instance term was coined. Leveraging a common hypervisor for this did not seem very exciting, so a Docker container was picked as a form factor of choice. This approach leverages a proven application virtualization framework and enables future portability beyond the hardware appliances. Container-based FTD instances on Firepower 4100 and 9300 appliances would become available first, but we envision building a similar ASA package with mix-and-match capabilities in the future.

Given our desire to provide complete data plane and management plane separation, each FTD instance would get a completely independent CPU, memory, and disk reservation. Unequally sized instances can be deployed, and the firewall administrator gets to decide a CPU core allocation for each instance – memory and disk are sized automatically based on this assignment. This is important to ensure that a single FTD instance cannot impact any other instances running on the same module or appliance. Given a finite number of available CPU cores, it obviously puts a constraint on the maximum total number of instances that can reside on a particular Firepower appliance. As we had established earlier, a total tenant count with a threat-centric security application is significantly lower than with a basic stateful firewall on the same hardware platform. As such, the full resource separation requirement is more important to most customers than scaling to hundreds of oversubscribed virtual firewalls.

Each FTD container behaves like a separate firewall with its own software image. This means that individual instances can be upgraded, downgraded, or rebooted completely independently. One would no longer have to stand up a separate physical appliance to test software upgrades on a single tenant. Furthermore, each FTD instance would have dedicated management CPU cores to ensure no contention between different tenants during configuration deployment, event generation, and monitoring. An administrator can even assign different FTD containers on a single blade to be managed by different FMC appliances. Most importantly, each instance would support the same complete feature set as a full-module FTD application – no more exceptions for multi-tenancy.

In order to support the new multi-instance capability, Firepower 4100 and 9300 platforms would introduce several new network interface assignment models. Physical and Etherchannel interfaces can be shared between two or more instances or assigned exclusively to a single FTD container. Furthermore, one would gain an ability to create VLAN subinterfaces directly on the chassis Supervisor and assign them to instances on the same shared or unique basis. Needless to say, the instances would be able to communicate to each other directly on the shared data interfaces or VLAN subinterfaces – this includes supporting inter-instance multicast connectivity for dynamic routing. A management interface can be shared across multiple FTD containers as well, but inter-instance communication would be blocked in order to support the fully isolated model.

The following figure illustrates a hypothetical deployment where a single Firepower module runs a set of unequally sized ASA and FTD instances with a combination of shared and unique interfaces:

Looking Forward

The Firepower multi-instance capability definitely represents a unique and novel approach to deploying secure multi-tenancy. I am obviously very excited about this idea, and there are many new directions that it opens for us and our customers. As we are finalizing this feature for the public release, feel free to leave a comment about what additional details you would like me to cover in the next post on this topic.