Is Your Firewall Permitting and Denying the Correct Flows?

Two days prior, a large US city fell victim to a ransomware attack that disabled a sizable portion of the municipal network. I found myself on an airplane a few hours later.

Our first order of business was to quarantine potentially infected systems away from known-clean systems. In the interest of time, we installed a large Cisco ASA firewall into the datacenter distribution layer as a crude segmentation barrier. Once management connectivity was established, I was tasked with firewall administration, a job that is generally monotonous, thankless, and easy to scapegoat when things go wrong.

Long story short, the network was changing constantly. Applications were being moved between subnets, systems were being torn down and rebuilt, and I was regularly tasked with updating the firewall rules to permit or deny specific flows. After about 30 minutes, I decided to apply some basic automation. Managing firewall configurations using Python scripts is powerful but not particularly new or interesting, so I won’t focus on that aspect today.

Instead, consider the more difficult question. How do you know that your firewall is permitting and denying the correct flows? Often times, we measure the effectiveness of our firewall by whether the applications are working. This is a good measure of whether the firewall is permitting the desirable flows, but NOT a good measure of whether the firewall is denying the undesirable flows. Answering this question was critical to our incident response efforts at the customer site.

How would you solve this problem?

I decided to use the `packet-tracer` command built into the ASA command line. The command supports native `xml` formatting, which makes it easy to parse into Python data structures. Using Python parlance, the output contains a list of dictionaries, each one describing a phase in the ASA processing pipeline. Stages might include an ACL check, route lookup, NAT, QoS, and more. After the phase list, a summary dictionary indicates the final result.

ASA1#packet-tracer input inside tcp 10.0.0.1 50000 192.168.0.1 80 xml
<Phase>
  <id>1</id>
  <type>ROUTE-LOOKUP</type>
  <subtype>Resolve Egress Interface</subtype>
  <result>ALLOW</result>
  <config/>
  <extra>found next-hop 192.0.2.1 using egress ifc management</extra>
</Phase>
<Phase>
  <id>2</id>
  <type>ACCESS-LIST</type>
  <subtype/>
  <result>DROP</result>
  <config>Implicit Rule</config>
  <extra>deny all</extra>
</Phase>
<result>
  <input-interface>UNKNOWN</input-interface>
  <input-status>up</input-status>
  <input-line-status>up</input-line-status>
  <output-interface>UNKNOWN</output-interface>
  <output-status>up</output-status>
  <output-line-status>up</output-line-status>
  <action>DROP</action>
  <drop-reason>(acl-drop) Flow is denied by configured rule</drop-reason>
</result>

Now, how to automate this? I decided to use a combination of Python packages:

1. Nornir, a task execution framework with concurrency support
2. Netmiko, a library for accessing network device command lines

The high-level logic is simple. On a per firewall basis, define a list of `checks` which contain all the inputs for a `packet-tracer` test. Each check also specifies a `should` key which helps answers our original business question; should this flow be allowed or dropped? This allows us to test both positive and negative cases explicitly. Checks can be TCP, UDP, ICMP, or any arbitrary IP protocol. The checks can also be defined in YAML or JSON format for each host. Here’s an example of a `checks` list for a specific firewall:

---
checks:
  - id: "DNS OUTBOUND"
    in_intf: "inside"
    proto: "udp"
    src_ip: "192.0.2.2"
    src_port: 5000
    dst_ip: "8.8.8.8"
    dst_port: 53
    should: "allow"
  - id: "HTTPS OUTBOUND"
    in_intf: "inside"
    proto: "tcp"
    src_ip: "192.0.2.2"
    src_port: 5000
    dst_ip: "20.0.0.1"
    dst_port: 443
    should: "allow"
  - id: "SSH INBOUND"
    in_intf: "management"
    proto: "tcp"
    src_ip: "fc00:192:0:2::2"
    src_port: 5000
    dst_ip: "fc00:8:8:8::8"
    dst_port: 22
    should: "drop"
  - id: "PING OUTBOUND"
    in_intf: "inside"
    proto: "icmp"
    src_ip: "192.0.2.2"
    icmp_type: 8
    icmp_code: 0
    dst_ip: "8.8.8.8"
    should: "allow"
  - id: "L2TP OUTBOUND"
    in_intf: "inside"
    proto: 115
    src_ip: "192.0.2.1"
    dst_ip: "20.0.0.1"
    should: "drop"
...

Inside of a Nornir task, the code iterates over the list of checks, assembling the proper `packet-tracer` command from the check data, and issuing the command to the device using Netmiko. Note that this task runs concurrently on all firewalls in the Nornir inventory, making it a good fit for networks with distributed firewalls. Below is a simplified version of the Python code to illustrate the high-level logic.

def run_checks(task):
    # Iterate over all supplied checks
    for chk in checks:
        # Build the string command from check details (not shown)
        cmd = get_cmd(chk)
        # Use netmiko to send the command and collect output
        task.run(
            task=netmiko_send_command,
            command_string=cmd
        )

Behind the scenes, the code transforms this XML data returned by the ASA into Python objects. Here’s what that dictionary might look like. It contains two keys: `Phase` is a list of dictionaries representing each processing phase, and `result` is the final summarized result/

{
  "Phase": [
    {
      "id": 1,
      "type": "ROUTE-LOOKUP",
      "subtype": "Resolve Egress Interface",
      "result": "ALLOW",
      "config": None,
      "extra": "found next-hop 192.0.2.1 using egress ifc management"
    },
    {
      "id": 2,
      "type": "ACCESS-LIST",
      "subtype": None,
      "result": "DROP",
      "config": "Implicit Rule",
      "extra": "deny all"
    }
  ],
  "result": {
    "input-interface": "UNKNOWN",
    "input-status": "up",
    "input-line-status": "up",
    "output-interface": "UNKNOWN",
    "output-status": "up",
    "output-line-status": "up",
    "action": "DROP",
    "drop-reason": "(acl-drop) Flow is denied by configured rule"
  }
}

In the interest of brevity, I won’t cover the extensive unit/system tests, minor CLI arguments, and dryrun process in this blog. Just know that the script will automatically output three different files. I used the new “processor” feature of Nornir to build this output. Rather than traversing the Nornir result structure after the tasks have completed, processors are event-based and will run at user-specified points in time, such as when a task starts, a task ends, a subtask starts, a subtask ends, etc.

One of the output formats is terse, human-readable text which contains the name of the check and the result. If a check should be allowed and it was allowed, or if a check should be dropped and it was dropped, is it considered successful. Any other combination of expected versus actual results indicates failure. Other formats include comma-separated value (CSV) files and JSON dumps that provide even more information from the `packet-tracer` result. Here’s the `terse` format when executed on two ASAs with hostnames `ASAV1` and `ASAV2`:

ASAV1 DNS OUTBOUND -> FAIL
ASAV1 HTTPS OUTBOUND -> PASS
ASAV1 SSH INBOUND -> PASS
ASAV1 PING OUTBOUND -> PASS
ASAV1 L2TP OUTBOUND -> PASS
ASAV2 DNS OUTBOUND -> PASS
ASAV2 HTTPS OUTBOUND -> FAIL
ASAV2 SSH INBOUND -> PASS
ASAV2 PING OUTBOUND -> PASS
ASAV2 L2TP OUTBOUND -> PASS

For the visual learners, here’s a high-level diagram that summarizes the project’s architecture. After Nornir is initialized with the standard `hosts.yaml` and `groups.yaml` inventory files, the host-specific checks are loaded for each device. Then, Nornir uses Netmiko to iteratively issue each `packet-tracer` command to each device. The results are recorded in three different output files which aggregate the results for easy viewing and archival.

If you’d like to learn more about the project or deploy it in your environment, check it out here on the Cisco DevNet Code Exchange. As a final point, I’ve built Cisco FTD support into this tool as well, but it is experimental and needs more in-depth testing. Happy coding!