Network Automation with Cisco DNA Center SDK – Part 2

In Part 1 of this series we went through setting up the SDK; importing into your python project; and making your first call using the SDK to authenticate against an instance of the DevNet Sandbox;

Let’s roll up our sleeves and dig into the next part of utilizing the SDK. In this installment of the blog series, we will look at how simple it is to leverage python to programmatically run IOS commands throughout your entire infrastructure with DNAC’s command runner APIs. We will also assume you already have installed the SDK and understand how authentication works.

What is Command Runner?

Command Runner is a feature in Cisco DNA Center that allows you to execute a handful of read-only (for now) IOS commands on the devices managed by DNAC

Here is how you can get a list of all supported IOS commands.

commands = dnac.command_runner.get_all_keywords_of_clis_accepted()

dnac – Our connection object we created from Part 1 of the series

command_runner – Command runner class. Calling it will allow us to access the underlying methods

get_all_keywords_of_clis_accepted() – This is the method we are after to display a list of all supported keywords.

Makes sense? Now that we understand what Command Runner is, let’s dig into using the APIs to build a simple use case.

Command Runner flow

The use case we are about to build together is a simple configuration backup. In order to accomplish this task we will need to:

1. Retrieve a list of all managed devices

2. Execute a `show run` on each device using Command Runner APIs

3. Retrieve the results and save them to file

But before we do so, understanding Command Runner flow is prudent.

Cisco DNA Center API calls are asynchronous, which means for each executed task a Task id is created. Upon task completion content can be retrieved from /file endpoint.

Endpoints and methods used

◉ POST /dna/system/api/v1/auth/token

◉ GET /dna/intent/api/v1/network-device

◉ POST /dna/intent/api/v1/network-device-poller/cli/read-request

◉ GET /dna/intent/api/v1/task/{task_id}

◉ GET /dna/intent/api/v1/file/{file_Id}

This sounds too complex, right? Not really, with the help of our handy dandy SDK we are able to handle all of this very easily.

Let’s take it step by step

Authenticate

– Create a new connection object and assign it to a variable

dnac = DNACenterAPI(username=dnac_creds['username'], password=dnac_creds['password'], base_url=dnac_creds['url'])

Retrieve list of devices

– Using the devices class to call get_device_list() method to retrieve a list of all managed devices.

– Upon 200 OK loop through the list of Switches and Hubs and extract the device id, we need to leverage it to programmatically run the command on each device

– Access device id variable via device.id pass it to and call cmd_run() function

def get_device_list():

    devices = dnac.devices.get_device_list()

    devicesuid_list = []

    for device in devices.response:

        if device.family == 'Switches and Hubs':

            devicesuid_list.append(device.id)

    print("Device Management IP {} ".format(device.managementIpAddress))

    cmd_run(devicesuid_list)

Execute Command Runner

– As we iterate over each device, we will need to execute show run command. to do so use command_runner class and call read run_read_only_commands_on_devices() method. This method requires two inputs of type list: commands and deviceUuids

– Upon execution DNAC will return a taskId (asynchronous, remember?)

– Check its progress via task class by calling get_task_by_id() method. Once the task has been successfully executed (you can use the built-in error handling within the SDK to check but that’s for another blog post) grab the returned fileId

– Now simply access the file class and call the download_a_file_by_fileid() method et VOILA!

def cmd_run(device_list):

    for device in device_list:

        print("Executing Command on {}".format(device))

        run_cmd = dnac.command_runner.run_read_only_commands_on_devices(commands=["show run"],deviceUuids=[device])

        print("Task started! Task ID is {}".format(run_cmd.response.taskId))

        task_info = dnac.task.get_task_by_id(run_cmd.response.taskId)

        task_progress = task_info.response.progress

        print("Task Status : {}".format(task_progress))

        while task_progress == 'CLI Runner request creation':

            task_progress = dnac.task.get_task_by_id(run_cmd.response.taskId).response.progress

        task_progress= json.loads(task_progress)

        cmd_output = dnac.file.download_a_file_by_fileid(task_progress['fileId'])

        print("Saving config for device ... \n")

Congratulation!

That was a lot! Luckily the SDK handled a lot of the heavy lifting for us here. This is a great example of configuration management. You could use this as a base to start building out a simple configuration drift monitoring tool given that the config is returned as JSON data. We can easily use JSON query to check for any configuration drift and automatically rebase it to the original config. This can be taken a step further even by leveraging Git for version control of your device config.

Continue reading

Introduction to Programmability – Part 3

This is Part 3 of the “Introduction to Programmability” series. If you haven’t already done so, I strongly urge you to check out Parts 1 & 2 before you proceed. You will be missing on a lot of interesting information if you don’t.

Part 1 of this series defined and explained the terms Network Management, Automation, Orchestration, Data Modeling, Programmability and API. It also introduced the Programmability Stack and explained how an application at the top layer of the stack, wishing to consume an API exposed by a device at the bottom of the stack, does that.

Part 2 then introduced and contrasted two types of APIs: RPC-based APIs and RESTful APIs. It also introduced the NETCONF protocol, which is an RPC-based protocol/API, along with the (only) encoding that it supports and uses: XML.

Note: You will notice that I use the words API and protocol interchangeably. As mentioned in Part 2, a NETCONF API means that the client and server will use the NETCONF protocol to communicate together. The same applies to a RESTCONF API. Therefore, both NETCONF and RESTCONF may be labelled as protocols, or APIs.

In this part of the series, you will see the other type of APIs in action, namely RESTful APIs. You will see first how vanilla HTTP works. Then we will build on what we introduced in Part 2 and dig just a little deeper into REST. Then explain the relationship between HTTP, REST and RESTful APIs. I like to classify RESTful API into two types: Industry-standard and native (aka vendor/platform-specific). We will briefly cover RESTCONF, an industry-standard API as well as NX-API REST, a native API exposed by programmable Nexus switches. Finally, you will see how to consume a RESTful API using Python.

On a side note, you may be wondering how so much information will be covered in one blog post. Well, the challenge has always existed between depth and breadth with respect to topic coverage. In this series, I attempt to familiarize you with as many topics as possible and answer as many common questions related to programmability as feasible. The intention is not for you to come out of this 15-minute read an expert, but to be able to identify concepts and technologies that thus far have sounded foreign to you as a network engineer.

HTTP

As a network engineer, before I got into network programmability many many years ago, I knew that HTTP was the protocol on which the Internet was based. I knew, as required by my work, that HTTP was a client-server protocol that used TCP port 80 (and 443 in the case of HTTPS). I also knew it had something to do with the URIs I entered into my web browser to navigate to a web page. That was it.

But what really is HTTP ?

HTTP stands for HyperText Transfer Protocol. Hypertext is text that contains one or more hyperlinks. A hyperlink is a reference or pointer to data known as the resource or the target of the hyperlink. The text of the hyperlink itself is called the anchor text. That target may be a number of things such as a webpage on the Internet, a section in a Word document or a location on your local storage system.

A little piece of trivia: In 1965 an American scientist called Ted Nelson coined the term hypertext to describe non-linear text. Non-linear refers to a lack of hierarchy for the links between the documents. Then in 1989, Sir Timothy Berners-Lee, wrote the first web client and server implementation that utilized hypertext. That protocol would be used to fetch the data that a hyperlink pointed to and eventually became HTTP. Today, Sir Timothy is best known as the inventor of the World Wide Web.

Therefore, pressing on the anchor text https://blogs.cisco.com/developer/intro-to-programmability-2 will send a request to the blogs.cisco.com server to fetch the resource at /developer/intro-to-programmability-2, which is the HTML content of the webpage at that URI. This content will be parsed and rendered by the web browser and displayed in the browser window for you to view.

So an HTTP workflow involves a client establishing a TCP connection to an HTTP server. This connection is done over port 80 by default, but the port is usually configurable. Once the TCP session is up, the client sends a number of HTTP request messages. The server responds to each request message with a response message. Once the HTTP transactions are completed, the TCP session is torn down by either of the endpoints.

A client HTTP request message includes a Universal Resource Identifier (URI) that is a hierarchical address composed of segments separated by a slash (/). This URI identifies the resource on the server that the client is targeting with this request. In the realm of network programmability, the resource identified by a URI may be the interface configuration on a switch or the neighbors in the OSPF neighbor table on a router.

The client request message will also include an HTTP method that indicates what the client wishes to do with the resource targeted by the URI in the same request. An example of a method is GET which is used to retrieve the resource identified by target URI. For example, a GET request to the URI identifying the interface configuration of interface Loopback 100 will return the configuration on that interface. A POST method, on the other hand, is used to edit the data at the target URI. You would use the POST method to edit the configuration of interface Loopback 100.

In addition to the URI and method, an HTTP request includes a number of header fields whose values hold the metadata for the request. Header fields are used to attach information related to the HTTP connection, server, client, message and the data in the message body.

Figure 1 shows the anatomy of an HTTP request. At the top of the request is the start line composed of the HTTP method, URI and HTTP version. Then comes the headers section. Each header is a key-value pair, separated by a colon. Each header is on a separate line. In more technical terms, each header is delimited by a Carriage Return Line Feed (CRLF). The headers section is separated from the message body with an empty line (two CRLFs). In the figure, the message body is empty, since this is a GET request: the client is requesting a resource from the server, in this case, the operational data of interface GigabitEthernet2, so there is no data to send in the request, and hence, no message body.

Figure 1 – Anatomy of an HTTP request

When a server receives a request from the client, it processes the request, and sends back an HTTP response message. An HTTP response message will include a status code that indicates the result of processing the request, and a status text that describes the code. For example, the status code and text 200 OK indicate that the request was successfully processed, while the (notorious) code and text 404 Not Found indicate that the resource targeted by the client was not found on the server.

The format of a response message is very similar to a request, except that the start line is composed of the HTTP version, followed by a status code and text. Also, the body is usually not empty. Figure 2 shows the anatomy of an HTTP response message.

Figure 2 – Anatomy of an HTTP response message

Studying and hence understanding and using HTTP revolves around the following points:

– Target URI: You need to know the correct syntax rules of a URI, such as which characters are allowed and which are not, and what each segment of the URI should contain. URI segments are called scheme, authority, path, query and fragment. You also need to understand the correct semantics rules of a URI, that is, to be able to construct URIs to correctly target the resources that you want to operate on. URI syntax rules are universal. Semantics rules, on the other hand, depend on which protocol you are working with. In other words, a syntactically correct URI that targets a specific resource using RESTCONF will not be the same URI to target that same resource on that same device using another RESTful API, such as NX-API REST.

– Request method: You need to know the different request methods and understand the effect that each would have on a resource. GET fetches a resource (such as a web page or interface configuration) while POST edits a resource (such as add a record to a database, or change the IP address on a router interface). Commonly used methods are GET, HEAD, OPTIONS, POST, PATCH PUT and DELETE. The first three are used to retrieve a resource while the other four are used to edit, replace or delete a resource.

– Server status codes: Status codes returned by servers in their HTTP response messages are classified into the following sub-categories:

◉ 1xx: Informational messages to the client. The purpose of these response messages is to convey the current status of the connection or transaction in an interim response, before the final response is sent to the client.

◉ 2xx: The request was successfully processed by the server. Most common codes in this category are 200 (OK) and 201 (Created). The latter is used when a new resource is created by the server as a result of the request sent from the client.

◉ 3xx: Used to redirect the client, such as when the client requests a web page and the server attempts to redirect the client to a different web page (common use-case is when a web page owner changes the location of the web page and wishes to redirect clients attempting to browse to the old URI).

◉ 4xx: Signals that there is something wrong with the request received from the client. Common codes in this category are 401 (Bad Request), 403 (Forbidden), and 403 (Not Found).

◉ 5xx: Signals an error on the server side. Common status codes in this category are 500 (Internal Error), 503 (Service Unavailable), and 504 (Gateway Timeout).

– Message body: Understanding how to construct the message body. If model-driven programmability is used, the message body will depend on two things:

◉ Syntax rules governed by the encoding used: a message encoded in XML will have different syntax rules than a message encoded in JSON, even if both are intended to accomplish the same task

◉ Semantics rules governed by the data model used: You may target the same resource and accomplish the same end result using two (or more) different message bodies, each depending on the hierarchy of elements defined by the referenced data model.

– Headers: Understanding which headers to include in your request message is very important to get the results you want. For example, in Figure 1 the first header right after the start line Accept: application/yang-data+json is the client’s way of telling the server (the DevNet IOS-XE router/sandbox in this case) that it will only accept the requested resource (the interface operational data) encoded in JSON. If this field was application/yang-data+xml, the server’s response body would have been encoded in XML instead. Header values in response messages also provide valuable information related to the server on which the resource resides (called origin server), any cache servers in the path, the resources returned, as well as information that will assist to troubleshoot error conditions in case the transaction did not go as intended.

HTTP started off at version 0.9, then version 1.0. The current version is 1.1 and is referred as HTTP/1.1. Most of HTTP/1.1 is defined in the six RFCs 7230 – 7235, each RFC covering a different functional part of the protocol.

HTTP/2 is also in use today, however, RFC 7540 states that “This specification [HTTP/2.0] is an alternative to, but does not obsolete, the HTTP/1.1 message syntax. HTTP’s existing semantics remain unchanged.” This means that HTTP/2.0 does not change the message format of HTTP/1.1. It simply introduces some enhancements to HTTP/1.1. Therefore, everything you have read so far in this blog post remains valid for HTTP/2.

HTTP/2 is based on a protocol called SPDY developed Google. HTTP/2 introduces a new framing format that breaks up an HTTP message into a stream of frames and allows multiplexing frames from different streams on the same TCP connection. This, along with several other enhancements and features promise a far superior performance over HTTP/1.1. The gRPC protocol is based on HTTP/2.

It may come as a surprise to some, but HTTP/3 is also under active development, however, it is not based on TCP altogether. HTTP/3 is based on another protocol called QUIC initially developed by, as you may have guessed, Google, then later adopted by the IETF and described in draft-ietf-quic-transport. HTTP/3 takes performance to whole new level. However, HTTP/3 is still in its infancy.

HTTP uses the Authorization, WWW-Authenticate, Proxy-Authorization and Proxy-Authenticate headers for authentication. However, in order to provide data confidentiality and integrity, HTTP is coupled with Transport Layer Security (TLS 1.3). HTTP over TLS is called HTTPS for HTTP Secure.

But what does HTTP have to do with REST and RESTful APIs ?

As you have read in Part 2 of this series, REST is a framework for developing APIs. It lays down 6 constraints, 5 mandatory and 1 optional. As a reminder, here are the constraints:

◉ Client-Server based

◉ Stateless

◉ Cacheable

◉ Have a uniform interface

◉ Based on a layered system

◉ Utilize code-on-demand (Optional)

In a nutshell, HTTP is the protocol that is leveraged to implement an API that complies with these constraints. But again, what does all this mean?

As you already know by now, HTTP is a client-server protocol. That’s the first REST constraint.

HTTP is a stateless protocol, as required by the second constraint, because when a server sends back a response to a client request, the transaction is completed and no state information pertaining to this specific transaction is maintained on the server. Any single client request contains all the information required to fully understand and process this request, independent of any previous requests.

Ever heard of cache servers ? An HTTP resource may be cached at intermediate cache servers along the path between the client and server if labeled as cacheable by the sending endpoint. Moreover, HTTP defines a number of header fields to support this functionality. Therefore, the third REST constraint is satisfied.

HTTP actually does not deal with resources, but rather with representations of these resources. Data retrieved from a server may be encoded in JSON or XML. Each of these is a different representation of the resource. A client may send a POST request message to edit the configuration of an interface on a router, and in the process, communicates a desired state for a resource, which, in this case, is the interface configuration. Therefore, a representation is used to express a past, current or desired state of a resource in a format that can be transported by HTTP, such as JSON, XML or YAML. This is actually where the name REpresentational State Transfer (REST) comes from.

The concept of representations takes us directly to the fourth constraint: regardless of the type of resource or the characteristics of the resource representation expressed in a message, HTTP provides the same interface to all resources. HTTP adheres to the fourth constraint by providing a uniform interface for clients to address resources on servers.

The fifth constraint dictates that a system leveraging RESTful APIs should be able to support a layered architecture. A layered architecture segregates the functional components into a number of hierarchical layers, where each layer is only aware of the existence of the adjacent layers and communicates only with those adjacent layers. For example, a client may interact with a proxy server, not the actual HTTP server, while not being aware of this fact. On the other end of the connection, a server processing and responding to client requests in the frontend may rely on an authentication server to authenticate those clients.

The final constraint, which is an optional constraint, is support for Code on Demand (CoD). CoD is the capability of downloading software from the server to the client, to be executed by the client, such as Java applets or JavaScript code downloaded from a web site and run by the client web browser.

Therefore, by providing appropriate, REST-compliant transport to a protocol in order to expose an API to the external world, HTTP makes that protocol or API RESTful.

Are you still wondering what is HTTP, REST and RESTful APIs ?

JSON – JavaScript Object Notation

Similar to XML, JSON is used to encode the data in the body of HTTP messages. As a matter of fact, the supported encoding is decided by the protocol used, not by HTTP. NETCONF only supports XML while RESTCONF supports both XML and JSON. Other APIs may only support JSON. Since XML was covered in Part 2 of this series, we will cover JSON in this part.

Unlike XML, that was developed to be primarily machine-readable, JSON was developed to be a human-friendly form of encoding. JSON is standardized in RFC 8259. JSON is much simpler than XML and is based on four simple rules:

1. Represent your objects as key-value pairs where the key and value are separated with a colon

2. Enclose each object in curly braces

3. Enclose arrays in square brackets (more on arrays in minute)

4. Separate objects or array values with commas

Let’s start with a very simple example – an IP address:

{“ip”: “10.20.30.40”}

The object here is enclosed in curly braces as per rule #2. The key (ip) and value (10.20.30.40) are separated by a colon as per rule #1. Keep in mind that the key must be a string and therefore will always be enclosed in double quotes. The value is also a string in the example since it is enclosed in double quotes. Generally, a value may be any of the following types:

◉ String: such as “Khaled” – always enclosed in double quotes

◉ Number: A positive, negative, fraction, or exponential number, not enclosed in quotes

◉ Another JSON object: shown in the next example

◉ Array: An ordered list of values (of any type) such as [“Khaled”,“Mohammed”,“Abuelenain”]

◉ Boolean: true or false

◉ null: single value of null

A very interesting visual description of value types is given here: https://www.json.org/.

Now assume that there is an object named address that has two child JSON objects, ip and netmask. That will be represented as follows:

{

  "address": {

    "ip": "100.100.100.100",

    "netmask": "255.255.255.255"

  }

}

Notice that the objects ip and netmask are separated by a comma as per rule #4.

What if the address object needs to hold more than one IP address (primary and secondary) ? Then it can be represented as follows:

{

  "address": [

    {

      "ip": "100.100.100.100",

      "netmask": "255.255.255.255"

    },

    {

      "ip": "200.200.200.200",

      "netmask": "255.255.255.255"

    }

  ]

}

In this example, address is a JSON object whose value is an array, therefore, everything after the colon following the key is enclosed in square brackets. This array has two values, each an JSON object in itself. So this is an array of objects. Notice that the in addition to the comma separating the ip and netmask objects inside each object, there is also a comma after the closing curly brace around the middle of the example. This comma separates the two values of the array.

And that’s about all you need to know about JSON !

Standards-based vs. Native RESTful APIs: RESTCONF & NX-API REST

As you have seen in the previous section, any RESTful protocol/API employing HTTP at the Transport Layer (of the programmability stack – NOT the OSI 7-layer model) will need to define three things:

1. What encoding(s) does it supports (XML, JSON, YAML, others) ?

2. How to construct a URI to target a specific resource ? A URI is a hierarchical way of addressing resources, and in its absolute form, a URI will uniquely identify a specific resource. Each protocol will define a different URI hierarchy to achieve that.

3. Which data models are supported and, combined with point number 1 above, will decide what the message body will look like.

RESTCONF is a standards-based RESTful API defined in RFC 8040. RESTCONF is compatible with NETCONF and is sometimes referred to as the RESTful version of NETCONF. This means that they can both coexist on the same platform without conflict. Although RESTCONF supports a single “conceptual” datastore, there are a set of rules that govern the interaction of RESTCONF with NETCONF with respect to datastores and configuration. While NETCONF support XML only, RESTCONF supports both XML and JSON. RESTCONF supports the same YANG data models supported by NETCONF. Therefore, a message body in RESTCONF will be model-based just as you have seen with NETCONF, with a few caveats. However, RESTCONF only implements a subset of the functions of NETCONF.

The architectural components of RESTCONF can be summarized by the 4-layer model in Figure 3. The 4 layers are Transport, Messages, Operations and Content. Just like NETCONF.

Figure 3 – The RESTONF architectural 4-Layer model

Now to the RESTful part of RESTCONF. RESTCONF supports all HTTP methods discussed so far. The key to understanding RESTCONF then is to understand how to construct a URI to target a resource. While it is out of scope of this (very) brief introductory post to get into the fine details of the protocol, it is important to get at least a glimpse of RESTCONF URI construction, as it is the single most important factor differentiating the protocol right after its compatibility with NETCONF. The resource hierarchy for RESTCONF is illustrated in Figure 4.

Figure 4 – Resource hierarchy in RESTCONF

The branch of this hierarchy that relates to configuration management and datastores is

API -> Datastore -> Data. A URI in RESTCONF has the general format of

https://device_address:port/API_Resource/Datastore_Resource/Resource-Path

Without getting into too much details, the Cisco implementation of RESTCONF uses the string restconf as the value of the API Resource and the string data as the value of the Datastore Resource. So on the DevNet IOS-XE Sandbox, for example, all RESTCONF URIs will start with https://sandbox-iosxe-latest-1.cisco.com:443/restconf/data/. In the next section you see how to configure a loopback address using a RESTCONF URI and a YANG data model.

Now on the other side of the spectrum are native RESTful APIs. Native RESTful APIs are vendor-specific and are usually platform specific as well. On example of a RESTful API that is widely used by the programmability community is NX-API REST that is exposed by programmable Nexus switches. NX-API REST is a RESTful API that uses HTTP request and response messages composed of methods, URIs, data models and status codes, like all other RESTful APIs. However, this API uses a Cisco-specific data model called the Management Information Tree (MIT). The MIT is composed of Managed Objects (MO). Each MO represents a feature or element on the switch that can be uniquely targeted by a URI.

When the switch receives an HTTP request to an NX-API REST URI, an internal Data Management Enginer (DME) running on the switch validates the URI, substitutes missing values with default values, where applicable, and, if the client is authorized to perform the method stated in the client request, the MIT is updated accordingly.

Similar to RESTCONF, NX-API REST supports payload bodies in both XML and JSON.

RESTful APIs and Python

The requests package has been developed to abstract the implementation of an HTTP client using Python. The Python Software Foundation recommends using the requests package whenever a “higher-level” HTTP client-interface is needed (https://docs.python.org/3/library/urllib.request.html).

The requests package is not part of the standard Python library, therefore it has to be manually installed using pip. Example 1 shows the installation of requests using pip3.7.

Example 1 Installing the requests package using pip

[kabuelenain@server1 ~]$ sudo pip3.7 install requests

Collecting requests

  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl

Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests) (2.7)

Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests) (2018.10.15)

Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests) (3.0.4)

Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests) (1.24.1)

Installing collected packages: requests

Successfully installed requests-2.22.0

[kabuelenain@server1 ~]$

After the requests package is installed, you are ready to import it into your code. Using the requests package is primarily based on creating a response object, and then extracting the required information from that response object.

The simplified syntax for creating a response object is:

Response_Object = requests.method(uri, headers=headers, data=message_body)

To create an object for a GET request, in place of requests.method, use requests.get. For a POST request, use requests.post, and so forth.

Replace the uri parameter in the syntax with the target URI. The headers parameter will hold the headers and the data parameter will hold the request message body. The uri should be a string, the headers parameter should be a dictionary and the data parameter may be provided as a dictionary, string, or list. The parameter data=payload may be replaced by json=payload, in which case the payload will be encoded into JSON automatically.

Some of the information that you can extract from the Response Object is:

◉ Response_Object.content: The response message body (data) from the server as a byte object (not decoded).

◉ Response_Object.text: The decoded response message body from the server. The encoding is chosen automatically based on an “educated guess”.

◉ Response_Object.encoding: The encoding used to convert Response_Object.content to ◉ Response_Object.text. You can manually set this to a specific encoding of your choice.

◉ Response_Object.json(): The decoded response message body (data) from the server encoded in json, if the response resembles a json object (otherwise an error is returned).

◉ Response_Object.url: The full (absolute) target uri used in the request.

◉ Response_Object.status_code: The response status code.

◉ Response_Object.request.headers: The request headers.

◉ Response_Object.headers: The response headers.

In Example 2, a POST request is sent to the DevNet IOS-XE Sandbox to configure interface Loopback123. Looking at the URI used, you can guess that the Python script is consuming the RESTCONF API exposed by the router. Also, from the URI as well as the message body, it is evident that the YANG model used in this example is ietf-interfaces.yang (available at https://github.com/YangModels/yang/tree/master/vendor/cisco/xe/1731).

Example 2 POST request using the requests package to configure interface Loopback123

#!/usr/bin/env python3

import requests

url = 'https://sandbox-iosxe-latest-1.cisco.com:443/restconf/data/ietf-interfaces:interfaces/'

headers = {'Content-Type': 'application/yang-data+json',

    'Authorization': 'Basic ZGV2ZWxvcGVyOkMxc2NvMTIzNDU='}

payload = '''

{

  "interface": {

    "name": "Loopback123",

    "description": "Creating a Loopback interface using Python",

    "type": "iana-if-type:softwareLoopback",

    "enabled": true,

    "ietf-ip:ipv4": {

      "address": {

        "ip": "10.0.0.123",

        "netmask": "255.255.255.255"

      }

    }

  }

}

'''

Response_Object = requests.post(url,headers=headers,data=payload,verify=False)

print('The server response (data) as a byte object: ','\n\n',Response_Object.content,'\n')

print('The decoded server response (data) from the server: ','\n\n',Response_Object.text,'\n')

print('The encoding used to convert Response_Object.content to Response_Object.text: ','\n\n', Response_Object.encoding,'\n')

print('The full (absolute) URI used in the request: ','\n\n',Response_Object.url,'\n')

print('The response status code: ','\n\n',Response_Object.status_code,'\n')

print('The request headers: ','\n\n',Response_Object.request.headers,'\n')

print('The response headers :','\n\n',Response_Object.headers,'\n')

Example 3 shows the result from running the previous script.

Example 7-3 Running the script and creating interface Loopback123

[kabuelenain@server1 Python-Scripts]$ ./int-loopback-create.py

/usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

  InsecureRequestWarning)

The server response (data) as a byte object: 

 b''

The decoded server response (data) from the server: 

The encoding used to convert Response_Object.content to Response_Object.text: 

 ISO-8859-1

The full (absolute) URI used in the request: 

 https://sandbox-iosxe-latest-1.cisco.com:443//restconf/data/ietf-interfaces:interfaces/

The response status code: 

 201

The request headers: 

 {'User-Agent': 'python-requests/2.20.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/yang-data+json', 'Authorization': 'Basic ZGV2ZWxvcGVyOkMxc2NvMTIzNDU=', 'Content-Length': '364'}

The response headers :

 {'Server': 'nginx/1.13.12', 'Date': 'Fri, 13 Nov 2020 11:00:28 GMT', 'Content-Type': 'text/html', 'Content-Length': '0', 'Connection': 'keep-alive', 'Location': 'https://sandbox-iosxe-latest-1.cisco.com/restconf/data/ietf-interfaces:interfaces/interface=Loopback123', 'Last-Modified': 'Fri, 13 Nov 2020 11:00:14 GMT', 'Cache-Control': 'private, no-cache, must-revalidate, proxy-revalidate', 'Etag': '"1605-265214-914179"', 'Pragma': 'no-cache'}

[kabuelenain@server1 Python-Scripts]$

As you can see, the status code returned in the server response message is 201 (Created) which means that the Loopback interface was successfully created. You may have noticed that the message body (actual data in the message) is empty, since there is nothing to return back to the client. However, the Location header in the response headers (highlighted in the example) returns a new URI that points to the newly created resource.

Continue reading

Introduction to Programmability – Part 1

Are you a network engineer and have had to repeat the same boring task at work, every day? Do you feel that there must be a way for you to do a task once, and then “automate” it? Theoretically, an infinite number of times? Or, have you been spending more time cleaning up and correcting configuration mistakes than you spend implementing those configurations? Or maybe you have been hearing a lot about this hot new “thing” called network programmability, but in the middle of the hype, could not figure out what exactly it is?

If any of those cases (and many others) apply to you, then you are in the right place. The fact that you are here, reading this now, means you know that there is probably a solution to your problem(s) in the realm of automation and/or programmability. In this case, buckle up because you are in for a ride!

If you are a network engineer and browsed to this page by mistake, I still urge you to read on. Netflix, Youtube, Facebook and Twitter will still be there when you are done. (Or not.) This is more fun. Trust me!

A Few Definitions For The Road

Before we dive into the nuances of network programmability and automation, let’s clear up some confusion. I hate nothing in the world more than definitions – well, maybe greasy pizza – but this is a necessary evil! In order to start clean, you must understand each of the following: network management, automation, orchestration, modeling, programmability and APIs.

Network Management is an umbrella term that covers the processes, tools, technologies, and job roles, among other things, required to manage a network and the lifecycle of the services offered by that network.

Many standards and frameworks exist today to define the different components of network management. One of them is FCAPS, where the acronym stands for Fault, Configuration, Accounting, Performance and Security Management. FCAPS is geared towards managing the systems that constitute the network.

Another is ITIL. The acronym stands for Information Technology Infrastructure Library and covers an extensive number of practices for IT Services Management (ITSM), which is basically the lifecycle of the services provided by the network. ITIL is divided into 5 major practices: Service Strategy, Service Design, Service Transition, Service Operation, and Continual Service Improvement. Each practice is divided into smaller sub-practices. For example, Service Design includes Capacity Management, Availability Management and Service Catalogue Management while Service Operation includes Incident Management, Problem Management and Request Fulfilment. Some people make a career being ITIL practitioners.

The Merriam-Webster dictionary defines Automation as “the technique of making an apparatus, a process, or a system operate automatically”. In other words, having a system of some sort do the work for you, work that you would otherwise do manually. However, you will have to tell this system what is it that you want to get done, and sometimes, how to do it.

So, configuring a network of routers with dynamic routing protocols, so that these routers speak with each other and figure out the shortest path per destination, is a form of automation. The alternative would be having someone do the calculations manually on a piece of paper, and then configuring static routes on each router. And so is writing a program that configures a VLAN on your switch – or your 500 switches – without someone having to log in to each switch individually and configuring the VLAN via the CLI.

As you have already guessed, the power of automation is not intrinsically in the automation itself. Logging into one switch manually and configuring one VLAN is probably much faster than writing a Python program to do that for you. So why automate? Obviously, the importance of automation is its application to repetitive tasks.

Automation will not only save the time you will spend repeating a task, it also maintains consistency and accuracy of performing that task, over all its iterations. It does not matter if you have 10 or 500 switches. The program you wrote will always go through the exact same steps for eachevery switch, with the exact same result. Every time. Of course, the assumption here is that no errors will happen because of factors external to your program, such as an unreachable switch, wrong credentials configured on a switch, or a switch with a corrupted IOS. Although, you can write a program to detect and mitigate these error conditions!

When you have several systems working together to get a job done, there is typically a need for a system, or a function, to coordinate the execution of the tasks performed by the different systems towards getting this job completed. This coordination function is called Orchestration.

For example, a private or public cloud that provides virtual machines to its users will include different systems to provision the network, compute, virtualization, operating systems, and maybe the applications, for those VMs. Orchestration will provide the function of coordination between all the different systems and applications to get the VM up and running.

Automation and orchestration work well in tandem. Automation covers single tasks. Using software to configure a VLAN on a switch is automation, and so is provisioning a VM over ESXi, or installing Linux on that VM. Orchestration, on the other hand, is the function of coordinating the execution of these automated tasks, in a specific sequence, each task using its own software and each on its respective system. The scope of automation involves single tasks. The scope of orchestration involves a workflow of tasks.

The concept of Modeling Data is not a new concept and is not exclusive to networks or even automation. Data modeling is very involved and is a major branch of data science. For the humble purpose of this blog, let’s use an example to demonstrate what a model is. In Example 1, you can see a configuration snippet of BGP on an IOS-XR router in the left column. In the right column, the specific values for this particular device were removed and replaced with a description of what should be there. A template of sorts.

Example 1 – BGP configuration snippet on IOS-XR and the corresponding data model in tree notation

As you can see, a model is a little more than just a template.

A model describes data types. An IP address is composed of four octets separated by the “.” character and each octet has a value between 0 and 255. An ASN is an integer between 1 and 65535. These two objects, an IP address and ASN, are leaf objects, each having a specific type and each instance of that leaf has a value.

As you have already guessed, a model also describes the data hierarchy. Address families are child objects to the main BGP process. Then the networks injected for a particular address family are children objects to that address family. And the same applies to neighbors: there are global neighbors that are children to the BGP process, and then there are neighbors defined under the different VRFs. You get the point.

So, in order to reflect hierarchy, other object types may exist in a model besides a leaf, such as leaf lists, lists and containers. A leaf-list is a list of leafs. For example, a snmp server configured on router is a leaf object. A list of snmp servers make up a leaf list. All leafs under a leaf list are of the same type.

A list is a group of other objects and has many instances. For example, the VRF in Example 1 is a list. It has children objects of different types (some of which are themselves lists), and at the same time you may have more than one VRF configured under the BGP process. A container is a group of objects of different types, but a container will only have a single instance. An example of a container is the BGP process itself. This is an over simplification of what a data model is just to elaborate on the concept.

Data models used in the arena of network programmability are described using a language called YANG. A “YANG model” is nothing more than a data model described using YANG.

Defining Programmability is not as easy as the previous terms. The reason for this is that the term is used across a very wide spectrum, and means different things depending on what context it is used in. Programming a device or a system basically means giving it instructions to do what you want it to do. A programmable device is a device that can execute different tasks based on the instructions it is given. In the world of electronics, an ASIC (Application Specific Integrated Circuit) is a chip that does one specific function. If this ASIC is built to accept two numbers as input and add those two numbers, it will always do that. A Microprocessor, on the other hand, accepts instructions describing what you want it to do with the input it is given. You can program it to add two numbers, multiply them, or subtract one from the other. A microprocessor is a programmable device while an ASIC is not.

But doesn’t this equate programming to configuration? Configuring a network device is basically telling that device what to do … right? Well, that is tricky question, and it is here that we discuss programmability as used today in the context of network automation.

Programmability for the purpose of network automation is basically the capability to retrieve data, whether configuration or operational data, from a system, or push configuration to a system, using an Application Programming Interface, or API for short. An API is an interface to a system that is designed for software interaction with this system. Contrast this to a router CLI. A CLI requires human interaction to be useful. An API on that same router would be designed so that a Python program, for example, can interact with the router without any human intervention.

But what really is an API?

An API is software running on a system. This software provides a particular function to other software, while not exposing this other software to how this function is performed. An API will typically have a predefined way to reach it, for example, over a specific TCP port. The API will also define the format of the data that it accepts, as well as the data it sends back. It may also define different message types and specific syntax and semantics to avail the services provided by the API. A device that implements an API is said to expose an API to be consumed by other software.

Now to connect the dots. Orchestration coordinates a number of automated tasks in a workflow to implement one of the disciplines of network management. Automation may leverage an API exposed by a device in order to manage this device programmatically. When a data model is used as a reference during programmatic access, the device is said to leverage model-driven programmability.

Quite a mouthful!

Network Programmability: The Details

Network Programmability as a practice is best summarized by the Programmability Stack in Figure 1 below. The stack defines six layers:

1. Application

2. Model

3. Protocol

4. Encoding

5. Transport

6. Infrastructure

Figure 1 – The Network Programmability Stack

(Disclaimer: The Programmability Stack has not been standardized by any industry-recognized entity such as the OSI 7-Layer Model. Therefore, you will probably run into a number of variations of this stack as you progress in your studies of programmability. I found that the one I have drawn here is the best version for the sake of getting the point across. Feel free to contrast it to other version you find elsewhere and tell me what you think in the comments below.)

At the top of the stack is an application that may be a simple python script or a sophisticated network management system such as Cisco Prime. At the bottom of the stack is the device exposing an API. In order for the application to programmatically speak with the device’s API, it will leverage a choice of components at the different layers of the stack.

The application will choose a model. Different types of models exist, the majority today described in YANG. A model may be vendor-specific or standards-based. In either case, the model will define the structure of the data that the application sends to or receives from the device (through the API).

The application will have to choose a protocol that defines the message types as well as the syntax and semantics used in those messages. There are three primary protocols used today for network programmability: NETCONF, RESTCONF and gRPC.

NETCONF, for example, defines three message types: hello, rpc and rpc-reply. It also defines specific operations that may be used in the rpc message to perform tasks such retrieving operational data from a device or pushing configuration to a device. The messages will use specific, well-defined syntax.

The protocols themselves are sometimes described as the APIs. Don’t get confused just yet ! When a device exposes a NETCONF API, then the application will have no choice but to use the NETCONF protocol to speak with the device. The same applies to the other protocols.

A protocol will have a choice of a data format, typically referred to as the data encoding. The most common data encodings in use today are XML, JSON, YAML and GPB. For example, NETCONF will send and receive data only in the form of XML documents. RESTCONF supports both, XML and JSON.

Then this data will be transported back and forth between the application and the device that is exposing the API using a transport protocol. For example, NETCONF uses SSH while RESTCONF uses HTTP.

This is network programmability in a nutshell!

Continue reading

An Introduction Into Kubernetes Networking – Part 4

Rule based routing

The final topic that we’ll cover in this series is rule based routing (HTTP hosts and paths) using the Kubernetes Ingress. An Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

“Ingress can provide load balancing, SSL termination and name-based virtual hosting.”

https://kubernetes.io/docs/concepts/services-networking/ingress/

There are a number of ways to configure a Kubernetes Ingress and in this example we’ll use a fanout. A fanout configuration routes traffic from a single IP address to more than one service.

From the YAML file above we can see a set of rules defining two http paths, one to a Guestbook and one to a different application called the Sockshop.

Kubernetes Ingress Controller

Just like we learnt Kubernetes Services require an external loadbalancer, a Kubernetes Ingress itself does not provide the rule based routing. Instead it relies on an Ingress Controller to perform this function.

There are many ingress controller options available. In our lab we are using Cisco Container Platform which automatically deploys an Nginx Ingress Controller to each new Kubernetes tenant cluster.

https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

In the following screenshots you’ll see that we have an ingress controller, nginx-ingress-controller-xxxxx, running on each node. We also have a service of type LoadBalancer which will direct our incoming traffic into the Nginx controller.

Similar to how MetalLB worked for Kubernetes Services, the Nginx controller will look for any changes to the Kubernetes Ingress. When a new ingress is configured the Nginx configuration will be updated with the routing rules which have been configured in the ingress YAML file (see above for example YAML).

Each ingress controller also has options to provide annotations for custom configuration of the specific controller. For example here are the Nginx annotations you can use.

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

In this lab Nginx has noticed the new ingress defined and has created the routing rules and annotions as part of it’s configuration. We can confirm this by looking at the nginx.conf file on each nginx-ingress-controller-xxxxx pod.

Since the ingress controller is running in multiple pods we can use the Kubernetes Services outlined above to provide access. In our case we have a LoadBalancer type service configured to direct external traffic to one of the available Nginx controller pods.

From there the Nginx controller will redirect based on the path, either to the guestbook frontend service or the sockshop service. These will in turn forward the traffic onto an available pod managed by the respective service.

Why should I use an ingress?

Besides the routing rules that we’ve just described, a Kubernetes Ingress allows us to conserve IP addresses. When we use a service of type LoadBalancer we require an externally routable address for each service configured. Assigning these addresses on premises may not have a big impact however usually there is a cost associated to each IP address in a public cloud environment.

When using an ingress we can have a single external IP address assigned (for the ingress service), and each service behind the ingress can use a ClusterIP. In this scenario the services are only accesible through the ingress and therefore don’t require a public IP address.

As we’ve just alluded to the Kubernetes Ingress also provides a single ingress point to which we can define our routing rules and other configuration such as TLS termination.

Continue reading

An Introduction Into Kubernetes Networking – Part 3

Tracking Pods and Providing External Access

In the previous section we learnt how one pod can talk directly to another pod. What happens though if we have multiple pods all performing the same function, as is the case of the guestbook application. Guestbook has multiple frontend pods storing and retrieving messages from multiple backend database pods.

◉ Should each front end pod only ever talk to one backend pod?

◉ If not, should each frontend pod have to keep its own list of which backend pods are available?

◉ If the 192.168.x.x subnets are internal to the nodes only and not routeable in the lab as previously mentioned, how can I access the guestbook webpage so that I can add my messages?

All of these points are addressed through the use of Kubernetes Services. Services are a native concept to Kubernetes, meaning they do not rely on an external plugin as we saw with pod communication.

There are three services we will cover:

◉ ClusterIP
◉ NodePort
◉ LoadBalancer

We can solve the following challenges using services.

◉ Keeping track of pods
◉ Providing internal access from one pod (e.g. Frontend) to another (e.g. Backend)
◉ Providing L3/L4 connectivity from an external client (e.g. web browser) to a pod (e.g. Frontend)

Labels, Selectors, and Endpoints

Labels and selectors are very important concepts in Kubernetes and will be relevant when we look at how to define services.

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/

◉ “Labels are key/value pairs that are attached to objects, such as pods [and] are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users. Unlike names and UIDs, labels do not provide uniqueness. In general, we expect many objects to carry the same label(s)”

◉ “Via a label selector, the client/user can identify a set of objects”

Keeping track of pods

Here is the deployment file for the Guestbook front-end pods.

As you can see from the deployment, there are two labels, “app: guestbook” and “tier: frontend“, associated to the frontend pods that are deployed. Remember that these pods will receive an IP address from the range 192.168.x.x

Here is the first service created (ClusterIP). From this YAML output we can see the service has a selector and has used the same key/value pairs (“app: guestbook” and “tier: frontend“) as we saw in our deployment above.

When we create this service, Kubernetes will track the IP addresses assigned to any of the pods that use these labels. Any new pods created will automatically be tracked by Kubernetes.

So now we’ve solved the first challenge. If we have 100s of frontend pods deployed do we need to remember the individually assigned pod addresses (192.168.x.x)?

No, Kubernetes will take care of this for us using services, labels, and selectors.

Providing internal access from one pod (e.g. Frontend) to another (e.g. Backend)

Now we know Kubernetes tracks pods and its associated IP address. We can use this information to understand how our frontend pod can access any one of the available backend pods. Remember each tier could potentially have 10, 100s or even 1000s of pods.

If you look at the pods or processes running on your Kubernetes nodes you won’t actually find one named “Kubernetes Service”. From the documentation below, “a Service is an abstraction which defines a logical set of Pods and a policy by which to access them”

https://kubernetes.io/docs/concepts/services-networking/service/

So while the Kubernetes Service is just a logical concept, the real work is being done by the “kube-proxy” pod that is running on each node.

Based on the documentation in the link above, the “kube-proxy” pod will watch the Kubernetes control plane for changes. Every time it sees that a new service has been created, it will configure rules in IPTables to redirect traffic from the ClusterIP (more on that soon) to the IP address of the pod (192.168.x.x in our example).

*** IMPORTANT POINT: *** We’re using IPTables however please see the documentation above for other implementation options

What is the ClusterIP?

The Kubernetes ClusterIP is an address assigned to the service which is internal to the Kubernetes cluster and only reachable from within the cluster.

If you’re using Kubeadm to deploy Kubernetes then the default subnet you will see for the ClusterIP will be 10.96.0.0/12

https://github.com/kubernetes/kubernetes/blob/v1.17.0/cmd/kubeadm/app/apis/kubeadm/v1beta2/defaults.go#L31-L32

So joining the pieces together, every new service will receive an internal only ClusterIP (e.g. 10.101.156.138) and the “kube-proxy” pod will configure IPTables rules to redirect any traffic destined to this ClusterIP to one of the available pods for that service.

DNS services

Before we continue with services, it’s helpful to know that not only do we have ClusterIP addresses assigned by Kubernetes, we also have DNS records that are configured automatically. In this lab we have configured CoreDNS.

“Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names.”

“Every Service defined in the cluster . . . is assigned a DNS name. By default, a client Pod’s DNS search list will include the Pod’s own namespace and the cluster’s default domain.

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

When we deploy the backend service, not only is there an associated ClusterIP address but we also now have a record, “backend.default.svc.cluster.local”. “Default” in this case being the name of the Kubernetes namespace in which the backend pods run. Since every container is configured to automatically use Kubernetes DNS, the address above will resolve correctly.

Bringing this back to our example above, if the frontend pod needs to talk to the backend pods and there are many backend pods to choose from, we can simply reference “backend.default.svc.cluster.local” in our applications code and this will resolve to the ClusterIP address which is then translated to one of the IP addresses of these pods (192.168.x.x)

NAT

We previously learnt in the pod to pod communication section that Kubernetes requires network connectivity be implemented without the use of NAT.

This is not true for services.

As mentioned above, when new services are created IPTables rules are configured which translate from the ClusterIP address to the IP address of the backend pod.

When traffic is directed to the service ClusterIP, the traffic will use Destination NAT (DNAT) to change the destination IP address from the ClusterIP to the backend pod IP address.

When traffic is sent from a pod to an external device, the pod IP Address in the source field is changed (Source NAT) to the nodes external IP address which is routeable in the upstream network.

Providing L3/L4 connectivity from an external client (e.g. web browser) to a pod (e.g. Frontend)

So far we’ve seen that Kubernetes Services continuously track which pods are available and which IP addresses they use (labels and selectors). Services also assign an internal ClusterIP address and DNS record as a way for internal communications to take place (e.g. frontend to backend service)

What about external access from our web browser to the frontend pods hosting the guestbook application?

In this last section covering Kubernetes Services we’ll look at two different options to provide L3/L4 connectivity to our pods.

NodePorts

As you can see from the service configuration, we have defined a “kind: Service” and also a “type: NodePort“. When we configure the NodePort service we need to specify a port (default is between 30000-32767) to which the external traffic will be sent. We also need to specify a target port on which our application is listening. For example we have used port 80 in the guestbook application.

When this service has been configured we can now send traffic from our external client to the IP address of any worker nodes in the cluster and specify the NodePort we have chosen (<NodeIP>:<NodePort>).

In our example we could use https://10.30.1.131:32222 and have access to the guestbook application through a browser.

Kubernetes will forward this traffic to one of the available pods on the specified target port (in this case frontend pods, port 80).

Under the hood Kubernetes has configured IPTables rules to translate the traffic from our worker node IP address/NodePort to our destination pod IP address/port. We can verify this by looking at the IPTables rules that have been configured.

LoadBalancer

The final topic in the Kubernetes Services section will be the LoadBalancer type which exposes the service externally using either a public cloud provider or an on premises load balancer.

https://kubernetes.io/docs/concepts/services-networking/service/

Unlike the NodePort service, the LoadBalancer service does not use the IP address from the worker nodes. Instead it relies upon an address selected from a pool that has been configured.

This example uses the Cisco Container Platform (CCP) to deploy the tenant clusters and CCP automatically installs and configures MetalLB for the L3/L4 loadbalancer. We have also specified a range of IP addresses that can be used for the LoadBalancer services.

“MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.”

https://metallb.universe.tf/

As you can see from YAML above, we configure the service using “type: LoadBalancer” however we don’t need to specify a NodePort this time.

When we deploy this service, MetalLB will allocate the next available IP address from the pool of addresses we’ve configured. Any traffic destined to the IP will be handled by MetalLB and forwarded onto the correct pods.

We can verify that MetalLb is assigning IPs correctly by looking at the logs.

Continue reading