Introduction to Terraform and ACI – Part 5

If you haven’t already seen the Introduction to Terraform posts, please have a read through. Here are links to the first four posts:

1. Introduction to Terraform

2. Terraform and ACI​​​​​​​

3. Explanation of the Terraform configuration files

4. Terraform Remote State and Team Collaboration

So far we’ve seen how Terraform works, ACI integration, and remote state. Although not absolutely necessary, sometimes it’s useful to understand how the providers work in case you need to troubleshoot an issue.​​​​​​​ This section will cover at a high level how Terraform Providers are built, using the ACI provider as an example. 

Code Example

https://github.com/conmurphy/intro-to-terraform-and-aci-remote-backend.git

For explanation of the Terraform files see Part 3 of the series. The backend.tf file will be added in the current post.

Lab Infrastructure

You may already have your own ACI lab to follow along with however if you don’t you might want to use the ACI Simulator in the Devnet Sandbox.

ACI Simulator AlwaysOn – V4

Terraform File Structure

As we previously saw, Terraform is split into two main components, Core and Plugins. All Providers and Provisioners used in Terraform configurations are plugins.  ​​​​

Data sources and resources exist within a provider and are responsible for the lifecycle management of the specific endpoint. For example we will have a look at the resource, “aci_bridge_domain“, which is responsible for creating and managing our bridge domains.

The code for the Terraform ACI Provider can be found on Github

https://github.com/CiscoDevNet/terraform-provider-aci

There are a number of files in the root directory of this repo how the ones we are concerned with are “main.go“, the “vendor” folder, and the “aci” folder. 

◉ main.go: This file creates a valid, executable Go binary that Terraform Core can consume.

◉ vendor: This folder contains all of the Go modules which will be imported and used by the provider plugin. The key module for this provider is the ACI Go SDK which is responsible for making the HTTP requests to APIC and returning the response.

◉ aci: This is the important directory where all the resources for the provider exist.

Within the ACI  folder you’ll find a file called “provider.go“. This is a standard file in the Terraform providers and is responsible for setting the properties such as username, password, and URL of the APIC in this case.

It’s also responsible for defining which resources are available to configure in the Terraform files, and linking them with the function which implements the Create, Read, Update, and Delete (CRUD) capability.

In the aci folder you’ll also find all the data sources and resources available for this provider. Terraform has a specific structure for the filename and should start with data_source_ or resource_

Let’s look at the resource, “resource_aci_fvbd“, used to create bridge domains.

◉ On lines 10 and 11 the ACI Go SDK is imported. 

◉ The main function starts on line 16 and is followed by a standard Terraform configuration

    ◉ Lines 18 – 21 define which operations are available for this resource and which function should be called. We will see these four further down the page.

◉ Lines 29 – 59 in the screenshot are setting the properties which will be available for the resource in the Terraform configuration files.

TROUBLESHOOTING TIP: This is an easy way to check exactly what is supported/configurable if you think the documentation for a provider is incorrect or incomplete. 

We’ve now reached the key functions in the file and these are responsible for implementing the changes. In our case creating, reading, updating, and destroying a bridge domain.

If you scroll up you can confirm that the function names match those configured on lines 18-21 

Whenever you run a command, e.g. “terraform destroy“, Terraform will call one of these functions. 

Let’s have a look at what it’s creating.

First the ACI Go SDK has to be setup on line 419

Following on from that the values from your configuration files are retrieved so Terraform can take the appropriate action. For example in this screenshot the name we’ve configured, “bd_for_subnet“, will be stored in the variable, “name“. 

Likewise for the description, TenantDn, and all other bridge domain properties we’ve configured.

Further down in the file you’ll see the ACI Go SDK is called to create a NewBridgeDomain. This object is then passed to a Save function in the SDK which makes a call to the APIC to create the bridge domain

Continuing down towards the end of the create function you’ll see the ID being set on line 726. Remember when Terraform manages a resource it keeps the state within the terraform.tfstate file. Terraform tracks this resource using the id, and in the case of ACI the id is the DistinguishedName.

It’s not only the id that Terraform tracks though, all the other properties for the resource should also be available in the state file. To save this data there is another function, setBridgeDomainAttributes, which sets the properties in the state file with the values that were returned after creating/updating the resource. 

So when Terraform creates our bridge domain, it saves the response properties into the state file using this function.

TROUBLESHOOTING TIP: If resources are always created/updated when you run a terraform apply even though  you haven’t changed any configuration, you might want to check the state file to ensure that all the properties are being set correctly.

Source: cisco.com

Continue reading

The Best Kept Secret in Mobile Networks: A Million Saved is a Million Earned

I’m about to let you in on a little secret, actually it’s a big secret. What if I told you that you could save millions – even tens of millions – of dollars in two hours or less? If you were the CFO of a Mobile Operator, would this get your attention? What if I told you that the larger and busier your mobile network is, the more money you could save? What if the savings were in the hundreds of millions or even billions of dollars?

Now that I have your attention, let me tell you a bit about the Cisco Ultra Traffic Optimization (CUTO) solution. I could tell you that this is a vendor-agnostic solution for both the RAN and the mobile packet core. I could tell you how CUTO uses machine learning algorithms or about proactive cross-traffic contention detection. I could tell you about elephant flows and how the CUTO software optimizes the packet scheduler in RAN networks. I could write a whole blog on the CUTO technology and how it works, but I won’t. I will leave out the technical details for another time to share.

What I want to share today are two important facts about our CUTO solution:

1) We have helped multiple operators install and turn on CUTO, networkwide in less than 2 hours.

2) Real world deployments are demonstrating material savings, including a recent trial with a large Tier 1 operator that resulted in a calculated savings of several billions of dollars.

Out of all the segments of the network that operators are investing in, spectrum and RAN tend to be a top priority, both from a CAPEX and an OPEX standpoint. This is because mobile network operators have thousand’s, if not tens of thousands, of cell towers with accompanying RAN equipment. The amount of equipment required, and the cost of a truck roll per site leads to enormous expenses when you want to upgrade or augment the RAN network. With network traffic growing faster every year, the challenge to stay ahead of demand also grows. Major RAN network augmentations can take months, if not years to complete, especially when you factor in governmental regulations and permitting processes. This is where one aspect of the CUTO solution stands out, its ease and speed of deployment. CUTO’s purpose is to optimize the RAN network and improve the efficiency of the use of the spectrum. Instead of sending an army of service trucks and technicians to each and every cell tower, CUTO is deployed in the Core of the network, which is a very small number of sites (data centers). Making things even easier, CUTO can be deployed on Common Off the Shelf Servers (COTS), or better yet it can be deployed on the existing Network Function Virtual Infrastructure (NFVI) stack already in the mobile network core. Installing and deploying CUTO is as easy as spinning up a few virtual machines. In less than 2 hours, real operators on live networks have managed to install and deploy CUTO networkwide. Service providers talk a lot about MTTD (mean time to detect an issue) and MTTR (mean time to repair it), but with CUTO they can talk about MTTME – mean time to millions earned (saved).

If you look at the present mode of operation (PMO) for most mobile operators, there’s a very typical workflow that occurs in RAN networks. Customer’s consumption of video and an insatiable appetite for bandwidth eventually leads to a capacity trigger in the network, alerting the mobile network operator that they have congestion in a cell site. To handle the alert, operators typically have three options:

1) They might be able to “re-farm” spectrum and transition 3G spectrum to 4G spectrum. If this is an option, it typically leads to about a 40% spectrum gain at the price tag of about 22K in CAPEX per site and with very nominal OPEX costs. This option is relatively quick, simple, and leads to a good capacity improvement.

2) They might be able to deploy a new spectrum band or increase the Antenna Sectorization density. This option leads to about 20%-30% of the capacity gain but comes at a price tag of about 80K in CAPEX per site and about 20K in OPEX. This is a relatively long and costly process for a marginal capacity improvement and finding high quality “beachfront” spectrum is impossible in many markets around the world.

3) If neither option 1 or 2 are possible, the Operator would need to build a new cell site and cell split (tighten reuse). A new site leads to about 80% capacity gain depending on the site’s placement, user distribution, terrain, shadowing, etc., and comes at a cost of roughly 250K in CAPEX and about 65K/year in OPEX. This is an extremely long process (permitting, etc.) and very expensive.

Unfortunately, the spectrum is a finite resource, the opportunity for operators to choose option 1 or 2 is becoming scarce. Just five years ago, about 50% of cell sites with congestion were candidates for option 1, 30% are candidates for option 2, and only 20% required option 3. Five years from now virtually no cell sites will be candidates for option 1, maybe 10% will be candidates for option 2, and over 90% will require the very time-consuming and costly option 3.

CUTO offers mobile operators an alternative, helping to optimize traffic and reduce the congestion in the cell sites, which can significantly reduce the number of sites requiring capacity upgrades. During a recent trial to measure the efficacy of CUTO in the real-world and at scale, we deployed it with a Tier 1 operator. Near immediately, we saw a 15% reduction in the number of cell sites triggering a capacity upgrade due to consistent congestion. Here are some real-world numbers of how we were able to calculate the savings based on that 15% reduction:

The operator in this use case has:

• ~40M subscribers

• 10,000 sites triggering a need for a capacity upgrade

• Annual data traffic growth rate of 25%

The assumptions we agreed to with the operator were:

• Blended Incremental CAPEX/Upgrade (options 1, 2, and 3) = $100K CAPEX

• Blended Incremental OPEX/Site/Year (options 1, 2, and 3) = $20K OPEX/year

In this real-world example, you can see that CUTO saves this operator over $1.8B of CAPEX and $837M of OPEX over 5 years. I like to consider that as “adult money.” I recognize that these numbers are enormous and because of that, you may be skeptical. I was skeptical until I saw the results of the real-world trial for myself. I expect there to be plenty of FUD coming from the folks that are at risk of missing out on significant revenue because of your deployment of CUTO. Here’s my answer to those objections:

• Even if you cut these numbers in half or more, my guess is that you are looking at a material impact on your P&L.

• See it firsthand, don’t just take my word for it, ask your local Cisco Account Manager for a trial of CUTO and look at the savings based on your network, your mix of options 1, 2, or 3, and your costing models.

We all know that 5G will be driving new use cases, the need for more bandwidth, and we know that video will only become more ubiquitous. Mobile operators will require more and more cell sites and those sites will continue to fill up in capacity over time. Why not get ahead of the problem, and see what sort of MTTME your organization is capable of when your organization embraces a highly innovative software strategy?

Continue reading

Introduction to Terraform with ACI – Part 4

If you haven’t already seen the Introduction to Terraform post, please have a read through. This section will cover the Terraform Remote Backend using Terraform Cloud.​​​​​​​

1. Introduction to Terraform

2. Terraform and ACI​​​​​​​

3. Explanation of the Terraform configuration files

Code Example

https://github.com/conmurphy/intro-to-terraform-and-aci-remote-backend.git

For explanation of the Terraform files see the following post. The backend.tf file will be added in the current post.

Lab Infrastructure

You may already have your own ACI lab to follow along with however if you don’t you might want to use the ACI Simulator in the Devnet Sandbox.

ACI Simulator AlwaysOn – V4

Terraform Backends

An important part of using Terraform is understanding where and how state is managed. In the first section Terraform was installed on my laptop when running the init, plan, and apply commands. A state file (terraform.tfstate) was also created in the folder in which I ran the commands. ​​​​​​​

It’s fine when learning and testing concepts however does not typically work well in shared/production environment. What happens if my colleagues also want to run these commands? Do they have their own separate state file?​​​​​​​

These questions can be answered with the concept of the Terraform Backend.

“A backend in Terraform determines how state is loaded and how an operation such as apply is executed. This abstraction enables non-local file state storage, remote execution, etc.

Also Read: 300-420: Designing Cisco Enterprise Networks (ENSLD)

Here are some of the benefits of backends:

◉ Working in a team: Backends can store their state remotely and protect that state with locks to prevent corruption. Some backends such as Terraform Cloud even automatically store a history of all state revisions.

◉ Keeping sensitive information off disk: State is retrieved from backends on demand and only stored in memory. If you’re using a backend such as Amazon S3, the only location the state ever is persisted is in S3.

◉ Remote operations: For larger infrastructures or certain changes, terraform apply can take a long, long time. Some backends support remote operations which enable the operation to execute remotely. You can then turn off your computer and your operation will still complete. Paired with remote state storage and locking above, this also helps in team environments.”

https://www.terraform.io/docs/backends/index.html

As you can see from the Terraform documentation, there are many backend options to choose from.

In this post we’ll setup the Terraform Cloud Remote backend.

https://www.terraform.io/docs/backends/types/remote.html

We will use the same Terraform configuration files as we saw in the previous posts, with the addition of the “backend.tf “ file. See the code examples above for a post explaining the various files.

For this example you will need to create a free account on the Terraform Cloud platform

◉ Create a new organization and provide it a name

◉ Create a new CLI Driven workspace

◉ Once created, navigate to the “General” page under “Settings”

◉ Change the “Execution Mode” to “Local”

You have two options with Terraform Cloud

◉ Remote Execution – Let Terraform Cloud maintain the state and run the plan and apply commands

◉ Local Execution – Let Terraform Cloud main the state but you run the plan and apply  commands on your local machine

In order to have Terraform Cloud run the commands you will either need public access to the endpoints or run an agent in your environment (similar to Intersight Assist configuring on premises devices)

Agents are available as part of the Terraform Cloud business plan. For the purposes of this post Terraform Cloud will manage the state while we will run the commands locally.

◉ Navigate back to the production workspace and you should see that the queue and variables tabs have been removed.

◉ Copy the example Terraform code and update the backend.tf file (the Terraform files can be found in the Github repo above)

◉ Navigate to the Settings link at the top of the page and then API Tokens

◉ Create an authentication token

◉ Copy the token

◉ On your local machine create a file (if it doesn’t already exist) in the home directory with the name .terraformrc

◉ Add the credentials/token information that was just create for your organization. Here is an example

CONMURPH:~$ cat ~/.terraformrc

credentials "app.terraform.io" {

  token="<ENTER THE TOKEN HERE> "

}

◉ You should now have the example Terraform files from the Github repo above, an updated backend.tf file with your organization/workspace, and a .terraformrc file with the token to access this organization

◉ Navigate to the folder containing the example Terraform files and your backend.tf file

◉ Run the terraform init command. If everything is correct you should see the remote backend initialised and the ACI plugin installed

◉ Run the terraform plan and terraform apply commands to apply to configuration changes.

◉ Once complete, if the apply is successful have a look at your Terraform Cloud organization.

◉ In the States tab you should now see the first version of your state file. When you look through this file you’ll see it’s exactly the same as the one you previously had on your local machine, however now it’s under the control of Terraform Cloud​​​​​​​

◉ Finally, if you want to collaborate with your colleagues, you can all run the commands locally and have Terraform Cloud manage a single state file. (May need to investigate locking depending on how you are managing the environment)

Source: cisco.com

Continue reading

Introduction to Terraform with ACI – Part 3

If you haven’t already seen the previous Introduction to Terraform posts, please have a read through. This “Part 3” will provide an explanation of the various configuration files you’ll see in the Terraform demo.

Introduction to Terraform

Terraform and ACI

Code Example

https://github.com/conmurphy/terraform-aci-testing/tree/master/terraform

Configuration Files

You could split out the Terraform backend config from the ACI config however in this demo it has been consolidated. 

config-app.tf

The name “myWebsite” in this example refers to the Terraform instance name of the “aci_application_profile” resource. 

The Application Profile name that will be configured in ACI is “my_website“. 

When referencing one Terraform resource from another, use the Terraform instance name (i.e. “myWebsite“).

config.tf

Only the key (name of the Terraform state file) has been statically configured in the S3 backend configuration. The bucket, region, access key, and secret key would be passed through the command line arguments when running the “terraform init” command. See the following for more detail on the various options to set these arguments.

https://www.terraform.io/docs/backends/config.html

terraform.tfvars

variables.tf

We need to define the variables that Terraform will use in the configuration. Here are the options to provide values for these variables:

◉ Provide a default value in the variable definition below

◉ Configure the “terraform.tfvars” file with default values as previously shown

◉ Provide the variable values as part of the command line input

$terraform apply –var ’tenant_name=tenant-01’

◉ Use environmental variables starting with “TF_VAR“

$export TF_VAR_tenant_name=tenant-01

◉ Provide no default value in which case Terraform will prompt for an input when a plan or apply command runs

versions.tf

Source: cisco.com

Continue reading

Win with Cisco ACI and F5 BIG-IP – Deployment Best Practices

Applications environments have different and unique needs on how traffic is to be handled. Some applications, due to the nature of their functionality or maybe due to a business need do require that the application server(s) are able to view the real IP address of the client making the request to the application.

Now, when the request comes to the F5 BIG-IP, it has the option to change the real IP address of the request or to keep it intact. To keep it intact, the setting on the F5 BIG-IP ‘Source Address Translation’ is set to ‘None’.

As simple as it may sound to just toggle a setting on the F5 BIG-IP, a change of this setting causes significant change in traffic flow behavior.

Let us take an example with some actual values. Starting with a simple setup of a standalone F5 BIG-IP with one interface on the F5 BIG-IP for all traffic (one-arm)

◉ Client – 10.168.56.30

◉ BIG-IP Virtual IP – 10.168.57.11

◉ BIG-IP Self IP – 10.168.57.10

◉ Server – 192.168.56.30

Scenario 1: With SNAT

From Client: Src: 10.168.56.30           Dest: 10.168.57.11

From BIG-IP to Server: Src: 10.168.57.10 (Self-IP)     Dest: 192.168.56.30

In above scenario, the server will respond back to 10.168.57.10 and F5 BIG-IP will take care of forwarding the traffic back to the client. Here, the application server has visibility to the Self-IP 10.168.57.10 and not the client IP. 

Scenario 2: No SNAT

From Client: Src: 10.168.56.30           Dest: 10.168.57.11

From BIG-IP to Server: Src: 10.168.56.30       Dest: 192.168.56.30

In this scenario, the server will respond back to 10.168.56.30 and here is where comes in the complication, as the return traffic needs to go back to the F5and not the real client. One way to achieve this is to set the default GW of the server to the Self-IP of the BIG-IP and then the server will send the return traffic to the BIG-IP. BUT what if the server default gateway is not to be changed for whatsoever reason.  Policy based redirect will help here. The default gateway of the server will point to the ACI fabric, and the ACI fabric will be able to intercept the traffic and send it over to the BIG-IP.

With this, the advantage of using PBR is two-fold

◉ The server(s) default gateway does not need to point to F5 BIG-IP, but can point to the ACI fabric

◉ The real client IP is preserved for the entire traffic flow

◉ Avoid server originated traffic to hit BIG-IP, resulting BIG-IP to configure a forwarding virtual to handle that traffic. If server originated traffic volume is high, it could result unnecessary load the F5 BIG-IP

Before we get deeper into the topic of PBR below are a few links to help you refresh on some of the Cisco ACI and F5 BIG-IP concepts

◉ Cisco ACI fundamentals

◉ SNAT and Automap

◉ F5 BIG-IP modes of deployment

Now let us look at what it takes to configure PBR using a Standalone F5 BIG-IP Virtual Edition in One-Arm mode.

To use the PBR feature on APIC – Service graph is a MUST

Details on L4-L7 service graph on APIC

To get hands on experience on deploying a service graph (without pbr)

Configuration on APIC

1) Bridge domain ‘F5-BD’

◉ Under Tenant->Networking->Bridge domains->’F5-BD’->Policy

◉ IP Data plane learning – Disabled

2) L4-L7 Policy-Based Redirect

◉ Under Tenant->Policies->Protocol->L4-L7 Policy based redirect, create a new one

◉ Name: ‘bigip-pbr-policy’

◉ L3 destinations: F5 BIG-IP Self-IP and MAC

◉ IP: 10.168.57.10

◉ MAC: Find the MAC of interface the above Self-IP is assigned from logging into the F5 BIG-IP (example: 00:50:56:AC:D2:81)

3) Logical Device Cluster- Under Tenant->Services->L4-L7, create a logical device

◉ Managed – unchecked

◉ Name: ‘pbr-demo-bigip-ve`

◉ Service Type: ADC

◉ Device Type: Virtual (in this example)

◉ VMM domain (choose the appropriate VMM domain)

◉ Devices: Add the F5 BIG-IP VM from the dropdown and assign it an interface

◉ Name: ‘1_1’, VNIC: ‘Network Adaptor 2’

◉ Cluster interfaces

◉ Name: consumer, Concrete interface Device1/[1_1]

◉ Name: provider, Concrete interface: Device1/[1_1]

4) Service graph template

◉ Under Tenant->Services->L4-L7->Service graph templates, create a service graph template

◉ Give the graph a name:’ pbr-demo-sgt’ and then drag and drop the logical device cluster (pbr-demo-bigip-ve) to create the service graph

◉ ADC: one-arm

◉ Route redirect: true

5) Click on the service graph created and then go to the Policy tab, make sure the Connections for the connectors C1 and C2 and set as follows:

◉ Direct connect – True

◉ Adjacency type – L3

6) Apply the service graph template

◉ Right click on the service graph and apply the service graph

◉ Choose the appropriate consumer End point group (‘App’) provider End point group (‘Web’) and provide a name for the new contract

◉ For the connector

◉ BD: ‘F5-BD’

◉ L3 destination – checked

◉ Redirect policy – ‘bigip-pbr-policy’

◉ Cluster interface – ‘provider’

Once the service graph is deployed, it is in applied state and the network path between the consumer, F5 BIG-IP and provider has been successfully setup on the APIC

Configuration on BIG-IP

1) VLAN/Self-IP/Default route

◉ Default route – 10.168.57.1

◉ Self-IP – 10.168.57.10

◉ VLAN – 4094 (untagged) – for a VE the tagging is taken care by vCenter

2) Nodes/Pool/VIP

◉ VIP – 10.168.57.11

◉ Source address translation on VIP: None

3) iRule (end of the article) that can be helpful for debugging

Few differences in configuration when the BIG-IP is a Virtual edition and is setup in a High availability pair

1) BIG-IP: Set MAC Masquerade

2) APIC: Logical device cluster

◉ Promiscuous mode – enabled

◉ Add both BIG-IP devices as part of the cluster

3) APIC: L4-L7 Policy-Based Redirect

◉ L3 destinations: Enter the Floating BIG-IP Self-IP and MAC masquerade

Configuration is complete, let’s look at the traffic flows.

Client-> F5 BIG-IP -> Server

Server-> F5 BIG-IP -> Client

In Step 2 when the traffic is returned from the client, ACI uses the Self-IP and MAC that was defined in the L4-L7 redirect policy to send traffic to the BIG-IP.

iRule to help with debugging on the BIG-IP

Output seen in /var/log/ltm on the BIG-IP, look at the event <SERVER_CONNECTED>

Scenario 1: No SNAT -> Client IP is preserved

If you are curious of the iRule output if SNAT is enabled on the BIG-IP – Enable AutoMap on the virtual server on the BIG-IP

Scenario 2: With SNAT -> Client IP not preserved.

Continue reading

For Banks – The Contact Center is Your Best Friend

For years, the album that sold the most units was Carole King’s “Tapestry”. Estimates are that this record has sold more than 25 million copies. Rife with well-known songs, an interesting comment made by one of the initial reviewers in 1971 called the song “You’ve Got a Friend” the “core” of and “essence” of the album. It didn’t hurt that James Taylor’s version also became a monster hit. For banks, they too have a friend – in their contact centers.

The malls emptied, and the contact centers filled up

The last twelve months have initiated a renaissance in contact center operations. While the modernization of contact centers had been on a steady march, the realities of 2020 suddenly presented a giant forcing function changing the customer engagement landscape in a dramatic fashion. In one fell swoop, 36 months of planned investment in modernizing contact centers accelerated into a single 12-month period. As the physical world was shut down, the digital world ramped up dramatically. Banks saw branch visits slow to a crawl, and digital and contact center interactions increased by orders of magnitude. In addition, up to 90% of contact center agents were sent home to work, with estimates that a majority of them will stay there over time as indicated by this Saddletree Research analysis:

Prior planning prevented poor performance

Fortunately, banks and credit unions were one of the key vertical markets that were relatively prepared for 2020 and were able to lean into the challenges presented, though this was not to say things went perfectly. What was behind this preparation and what were these organizations doing prior and during the crisis? And what should they do in the years ahead?

The “Digital Pivot” paid huge dividends

At their core, banks and credit unions collect deposits and loan them out at (hopefully) a profit. With money viewed as a commodity, financial services firms were one of the first industries to understand the only two sustainable differentiators they possessed were the customer experience they delivered, and their people. It is interesting that these are the main two ingredients which comprise a contact center!

For many banks prior to 2010, the biggest challenge for contact center operations consisted of navigating mergers and acquisitions when combining operations. Normalizing operations during mergers often manifested itself in a giant IVR farms meant to absorb large amounts of voice traffic. Prior form factors for self-service were not know as “low-effort” propositions, and customer experience scores suffered for years. Banks as an aggregate industry dropped below all industry averages for customer experience, after leading for years.

The mobile revolution presented a giant reset for banking customer experience. Financial institutions by and large have done an excellent job of adopting mobile applications to the delight of their customers. In response, customer experience scores in banking have steadily risen the past 10 years, and banks are near the top quartile again, only trailing consumer electronics firms and various retailers.

Banks are more like a contact center than you think

Banks and contact centers have very common characteristics. Both wrap themselves in consumer-friendly self-service applications which automate formerly manual processes that required human assistance. These include popular customer engagement platforms such as mobile applications and ATMs. In the contact center this dynamic involves speech recognition, voice biometrics, and intelligent messaging.

As self-service has become increasingly popular, live interactions that are left over for both the branch and the contact center have become more complex, difficult to solve on the first try, and requiring collaborative, cross business resolution by the individual servicing the customer. These types of interactions are known as “outliers”. In this situation the contact center becomes in essence, a “digital backstop” where the consumer interacts with self service first and then and only then seeks live assistance.

Prior planning prevents poor performance part II

The digital tsunami started in 2010 via the mass adoption of mobile applications by banks, giving this industry in particular a significant head start on the “outlier” dynamic. Therefore in 2020 when the shopping malls emptied out and contact centers filled up, banks had already been operating tacitly in the “outlier” model for a number of years and were in a better position to succeed. Applications such as intelligent call back, integrated consumer messaging, work at home agents, voice biometrics, A.I. driven intelligent chat bots, and seamless channel shift from mobile applications to the contact center were already in place to some extent for leading financial institutions.

Thinking ahead

With much of the focus on contact center, automation in banking has been able to extend A.I. into the initial stages of customer contact. The road ahead will include wrapping A.I. driven intelligence to surround contact center resources during an interaction, essentially creating a new category of resources known as “Super Agents”. In this environment, all agents in theory can perform as the best agents because learnings from the best performers are automatically applied throughout the workforce. In addition, Intelligent Virtual Assistants, or IVAs, will act as “digital twins” for contact center agents – automatically looking for preemptive answers to customers questions, and automating both contact transcripts and after call work documentation and follow up.

Yes, if you’re a bank, you have a friend in your contact center

Banks made the pivot to delivering better customer experience in their contact centers during the “Digital Pivot” in the early 2010s. From there, banks made steady progress to reclaim their CX leadership and delivering excellent customer experiences. The realities of 2020 accelerated contact center investment by at least 36 months into a 12-month window. Banks which had established leadership utilized this forcing function to accelerate a next generation of customer differentiators, firmly entrenched in themselves as category leaders in the financial services industry. Other institutions can utilize these unique times to play rapid catch-up. Who benefits? Their customers.

Source: cisco.com

Continue reading