What DevSecOps Means for Your CI/CD Pipeline

The CI/CD (Continuous Integration/Continuous Deployment) pipeline is a major ingredient of the DevOps recipe. As a DevSecOps practitioner, you need to consider the security implications for this pipeline. In this article, we will examine key items to think about when it comes to DevSecOps and CI/CD.

The type of CI/CD pipeline you choose—whether it’s managed, open source, or a bespoke solution that you build in-house—will impact whether certain security features are available to you out of the box, or require focused attention to implement.

Let’s dive in

Secret management for your CI/CD pipeline

Your CI/CD pipeline has the keys to the kingdom: it can provision infrastructure and deploy workloads across your system. From a security perspective, the CI/CD pipeline should be the only way to perform these actions. To manage your infrastructure, the CI/CD pipeline needs the credentials to access cloud service APIs, databases, service accounts, and more—and these credentials need to be secure.

Managed or hosted CI/CD pipelines provide a secure way to store these secrets. If you build your CI/CD solution, then you’re in charge of ensuring secrets are stored securely. CI/CD secrets should be encrypted at rest and only decrypted in memory, when the CI/CD pipeline needs to use them.

You should tightly lock down access to the configuration of your CI/CD pipeline. If every engineer can access these secrets, then the potential for leaks is huge. Avoid the temptation to let engineers debug and troubleshoot issues by using CI/CD credentials.

Some secrets (for example, access tokens) need to be refreshed periodically. CI/CD pipelines often use static secrets—which have much longer lifetimes, and so don’t need regular refreshing—to avoid the complexities of refreshing tokens.

Injecting secrets into workloads

Cloud workloads themselves also use secrets and credentials to access other resources and services that their functionality depends on. These secrets can be provided in several ways. If you deploy your system as packages using VM images or containers, then you can bake the secrets directly into the image, making them available in a file when the workload runs.

Another approach is to encrypt the secrets and store them in source control. Then, inject the decryption key into the workload, which can subsequently fetch, decrypt, and use the secrets.

Kubernetes allows for secrets that are managed outside of the workload image but exposed as an environment variable or a file. One benefit of secrets as files is that secret rotation doesn’t require re-deploying the workload.

Infrastructure as code: a security perspective

Infrastructure as code is not only an operational best practice; it is also a security best practice. 

software systems = infrastructure + workloads

When ad hoc changes are made to infrastructure configurations, this drift can introduce security risks. When resources are provisioned without any auditing or governance, it becomes difficult to maintain proper security measures across all resources.

Manage your infrastructure just like you manage your code. Use declarative configurations (like those of  Terraform, AWS CloudFormation, or Kubernetes CRDs). Review and audit every change.

Bring your own security tools

CI/CD pipelines are flexible. Generally speaking, they let you execute a sequence of steps and manage artifacts. The steps themselves are up to you. As a security engineer, you should take advantage of the security tools that already exist in your environment (especially in the cloud). For example, GitHub and GitLab both scan your commits for the presence of secrets or credentials. Some managed CI/CD solutions build in API scanning or application security scans. However, you may also prefer to add tools and checks into the mix.

You could also add static code analysis (like SonarQube) to ensure that code adheres to conventions and best practices. As another example, you mayincorporate vulnerability scanning (like Trivy or Grype) to your CI/CD pipeline, checking container images or third-party dependencies for security flaws.

Comprehensive detection and response

Application observability, monitoring, and alerting are fundamental DevOps Day 2 concerns. Although your CI/CD pipeline is not directly involved in these activities, you should use your CI/CD pipeline to deploy the security tools you use for these purposes. From the point of view of the CI/CD pipeline, these are just additional workloads to be deployed and configured.

Your CI/CD pipeline should include early detection of security issues that trigger on every change that affects workloads or infrastructure. Once changes are deployed, you need to run periodic checks and respond to events that happen post-deployment.

In case of faulty CI/CD, break glass

The CI/CD pipeline is a critical part of your system. If your CI/CD is broken or compromised, your application may continue to run, but you lose the ability to make safe changes. Large scale applications require constant updates and changes. If a security breach occurs, you need to be able to shut down and isolate parts of your application safely.

To do so, your CI/CD pipeline must be highly available and deployed securely. Whenever you need to update, rollback, or redeploy your application, you depend on your CI/CD pipeline.

What should you do if your CI/CD pipeline is broken? Prepare in advance for such a case, determining how your team and system will keep operating (at reduced capacity most likely) until you can fix your CI/CD pipeline. For complicated systems, you should have runbooks. Test how you will operate when the CI/CD is down or compromised.

Source: cisco.com