The DevSec Blueprint's AWS DevSecOps Pipeline

Introduction

While exploring the world of DevSecOps, I came across Damien Burks’ The DevSec Blueprint - a comprehensive, free, and open-source learning guide that provides the foundational skills and knowledge needed to break into the field. After grasping the basics, I was eager to get hands-on and build a pipeline in AWS. I decided to dive in by following a guide straight from the DevSec Blueprint website. In this post, I’ll share my experience setting everything up, along with some lessons learned that might be helpful for anyone following a similar path.

Setting up Dependencies

First things first, I began by setting up the dependencies listed below.

• Setup GitHub Repos

• Setup Snyk Account

• Setup Terraform Cloud

• Configure Deployment Role in AWS

• Configure Secrets and Environment Variables

Setup GitHub Repos

I forked both the repos AWS DevSecOps Pipeline and Awesome FastAPI to my personal github account and then cloned both the forked repos to my local machine. I used GitHub CLI to interact with my GitHub account.

gh repo clone mdcn47/aws-devsecops-pipeline

gh repo clone mdcn47/awsome-fastapi

Setup Snyk Account

I signed up for Snyk using my GitHub account. Once logged-in, I clicked on my name in the bottom-left corner of the page and selected Account Settings. In the General section, I located the Auth Token field, generated a token, and noted it down for later integration. After that, I navigated to the Organization Settings page to obtain the Organization ID as well.

Setup Terraform Cloud

I signed up for Terraform Cloud, also using my GitHub account. Once I logged-in, I created an organization named MDCN which was pretty straightforward.

Configure Deployment Role in AWS

In this step, I’m setting up an AWS IAM role with OpenID Connect (OIDC), so Terraform Cloud can assume the role and deploy infrastructure changes on my behalf.

Adding Terraform Cloud as an OIDC provider

Creating an IAM Role for Terraform

I’ve begun by filling out the trust relationship fields, as shown below.

Following that, I created the role named terraform-cloud-deployer-oidc, attached the AdministratorAccess policy to it, and recorded its ARN for use in later steps.

Configure Secrets and Environment Variables

I started by defining the Organizational Variable Set in Terraform Cloud, which included creating the variables TFC_AWS_PROVIDER_AUTH and TFC_AWS_RUN_ROLE_ARN.

The following step was not documented in the guide; I identified it while working through the implementation.

Since we'll be using GitHub Actions to automate deployments when changes are pushed to the main branch, we need to generate a Terraform API token and add it as a repository secret in GitHub.

So I generated a Terraform API token using the steps shown below. Be sure to store the token securely, as it will only be displayed once.

Then I added the Terraform API token as a secret to the forked aws-devsecops-pipeline repository.

Deploying and Testing

In this final stage, I deploy and configure the infrastructure, run the pipeline, and analyze the output.

Deploying and Configuring the Pipeline

I logged into Terraform Cloud and created a project called devsecops-aws under my organization, MDCN. Within this project, I set up two workspaces: aws-devsecops-eks-cluster and aws-devsecops-pipelines.

Note*: In the guide, the workspace names included the organization name as a prefix. However, this will lead to errors later when running the workflow, since the workspace names in the Terraform configuration files (provider.tf) and the GitHub Actions workflow files (terraform-apply-eks.yml and terraform-apply-pipelines.yml) do not match.*

However, there's an issue: Terraform Cloud is unable to validate the configurations because the organization name in my account differs from the one specified in the Terraform code of the forked repository.

So, I used grep to search the local repository for the string dsb - the organization name used in the guide - to identify all the files that needed to be updated.

grep -rniw . -e "dsb"

I replaced all instances of dsb with mdcn in the relevant files and pushed the changes to the repository.

Next, I created two variables, SNYK_ORG_ID and SNYK_TOKEN, in the aws-devsecops-pipelines workspace and populated them with their respective values.

I then tried to deploy the changes using GitHub Actions.

However, the deployment failed at the Create Apply Run step. From the error message, it seems the connection to the AWS account failed.

While troubleshooting this issue, I noticed another required change in the main.tf file - replacing the AWS username from damien to mdcn47 (my AWS IAM user). I made the change right away.

After digging through the Terraform code, IAM role settings, and Terraform Cloud configuration, I finally tracked down the issue. I had originally created my Terraform Cloud project as aws-devsecops, but when setting up the IAM role I followed the guide and used AWS as the project name. This mismatch caused the trust policy to reference the wrong project. To fix it, I simply renamed my Terraform Cloud project to AWS.

I tested the connection by running terraform plan locally.

With that fixed, it’s a good time to kick off another deployment using GitHub Actions. This time the failure happened during the Apply step. Turns out the AMI type specified in the configuration is incompatible with the current Kubernetes version.

So I updated the code, swapping out ami_type = AL2_x86_64 for ami_type = AL2023_x86_64_STANDARD in the aws_eks_node_group resource block (line number 98).

The git push automatically triggered the workflow again and this time both the eks-cluster and pipelines terraform ran successfully.

I logged into my AWS account, checked the EKS console, and confirmed that the cluster, nodes, and pods were all successfully created.

However, when I checked AWS CodePipeline, I noticed that the awsome-fastapi application failed to deploy because of a GitHub connection error.

Navigating to Settings > Connections clearly indicates that the connection is in a pending status.

I followed the steps in the guide and was able to set up the connection without any issues.

The pipeline is now operational and configured to automatically detect and deploy changes from the GitHub repository.

Running the Pipeline and Analyzing Outputs

I triggered the pipeline by navigating to the CodePipeline dashboard, selecting the awsome-fastapi pipeline, and clicking Release change.

Clicking that button kicks off the pipeline - it’s expected to pull the latest code from GitHub, build the project, run tests and security scans, and finally deploy the application into the EKS cluster. However, the pipeline failed at the SnykSecurityScan step within the Test phase.

Digging into the error, I found that it failed while executing the command snyk auth ${SNYK_TOKEN}. A bit of research suggested that the curly braces {} might not be necessary, and removing them could fix the issue.

I went ahead and updated the sastscanning.yml file inside codepipeline/buildspecs and pushed the changes to the repository.

The new commit triggered another pipeline run. For the application deployment, I manually triggered the pipeline again by clicking Release change in CodePipeline. This time, Snyk authenticated successfully, but the pipeline failed as expected because it detected a critical severity vulnerability, exceeding the high threshold we had set.

The Snyk CLI exits with status code 1 whenever it finds vulnerabilities that meet or exceed the severity threshold we’ve set (high in our case). However, for testing the application deployment, I want the pipeline to continue while still reporting any issues. To achieve this, I appended || true to the command, making it: snyk test --file=requirements.txt --severity-threshold=high || true.

I updated the sastscanning.yml file once more and pushed the changes to the repository.

With this run, every test passed and the app was deployed without any issues.

While going through the execution logs, I noticed that the Snyk Code scan hadn’t run because it wasn’t enabled in my Snyk account. I quickly turned it on and reran the deployment.

Synk Scan Results

Trivy Scan Results (Container Security Scan)

With the application successfully deployed, I wanted to verify that it was running smoothly without any issues. I opened my EKS cluster (mdcn-devsecops-cluster), navigated to Resources > Services > awsome-fastapi, grabbed the load balancer URL, and accessed it in the browser. Since the app was running on port 80, I used HTTP. I was greeted with a Hello World message, confirming that the app was up and running without any issues.

It’s been a great learning journey, but I can’t stop here - I need to tear down the AWS infrastructure from this exercise before it surprises me with a painful credit card bill.

I navigated to the pipeline and eks-cluster Terraform directories and ran terraform destroy --auto-approve.

And that’s a wrap! The pipeline ran, the app deployed successfully, I learned a ton, and all AWS resources are now safely cleaned up - no surprises for my credit card.