![](https://crypto4nerd.com/wp-content/uploads/2023/06/1OeaW2TjBKugv3zW3MDQlnw-1024x457.png)
Before going ahead with the steps, one needs the following things :
- GCP account login (obviously)
- GCloud CLI
Feel free to go through the below post to get it setup.
Setup steps
For the steps to be performed, we’ll be using the following github gist from Anto whose video has been linked above.
Through this implementation, we’re aiming to use two service accounts for our terraform pipeline (one for plan and one for apply) and they will be using the GitHub Workload identity pool.
Please note that the gist also contains setting up a GCS bucket for storing state which we’ll be tackling in the lower half of this article.
Creating the workload pool
Please note to export the PROJECT_ID variable of your GCP project before running.
Post setting up gcloud CLI, run the below commands to create the workload identity pool and the provider, in this case, Github supports OIDC at the time of writing which is required for WIF.
gcloud iam workload-identity-pools create github
--project=$PROJECT_ID
--location="global"
--description="GitHub pool"
--display-name="GitHub pool"gcloud iam workload-identity-pools providers create-oidc "github"
--project="${PROJECT_ID}"
--location="global"
--workload-identity-pool="github"
--display-name="GitHub provider"
--attribute-mapping="google.subject=assertion.sub,attribute.workflow_ref=assertion.job_workflow_ref,attribute.event_name=assertion.event_name"
--issuer-uri="https://token.actions.githubusercontent.com"
In the second command line snippet, the Google Cloud SDK (gcloud) is utilized to create an OpenID Connect (OIDC) provider named “GitHub” within the specified project and global location. This provider is seamlessly integrated into Google Cloud’s Workload Identity Pools, empowering users to securely access resources through GitHub identities.
The command elegantly defines key attributes, enabling a smooth mapping of critical information, such as “google.subject,” “attribute.workflow_ref,” and “attribute.event_name,” facilitating efficient access and authorization. Additionally, the snippet establishes a secure connection by setting the issuer URI to “https://token.actions.githubusercontent.com,” ensuring robust and reliable communication between services.
Setting up service accounts
Now that we have created the workload pool for Github, we would need to create two service accounts, one for running terraform apply and one for running terraform plan.
Make sure to export the following variables :
1. GITHUB_REPO_NAME : Name of the GitHub Repo which houses your terraform code
2. GITHUB_USER : Your GitHub username
gcloud iam service-accounts create tf-plan-sa
--project=$PROJECT_ID
--description="SA use to run Terraform Plan"
--display-name="Terraform Planner"gcloud iam service-accounts create tf-apply-sa
--project=$PROJECT_ID
--description="SA use to run Terraform Apply"
--display-name="Terraform Applier"
gcloud iam service-accounts add-iam-policy-binding "tf-plan-sa@${PROJECT_ID}.iam.gserviceaccount.com"
--project="${PROJECT_ID}"
--role="roles/iam.workloadIdentityUser"
--member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/github/attribute.repository_name/${GITHUB_REPO_NAME}/attribute.actor/${GITHUB_USER}/attribute.event_name/pull_request"
--role="roles/iam.serviceAccountTokenCreator"
gcloud iam service-accounts add-iam-policy-binding "tf-apply-sa@${PROJECT_ID}.iam.gserviceaccount.com"
--project="${PROJECT_ID}"
--role="roles/iam.workloadIdentityUser"
--member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/github/attribute.workflow_ref/${GITHUB_USER}/${GITHUB_REPO_NAME}/.github/workflows/terraform.yaml@refs/heads/main"
--role="roles/iam.serviceAccountTokenCreator"
The commands can be referred from the following readme that was prepared :
Post doing the whole setup , it will look like this and following roles would need to be added. The below roles worked for me post adding.
Now that we have the service accounts readied up and binded , you can proceed with the following step of linking the service accounts to the pool and we’re good to go:
Before connection:
Post connecting the service accounts:
Post that is done , we can now proceed with setting up terraform and creating the GCS backend.
There are two ways to achieve the same:
- Run the create GCS bucket shell command and then point the backend block of terraform code to the created bucket.
- Create the bucket via Terraform locally and then migrate the state to the remote bucket for later use.
We’ll be doing step 1 of the process for this article for simplicity.
Make sure to export STATE_BUCKET and PROJECT_ID variables before running and this will create a GCS bucket to hold the terraform state file.
gcloud storage buckets create gs://$STATE_BUCKET --project=$PROJECT_ID --default-storage-class=STANDARD --location=EUROPE-WEST1 --uniform-bucket-level-access
Now that it has been done, we can move over to setting up Terraform and the GitHub actions pipeline for provisioning changes in a CI/ CD manner.
Terraform installation
Hashicorp has detailed the setup guide for installing Terraform locally in the following document which is a good read :
Post installing terraform , you would need to add the following block to make the state file be stored on remote GCS backend.
terraform {
# Comment out if storage needs to be removed and recreated again
backend "gcs" {
bucket = "tf-state-tracker"
}
}
Post terraform has been installed, its time to move on to creating the GitHub actions pipeline to make use of the WIF setup we did earlier.
The final pipeline code would look something like this.
name: Terraform GCP Pipelineon:
push:
branches:
- "main"
paths:
- 'terraform/**'
- '.github/workflows/terraform.yaml'
pull_request:
branches:
- "main"
paths:
- 'terraform/**'
- '.github/workflows/terraform.yaml'
jobs:
terraform:
permissions:
contents: 'read'
id-token: 'write'
pull-requests: 'write' #Needed to comment on the PR
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- id: 'plannerAuth'
if: github.ref != 'refs/heads/main'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth@v1.0.0'
with:
workload_identity_provider: 'projects/221017926326/locations/global/workloadIdentityPools/github/providers/github'
service_account: 'tf-plan@exemplary-tide-379122.iam.gserviceaccount.com'
- id: 'applierAuth'
if: github.ref == 'refs/heads/main'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth@v1.0.0'
with:
workload_identity_provider: 'projects/221017926326/locations/global/workloadIdentityPools/github/providers/github'
service_account: 'tf-apply@exemplary-tide-379122.iam.gserviceaccount.com'
- uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.4.5
- id: fmt
name: Terraform fmt
working-directory: terraform
run: terraform fmt -check
- id: init
name: Terraform Init
working-directory: terraform
run: terraform init -input=false -backend-config="bucket=tf-state-tracker"
# - id: validate
# name: Terraform Validate
# run: terraform validate -no-color
- id: plan
name: Terraform Plan
working-directory: terraform
run: terraform plan -no-color
continue-on-error: true
- uses: actions/github-script@v6
if: github.event_name == 'pull_request'
env:
PLAN: "terraformn${{ steps.plan.outputs.stdout }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Format and Style 🖌`${{ steps.fmt.outcome }}`
#### Terraform Initialization ⚙️`${{ steps.init.outcome }}`
#### Terraform Validation 🤖`${{ steps.validate.outcome }}`
#### Terraform Plan 📖`${{ steps.plan.outcome }}`
<details><summary>Show Plan</summary>
```n
${process.env.PLAN}
```
</details>
*Pushed by: @${{ github.actor }}, Action: `${{ github.event_name }}`*`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
- name: Terraform Plan Status
if: steps.plan.outcome == 'failure'
run: exit 1
- name: Terraform Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
working-directory: terraform
run: terraform apply -auto-approve -input=false
Lets try to go step-by-step and understand what each section does.
The “on” block
name: Terraform GCP Pipelineon:
push:
branches:
- "main"
paths:
- 'terraform/**'
- '.github/workflows/terraform.yaml'
pull_request:
branches:
- "main"
paths:
- 'terraform/**'
- '.github/workflows/terraform.yaml'
The above section of the pipeline does the following:
It triggers if the changes are pushed or raised pull requests on the “main” branch.
Under the “paths” block, it checks for the path where the terraform code lies, in this case, it’s under terraform directory.
Alongside the path to code, we have highlighted the YAML file to also be included whenever a change is pushed/raised pull request to the main branch for this pipeline as well.
The “jobs” block
jobs:
terraform:
permissions:
contents: 'read'
id-token: 'write'
pull-requests: 'write' #Needed to comment on the PR
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- id: 'plannerAuth'
if: github.ref != 'refs/heads/main'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth@v1.0.0'
with:
workload_identity_provider: 'projects/221017926326/locations/global/workloadIdentityPools/github/providers/github'
service_account: 'tf-plan@exemplary-tide-379122.iam.gserviceaccount.com'- id: 'applierAuth'
if: github.ref == 'refs/heads/main'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth@v1.0.0'
with:
workload_identity_provider: 'projects/221017926326/locations/global/workloadIdentityPools/github/providers/github'
service_account: 'tf-apply@exemplary-tide-379122.iam.gserviceaccount.com'
- uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.4.5
- id: fmt
name: Terraform fmt
working-directory: terraform
run: terraform fmt -check
- id: init
name: Terraform Init
working-directory: terraform
run: terraform init -input=false -backend-config="bucket=tf-state-tracker"
# - id: validate
# name: Terraform Validate
# run: terraform validate -no-color
- id: plan
name: Terraform Plan
working-directory: terraform
run: terraform plan -no-color
continue-on-error: true
- uses: actions/github-script@v6
if: github.event_name == 'pull_request'
env:
PLAN: "terraformn${{ steps.plan.outputs.stdout }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Format and Style 🖌`${{ steps.fmt.outcome }}`
#### Terraform Initialization ⚙️`${{ steps.init.outcome }}`
#### Terraform Validation 🤖`${{ steps.validate.outcome }}`
#### Terraform Plan 📖`${{ steps.plan.outcome }}`
<details><summary>Show Plan</summary>
```n
${process.env.PLAN}
```
</details>
*Pushed by: @${{ github.actor }}, Action: `${{ github.event_name }}`*`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
- name: Terraform Plan Status
if: steps.plan.outcome == 'failure'
run: exit 1
- name: Terraform Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
working-directory: terraform
run: terraform apply -auto-approve -input=false
This block has the following sections :
jobs: terraform:
- Specifies the start of the Terraform job in the pipeline.
permissions:
- Defines the permissions required for different actions on the repository.
2. Specifies the level of access for contents
, id-token
, and pull-requests
.
runs-on: ubuntu-latest
- Sets the operating system environment for the job to Ubuntu.
steps:
- Begins the list of steps to be executed in the job.
uses: actions/checkout@v3
- Checks out the repository code to the runner machine.
id: ‘plannerAuth’
- Assigns an ID to the following step for identification.
if: github.ref != ‘refs/heads/main’
- Specifies a conditional statement for executing the step only if the branch is not the main branch.
name: ‘Authenticate to Google Cloud’
- Provides a name for the step, indicating its purpose.
uses: ‘google-github-actions/auth@v1.0.0’
- Utilizes the Google Cloud authentication action for authenticating with Google Cloud.
with:workload_identity_provider:
- Specifies the Google Cloud Workload Identity Provider to use for authentication.
service_account:
- Defines the service account to be used for authentication.
id: ‘applierAuth’
- Assigns an ID to the following step for identification.
name: ‘Authenticate to Google Cloud’
- Provides a name for the step, indicating its purpose.
terraform_version: 1.4.5
- Specifies the version of Terraform to be installed and used.
id: fmt
- Assigns an ID to the following step for identification.
working-directory: terraform
- Sets the working directory for the step to the
terraform
directory.
run: terraform fmt -check
- Executes the Terraform command
fmt -check
to check the formatting of the Terraform code.
uses: actions/github-script@v6:
- Executes GitHub Actions script using JavaScript to create a comment on the PR with the Terraform plan details.
name: Terraform Plan Status:
- Checks if the Terraform plan step has failed and exits the job with an error code if it did.
name: Terraform Apply:
- Applies the Terraform changes automatically when there is a push to the “main” branch.
Conclusion
You’ve reached the end of this transformative guide, equipping yourself with the knowledge and tools to implement Workload Identity Federation (WIF) and establish a Terraform pipeline on Google Cloud Platform.
These vital building blocks have empowered you to enhance access management, automate infrastructure provisioning, and ensure smoother collaboration within your team. But we’re just scratching the surface.
In the upcoming article, we’ll delve into the realm of Cloud Composer, unraveling the secrets of orchestrating data pipelines and unlocking new levels of productivity. Brace yourself for an exhilarating exploration of GCP’s cutting-edge capabilities.”
Please do share and clap if you loved this article, also in case you missed part 1, I have attached the same below, feel free to give it a read and let me know how it is 😃.
We’ll meet again and complete this awesome journey to learn and grow as a data scientist by doing projects 👋 .