This topic is nothing new. There are many articles and tutorials on the internet about this. I’ll be setting up a Terraform project to explore GCP. I cover the basics of Terraform and GCP.
Warning
I replaced the real project ID with project-id in the following examples. You need to replace it with your project ID.
Prerequisites
Create GCP Project
First, you must create a GCP account, then create a project and set up billing on that project. If you already have a Gmail account, you can use it to make a GCP account. Head to the Google Cloud Console and create a new project.
My plan is to pay for the services I use. I need to feel the pain and learn more about cost optimization. You can be more intelligent than me and use the free tier for the first year. After that, you will be charged for the services you use. You can find more information about the free tier here.
Install And Set Up The gcloud CLI Tool
The gcloud
CLI is a part of the Google Cloud SDK. You can find the installation instructions for gcloud
CLI here.
Next, you need to set up the gcloud
CLI tool. Docs are here.
For future reference, you can find the list of all gcloud
commands here.
After initialization, you can always check what you set with the following command:
$ gcloud config list [SECTION/PROPERTY] [--all]
Install Terraform CLI Tool
You can find the Terraform installation instructions here. But I’m not following the instructions.
I’m using the tool tfenv to manage Terraform versions. Other tools can do that. You can use asdf
, too.
I saw that asdf
can do more than manage Terraform versions.
Let me explain how I do it since I use tfenv
. I assume you already installed it. I use tfenv
to install the desired version of Terraform. And to set it as default:
$ tfenv install 1.5.5
$ tfenv use 1.5.5
You can check the version with the following command:
$ terraform --version
I’ll explain later why I use tfenv
.
Other Tools I Use
Other tools I’ll be using are:
- task - a task runner and a replacement for
make
- git - version control system
- direnv - unconsciously set and unset environment variables
Setting Up The Project
I’m using this layout for now:
. # project root
├── README.md # project README
├── LICENSE # project LICENSE
├── .gitignore # project .gitignore
├── Taskfile.yml # project taskfile
└── gcp # GCP Terraform code
└── base # base GCP Terraform code
├── README.md # base README
├── .terraform-version # base terraform version
├── main.tf # base main.tf
├── outputs.tf # base outputs.tf
├── variables.tf # base variables.tf
├── terraform.tfvars # base terraform.tfvars
└── provider.tf # base provider.tf
Let me quickly explain. The gcp
directory is the root directory for all GCP Terraform code. The base
directory is the root directory for all base GCP Terraform code.
That base code would cover the following, for now:
- Creating a bucket for storing Terraform state
- Creating Key and KeyRing for encrypting Terraform state file
- Setting up Terraform’s backend
- Setting up Terraform provider
- Setting up Terraform base variables
Just create the directories and files, and we are ready to go.
Info
Oh, one small thing. The file .terraform-version is used by tfenv to set and pin the Terraform version. Even if I set with tfenv the default Terraform version, I use this file to say which Terraform version should be used for the base. If you are using tfenv do:
$ cd gcp/base
$ tfenv pin
Pinned version by writing "1.5.5" to /somepath/gcp/base/.terraform-version
With this, it does not matter what is the default Terraform version. The tfenv will use the version from the .terraform-version file. If, for some reason, you do not have the desired version, tfenv will install it for you.
Setting Up Provider
The first thing we need to do is to set up the provider. I’m using the latest version of the provider. You can find the latest version here:
provider "google" {
project = var.project_id
region = var.region
zone = var.zone
}
terraform {
required_version = ">=1.5.5"
required_providers {
google = {
source = "hashicorp/google"
version = "4.77.0"
}
}
backend "local" {
path = "local-state.tfstate"
}
}
I’m using the local
backend for now. I plan to switch to the gcs
backend later. The local
backend is used for storing the Terraform state locally. The state
file is called local-state.tfstate
. And it will contain the Terraform state for the base. It is not encrypted. My plan is to keep it safe and encrypted.
Block terraform
sets the required Terraform provider version, provider, and backend. You can find more here.
The provider
block configures the provider by setting the project ID, region, and zone. You probably noticed that I’m using variables.
Setting Up Variables
I’m using variables for defining variables I’ll be using in the base. I’m using the following variables:
variable "project_id" {
description = "project id"
type = string
}
variable "region" {
description = "default region"
type = string
}
variable "zone" {
description = "default zone"
type = string
}
Once you defined the variables, you need to set them. You can do that in the terraform.tfvars
file:
project_id = "my-project"
region = "europe-west1"
zone = "europe-west1-b"
You should avoid committing the terraform.tfvars
file. It may contain sensitive information. I’ll show you how to handle that later.
But for now, you must create the terraform.tfvars
file and set the variables. For the project_id
variable, you need to set the project ID you created earlier.
Choose the region and zone that is closest to you. You can find the list of regions and zones here.
Info
Is this important? Yes, it is.
You need to choose the region and zone carefully, considering reliability, availability, and cost into account. For your users to have the best experience, your servers must be close to them and deliver the content quickly. It would be best if you considered the cost, too.
To learn more about choosing the correct region and zone, you can find more here in Google documentation.
Init, Plan, and Apply
At this moment, I can initialize the Terraform workflow:
$ terraform init
Initializing the backend...
Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Finding hashicorp/google versions matching "4.77.0"...
- Installing hashicorp/google v4.77.0...
- Installed hashicorp/google v4.77.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
This will create a directory .terraform
and file .terraform.lock.hcl
file. You should include .terraform.lock.hcl
in your commit. It is used to lock the provider version. If you run
terraform init
again, it will use the same provider version. The folder .terraform
stores the provider plugins and other files required for Terraform to work.
Next, I can run terraform plan
:
$ terraform plan
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
This will show me what Terraform will do. Since I did not create any resources, it will not do anything. But it is good to run terraform plan
before terraform apply
.
Next, I can run terraform apply
:
$ terraform apply
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Well, this is expected. I did not create any resources. If there are any changes, Terraform will show you what it will do and ask you to confirm them. If you want to apply the changes, type yes
and hit enter.
Setting Up The Bucket
From the terraform init
output, I can see that the backend is configured. It is configured to use the local
backend. My goal is to use the gcs
backend. That means I want to store the
Terraform state file in the GCP bucket. Let’s look at how to do it:
data "google_storage_project_service_account" "gcs_account" {
}
resource "google_kms_key_ring" "tf_states" {
name = "terraform-state-key-ring"
location = "us"
lifecycle {
prevent_destroy = true
}
}
resource "google_kms_crypto_key" "tf_states" {
name = "terraform-state-key"
key_ring = google_kms_key_ring.tf_states.id
rotation_period = "100000s"
lifecycle {
prevent_destroy = true
}
}
resource "google_kms_crypto_key_iam_binding" "binding" {
crypto_key_id = google_kms_crypto_key.tf_states.id
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
members = ["serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"]
}
resource "google_storage_bucket" "tf_states_bucket" {
name = "terraform-states-${var.project_id}"
force_destroy = false
location = "us"
storage_class = "STANDARD"
versioning {
enabled = true
}
encryption {
default_kms_key_name = "your-crypto-key-id"
}
depends_on = [google_kms_crypto_key_iam_binding.binding]
lifecycle {
prevent_destroy = true
}
}
When working with Terraform, you are calling the API of the provider. With each call, you are pushing configuration to the provider. The provider will then create, update, or delete your resources.
That is a lot of code. Let me explain.
First, I’m creating a Key(google_kms_crypto_key
) and Key Ring(google_kms_key_ring
) for encrypting the Terraform state file. These resources are coming from Google KMS. The service allows you to work with cryptographic keys and execute cryptographic operations.
Next is adding the Role roles/cloudkms.cryptoKeyEncrypterDecrypter
to the Service Account. Which is needed to encrypt/decrypt the state file. In the end, I’m creating a bucket for storing the
Terraform state file, and I’m using the Key for encrypting the Terraform state file. There is a small gotcha. I need a Service Account for encrypting the Terraform state file. But it does not exist
yet. So, I’ll use an automatic Google Cloud Storage service account to get it when the resource is created. I’m using a data
resource for that.
I’m using lifecycle.prevent_destroy
to ensure the resources are not accidentally deleted.
If I now run the terraform plan
I’ll see the following:
$ terraform plan
data.google_storage_project_service_account.gcs_account: Reading...
data.google_storage_project_service_account.gcs_account: Read complete after 1s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# google_kms_crypto_key.tf_states will be created
+ resource "google_kms_crypto_key" "tf_states" {
+ destroy_scheduled_duration = (known after apply)
+ id = (known after apply)
+ import_only = (known after apply)
+ key_ring = (known after apply)
+ name = "terraform-state-key"
+ purpose = "ENCRYPT_DECRYPT"
+ rotation_period = "100000s"
}
# google_kms_crypto_key_iam_binding.binding will be created
+ resource "google_kms_crypto_key_iam_binding" "binding" {
+ crypto_key_id = (known after apply)
+ etag = (known after apply)
+ id = (known after apply)
+ members = [
+ "serviceAccount:service-964026605440@gs-project-accounts.iam.gserviceaccount.com",
]
+ role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
}
# google_kms_key_ring.tf_states will be created
+ resource "google_kms_key_ring" "tf_states" {
+ id = (known after apply)
+ location = "us"
+ name = "terraform-state-key-ring"
+ project = (known after apply)
}
# google_storage_bucket.tf_states_bucket will be created
+ resource "google_storage_bucket" "tf_states_bucket" {
+ force_destroy = false
+ id = (known after apply)
+ labels = (known after apply)
+ location = "US"
+ name = "terraform-states-project-id"
+ project = (known after apply)
+ public_access_prevention = (known after apply)
+ self_link = (known after apply)
+ storage_class = "STANDARD"
+ uniform_bucket_level_access = (known after apply)
+ url = (known after apply)
+ encryption {
+ default_kms_key_name = "your-crypto-key-id"
}
+ versioning {
+ enabled = true
}
}
Plan: 4 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
Ok, let’s try to apply the changes:
$ terraform apply
...
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
google_kms_key_ring.tf_states: Creating...
Error: Error creating KeyRing: googleapi: Error 403: Google Cloud KMS API has not been used in project 3534534535 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=3534534535 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
with google_kms_key_ring.tf_states,
on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
4: resource "google_kms_key_ring" "tf_states" {
I had to cut output. But you can see the error. I need to enable the Cloud KMS API. This will be a standard message if you try to use the API, which is not enabled. You can enable it by following the link in the error message. Or you can enable it with the following command:
$ gcloud services enable cloudkms.googleapis.com
Tip
You can see all the APIs you enabled with the following command:
$ gcloud services list --enabled
And list all available APIs with the following command:
$ gcloud services list --available
I already have enabled Google Storage API. But if you did not, you need to enable it, too. Try to figure out how. Ok, let’s try again:
$ terraform apply
...
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
google_kms_key_ring.tf_states: Creating...
╷
│ Error: Error creating KeyRing: googleapi: Error 409: KeyRing projects/project-id/locations/us/keyRings/terraform-state-key-ring already exists.
│
│ with google_kms_key_ring.tf_states,
│ on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
│ 4: resource "google_kms_key_ring" "tf_states" {
I was impatient and ran terraform apply
again. Google API responded that it was not enabled, but actually, it was. And not only that, it created the Key and KeyRing.
This situation is rare. But it can happen. We can fix it by importing created resources. Let’s do this:
$ terraform import google_kms_key_ring.tf_states project-id/us/terraform-state-key-ring
How do I know that? Well, I read the error message and the provider’s documentation. You’ll have a section Import in Terraform documentation for each resource. There, you’ll be able to see how to import it.
Now, let’s try again:
$ terraform apply
...
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
data.google_storage_project_service_account.gcs_account: Reading...
google_kms_key_ring.tf_states: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring]
data.google_storage_project_service_account.gcs_account: Read complete after 0s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]
google_kms_crypto_key.tf_states_key: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key]
google_kms_crypto_key_iam_binding.binding: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key/roles/cloudkms.cryptoKeyEncrypterDecrypter]
google_storage_bucket.tf_states_bucket: Creating...
google_storage_bucket.tf_states_bucket: Creation complete after 2s [id=terraform-states-project-id]
The bucket is created. You’ll see it if you go to GCP console. You can check bucket properties and configuration there in the console or with the following command:
$ gsutil ls -L -b gs://terraform-states-project-id
You can see that the bucket is encrypted with the Key we created. And to check the Key properties and configuration in the console or with the following command:
$ gcloud kms keys describe terraform-states-key --location us --keyring terraform-state-key-ring
You can list the content of the bucket with the command:
$ gsutil ls gs://terraform-states-project-id
But you’ll get nothing because it is empty. I still need to move Terraform’s state file to the bucket.
Setting Up The Backend
If you remember, I’m using the local
backend. If you check the local directory, you’ll see a file called local-state.tfstate
. That is the Terraform state file. And that file I want to move to
the bucket and encrypt it there. I can do that with the following code:
backend "gcs" {
bucket = "terraform-states-project-id"
prefix = "terraform/state"
}
And removing this one in provider.tf
:
backend "local" {
path = "local-state.tfstate"
}
Now, if I run terraform init
, I’ll see the following:
terraform init
Initializing the backend...
╷
│ Error: Backend configuration changed
│
│ A change in the backend configuration has been detected, which may require migrating existing state.
│
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".
╵
Well, let’s do as it says:
$ terraform init -migrate-state
When asked, type yes
and wait to finish. Let’s check if we have anything in the bucket:
$ gsutil ls gs://terraform-states-project-id/terraform/state
You’ll see that the bucket has a file called default.state
. That is the Terraform state file. It is encrypted. You can check it with the following command:
$ gsutil cat gs://terraform-states-project-id/terraform/state/default.tfstate | jq .
Conclusion
With I just set up a project. With it, I can start exploring GCP. We touched basic Terraform CLI commands(init
, plan
, apply
, and import
), Terraform variables,
provider, backend, and resources. I hope you learned something new. I know I did. I’ll continue exploring GCP with Terraform in the next article.