Should I put the Terraform lock file in GitHub ?

Should I put Terraform lock file in GitHub ?

The above question was discussed with DevOps engineers when they faced an issue while running the terraform workflow recently. It was evident that something was broken but could not identity the cause of it.

Some of the comments during our discussion were as below.

• There is no code change in the terraform configuration so why this issue popped up now ?
• Are you putting the lock file in GitHub ….No. We are not putting lock file in the GitHub
• What is lock file ?
• I don’t think it should be kept in the version control as we have not kept it and not faced any issue in the past. This could be your local setup issue
• No…No.. we should be using versions constraints in the terraform
• Isn’t the versions specified for the required providers is good enough ?

This was due to not having terraform lock file in GitHub .

Project Structure

The sample terraform project structure is devised as follows to understand the concept.

/ 
- main.tf
- terraform.tf

# terraform.tf

terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "3.1.0"
    }

    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.5.0"
    }
  }

  required_version = "~> 1.2"
}

main.tf
# empty - we don't need anything here to understand the concept

Run Terraform workflow.

terraform init

terraform init output

If you notice terraform init command

• Finds and installs the specific versions of the providers matching the criteria
• Provides a message that it has created a lock file .terraform.lock.hcl and also states that include this file in your version control repository.

However the question comes — if you have already provided version constraints in the terraform.tf then why it is needed ? Isn’t that enough ?

Let us look at the lock file first. The lock file is named as .terraform.lock.hcl file. The content of the file is as shown below.

Note the message

Terraform pulling the version > 4.5.0

Terraform pulling the exact version

In short the terraform does the following

• Creates the .terraform.lock.hcl
• Pulls the provider matching the version constraint and records the version details in the lock file
• Note that the required_version = ~> 1.2 is not recorded as it is not the provider but the version constraint for Terraform itself

Now let us understand why lock file has to be kept in Github ?

There are two providers used in the example random and aws

// Traform pulls the providers matching the constraints for respective providers. 
version = "3.1.0" // ensures that 3.1.0 is used 

version = ">= 4.5.0" // at least 4.5.0 ..... 
// ...it could be 4.5.x, 5.5.x - There is no restriction on the upper version
// ...Underlying provider adds new features and new services as it evolves ...
// ...resulting in new provider versions

If you look at version constraints for AWS provider it is specified as version => 4.5.0 so terraform is going to pull the version which is greater than 4.5.0 . This could be 4.5.x, 5.5.x. There is no restriction on the upper version. When I ran terraform init terraform downloaded the latest version available at the time of execution of which was 5.73.0.

Just consider the situation that you or any other developer runs the terraform init after few days or months then terraform init will download the latest version available at that time . This could be 5.73.3 or 5.74.0 and so on. This may lead to unintended consequences due to some change in the underlying provider which may break the things. This was precisely the case with DevOps team. After changing to earlier provider versions it got fixed !

Terraform downloaded the latest versions of the providers which caused issues and terraform plan started failing. It was discovered that the the terraform lock file was not there in the source control so terraform did not know the version to use. If there would have been lock file then terraform would have ensured the same version is downloaded which was used earlier by avoiding unintended consequences in future runs.

Secondly in practice I have seen many developers don’t put any version constraint in the in the terraform.tf when they start using terraform it. So if multiple developers are developing the terraform configuration and if it is done over a period of time then you are developing and testing against the latest versions all the time . If anything breaks then you need to really go back and figure out what went wrong. This could be avoided if you simply put the lock file in source control repository such as Github.

Summary

In order to ensure that the terraform downloads the same provider with which you have developed and tested the terraform configuration it is essential to put this lock file in Github.

• Lock file is only concerned with provider dependencies
• Records the providers and its version
• Hash values are calculated from the binary of the file (not discussed in detail here)
• lock file is updated by Terraform (terraform init)
• Lock file should not be edited manually
• As a thumb rule Never delete the lock file