Terraform: error configuring S3 Backend: no valid credential sources for S3 Backend found.

Terraform backend configuration for remote storage may be quite challenging if the correct parameters are not passed.

We can get multiple errors while executing the terraform init command, depending upon the configuration arguments we miss or based upon the permissions defined for our AWS profile or AWS User/Role.

Terraform Backend State Configuration

Terraform Backend Configuration Document

Here, we will focus on valid credential errors.

Error: Error configuring S3 Backend: no valid credential sources for S3 Backend found.

Terraform backend configuration code: The backend was defined as below, we specified the bucket, key, and region.

terraform {
  required_providers {
    aws = {
        source = "hashicorp/aws"
    }
  }
  backend "s3" {
    bucket = "terraform-backend"
    key = "misc-infra-terraform-state"
    region = "us-east-1"
  }
}

When we ran the terraform init command for the above configuration and got the below error,

terraform s3 backend error


E:\terraform>terraform init

Initializing the backend...
╷
│ Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
│
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│       For verbose messaging see aws.Config.CredentialsChainVerboseErrors

In our case, the fix was pretty easy, the above terraform backend block had missing AWS credentials, we just passed the credential details, and it worked like charm!

Updated S3 Backend configuration

terraform {
  required_providers {
    aws = {
        source = "hashicorp/aws"
    }
  }
  backend "s3" {
    bucket = "terraform-backend"
    key = "misc-infra-terraform-state"
    region = "us-east-1"
    profile = "myaws_profile"
  }
}

Terraform – GitHub Repository Creation

Git is one of the official providers supported by Terraform. We can manage GitHub through Terraform, like repository creation, GitHub Actions, etc.

The official terraform git document can be here

In our below example, we will try to create a basic private repository in our GitHub account.

Authentication Method

Terraform supports two types of Authentication for Github.

  • OAuth /Personal Access Token
  • GitHub App Installation

OAuth /Personal Access Token

This is the easiest method for authenticating Git using Terraform. We will need to create a token in our GitHub account and pass the Token value to our Terraform provider block.

Steps to Generate OAuth /Personal Access Token

  • Log in to your GitHub account at, https://github.com
  • Click on the Left Top side profile icon and on the navigation bar, click on setting
terraform GitHub repository setting
  • Now, on the right-hand side profile navigation panel, click on the Developer settings
terraform git repository developer setting
  • On the Developers setting Page, select Personal access tokens, then select Tokens (classic)
  • You can select Fine-grained Tokens, but during this document creation it was a beta version, so we will go with classic tokens.
  • Click on Generate new token and select Classic Tokens.
terraform GitHub repository classic token
  • On the New personal access token (classic) generate the page, and update the following.
    • Note – Your Token usage description.
    • Expiration – Set the date to a minimum, for security purposes.
    • Select Scopes – select the permission/role you want to grant to your token, do not select all roles. select only the minimum roles which are required.
terraform GitHub token creation page
  • Once you update the above details, click on Generate token
  • Copy the generated token, as we will need this token in our terraform code
terraform GitHub token example

Note: The token shown in the above example has been deleted and no longer exist

Terraform Code

Define the provider block using GitHub as the provider, in our example, we have defined in provider.tf file.

terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "~> 5.0"
    }
  }
}

provider "github" {
  token = var.token
}

Now define your variables in variable.tf file.

variable "token" {
  description = "Enter your git token"
  default     = "ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxrfR1c"
}

Let us define the GitHub repositroy resource block in main.tf file. The output for our configuration will be the HTTP clone URL of our newly created repository

resource "github_repository" "my-terraform-repo" {
  name        = "my-terraform-repo"
  description = "Git repository created using Terraform"
  visibility  = "private"
}

# get the http clone url 
output "get_clone_url" {
  value = github_repository.my-terraform-repo.http_clone_url
}

We have completed the configuration, now let us init our terraform to download the git modules and plugins.

terraform init 
terraform GitHub init

The GitHub plugins/modules are downloaded, and now we will run the plan for an overview of our GitHub repository.

terraform plan -out gitrepo.tfplan
terraform GitHub plan

To create the GitHub repository, we will run the apply command with the above plan file.

terraform apply gitrepo.tfplan
terraform GitHub apply

The terraform has created our repository successfully, we can cross-verify them by validating them in our GitHub.

Terraform – Destroy Command

Terraform destroy command, is a way to destroy the infrastructure created using Terraform.

Often we need to destroy all or specific resources that we are managing using the terraform. terraform destroy is a way we can destroy the managed resource.

Syntax

terraform destroy <options> <resource_id>

To delete all managed resources, we can directly use the command, You will prompt for confirmation prior to destroying the managed resources.

Option 1:

terraform destroy

As we can see in the below output, terraform will list the resources to be destroyed and will prompt for confirmation.

Note: The confirmation should be case sensitive “yes“, any other word or even “Yes/YES” etc will result in cancellation of destruction.

terraform destroy command

Option 2:

To destroy specific resources from your managed resource use the flag –target along with the destroy command.

Syntax

terraform destroy --target <resource_id>

If you are not aware of your <resource_id>, there is an easy way to get the resource id. you will need to run the state list command. The state list command will list all the resources under your state file.

terraform state list

Once you have the resources listed, copy the resource id and use it in your destroy command as the target resource.

terraform destroy --target aws_vpc.siva_terraform_vpc[0]
terraform destroy command -target

If you do not want a user prompt/confirmation for resource deletion, you will need to pass the flag –auto-approve.

Syntax

terraform destroy --auto-approve
terraform destroy command –aoto-approve

As we can see all the resources will get terminated without any user confirmation.

Often we may want to delete all resources, except a few. In such a case, we can use the terraform state rm command to remove the resource from the state file and then run the terraform destroy command.

Syntax

terraform state rm <resource_id>
terraform state rm aws_vpc.siva_terraform_vpc[0]
terraform destroy command state rm

Now, if you run the terraform destroy command, all resources except the one we have removed from the state file will get terminated as terraform will not consider the resource as managed by terraform.

Read further,
Terraform Destroy

Dynamic Terraform State Configuration

Terraform doesn’t support passing dynamic variables for state files or backend blocks.

For example, let us consider we have a backend defined like the below,

Now, we have a scenario where we do not want to pass the values in the backend block in a static format, instead, we want the value to be dynamic.

We will rewrite the code and create two variables to pass the bucket name and bucket object (key) and pass these values in the backend block as variables instead of static values.

The above terraform configuration when we run the terraform init command will throw errors, as variables not allowed in the backend block.

E:\Projects\terraform>terraform init -migrate-state

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

╷
│ Error: Variables not allowed
│
│   on main.tf line 9, in terraform:
│    9:     bucket = var.backend_bucket #"s3-backend-demo"
│
│ Variables may not be used here.
╵

╷
│ Error: Variables not allowed
│
│   on main.tf line 10, in terraform:
│   10:     key = var.backend_key #"backend"
│
│ Variables may not be used here.
╵
Error while using dynamic variables for backend block

To resolve the issue, terraform allows us to pass the backend values dynamically through backend-config arguments via command prompt.

In your backend configuration block, remove the details which you want to pass dynamically. Like in our example we are using s3 as the backend, passing bucket name and object (key) values.

We will remove those attributes from the backend configuration block and instead we will pass them through backend-config arguments.

Passing backend configuration arguments values through a command prompt,

terraform init -backend-config="bucket=s3-backend-demo" -backend-config="key=backend"

In case we have an existing state file and we want to update the new backend arguments, you will need to run the init command with -migrate-state arguments.

terraform init -migrate-state -backend-config="bucket=s3-backend-demo" -backend-config="key=backend"

Terraform – Security Group

The Security Group acts as a virtual firewall, controlling the traffic flowing in and out of your EC2 resource associated with it.

VPS when created have default security groups, we can add additional security groups as per our requirements.

Security Groups operate at the instance level, whereas ACL acts over VPCs. Security Group supports allow rules only and are Stateful.

Security Groups consist of rules, which control traffic based upon protocols and port numbers, there are separate sets of rules for inbound and outbound traffic.

Security Groups allow you to restrict those rules to a certain IP CIDR range or you can allow traffic to all, 0.0.0.0/0 (IP4) and ::/0 (IP6)

We can create a Security Group through various methods, including AWS console as well as using IaaC (Infrastructure as Code).

Create Security Group through AWS Console

  • Login to your AWS Console, and navigate to VPC or EC2 service.
  • Click on the Security Group on the left navigation bar. All existing Security Groups will be listed.
  • Click on Create new Security Group, and enter the Security Group Name and Description.
  • Add the Protocol, Port, CIDR Range, etc., and click on save.

That’s it and your security group is created through GUI, now we will learn how we can use IaaC to create a Security Group in AWS using Terraform.

Create Security Group through Terraform (IaaC)

In our example, we will create a Security Group for the LAMP server and will allow traffic for ports 80 (HTTP), 443 (HTTPS), 22 (SSH), and 3306 (MySQL).

We will be creating a Security Group using different methods,

Method 1

In Method one let us go in the simplest way, we will have multiple blocks of ingress rules.

Files to be created

  • data.tf
  • variable.tf
  • provider.tf
  • securitygroup.tf

Provider.tf, Let’s update the Terraform provider information, so Terraform knows which cloud provider we are using. The provider files contain two blocks Terraform and Provider.

// Terraform block,define the provider source and version
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

// Define provider 
provider "aws" {
  shared_credentials_file = var.shared_credentials_files
  region                  = var.region
  profile                 = var.profile

  default_tags {
    tags = {
      "Resource" = "Security Group TF"
    }
  }
}

// local variables, can be referenced 
locals {
  common_tags = {
    "Project"   = "TF Modules"
    "Owner"     = "CubLeaf"
    "WorkSpace" = "${terraform.workspace}"
  }
}

Variables.tf, declare the variables used in the configuration. These variables are also named input variables.

variable "region" {
  description = "Enter the region"
  default     = "us-east-1"
}

// update the path of credential file 
// replace the below path with your credential file
variable "shared_credentials_files" {
  description = "Enter the credential file"
  default     = "C:\\Users\\username\\.aws\\credentials"
}

// Enter the profile of aws you want to use
variable "profile" {
  description = "Enter the profile name"
  default     = "myprofile"
}

variable "port" {
  description = "Enter the ports to be configured in SG Ingress rule"
  default     = [80, 22, 443, 3306]
}

variable "protocol" {
  description = "Ports for LAMP Server"
  type        = map(any)
  default = { "80" = "HTTP"
    "443" = "HTTPS"
    "22"  = "SSH"
  "3306" = "MYSQL" }
}

Data.tf, These are return values, in our case we want the default VPC id, to reference it in our security group resource, instead of hard coding we can use the data block to reference the existing resource. It’s very similar to traditional function return values.

// Get the default VPC ID
data "aws_vpc" "get_vpc_id" {
  default = true
}

Securitygroup.tf, The actual resource “aws_security_group” that we are going to create will be declared here,

resource "aws_security_group" "lamp_securitygroup_basic" {
  name        = "LAMP Security Group Basic"
  description = "LAMP Server Security Group Basic ${terraform.workspace}" // call the workspace 
  vpc_id      = data.aws_vpc.get_vpc_id.id  // calling from the data block
  tags        = local.common_tags

  // in-bound rules
  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  // outbound rules
  egress {

    from_port        = 0
    to_port          = 0
    protocol         = -1
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

Now, as we have the code/configuration ready, let us initiate and create the Security Groupby executing the terraform plan and applying the plan command.

Terraform – Dynamic Block

A Dynamic block is similar to for expression but creates nested blocks instead of a complex typed value. It iterates over a given complex value and generates a nested block for each element of that complex value. Unlike the count block, which will iterate over the resource, the dynamic will reside inside the resource block.

Dynamic blocks are supported inside the following blocks,

  • Resource
  • Data
  • Provider
  • Provisioner

Dynamic blocks can be used for resources like Setting, Security Group, etc.

Let us consider the example of creating a security group with multiple ingress rules for ports (80, 443, 3306, and 22). In an ideal scenario, we will end up copy-pasting the ingress rule for each port in the security group resource block.

Security Group (Without Dynamic Block)

// Security Group without Dynamic Block
resource "aws_security_group" "lamp_securitygroup_basic" {
  name        = "LAMP Security Group Basic"
  description = "LAMP Server Security Group Basic ${terraform.workspace}" // call the workspace 
  vpc_id      = data.aws_vpc.get_vpc_id.id
  tags        = local.common_tags
  // in-bound rules
  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  ingress {
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  ingress {
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  // outbound rules
  egress {
    from_port        = 0
    to_port          = 0
    protocol         = -1
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

The above code will work perfectly fine, but as your security group rules go on increasing, you will need to copy-paste the ingress block for each rule, which will make the code hard to maintain and update.

To overcome this issue, Terraform provides us with a dynamic block. We can write the same code using dynamic block,

Security Group (Dynamic Block)

// Data block for VPC ID
data "aws_vpc" "get_vpc_id" {
  default = true
}
// variable declaration 
variable "port" {
  description = "Enter the ports to be configured in SG Ingress rule"
  default     = [80, 22, 443, 3306]
}
// Create Security Group uisng Dynamic Block
resource "aws_security_group" "lamp_sg" {
  name        = "LAMP Security Group"
  description = "LAMP Server Security Group ${terraform.workspace}" // call the workspace 
  vpc_id      = data.aws_vpc.get_vpc_id.id
  tags        = local.common_tags
  dynamic "ingress" {
    for_each = var.port
    content {
      description      = "Security rule for inbound ${ingress.value}"
      from_port        = ingress.value // The block should use block name to fetch the value and key
      to_port          = ingress.value
      protocol         = "tcp"
      cidr_blocks      = ["0.0.0.0/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
  }
  // outbound rules
  egress {
    from_port        = 0
    to_port          = 0
    protocol         = -1
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

In our example, we are producing a nested block of ingress rules,

  • The label of our dynamic block is “ingress”, you can change the label, as you per the requirement, like if you want to generate “setting” nested blocks we can name the dynamic block as “setting”.
  • For_each – Argument will provide the complex argument to iterate over.
  • Content Block – defines the body of each generated block. In our example it will be the ingress block, iterating over multiple ports.

In multi-level nested block, we can have multi-level nested dynamic blocks, ie the nested dynamic bock within the dynamic block.

Terraform Module

Modules are defined with module blocks. They are used for the reusability of the code, suppose you have a stack that can be re-used.  In such a case instead of copy-paste the same resource code, again and again, we define a module.

Any folder containing a configuration file is by default considered as a module in Terraform.

These modules are referenced in the module code block.

Modules do contain input and outputs.

The module looks similar to resources, they just don’t have types. Each module must have unique names, in the configuration. Modules only have one required parameter, i.e. source.

Modules can be stored in local as well as remote locations,

  • Github
  • Local
  • Terraform Registry
  • Gitbucket
  • HTTP URL
  • S3 Bucket
  • GCS Bucket

Note – You may need to run the terraform init command to call the module file. In case of any update in the module file, you will need to run terraform init command in order to get those module updates reflected in your terraform configuration, failure to do so will result in the below error.

$ terraform init

Terraform Destroy

  • Destroy command will be used to destroy the created configuration.
  • You can use target flag to specify which resource you want to delete.
  • $ terraform  destroy – target < flagname>
  • You can also pass the -out flag to pass the output to a file.
  • $ terraform destroy –out <filename.tfplan>

Terraform Graph

  • Terraform creates a dependency graph when we execute the plan command, you can view the dependency graph using the graph command.
  • $ terraform graph
  • You can export the graph data using command, these graphs can be visualised using tools graphviz.
  • $ terraform graph > <filename>.dot
  • The graph will be displayed in below format,
  • You can also run dot command; dot command is part of the tool. The output will be saved in svg format.



Terraform Variable

  • Variable helps in centralize and manage values in the configuration.
  • The variable block consist of
    • Type
    • Description
    • Default values.
  • We can create a file variable.tf or default file name terraform.tfvars. We can also specify flag for variable files.
  • For any other file name used to define variable, use the command flag, –var-file. You can specify more than one file, but they will be executed in sequence.
  • All environment variables are defined using prefix TF_VAR_<variable name>
  • Terraform as a built in variable called as PATH it’s basically used in modules.
  • We can have below variable types in Terraform
    • String – Can be Boolean True or False or simple string
    • Map – A collection type (Associate array or Hash Table, similar to dictonary of Python)
    • List – A collection type (List of item, starting with index Zero).
  • We can call variables four ways,Through command prompt
    • Environment variables
    • Default variables
    • Variable defined in files
  • Variables have name and optional parameters,
    • Type – if type is omitted then terraform consider it as by default string.
    • Default
    • Description
  • Variables definition,
  • To call the variable in terraform main file. We need to use the keyword var as prefix
  • Syntax : var.<variable_name>
  • Calling the variables in main terraform file,