AWS EC2 – Unable to install/download packages from amazon repo to EC2 instance

We may have faced this issue on connecting to the Amazon repo to download/install packages, like Mysql, Apache, Nginx, etc.

We may get a connection time-out error when we run the sudo apt install mysql-server command. The error can occur for any package, in our example, we are installing a MySQL server.

Error

root@ip-10-9-9-58:/home/ubuntu# sudo apt install mysql-server

Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.91.65.63), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.207.133.243), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (54.87.19.168), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (54.165.17.230), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (3.87.126.146), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (3.209.10.109), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (18.232.150.247), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (34.201.250.36), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (34.237.137.22), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.73.36.184), connection timed out

The issue may occur in private or public instances. The most common solution is to first check the security groups assigned to your instance.

Steps

Login to your AWS console > EC2 > Security Groups,

Click on the Edit Outbound rules tab, then on Edit outbound rules.

Ensure we have HTTP TCP protocol for port 80 opened on outbound rules of the assigned security group.

If the rule for HTTP TCP Port 80 is missing, add the new rules in the similar format specified in the above image and save the changes.

Now, try to install the package, it should connect over the internet and install the package successfully.

Restricted Outbound Access

To solve the issue, above we have allowed all outbound traffic, in some cases due to security restrictions the organization may not allow you to open outbound traffic to all IP ranges.

The best practice says we should have minimum permissions.

To accomplish our security goals, we can restrict the outbound traffic to a certain Amazon repo mirrors IPs.

As we saw in the above error, Amazon tries to hit several of its mirrors to download and install the package. We need to copy any of these mirror IP addresses and use them to restrict outbound traffic in our security group.

root@ip-10-9-9-58:/home/ubuntu# sudo apt install mysql-server

Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.91.65.63), connection timed out

In this example, we will open outbound only for IP 52.91.65.63.

Login to your AWS console > EC2 > Security Groups, select the assigned security group, and click on the Edit Outbound rules tab, then on Edit outbound rules.

Select the HTTP TCP port 80 rule and change 0.0.0.0/0 to 52.91.65.63/32. Save the changes, this will restrict the outbound rule for HTTP TCP port 80 to only one IP address, 52.91.65.63.

Note: We need to add a CIDR range even when we have one single IP address, Security Group does not allow us to add just a single IP address without a CIDR range. Even for a single IP address, we are required to add a CIDR block.

In our example for our single IP address, we have added a CIDR range /32.

You can change the CIDR block range based on your IP requirements.

Terraform – Security Group

The Security Group acts as a virtual firewall, controlling the traffic flowing in and out of your EC2 resource associated with it.

VPS when created have default security groups, we can add additional security groups as per our requirements.

Security Groups operate at the instance level, whereas ACL acts over VPCs. Security Group supports allow rules only and are Stateful.

Security Groups consist of rules, which control traffic based upon protocols and port numbers, there are separate sets of rules for inbound and outbound traffic.

Security Groups allow you to restrict those rules to a certain IP CIDR range or you can allow traffic to all, 0.0.0.0/0 (IP4) and ::/0 (IP6)

We can create a Security Group through various methods, including AWS console as well as using IaaC (Infrastructure as Code).

Create Security Group through AWS Console

  • Login to your AWS Console, and navigate to VPC or EC2 service.
  • Click on the Security Group on the left navigation bar. All existing Security Groups will be listed.
  • Click on Create new Security Group, and enter the Security Group Name and Description.
  • Add the Protocol, Port, CIDR Range, etc., and click on save.

That’s it and your security group is created through GUI, now we will learn how we can use IaaC to create a Security Group in AWS using Terraform.

Create Security Group through Terraform (IaaC)

In our example, we will create a Security Group for the LAMP server and will allow traffic for ports 80 (HTTP), 443 (HTTPS), 22 (SSH), and 3306 (MySQL).

We will be creating a Security Group using different methods,

Method 1

In Method one let us go in the simplest way, we will have multiple blocks of ingress rules.

Files to be created

  • data.tf
  • variable.tf
  • provider.tf
  • securitygroup.tf

Provider.tf, Let’s update the Terraform provider information, so Terraform knows which cloud provider we are using. The provider files contain two blocks Terraform and Provider.

// Terraform block,define the provider source and version
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

// Define provider 
provider "aws" {
  shared_credentials_file = var.shared_credentials_files
  region                  = var.region
  profile                 = var.profile

  default_tags {
    tags = {
      "Resource" = "Security Group TF"
    }
  }
}

// local variables, can be referenced 
locals {
  common_tags = {
    "Project"   = "TF Modules"
    "Owner"     = "CubLeaf"
    "WorkSpace" = "${terraform.workspace}"
  }
}

Variables.tf, declare the variables used in the configuration. These variables are also named input variables.

variable "region" {
  description = "Enter the region"
  default     = "us-east-1"
}

// update the path of credential file 
// replace the below path with your credential file
variable "shared_credentials_files" {
  description = "Enter the credential file"
  default     = "C:\\Users\\username\\.aws\\credentials"
}

// Enter the profile of aws you want to use
variable "profile" {
  description = "Enter the profile name"
  default     = "myprofile"
}

variable "port" {
  description = "Enter the ports to be configured in SG Ingress rule"
  default     = [80, 22, 443, 3306]
}

variable "protocol" {
  description = "Ports for LAMP Server"
  type        = map(any)
  default = { "80" = "HTTP"
    "443" = "HTTPS"
    "22"  = "SSH"
  "3306" = "MYSQL" }
}

Data.tf, These are return values, in our case we want the default VPC id, to reference it in our security group resource, instead of hard coding we can use the data block to reference the existing resource. It’s very similar to traditional function return values.

// Get the default VPC ID
data "aws_vpc" "get_vpc_id" {
  default = true
}

Securitygroup.tf, The actual resource “aws_security_group” that we are going to create will be declared here,

resource "aws_security_group" "lamp_securitygroup_basic" {
  name        = "LAMP Security Group Basic"
  description = "LAMP Server Security Group Basic ${terraform.workspace}" // call the workspace 
  vpc_id      = data.aws_vpc.get_vpc_id.id  // calling from the data block
  tags        = local.common_tags

  // in-bound rules
  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  // outbound rules
  egress {

    from_port        = 0
    to_port          = 0
    protocol         = -1
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

Now, as we have the code/configuration ready, let us initiate and create the Security Groupby executing the terraform plan and applying the plan command.

Terraform – Dynamic Block

A Dynamic block is similar to for expression but creates nested blocks instead of a complex typed value. It iterates over a given complex value and generates a nested block for each element of that complex value. Unlike the count block, which will iterate over the resource, the dynamic will reside inside the resource block.

Dynamic blocks are supported inside the following blocks,

  • Resource
  • Data
  • Provider
  • Provisioner

Dynamic blocks can be used for resources like Setting, Security Group, etc.

Let us consider the example of creating a security group with multiple ingress rules for ports (80, 443, 3306, and 22). In an ideal scenario, we will end up copy-pasting the ingress rule for each port in the security group resource block.

Security Group (Without Dynamic Block)

// Security Group without Dynamic Block
resource "aws_security_group" "lamp_securitygroup_basic" {
  name        = "LAMP Security Group Basic"
  description = "LAMP Server Security Group Basic ${terraform.workspace}" // call the workspace 
  vpc_id      = data.aws_vpc.get_vpc_id.id
  tags        = local.common_tags
  // in-bound rules
  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  ingress {
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  ingress {
    from_port        = 22
    to_port          = 22
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  ingress {
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  // outbound rules
  egress {
    from_port        = 0
    to_port          = 0
    protocol         = -1
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

The above code will work perfectly fine, but as your security group rules go on increasing, you will need to copy-paste the ingress block for each rule, which will make the code hard to maintain and update.

To overcome this issue, Terraform provides us with a dynamic block. We can write the same code using dynamic block,

Security Group (Dynamic Block)

// Data block for VPC ID
data "aws_vpc" "get_vpc_id" {
  default = true
}
// variable declaration 
variable "port" {
  description = "Enter the ports to be configured in SG Ingress rule"
  default     = [80, 22, 443, 3306]
}
// Create Security Group uisng Dynamic Block
resource "aws_security_group" "lamp_sg" {
  name        = "LAMP Security Group"
  description = "LAMP Server Security Group ${terraform.workspace}" // call the workspace 
  vpc_id      = data.aws_vpc.get_vpc_id.id
  tags        = local.common_tags
  dynamic "ingress" {
    for_each = var.port
    content {
      description      = "Security rule for inbound ${ingress.value}"
      from_port        = ingress.value // The block should use block name to fetch the value and key
      to_port          = ingress.value
      protocol         = "tcp"
      cidr_blocks      = ["0.0.0.0/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
  }
  // outbound rules
  egress {
    from_port        = 0
    to_port          = 0
    protocol         = -1
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

In our example, we are producing a nested block of ingress rules,

  • The label of our dynamic block is “ingress”, you can change the label, as you per the requirement, like if you want to generate “setting” nested blocks we can name the dynamic block as “setting”.
  • For_each – Argument will provide the complex argument to iterate over.
  • Content Block – defines the body of each generated block. In our example it will be the ingress block, iterating over multiple ports.

In multi-level nested block, we can have multi-level nested dynamic blocks, ie the nested dynamic bock within the dynamic block.

AWS Stateful vs Stateless

Stateful vs Stateless

AWS Security Group is Stateful and ACL is Stateless, when we open any port in Security Group (Inbound) the same port will get opened in the Outbound and vice versa, the same is not true for ACL, even when you open any port in Inbound, you will need to explicitly open the same in outbound, that’s why ACL is Stateless.