Error: creating Auto Scaling Group : ValidationError: You must use a valid fully-formed launch template. Value () for parameter groupId is invalid. The value cannot be empty

As a DevOps engineer, we do encounter such cryptic error messages while launching infrastructures. In the below post, we will try to understand one such error while creating an Auto Scaling Group in AWS cloud, with a possible workaround.

Error:
Error: creating Auto Scaling Group : ValidationError: You must use a valid fully-formed launch template. Value () for parameter groupId is invalid. The value cannot be empty
│ status code: 400, request id: aac314dd-082f-4634-82ca-8bd6

Backgroud

We are trying to create an Auto Scaling group with a launch template.

# Auto Scale Group
resource "aws_autoscaling_group" "osc_asg_application" {
  name                = "osc-asg-application"
  max_size            = 1
  min_size            = 1
  desired_capacity    = 1
  vpc_zone_identifier = [aws_subnet.osc_application.id]
  launch_template {
    id      = aws_launch_template.osc_application_launch_template.id
    version = "$Latest"
  }
  tag {
    key                 = "name"
    value               = "osc-asg-application"
    propagate_at_launch = true
  }
}
# Launch Template
resource "aws_launch_template" "osc_application_launch_template" {
  name                   = "osc-application-launch-template"
  image_id               = data.aws_ami.cloudfront_ami.image_id
  key_name               = "demokey"
  instance_type          = "t2.micro"
  security_group_names = [aws_security_group.osc_security_group.id]
  block_device_mappings {
    device_name = "/dev/sdf"
    ebs {
      volume_size = 8
      volume_type = "gp3"

    }
  }

  dynamic "tag_specifications" {
    for_each = var.tag_resource
    content {
      resource_type = tag_specifications.value
      tags = {
        Name = "osc-application-resource"
      }
    }
  }

  tags = {
    Name = "osc-application-launch-template"
  }
}

When we planned and applied the above terraform code, it failed with the following error code.


Error: creating Auto Scaling Group : ValidationError: You must use a valid fully-formed launch template. Value () for parameter groupId is invalid. The value cannot be empty
│ status code: 400, request id: aac314dd-082f-4634-82ca-8bd6b9fe69a6

Upon investigating the issue futher, we found the security group argument passed in launch template was wrong. We had passed the argument security_group_names instead of vpc_security_group_ids.

We made the changes in the launch template and updated the security group argument.

The updated working launch template code is as below,

resource "aws_launch_template" "osc_application_launch_template" {
  name                 = "osc-application-launch-template"
  image_id             = data.aws_ami.cloudfront_ami.image_id
  key_name             = "demokey"
  instance_type        = "t2.micro"
  vpc_security_group_ids = [aws_security_group.osc_security_group.id]
  block_device_mappings {
    device_name = "/dev/sdf"
    ebs {
      volume_size = 8
      volume_type = "gp3"

    }
  }

  dynamic "tag_specifications" {
    for_each = var.tag_resource
    content {
      resource_type = tag_specifications.value
      tags = {
        Name = "osc-application-resource"
      }
    }
  }

  tags = {
    Name = "osc-application-launch-template"
  }
}

AWS EC2 – Unable to install/download packages from amazon repo to EC2 instance

We may have faced this issue on connecting to the Amazon repo to download/install packages, like Mysql, Apache, Nginx, etc.

We may get a connection time-out error when we run the sudo apt install mysql-server command. The error can occur for any package, in our example, we are installing a MySQL server.

Error

root@ip-10-9-9-58:/home/ubuntu# sudo apt install mysql-server

Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.91.65.63), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.207.133.243), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (54.87.19.168), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (54.165.17.230), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (3.87.126.146), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (3.209.10.109), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (18.232.150.247), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (34.201.250.36), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (34.237.137.22), connection timed out Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.73.36.184), connection timed out

The issue may occur in private or public instances. The most common solution is to first check the security groups assigned to your instance.

Steps

Login to your AWS console > EC2 > Security Groups,

Click on the Edit Outbound rules tab, then on Edit outbound rules.

Ensure we have HTTP TCP protocol for port 80 opened on outbound rules of the assigned security group.

If the rule for HTTP TCP Port 80 is missing, add the new rules in the similar format specified in the above image and save the changes.

Now, try to install the package, it should connect over the internet and install the package successfully.

Restricted Outbound Access

To solve the issue, above we have allowed all outbound traffic, in some cases due to security restrictions the organization may not allow you to open outbound traffic to all IP ranges.

The best practice says we should have minimum permissions.

To accomplish our security goals, we can restrict the outbound traffic to a certain Amazon repo mirrors IPs.

As we saw in the above error, Amazon tries to hit several of its mirrors to download and install the package. We need to copy any of these mirror IP addresses and use them to restrict outbound traffic in our security group.

root@ip-10-9-9-58:/home/ubuntu# sudo apt install mysql-server

Could not connect to us-east-1.ec2.archive.ubuntu.com:80 (52.91.65.63), connection timed out

In this example, we will open outbound only for IP 52.91.65.63.

Login to your AWS console > EC2 > Security Groups, select the assigned security group, and click on the Edit Outbound rules tab, then on Edit outbound rules.

Select the HTTP TCP port 80 rule and change 0.0.0.0/0 to 52.91.65.63/32. Save the changes, this will restrict the outbound rule for HTTP TCP port 80 to only one IP address, 52.91.65.63.

Note: We need to add a CIDR range even when we have one single IP address, Security Group does not allow us to add just a single IP address without a CIDR range. Even for a single IP address, we are required to add a CIDR block.

In our example for our single IP address, we have added a CIDR range /32.

You can change the CIDR block range based on your IP requirements.

AWS EC2 – Windows SSH – Permissions for public / SSH key are too open

We all may have encountered issues of bad permission for the public key while accessing the Linux/Ubuntu/Unix box through windows 10 systems.

This issue you may face while using a new set of public keys.

Error

E:\AWS\key>ssh -i ./my-key.pem ubuntu@10.0.0.1
The authenticity of host '10.0.0.1 (10.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:HxAA3hSzLSd/TRcZtXUYrjfZ0C9jL7fXmAZigM5p3+I.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.0.0.1' (ECDSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for './my-key.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "./my-key.pem": bad permissions
ubuntu@10.0.0.1: Permission denied (publickey).

The fix is pretty simple, we should just set the right permissions of the pem (public key) file. The only mistake we do while fixing the above issue is not granting permission to the correct user.

We need to first ensure we have the correct user details which we have used for our windows system login.

To verify the user details run the below command in your command prompt,

E:\AWS\key>whoami
desktop-4455tbos\myuser

Copy the user details, we will require these details in our later steps.

Steps to set the pem (public key) file permission

Browse and navigate to your public key directory.

Right-click on the key file name and click on properties.

Select the Security Tab and click on Advance.

On Advanced Security Setting Panel, click on “Disable inheritance

On the Block Inheritance Tab, Select “Remove all inherited permissions from the object

All Existing permission will be removed, ensure the permission Text Area has zero entries as shown below,

Now Click on the “Add” button, and you should get the pop-up to add permissions and user. Click on “Select Principal

On the “Select User or Group” panel, Enter the username we got earlier and click on “check names“.

You should be able to see your selected username. Once validated click on OK.

On Basic permission, select and check “Full control” and apply the changes.

You should be able to view your username with all permissions on the key property tab.

Now, let us connect to ssh,

ssh -i ./my-key.pem ubuntu@<your instance IP>
E:\keys>ssh -i ./my-key.pem ubuntu@10.0.0.1
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-1020-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

We should be able to connect to our instance.

EC2 – Instance user data fail – [WARNING]: Failed to run module scripts-user

Recently we faced an issue while executing user_data on my EC2 instance. This instance was created using Ubuntu 20.04 AMI. The user_data failed to execute the commands without any specific error message.

We tried different solutions available online like below,

  • Added / Removed #!/usr/bin/bash -x
  • Added / Removed #!/usr/bin/bash -v -e
  • Changed the interpreter of bash from #!/usr/bin/bash to #!/bin/bash

None of the above fixes worked for us, so we troubleshot it further by analyzing the user_data logs on the server.

The logs of user_data execution on an Ubuntu server can be found at, /var/log/cloud-init-output.log file.

The error recorded in the file /var/log/cloud-init-output.log was as below,

Cloud-init v. 22.2-0ubuntu1~22.04.3 running 'modules:final' at Fri, 30 Sep 2022 04:56:41 +0000. Up 24.86 seconds.
2022-09-30 04:56:41,635 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2022-09-30 04:56:41,635 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 22.2-0ubuntu1~22.04.3 finished at Fri, 30 Sep 2022 04:56:41 +0000. Datasource DataSourceEc2Local.  Up 25.26 seconds

Based on the errors recorded in the cloud-init-output.log file, we inspected the user_data recorded on the server. The user_data details on the host server can be located at,

  • The cloud-config file is located at /var/lib/cloud/instance/cloud-config.txt
  • The user data can be found at, /var/lib/cloud/instance/scripts/
  • On certain instances, you may see symlinks with the instance id at the above script location. The user data are stored in files name part_001 etc.

The inspection of user data captured on the host server showed spaces on the first line near the bash interpreter #!/bin/bash

As we can see in the below image, these spaces were the culprits in our case.

To fix the issue, we removed those spaces, re-published the user_data, and launched a new instance. The new instance executed the user_data without any failures.

The new user_data captured/recorded on the newly launched EC2 Instance does not have any spaces as we can see in the below image,

EC2 – SSH access – Permission denied (publickey)

Error ec2-user@10.0.0.10: Permission denied (publickey).


Issue : We recently got an issue where we were not able to access the EC2 instance through SSH and got permission denied error (publickey).

The permissions of the pem file was correct, i.e 600. Upon further investigating the issue, we found the issue was with ownership of /home/ec2-user/.ssh/authorized_keys file, by default the file ownership should be ec2-user:ec2-user or ubuntu:ubuntu based upon the OS you are using. In our case the ownership of the file was changed, which blocked our access to ssh on the ec2 instance.

Solution: There are multiple fixes for such issues,

  • Access the ec2-instance through SSM, session manager through amazon console and update the ownership.
  • Run the SSM command on the respective instance to update the ownership of impacted file.

In both above solutions, we need SSM agent to be installed on the impacted instance, in our case the impacted instance didn’t had SSM agent installed.
To fix the issue, we used the below approach as we cannot use SSM command or session manager on the impacted instance.

  • We took snapshot of the impacted instance volume.
  • Stop the instance and update the user_data of the impacted instance with below details,

Content-Type: multipart/mixed; boundary=”//”
MIME-Version: 1.0

–//

Content-Type: text/cloud-config; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”cloud-config.txt”

# cloud-config
cloud_final_modules:
– [scripts-user, always]

–//
Content-Type:
text/x-shellscript; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash

chown root:root /home
chmod 755 /home
chmod 600 /home/ec2-user/.ssh/authorized_keys
chown ec2-user:ec2-user /home/ec2-user -R

–//

  • The above content will update user_data as well the cloud-config file.
  • The cloud config file is located at /var/lib/cloud/instance/cloud-config.txt
  • We also need to ensure that, all exiting user data are deleted or cleaned, as the above config update may trigger to execute the existing as well the new user_data, thus corrupting your application.
  • The old user data can be located at, /var/lib/cloud/instance/scripts/
  • On certain instance you may see symlinks with the instance id at the above script location. The user data are stored in files name’s part_001 etc.
  • If you don’t have any user data, just start the instance again and the new user_data will kick-in and update the ownership of /home/ec2-user/.ssh/authorized_keys file and thus allowing us to access the instance through ssh.

Follow the below steps only when you have user data impacting your application.

  • To delete the user data, create one more instance, and attach the impacted instance volume to the new instance.
  • Note- the new instance and the snapshot volume should be in the same availability zone.
  • Access the new instance through SSH, mount the volume using below command,

lsblk (will list all the drives)
mkdir /mnt
mount /dev/xvdf /mnt

  • Access the existing user_data at /var/lib/cloud/instance/scripts/, delete or move it some other location.
  • Detach the instance and reattach it to the impacted instance and start the instance.
  • The new user_data to update the ownership will kick in and will update the ownership of /home/ec2-user/.ssh/authorized_keys file, thus allowing you to login to the impacted instance.

EC2 – Overview

Elastic Cloud Compute is a virtual server based on Windows /Linux Platform in AWS.

  • Amazon Elastic Compute Cloud has reduced the time to get the server instance up and boot in minutes. This allows us to quickly scale up/down the capacity as per the computing requirements.

EC2 Types

  • On Demand – This allows you to pay fixed rate by hours/Seconds with no upfront commitments. This not available in seconds for windows servers.
    • This is perfect for users that want low cost and flexibility of EC2. Without any upfront payments or commitments.
    • Applications with short term, spiky or unpredictable loads that cannot be interrupted.
    • Application developed and tested on EC2 for the first time.
  • Reserved – Provides with capacity reservations and offers discount on the hourly charge for an instance,  1to 3-year terms
    • Application with steady state or predictable usage.
    • Application that requires reserve capacity.
    • User can pay upfront to reduce the total computing cost further down.
  • Types of RI
    • Standard RIs – Up to 75% off on-demand RI.
    • Convertible RIs – Up to 54% off on-demand RI. It allows you to change the attributes of the RI if the exchange results in creation of Reserved Instance of equal or greater value.
    • Scheduled RIs – These are time bound RIs, it will be available during the reserve time window. This option allows you to match your capacity reservation to a predictable recurring schedule that only requires a fraction of a day, a week, or a month.
  • Spot – Enables you to Bid whatever price you want for instance capacity, providing for even greater saving if the applications have flexible start and end time (No Peak Hour).
    • Application having flexible end and start time.
    • Applications those are only feasible at very low computing price.
    • Users with an urgent need for large amount of additional computing capacity.
    • If spot instance is terminated by Amazon, you will not be charged for the partial hour. But if you terminate the instance, you will be charged for that complete hour.
  • Dedicated Host – Physical EC2 server dedicated for your use. Dedicated Hosts can help you to save the cost by allowing using the existing server bound software.
    • Useful for regulatory requirement that doesn’t support multi-tenant or cloud deployments
    • Great for licensing which doesn’t support multi-tenant or cloud deployments.
    • Can be purchased on-Demand (Hourly)
    • Can be purchased as reservation for up to 70% off the On-Demand.
  • Amazon EBS allows you to create storage volume and attach them to Amazon EC2 Instance. Once attach you can create file system on top of these volumes, you can run them as database or use them in any other way you would use a block device.
  • Amazon EBS is placed in a specific AZ, where they are automatically replicated to protect you from failure of a single component.
  • EBS volume types
    • General Purpose SSD (GP2)
      • General purpose balances both price and performance.
      • Ratio of 3 IOPS per 3 GB with up to 10000 IOPS and the ability to burst up to 3000 IOPS for extended period for volumes at 3334 Gib and above.
    • Provisioned IOPS SSD (101)
      • Designed for I/O Intensive applications such as large relational or NoSQL database.
      • Use if you need more than 10,000 IOPS
      • Can provision up to 20000 IOPS per Volume.
    • Throughput optimized HDD (ST1)
      • These cannot be a boot volume, these are additional volumes attached to the root volume.
      • Big data, data warehousing, log processing
    • Cold HDD (SC1)
      • Lowest cost storage for infrequent access workload.
      • File server.
    • Magnetic (Standard)
      • Lowest cost per GB of all EBS volume Type that is a bootable.
      • Magnetic volume is ideal where the work load were data are accessed infrequently and application where the lowest cost is important.
  • Each subnet is assigned to one AZ, you cannot have more than 1 subnets per AZ.
    One Subnet = One AZ
  • AMI (Amazon Machine Images) – are snapshots of different flavor virtual machines.
  • On EC2 Create page, advance section we can add boot strap scripts. It may contain the Unix Bash commands.
  • On Add Storage page, we can add the root EBS volume as well as the additional volumes. In root Volume we can add only General Purpose (GP2), Provisioned IOPS SSD (IO1) and Magnetic.
  • These root volumes are used to load the OS.
  • Volumes are nothing but virtual HDD.
  • Delete on termination if checked will delete the volume assigned upon termination of the EC2 Instance.
  • Security Groups are Virtual files. It defines the traffic allowed from your web servers, like which port/protocol is allowed or denied for the instance.
  • 0.0.0.0/0 will allow access to all.
  • Public key and private key pair will be shared upon creation of EC2 Instance. Public key is your Padlock and private key is the key to Pad Lock.
  • These key pair needs to be downloaded and saved, as it will not be shown later. You can regenerate the keys.
  • Encryption can be done only for the attached volume, we cannot encrypt the Root volume device.
  • The EBS root volume can be encrypted using the third-party tool like Bit Locker. This can be achieved while creating the Amazon AMI or through the API.
  • Login through SSH terminal for EC2 Instance,
  • To connect through the Windows system to AWS EC2 instance, we need to use Putty and Putty keygen.
  • The Private and Public key generated by AWS EC2 are in pem extension format, and for putty we need ppk, here we use putty keygen tool to convert the. pem file to. ppk file.
  • To connect to putty, use IP address (Public IP address) of your instance as hostname
  • Windows 10 provides SSH access through CMD, you can directly run the below command to SSH your EC2 instance using Pem file.
  • ssh -i <path to your .pem file> ec2-user@<your EC2 Public IP address>
  • In case of ubuntu as OS, replace ec2-user with ubuntu@<your EC2 Public IP address>
  • Upgrading EBS volume Type
    • You cannot have the EC2 instance in one availability zone and the EBS Volume in another availability zone. Both should be in the same AZ.
    • We cannot modify the volume for Standard Volume Type, standard is pure magnetic volume. All other volume type can be modified.
    • To change the AZ for any volume, we will need to create a snapshot first for that EC2 instance and then create a volume for the snapshot. While creating volume for the snapshot, you can change the volume type as well as the AZ for the new volume.
    • Snapshot also allows the EC2 instance to move from one region to another one. Create a snapshot and then move the snapshot to the new region.
    • Through snapshot we can create images. These images are stored under AMI.
    • To delete the AMI, select the images and click on deregister.
  • Volume vs Snapshot – Security
    • Snapshots of Encrypted volumes are Encrypted automatically.
    • Volumes restored from encrypted snapshots are encrypted automatically.
    • You can share snapshots but only when they are un-encrypted.
    • These snapshots can be shared with other AWS account or can be made public.
  • Additional Details
    • Volumes Exist on EBS
    • Volumes are nothing but Virtual HDD
    • Snapshot exists on S3, they are point in time copy of the volumes.
    • Snapshots are incremental – That means only those blocks that have changed since your last snapshot are moved to S3.
    • First snapshot takes time to load, as it will capture all data.
    • To create a snapshot for the EBS volumes that serve as a root device, you should stop the instance before taking the snapshot.
    • However, you can take snaps while instance is running.   
    • You can create AMIs from the EBS-backed instance and snapshots.
    • You can change the EBS volume sizes on the fly, including changing the size and volume type.
    • To move an EC2 Volume from one AZ/Region to another, take a snap or an image of it, then copy it to the new AZ/Region.

Elastic Load Balancer

  • AWS offers three different types of load balancers in AWS, they are as below,
    • Application Load Balancer
    • Network Load Balancer
    • Classic Load Balancer
  • Application Load Balancer – They are best suited for load balancing the HTTP and HTTPS traffic. They operate at layer 7 (TCP layers) and are application-aware. They are intelligent and can create advance request routing, sending specific requests to specific web servers.
  • Note –
    • One subnet is equal to one availability zone.
    • You will need to register the target (Load balancer) to the instance to assign it instance.
  • Network Load Balancer – They are required for load balancing where extreme performance is required. They operate at layer 4. They are capable of handling millions of requests per second while maintaining ultra-low latency.
  • Classic Load Balancer – They are legacy Elastic load balancers. It can be used for load balancing HTTP/HTTPS applications and it uses layer 7 specific features, such as X-forwarder and sticky sessions. You can use strict layer 4 load balancing for applications that purely rely on the TCP protocol.
    • Classic load balancer error 504, if your application throws error code 504 (Gateway Timeout). This means the application may have issues at the application layer or database layer.
  • X-Forwarded-For – This helps to identify the public IP address under the private cluster. This is part of the classic Load balancer. ELB can pass the public IPv4 address of the client to the EC2 instance.

  • Notes:
    • Load balancers do not provide any public IP address; instead, they will have a DNS name. You will need to update the DNS in cname instead of IP address.
    • Instances monitored on the ELB are reported as
      • In-service
      • Out-service
    • Health checks verify if the instance is healthy by talking to them.

AMI Types (EBS vs Instance Store)

  • All AMIs are either categorized as Amazon EBS or backed by an instance store.
  • For EBS volume – The root device for an instance launched from the AMI is an Amazon EBS volume created from an amazon EBS snapshot.
  • For Instances Store Volume – The root device is an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3.
  • AMI can be selected based on
    • Region (Region and Availability zone)
    • Operating Systems
    • Architecture (32 and 64 bits)
    • Launch permission
    • Storage for root device (Root Device Volume)
      • Instance Store (Ephemeral Storage)
      • EBS backend volume
  • Operating System
    • AWS provides a variety of OS, which we can select and install as per our requirements.
  • Architecture Type
  • Root Device Type
  • EBS vs Instance Store
    • Instance store volumes are sometimes called Ephemeral storage.
    • Instance store volumes cannot be stopped You will lose the data on this instance if it is stopped.
    • The EBS-backed instances can be stopped, without data loss.
    • You can reboot both types of the Instance, without data loss.
    • Upon termination, both Root volumes will be deleted. But in EBS volumes, we can tell AWS to keep the root device volume.