Author: admin

AWS Stateful vs Stateless

admin

Stateful vs Stateless

AWS Security Group is Stateful and ACL is Stateless, when we open any port in Security Group (Inbound) the same port will get opened in the Outbound and vice versa, the same is not true for ACL, even when you open any port in Inbound, you will need to explicitly open the same in outbound, that’s why ACL is Stateless.

EC2 – SSH access – Permission denied (publickey)

admin

Error – ec2-user@10.0.0.10: Permission denied (publickey).


Issue : We recently got an issue where we were not able to access the EC2 instance through SSH and got permission denied error (publickey).

The permissions of the pem file was correct, i.e 600. Upon further investigating the issue, we found the issue was with ownership of /home/ec2-user/.ssh/authorized_keys file, by default the file ownership should be ec2-user:ec2-user or ubuntu:ubuntu based upon the OS you are using. In our case the ownership of the file was changed, which blocked our access to ssh on the ec2 instance.

Solution: There are multiple fixes for such issues,

  • Access the ec2-instance through SSM, session manager through amazon console and update the ownership.
  • Run the SSM command on the respective instance to update the ownership of impacted file.

In both above solutions, we need SSM agent to be installed on the impacted instance, in our case the impacted instance didn’t had SSM agent installed.
To fix the issue, we used the below approach as we cannot use SSM command or session manager on the impacted instance.

  • We took snapshot of the impacted instance volume.
  • Stop the instance and update the user_data of the impacted instance with below details,

Content-Type: multipart/mixed; boundary=”//”
MIME-Version: 1.0

–//

Content-Type: text/cloud-config; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”cloud-config.txt”

# cloud-config
cloud_final_modules:
– [scripts-user, always]

–//
Content-Type:
text/x-shellscript; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash

chown root:root /home
chmod 755 /home
chmod 600 /home/ec2-user/.ssh/authorized_keys
chown ec2-user:ec2-user /home/ec2-user -R

–//

  • The above content will update user_data as well the cloud-config file.
  • The cloud config file is located at /var/lib/cloud/instance/cloud-config.txt
  • We also need to ensure that, all exiting user data are deleted or cleaned, as the above config update may trigger to execute the existing as well the new user_data, thus corrupting your application.
  • The old user data can be located at, /var/lib/cloud/instance/scripts/
  • On certain instance you may see symlinks with the instance id at the above script location. The user data are stored in files name’s part_001 etc.
  • If you don’t have any user data, just start the instance again and the new user_data will kick-in and update the ownership of /home/ec2-user/.ssh/authorized_keys file and thus allowing us to access the instance through ssh.

Follow the below steps only when you have user data impacting your application.

  • To delete the user data, create one more instance, and attach the impacted instance volume to the new instance.
  • Note- the new instance and the snapshot volume should be in the same availability zone.
  • Access the new instance through SSH, mount the volume using below command,

lsblk (will list all the drives)
mkdir /mnt
mount /dev/xvdf /mnt

  • Access the existing user_data at /var/lib/cloud/instance/scripts/, delete or move it some other location.
  • Detach the instance and reattach it to the impacted instance and start the instance.
  • The new user_data to update the ownership will kick in and will update the ownership of /home/ec2-user/.ssh/authorized_keys file, thus allowing you to login to the impacted instance.

Scrapy – Shell Commands

admin

Scrapy Shell Commands

Fetch Command

fetch('http://chaistudy.com')

We can also fetch the website using the below command, it will be same as the above fetch command

scrapy shell "yourwebsite.com"

Response, response command is to get the response from the scrapped content.

response.xpath() or response.css()

Number of items in response

len(response.xpath( '//*[@id="your_css_id"]'))

While using Python scrapy, the xpath is the best option and when we want to fetch the id or class of css, use the below format.

ID:

response.xpath('//*[@id="your_css_id"]')

Class:

response.xpath('//*[@class="your_css_class_name"]')

To Get the text,

response.xpath('//*[@class="your_css_class_name"]/text()')

Extract all data,

response.xpath('//*[@id="your_css_id"]/text()').extract()

response.xpath('//*[@id="your_css_id"]/text()').extract_all()

Extract First Element,

response.xpath('//*[@id="your_css_id"]/text()').extract_first()

YAML – Basics

admin

We can define key/value pair in simple format.

We can define key/value pair in simple format.

We can create list for objects as well for simple values.

Below userdata is the object and the starting with – are the list items of the object.

We can create simple list for object, here we are not using key pair but all values are passed as list.

Note: – represents list items

We can have a nested list, as shown below,

We can define list in three different types,

Method 1

Method 2

Method 3

Linux/Unix – sshd_config

admin

The entries in sshd_config can be updated, the file is genrally located at /etc/ssh/sshd_config.

Example

Match User john,mike
AuthenticationMethods publickey,privatekey

Note: The users should be separated with comma and should not have spaces in between.

The Match keyword will match user to restrict the arguments for the defined user.

In our case the AuthenticationMethod is applicable only for john and mike

Linux – Add User

admin

To Add User in Linux,

useradd -g johngroup-s /bin/bash -m -d /home/john -u 199999 john

The above command will create user John which has a specific uid 199999, under group johngroup along with a home folder in /home/john and will also add an entry in ssh /bin/bash for the respective user.

Command Arguments

  • -g To create user in specific group
  • -s To create user entry in ssh
  • -u user with specific uid
  • -m and -d to create with home directory

ElastiCache

admin
  • ElastiCache is a web service that makes it easy to deploy, operate and scale in-memory cache in the cloud.
  • ElastiCache improves the performance of the application by allowing retrieving the information from the fast, managed in-memory cache then the slow disk based database.
  • ElastiCache can improve the latency and throughput for the read-heavy application workloads or compute intensive work loads.
  • Caching improves application performance by storing critical piece of data in memory for low latency access.
  • ElastiCache has to be used with RDS
  • ElastiCache is very good choice if your database is particularly read-heavy and not prone to frequent change.
  • Redshift is very good for OLAP transactions.

Types of ElastiCache

  • MemCached
    • A widely adapted memory objects cache system. ElastiCache protocol is complaint with the MemCached.
    • All tools that work with MemCached environments will work in ElastiCache.
    • MemCache supports Multi-thread
    • MemCache doesnt have Multi-AZ capacity
  • Redis
    • A popular open-source in-memory key–value store that supports data structure such as sorted sets and list.
    • It supports Sorted set and list.
    • ElastiCache supports Master-Slave replication and Multi AZ which can be achieve cross AZ redundancy

Caching Strategy

Two types of strategies are available

  • Lazy loading
  • Write Throughput

Lazy Loading

  • Loads the data in cache only when required.
  • If the data is in cache, Elastic Cache will return the data or else it will return Null.
  • Lazy loading with TTL
    • Specify the number of seconds until the key or data expires to avoid keeping stale data in cache.
    • Lazy Loading treats an expired key as a cache miss and causes the application to retrieve the data from the database and write it back to the cache with a new TTL.
    • Lazy Loading does not eliminate the stale data but to avoid the stale data.

Write Through Cache

Write Through Cache adds and updates the cache whenever the data is written to the database.