Category: AWS

Terraform Variable

admin
  • Variable helps in centralize and manage values in the configuration.
  • The variable block consist of
    • Type
    • Description
    • Default values.
  • We can create a file variable.tf or default file name terraform.tfvars. We can also specify flag for variable files.
  • For any other file name used to define variable, use the command flag, –var-file. You can specify more than one file, but they will be executed in sequence.
  • All environment variables are defined using prefix TF_VAR_<variable name>
  • Terraform as a built in variable called as PATH it’s basically used in modules.
  • We can have below variable types in Terraform
    • String – Can be Boolean True or False or simple string
    • Map – A collection type (Associate array or Hash Table, similar to dictonary of Python)
    • List – A collection type (List of item, starting with index Zero).
  • We can call variables four ways,Through command prompt
    • Environment variables
    • Default variables
    • Variable defined in files
  • Variables have name and optional parameters,
    • Type – if type is omitted then terraform consider it as by default string.
    • Default
    • Description
  • Variables definition,
  • To call the variable in terraform main file. We need to use the keyword var as prefix
  • Syntax : var.<variable_name>
  • Calling the variables in main terraform file,

AWS Terraform Authentication

admin
  • To run terraform on your local system for your AWS, you will need to first configure AWS on your local system.
  • Create a folder .aws/credential and store the AWS credential details at C:\Users\<yourhome folder>\.aws
  • You can do the above also through AWS command line > aws configure
  • You can define credentials in four different ways,
    • Static
    • Environment Variable
    • Shared Credential file
    • EC2 Role 
  • Static – it is basic method, we directly hard code the credentials in the .tf file provider section.
  • Environment Variable – Run the below command for windows machine, for Linux system use export commands.
  • Once configured the environmental variables, you can all the keys through provider sections.

  • Shared Credential files – You can access the credential files created using aws configure command. The file is by default at location $HOME/.aws/credentials.  You can call the shared file like below in the provider section.
  • EC2 Role – If running under EC2, then you can directly create and assign role to EC2, this avoids use of hard coded credentials.
  • Assume role – We can provide role ARN, terraform will attempt this role. You can call the role ARN in provider section.

DynamoDB Accelerator (DAX)

admin

DyanmoDB Accelerator (DAX)

  • It is a fully managed, clustered in-memory cache for DynamoDB.
  • It is only used for Read operation, can increase 10x Read operation performance.
  • Microsecond performance for millions of request per second.
  • Ideal for Read-heavy or Brusty workloads

How it works

  • DAX is a write through cache service, i.e. it writes the data in cache as well as the backend DynamoDB table.
  • Your application and point to the DAX cluster instead of the actual backend table.
  • If the data you are looking in cache, the data is returned back from the cache and not from the actual table.
  • If the item is not in the cache, the DAX will perform eventually consistent GetItem operation against the DynamoDB.
  • Retrieval of the data from the DAX reduces the read load from actual DynamoDB table.
  • It also helps in reducing the Provisioned read capacity.

Not Suitable for

  • It is suitable for eventually consistent read operation, so it will not work with the application using Strongly Consistent.
  • Write intensive application
  • Application with fewer read operations
  • Application that do not require microsecond read operations.

Database Overview

admin
  • AWS database types
    • RDS – OLTP        
      • SQL
      • MySQL
      • PostgreSQL
      • Oracle
      • Aurora
      • MariaDB
    • No SQL Database
      • DynamoDB
    • RedShift – OLAP
    • Elastic Cache – In Memory Cache
      • MemCached
      • Redis
  • Encryption of the Database
    • Encryption at REST is supported for MySQL, Oracle, SQL, PostgreSQL, MariaDB and Aurora.
    • Encryption is done by AWS KMS service (Key Management Service).
    • Once the RDS instance is encrypted. The data stored at REST in the underlying storage is encrypted.
    • This is applicable for Automated, Read Replicas and Snapshots.
    • At present, Encryption of the existing database is not supported. To use Amazon RDS encryption for the existing database, you must first create a snapshot, make a copy of that snapshot and encrypt the copy. 

Cloud-front – CDN (Content Delivery Network)

admin
  • CDN is a content delivery network is a system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage and a content delivery server.
  • Amazon CloudFront, you use key pairs to create signed URLs for private content, such as when you want to distribute restricted content that someone paid for.
  • IAM users can’t create CloudFront key pairs. You must log in using root credentials to create key pairs.

  • CloudFront Key Terminologies
    • Edge location – This is the location where the content will be cached This is different than the AWS Region/AZ (Availability Zone).
    • We can see the available Edge location on AWS site.
    • Origin – This is the origin of the actual files, that CDN will distribute. This can be a S3 Bucket, an EC2 instance, an Elastic Load Balancer or Route53.
    • Distribution – This is the name given to CDN which has a collection of Edge Location.
    • Web Distribution – This used only for the Websites
    • RTMP – used for the Media files.

  • CDN Example
  • The requester will first hit the edge location and then the Edge location will contact the actual server.
  • The data will be cached at the Edge location till TTL and then will be passed to the requester.
  • The first request will be delayed one as it goes around through the requester – Edge – Server, but the next request will be faster as the data will be directly fetched from the Edge location.

CDN Usage

  • CDN can be used to deliver, the entire website, including static, dynamic, Streaming and Interactive content.
  • Request will be automatically routed to the nearest Edge location.
  • CDN is optimized to work with different AWS services,
    • S3
    • Elastic Load balancer
    • EC2
    • Route53
    • It also works seamlessly with non-AWS origin servers, which stores the original, definitive versions of your files.
  • Edge location are for both Read and Write operations i.e. we can put an object to an Edge location.
  • Every object cached has TTL, to clear the cache, it will be a chargeable service.
  • CDN provides two different types,
    • RTMP
    • Web Distribution
  • To restrict the user, select option pre-signed URL or signed cookies. This will restrict the user access to pre-verified URL and sharing the content will be restricted.
  • WAF (Web Application Firewall), this blocks the Cross scripting, SQL injections.
  • We can have shared SSL certificate or the dedicated one assigned to domain.
  • Geo restriction allows us to whitelist or blacklist the countries. We can either blacklist or whitelist. We cannot use both at the same time.
  • Create Invalidation is used to remove the object/content from the Edge location cache.

We can force to load the content through CloudFront, accessing the content directly through the S3 DNS URL will throw an error.

API Gateway Overview

admin

API Gateway

  • We have two type of the API
    • REST API – Representational State Transfer. It uses JSON. It is also used in AWS.
    • SOAP API – The existing one, it uses XML.
  • AS API Gateway is a fully manage services used by the developers to publish, maintain, monitor and secure API at any scale.
  • The code can be in EC2 or in serverless Lambda or DynamoDB.
  • It provides us HTTPS endpoints to define a RESTFUL API.
  • Each API end point can point to different targets.
  • It is scalable.
  • We can track and control usage using the API key.
  • Throttle request to prevent attack.
  • We can maintain multiple version of the API.
  • You can use cloudwatch to monitor the logs of API Gateway.
  • Although API gateway supports REST method, it do support legacy SOAP method. But that will be custom configuration.

Configuring the API

  • Define the API (Container)
  • Define the resources and nested resources (URL paths)
  • For each resources
    • Select the supported HTTP method
    • Set the security
    • Choose target (EC2, Lambda or DyanamoDB)
    • Set request and response transformation.
  • Deploy API to Stage
  • It will use the API gateway domain by default
  • You can use custom domain
  • Now supports AWS certificate manager (get free SSL/TLS certs)

API caching

  • It is very similar to cache, instead of hitting to the target, the frequent requests are cached at endpoint and if someone requests the same information, the cached information is sent back.
  • It has a TTL set.
  • This decreases the latency.

Same-origin policy

  • It is an important concept in web application security.
  • It allows first webpage scripts to access the second page script, only if they are in the same region.
  • This is managed and handled by the browsers
  • This helps in avoiding the XSS attacks
  • This is ignored by tool like Postman and CURL.

CORS Cross-Origin Resource Sharing

  • This is one way in which the server at one end can relax the same origin policy.
  • This will allow the restricted resource on one web application to be shared /requested from another domain outside the domain through which the request is been served.
  • Browser makes an HTTP OPTIONS call for an URL. OPTIONS are GET, PUT, POST
  • Server returns the option that says “These domains are approved to GET this URL”
  • Error – “Origin policy cannot be read at the remote resource “; you will need to enable CORS at the AI Gateway.

Advance API Gateway

  • We can use the API Gateway Import feature to import from an external definition into API gateway.
  • With import API you can either create a new API by submitting a POST request that includes swagger definition in the payload and end point configuration or you can update an existing API by using a PUT request that contains a swagger definition in payload.
  • You can update the existing API definition by overwriting it or you can merge a definition with existing one.
  • You can specify the options in mode query parameter in the request URL.

API throttling

  • By default, API gateway has a steady state request rate to 10000 requests per second.
  • The maximum concurrent request is 5000 request across all API with an account.
  • If the request exceeds 5000 or 10000 limit, you will get an error “429 too many request”

IAM Overview

admin

IAM (Identity Access Management)

  • It allows the management of the users and their level of access to the AWS console.
  • It gives centralized control to your AWS console.
  • We can assign Granular permission.
  • Shared access to AWS console.
  • Identity federation (You can integrate it with AD, Facebook, LinkedIn, etc.).
  • Multi-factor Authentication.
  • It also allows temporary access for users/devices and services where necessary (Web application and Mobile application)
  • You can set up your own password rotary policy.
  • Integrates with many different AWS services
  • Supports PCI DSS Compliance.
  • Critical terms
    • Users – The end users.
    • Roles – You create roles and then assign them to AWS resources.
    • Groups – A collection of users under one set of permissions.
    • Policies – documents that define one or more permissions, they can be applied to users/groups/roles. All can share the same policies. It sits on top of all.
  • IAM doesn’t have regions, it is applied universally.  It shows as Global.
  • The root account is the email address used to sign up for the AWS account and has full access.
  • MFA can be a virtual device or a physical device. For Android, it will be google authenticator.
  • Groups
    • You can create a custom group name.
    • Once the group is created you will need to attach the policy to the group.
    • The policy is pre-published by AWS.
    • Policy documents are in key-value pair (JSON) format.
  • Users
    • Users when created generates Access Key and Secret Access Key. Both are used when we connect AWS programmatically.
    • These are shown only once when you first time creates them in AWS.
    • If the keys are lost, it can be regenerated.
    • The password can be used with the email address to log in AWS console.

Permission can be directly attached to the users, so permission works for both groups and users.

Types of IAM policy

  • Managed Policy
  • Customer managed Policy
  • Inline Policy

Managed Policy

  • These are the policy created by AWS, we cannot update anything in these managed policies.

Custer managed Policy

  • Customer-managed policies are the ones created by customers, managed and updated by them. They can be attached only to their account.

Inline Policy

  • The inline policies are actually embedded within the user, group, and role to which it applies.
  • It has a strict one-to-one relationship between the entity and policy.


Kinesis Overview

admin
  • Streaming data, it’s a data generated continuously with thousands of sources, which sends the data in small size and record continuously. E.g. listed below,
    • Purchases from online store amazon.com
    • Stock price
    • Game data
    • Social network data
    • Geospatial data (Uber data) constantly sends the data about the cab and users location.
    • IOT data (Sensor Data).
  • Kinesis is a platform where we send the stream data, it helps to load and analyze the streaming data. It also helps in building your own custom applications as per the business needs.
  • Kinesis offers three core services
    • Kinesis stream
    • Kinesis Firehose
    • Kinesis Analytics
  • Kinesis retains the data for 24 hours and you can increase that to 7 days retention.
  • Kinesis Stream
    • They store the data in Shard, then the data is passed to EC2.
    • Once processed in EC2, the data can be moved further to different storage media like DynamoDB, S3, EMR and Redshift.
    • Kinesis streams consists of Shards.
    • 5 transactions per second to read.
    • 2 MB /Sec max data read rate.
    • 1000 records per second to write up to maximum 1 MB per second including the partition key.
    • The total capacity of the stream is the sum of the capacities of its Shard.
  • Kinesis Firehose
    • In Firehose, the data is acquired from different media like your personal laptop/S3 etc. and passed to Firehose, in Firehose it is analyzed and processed and then passed to S3. The data analytics is optional.
    • The data is retained for 24-hour default and can be extended till 7 days.
    • You can pass the data to Redshift but it should go through the S3. You can also pass the data to elastic search.
  • Kinesis analytics
    • It allows you to run the SQL queries; you can store the data of SQL further to S3/Redshift and Elasticsearch Cluster.
    • It is a web analysis tool used with SQL type query.

SWF (Simple Workflow Service)

admin
  • In SWF the task is assigned only once and never duplicated.
  • SWF coordinates work across distributed application components.
  • It helps in implementing complex business processes and work flows.
  • It is suitable for long running execution and not for short batch jobs etc.
  • Enables complex interaction within different application on different platform, AWS or on-premise infrastructure and between different users.
  • Maximum work flow can be 1 year and it is measured in seconds.
  • Features of SWF
    • Tasks executed with no duplicate
    • Routing and queueing of the task handled by the SWF.
    • It also provides timeout and status of the task.
    • The task workflow can have child process, and a list of hierarchy process.
    • It handles user data Input and execution result output.
  • SWF components
    • A workflow is control flow logic for execution of the task.
    • A domain contains of workflow or workflows.
    • Tasks can be performed by executable code, a webservices call or end user input. They can be performed parallel or serially.
    • Actors interact directly with SWF to coordinate tasks.
  • SWF Actors
    • Actors can be workflow starter, decider or activity workers
    • Starters can initiate the execution of work flow.
    • Deciders implement work flow logic and notify SWF changes during the workflow execution
    • Activity Workers perform activity tasks of the workflow.   
  • SWF Task
    • Must be registered using either the console or the RegisterActivityType action (API/CLI).
    • When scheduled you can specify a task list (queue)
    • Decision and Activity task has separate lists (queue)
    • Task can be assigned to activity workers through task routing if required.
  • Application communication can be established to SWF through,
    • SDK
    • SWF API HTTP POST
    • Flow Framework (Java or Ruby)
    • CLI
  • SWF and SQS
    • SWF presents a task oriented API and SQS offers a message oriented API
    • SWF ensures that the task is assigned only once and never duplicated, with SQS you may get duplicate message and you will need to ensure that a message is process only once.
    • SWF keeps tracks of all the tasks and events in an application, in SQS we should implement our own application level tracking especially if the application uses multiple queues.

SNS (Simple Notification Service)

admin
  • SNS is a push mechanism, as supposed to be SQS which is pull mechanism.
  • It pushes cloud notification directly to the mobile devices.
  • SNS can deliver the message via SMS/Email to SQS or any HTTP end point.
  • To avoid message loss, message published in the SNS are stored redundantly across multiple availability zone.  
  • SNS Topic
    • SNS allows to group multiple recipients using the topics. A topic is an access point for allowing recipients to dynamically subscribe for identical copies of the same notification.
    • One topic can support deliveries to multiple end point types – e.g. you can group together iOS, Android and SMS recipients.
  • SNS Benefits
    • Instantaneous push based delivery (no Pulling)
    • Simple API and easy integration with applications.
    • Flexible message delivery over multiple transport protocol.
    • Inexpensive, pay-as-you-go model with no up-front costs
    • Web-based AWS management console offers simplicity of a point-and-click interface.
  • SNS vs SQS
    • Both are messaging services in AWS.
    • SNS is Push
    • SQS is Pull
  • SNS Pricing
    • $0.50 per 1 million SNS Requests.
    • $0.06 per 100000 Notification deliveries over HTTP.
    • $0.75 per 100 notifications delivered over SMS.
    • $2.00 per 100000 notifications delivered over email.