Database Overview

admin
  • AWS database types
    • RDS – OLTP        
      • SQL
      • MySQL
      • PostgreSQL
      • Oracle
      • Aurora
      • MariaDB
    • No SQL Database
      • DynamoDB
    • RedShift – OLAP
    • Elastic Cache – In Memory Cache
      • MemCached
      • Redis
  • Encryption of the Database
    • Encryption at REST is supported for MySQL, Oracle, SQL, PostgreSQL, MariaDB and Aurora.
    • Encryption is done by AWS KMS service (Key Management Service).
    • Once the RDS instance is encrypted. The data stored at REST in the underlying storage is encrypted.
    • This is applicable for Automated, Read Replicas and Snapshots.
    • At present, Encryption of the existing database is not supported. To use Amazon RDS encryption for the existing database, you must first create a snapshot, make a copy of that snapshot and encrypt the copy. 

Cloud-front – CDN (Content Delivery Network)

admin
  • CDN is a content delivery network is a system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage and a content delivery server.
  • Amazon CloudFront, you use key pairs to create signed URLs for private content, such as when you want to distribute restricted content that someone paid for.
  • IAM users can’t create CloudFront key pairs. You must log in using root credentials to create key pairs.

  • CloudFront Key Terminologies
    • Edge location – This is the location where the content will be cached This is different than the AWS Region/AZ (Availability Zone).
    • We can see the available Edge location on AWS site.
    • Origin – This is the origin of the actual files, that CDN will distribute. This can be a S3 Bucket, an EC2 instance, an Elastic Load Balancer or Route53.
    • Distribution – This is the name given to CDN which has a collection of Edge Location.
    • Web Distribution – This used only for the Websites
    • RTMP – used for the Media files.

  • CDN Example
  • The requester will first hit the edge location and then the Edge location will contact the actual server.
  • The data will be cached at the Edge location till TTL and then will be passed to the requester.
  • The first request will be delayed one as it goes around through the requester – Edge – Server, but the next request will be faster as the data will be directly fetched from the Edge location.

CDN Usage

  • CDN can be used to deliver, the entire website, including static, dynamic, Streaming and Interactive content.
  • Request will be automatically routed to the nearest Edge location.
  • CDN is optimized to work with different AWS services,
    • S3
    • Elastic Load balancer
    • EC2
    • Route53
    • It also works seamlessly with non-AWS origin servers, which stores the original, definitive versions of your files.
  • Edge location are for both Read and Write operations i.e. we can put an object to an Edge location.
  • Every object cached has TTL, to clear the cache, it will be a chargeable service.
  • CDN provides two different types,
    • RTMP
    • Web Distribution
  • To restrict the user, select option pre-signed URL or signed cookies. This will restrict the user access to pre-verified URL and sharing the content will be restricted.
  • WAF (Web Application Firewall), this blocks the Cross scripting, SQL injections.
  • We can have shared SSL certificate or the dedicated one assigned to domain.
  • Geo restriction allows us to whitelist or blacklist the countries. We can either blacklist or whitelist. We cannot use both at the same time.
  • Create Invalidation is used to remove the object/content from the Edge location cache.

We can force to load the content through CloudFront, accessing the content directly through the S3 DNS URL will throw an error.

API Gateway Overview

admin

API Gateway

  • We have two type of the API
    • REST API – Representational State Transfer. It uses JSON. It is also used in AWS.
    • SOAP API – The existing one, it uses XML.
  • AS API Gateway is a fully manage services used by the developers to publish, maintain, monitor and secure API at any scale.
  • The code can be in EC2 or in serverless Lambda or DynamoDB.
  • It provides us HTTPS endpoints to define a RESTFUL API.
  • Each API end point can point to different targets.
  • It is scalable.
  • We can track and control usage using the API key.
  • Throttle request to prevent attack.
  • We can maintain multiple version of the API.
  • You can use cloudwatch to monitor the logs of API Gateway.
  • Although API gateway supports REST method, it do support legacy SOAP method. But that will be custom configuration.

Configuring the API

  • Define the API (Container)
  • Define the resources and nested resources (URL paths)
  • For each resources
    • Select the supported HTTP method
    • Set the security
    • Choose target (EC2, Lambda or DyanamoDB)
    • Set request and response transformation.
  • Deploy API to Stage
  • It will use the API gateway domain by default
  • You can use custom domain
  • Now supports AWS certificate manager (get free SSL/TLS certs)

API caching

  • It is very similar to cache, instead of hitting to the target, the frequent requests are cached at endpoint and if someone requests the same information, the cached information is sent back.
  • It has a TTL set.
  • This decreases the latency.

Same-origin policy

  • It is an important concept in web application security.
  • It allows first webpage scripts to access the second page script, only if they are in the same region.
  • This is managed and handled by the browsers
  • This helps in avoiding the XSS attacks
  • This is ignored by tool like Postman and CURL.

CORS Cross-Origin Resource Sharing

  • This is one way in which the server at one end can relax the same origin policy.
  • This will allow the restricted resource on one web application to be shared /requested from another domain outside the domain through which the request is been served.
  • Browser makes an HTTP OPTIONS call for an URL. OPTIONS are GET, PUT, POST
  • Server returns the option that says “These domains are approved to GET this URL”
  • Error – “Origin policy cannot be read at the remote resource “; you will need to enable CORS at the AI Gateway.

Advance API Gateway

  • We can use the API Gateway Import feature to import from an external definition into API gateway.
  • With import API you can either create a new API by submitting a POST request that includes swagger definition in the payload and end point configuration or you can update an existing API by using a PUT request that contains a swagger definition in payload.
  • You can update the existing API definition by overwriting it or you can merge a definition with existing one.
  • You can specify the options in mode query parameter in the request URL.

API throttling

  • By default, API gateway has a steady state request rate to 10000 requests per second.
  • The maximum concurrent request is 5000 request across all API with an account.
  • If the request exceeds 5000 or 10000 limit, you will get an error “429 too many request”

IAM Overview

admin

IAM (Identity Access Management)

  • It allows the management of the users and their level of access to the AWS console.
  • It gives centralized control to your AWS console.
  • We can assign Granular permission.
  • Shared access to AWS console.
  • Identity federation (You can integrate it with AD, Facebook, LinkedIn, etc.).
  • Multi-factor Authentication.
  • It also allows temporary access for users/devices and services where necessary (Web application and Mobile application)
  • You can set up your own password rotary policy.
  • Integrates with many different AWS services
  • Supports PCI DSS Compliance.
  • Critical terms
    • Users – The end users.
    • Roles – You create roles and then assign them to AWS resources.
    • Groups – A collection of users under one set of permissions.
    • Policies – documents that define one or more permissions, they can be applied to users/groups/roles. All can share the same policies. It sits on top of all.
  • IAM doesn’t have regions, it is applied universally.  It shows as Global.
  • The root account is the email address used to sign up for the AWS account and has full access.
  • MFA can be a virtual device or a physical device. For Android, it will be google authenticator.
  • Groups
    • You can create a custom group name.
    • Once the group is created you will need to attach the policy to the group.
    • The policy is pre-published by AWS.
    • Policy documents are in key-value pair (JSON) format.
  • Users
    • Users when created generates Access Key and Secret Access Key. Both are used when we connect AWS programmatically.
    • These are shown only once when you first time creates them in AWS.
    • If the keys are lost, it can be regenerated.
    • The password can be used with the email address to log in AWS console.

Permission can be directly attached to the users, so permission works for both groups and users.

Types of IAM policy

  • Managed Policy
  • Customer managed Policy
  • Inline Policy

Managed Policy

  • These are the policy created by AWS, we cannot update anything in these managed policies.

Custer managed Policy

  • Customer-managed policies are the ones created by customers, managed and updated by them. They can be attached only to their account.

Inline Policy

  • The inline policies are actually embedded within the user, group, and role to which it applies.
  • It has a strict one-to-one relationship between the entity and policy.


Kinesis Overview

admin
  • Streaming data, it’s a data generated continuously with thousands of sources, which sends the data in small size and record continuously. E.g. listed below,
    • Purchases from online store amazon.com
    • Stock price
    • Game data
    • Social network data
    • Geospatial data (Uber data) constantly sends the data about the cab and users location.
    • IOT data (Sensor Data).
  • Kinesis is a platform where we send the stream data, it helps to load and analyze the streaming data. It also helps in building your own custom applications as per the business needs.
  • Kinesis offers three core services
    • Kinesis stream
    • Kinesis Firehose
    • Kinesis Analytics
  • Kinesis retains the data for 24 hours and you can increase that to 7 days retention.
  • Kinesis Stream
    • They store the data in Shard, then the data is passed to EC2.
    • Once processed in EC2, the data can be moved further to different storage media like DynamoDB, S3, EMR and Redshift.
    • Kinesis streams consists of Shards.
    • 5 transactions per second to read.
    • 2 MB /Sec max data read rate.
    • 1000 records per second to write up to maximum 1 MB per second including the partition key.
    • The total capacity of the stream is the sum of the capacities of its Shard.
  • Kinesis Firehose
    • In Firehose, the data is acquired from different media like your personal laptop/S3 etc. and passed to Firehose, in Firehose it is analyzed and processed and then passed to S3. The data analytics is optional.
    • The data is retained for 24-hour default and can be extended till 7 days.
    • You can pass the data to Redshift but it should go through the S3. You can also pass the data to elastic search.
  • Kinesis analytics
    • It allows you to run the SQL queries; you can store the data of SQL further to S3/Redshift and Elasticsearch Cluster.
    • It is a web analysis tool used with SQL type query.

SWF (Simple Workflow Service)

admin
  • In SWF the task is assigned only once and never duplicated.
  • SWF coordinates work across distributed application components.
  • It helps in implementing complex business processes and work flows.
  • It is suitable for long running execution and not for short batch jobs etc.
  • Enables complex interaction within different application on different platform, AWS or on-premise infrastructure and between different users.
  • Maximum work flow can be 1 year and it is measured in seconds.
  • Features of SWF
    • Tasks executed with no duplicate
    • Routing and queueing of the task handled by the SWF.
    • It also provides timeout and status of the task.
    • The task workflow can have child process, and a list of hierarchy process.
    • It handles user data Input and execution result output.
  • SWF components
    • A workflow is control flow logic for execution of the task.
    • A domain contains of workflow or workflows.
    • Tasks can be performed by executable code, a webservices call or end user input. They can be performed parallel or serially.
    • Actors interact directly with SWF to coordinate tasks.
  • SWF Actors
    • Actors can be workflow starter, decider or activity workers
    • Starters can initiate the execution of work flow.
    • Deciders implement work flow logic and notify SWF changes during the workflow execution
    • Activity Workers perform activity tasks of the workflow.   
  • SWF Task
    • Must be registered using either the console or the RegisterActivityType action (API/CLI).
    • When scheduled you can specify a task list (queue)
    • Decision and Activity task has separate lists (queue)
    • Task can be assigned to activity workers through task routing if required.
  • Application communication can be established to SWF through,
    • SDK
    • SWF API HTTP POST
    • Flow Framework (Java or Ruby)
    • CLI
  • SWF and SQS
    • SWF presents a task oriented API and SQS offers a message oriented API
    • SWF ensures that the task is assigned only once and never duplicated, with SQS you may get duplicate message and you will need to ensure that a message is process only once.
    • SWF keeps tracks of all the tasks and events in an application, in SQS we should implement our own application level tracking especially if the application uses multiple queues.

SNS (Simple Notification Service)

admin
  • SNS is a push mechanism, as supposed to be SQS which is pull mechanism.
  • It pushes cloud notification directly to the mobile devices.
  • SNS can deliver the message via SMS/Email to SQS or any HTTP end point.
  • To avoid message loss, message published in the SNS are stored redundantly across multiple availability zone.  
  • SNS Topic
    • SNS allows to group multiple recipients using the topics. A topic is an access point for allowing recipients to dynamically subscribe for identical copies of the same notification.
    • One topic can support deliveries to multiple end point types – e.g. you can group together iOS, Android and SMS recipients.
  • SNS Benefits
    • Instantaneous push based delivery (no Pulling)
    • Simple API and easy integration with applications.
    • Flexible message delivery over multiple transport protocol.
    • Inexpensive, pay-as-you-go model with no up-front costs
    • Web-based AWS management console offers simplicity of a point-and-click interface.
  • SNS vs SQS
    • Both are messaging services in AWS.
    • SNS is Push
    • SQS is Pull
  • SNS Pricing
    • $0.50 per 1 million SNS Requests.
    • $0.06 per 100000 Notification deliveries over HTTP.
    • $0.75 per 100 notifications delivered over SMS.
    • $2.00 per 100000 notifications delivered over email.

Elastic Transcoder

admin
  • Elastic Transcoder is a media Transcoder in cloud.
  • Converts the original media files into different formats which will be compatible with smartphones, tablets and PCs. It optimizes the source media file.
  • Provides transcoding presets for popular output formats, which means we don’t need to worry about which setting works best for devices.
  • Pay based on the minutes and the resolution of the media which is transcoded.
  • How it works,
  • Upload the media file in the S3 bucket, Pass it to the lambda function, lambda function will invoke Elastic Transcoder.
  • Elastic Transcoders will then further process and convert it to different media format.
  • Then it is moved back to S3, it also creates thumbnail of the media file.

API Gateway

admin

API Gateway

  • API caching will cache the endpoint response. Thus, reducing the number of requests.
  • Feature of API Cache
    • Its low cost and Efficient
    • Scales Effortlessly and automatically.
    • Increase performance.
    • You can throttle requests to prevent requests.
    • Connects to cloud watch to log all requests.
  • Same origin policy
    • It is an important web application security model, under this policy a web browser permits scripts contained in a first web page to access data in a second web page, only if both web pages have the same origin (Which means same domain name).
  • CORS (Cross-Origin Resource Sharing)
    • CORS is on the way where the server at the other end (Not the client code in the browser) can relax the same origin policy.
    • This policy is a mechanism that allows restricted resources (e.g. font) on a web page to be requested from another domain outside the domain from which the first resource was served.
    • If you are using JavaScript/AJAX that uses multiple domains with API Gateway, ensure that you have enabled CORS on API Gateway.

The flow of the API Gateway or how it is implemented.

ECS – Elastic Container Service

admin

ECS is a container orchestration service that supports Docker and Windows containers. It has deep integration with AWS services IAM, VPC, and Route53.

ECS is a fully managed clustered platform that allows us to run the Docker images in the cloud.

  • The container is more like a virtual operating environment.
  • It has a standardized unit with everything the software needs to be run
  • It is also well suited for microservice, applications are created independently with stateless components

The architecture of Docker Container

Advantages of Container and Micro Services

  • Highly Scalable, we can scale only those resources which have max out, no need to scale out the whole application infrastructure.
  • Fault Tolerance, Error on one component doesn’t impact the other components.
  • Easy to maintain, Easier to maintain compared to the monolithic applications.

Fargate or EC2

  • We have the option to choose the container service,
  • The cluster of the virtual machines, ECS will allow us to run the container on the cluster of virtual machines. It provides you the control over installation, configuration and management of your environment.
  • Fargate can be used for serverless container where you don’t need to worry about the underlying EC2 instance.

ECR – ECR stores the container images, which will be used by the Docker.

ECS Application

  • ECS is used by AWS Sagemaker, LEX and Amazon.com which use ECS for the recommendation engine.
  • To prepare your application to run on Amazon ECS, you must create a task definition.
  • The task definition is a text file (in JSON format) that describes one or more containers (up to a maximum of ten) that form your application.
  • It specifies various parameters for your application. For example, these parameters can be used to indicate which containers should be used, which ports should be opened for your application, and what data volumes should be used with the containers in the task.
  • An Amazon ECS cluster is a logical grouping of tasks or services.