Category: S3 (Simple Storage Service)

S3

AWL CLI Paging

admin
  • Paging controls the number of output shown on the CLI prompt, when we run the cli commands.
  • The default page size is 100
  • If you run a command which has 300 object, CL will make three different API calls but the output will show all in same cmd window as a single API. You may need to pass the next token value depending upon the command you run.
  • In certain cases if the object exceeds 1000 limit, we may get time out error.
  • To fix this error we will use the flag –page-size and provide the number less than the limit.
  • Still the CLI will fetch all records but with more number of API calls fetching records in smaller number.
  • Use –max-item flag to limit the CLI output

CLI paging commands

As shown below example, we are using paging to fetch the s3 bucket list.

  • aws s3api list-objects --bucket <YOUR_BUCKET_NAME>
  • aws s3api list-objects --bucket <YOUR_BUCKET_NAME> --page-size 5
  • aws s3api list-objects --bucket <YOUR_BUCKET_NAME> --max-items 1

Cloud-front – CDN (Content Delivery Network)

admin
  • CDN is a content delivery network is a system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage and a content delivery server.
  • Amazon CloudFront, you use key pairs to create signed URLs for private content, such as when you want to distribute restricted content that someone paid for.
  • IAM users can’t create CloudFront key pairs. You must log in using root credentials to create key pairs.

  • CloudFront Key Terminologies
    • Edge location – This is the location where the content will be cached This is different than the AWS Region/AZ (Availability Zone).
    • We can see the available Edge location on AWS site.
    • Origin – This is the origin of the actual files, that CDN will distribute. This can be a S3 Bucket, an EC2 instance, an Elastic Load Balancer or Route53.
    • Distribution – This is the name given to CDN which has a collection of Edge Location.
    • Web Distribution – This used only for the Websites
    • RTMP – used for the Media files.

  • CDN Example
  • The requester will first hit the edge location and then the Edge location will contact the actual server.
  • The data will be cached at the Edge location till TTL and then will be passed to the requester.
  • The first request will be delayed one as it goes around through the requester – Edge – Server, but the next request will be faster as the data will be directly fetched from the Edge location.

CDN Usage

  • CDN can be used to deliver, the entire website, including static, dynamic, Streaming and Interactive content.
  • Request will be automatically routed to the nearest Edge location.
  • CDN is optimized to work with different AWS services,
    • S3
    • Elastic Load balancer
    • EC2
    • Route53
    • It also works seamlessly with non-AWS origin servers, which stores the original, definitive versions of your files.
  • Edge location are for both Read and Write operations i.e. we can put an object to an Edge location.
  • Every object cached has TTL, to clear the cache, it will be a chargeable service.
  • CDN provides two different types,
    • RTMP
    • Web Distribution
  • To restrict the user, select option pre-signed URL or signed cookies. This will restrict the user access to pre-verified URL and sharing the content will be restricted.
  • WAF (Web Application Firewall), this blocks the Cross scripting, SQL injections.
  • We can have shared SSL certificate or the dedicated one assigned to domain.
  • Geo restriction allows us to whitelist or blacklist the countries. We can either blacklist or whitelist. We cannot use both at the same time.
  • Create Invalidation is used to remove the object/content from the Edge location cache.

We can force to load the content through CloudFront, accessing the content directly through the S3 DNS URL will throw an error.