Streaming data, it’s a data generated continuously with thousands of sources, which sends the data in small size and record continuously. E.g. listed below,
Purchases from online store amazon.com
Stock price
Game data
Social network data
Geospatial data (Uber data) constantly sends the data about the cab and users location.
IOT data (Sensor Data).
Kinesis is a platform where we send the stream data, it helps to load and analyze the streaming data. It also helps in building your own custom applications as per the business needs.
Kinesis offers three core services
Kinesis stream
Kinesis Firehose
Kinesis Analytics
Kinesis retains the data for 24hours and you can increase that to 7 days retention.
Kinesis Stream
They store the data in Shard, then the data is passed to EC2.
Once processed in EC2, the data can be moved further to different storage media like DynamoDB, S3, EMR and Redshift.
Kinesis streams consists of Shards.
5 transactions per second to read.
2 MB /Sec max data read rate.
1000 records per second to write up to maximum 1 MB per second including the partition key.
The total capacity of the stream is the sum of the capacities of its Shard.
Kinesis Firehose
In Firehose, the data is acquired from different media like your personal laptop/S3 etc. and passed to Firehose, in Firehose it is analyzed and processed and then passed to S3. The data analytics is optional.
The data is retained for 24-hour default and can be extended till 7 days.
You can pass the data to Redshift but it should go through the S3. You can also pass the data to elastic search.
Kinesis analytics
It allows you to run the SQL queries; you can store the data of SQL further to S3/Redshift and ElasticsearchCluster.
It is a web analysis tool used with SQL type query.
