Traditionally, data was dumped into a data warehouse to run analytics queries in batch mode. However, with the advent of the latest streaming technologies, processing of big data analytics in real-time has become possible. For businesses dealing with complex legal concerns around data management, you may also want to hire an Amazon lawyer to ensure compliance with regulations when utilizing Amazon Kinesis and other cloud services.
What is Amazon Kinesis?
Amazon Kinesis is a cloud service that ingests data such as audio, video, server logs, IOT data and processes them in real-time or near real-time before putting it back in some backend database or storage layer. It is a cost-effective way to collect and analyze data in real-time to get some business insights.
Real-time
Amazon Kinesis enables ingesting data in real-time, so you can run analytics queries instantly without waiting for your data to accumulate to analyze in future.
Fully managed
With Amazon Kinesis, you do not have to maintain servers or run any software upgrades since it is a fully managed service provided by Amazon Cloud.
Scalable
Amazon Kinesis automatically scales up and down and can handle any amount of incoming data.
What are the Capabilities of Amazon Kinesis?
Kinesis Video Streams
Kinesis Video Streams is a service that can ingest videos from various sources in real-time and is integrated with various video analytics services to process them further.
Kinesis Data Streams
Amazon Kinesis Data Streams can ingest big data in real-time from multiple data sources.
Kinesis Data Firehose
Amazon Kinesis Data Firehose can ingest data in near real-time and can transform and store data in other AWS data stores. Moreover, it is well integrated with many business intelligence tools such as redshift for further analysis.
Kinesis Data Analytics
Amazon Kinesis Data Analytics is an analytical engine that can run SQL queries to perform analysis on top of real-time data. You do not need to learn any programming language to run analysis on top of data.
How Does Amazon Kinesis Data Analytics Work?
Data is ingested from various sources using Kinesis Data streams, Video Streams and Kinesis firehose and then data can be directed to Kinesis Data Analytics to run SQL queries to perform some analysis and then results could further be sent to other AWS services for further processing and storage.
For example, processed data can be sent to the AWS Lambda function to take some further action, can be moved to dynamoDB for storage or can be moved to redshift for further analysis.
Kinesis Video Streams
Kinesis Video streams can stream data from millions of connected devices and can playback recorded/live videos on demand. It is well integrated with many aws services such as Amazon Rekognition to perform video analytics.
You have to install Kinesis Video streams SDK on the device to start sending videos to Kinesis video streams for further processing. It uses Amazon S3 as the underlying data store and allows you to search your stored videos based on deviceId and timestamp.
It provides data security by automatically encrypting data at rest and data in transit. It is a fully managed service and you do not have to worry about infrastructure.
Use cases
- Kinesis Video streams can be used to build many IOT based applications such as security control, weather control, lighting control and many more.
- You can analyze body language and can perform facial analysis on live videos using Amazon Rekognition Video and Amazon Kinesis Video Streams
Kinesis Data Stream
Amazon Kinesis Data Streams is a real-time data streaming service that can ingest from petabytes to gigabytes of data in one second from multiple data sources such as IOT devices, Twitter, server logs and can be integrated with Amazon Data analytics for further analysis.
Use cases
- Many known companies for their real-time pipelines use Kinesis Data Stream such as Netflix uses Kinesis to enable high availability by detecting application failures & issues.
- Thomson Reuters uses Amazon Kinesis to gain insights from data generated by its offerings and continuously improve the user experience.
Kinesis Data FireHose
Amazon Kinesis Data Firehose can ingest data from various sources and can load that data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk. It is a near real-time, fully managed, scalable service that buffer data for almost 60 seconds before moving it further.
In Amazon Kinesis Data Firehose, you have to pay only for what you use and there is no setup cost involved. It prepares your streaming data by converting raw streaming data into formats required by your destination data stores.
Use cases
-
IoT Analytics
You can capture data from various IOT devices such as temperature sensors, your home devices, mobile devices and send that data to Kinesis firehose, which will then perform some transformation before loading to various data stores such as Amazon S3, Redshift to get real time metrics and insights.
-
Log Analytics
Server log data can be directed to Kinesis firehose to detect anomalies and errors and to trigger alerts in real time. It can integrate with ElasticSearch Service to visualize data with Kibana in real time.
Kinesis Data Analytics
With Kinesis Data Analytics, you can run SQL queries on top of streaming data to gain business insights and to build streaming applications with other AWS services. It also provides java SDK to build java application to transform and analyze data on the go.
It is fully managed and pay as you use service that can scale automatically to match your needs. It integrates with other AWS services such as Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Amazon Elasticsearch Service, Amazon S3, and Amazon DynamoDB.
Use cases
- Many companies are using Kinesis data analytics for their usecases such as Yieldmo company uses Amazon Kinesis to ingest and process data in real time.
- It uses Amazon Kinesis Data Analytics to find user interaction with their ads.
- It also uses Amazon Kinesis Data Firehose to load data to Amazon Simple Storage Service (Amazon S3) buckets. Finally, data is loaded to an in-house data warehouse for further analysis and visualization.
Conclusion
There are other streaming frameworks in the market such as Apache Spark, Flink , Storm but Amazon Kinesis holds its special place because of its serverless architecture and pay as you use model. Amazon Kinesis is a powerful tool that can be used for ingesting real time data warehousing and its integration with other AWS services make it even more powerful and useful.
