Cyber Security
About Client
Our client is a cyber security services provider company. Our client has created a Risk engine which will prepare a detailed report by analyzing different data received from different sources. Client has developed their proprietary script which used all the data received from different logs to analyze the data and generate final report for any kind of cyber threat or any kind of unauthorized accesses.
Business Context
Our client came to Aegis for following.
- Create a pipeline for processing data from streaming system as well as batch processing.
- Design data lake for generating final required report based on the different data inputs.
- Design and implement proper data handling policy, to align with different data handling policy.
- Design and implement proper security policy so only authorized process and person can access the data.
Solution Provided
- As per the data size we had to go with Bigdata technology stack. We used
- Kafka to handle the incoming streaming data.
- Nifi for processing real time data.
- Airflow for processing batched data.
- We used PySpark to develop our DAG and use the same in Nifi and Airflow.
- For database we used HBase and Hive.
- Data from Nifi and Airflow was pushed to Hive.
- Final processed data was stored in HBase.
- We helped them to setup full E2E pipeline and designing data lake.
- We created different spark codes which is used in their pipeline which handles all the data related modification and encryption/decryption for complying to various data policies like GDPR.
- We setup Apache Ranger required for data security.
