In many data centers, a different type of server generates a large amount of data (events, in this case, is the status of the server in the data center) in real-time. There is always a need to process these data in real-time and generate insights that will be used by the server/data center monitoring people and they have to track these server's status regularly and find the resolution in case of issues occurring, for better server stability. Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.

Hence we want to build the real-time data pipeline using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django, and Flexmonster on Docker to generate insights out of this data. The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker. Data visualization is built using the Django web framework and Flexmonster.