Project Name
Spark Streaminglens
Mentor Details
Name:
Abhishek Dixit
Organisation:
Qubole
Designation:
Member Technical Staff
Project Description
Spark structured streaming, is one of the most popular frameworks used to build streaming pipelines today. However, configuring spark streaming pipelines can be challenging and can lead to a lot of resource wastage if done improperly. A few months back, we had built a tool Streaminglens which generates insights about the scalability limits of a spark streaming pipeline. The purpose of this project is to build new features in Streaminglens to make the tool more powerful.
Key Objectives:
* Adding new file reporting framework in Streaminglens (Bootstrap task)
* Adding Support for Pluggable AutoScaling in Streaminglens
* Adding Support to read metrics through Checkpoint Files
* Building Test Framework for the Project
Necessary Requirements: Prior Development Experience with any Objected Oriented Program Language (Java, Scala, etc)
Preferred Requirements:
* Knowledge / Development Experience with Scala
* Experience of writing Unit Test Frameworks in Java / Scala
* Basic knowledge of Spark / Spark Structured Streaming will be helpful.
The project will give hands-on experience of writing production-level code and understanding of Big Data and Streaming Systems.
Programming Languages
Scala
Project Pre-requisites
Prior Development Experience with any Objected Oriented Program Language (Java, Scala, etc)
Project Duration (in Months)
1-2 months
Number of openings
1
Project Difficulty
Moderate
Additional Information
Github:
Proposal requirements
Please share Github links of your existing projects.