Member Technical Staff
Spark structured streaming, is one of the most popular frameworks used to build streaming pipelines today. However, configuring spark streaming pipelines can be challenging and can lead to a lot of resource wastage if done improperly. A few months back, we had built a tool Streaminglens which generates insights about the scalability limits of a spark streaming pipeline. The purpose of this project is to build new features in Streaminglens to make the tool more powerful.
* Adding new file reporting framework in Streaminglens (Bootstrap task)
* Adding Support for Pluggable AutoScaling in Streaminglens
* Adding Support to read metrics through Checkpoint Files
* Building Test Framework for the Project
Necessary Requirements: Prior Development Experience with any Objected Oriented Program Language (Java, Scala, etc)
* Knowledge / Development Experience with Scala
* Experience of writing Unit Test Frameworks in Java / Scala
* Basic knowledge of Spark / Spark Structured Streaming will be helpful.
The project will give hands-on experience of writing production-level code and understanding of Big Data and Streaming Systems.
Prior Development Experience with any Objected Oriented Program Language (Java, Scala, etc)
Project Duration (in Months)
Number of openings
Please share Github links of your existing projects.