Project Name

Spark Streaminglens

Mentor Details

Name: 

Abhishek Dixit

Organisation:

Qubole

Designation:

Member Technical Staff

Project Description

Spark structured streaming, is one of the most popular frameworks used to build streaming pipelines today. However, configuring spark streaming pipelines can be challenging and can lead to a lot of resource wastage if done improperly. A few months back, we had built a tool Streaminglens which generates insights about the scalability limits of a spark streaming pipeline. The purpose of this project is to build new features in Streaminglens to make the tool more powerful.

Key Objectives:
* Adding new file reporting framework in Streaminglens (Bootstrap task)
* Adding Support for Pluggable AutoScaling in Streaminglens
* Adding Support to read metrics through Checkpoint Files
* Building Test Framework for the Project

Necessary Requirements: Prior Development Experience with any Objected Oriented Program Language (Java, Scala, etc)

Preferred Requirements:
* Knowledge / Development Experience with Scala
* Experience of writing Unit Test Frameworks in Java / Scala
* Basic knowledge of Spark / Spark Structured Streaming will be helpful.

The project will give hands-on experience of writing production-level code and understanding of Big Data and Streaming Systems.

Programming Languages

Scala

Project Pre-requisites

Prior Development Experience with any Objected Oriented Program Language (Java, Scala, etc)

Project Duration (in Months)

1-2 months

Number of openings

1

Project Difficulty

Moderate

Additional Information

Proposal requirements

Please share Github links of your existing projects.

Have questions or feedback? Interested in working with us?  Email us at connectinternlink@gmail.com