BiWeekly GSoC Updates Blog Post

Data Intensive Research in Astronjomy and Astrophysics — Data-Intensive Research in Astronomy and Cosmology

In all my previous internships/contract-work I have worked with only the web stack and that too on the web front end specifically.
All through the past years, I have been part of creating large scale consumer-facing web apps mostly with Javascript and related frameworks like React, Vue, and Angular.

But when I started working with DiRAC’s Data Engineering team, I had to work on a totally different tech stack!

The Data Engineering team worked on Jupyter Lab and Jupyter Notebooks. They also worked mostly with Python/Go-related tech stack and frameworks and on very impressive DevOps stuff on AWS and also on their own EPYC servers.

Although I had experience in Python mostly from solving LeetCode questions, but not a development experience with Python Frameworks. And I never heard of the Jupyter ecosystem before working at DiRAC. And the biggest DevOps stuff I have had ever done was hosting my projects on simple hosting services like Netlify.

So all these terms like Jupyter Notebook, Jupyter Lab, EPYC, Pyspark, Tornado, were totally alien to me.

It was a new learning curve for me but definitely very interesting, intriguing, and challenging at the same time.

Also and most importantly, my mentors were super supportive and they communicated everything to the minutest details and helped me with links of all the necessary documentation and explained the workflows over video calls.

Well, apart from tech stack, I came to know a lot of people from DiRAC as we interacted over the weekly data engineering meetings.
I had interactions with Colin, Dino, Andy, Chris, and Brigitta who are working on the Data Engineering team.

Regarding my GSoC project, we are trying to build an MVP and launch a minimal version by the end of next week.