Google Summer of Code 2022 — Building, Testing, and Wrapping Helioviewer API in a Python package

Hello everyone! This summer my project proposal to OpenAstronomy got accepted. I recently posted a series of blogs sharing my GSoC experience and the work I have done. This blog post details my experience of participating in the program and the contributions that I made to the Project-Helioviewer. If you haven’t read my previous blog posts, no worries, I will try to sum up all of them in this blog.

Starting Point

I started contributing to open-source projects in September 2021, my friend Vivek Agrawal told me about the Google Summer of Code program, and since then I had an eye for GSoC 2022. I did all the research and through the timeline, I knew that on the 7th of March Google will release the list of accepted organizations participating as mentors in 2022, I continued building my developer skills.

On March 7th, I searched for organizations that matched my interest, i.e, science. When I saw openAstronomy at that moment I was like: what the heck? astronomy and open source together? 👾👾👾

I set my goal, focused only on one project and started to understand the project idea. I joined the mailing list and connected with the project mentors and talked to them about my implementation ideas. While I prepared the proposal, I also made the following contributions to SunPy:

#5980 (code) and #6024 (updating docs), I wrote a test case for SunPy.timeseries.
#5961 adds an example to show “how to reproject a map to a map projection with a custom origin”.

In the one month before the results, to learn more about software testing and get familiar with pytest, I searched for issues labeled “Tests” in SunPy and submitted this pull request🚀.

The Community Bonding Period

After getting selected for the program, I continued my contribution to SunPy. I worked on a python script that updates vendor code automatically in this pull request. In a meeting, I met my mentor Nabil Freij, Jack Ireland, and Daniel Garcia Briseno and we discussed the project plan.

Introduction

In this program, I wrapped the Helioviewer API in a Python package. The development of this project is supported by the 2022 Google Summer of Code, the umbrella OpenAstronomy, SunPy Project, and the Helioviewer Project.

HvPy

Helioviewer.org is part of the Helioviewer Project, an open-source project for the visualization of solar and heliospheric data. The Helioviewer Project is funded by ESA and NASA.

HvPy is a python interface for interacting with Helioviewer’s Web API. This means all API calls are done by sending an HTTP request and parsing the response. Since this action is common for all API requests, it is encapsulated by the API core in HvPy. Certain parameters are provided to the core via an instance of HvpyParameters. This instance contains all information necessary to perform the API request. It contains the target URL, the input parameters, and the expected output type. Using this information, the HTTP Requester will be able to perform the request and coerce the result into the desired type.

Request Workflow

Each request takes a set of Input Parameters. The metadata defined in the HvpyParameters class is needed for an API request. It contains the InputParameters , expected OutputType and the API URL for the request. Using this information, the module core performs the HTTP request and parses the result into the expected type.

Front-end

Much of the design mentioned above is for internal use when adding new API endpoints or modifying the internal behavior of this module. The internal design allows for ease of adding new APIs, validating input parameters, and keeping the code DRY.

The actual front end that users will interface with lives in facade.py as it is the façade that hides this internal design. This module contains the API interface in its simplest form. It is responsible for taking user input, constructing the HvpyParameters instance, and passing it along to the core to perform the request.

Utils

We got clean and simple front-end functions for every endpoint. For some endpoint to make a query you need to design a string called LayerString and EventString which looks like: —

layers = "[9,1,100],[19,1,50]"
events = "[AR,all,1],[CD,all,1]"

Right, it is difficult to write this string for a beginner. So to save users from this difficult process we have these utility functions.

Creating the layer string
— This function takes a list of tuples of the form (Source id, Opacity). We defined an Enum for the users to get the source id of the required data source. Visit here to see all the possible Data Sources supported by Helioviewer. Let's see how we create a layer string for AIA_131 with 100% opacity along with an HMI Magnetogram with 50% opacity.

>>> from hvpy import DataSource, create_layers
>>> create_layers([(DataSource.AIA_131, 100), (DataSource.HMI_MAG, 50)])
‘[9,1,100],[19,1,50]’

Creating the events string
— This function takes a list of tuples of the form (event type, recognition methods). We defined an Enum for the users to get event types. Here is a list of all the supported Event Types by Helioviewer. To generate an event string for Active Region and Coronal Dimming we do: —

>>> from hvpy import EventType, create_events
>>> create_events([EventType.ACTIVE_REGION,
EventType.CORONAL_DIMMING])
‘[AR,all,1],[CD,all,1]’

Saving the file
— Some endpoint in response returns binary data. It may be JPEG, JPEG2000, mp4, PNG, etc. To write these binary data in your harddisk we have a helper function called save_file. This function takes in the following arguments: —

save_file(data: bytearray, filename: Union[Path, str], overwrite: bool = False)

The overwrite flag will overwrite the existing file with the same name and Path.

Helpers

To create a movie (mp4, WebM, Flv) we need to query the queueMovie endpoint and then use source id of the response to download the movie using the downloadMovie endpoint. To automate this process we have a helper function called createMovie.

Here is the function in use: —

https://medium.com/media/b982e24dd91b304503327c9a6e2d377f/href

And this is the result 👇

2022-9-1_2022-9-5.mp4

Week 10

In week 9 we decided to use the beta URL for query and by the end of the week, it is ready for testing (for more details read my week 7–9 blog post).

So the next task was to implement the remaining endpoint backends and bring them to the façade. I opened the following PRs for the remaining endpoints.

#57 — downloadMovie (this takes in a movie IDand returns binary data).
#58 — shortenURL (shorten a Helioviewer.org URL with the bit.ly URL shortening web service) and getNewsFeed (get the XML RSS feed of the official Helioviewer Project Blog).
#59 — getTile (requests a single image tile to be used in the Helioviewer.org Viewport. Tiles are 512x512 pixel PNG images, generated for a given image scale from the intermediary JPEG2000 image files).

Yay!! the last endpoint got merged.

There is a huge list of data sources to capture the sun image, for example, SDO, SOHO, Hinode, etc. We use these data source IDs to query the server. So to keep the process user-friendly we decided to store all the IDs in a Enum. This way one needs to remember the data source ID.

#61 — Adds Enum for the DataSource.

Week 11

After we have the DataSource Enum, I implemented the create_layers function that we talked about above.

#65 — Function to generate the layer string.

We also have a list of feature/event types, for example, Active region, Bright point, Flare, etc. Each event types have a Feature code. We did the same with Event Types as we did with the data sources. We decided to store all the feature code in a Enum.

#70 — Adds Enum for EventTypes.

Week 12

Now we got the Enum for EventType, I implemented the create_event function to automate the creation of an event string for the query.

#71 — create_event function to create event string.

After this, I implemented the save_file function which we talk about above.

#72 — Function to save the binary response.

Week 13

Everything is done, now I implemented the createMovie function for the users to automate the lengthy process of creating a movie.

#73 — The createMovie function.

After PR #73 got merged, we had the first release on PyPI. 🎉

Now, you can install it and make a movie yourself 😉

~ pip install hvpy

I hope this project will help researchers within the field of solar physics and helioviewer.org users to carry out their tasks easily and effectively.

Reference to the previous blog posts and other resources

Blog post 1, Starting my summer with SunPy —
How I got selected in GSoC and my work experience of week 1 and week 2.
Blog post 2, Facade for the API wrapper —
Detailed work experience from week 3 to week 6.
Blog post 3, Moving close to the First Release —
Detailed work experience from week 7 to week 9.
Contribute to hvpy
Documentation

Closing Thoughts

First of all, I would like to thank with all my heart Nabil Freij, Daniel Garcia Briseno, and Jack Ireland for their amazing guidance throughout the program. Without their guidance and ideas, I would have not been able to implement this project.

These three months of work with Project-Helioviewer have made me a very confident backend developer. I learned a lot of new things about programming and learned about my mistakes. I learned how we design a project and execute the project plan.

I enjoyed every moment of this program.

Thank you SunPy and The Helioviewer Project ❤

Cheers!