Universe OpenAstronomy (Posts by Manit Singh)

During GSoC 2024, I made several key contributions to the sunpy-soar project:

Manit Singh — Fri, 23 Aug 2024 16:10:53 GMT

Initial Implementation of Metadata for Remote Sensing Instrument(merged):

PR #118: This was my initial pull request where I established join operations for tables and implemented metadata for wavelength and detector for remote sensing instruments.

2. Gallery Examples and How-to Guide for recent implementations(merged):

PR #127: In this pull request, I added gallery examples and a how-to guide showcasing the newly implemented wavelength and detector metadata.

3. Error Handling for SOAR Server Downtime(merged):

PR #135: This update involved catching server errors thrown by SOAR when it’s down, enhancing the robustness of the system.

4. Distance Filtering Query Support(merged):

PR #137: I added support for a new query method, REQUEST="doQueryFilteredByDistance", enabling distance-based filtering for queries.

5. Observation Mode Metadata (Not Yet Merged):

PR #140: This pull request introduced Observation Mode as a new metadata. After discussions with my mentors, we decided not to merge this feature for now.

6. Field of View (FoV) Values Extraction(Not yet merged):

PR #143: I added a third join table to extract Field of View (FoV) values with “earth” and “sun” references, allowing users to plot these on a SunPy map.

7. Developer Guides for Future Development(Not yet merged)

PR #147: This pull request added a section of developer guides designed to assist future contributors in developing sunpy-soar.

8. External Method for URL Construction (closed):

PR #129: This pull request proposed using an external method for making URL calls instead of manually constructing them. However, after further discussion with my mentors, we realized this approach could limit our ability to add future functionality, so we decided not to close it.

Enhancing SOAR Queries: Improved Error Handling and Support for Distance-Based Filtering

Manit Singh — Tue, 23 Jul 2024 15:50:13 GMT

Improved Error Handling and Support for Distance-Based Filtering

Introduction

Over the past few weeks, I’ve been working on addressing and enhancing certain functionalities within the sunpy-soar package. This post delves into the two main improvements I’ve implemented: better error handling for server downtime and the introduction of support for distance-based query filtering.

Improved Error Handling for Server Downtime

Previously, when the SOAR server was down, a generic JSONDecodeError would be raised. This was less than ideal as it did not provide a clear indication of what the actual issue was. To improve this, I worked on implementing a more descriptive error message that would be raised in such scenarios.

r = requests.get(f"{tap_endpoint}/sync", params=payload)
try:
    response_json = r.json()
except JSONDecodeError:
    msg = "Server returned an invalid JSON response. The SOAR server may be down or not functioning correctly."
    raise RuntimeError(msg)

With this change, users will now see a RuntimeError with a clear message indicating that the server may be down or not functioning correctly, which makes troubleshooting much easier.

Implementing Support for Different REQUEST Types

After resolving the error handling issue, I moved on to implementing support for different REQUEST types. Sunpy-soar initially only supported the doQuery REQUEST type. However, there was a need to expand this to support the doQueryFilteredByDistance REQUEST type as well.

What is doQueryFilteredByDistance?

The doQueryFilteredByDistance REQUEST type allows for filtering the query results based on a specified distance range. The main change here is setting the REQUEST parameter to doQueryFilteredByDistance and appending &DISTANCE(distancemin,distancemax) to the query.

Example Query:

SELECT * FROM soar.v_sc_data_item WHERE instrument='MAG' AND level='LL02'

With Distance Filtering:

SELECT * FROM soar.v_sc_data_item WHERE instrument='MAG' AND level='LL02' AND DISTANCE(0.28,0.49)

Conclusion

These enhancements significantly improve the functionality and user experience of the sunpy-soar package. The improved error handling provides clearer feedback to users when the SOAR server is down, and the support for doQueryFilteredByDistance allows for more refined queries based on distance, opening up new possibilities for data analysis.

Midterm Evaluations Are Close!

Manit Singh — Sun, 07 Jul 2024 15:02:57 GMT

So, in the past two weeks, upon discussion with my mentors, I started working on the stretch goal, which involves using Astroquery’s TAP Plus for making the URL calls. Additionally, I had to clean up some code and remove the instruments SPICE and PHI from returning wavelength data due to their individual issues.

For SPICE, the `wavemin’ and `wavemax` values are only available for the first spectral window in SOAR, so the data is not entirely representative of what a user might be searching for. Similarly, for PHI, the issue is that two different units are used to represent wavelength, which results in very
different `wavemin’ and `wavemax` data due to the data being in both angstrom and nanometer units. Unfortunately, this is not specified anywhere in SOAR, so to ensure that `sunpy-soar` does not
return any misleading data to the user, we’ve decided to exclude the wavelength data for these instruments.

Coming to the change in table fetching methods. TAP Plus basically uses SQL to fetch tables, which is different from the normal TAP that uses ADQL to fetch tables. This avoids the entire process of manually constructing Astropy tables after calling the SOAR API with TAP.

For comparison, this is how the method looks with TAP:

@staticmethod
    def _do_search(query):
        """
        Query the SOAR server with a single query.

        Parameters
        ----------
        query : list[str]
            List of query items.

        Returns
        -------
        astropy.table.QTable
            Query results.
        """
        tap_endpoint = "http://soar.esac.esa.int/soar-sl-tap/tap"
        payload = SOARClient._construct_payload(query)
        # Need to force requests to not form-encode the parameters
        payload = "&".join([f"{key}={val}" for key, val in payload.items()])
        # Get request info
        r = requests.get(f"{tap_endpoint}/sync", params=payload)
        log.debug(f"Sent query: {r.url}")
        r.raise_for_status()

        # Do some list/dict wrangling
        names = [m["name"] for m in r.json()["metadata"]]
        info = {name: [] for name in names}

        for entry in r.json()["data"]:
            for i, name in enumerate(names):
                info[name].append(entry[i])

        if len(info["begin_time"]):
            info["begin_time"] = parse_time(info["begin_time"]).iso
            info["end_time"] = parse_time(info["end_time"]).iso

        result_table = astropy.table.QTable(
            {
                "Instrument": info["instrument"],
                "Data product": info["descriptor"],
                "Level": info["level"],
                "Start time": info["begin_time"],
                "End time": info["end_time"],
                "Data item ID": info["data_item_id"],
                "Filename": info["filename"],
                "Filesize": info["filesize"],
                "SOOP Name": info["soop_name"],
            },
        )
        if "detector" in info:
            result_table["Detector"] = info["detector"]
        if "wavelength" in info:
            result_table["Wavelength"] = info["wavelength"]
        result_table.sort("Start time")
        return result_table

This is how the method looks with Astroquery TAP plus.

`def _do_search(query):
        """
        Query the SOAR server with a single query.

        Parameters
        ----------
        query : list[str]
            List of query items.

        Returns
        -------
        astropy.table.Table
            Query results.
        """
        tap_endpoint = "http://soar.esac.esa.int/soar-sl-tap/tap"
        sql_query = SOARClient._construct_payload(query)
        soar = TapPlus(url=tap_endpoint)
        job = soar.launch_job_async(sql_query)
        results = job.results
        new_colnames = {
            "instrument": "Instrument",
            "descriptor": "Data product",
            "level": "Level",
            "begin_time": "Start time",
            "end_time": "End time",
            "filename": "Filename",
            "filesize": "Filesize",
            "soop_name": "SOOP Name",
        }
        new_colnames.update({k: k.capitalize() for k in ["wavelength", "detector"] if k in results.colnames})
        for old_name, new_name in new_colnames.items():
            results.rename_column(old_name, new_name)
        results.sort("Start time")
        return results

Even the query construction methods, they looks a lot more cleaner. As the final query generated is an SQL query not an ADQL query.

It’s going good.

Manit Singh — Sun, 23 Jun 2024 07:21:59 GMT

It’s going good.

The past two weeks were dedicated to the implementation of wavelength functionality. Now, wavelength has been successfully implemented for the applicable instruments. Along with that, a how-to guide has been created to assist users in navigating querying over wavelength for different instruments. Additionally, a gallery example for wavelength and detector has been added.

What’s the direction of implementation:

The implementation of wavelength can be divided into two parts:

Instruments with a “wavelength” column in SOAR: These remote sensing instruments are EUI, SOLOHI, and METIS. For these instruments, we query on the basis of wavelength.

 instrument = a.Instrument("EUI")
 time = a.Time("2023-04-03 15:00", "2023-04-03 16:00")
 level = a.Level(1)
 wavelength = a.Wavelength(304 * u.AA)
 res = Fido.search(instrument & time & level & wavelength)

Result:

A range of wavelength can also be passed, which will be considered wavemin and wavemax for these three instruments.

wavelength = a.Wavelength(171 * u.AA, 185 * u.AA)

Result:

2. For the instruments PHI and SPICE, we don’t have a “wavelength” column in their instrument table, so we use wavemin and wavemax for querying.

However, there is a problem with SPICE. Since the range of wavelength is only given for the first spectral window of the data, to ensure the data is not misleading to the user, we do not return any wavelength values.

 instrument = a.Instrument("PHI")
 time = a.Time("2023-02-01", "2023-02-02")
 level = a.Level(2)
 wavelength = a.Wavelength(6173.065 * u.AA, 6173.501 * u.AA)
 res = Fido.search(instrument & time & level & wavelength)

Result:

Although passing a range of wavelengths is preferred for PHI, if only one value is passed, it will be taken as wavemin, and filtering will be done based on it. Additionally, the corresponding wavemax will be provided in the output table.

wavelength = a.Wavelength(6173.065 * u.AA)

Result:

Challenges that still needs working:

There is an issue with PHI’s wavelength data as well. The wavelengths returned are sometimes in the order of 6173 and sometimes 617.3, which are essentially just different units of similar wavelength data, but this is not specified in SOAR.

A Great Start to GSoC with SunPy

Manit Singh — Sun, 02 Jun 2024 08:15:15 GMT

Hello, everyone! My name is Manit Singh, and I’m thrilled to share my journey as a participant in Google Summer of Code (GSoC) with SunPy. Getting selected for this prestigious program has been a significant milestone in my career, and I am excited to contribute to the Enhancing SOAR Metadata project.

Community Bonding Period

Introduction to the Community Bonding Period

The community bonding period was an essential phase where I got to know my mentors, understand the project’s goals in-depth, and prepare for the coding period.

Meetings and Discussions

During this period, I had several meetings and discussions with my mentors and community members. These interactions were crucial for:

Understanding the Project Goals: The project aims to increase the scope of SOAR data in sunpy-soar.
Clarifying Initial Approaches: My initial approach was to implement wavelength and detector first, and then discuss the implementation of the next set of metadata.

From these initial discussions, we concluded that the Detector should be implemented first, after creating methods to support the joining of instrument and data tables (i.e., v_<instrument>_<ll/sc>_fits and v_<ll/sc>_data_item).

First Coding Week: Implementing the Detector in the SOAR Project

Objectives

The primary objective for the first week was to add methods to support querying data involving instrument and data tables both and implement the detector component in the SOAR project. Although the detector is part of the SOAR product, my mentors suggested that there are many people who might still want to query over the detector.

Implementation Details

As discussed in the meetings held during the community bonding period, I worked on constructing methods to enable joining two different tables. Initially, it all looked very messy, but with time and refactoring, a quite simple join constructing method was created.

Implementing Detector: Similar to how other metadata in the sunpy.net attribute system was implemented, I implemented the Detector. The only major difference was taking data from two tables and joining them.
Testing: I added tests to ensure that the construct method designed for joining tables could give the desired query outputs and that the detector was working fine for instruments with multiple dimensional data. (which depends on the dimensional index).

Challenges and Solutions

Different Dimension Index for instruments:

Instruments collect a wide variety of data types that can be multidimensional. For example, for SPICE, there are 4 dimensions, resulting in 4 similar rows of data, one for each dimension. This could be confusing for the user unless the dimension index is also shown in a column. For the STIX instrument, there are 0 dimensions, meaning it does not have any detector column in its instrument fits table. For other remote sensing instruments, there are 2 dimensions.

Solution:

# As there are no dimensions in STIX, the dimension index need not be included in the query for STIX.
if "stx" not in instrument_table:
# To avoid duplicate rows in the output table, the dimension index is set to 1.
final_query += "h2.dimension_index='1'+AND+"

In the construct methods of the ADQL query I added dimension_index to be taken 1 by default for all instruments other than STIX, this results in no repetition of data.
For STIX instrument, since there are no dimension it just works simply okay without taking any dimension in the join query constructed after construct methods

Outcomes

code:

instrument = a.Instrument("METIS")
time = a.Time("2022-06-02 0:00","2022-06-02 1:00")
level = a.Level(2)
detector=a.Detector("VLD")

By the end of the first week, the detector component was successfully implemented in sunpy-soar. The pull request is currently awaiting review from all the mentors to ensure any discrepancies in the implementation are addressed and to gather additional insights. :)

Conclusion

The journey so far has been incredibly enriching. The support from my mentors and the community has been invaluable. I am looking forward to the upcoming weeks and continuing to contribute to the SOAR project.