Universe OpenAstronomy (Posts about gnuastro)

Final GSoC Report

DeadSpheroid — Sun, 06 Oct 2024 21:30:00 GMT

In this post, I'll be discussing my GSoC'24 project, the goals set, work done, and future scope

About OpenAstronomy

OpenAstronomy is a collaboration between open source astronomy and astrophysics projects to share resources, ideas, and to improve code.

OpenAstronomy consists of many different projects: astropy, sunpy, stingray, radis

and of course GNU Astronomy Utilities (Gnuastro)

About Gnuastro

Gnuastro is an official GNU package that consists of many CLI programs as well as library functions for manipulation and analysis of astronomical data. Something important to note about Gnuastro is that it is written entirely in C99 and shell script. The Gnuastro team meets every Tuesday to exchange notes and review progress

Goals of the project

Going into this project, there were two main goals decided

Setting up the low level wrapper infrastructure for using OpenCL within Gnuastro, putting minimal requirements on the developers/users to know OpenCL.
Parallelizing Gnuastro subroutines using the aforementioned wrappers to offload compute-heavy tasks to the GPU.

The first goal deals with how users of the library would interact with the OpenCL modules. Ideally, you would want the users to have no knowledge about OpenCL and only interact with it through GNUAstro.

The second goal deals with analyzing parts of the Gnuastro library, identifying easy to parallelize sections and writing optimised OpenCL Kernels for them, leveraging the wrapper infrastructure for execution.

The majority of my work lives here

Pre GSoC

Prior to GSoC, my experience consisted mostly of Deep Learning and Natural Language Processing. My knowledge of GPU Processing was limited and naive.

During the proposal drafting period, candidates were asked to submit a task, which was implementing simple image convolution on both CPU and GPU using OpenCL.

During GSoC

Work on GSoC kicked off around May 1st with the first objective being

Build System Integration

Now, Gnuastro being a GNU project uses the GNU Build System, GNU Autotools(Autoconf, Automake, Libtool).

This was a completely new build system for me to work with and I had to get the library to include OpenCL and link against the OpenCL library at compile time.

Thanks to some helpful pointers from Mohammad, I was able to grasp the working pretty quickly and was able to set up Gnuastro to include and build with OpenCL if it was detected on the system.

For more information, you can read here

Wrapper Infrastructure

The next goal was to create wrappers around the OpenCL C API, for various operations(data transfer, launching kernels, querying devices).

This was done in the form of a new Gnuastro module called cl-utils.c which contained

Initialisation

Functions dealing with initialising OpenCL and creating, destroying OpenCL objects

Data Transfer Functions

Functions dealing with data transfer to and from the GPU. This is one of the biggest overheads in GPU Programming. Additionally, OpenCL presented another challenge in the form of transferring structs to the GPU, which was problematic as one of Gnuastro’s most important data structures gal_data_t could not be directly transferred.

So, this module provides two ways of transferring data to the GPU

OpenCL Buffers and
OpenCL SVM

Buffers are intended to be used for simple data structures, while OpenCL SVM is better with more complex data structures involving internal pointers.

The interface looks something like this

void
gal_cl_write_to_device (cl_mem *buffer, void *mapped_ptr,
cl_command_queue command_queue);

void *
gal_cl_read_to_host (cl_mem buffer, size_t size,
cl_command_queue command_queue);
.
.
.
gal_data_t *
gal_cl_alloc_svm (size_t size_of_array, size_t size_of_dsize,
cl_context context, cl_command_queue command_queue);

void
gal_cl_map_svm_to_cpu (cl_context context, cl_command_queue command_queue,
void *svm_ptr, size_t size);

For more information on the two, and a comparison see here

Executing Kernels

Now, the main code running on the GPU is the OpenCL Kernel, usually defined in a .cl file and compiled at runtime.

The idea when making this module, was to keep the interface as similar to the original pthreads gal_threads_spin_off() interface that Gnuastro already had. So i created a gal_cl_threads_spinoff() function, taking information like the kernel filepath, number of inputs, list of inputs, number of threads executed and more.

typedef struct clprm
{
char               *kernel_path; /* Path to kernel.cl file */
char               *kernel_name; /* Name of __kernel function */
char             *compiler_opts; /* Additional compiler options */
cl_device_id          device_id; /* Device to be targeted */
cl_context              context; /* Context of OpenCL in use */
int             num_kernel_args; /* Number of total kernel arguments */
int                num_svm_args; /* Number of SVM args*/
void              **kernel_args; /* Array of pointers to kernel args */
size_t       *kernel_args_sizes; /* Sizes of non SVM args */
int          num_extra_svm_args; /* Number of implicit SVM args */
void           **extra_svm_args; /* Array of pointers to these args */
int                    work_dim; /* Work dimension of job - 1,2,3 */
size_t        *global_work_size; /* Array of global sizes of size work_dim */
size_t         *local_work_size; /* Array of local sizes of size work_dim */
} clprm;

These wrappers were not developed all at once, but rather in conjunction with the next section, writing wrappers as and when I needed them.

Parallelized Subroutines

To achieve the goal of GPU acceleration, first we needed to identify parts of the library that could be parallelized. Its important to note that not everything can be parallelized, and just because something can be, doesnt mean it should be.

The most obvious candidate for this of course was 2D Image Convolution, already implemented in Gnuastro in the astconvolve module.

Same code on CPU and GPU

The initial idea was to have the exact same code running on both the CPU(via pthread) and the GPU(via OpenCL). This is possible because OpenCL Kernels are based on OpenCL C which is a variant(kind of a subset) of C99.

This is because Gnuastro is a “minimal dependencies” package and having two separate implementations would greatly overcomplicate the codebase.

However for the time being, this idea was shelved, till I had a working implementation of convolution in OpenCL.

Convolution

I got to work creating a new module cl-convolve.c containing the new implementation of convolution gal_convolve_cl()

The exact code can be viewed here, but in short

Transfer input, kernel and output images to GPU
Spin off a thread for each pixel in the input, convolving that particular pixel.
Copy the output image back to CPU

Additional features

However, Gnuastro doesn’t use a simple 2D convolution, it also performs an additional three important tasks:

Edge Correction: Pixels near the edge use a different kernel weight than others. More info here
NAN Checking: Often, images captured by astronomical cameras, have missing pixels(represented as NANs). These pixels are to be ignored.
Channels: Cameras use multiple different sensors to capture images, and convolution should not mix pixels from different sensors. For a better idea, read Gnuastro’s explanation

The first two were rather easy to implement, but the third was a bit troublesome, especially because the existing implementation of gnuastro was complex and hard to understand.

Eventually however, with a little bit of math it was possible, and the final kernel looked like this

After these parts were completed, now, all that was left was to actually integrate it properly with Gnuastro.

Optimised Convolution

The power of GPUs comes not from the many threads that are launched, but rather from the many optimisations possible, from organising threads into blocks, to special kinds of memory. I decided to try optimising Convolution based on Labeeb’s suggestion of using shared memory.

However most of the optimisation out there are for CUDA, not OpenCL, but the principles in question were the same. Thanks to this article, I was able to implement an optimised 2Dconvolution kernel in OpenCL.

The results of the optimisation were surprisingly positive: For a 5000 x 5000 image, times recorded for the convolution operation(excluding data reading/writing in seconds were)

	Pthread	OpenCL-CPU	OpenCL-GPU
w/out optimisations	1.014374	0.918015	0.025869
w/ optimisations	1.053622	0.326756	0.004184

Thats a speedup of ~6.2 times over the non optimised GPU run, and ~242 times over the existing pthread implementation in Gnuastro!

Further optimisations are possible using special native functions like MUL24 and constant memory. But the details of those and how these optimisation work is a topic for a separate post.

Revisiting Same Code on CPU vs GPU

After a discussion, it was decided that the best path forward for OpenCL in Gnuastro would be to completely replace the existing pthread implementation.

In essence, the existing “convoluted” convolution implementation would be replaced with my new one, allowing the same code to be ran in 3 different ways:

With OpenCL on the GPU
With OpenCL on the CPU
With GCC+Pthreads on the CPU

This decision was made to adhere to the Gnuastro philosophy of “Minimal Dependencies” so the user does not have to install many packages just to use the library.

It was challenging, owing to the different styles in which we write code for a CPU device versus a GPU device. But I managed to get a partially working version using some C macros here and there to do so. It still fails some Gnuastro tests, which is yet to be resolved.

However, doing so prevents the library from utilising the full power of GPUs with several GPU specific optimisations seen previously.

Using the OpenCL modules in your program

Finally, when a user wants to use Gnuastro’s OpenCL capabilities within their own programs, the flow followed would look like:

Intialize OpenCL
Transfer Input to Device
Write an OpenCL Kernel
Spinoff Threads
Copy Output back to Host

Lets take an example where we need to simply add two fits images.

Initialize OpenCl

  cl_context context;
cl_platform_id platform_id;
cl_device_id device_id;

gal_cl_init (CL_DEVICE_TYPE_GPU, &context, &platform_id, &device_id);
cl_command_queue command_queue
= gal_cl_create_command_queue (context, device_id);

This initializes and OpenCL context, among other objects for use in future function calls.

Transfer Input to Device

Make use of gal_cl_copy_data_to_gpu() to transfer the loaded fits files to the GPU, passing the previously initialized context and command queue. Make sure the command queue finishes before proceeding ahead through gal_cl_finish_queue()

  gal_data_t *input_image1_gpu
= gal_cl_copy_data_to_gpu (context, command_queue, input_image1);
gal_data_t *input_image2_gpu
= gal_cl_copy_data_to_gpu (context, command_queue, input_image2);
gal_data_t *output_image_gpu
= gal_cl_copy_data_to_gpu (context, command_queue, output_image);

gal_cl_finish_queue (command_queue);

Write an OpenCL Kernel

First, any custom structs you use, must be defined in the kernel, here we define gal_data_t.

Then, you create the “per thread” function that will be executed, prefixed by __kernel and always returning void.

In the arguments, mention the pointers to the inputs/outputs, as well as a __global identifier, since your input is acessible by all threads.

Make use of OpenCl’s get_global_id(0) to get the thread id along the 0th dimension.

Perform the core operation of your program.

Putting it all together, it looks like this:

typedef struct  __attribute__((aligned(4))) gal_data_t
{
/* Basic information on array of data. */
void *restrict array; /* Array keeping data elements.               */
uchar type;         /* Type of data (see 'gnuastro/type.h').      */
size_t ndim;          /* Number of dimensions in the array.         */
size_t *dsize;        /* Size of array along each dimension.        */
size_t size;          /* Total number of data-elements.             */
.
.
.
/* Pointers to other data structures. */
struct gal_data_t *next;  /* To use it as a linked list if necessary.   */
struct gal_data_t *block; /* 'gal_data_t' of hosting block, see above.  */
} gal_data_t;

__kernel void
add(__global gal_data_t *input_image1,
__global gal_data_t *input_image2,
__global gal_data_t *output_image)
{
int id = get_global_id(0);

float *input_array1 = (float *)input_image1->array;
float *input_array2 = (float *)input_image2->array;
float *output_array = (float *)output_image->array;

output_array[id] = input_array1[id] + input_array2[id];
return;
}

Spin Off Threads

Make use of the clprm struct defined in gnuastro/cl-utils.h to group all the relevant parameters.

Kernel Path is the filepath to the OpenCL Kernel you just wrote.

Kernel Name is the name of the function you defined with __kernel earlier.

Compiler Options is a string of any special compiler options like macros/debug options you wish to use for the kernel.

Device Id & Context are the objects intialized in the first step.

Number of Kernel Arguments is the number of kernel arguments.

Number of SVM Arguments is the number of arguments that use SVM(all the gal_data_t’s)

Kernel Arguments is an array to void pointers of kernel arguments.

Number of Extra SVM Arguments is the number of arguments that are implicitly referenced with a struct. For example, input_image1_gpu is directly referenced as a kernel argument, but the input_image1_gpu->array is implicitly referenced.

Extra SVM Arguments is an array of void pointers to the aforementioned special arguments.

Work Dim is the number of dimensions of the threads (1, 2, 3) For example, an array would have 1 dimension(0,1,2,…34,35,36) x an image would have 2 dimensions(0:0, 0:1, 1:0, 1:1,….) x:y a volume would have 3 dimensions.

Global Work Size is the total number of threads spun off

Local Work Size is the number of threads in a block on one GPU core. Leaving it blank lets the device choose this number.


clprm *sprm = (clprm *)malloc (sizeof (clprm));

void *kernel_args[] = { (void *)input_image1_gpu, (void *)input_image2_gpu,
(void *)output_image_gpu };

void *svm_ptrs[]
= { (void *)input_image1_gpu->array, (void *)input_image2_gpu->array,
(void *)output_image_gpu->array };

size_t numactions = input_image1->size;

sprm->kernel_path = "./lib/kernels/add.cl";
sprm->kernel_name = "add";
sprm->compiler_opts = "";
sprm->device_id = device_id;
sprm->context = context;
sprm->num_kernel_args = 3;
sprm->num_svm_args = 3;
sprm->kernel_args = kernel_args;
sprm->num_extra_svm_args = 3;
sprm->extra_svm_args = svm_ptrs;
sprm->work_dim = 1;
sprm->global_work_size = &numactions;
sprm->local_work_size = NULL;

gal_cl_finish_queue (command_queue);

Copy Output back to Host

gal_cl_read_data_to_cpu(context, command_queue, output_image_gpu);

The complete program can be accessed here

Post GSoC

Now, that the wrapper infrastructure is set up and convolution is implemented, whats left is to test the implementation against real life scenarios to make sure it lives up to the expectations of the Gnuastro users. We also need to come up with a consistent way to execute the same kernel on both OpenCL and GCC, as mentioned earlier.

Additionally, now that work on one module is complete, it opens the scope for more modules to be implemented on the GPU (like statistics, interpolation and more)

Acknowledgements

GSoC has been an incredible learning experience for me both from a technical view and from a personal view.

On the technical side, I learned a lot about one of my favourite domains in Low Level Programming, GPU Programming and my understanding of how to write libraries that are easy to use, performant and above and all else, FOSS, improved tremendously. It’s one thing when you learn and write code for your own personal projects, but it’s a completely different experience contributing to something like Gnuastro.

On the personal side, the weekly meetings with the Gnuastro team were always extremely engaging and i got to learn a lot from the team, Giacomo’s work on Astrometry, Alvaro’s work on Deconvolution and Ronald too. Their feedback on stuff like debugging using valgrind/gdb and references to other projects using OpenCL, alongside other topics has been invaluable.

Above and all else im thankful to my mentor Mohammad Akhlagi. Its been amazing getting to interact with someone so experienced and I learned a lot from him, ranging from Astronomy to Hacking the GNU C Library. He was always patient and understanding of my other responsibilities and allowed me to work at my own pace. I’m grateful to him for the opportunity to be a part of the Gnuastro community.

Finally, I can’t explain how indebited I am to my mentor Labeeb Asari. His knowledge about GPU Programming has been vital to my work on this project and I’m grateful to him for introducing me to the Gnuastro team. From Project X, to GSoC to college in general, he has been a big help in everything I’ve done and im glad to have him as a mentor and friend.

A huge thank you to the Google Summer of Code Team for undertaking this wonderful initiative and I hope they continue this program in future years.

Towards New Speeds

DeadSpheroid — Sun, 28 Jul 2024 21:30:00 GMT

In this post, I hope to illustrate how GPU's actually accelerate parallel operations

Let’s get started

Firstly, how do GPU’s really work? Well, at a lower level, the architecture looks something like this…

Focus on the differences between the two, CPUs have few, very highly specialised and refined cores. GPUs on the other hand have hundreds of more primitive, yet powerful cores.

This is the reason why GPUs can’t do I/O and stuff, its because they are meant solely for mathematical operations.

But what’s the point of so many cores??

This is where SIMD or Single Instruction Multiple Data processing comes in handy. See, many operations(image and volume ones notoriously) are extremely taxing for the CPU to perform. Imagine being part of a team of 8/16 people, stamping a sheet of paper, except you have 10^6 sheets to stamp. Even if you took 1 ms/sheet, it would still take you insanely long to finish your jobs. And I mean, you have better things to do right?

Well what if you could hire 10^3 people for cheap and give each of them a sheet of paper to stamp? Wouldn’t that greatly speed things up? This is the core idea behind SIMD processing, you have an operation that is to be done thousands of times, over and over again, just on different data.

So you give each GPU core a part of the data and let it do the job, since the GPU has so many cores, it’s not really a problem.

So whats the catch?

Continuing the stamping sheet analogy, giving the sheets to 10^3 workers is challenging and time-consuming. In other words, data transfer is a problem, since GPU VRAM is separate from CPU RAM

Additionally, parallel programming forces you to think in an additional dimension, because your code is being executed 100s of times at the same time. This makes writing efficient kernels difficult, since branching is frowned on at the GPU, and you need some way of preventing data races.

That is to say nothing of the increased power consumption and high cost of hardware.

Despite all of this however, GPUs are still heavily favoured, because the speed-up they offer greatly outweights the rest.

Well, how do I use my GPU?

Let’s take an example, hopefully youre already familiar with image convolution. If not, the image below explains it well

For each iteration, we center the kernel over a pixel, multiply the overlapping values, add them up and then (optionally)divide by sum of kernel values. On CPU? For an m x n image and a k x k kernel, this is an O(mn * k^2) operation, meaning the time taken for convolution for a given data size increases tremendously.

This can be greatly lessened using a GPU.

But for that, we need to first identify the stamping task here, the tedious computation which is easy to do, but time consuming.

Image Convolution

If you thought of the matmul operation happening at each pixel, you’d be correct! This is one of the easiest ways to parallelise convolution. We’re repeatedly performing matrix multiplication, with different pixels at the center each time

Therefore SIMD can be applied here, the instruction being matmul and the data being all the pixels.

On a lower level, this is represented by the diffferent thread_id given to each thread on the GPU. This id represents a one to one mapping of an integer to input data elements. Incase of Images, it is the pixel number that is at the center.

Here’s an example of how convolution can be parallelized, note that this is not the most efficient and is nowhere near perfect, but it is simple enough to understand.

__kernel void
convolution(
__global float *image_array, __global size_t *image_dsize,
__global float *kernell_array, __global size_t *kernell_dsize,

__global float *output) {

/* get the image and kernel size */
int image_height = image_dsize[0];
int image_width = image_dsize[1];
int kernell_height = kernell_dsize[0];
int kernell_width = kernell_dsize[1];

/* get the local group id */
int id = get_global_id(0);
int row = id / image_width;
int col = id % image_width;

if (row < image_height && col < image_width) {

float sum = 0;
/* matmul operation as normal*/
for (int y = -kernell_height / 2; y <= kernell_height / 2; y++) {
for (int x = -kernell_width / 2; x <= kernell_width / 2; x++) {
if (row + y >= 0 && row + y < image_height && col + x >= 0 &&
col + x < image_width) {
sum += (image_array[(row + y) * image_width + col + x] *
kernell_array[(y + kernell_height / 2) * kernell_width + x +
kernell_width / 2]);
}
}
}
output[row * image_width + col] = sum;
}
}

A new outlook

Of course, the world would be a lovely place if everything could be parallelised as easily as this. Realistically, parallelizing these operations is hard, but you have tools like local groups, barriers, etc to help you out!

Exploring OpenCL memory management

DeadSpheroid — Sat, 13 Jul 2024 21:30:00 GMT

In this post, I hope to give a high level understanding of OpenCL's Memory Mechanisms

The basics

Firstly, its important to have a basic understanding of the hardware involved. Keeping it simple, each OpenCL device represents a different set of hardware, each with its own RAM. My own laptop has a 16GB CPU RAM, and 6GB VRAM

Now, at the heart of C, we have pointers, without them well, you can’t really get much done in C. The pointers we convetionally use are pointers to CPU RAM.

So what would happen if you try to pass a CPU Pointer to the GPU? Well, of course, it wont work, the GPU simply segfaults, as it cannot understand the pointer given to it. But we still need to use pointers, we can’t just abandon them. So how do we do this?

Buffers

At the simplest level, we have OpenCL Buffers. These buffers are chunks of memory allocated on the OpenCL device as well as on host memory.

cl_mem clCreateBuffer(
cl_context context,
cl_mem_flags flags,
size_t size,
void *host_ptr,
cl_int *errcode_ret)

Creating a buffer is easy, its figuring out what kind of buffer you need that is important.

Broadly speaking there are 3 types of buffers based on the flags passed:

CL_MEM_USE_HOST_PTR

This tells OpenCL to use the host pointer provided as the underlying memory on host.

CL_MEM_COPY_HOST_PTR

This flag tells OpenCL to make a new buffer and fill it with the memory pointed to by host pointer.

CL_MEM_ALLOC_HOST_PTR

This one is the same as CL_MEM_USE_HOST_PTR, but the allocation of host pointer is also done by OpenCL

But which one should you use? Well, if you desire a zero copy buffer, i.e. create a buffer without copying memory, especially memory on host, then CL_MEM_USE_HOST_PTR(if you have the memory already initialised) or CL_MEM_ALLOC_HOST_PTR(if you plan to initialise the buffer afterward) The concept of a zero copy buffer is super helpful when you are targeting the same host CPU as an OpenCL device.

I mean, you already have the data in CPU RAM, why would you make another copy in CPU RAM by creating a new buffer?

Literacy for Buffers

The above is enough when you are operating on the same device, which is seldom the case with OpenCL. But when you work with GPUs, you need to get the memory into GPU VRAM somehow.

And this is impossible(maybe) without copying the data over.

So how do you copy the data over to GPU VRAM? Well, after allocating a buffer as seen before, OpenCL will try to recreate the host side buffer on the device as well.

But when you update the host side buffer(like reading in input), youd want it to reflect on device as well. Similary, when your device is done processing, you need to get the output from device memory to host memory.

There are two main ways to do this:

Read/Write Buffer

You have a buffer on host memory and on device memory that mirror each other.

cl_int clEnqueueReadBuffer(
cl_command_queue command_queue,
cl_mem buffer,
cl_bool blocking_read,
size_t offset,
size_t size,
void* ptr,
cl_uint num_events_in_wait_list,
const cl_event* event_wait_list,
cl_event* event);

cl_int clEnqueueWriteBuffer(
cl_command_queue command_queue,
cl_mem buffer,
cl_bool blocking_write,
size_t offset,
size_t size,
const void* ptr,
cl_uint num_events_in_wait_list,
const cl_event* event_wait_list,
cl_event* event);

OpenCL provides Read/Write commands to force overwrite of one buffer over the other, and in this way, data transfer is achieved.

Map/Unmap Buffer

There is a single buffer on device memory, that is presented to CPU when demanded So “mapping” a buffer will bring it from device memory into host RAM. Then any changes made will be saved in host RAM. Finally, once done with changes, you may “unmap” the buffer, which writes all changes made back to device memory

void* clEnqueueMapBuffer(
cl_command_queue command_queue,
cl_mem buffer,
cl_bool blocking_map,
cl_map_flags map_flags,
size_t offset,
size_t size,
cl_uint num_events_in_wait_list,
const cl_event* event_wait_list,
cl_event* event,
cl_int* errcode_ret);

cl_int clEnqueueUnmapMemObject(
cl_command_queue command_queue,
cl_mem memobj,
void* mapped_ptr,
cl_uint num_events_in_wait_list,
const cl_event* event_wait_list,
cl_event* event);

In this way, data transfer is achieved

To map or not to map?

To be honest, performance differences are very minute, atleast from my tests with the gnuastro library. However Map and Unmapping makes a world of difference as compared to Read/Write when it comes to simplicity

The problem with buffers

No matter what you do, when working with buffers, you always end up copying the data For example, you load an image into CPU RAM, but actually want to work with it on the GPU. So, you end up copying the image into GPU RAM. In the end, you process the same data twice, once while loading and once while copying

For small images(2000 x 2000) this is barely noticeable But gnuastro, and the people using gnuastro deal with astronomical images of incredibly large sizes(i’ve heard 30GB just for one image).

So, most certainly, any time you save by using parallelised processing on the GPU, is lost and maybe even worsened by the data transfer times. Then, using the GPU is almost pointless, unless you use the same data over and over again

“Well, cant I just load the data on the GPU directly?” Thats not possible, atleast not to my knowledge. This is the tradeoff with GPUs. On a CPU, you have 4/8/16 highly specialised and capable cores(math, I/O), while on the GPU you have 1000s of some very primitive math operations(only math, no I/O) So you always have to load it into CPU RAM first and then go to GPU RAM.

So how can we fix this problem? Well, one of the options is to use Shared Virtual Memory(OpenCL SVM), which enables the GPU to directly access CPU RAM and play with CPU pointers.

However, I still have yet to test SVM in the context of gnuastro, to see if its useful. Besides, SVM also fixes the problem of structs containing pointers(for another post). Documentation for OpenCL is already sparse, and to add insult to injury, documentation on OpenCL SVM is even more sparse. But I like the challenge…

Deeper into OpenCL

DeadSpheroid — Sat, 22 Jun 2024 19:45:00 GMT

In this post, I hope to give a high level understanding of OpenCL and its workings

Setting it up

OpenCL is relatively easy to get up and running on your system.

For users: All you need, is the OpenCL runtime for your device! In case of Nvidia, this comes with the Nvidia drivers, while for Intel CPUs, this has to be manually installed by a package manager.
For developers: You will need the OpenCL library to link against, and the OpenCL headers as well, again easily available in your package manager.

Here’s a new perspective

OpenCL presents a general interface to the developer, no matter what the device or the architecture.

Firstly, we have the host, which is responsible for all the book-keeping, and task scheduling on the OpenCL device.

Then, we have our OpenCL device, which is divided into a number of compute units. Each Compute Unit (CU) is further divided into a number of processing elements.

But what do these words actually mean?

Well, a Processing Element(PE) is a single unit, that is responsible for executing a single thread(also called a work item). Think of a single function being executed.

Each PE has its own private memory, not accessible by anyone, but this PE

A bunch of processing elements are grouped together to form a compute unit which, at a time, executes a single work group(grouping of many work items).

The CUs all share a global memory, accessible by anyone

So for a CPU, the maximum number of CU s is the number of CPU cores!

But why do you want work groups? Why not have work items only?

Well, having this grouping of work items, allows for a greater deal of complexity, because we can synchronize across items in a work group, have a local memory only for this work group, and more…

A complete walkthrough

Let’s look at a typical workflow for an OpenCL program

Initialisation

First we need to check the currently available OpenCL platforms, which are basically implementations of OpenCL available on your system

For example, you can have both Intel OpenCL and POCL OpenCL for your i7 CPU.

Then from these platforms, you need to choose a device to execute on. OpenCL supports CPUs, GPUs, FPGAs, and all sorts of accelerators.

Context

Once you have the platform and device you wish to use, you need to create an OpenCL context, which will handle everything for that particular platform and device.

Command Queue

Then, you have to create a command queue, which, as the name suggests, will store any commands(kernels) you queue for execution, and dispatch them in order(or even out of order if you like!).

Kernel

After the command queue, you must compile the kernel source code(the api provides functions to do this), so that it can be executed later.

Memory

Finally, one of the most important parts of this entire process, is passing the input to the OpenCL device.

Now, initially the data is stored on your CPU RAM, which is unfortunately inaccessible to your GPU.

Therefore you need to copy the data to your GPU RAM, using the cl_mem interface that OpenCL provides.

However, if you know that the device being used is the same CPU, then this copy can be skipped, to save time, using the CL_MEM_USE_HOST_PTR flag while creating a cl_mem object.

Execution

At the end, you can use the command queue created earlier along with the cl_mem created previously to execute the compiled kernel on the device

Subsequently don’t forget to copy the output data back to CPU RAM, if the execution was done on GPU.

However, there’s still a ton of unexplained stuff like, “How do you save the time wasted in copying data to the device?” or “Can you pass any data to the device? Even structs?”

We’ll explore OpenCL more in subsequent posts.

OpenCL, meet the Gnuastro Build System

DeadSpheroid — Sat, 08 Jun 2024 23:45:00 GMT

In this post, I hope to summarize the work done so far towards my GSoC project for integrating OpenCL with the Gnuastro library and my relatively limited understanding of OpenCL.

What is OpenCL?

Open Computing Language is a framework for writing programs that execute across heterogenous platforms. In simpler terms, OpenCL provides a standard interface for programmers to execute the same code across multiple devices, be it a CPU or a GPU or any other accelerator.

It comprises of the OpenCL standard which is maintained by Khronos, and implemented by the various hardware manufacturers and by the open source community across a wide variety of devices.

Most modern devices all support OpenCL in some format or the other. Intel/Nvidia for example provide their own propietary implementations. On the other hand, POCL an open source project provides implementations for those that dont have actively maintained propietary ones, like AMD.

Why OpenCL?

Unlike certain propietary frameworks cough CUDA cough, OpenCL is not constrained to any particular manufacturer. You can target any GPU/CPU as long as you get the OpenCL implementation for that device. This is made easy thanks to projects like POCL.

The performance of CUDA versus OpenCL is heavily debated and leans towards CUDA for Nvidia hardware, but the difference depends on the use case and isn’t too much of a concern as compared to the way they are used.

The downside of OpenCL is the smaller community and the lack of many modern features that CUDA brings.

The inner workings of OpenCL, how I managed to set it up, how the OpenCL C API works is another long story and is deserving of its own post.

Kickoff with Gnuastro

The first goal for the project was to figure out a way to integrate OpenCL with the Gnuastro build system.

Gnuastro like many other free software uses the GNU Build System also called GNU Autotools

The three major components of Autotools are:

Autoconf

At the heart of Autotools, we have Autoconf, which generates a single configure script from a configure.ac file.

This configure script scans the environment for various files and libraries, specific versions of them, the hardware being used, and more. Then, it configures the build of the project in certain ways enabling/disabling certain parts depending on what was found and what wasnt.

In this way, the portability of any project can be ensured by simply distributing the configure script, along with the Makefile.ins.

Automake

Automake makes use of information found by configure and generates the Makefiles necessary to build the project.

To be more precise, it parses Makefile.ams into Makefile.ins which are in turn parsed by configure to produce the final Makefiles. Automake also performs automatic dependency tracking, so that recompilling isn’t done unless required.

All you have to do, is specify the name and each of the sources involved in the library/binary, and Automake does the rest.

Libtool

Libtool is responsible for abstracting the library creation process, since different platforms handle static/dynamic libraries differently.

I mainly worked with Automake and Autoconf during integration and didn't really touch Libtool.

Stepping into Integration

Inside `configure.ac`

Without getting into detail, when checking for the presence of OpenCL, it suffices to check for libOpenCL.so and the CL.h header file.

That is, Gnuastro should be able to include the OpenCL header file to use its C API, and then later link against the OpenCL library.

Luckily for us, Gnulib provides a simple AC_LIB_HAVE_LINKFLAGS macro which takes as input, a library name and a test code and tries to find the library and compile/link the test code.

Upon successfully executing, it sets certain variables, so we can modify further building on the basis of finding OpenCL.

AC_LIB_HAVE_LINKFLAGS([OpenCL], [], [#include <CL/cl.h>])
AS_IF([test "x$LIBOPENCL" = x],
[
if successfull ...
],
[
LIBS="$LIBOPENCL $LIBS"
has_ocl=1;
if unsuccessfull ...
])
AM_CONDITIONAL([COND_HASOPENCL], [test "x$has_ocl" = "x1"])

After making these modifications to configure.ac, we can now test whether OpenCL was found inside the various Makefile.ams and accordingly change the build.

Inside `Makefile.am`

Now, we can use the variable we set previously in configure.ac and either include or exclude the OpenCL modules from being compiled and included in the library.

if COND_HASOPENCL
$(info "Found OpenCL")
MAYBE_CL = cl_utils.c
MAYBE_CL_H = $(headersdir)/cl_utils.h
MAYBE_CONVOLVE_CL = cl_convolve.c
else
$(info "What is Opencl?")
endif

libgnuastro_la_SOURCES = \
$(MAYBE_NUMPY_C) \
$(MAYBE_WCSDISTORTION) \
$(MAYBE_CL) \
$(MAYBE_CONVOLVE_CL) \
arithmetic.c \
arithmetic-and.c \
...

Additionally, we need to save this variable in Gnuastro’s config.h file for later use to prevent other modules from mistakenly including the OpenCL ones incase OpenCL was not compiled.

checking for build system… yes

Now when someone builds Gnuastro, if OpenCL is present on their system, then the OpenCL relevant files are compiled and included in the library.

On the other hand, if OpenCL is absent, then the library is built as normal, as if OpenCL never existed.

Finally, we can get started with the actual OpenCl part and we’ll have a look at Image Convolution(astconvolve) in the next post…

OpenCL, meet the Gnuastro Build System

DeadSpheroid — Sat, 08 Jun 2024 23:00:00 GMT

In this post, I hope to summarize the work done so far towards my GSoC project in integrating OpenCL with the Gnuastro library and my relatively limited understanding of OpenCL.

Final GSoC Report

Labib Asari — Mon, 21 Aug 2023 23:00:00 GMT

I will be discussing the goals of my GSoC project, how I spent my time and what I learned during this period. I will also be discussing the future of my project and what I plan to do next.

Goals of my GSoC

The original Google Summer of Code project this was year was to :

Redisgn the error handling inside Gnuastro C library.
Adding wrappers for Gnuastro library functions in pyGnuastro.

Prior to GSoC, my experience mostly consisted of Deep Learning and Computer Vision. I had a good high-level understanding of how GPUs were leveraged for the compute intensive tasks in various libraries and frameworks in these domains. I had started exploring the lower-level abstractions over GPUs using the CUDA framework.

In the early weeks of February, I delivered a presentation to the Gnuastro development team. The point of this presentation was a proposal outlining the integration of GPU support into Gnuastro — an idea borrowed from the Machine Learning world but with huge advancement potential in the feild of Astronomy. Both of these domains process huge amounts of data. Both of these domains are characterized by the processing of substantial volumes of data.

My mentor Mohammad Akhlaghi was very supportive of this idea and gave me the go ahead to start working on it.

And so, we had a 3rd goal for this GSoC project :

Adding GPU support to Gnuastro.

Work Done Throughout the GSoC

Error Handling

Need : All of Gnuastro’s library functions performed error handling using error(EXIT_FAILURE, ....); thus exiting the program whenever an error was encountered with a detailed error message. This wasn’t a problem for the Gnuastro programs however for other callers like pyGnuastro, this is problematic as it exits from the entire Python environment.

The new error handling mechanism defines a module error.h new data structure gal_error_t. The exact contents of this structure have gone through multiple iterations but the final one is :

The user should define a gal_error_t before the function call and pass it as an argument to the function(every function in Gnuastro will have an extra argument now).

During the function execution, if any error occurs, it will populate the gal_error_t with the error message and the error code. The user can then check the error code and the error message to determine what went wrong.

Corresponding functions are added in error.h for writing and managing the structure. Some methods are also provided for Python interface.

After the module was finished, Mohammad implemented the new error mechanism inside the cosmology.c module, and then I used it to update the corresponding cosmology module in pyGnuastro. This solved the main the problem of python environment exiting on any error, instead errors were being reported inside the python shell.

This completed setting-up the low level infrastructure for the new error handling mechanism. This can be now used by other modules of Gnuastro to update what happens when an error occurs. Implementing the high level error function calls, deciding the exact error type and defining what message should be shown, would be best done by the original authors of the modules.

The new error handling mechanism currently lives at the Gnuastro repository.

pyGnuastro

Apart from implementing the new error handling mechanism in existing modules of pyGnuastro, I worked on 2 major things

Implemented speclines module in pyGnuastro : this is a simple module without any complex data structures. I tried this first when I was learning about the C-Python API. It gave me a good grasp of how and what’s going on in the existing pyGnuastro implementation.
GAL_DATA_T for Python : The core data structure of Gnuastro - gal_data_t is a C struct. Any external data is represented using this structure. It was crucuial to had a similar structure in Python. Previously Jash had worked on loading and saving fits file made use of the Numpy-C API to to convert the raw data inside the gal_data_t to a Numpy array. This was an extremely clever and efficient idea, however it skipped all the other details inside gal_data_t. We had to find a way to represent the entire gal_data_t in Python. The normal way to create a new data structure in Python would be to create a new class. However, the wrappers are written in C language and we don’t get access to the Python interpreter. I took some more inspiration from Numpy on how they created a new Python - their core data structure : numpy.ndarray - using the C-Python API. I then discovered the API allows us to define custom objects which may be used a data type for the Python interpreter. I learnt and used them to have a corresponding pygnuastro.data for pyGnuastro. It basically acted as a new data type in python similar to numpy.ndarray, had other details of gal_data_t.After this we had details of gal_data_t in python but we were missing on Jash’s idea of utilizing Numpy in pyGnuastro. I spent some time to make sure we can still utilize numpy’s speed inside pyGnuastro, The C-Python API is versatile and it allows having complex objects as sub-objects to other objects. Eventually we had the array(raw data) being represented as a numpy.ndarray! This meant we had both the speed of numpy and the details of gal_data_t in pyGnuastro’s pygnuastro.data. This was a major milestone in pyGnuastro.

GPUs in Gnuastro

Gnuastro is an astronomical data analysis and manipulation library. Astronomical data is usually very large in size, and thus computationally intensive. If the operations performed on this data are parallelizable, then GPUs can significantly speed up the processing.

I started my work on GPUs right after Mohammad approved my initial idea. Here’s a summary/story of all the work done for GPU support :

Learning about build systems : After GPU support idea was accepted, my mentor suggested we should first setup the build system so CUDA modules can be integrated smoothly in the future. Gnuastro uses Autotools for its build system. I started by learning about autoconf, automake and libtool.
Linking Gnuastro with CUDA runtime : CUDA SDK provides a runtime library - cudart which the necessay component to initiate communication with the GPU drivers. The runtime library is distributed as both a static and shared object file. This made things easier as we could link the runtime library statically with the Gnuastro library, making cudart part of Gnuastro. I modified the configure script to link the runtime library statically with Gnuastro. This was also the time I learnt extensively about how low level system libraries are built, linked and distributed.
Struggling with Libtool : I then tried to implement some simple matrix functions in CUDA and integrate them with Gnuastro. CUDA source code is compiled by nvcc compiler. However during linking, libtool assumes that all source files are compiled by gcc. It ignored all the CUDA source files. After writing dedicated rules for CUDA source compilation in the Makefile, the CUDA source was getting compiled, but not being linked to the Gnuastro. Libtool only links files having a corresponding libtool object(.lo files) and they’re created by libtool for each source file handled by it(which in our case were gcc compiled files).
AutoMake developers rescuing us : After trying and struggling with libtool for a few days, my mentor suggested that I contact the AutoMake developers to seek some help. I mailed them a small demonstration of what I was trying to do and waited for there response. After a few days, I received a reply from them. The fix was actually simple, automake had special variables(LD_ADD) which directly communicates with the GNU linker (ld) and I just had to add CUDA object files to this variable. It worked and we finally had a working CUDA module in Gnuastro which used GPU for execution!

It was around 1st week of April now, I made my final proposal submission and had fingers crossed for getting selected in GSoC.

As mentioned in the GSoC proposal, we had to first focus on the Error handling and Python wrappers, so I started working on these two goals (I was also indeed selected for GSoC in the meantime!).

Convolutions on GPU : After getting back to working with GPUs in around June-July, I started with implementing the convolution function in CUDA. Convolution is a direct operation as well as a subroutine to other operations in Gnuastro. The results of CUDA convolution were remarkable. We got upto 400x speed up on convolution operation! My mentor then suggested me since the speedup is very significant, I should prioritise getting more of GPU work done. Read more about Convolution on GPU in my blog here.
Adapting OpenCL : CUDA is a proprietary framework by Nvidia. It only works on Nvidia GPUs. We wanted to make Gnuastro GPU support available to all users, irrespective of the GPU they have. This is where OpenCL comes in. OpenCL is an open standard for parallel programming of heterogeneous systems. It is supported by all major GPU vendors. I started learning about OpenCL and how it works at a low level. I also started learning about the OpenCL C99 programming standard. Read more about starting with OpenCL in my blog here.
Integrating OpenCL : OpenCL was initially hard to learn, but I managed to integrate that with Gnuastro right before my GSoC’s official timeline was about to end! I have a pretty detailed blog on the the entire integration process here.
Same code on CPU and GPU : After we had success with OpenCL, my mentor recommended we should try executing the exact same code on CPU and GPU - to show the concept of executing same instructions both processors and seeing the speed-up on GPUs. This was never done in the field of Astronomy so it’d have been a great demonstration. This was quite challenging as GPUs are programmed with different frameworks and have some extra components in code for management. Usually in Machine Learning frameworks, the GPU and CPU modules are generally written seperately(Infact Tensorflow used to have different package altogether for GPU until 2.0) However the good part is, most of the GPU frameworks are derived from C/C++ language and have . I spent my last week of GSoC trying to implement the core logic in a Macro which will be shared by both OpenCL kernels and C library and had success, this can be accessed here.

Future : The future of this project is very bright. I have set up the bare-bone GPU integration already, I’ll continue to add GPU modules building upon it.. We have a working OpenCL integration. We have a working CUDA integration. We have a working CPU-GPU code sharing. I mentioned certain challenges we are currently facing in my opencl_integration blog. I’ll continue to figure out a solution for them and adding support for further modules on GPU.

Acknowledgements

GSoC has been a great learning experience for me. I’m extremely grateful to everyone who was part of this journey.

I would like to thank my mentor Mohammad Akhlaghi for his constant support and guidance throughout the project. He has been very patient right from the beginning, beleived in me when I did not have a clear idea on how I’d approach all the goals. He allowed me work on my pace, explore and learn things as needed and has always pulled me out of the rabbit hole whenever I got stuck. Everytime I join a meeting with him, I learn something new. I’m very grateful to him for giving me this opportunity to work on this project.

I am Graciously thankful to Jash Shah for introducing me to the Gnuastro development team and walking me through the existing work on error handling and pyGnuastro. It provided me a huge boost was extremely valuable. He’s always been attentive to my small queries and has supported me through multiple challenges. In general, Im very grateful to have him as a mentor and freind.

I would also like to thank the Gnuastro development team for their support and feedback throughout the project. Its been such a wonderful time working with them. I have learnt a ton from attending Pedram’s work on adding Sql to Gnuastro, Fathma’s work on Tiff files and Curl library, Faezeh’s work on implementing Convolutional Neural Networks in Gnuastro. They’ve always been crucial in providing feedback and suggestions on my work. I’m very grateful to them for their support. I am genuinely grateful for the opportunity to collaborate with such a talented and committed group, and I look forward to work and grow with them in the future.

I would also like to thank the Google Summer of Code team for taking the wonderful initiative and giving me this opportunity to work on this project.

Integrating OpenCL with Gnuastro

Labib Asari — Fri, 11 Aug 2023 23:00:00 GMT

Background

In the last post, I discussed what is OpenCL and why we chose to integrate it with Gnuastro. In this post, I’ll be discussing the actual implementation and the challenges I faced.

Programming in OpenCL

The OpenCL 3.0 standard has done a great job of simplifying the programming model. The OpenCL 3.0 API is a header-only library that provides a modern, object-oriented interface to the OpenCL runtime. It is designed to be easy to use and provides a abstraction of the OpenCL runtime, making it easier to write portable code across different OpenCL implementations. We still have to communicate with the driver (unlike CUDA) at a low level, but this becomes a mandatory step when we want to run our code on different hardware (CUDA always expects an NVIDIA device).

Here’s a general overview of steps to be followed when writing an using OpenCL :

Check for available Platforms : A platform is a collection of OpenCL devices. A platform can be a CPU, GPU, or an FPGA (Remember OpenCL can work with any platform!). This is done specifically to identify which OpenCL implementation will be used during runtime. We can query the system for available platforms using the clGetPlatformIDs function. This function returns a list of platforms available on the system.
Check for available devices : A device is a physical device that can execute OpenCL kernels. A device can be a CPU, GPU, or an FPGA. We can query the system for available devices using the clGetDeviceIDs function. This function returns a list of devices available on the system.
Create a context : A context is a container for all the OpenCL objects. It is used to manage the memory, command queues, and other OpenCL objects. It is created by passing a list of devices to the constructor. Since OpenCL can work with multiple devices, we can create a context with multiple devices. This is useful when we want to run our code on multiple devices at the same time.
Create a command queue : A command queue is used to queue up commands for the device to execute. The command queue is used to give commands to the device. The device executes the commands in the order they are received. The commands can be kernel execution, memory transfer, or any other OpenCL command. We can also create multiple command queues. This is useful when we want to run to multiple commands. Command queues in OpenCL are asynchronous by default. This means that the commands are queued up and the control is returned to the host. The host can then continue with other tasks. We can also create a synchronous command queue. This means that the commands are queued up and the control is returned to the host only when the commands are executed.
Load the Kernel : A kernel is a function that is executed on the device. It is written as per the C99 standard. We can load the kernel from a file or we can write the kernel inline. To maintain portablitiy, OpenCL kernels are generally compiled at runtime using clBuildProgram. We can also compile the kernel offline. This is useful when we want to compile the kernel for a specific device.
Copy Data to device memory : All the data used in kernel, must be on the device memory. So we have to copy the data from the host to the device memory. We can do this using the clCreateBuffer function. This function creates a buffer on the device memory. We can then copy the data from the host to the device using the clEnqueueWriteBuffer function. This function copies the data from the host to the device.
Launch the kernel : We can launch the kernel by passing the kernel object to the command queue. We have to set the arguments for the kernel seperately, using the clSetKernelArg function. We can also set the global and local work size. The global work size is the total number of work items that will be executed. The local work size is the number of work items that will be executed in a work group. The global work size should be a multiple of the local work size. If the global work size is not a multiple of the local work size, then the global work size is rounded up to the next multiple of the local work size.
Read the output : We can read the output from the device using the clEnqueueReadBuffer function. This function copies the data from the device to the host.

Implementation

Among all the steps mentioned above, everything up till loading the kernel is common to all the programs that’ll be using OpenCL. So we defined a gpu_utils module which is responsible for querying for the available platforms and devices, creating the context and command queue, loading and compiling the kernel. The only external data it requires is the path to the kernel file. This is provided as an input. It also provides utility functions to copy specific data types to and from device memory.

There’ll be 2 types of OpenCL program in Gnuastro :

Programs using OpenCL to speed-up existing operations inside Gnuastro.
User defined OpenCL kernels, responsible for performing a custom task.

Programs using OpenCL to speed-up existing operations inside Gnuastro

These programs will be using OpenCL to speed-up existing operations inside Gnuastro. For example, we can use OpenCL to speed-up the astconvolve operation by passing an extra --gpu. For these programs, the OpenCL kernels will be part of the Gnuastro Library.

The general flow of the program then becomes :

The user passes the input data for a specific operation, and also choses the local and global work size.
The program then initializes the device using gpu_utils module by providing the kernel file from the library, which does everything and returns a cl_kernel (which is essentially the compiled kernel).
Data transfer from CPU to device (GPU) is done using the functions provided by gpu_utils module.
The kernel is launched using with the provided global and local work size.
Data is copied back to CPU memory and returned to the user.

User defined OpenCL kernels, responsible for performing a custom task

These programs will be using OpenCL to perform a custom task. For example, we can use OpenCL to perform a custom convolution operation by passing a custom kernel. For these programs, the OpenCL kernels will be provided by the user. The exact design details yet to be determined for this.

Results

Input image is 10,000 x 20,000 random image with normal distribution. Kernel is 7 x 7 standard convolution kernel. CPU : Intel(R) Core(TM) i5-9300HF CPU @ 2.40GHz GPU : NVIDIA GeForce GTX 1650

Convolution using existing convolution in Gnuastro :

Convolution on OpenCL :

Result

The speed up for convolution operation is specifically ranges from 300-500x, but for the entire operation its around 3-5x due to the overhead of copying data to and from the device. Overcoming this is a big and important challenge!

Challenges

No GAL_DATA_T inside OpenCL kernel! : Inside OpenCL, cl_mem is the primary object used to represent memory objects such as buffers and images. It is used to allocate memory on the device. Regardless of where the data is coming from on device (arrays, structs, etc), it’s all converted into a cl_mem object when copied to the device.

However inside Gnuastro, the core data structure is gal_data_t which is essentially just a C struct.

Why is this a problem? Well the raw data of the input image/table is not contained inside the gal_data_t. It merely consists a pointer to that data! So wehn we copy the gal_data_t to device, the raw data(which is huge) is not copied. (It lives on the CPU memory, and hence cant use CPU pointers on GPU memory).

What about copying the raw data seperately on the GPU memory, and then replacing the pointer inside gal_data_t with a pointer which has the address on the GPU memory? Well, this is not possible either. Why? See, when we are on CPU, we’ve a good gal_data_t struct which is a single big object with ‘sub-objects’(one of which is the pointer). But on GPU, we’ve a cl_mem which is an object, but unlike structs, it cant have sub-objects!

How do we solve this? Currently all the required pointers inside gal_data_t are passed as seperate arguments to the kernel. After a careful study of the internal implementation of the cl_mem object, we’ll see if we can directly pass the gal_data_t to the kernel.

Data Transfer Overhead : As mentioned multiple times, for using GPUs, we must copy data to and from the GPU memory. Astronomical datasets are huge, and copying them for each operation is a big overhead! Infact the data transfer overhead is so huge, that the actual operation is much faster than the data transfer. Adding more to that, its not just faster, its much much faster! So much so that around 95% of the time is spent in copying data to and from the GPU memory. It reduces performance by ~100x! It can’t continue this way!

One solution we’ve figured is, when the External data is loaded for the first time in the program, we load it on the GPU memory instead of the CPU memory. This way, for each subsequent operation, we dont have to copy the data from CPU to GPU memory. After all the operations are done, we’ll copy the result back to CPU memory and save it to the disk. This will avoid almost all the Data Transfer overhead.

This is about the same approach used by Machine Learning Libraries such as Tensorflow. Basically during initialization, it occupies all the GPU memory it can, and keeps it occupied. All the operations, their results and the subsequent operations are done on the GPU memory itself.

Moving towards OpenCL

Labib Asari — Thu, 27 Jul 2023 23:00:00 GMT

Background

So far, all my work on GPUs has been using CUDA. But CUDA is proprietary to NVIDIA and only works on NVIDIA GPUs. So, I’ve been working on moving the code to OpenCL, which is an open standard for parallel programming on heterogeneous systems.

OpenCL

OpenCL(Open Computing Language) is an open standard for cross-platform, parallel programming of diverse accelerators(CPUs, GPUs, FPGAs, etc) found in supercomputers, cloud servers, personal computers, mobile devices and embedded platforms. Note the 2 key points -

open standard : this means that the specification and documentation of the technology are publicly available and can be accessed by anyone.
cross-platform : this means that it can run on multiple operating systems and hardware architectures without requiring major modifications to the code.

This makes OpenCL a very attractive option for developers who want to write code that can run on a wide range of devices. From Gnuastro’s perspective, this means that we can write code that can run on multiple GPU manufactureres, as well as CPUs and other accelerators. Our GPU kernels will be portable to any system, regardless of its configuration!

Next point to consider is OpenCL is a standard. It is different from CUDA in this regard. CUDA is a framework, whereas OpenCL is a standard. What does this mean?

The OpenCL standard refers to the specification and guidelines set forth by the Khronos Group which is responsible for developing and maintaining the standard. The OpenCL standard defines the API, data types, functions, and programming model that developers must follow when writing code for OpenCL. It is a formal document that ensures uniformity and compatibility across different OpenCL implementations.

OpenCL is not an open-source library! It basically defines how the library should behave(big simplification!).

So what can we do with the standard alone? Not much! We need an implementation of the standard.

This also reminds me of the question I once had - What do you need to create a new programming language? My first guess was a compiler! My thought process was if a program(compiler in this case) can understand my High level language and convert it to corresponding machine code, then I can write programs in that high level language for any task! So all I’d need is a compiler for that language. Its close, but not totally accurate.

You dont actually need a compiler for a new programming language. You ONLY need a specification for it. The specification will define the syntax and semantics(rules) of the language. You only need a compiler when you want to run programs using your language!(what good is a language if you cant run programs using it? haha)

Similaraly OpenCL defines a set of rules which specify how it will behave. But to use OpenCL we need an implementation of this standard.

OpenCL implementations are software packages developed by hardware manufactureres that provide the necessary drivers and runtime libraries for running OpenCL applications on their specific hardware. Each hardware vendor is responsible for creating their own OpenCL implementation that conforms to the OpenCL standard. This means that each implementation may have its own unique features and quirks, but they all adhere to the same standard.

There are many different implementations available for it! (find the full list here or here).

Basically each of the hardware manfacturers provide an implementation of the OpenCL standard for their hardware. This implementation is usually provided as a framework. Depending on what hardware you have on your system, you can choose the corresponding framework to use.

How does OpenCL work?

Here’s waht a typical OpenCL system looks like :

OpenCL programs consist of two parts: host code and device code. The host code is written in C or C++ and runs on the host, while the device code is written in OpenCL C and runs on the device. The host code is responsible for setting up the OpenCL environment, creating the context, compiling the device code, and executing the kernels on the device.

The device code is compiled at runtime by the host code. This means that the host code must be compiled first, and then the device code can be compiled. The host code is compiled using a standard C/C++ compiler, while the device code is compiled using the OpenCL compiler. The OpenCL compiler is provided by the OpenCL implementation and is responsible for compiling the device code into binary code that can be executed on the device.

How does the OpenCL library interact with the hardware? Its made possible through OpenCL-ICD.

OpenCL ICD stands for OpenCL Installable Client Driver. It is a component of the OpenCL

It enables multiple manufacturers OpenCL drivers to coexist on a single system. Instead of having a single monolithic OpenCL driver, an ICD allows different manufactureres (e.g., NVIDIA, AMD, Intel) to provide their own separate OpenCL implementation as dynamically loadable libraries. This means that developers can select the appropriate OpenCL driver at runtime without needing to modify their applications.

The ICD mechanism is crucial for achieving portability and flexibility in developing applications using computational power of various devices from different manufacturers.

OpenCL Programming Model

The Programming Model of OpenCL is very similar to CUDA which I covered in my previous post. However CUDA has a lot of abstraction since it has its own runtime library which communicates with the driver. In OpenCL there’s direct communication with the drivers and the host code is responsible for setting up the environment so its a bit more lower level than CUDA.

Some of the key terms in OpenCL are :

Work Item: Basic unit of work on a compute device
Kernel: The code that runs on a work item (Basically a C function)
Program: Collection of kernels and other functions
Context: The environment where work-items execute (Devices, their memories and command queues)
Command Queue: Queue used by the host to submit work (kernels, memory copies) to the device.

I’ll cover the programming aspect of OpenCL in more detail in my next post.

GPUs and Convolutions in Gnuastro

Labib Asari — Mon, 03 Jul 2023 23:00:00 GMT

Background

This is an overview of what I’ve been upto for the past 2 weeks. Doesn’t go into much technical details and the actual code but just walks through the general idea.

Convolution is a fundamental operation in various domains, such as image processing, signal processing, and deep learning. It is an important module in Gnuastro and is also used as a subroutine in other modules.

Convolutional operations can be broken down into smaller tasks, such as applying the kernel to different portions of the input data. By utilizing multiple threads, each thread can independently process a subset of the input, reducing the overall execution time. This parallelization technique is particularly effective when dealing with large input tensors or performing multiple convolutions simultaneously.

While traditional CPUs (Central Processing Units) excel at performing a wide range of tasks, they are not specifically designed for heavy parallel computations like convolutions. On the other hand, GPUs (Graphics Processing Units) are highly optimized for parallel processing, making them ideal for accelerating convolutional operations.

GPUs vs CPUs Architecture

Cores and Parallelism :

CPUs have fewer, more powerful cores optimized for sequential processing, while GPUs have thousands of smaller cores designed for parallel processing. This parallelism allows GPUs to perform computations on multiple data elements simultaneously, leading to significant speedup in parallelizable tasks like graphics rendering and deep learning.

Memory Hierarchy :

CPUs typically have larger caches and more advanced memory management units (MMUs), focusing on low-latency operations and complex branch prediction. GPUs, prioritize high memory bandwidth and utilize smaller caches to efficiently handle large amounts of data simultaneously, crucial for tasks like image processing and scientific simulations.

Emphasis :

CPUs are designed with an emphasis on executing single threads - very fast. GPUs are designed with an emphasis on executing on executing multiple threads.

Programming Model

For Programming GPUs, several frameworks (high level APIs) are available

CUDA - developed by NVIDIA for its GPUs.
OpenCL - Open Source, Cross Platform parallel programming standard for diverse accelerators.
HIP - developed by AMD, portable.
and many more….

CUDA

The CUDA platform consists of a programming language, a compiler, and a runtime library.

Programming Language - Based on C, has extensions to write code for GPU.
Compiler - Based on clang, offloads host code to system compiler and translates device code into binary code that can be executed on the GPU.
Runtime Library - Provides the necessary functions and tools to manage the execution of the code on the GPU (interacts with the driver).

Note : When we have multiple devices(GPUs, FPGAs, etc) on a single system, which can execute tasks apart from the main CPU, they’re generally referred to as device whereas the main CPU is referred to as host.

CUDA Programs

CUDA programs consists of normal host code along with some kernels. Kernels are like other functions, but when you call a kernel, they’re executed N times parallely by N different CUDA threads, as opposed to only once like normal functions. They’re defined using the __global__ keyword.

Eg :

Normally, we put the above piece of code inside a loop, so all elements are covered.

With GPUs, there’s no need for loops - for N elements, we launch N threads each of which add 1 element at the same time!

CUDA Execution Configuration

Can we launch an arbitrary large number of threads? Technically No

The maximum allowed threads depend on your GPUs compute capability.
But generally it’s so large, it always covers all your elements
For Compute Capability > 3.0
- Max Number of threads : (2^31)(2^16)(2^16)(210) = 2^42!

Threads and Blocks :

All threads are organized into groups called - Block.
All blocks are organized into groups called - Grid.

Blocks and Grids could be a 1D, 2D or 3D structures.

When calling a GPU kernel, we specify the structure of each block, number of blocks, and number of threads/block - This is called the Execution Configuration.

Example :

The above code Launches 32321 = 1024 blocks Each having 1616 = 256 threads Total no. of threads = 1024256.

CUDA Memory Hierarchy

CUDA threads may access data from multiple memory spaces during their execution as illustrated above.

Local memory for each thread.
Shared memory b/w all threads of same block.
Global memory b/w all blocks.

CUDA Hardware abstraction

The entire GPU is divided into several Streaming MultiProcessors (SMs). They have different architecture than a typical CPU core. Each SM has several CUDA cores, which are the actual processing units.

It is designed with SIMT/SIMD philosophy, which allow execution of multiple threads concurrently on them. One Block is executed at a time on a single SM.

CUDA Developing Workflow

Results of Convolution on GPU for Gnuastro

All tests were performed on a system with the following specifications:

CPU :

Intel(R) Core(TM) i5-9300HF CPU @ 2.40GHz
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
CPU max MHz: 4100.0000
CPU min MHz: 800.0000

GPU :

NVIDIA GeForce GTX 1650
Turing Architecture
Driver Version: 535.54.03
CUDA Version: 12.2
VRAM : 4GB
Compute Capability : 7.5

The input image was a 10k x 20k FITS file with 32-bit floating point values. The kernel was a 3x3 matrix with 32-bit floating point values.

CPU Multi-threaded

GPU

The overall speedups seems to only be 6X but this also counts the time taken to transfer the data from CPU to GPU and back. If we only consider the time taken to perform the convolution, the speedup is around ~700X!.

Universe OpenAstronomy (Posts about gnuastro)

Final GSoC Report

About OpenAstronomy

About Gnuastro

Goals of the project

Pre GSoC

During GSoC

Build System Integration

Wrapper Infrastructure

Initialisation

Data Transfer Functions

Executing Kernels

Parallelized Subroutines

Same code on CPU and GPU

Convolution

Additional features

Optimised Convolution

Revisiting Same Code on CPU vs GPU

Using the OpenCL modules in your program

Initialize OpenCl

Transfer Input to Device

Write an OpenCL Kernel

Spin Off Threads

Copy Output back to Host

Post GSoC

Acknowledgements

Towards New Speeds

Let’s get started

But what’s the point of so many cores??

So whats the catch?

Well, how do I use my GPU?

Image Convolution

A new outlook

Exploring OpenCL memory management

The basics

Buffers

CL_MEM_USE_HOST_PTR

CL_MEM_COPY_HOST_PTR

CL_MEM_ALLOC_HOST_PTR

Literacy for Buffers

Read/Write Buffer

Map/Unmap Buffer

To map or not to map?

The problem with buffers

Deeper into OpenCL

Setting it up

Here’s a new perspective

A complete walkthrough

Initialisation

Context

Command Queue

Kernel

Memory

Execution

OpenCL, meet the Gnuastro Build System

What is OpenCL?

Why OpenCL?

Kickoff with Gnuastro

Autoconf

Automake

Libtool

Stepping into Integration

Inside configure.ac

Inside Makefile.am

checking for build system… yes

OpenCL, meet the Gnuastro Build System

Final GSoC Report

Goals of my GSoC

Work Done Throughout the GSoC

Error Handling

pyGnuastro

GPUs in Gnuastro

Acknowledgements

Integrating OpenCL with Gnuastro

Background

Programming in OpenCL

Implementation

Programs using OpenCL to speed-up existing operations inside Gnuastro

User defined OpenCL kernels, responsible for performing a custom task

Results

Inside `configure.ac`

Inside `Makefile.am`