A real-time post-correlation beamformer and correlator for the SPOTLIGHT

Abstract

SPOTLIGHT is a time-domain survey instrument to perform a real-time commensal search for FRBs (Fast Radio Bursts) and Pulsars with a PetaFlop system hosting 60 CDAC’s indigenously developed Rudra servers equipped with 90 A100 GPUs connected to a 2 PB storage over a 100 Gbps Infiniband network. The system executes real-time HPC and AI applications to ensure simultaneous time-domain detection and arc-second imaging localisation of the detected bursts across the GMRT observing band of 300 to 1460 MHz. The 60 Rudra servers are arranged in three sub-clusters implementing the correlator, transient search and imaging analysis. The first cluster, consisting of 16 servers, performs correlation and beamformation forming 2000 beams at 1.3 millisecond time-resolution and sends the 2000 beams to the 24 servers of the second cluster for multi-beam transient detection pipeline to search for FRBs using Machine Learning for classification. The 2000 beams are optimally located well within the half-power beamwidth of the primary beam of a GMRT antenna. Imaging localisation on the detected FRBs and candidate confirmation is done on the 20 servers of the third cluster.

The first stage of the system is equipped to accept Nyquist sampled digitised data of baseband signals of both polarisations of all antennas of GMRT at the rate of 25 GB/s over 32 ten gigabit fibre links to perform correlation and beamformation. The data ingestion to the SPOTLIGHT system is done via UDP packet broadcasting from the GMRT samplers, ensuring piggy-back operation with the GMRT Wide-band Backend. The received digitised data on each server over two ten gigabit fibre links is buffered, time-sliced and shared between the servers on a 100 Gbps network such that each server gets a slice of contiguous time series digitised data of all antennas, on which it performs operations like FFT, Multiplication and Accumulation (MAC) and beamformation using dual A100 GPUs. MAC is designed to give visibilities at 1.3 ms time resolution. The post-correlation beams are formed using the phased addition of the 1.3 ms time resolution visibilities. The post-correlation beams reduce the effect of radio frequency interferences and systematics in the data by eliminating the auto-correlation products (Figure 1). Cleaning the dynamic spectrum and more importantly, the dispersion measure-time image significantly increases the efficiency of the AI classifier. The beam-steering implemented at the post-correlation stage significantly minimizes computational costs. The 2000 beams thus formed are distributed over the servers of the second stage such that each server receives data of ~84 beams. The beam data received is buffered in a shared memory which is accessed by the multi-beam transient detection pipeline for pulsar and FRB searches.

The 1.3 ms time resolution visibilities formed in the first stage are integrated up to 1 second or higher and sent to a dedicated host node for archiving. This visibility data at this reduced time-resolution is used to detect the non-working antennas to remove them from the array and also to perform real-time phase and amplitude calibration of the data. In the first stage, in parallel to performing correlation and beamforming, the digitised Nyquist data is written to a shared memory. Upon detection of a transient event in the second stage, this baseband shared memory is accessed to write the Nyquist sampled voltages to the 2 PB storage over 100 Gbps network. The availability of the full Nyquist samples from the full array covering the dispersion sweep of the FRBs opens up the possibility of probing the underlying emission physics aided with full polarisation information and the propagation effects at an unprecedented sensitivity. In addition to this baseband buffering, in the second stage, the 1.3 ms time resolution visibilities are buffered in another shared memory which is accessed to write these visibilities to the 2 PB storage upon detection of a burst. This facilitates real-time imaging localisation of the FRB at arc-second resolution, enabling the host galaxy association to use the burst as a cosmological tool. The sizes of shared memory for the beam data (~ 2.4 TB) and 1.3 millisecond resolution visibilities (~ 5 TB) in the second stage and the baseband shared memory (~ 4.8 TB at 4-bits per sample) in the first stage are large enough to hold the data on Random Access Memory to accommodate detection of bursts even at high dispersion measure.

The SPOTLIGHT system is being realized in a phased manner. The entire hardware installation was completed by October 2024. Following this, the software pipelines are being tested with an increasing number of beams progressively. The real-time system currently being tested at the GMRT can generate 800 beams, along with all promised data products, using 32 A100 GPUs, achieving 40% of the final specification.