The South Pole Telescope in docking position during EHT preparations in January 2017. Image credit: Daniel Michalik / South Pole Telescope
Though they were capable of increasing the capacity and the telescope can collect data over the cosmological distances, there also remained more challenges. During the course, each observation generated about 1 petabyte of data which is much heavy to transfer using digital techniques. So, the scientist has to collect the collected data manually by using high-tech hard-drive to a common data centre and then process them as the determined standards.
To transfer data they used the traditional method of data transfer by carrying each harddrive from the observation station to the processing center. It has been heard from an involved person that due to the extreme coldness, they shifted the announcement by six months as one of the observatory and data collection center was at Antarctica.
To make sure that the designed system fully works, the scientific team also took consideration of the high altitude observatories. The system requirements also depend on the altitude. So, for the operation at high altitude, helium-filled hermetically sealed hard drives are used as a storage device. Using this will avoid the need to build pressure chambers around the recorders. As a consequence hard drives collectively accumulated over 10,000 hr of recording at up to 5100 m altitude without a disk failure during the 2017 observations.
Related: First Image of Black Hole (Part-Two); How Scientists saw the Unseen Black Hole?
Dimitrios Psaltis, EHT project scientist and an astronomy professor at the University of Arizona said, "There's no way that we can transfer this data through the internet." Psaltis added, "So, what we actually do is, we take our hard drives and we FedEx them from place to place. This is much faster than any cable that you can ever find."
In order to reconstruct the brightness distribution of an observed source, Very-long-baseline interferometry (VLBI) requires cross-correlation between the individual signals recorded independently at each station, brought to a common time reference using local atomic clocks paired with the Global Positioning System (GPS) for coarse synchronization. The resulting complex correlation coefficients need to be calibrated for residual clock and phase errors and then scaled to physical flux density units using time-dependent and station-specific sensitivity estimates. Once this process was completed, further analysis and refinement of the image were done by using model-dependent self-calibration techniques.
The data M87 was taken by seven stations at five geographic sites on 2017 April 5, 6, 10, and 11. All stations except ALMA and the JCMT recorded two polarizations and two frequency bands. ALMA recorded dual linear polarization that was later converted to a circular basis via PolConvert and the JCMT recorded a single circular polarization.
To work on the requirements they outlined processing pipelines and characterization of the data that occurred over a series of five internal engineering releases, during which site-specific data issues were identified and mitigated in correlation and post-processing. They developed three independent processing pipelines for the initial fringe detection, phase calibration, and reduction of EHT data.
It was also more important to calibrate a meaningful image form those collected huge data. For this work, the EHT imaging team was subdivided into four separate imaging teams. To make sure that the image from one team does not influence the other, the four teams were blind to each other’s work and prohibited from discussing their imaging results and even from discussing aspects of the data. Once they determine the ‘fiducial’ image of M87 from each imaging method on each observing day they performed additional validation tests. For this, they compared the residual telescope gains estimated for M87 with those for the calibrator source (in this case 3C 279) to determine whether or not the significant variation seen in the inferred LMT residual gains are consistent between the two sources. After this comparison, they also performed data resampling tests, and for this, they used identical scripted reconstructions on data sets that exclude individual sites. This will help them to assess how sensitive the image morphology is to the loss of different combinations of data.
References and Further Readings:
-
First M87 Event Horizon Telescope Results. III. Data Processing and Calibration, DOI: 10.3847/2041-8213/ab0c57
-
First M87 Event Horizon Telescope Results. IV. Imaging the Central Supermassive Black Hole, DOI: 10.3847/2041-8213/ab0e85