Challenges for calibration and analysis pipelines
Thijs van der Hulst
Kapteyn Institute
Thijs van der Hulst
PHISCC2015 Rutgers University 16 - 18 March
Challenges for calibration and analysis pipelines
can clever algorithms replace a life time experience?
Thijs van der Hulst
Kapteyn Institute
Thijs van der Hulst
PHISCC2015 Rutgers University 16 - 18 March
Challenges for calibration and analysis pipelines
can clever algorithms replace a life time experience?
•
•
•
•
Thijs van der Hulst
From visibilities to science ready data
Forthcoming HI surveys
Source finding and characterisation
Visualisation challenges
Thijs van der Hulst
Kapteyn Institute
PHISCC2015 Rutgers University 16 - 18 March
Steps from visibilities to science ready data
•
flagging for RFI, system problems, etc.
baseline length =>
!
!
!
!
!
!
!
•
applying amplitude and phase (self) calibration
•
continuum subtraction
!
!
courtesy Tom Oosterloo
frequency =>
•
frequency =>
frequency =>
imaging including deconvolution
Thijs van der Hulst
PHISCC2015 Rutgers University 16 - 18 March
Number of detections in planned HI surveys
rendering by Davide Punzo
APERTIF APERTIF DINGO
Survey: Wallaby shallow
survey medium deep
>10
beams
800
300
30
3
3-10
beams
35000
10000
2000
200
<3
beams
500000
125000
25000
5000
estimates from Duffy et al. MNRAS 426, 3385, 2012
Thijs van der Hulst
PHISCC2015 Rutgers University 16 - 18 March
Required tools
•
•
•
•
•
Data flagging (good experience with LOFAR)
Continuum subtraction (requires knowledge of the HI emission)
Source finding (demo by Paolo Serra)
Source characterisation (talk by Nadine Giese)
Interactive and quantitative visualisation (talk by Davide Punzo)
and
•
•
•
Thijs van der Hulst
Algorithms must be able to handle ~ 4 Tb/day
Integrate source finding and HI imaging pipeline
Visualisation/modelling capabilities required at desk top level
PHISCC2015 Rutgers University 16 - 18 March
Source finding with SoFiA
(Serra et al. MNRAS 448, 1929, 2015)
Thijs van der Hulst
24 pointing mosaic with the WSRT
PHISCC2015 Rutgers University 16 - 18 March
An example data cube of ~ 8 square degrees
Rendering by
Davide Punzo
•
•
Thijs van der Hulst
Perseus-Pisces filament
Ramatsoku et al. 2015
one APERTIF pointing will deliver ~100 detections occupying ~ 109 voxels
one pointing has ~1011 voxels (~1 TB): so 99% is noise
PHISCC2015 Rutgers University 16 - 18 March
Characterizing 3D structures automatically
galaxies in the Ursa Major cluster (courtesy Busekool and Verheijen)
Thijs van der Hulst
rendering by Davide Punzo
PHISCC2015 Rutgers University 16 - 18 March
3D Visualisation and smoothing
original cube
UGC12808
WHISP
rendering by Davide Punzo
SDSS
Thijs van der Hulst
PHISCC2015 Rutgers University 16 - 18 March
3D Visualisation and smoothing
smoothed cube
UGC12808
SDSS
WHISP
rendering by Davide Punzo
Thijs van der Hulst
PHISCC2015 Rutgers University 16 - 18 March
Summary
HI surveys require clever pipelines dealing with:
!
•
•
•
•
RFI rejection
selfcalibration, in some cases direction dependent
continuum subtraction avoiding line emission
high fidelity, fast deconvolution
!
and
!
!
• 3D source finding and source characterisation
• error analysis and reliability assessment of results
• quantitative and interactive 3D/2D visualisation tools
Thijs van der Hulst
subject of PhD of Nadine Giese and Davide Punzo
PHISCC2015 Rutgers University 16 - 18 March