1. No category

EISCAT Metadata: current status and future
requirements for 3D data
Carl-Fredrik Enell, EISCAT HQ
EGI-CC and NEIC projects
EISCAT 3D User meeting, Uppsala May 2015
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
1 / 26
1
Current EISCAT data
2
Current metadata formats
3
Requirements on EISCAT 3D data format
4
Requirements on EISCAT 3D metadata
5
E-infrastructure projects
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
2 / 26
Outline
1
Current EISCAT data
2
Current metadata formats
3
Requirements on EISCAT 3D data format
4
Requirements on EISCAT 3D metadata
5
E-infrastructure projects
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
3 / 26
Levels of data
1
Voltage domain (“raw”)
2
ACF domain or similar (“correlated”)
3
Analysed products (fitted physical parameters)
4
Visualisations and publications
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
4 / 26
EISCAT instruments
Ionosondes
UHF radar
VHF radar (tristatic)
Svalbard radar
HF radar (Heating)
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
5 / 26
Ionosonde data
Dynasonde (DSND) analysis
Tromsø and Svalbard
Level 1 (raw samples) data: archive in Tromsø (CDs)
Levels 2, 3 (identified echoes, analysed parameters) SQL database
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
6 / 26
Incoherent scatter radar data
“Raw”
Old system (mainland pre-2000): converted tape data
New systems:
I
I
lag profile data (level 2) in .mat files
sometimes voltage domain (level 1)
Archive in Kiruna
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
7 / 26
Incoherent scatter radar data
Analysed
GUISDAP analysis in Madrigal database
http://www.eiscat.se/madrigal NCAR and HDF5 files
Vector velocities (not in present routines)
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
7 / 26
Outline
1
Current EISCAT data
2
Current metadata formats
3
Requirements on EISCAT 3D data format
4
Requirements on EISCAT 3D metadata
5
E-infrastructure projects
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
8 / 26
Ionosondes
List of CDs
http://www.eiscat.uit.no/heating/Dynasonde/logs/dyn_
soundings_index.html
SQL database with PHP interface
http://dynserv.eiscat.uit.no
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for 3D
meeting
data 2015
9 / 26
Heating / HF radar
Paper log, scanned at
http://www.eiscat.uit.no/DataBases/heating_logs/
Some EISCAT site log information
EROS experiment files since 2009
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
10 / 26
Incoherent scatter radars
Experiment configuration
EROS files
GUISDAP analysis configuration
Some documentation for some experiments
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
11 / 26
Incoherent scatter radars
Experiment scheduling
http://www.eiscat.se/schedule/schedule.cgi
Raw data archive (index of level 2+1 files)
SQL database
I
I
tape archive, index of experiment data
tape archive tapes, index of old tapes that have been converted
Access through schedule pages (URL above)
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
11 / 26
Incoherent scatter radars
Raw data files
.mat, compatible with Matlab v4, libraries for most programming
languages
Contain parameter block d parbl
I
I
Format of parameter block has changed
Somewhat different for UHF, VHF, ESR
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
11 / 26
Incoherent scatter radars
Analysed data: Madrigal database
Local data: NCAR and HDF5 files in directories
Sites share metadata: Text file indices
Search
I
I
Web interface
APIs
EISCAT: http://www.eiscat.se/madrigal
More info: http://www.openmadrigal.org
Data portal systems
ESPAS http://www.espas-fp7.eu
ENVRI project
Each tends to invent its own data ontology and metadata format although
based on some standards
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
11 / 26
Issues with current data and metadata systems
Confusing
Many systems to maintain
Do not scale to volumetric data
I
I
metadata of sites, beams. . .
data volumes
No persistent object identifiers
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
12 / 26
Outline
1
Current EISCAT data
2
Current metadata formats
3
Requirements on EISCAT 3D data format
4
Requirements on EISCAT 3D metadata
5
E-infrastructure projects
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
13 / 26
Scaling
Raw voltages
Beam-formed
I
I
M sites
N beams
Analysed
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
14 / 26
Data volumes
Site ring buffer > 80 TB
Archive
I
I
I
about 1 PB per year
Disks: few PB
Tapes: ≈20 PB
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
15 / 26
File format
Suggest HDF5
I
I
I
Standard
Flexible from user point of view
High performance
OpenRadar
http://www.openradar.org
I
Discussed earlier at user meetings
I
Defines HDF5 structure with metadata for voltage domain samples
I
Not for 3-D
I
Not for correlated or analysed products
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
16 / 26
Outline
1
Current EISCAT data
2
Current metadata formats
3
Requirements on EISCAT 3D data format
4
Requirements on EISCAT 3D metadata
5
E-infrastructure projects
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
17 / 26
Time stamps
Inter-site beamforming
Distributed network timing
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
18 / 26
Experiment mode
Beam forming
Pulse coding
Low or high power mode, etc
Make it possible to understand an experiment without going into
configuration
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
19 / 26
Object IDs
Timing foremost
Traceability, data provenience
Use standard ID scheme
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
20 / 26
Experiment coordination
Science needs multi-instrument observations
Timing most important way of finding data
I
Synchronization: GPS, network time
An interface to include info in EISCAT 3D metadata:
I
I
I
Used ionospheric parameters from ionosondes
Heater operation
Other complementary instrument operation if data available to EISCAT
How use object IDs
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
21 / 26
Outline
1
Current EISCAT data
2
Current metadata formats
3
Requirements on EISCAT 3D data format
4
Requirements on EISCAT 3D metadata
5
E-infrastructure projects
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
22 / 26
Initiatives in COOPEUS and ENVRI FP7
Roadmap (Anders Tjulin)
Common scheduling of all ISRs
Data formats
Data models
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
23 / 26
NEIC project
John White, CERN 50%
Specify computing requirements for sites
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
24 / 26
EGI-CC
EISCAT 3D support project
Data portal
I
I
Pilot for current EISCAT data
Develop EISCAT 3D data access
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
25 / 26
Summary
Current radar data formats, metadata and archives not suitable for
3D data
Work continues based on experience of present system and proposed
EISCAT 3D science
Running projects
I
I
I
EGI-CC
NEIC
ENVRIplus
C-F Enell
EISCAT Metadata: current status and future requirements
E3D user
for meeting
3D data 2015
26 / 26