Data standards description
The standards to be followed by the RPW data products are presented in the Metadata Definition for Solar Orbiter Science Data document issued by ESA. It includes the file naming convention, data versioning, data processing levels definition, data formats, metadata, etc.
Some standards are however reported below for convenience.
Data standards for RPW
Definition of processing levels
Level | Data type | Format and Metadata content |
---|---|---|
L0 | "Raw" data, unpacked and decompressed data | RPW L0 data are saved daily files in the HDF5 format. Metadata reflect the information that was available in the TM/TC packets generated/executed on-board the given day. |
L1 | "Engineering" data, uncalibrated | RPW L1 science data are saved in CDF format files. |
L2 | "Calibrated" data, science quality | RPW L2 science data are saved in CDF format files. |
L3 | Higher-level data | RPW L3 science data are saved in CDF format files. RPW L3 image data are saved in PNG format files. |
CAL | Calibration data | RPW calibration data are saved in CDF format files. |
ANC | Ancillary data | RPW team does not produce any ancillary data. Ancillary data publicly released by ESA for Solar Orbiter are SPICE kernels and CDF format files. |
Additionnaly, RPW team generates also CDF daily files storing the instrument subsytems housekeeping. The files are labelled using an internal level "HK" and are not released publicly.
File naming convention
Solar Orbiter data files are named according to the following convention:
source_level_descriptor_datetime_version_freefield.extension
Where the different fields that make up the filename are separated by underscores and are defined as follows:
Field | Description | Mandatory / Optional |
---|---|---|
source | Identifies the data as from Solar Orbiter. Must be ’solo’. | Mandatory |
level | Processing level. Content will either be L0, L1, L2, L3, ... according to the Solar Orbiter data processing level standard | Mandatory |
descriptor | The instrument (mandatory), and if appropriate the sensor/detector and/or data product, each separated by a hyphen. Including the instrument is mandatory. Detector and data product information can be included (or not) as required, e.g., metis, swa-eas-pad or epd-flux are all valid descriptors. However, the descriptor must uniquely identify the type of data in the file. The descriptor should only use lowercase letters or numbers and use no special characters. Level 3 data derived from multiple instruments have ’multi’ as the first part of descriptor. The meaning of the descriptor for ancillary data is slightly different, i.e., |
Mandatory |
datetime | The timestamp of the data in the file, formatted according to datetime standards for Solar Orbiter, corresponding to TIME_MIN in CDF keyword as the time at which the observation was made by the instrument/spacecraft. If CDF TIME_MAX keyword is applicable, it should be appended to the TIME_MIN, with a hyphen, to this datetime field. | Mandatory |
version | The version of the file, formatted as capital V + 2-character incremental number padded with 0, e.g., V01 | Mandatory |
freefield | An optional field of which the format is free, but decimal points or underscores cannot be used. | Optional |
extension | File extension. Must be ".cdf" for CDF format files and ".png" for PNG format files. | Mandatory |
Data versioning
The version of a RPW data product links to multiple types of versions linked to e.g., the software producing the product, the version of parameter configuration, the calibration data used to calibrate it, updates in metadata format, etc. The versions of the different ’actors’ that touched the data are described in the metadata. The filename however will contain a version number that is incremented whenever one or more of the individual versions got updated.
This data product version is formatted as ’V’ + and incremental number padded with ’0’ to 2 characters, as recommended by the ISTP Metadata Guidelines.
Data quality
CDF files shall include the QUALITY_FLAG and QUALITY_BITMASK variables to inform on data quality. They are both time dependent parameters and can apply to one or several science parameters.
QUALITY_FLAG shall be a CDF_UINT1 flag providing a human readable high-level information about the quality with the following values:
Quality | Meaning |
---|---|
0 | Bad data |
1 | Known problems, use at your own risk |
2 | Survey data, possibly not publication-quality |
3 | Good for publication, subject to PI approval |
4 | Excellent data which has received special treatment |
QUALITY_BITMASK shall be a CDF_UINT2 flag providing a computer readable quality information.
Table gives the bit meaning for RPW data products:
Bit order | Meaning |
---|---|
Discover more about data products here.