Differences

This shows you the differences between two versions of the page.

--- codes:ozoneproduct [2020/04/08 11:28]
jlsolano [Level 2.0]
+++ codes:ozoneproduct [2022/10/05 09:14] (current)
@@ Line 66: / Line 66: @@
      * //o3ds// and //o3ds_maxstd// values have to be set in [[devel:eubrewnetconfiguration|config]]
      * If not, a default value for std of 2.5 is used
-   - Filtered by ozone slant column / air mass
+     * For other measurements (ZS, FZ, FM, ...) you can set specific thresholds in //Data and Metadata// → //Data Processing// → //Rejection Criteria//. A std of 4 is used by default for ZS measurements.
-      * //mu_ds// and //mu_ds_value// values have to be set in [[devel:eubrewnetconfiguration|config]], and the model of the Brewer is Mk II or IV and the stray light correction has not been defined
+   - Filtered by ozone slant column / air mass
-      * If not, a default value of 3.5 is used
+     * For DS measurements, //mu_ds// and //mu_ds_value// values have to be set in [[devel:eubrewnetconfiguration|config]]
+       - If the limits are defined, they are used.
+       - If not, the following default values are used:
+         * Single (no stray correction): 3.5
+         * Single (with stray correction): 6
+         * Double: 6
+     * For other ozone measurements in which the Stray Light correction is not defined (ZS, FZ, FM, ...), 3.5 is the default value for Single Brewers, and 6 for Doubles. You can also set a specific value in //Data and Metadata// → //Data Processing// → //Rejection Criteria//
    - The measures has valid hg (step change lees than 2) before and after (see [[devel:brewerpythonmodule#check_hg|Brewer Python Module (Check HG)]].
    - Filtered by Minimun Values: lesser Ozone values than Maximun are discarded
@@ Line 90: / Line 96: @@
 Two Flags have been added to the O<sub>3</sub> and SO<sub>2</sub> for filter a correction representation:
-^ Flag ^ Applied ^ Value ^
+^ Flag type ^ Flag name ^ Description ^ Value ^
-| Filter Flag | Ozone standard deviation | 1 |
+| Data filter | FLAG_OZONE_STD_FILT | Ozone standard deviation | 1 |
-| Filter Flag | Max airmass | 2 |
+| Data filter | FLAG_AIRMASS_FILT | Max airmass | 2 |
-| Filter Flag | Valid HG | 4 |
+| Data filter | FLAG_HG_FILT | Valid HG | 4 |
-| Filter Flag | Min Value | 8 |
+| Data filter | FLAG_MIN_FILT | Min Ozone Value | 8 |
-| Filter Flag | Max Value | 16 |
+| Data filter | FLAG_MAX_FILT | Max Ozone Value | 16 |
-| Filter Flag | Exclusion List | 32 |
+| Data filter |  | Exclusion List | 32 |
-| Correction Flag | Standard Lamp | 1 |
+| Correction Flag |  | Standard Lamp | 1 |
-| Correction Flag | ETC filter dependent | 2 |
+| Correction Flag |  | ETC filter dependent | 2 |
-| Correction Flag | Straylight | 4 |
+| Correction Flag |  | Straylight | 4 |
 === Flag Process ===
@@ Line 195: / Line 201: @@
 The Level 2 data are produced when the operator activates the verification flag and validates (assign to level 2) the observations data between the current calibration and the previous verified one.
-===== WOUDC files =====
+----
-EUBREWNET can generate files with monthly data in the extCSV requested by the WOUDC, so that Brewer operators can later upload these files to the WOUDC FTP. See https://guide.woudc.org/en/ for more information on the WOUDC submission process.
+===== WOUDC extCSV files =====
-To generate the WOUDC files, use the URL ''http://www.eubrewnet.org/eubrewnet/woudc/csv'' with the following arguments:
+EUBREWNET can generate files with monthly data in the extCSV format requested by the WOUDC (see https://guide.woudc.org/en/)
-* ''brewerid'': Brewer serial number.
+To generate the WOUDC files, use the URL ''http://eubrewnet.aemet.es/eubrewnet/woudc/csv'' with the following arguments:
-* ''date'': month to process, in the format ''YYYY-MM''
+  * ''brewerid'': Brewer serial number.
-* ''level'': EUBREWNET product level to be included in the file, either 1.5 or 2.0
+  * ''date'': month to process, in the format ''YYYY-MM'', ''last'' for the last month according to today's date.
-For example, to get Brewer #157's L1.5 data for January 2019 in WOUDC format, one should use the URL
+For example, use
-http://www.eubrewnet.org/eubrewnet/woudc/csv?brewerid=157&date=2019-01&level=1.5
+http://eubrewnet.aemet.es/eubrewnet/woudc/csv?brewerid=157&date=2019-01
-The following arguments are optional and probably not useful for most users:
+to get Brewer #157's highest level data (1.5 near real time, 1.6 consolidated NRT, or 2.0 final) for January 2019.
-* Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.
+Note that EUBREWNET's 1.5, 1.6, and 2.0 data levels correspond to WOUDC's level 2.0 (see https://guide.woudc.org/en/#321-introduction). Files generated by EUBREWNET and submitted to the WOUDC will hence be stored at https://woudc.org/archive/Archive-NewFormat/TotalOzone_2.0_1/
-* ''version'': this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of April 2020)
+The following arguments are optional:
+  * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.
+  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version ''1.0'' is used (note this is the only version available to the public as of April 2020)
+  * ''file_version'': this is used in the Data Generation Version field of the extCSV file (see https://guide.woudc.org/en/#32-the-woudc-extcsv-data-file-metadata), and is written as ''major_version.minor_version''. If not specified, the file version is automatically determined depending on whether a file for the same date is already available at the WOUDC:
+    *  if the file exists, the ''minor_version'' will be increased by ''1'' (e.g., if the existing file is version ''3.2'', the new file will be ''3.3'')
+    * if no file exists, ''file_version'' defaults to EUBREWNET's processing algorithm version, which itself defaults to ''1.0'' (see above)
+===== GEOMS HDF files =====
+It is also possible to generate HDF files with monthly data in the GEOMS format (see https://evdc.esa.int/documents/1/geoms-1.0.pdf) accepted by the NDACC, AVDC, and other data centers.
+To generate the GEOMS files, use the URL ''http://eubrewnet.aemet.es/eubrewnet/geoms/hdf'' with the following arguments:
+  * ''brewerid'': Brewer serial number.
+  * ''date'': month to process, in the format ''YYYY-MM''
+  * ''level'': EUBREWNET product level to be included in the file, either 1.5 or 2.0
+For example, use
+http://eubrewnet.aemet.es/eubrewnet/geoms/hdf?brewerid=157&date=2019-02&level=1.5
+to get Brewer #157's L1.5 data for February 2019.
+The following arguments are optional:
+  * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.
+  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of April 2020)
+  * ''file_version'': this is used to fill the ''DATA_FILE_VERSION'' global attribute of the HDF file (see https://evdc.esa.int/documents/1/geoms-1.0.pdf, page 17, section 4.2.9). If not specified, it defaults to ''001''
+  * ''file_access'': this is used to fill the ''FILE_ACCESS'' global attribute (see https://evdc.esa.int/documents/1/geoms-1.0.pdf, page 20, section 4.3.3). If more than one data center is specified, please separate their acronyms using only commas, e.g. ''AVDC,NDACC'' (**do not** use semicolons! ''AVDC;NDACC'' will not be interpreted correctly by the server!). If not specified, it defaults to ''NDACC''
+  * ''individual'': write all individual measurements instead of daily means
+You can check the GEOMS-HDF files using the AVDC GEOMS QA checker available at https://avdc.gsfc.nasa.gov/index.php?site=1473794367
+===== NDACC AMES files =====
+EUBREWNET can also generate AMES files with monthly data in the format specified by NDACC (see http://ndaccdemo.org/data/formats)
+To generate the AMES files, use the URL ''http://eubrewnet.aemet.es/eubrewnet/ndacc/ames'' with the following arguments:
+  * ''brewerid'': Brewer serial number.
+  * ''date'': month to process, in the format ''YYYY-MM''
+  * ''level'': EUBREWNET product level to be included in the file, either 1.5 or 2.0
+For example, use
+http://eubrewnet.aemet.es/eubrewnet/ndacc/ames?brewerid=157&date=2019-02&level=1.5
+to get Brewer #157's L1.5 data for February 2019.
+As of September 2020, the error fields have a fixed value corresponding to 1% of the observation values.
+The following arguments are optional:
+  * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.
+  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of September 2020)
+  * ''qualityflag'': this is the 4-bytes value to write at the end of the first line in the NDACC file (see https://www.ndsc.ncep.noaa.gov/data/formats/header/#DFlag). Defaults to ''0001''
+  * ''individual'': write all individual measurements instead of daily means
-* ''file_version'': this is used in the Data Generation Version field of the extCSV file (see https://guide.woudc.org/en/#32-the-woudc-extcsv-data-file-metadata). If not specified, defaults to ''version'', which is EUBREWNET's processing algorithm version, which itself defaults to 1.0 (see above)

Eubrewnet Wiki

User Tools

Site Tools

Differences

Page Tools