Eubrewnet Wiki

User Tools

Site Tools

Trace: • ozoneproduct
codes:ozoneproduct

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
codes:ozoneproduct [2020/09/11 06:31] – [NDACC AMES files] jlsolanocodes:ozoneproduct [2025/10/07 15:59] (current) – [The Process] ajberjon
Line 66: Line 66:
      * //o3ds// and //o3ds_maxstd// values have to be set in [[devel:eubrewnetconfiguration|config]]      * //o3ds// and //o3ds_maxstd// values have to be set in [[devel:eubrewnetconfiguration|config]]
      * If not, a default value for std of 2.5 is used      * If not, a default value for std of 2.5 is used
-   - Filtered by ozone slant column / air mass  +     * For other measurements (ZS, FZ, FM, ...) you can set specific thresholds in //Data and Metadata// → //Data Processing// → //Rejection Criteria//. A std of 4 is used by default for ZS measurements. 
-      * //mu_ds// and //mu_ds_value// values have to be set in [[devel:eubrewnetconfiguration|config]], and the model of the Brewer is Mk II or IV and the stray light correction has not been defined +   - Filtered by ozone slant column / air mass 
-      * If not, a default value of 3.5 is used+     * For DS measurements, //mu_ds// and //mu_ds_value// values have to be set in [[devel:eubrewnetconfiguration|config]] 
 +       - If the limits are defined, they are used. 
 +       - If not, the following default values are used: 
 +         * Single (no stray correction): 3.5 
 +         * Single (with stray correction): 6 
 +         * Double: 6 
 +     * For other ozone measurements in which the Stray Light correction is not defined (ZS, FZ, FM, ...), 3.5 is the default value for Single Brewers, and 6 for Doubles. You can also set a specific value in //Data and Metadata// → //Data Processing// → //Rejection Criteria//
    - The measures has valid hg (step change lees than 2) before and after (see [[devel:brewerpythonmodule#check_hg|Brewer Python Module (Check HG)]].    - The measures has valid hg (step change lees than 2) before and after (see [[devel:brewerpythonmodule#check_hg|Brewer Python Module (Check HG)]].
    - Filtered by Minimun Values: lesser Ozone values than Maximun are discarded    - Filtered by Minimun Values: lesser Ozone values than Maximun are discarded
Line 84: Line 90:
           * If not or 0.0 value, ETC filter dependent correction is not applied           * If not or 0.0 value, ETC filter dependent correction is not applied
    - Stray Light correction    - Stray Light correction
-          * //stray_light_a// and //stry_light_b// values has to be set in [[devel:eubrewnetconfiguration|config]] +          * //stray_light_a// and //stry_light_b// values has to be set in [[devel:eubrewnetconfiguration|config]] as described in [[https://doi.org/10.5194/acp-18-10347-2018|Rimer et al. (2018)]] 
-          * If not provided or both of them are 0.0, straylight correction is not applied+          * Alternatively, the  method based on PHYCS ([[https://doi.org/10.5194/amt-16-4785-2023|Savastiouk et all., 2023]]) may be employed, using the //alpha// and //beta// parameters. 
 +          * If not provided or parameters are 0.0, straylight correction is not applied
  
  
 Two Flags have been added to the O<sub>3</sub> and SO<sub>2</sub> for filter a correction representation: Two Flags have been added to the O<sub>3</sub> and SO<sub>2</sub> for filter a correction representation:
  
-^ Flag ^ Applied ^ Value ^ +^ Flag type ^ Flag name ^ Description ^ Value ^ 
-| Filter Flag | Ozone standard deviation | 1 | +| Data filter | FLAG_OZONE_STD_FILT | Ozone standard deviation | 1 | 
-| Filter Flag | Max airmass | 2 | +| Data filter | FLAG_AIRMASS_FILT | Max airmass | 2 | 
-| Filter Flag | Valid HG | 4 | +| Data filter | FLAG_HG_FILT | Valid HG | 4 | 
-| Filter Flag | Min Value | 8 | +| Data filter | FLAG_MIN_FILT | Min Ozone Value | 8 | 
-| Filter Flag | Max Value | 16 | +| Data filter | FLAG_MAX_FILT | Max Ozone Value | 16 | 
-| Filter Flag | Exclusion List | 32 | +| Data filter |  | Exclusion List | 32 | 
-| Correction Flag | Standard Lamp | 1 | +| Correction Flag |  | Standard Lamp | 1 | 
-| Correction Flag | ETC filter dependent | 2 | +| Correction Flag |  | ETC filter dependent | 2 | 
-| Correction Flag | Straylight | 4 |+| Correction Flag |  | Straylight | 4 |
  
 === Flag Process === === Flag Process ===
Line 146: Line 153:
  
   - The $O_3_0$ is the [[codes:productsdescription#ozone_level_1|Level 1 ozone]]   - The $O_3_0$ is the [[codes:productsdescription#ozone_level_1|Level 1 ozone]]
-  - The SL correction use the R6 reference value and the measure SL is smoothing  using a Triangular moving average filter (see [[devel:brewerpythonmodule#sl_tma|Brewer Python Module (Triangular Moving Average)]])+  - The SL correction uses the R6 reference value and the R6 from SL measurements smoothed using a [[devel:brewerpythonmodule#sl_tma|Triangular Moving Average]]. The correction is only applied if the difference between the two R6 values is more than 5 units.
   - The ETC filter correction is applied using the ETC correction function, the correction $ETC_N$ only affects to the measurements performed with filter #N   - The ETC filter correction is applied using the ETC correction function, the correction $ETC_N$ only affects to the measurements performed with filter #N
   - The Stray Light ETC filter correction as depend of he ozone is an iterative process, in most cases only one iteration is needed (OSC<1600 DU) and is enough with two iterations for all the cases (up to OSC 2000DU). On the database two iterations are defined by default.   - The Stray Light ETC filter correction as depend of he ozone is an iterative process, in most cases only one iteration is needed (OSC<1600 DU) and is enough with two iterations for all the cases (up to OSC 2000DU). On the database two iterations are defined by default.
Line 194: Line 201:
  
 The Level 2 data are produced when the operator activates the verification flag and validates (assign to level 2) the observations data between the current calibration and the previous verified one. The Level 2 data are produced when the operator activates the verification flag and validates (assign to level 2) the observations data between the current calibration and the previous verified one.
 +
 +----
  
 ===== WOUDC extCSV files ===== ===== WOUDC extCSV files =====
Line 199: Line 208:
 EUBREWNET can generate files with monthly data in the extCSV format requested by the WOUDC (see https://guide.woudc.org/en/) EUBREWNET can generate files with monthly data in the extCSV format requested by the WOUDC (see https://guide.woudc.org/en/)
  
-To generate the WOUDC files, use the URL ''http://www.eubrewnet.org/eubrewnet/woudc/csv'' with the following arguments:+To generate the WOUDC files, use the URL ''http://eubrewnet.aemet.es/eubrewnet/woudc/csv'' with the following arguments:
  
   * ''brewerid'': Brewer serial number.   * ''brewerid'': Brewer serial number.
  
-  * ''date'': month to process, in the format ''YYYY-MM'' +  * ''date'': month to process, in the format ''YYYY-MM'', ''last'' for the last month according to today's date.
- +
-  * ''level'': EUBREWNET product level to be included in the file, either 1.5 or 2.0+
  
 For example, use For example, use
      
-http://www.eubrewnet.org/eubrewnet/woudc/csv?brewerid=157&date=2019-01&level=1.5+http://eubrewnet.aemet.es/eubrewnet/woudc/csv?brewerid=157&date=2019-01 
 + 
 +to get Brewer #157's highest level data (1.5 near real time, 1.6 consolidated NRT, or 2.0 final) for January 2019.
  
-to get Brewer #157's L1.5 data for January 2019.+Note that EUBREWNET's 1.5, 1.6, and 2.0 data levels correspond to WOUDC's level 1.0 (see https://guide.woudc.org/en/#321-introduction). Files generated by EUBREWNET and submitted to the WOUDC will hence be stored at https://woudc.org/archive/Archive-NewFormat/TotalOzone_1.0_1/ (**Note the WOUDC level in EUBREWNET's files was changed from 2.0 to 1.0 in August 2024**)
  
 The following arguments are optional: The following arguments are optional:
Line 217: Line 226:
   * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.   * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.
  
-  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of April 2020)+  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version ''1.0'' is used (note this is the only version available to the public as of April 2020)
  
-  * ''file_version'': this is used in the Data Generation Version field of the extCSV file (see https://guide.woudc.org/en/#32-the-woudc-extcsv-data-file-metadata). If not specified, it defaults to  EUBREWNET's processing algorithm version, which itself defaults to ''1.0'' (see above)+  * ''file_version'': this is used in the Data Generation Version field of the extCSV file (see https://guide.woudc.org/en/#32-the-woudc-extcsv-data-file-metadata), and is written as ''major_version.minor_version''. If not specified, the file version is automatically determined depending on whether a file for the same date is already available at the WOUDC:    
 +    *  if the file exists, the ''minor_version'' will be increased by ''1'' (e.g., if the existing file is version ''3.2'', the new file will be ''3.3'') 
 +    * if no file exists, ''file_version'' defaults to EUBREWNET's processing algorithm version, which itself defaults to ''1.0'' (see above) 
  
  
Line 226: Line 237:
 It is also possible to generate HDF files with monthly data in the GEOMS format (see https://evdc.esa.int/documents/1/geoms-1.0.pdf) accepted by the NDACC, AVDC, and other data centers. It is also possible to generate HDF files with monthly data in the GEOMS format (see https://evdc.esa.int/documents/1/geoms-1.0.pdf) accepted by the NDACC, AVDC, and other data centers.
  
-To generate the GEOMS files, use the URL ''http://www.eubrewnet.org/eubrewnet/geoms/hdf'' with the following arguments:+To generate the GEOMS files, use the URL ''http://eubrewnet.aemet.es/eubrewnet/geoms_uncert/hdf'' with the following arguments:
  
   * ''brewerid'': Brewer serial number.   * ''brewerid'': Brewer serial number.
Line 236: Line 247:
 For example, use For example, use
      
-http://www.eubrewnet.org/eubrewnet/geoms/hdf?brewerid=157&date=2019-02&level=1.5+http://eubrewnet.aemet.es/eubrewnet/geoms_uncert/hdf?brewerid=157&date=2019-02&level=1.5
  
 to get Brewer #157's L1.5 data for February 2019. to get Brewer #157's L1.5 data for February 2019.
Line 249: Line 260:
  
   * ''file_access'': this is used to fill the ''FILE_ACCESS'' global attribute (see https://evdc.esa.int/documents/1/geoms-1.0.pdf, page 20, section 4.3.3). If more than one data center is specified, please separate their acronyms using only commas, e.g. ''AVDC,NDACC'' (**do not** use semicolons! ''AVDC;NDACC'' will not be interpreted correctly by the server!). If not specified, it defaults to ''NDACC''   * ''file_access'': this is used to fill the ''FILE_ACCESS'' global attribute (see https://evdc.esa.int/documents/1/geoms-1.0.pdf, page 20, section 4.3.3). If more than one data center is specified, please separate their acronyms using only commas, e.g. ''AVDC,NDACC'' (**do not** use semicolons! ''AVDC;NDACC'' will not be interpreted correctly by the server!). If not specified, it defaults to ''NDACC''
 +
 +  * ''individual'': write all individual measurements instead of daily means
 +
 +You can check the GEOMS-HDF files using the GEOMS QA checker available at https://geoms-tool.nilu.no/geoms_file_format_checker
  
 ===== NDACC AMES files ===== ===== NDACC AMES files =====
Line 254: Line 269:
 EUBREWNET can also generate AMES files with monthly data in the format specified by NDACC (see http://ndaccdemo.org/data/formats) EUBREWNET can also generate AMES files with monthly data in the format specified by NDACC (see http://ndaccdemo.org/data/formats)
  
-To generate the AMES files, use the URL ''http://www.eubrewnet.org/eubrewnet/ndacc/ames'' with the following arguments:+To generate the AMES files, use the URL ''http://eubrewnet.aemet.es/eubrewnet/ndacc/ames'' with the following arguments:
  
   * ''brewerid'': Brewer serial number.   * ''brewerid'': Brewer serial number.
Line 264: Line 279:
 For example, use For example, use
      
-http://www.eubrewnet.org/eubrewnet/ndacc/ames?brewerid=157&date=2019-02&level=1.5+http://eubrewnet.aemet.es/eubrewnet/ndacc/ames?brewerid=157&date=2019-02&level=1.5
  
 to get Brewer #157's L1.5 data for February 2019. to get Brewer #157's L1.5 data for February 2019.
Line 279: Line 294:
  
   * ''individual'': write all individual measurements instead of daily means   * ''individual'': write all individual measurements instead of daily means
 +
codes/ozoneproduct.1599805884.txt.gz · Last modified: 2020/09/11 06:31 by jlsolano