Eubrewnet Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Ozone Level Definition ======

===== Level 0 =====
Are the O<sub>3</sub> and SO<sub>2</sub> comming from the Direct Sun measures from the Brewers.
Some considerations:
  * Standard Algorithm (as related in the BREWER MKIV SPECTROPHOTOMETER OPERATOR'S MANUAL (OM-BA-C231 REV B, August 15, 1999 [Appendix G])
  * Observations are not processed, are taken directly from B files. 
  * Values for O<sub>3</sub> and SO<sub>2</sub> come from the Direct Sun measures directly from the parsed B files
  * Constants and values used in the process come from the B file inst section (stored in the Brewer)
     * O<sub>3</sub> and SO<sub>2</sub> temperature coefficients, Neutral Density Filters...
     * Fixed Values: Interval Scaling, Rayleigh Coefficients...

===== Level 1 =====
Are the O<sub>3</sub> and SO<sub>2</sub> recalculations with the standard algorithm from the Direct Sun measures and applying a verified by an operator set of constants.
Some considerations:
  * The calulations use  [[devel:brewerpythonmodule#o3_so2|Brewer Python Module]] which implement the Standard Algorithm (the BREWER MKIV SPECTROPHOTOMETER OPERATOR'S MANUAL (OM-BA-C231 REV B, August 15, 1999 [Appendix G]). 
  * Values for O<sub>3</sub> and SO<sub>2</sub> come from the calculation of the Standard Algorithmn using Raw Counts from the Direct Sun measures and the Temperature from the Direct Sun Summaries.
  * with the exception of location of the measurement (Latitude, Longitude and Pressure) the configuration constants and values used in the process come from the Configurations uploaded and validated to the system by the Operators.
  * Latitude, Longitude and Pressure are taken from the B inst section of the B file.
  
==== The process ====
  - Get the DS and DS summaries measures of the day (coming from B file). If there are not available data, stop process.
  - Get the last available configuration with date equal or earlier. If there is not a available configuration, stop process.
  - Calulate the O<sub>3</sub> and SO<sub>2</sub> using the [[devel:brewerpythonmodule#o3_so2|Brewer Python Module]].
  - Group the individual measures in groups of fives and calculate standard deviation.
  - Store them into database.

==== Database ====
^  ozone_product_1_0 Table  ^^^^
^ Field ^ Description ^ Unit ^ Reference ^
| brewerid | Brewer identification number | | db.brewer_t |
| gmt | UT time of the measure in ISO 8601 format | GMT |[[http://www.iso.org/iso/es/home/standards/iso8601.htm|ISO 8601]]|
| configid | Configuration identification Numer | | db.configs_t |
| n_sum | Index of the daily summary | | |
| date_index | Continuous date index (0.0 = 0000-00-00T00:00:00Z) based in Matlab datenum | days | db.b_ds_t.date.toordinal + 366 + db.b_ds_t.mmmm_gmt / SECONDS_IN_A_DAY |
| sza | Solar zenith angle using time function | Degrees | [[devel:brewerpythonmodule#solar_zenith_angle|  Brewer Python Module (Solar Zenith Angle)]] |
| airmass | Calculated airmass using time function | | [[devel:brewerpythonmodule#brewer_airmass| Brewer Python Module (Brewer Airmass)]] |
| temperature | Instrument temperature | Degrees | db.b_ds_summary_t.temp |
| filt | Applied neutral density filter | | db.b_ds_t.nd_filter_position / 64 |
| o3 | Calculated Ozone value with Standard algorithm + attenuation filter correction [[codes:ozoneproduct#level_1|Ozone Level 1]] | Dobson Units | [[devel:brewerpythonmodule#o3_so2|Brewer Python Module (o3_so2)]] |
| std_o3 | Standard deviation of the group of measures | Dobson Units | Brewer Operator Manual (Section 9.2.6) | 
| so2 | Calculated so2 value with Standard algorithm + attenuation filter correction | Dobson Units | [[codes:ozoneproduct#level_1|Ozone Level 1]]|
| std_so2 | Standard deviation of the group of measures | Dobson Units | Brewer Operator Manual (Section 9.2.6) |
| latitude | Latitude of the Brewer Location | Degrees | db.b_t, db.configs_t |
| longitude | Longitude of the Brewer Location | Degrees | db.b_t, db.configs_t |
| press | Medium Pressure of the Brewer Location | Milibars | db.b_t, db.configs_t |
| ms8 | First double ratio | | [[devel:brewerpythonmodule#o3_so2|Brewer Python Module (o3_so2)]] |
| ms9 | Second double ratio | | [[devel:brewerpythonmodule#o3_so2|Brewer Python Module (o3_so2)]] |
| f1 | Corrected Operational-Wavelength counts per second for slit 1 (includes only instrumental corrections, does not include the Rayleigh correction) | | Brewer Operator Manual (Appendix G) |
| f2 | Corrected Operational-Wavelength counts per second for slit 2 (includes only instrumental corrections, does not include the Rayleigh correction) | | Brewer Operator Manual (Appendix G) |
| f3 | Corrected Operational-Wavelength counts per second for slit 3 (includes only instrumental corrections, does not include the Rayleigh correction) | | Brewer Operator Manual (Appendix G) |
| f4 | Corrected Operational-Wavelength counts per second for slit 4 (includes only instrumental corrections, does not include the Rayleigh correction) | | Brewer Operator Manual (Appendix G) |
| f5 | Corrected Operational-Wavelength counts per second for slit 5 (includes only instrumental corrections, does not include the Rayleigh correction) | | Brewer Operator Manual (Appendix G) |
| configdate | UT Date of the used configuration in ISO 8601 format | GMT | db.b_t, db.configs_t |
| configtype | Type of the used configuration: B header(1), ICF (2), Config (3) | | db.b_t, db.icf_t db.configs_t |
| process_date | Date and time when the product was processed |GMT |[[http://www.iso.org/iso/es/home/standards/iso8601.htm|ISO 8601]]| 

===== Level 1.5  REALTIME OBSERVATION =====

They are level 1.0 observations with filter and corrections  Level 1.0  O<sub>3</sub> and SO<sub>2</sub> observations

The list of config values and where to find them in the upload form can be found in [[devel:eubrewnetconfiguration|Eubrewnet Configuration]].

=== Filters ===
   - Filtered by ozone standard deviation.
     * //o3ds// and //o3ds_maxstd// values have to be set in [[devel:eubrewnetconfiguration|config]]
     * If not, a default value for std of 2.5 is used
   - Filtered by ozone slant column / air mass 
      * //mu_ds// and //mu_ds_value// values have to be set in [[devel:eubrewnetconfiguration|config]], and the model of the Brewer is Mk II or IV and the stray light correction has not been defined
      * If not, a default value of 3.5 is used
   - The measures has valid hg (step change lees than 2) before and after (see [[devel:brewerpythonmodule#check_hg|Brewer Python Module (Check HG)]].
   - Filtered by Minimun Values: lesser Ozone values than Maximun are discarded
     * Min value is set to 100
   - Filtered by Maximun Values: greater Ozone values than Maximun are discarded
     * Max value is set to 500
   - Filtered by the Exclusion List: measurements taken at a date and time included in the  [[devel:eubrewnetconfiguration&#exclusion_list|Exclusion List]] are discarded.
      
=== Corrections ===      
   - Standard lamp correction
          * //sl_o3_cor// has to be set in [[devel:eubrewnetconfiguration|config]] for calculate Standard Lamp Correction (Default)
          * If not, Standard Lamp correction is not applied 
   - Filter correction: ETC filter dependent correction
          * //etc_cor_f<sub>X</sub>// value has to be provided in [[devel:eubrewnetconfiguration|config]] (X filter)
          * If not or 0.0 value, ETC filter dependent correction is not applied
   - Stray Light correction
          * //stray_light_a// and //stry_light_b// values has to be set in [[devel:eubrewnetconfiguration|config]]
          * If not provided or both of them are 0.0, straylight correction is not applied


Two Flags have been added to the O<sub>3</sub> and SO<sub>2</sub> for filter a correction representation:

^ Flag type ^ Flag name ^ Description ^ Value ^
| Data filter | FLAG_OZONE_STD_FILT | Ozone standard deviation | 1 |
| Data filter | FLAG_AIRMASS_FILT | Max airmass | 2 |
| Data filter | FLAG_HG_FILT | Valid HG | 4 |
| Data filter | FLAG_MIN_FILT | Min Ozone Value | 8 |
| Data filter | FLAG_MAX_FILT | Max Ozone Value | 16 |
| Data filter |  | Exclusion List | 32 |
| Correction Flag |  | Standard Lamp | 1 |
| Correction Flag |  | ETC filter dependent | 2 |
| Correction Flag |  | Straylight | 4 |

=== Flag Process ===
In case of Filter flag, the process of setting it works as related below:
  - \begin{displaymath}Flag = 0 \end{displaymath} 
  - \begin{displaymath} \text{If } \sigma(Measures) > \text{Standard Deviation Value} \Rightarrow Flag = Flag + 1 \end{displaymath}
  - \begin{displaymath} \text{If } \max(\text{Measure Airmasses}) > \text{Max Airmass} \Rightarrow Flag = Flag + 2 \end{displaymath}
  - \begin{displaymath} \text{If } \text{Measure HG Test Failed} \Rightarrow Flag = Flag + 4 \end{displaymath}
  - \begin{displaymath} \text{If } Ozone < \text{Ozone Min Value} \Rightarrow Flag = Flag + 8 \end{displaymath}
  - \begin{displaymath} \text{If } Ozone > \text{Ozone Max Value} \Rightarrow Flag = Flag + 16 \end{displaymath}
  - \begin{displaymath} \text{If } measurement's\; date\; and\; time \in \text{Exclusion List}  \Rightarrow Flag = Flag + 32 \end{displaymath}
  - The Group of measures is discarded if at least one of the flags is not 0

In case of Correction flag, the process of setting it works as related below:
  - \begin{displaymath}Flag = 0 \end{displaymath} 
  - \begin{displaymath} \text{If SL Correction enabled }  Flag = Flag + 1 \end{displaymath}
  - \begin{displaymath} (\text{If ETC filter correction (config)} \ne 0) \And (\text{ETC filter Correction} \ne 0.0) \Rightarrow Flag = Flag + 2 \end{displaymath}
  - \begin{displaymath} (\text{If Straylight Parameters} \ne 0) \And (\text{Straylight Correction} \ne 0.0) \Rightarrow Flag = Flag + 4 \end{displaymath}




==== The Process ====
This function computes O<sub>3</sub> and SO<sub>2</sub> amount from Brewer measurements:

\begin{displaymath} 
    O_3 = O_3_0 + \Delta_{SL} -\Delta_{Filter}-\Delta_{StrayLight}
\end{displaymath}

\begin{displaymath} 
     O_3_0 = \frac{MS9 - ETC_0}{\mu \alpha }
 \end{displaymath}   
    
 \begin{displaymath}     
    \Delta_{SL}=\frac{R6_{ref}- R6}{\mu \alpha }
 \end{displaymath}
    
\begin{displaymath}        
    \Delta_{Filter}=\frac{ ECT_{N}}{\mu \alpha },
\end{displaymath}   

\begin{displaymath}     
    \Delta_{StrayLight_{N}}=\frac{ A*(\mu O_3_{N-1})^B}{\mu \alpha },      
\end{displaymath}     


  - The $O_3_0$ is the [[codes:productsdescription#ozone_level_1|Level 1 ozone]]
  - The SL correction use the R6 reference value and the measure SL is smoothing  using a Triangular moving average filter (see [[devel:brewerpythonmodule#sl_tma|Brewer Python Module (Triangular Moving Average)]])
  - The ETC filter correction is applied using the ETC correction function, the correction $ETC_N$ only affects to the measurements performed with filter #N
  - The Stray Light ETC filter correction as depend of he ozone is an iterative process, in most cases only one iteration is needed (OSC<1600 DU) and is enough with two iterations for all the cases (up to OSC 2000DU). On the database two iterations are defined by default.
    * **The configuration parameters A y B assume that osc ($\mu *O_3$) are in DU/1000**.
    * **We assume that the A parameter is negative in the configuration**.

\begin{displaymath}
\Delta_{StrayLight}=\frac{A*(\mu *〖(O_3_0〗+A*\frac{(\mu *O_3_0 )^B}{\alpha \mu}))^B}{\alpha \mu} $
\end{displaymath}

==== Database ====
==== Ozone Level 1.5 ====
^  ozone_product_1_5 Table  ^^^^
^ Field ^ Description ^ Unit ^ Reference ^
| brewerid | Brewer identification number | | db.brewer_t |
| gmt | UT time of the measure in ISO 8601 format | GMT |[[http://www.iso.org/iso/es/home/standards/iso8601.htm|ISO 8601]]|
| configid | Configuration identification Numer | | db.configs_t |
| n_sum | Index of the daily summary | | |
| date_index | Continuous date index (1.0 = 0001-01-01T00:00:00Z) based in python date.toordinal | days | db.b_ds_t.date.toordinal + db.b_ds_t.mmmm_gmt / SECONDS_IN_A_DAY |
| sza | Solar zenith angle using time function | Degrees | [[devel:brewerpythonmodule#solar_zenith_angle|  Brewer Python Module (Solar Zenith Angle)]] |
| airmass | Calculated airmass using time function | | [[devel:brewerpythonmodule#brewer_airmass| Brewer Python Module (Brewer Airmass)]] |
| temperature | Instrument temperature | Degrees | db.b_ds_summary_t.temp |
| filt | Applied neutral density filter | | db.b_ds_t.nd_filter_position / 64 |
| o3 | Calculated Ozone value with Standard algorithm + attenuation filter correction + config corrections [[codes:ozoneproduct#level_15_realtime_observation|Ozone Level 1.5]] | Dobson Units | [[devel:brewerpythonmodule#o3_so2_sl|Brewer Python Module (o3_so2_sl)]] |
| std_o3 | Standard deviation of the group of measures | Dobson Units | Brewer Operator Manual (Section 9.2.6) | 
| so2 | Calculated so2 value with Standard algorithm + attenuation filter correction [[codes:ozoneproduct#level_15_realtime_observation|Ozone Level 1.5]] | Dobson Units | [[devel:brewerpythonmodule#o3_so2_sl|Brewer Python Module (o3_so2_sl)]] |
| std_so2 | Standard deviation of the group of measures | Dobson Units | Brewer Operator Manual (Section 9.2.6) |
| latitude | Latitude of the Brewer Location | Degrees | db.b_t, db.configs_t |
| longitude | Longitude of the Brewer Location | Degrees | db.b_t, db.configs_t |
| press | Medium Pressure of the Brewer Location | Milibars | db.b_t, db.configs_t |
| o3_0 | Calculated Ozone value with Standard algorithm + attenuation filter correction [[codes:ozoneproduct#level_1|Ozone Level 1]]| Dobson Units | [[devel:brewerpythonmodule#o3_so2|Brewer Python Module (o3_so2)]] |
| so2_0 | Calculated so2 value with Standard algorithm + attenuation filter correction [[codes:ozoneproduct#level_1|Ozone Level 1]]| Dobson Units | [[devel:brewerpythonmodule#o3_so2|Brewer Python Module (o3_so2)]] |
| r6 | Standard Lamp Double Ratio 2 | | [[devel:brewerpythonmodule#sl_tma|Brewer Python Module (Triangular Moving Average)]] |
| r6_ref | Ozone value for Standard Lamp of Reference (Double Ratio 2) | | db.configs_t.r6_ref |
| r5 | Standard Lamp Double Ratio 1 | | [[devel:brewerpythonmodule#sl_tma|Brewer Python Module (Triangular Moving Average)]] | 
| r5_ref | Ozone value for Standard Lamp of Reference (Double Ratio 1) | | db.configs_t.r5_ref |
| filter_flag | APPLIED FLAGS: HG Filter, Max Airmass, O3 std (1 means True) | | [[codes:ozoneproduct#level_15_realtime_observation|Level 1.5 Realtime Observation]]|
| correction_flag | APPLIED CORRECTIONS: Straylight, ETC Correction, Standard Lamp correction(1 means True) | | [[codes:ozoneproduct#level_15_realtime_observation|Level 1.5 Realtime Observation]] |
| configdate | UT Date of the used configuration in ISO 8601 format | GMT | db.b_t, db.configs_t |
| configtype | Type of the used configuration: B header(1), ICF (2), Config (3) | | db.b_t, db.icf_t db.configs_t |
| process_date | Date and time when the product was processed |GMT |
[[http://www.iso.org/iso/es/home/standards/iso8601.htm|ISO 8601]]| 

===== Level 2.0 =====

**Level 2.0** are 1.5 observations validated with a posterior calibration. The brewer instrument use the measurements of the internal halogen lamp to track the Ozone calibration, but is usually difficult to know if the changes recorded by the SL test are due to changes on the lamp or real changes on the instrumental response. Only with a subsequent calibration/comparison we can validate the SL correction.

The Level 2 data are produced when the operator activates the verification flag and validates (assign to level 2) the observations data between the current calibration and the previous verified one.

----

===== WOUDC extCSV files =====

EUBREWNET can generate files with monthly data in the extCSV format requested by the WOUDC (see https://guide.woudc.org/en/)

To generate the WOUDC files, use the URL ''http://www.eubrewnet.org/eubrewnet/woudc/csv'' with the following arguments:

  * ''brewerid'': Brewer serial number.

  * ''date'': month to process, in the format ''YYYY-MM'', ''last'' for the last month according to today's date.

For example, use
  
http://www.eubrewnet.org/eubrewnet/woudc/csv?brewerid=157&date=2019-01

to get Brewer #157's highest level data (L1.5, L1.6, or L2.0) for January 2019.

Note that EUBREWNET's 1.5, 1.6, and 2.0 data levels correspond to WOUDC's level 2.0, see https://guide.woudc.org/en/#321-introduction. Files generated by EUBREWNET and submitted to the WOUDC will hence be stored at https://woudc.org/archive/Archive-NewFormat/TotalOzone_2.0_1/

The following arguments are optional:

  * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.

  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version ''1.0'' is used (note this is the only version available to the public as of April 2020)

  * ''file_version'': this is used in the Data Generation Version field of the extCSV file (see https://guide.woudc.org/en/#32-the-woudc-extcsv-data-file-metadata), and is written as ''major_version.minor_version''. If not specified, the file version is automatically determined depending on whether a file for the same date is already available at the WOUDC: 
  ** if the file exists, the ''minor_version'' will be increased by ''1'' (e.g., if the existing file is version ''3.2'', the new file will be ''3.3'')
  ** if no file exists, ''file_version'' defaults to EUBREWNET's processing algorithm version, which itself defaults to ''1.0'' (see above) 


===== GEOMS HDF files =====

It is also possible to generate HDF files with monthly data in the GEOMS format (see https://evdc.esa.int/documents/1/geoms-1.0.pdf) accepted by the NDACC, AVDC, and other data centers.

To generate the GEOMS files, use the URL ''http://www.eubrewnet.org/eubrewnet/geoms/hdf'' with the following arguments:

  * ''brewerid'': Brewer serial number.

  * ''date'': month to process, in the format ''YYYY-MM''

  * ''level'': EUBREWNET product level to be included in the file, either 1.5 or 2.0

For example, use
  
http://www.eubrewnet.org/eubrewnet/geoms/hdf?brewerid=157&date=2019-02&level=1.5

to get Brewer #157's L1.5 data for February 2019.

The following arguments are optional:

  * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.

  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of April 2020)

  * ''file_version'': this is used to fill the ''DATA_FILE_VERSION'' global attribute of the HDF file (see https://evdc.esa.int/documents/1/geoms-1.0.pdf, page 17, section 4.2.9). If not specified, it defaults to ''001''

  * ''file_access'': this is used to fill the ''FILE_ACCESS'' global attribute (see https://evdc.esa.int/documents/1/geoms-1.0.pdf, page 20, section 4.3.3). If more than one data center is specified, please separate their acronyms using only commas, e.g. ''AVDC,NDACC'' (**do not** use semicolons! ''AVDC;NDACC'' will not be interpreted correctly by the server!). If not specified, it defaults to ''NDACC''

You can check the GEOMS-HDF files using the AVDC GEOMS QA checker available at https://avdc.gsfc.nasa.gov/index.php?site=1473794367

===== NDACC AMES files =====

EUBREWNET can also generate AMES files with monthly data in the format specified by NDACC (see http://ndaccdemo.org/data/formats)

To generate the AMES files, use the URL ''http://www.eubrewnet.org/eubrewnet/ndacc/ames'' with the following arguments:

  * ''brewerid'': Brewer serial number.

  * ''date'': month to process, in the format ''YYYY-MM''

  * ''level'': EUBREWNET product level to be included in the file, either 1.5 or 2.0

For example, use
  
http://www.eubrewnet.org/eubrewnet/ndacc/ames?brewerid=157&date=2019-02&level=1.5

to get Brewer #157's L1.5 data for February 2019.

As of September 2020, the error fields have a fixed value corresponding to 1% of the observation values.

The following arguments are optional:

  * Instead of ''brewerid'', you can use one of ''gaw_id'', ''woudc_id'', or ''stationid'' to request the data for a station instead of a Brewer. ''stationid'' corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.

  * ''version'': this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of September 2020)

  * ''qualityflag'': this is the 4-bytes value to write at the end of the first line in the NDACC file (see https://www.ndsc.ncep.noaa.gov/data/formats/header/#DFlag). Defaults to ''0001''

  * ''individual'': write all individual measurements instead of daily means
Eubrewnet Wiki

User Tools

Site Tools

Page Tools
