Eubrewnet Wiki

User Tools

Site Tools

Trace: • ozoneproduct

Sidebar

Eubrewnet Wiki

Eubrewnet Documentation

  1. Eubrewnet
  2. Eubrewnet Community
  3. Configuration Upload Instructions

Content

  1. Eubrewnet Configuration
  2. Ozone Access Functions
  3. UV Access Functions
  4. AOD Access Functions
  5. Calibration Functions
  6. Data visualization
  7. Error Budget
  8. Code examples
  9. Brewer Python Library
  10. Ozone Product Process
  11. Ozone Products Description
  12. Brewer diagnostic tool
  13. Eubrewnet Report
  14. User Manual

Developers

  1. Wiki content
  2. Eubrewnet Products
  3. Generation of Products
  4. Subversion synchronization (campaigns)
  5. Automatic generation of weekly reports
  6. Eubrewnet Management
  7. Eubrewnet and Iberonesia Backup System
  8. DenyHosts at rbcce.aemet.es
  9. Administrator Manual
  10. Developer Manual
codes:ozoneproduct

This is an old revision of the document!

Table of Contents

Ozone Level Definition

Level 0

Are the O3 and SO2 comming from the Direct Sun measures from the Brewers. Some considerations:

  • Standard Algorithm (as related in the BREWER MKIV SPECTROPHOTOMETER OPERATOR'S MANUAL (OM-BA-C231 REV B, August 15, 1999 [Appendix G])
  • Observations are not processed, are taken directly from B files.
  • Values for O3 and SO2 come from the Direct Sun measures directly from the parsed B files
  • Constants and values used in the process come from the B file inst section (stored in the Brewer)
    • O3 and SO2 temperature coefficients, Neutral Density Filters…
    • Fixed Values: Interval Scaling, Rayleigh Coefficients…

Level 1

Are the O3 and SO2 recalculations with the standard algorithm from the Direct Sun measures and applying a verified by an operator set of constants. Some considerations:

  • The calulations use Brewer Python Module which implement the Standard Algorithm (the BREWER MKIV SPECTROPHOTOMETER OPERATOR'S MANUAL (OM-BA-C231 REV B, August 15, 1999 [Appendix G]).
  • Values for O3 and SO2 come from the calculation of the Standard Algorithmn using Raw Counts from the Direct Sun measures and the Temperature from the Direct Sun Summaries.
  • with the exception of location of the measurement (Latitude, Longitude and Pressure) the configuration constants and values used in the process come from the Configurations uploaded and validated to the system by the Operators.
  • Latitude, Longitude and Pressure are taken from the B inst section of the B file.

The process

  1. Get the DS and DS summaries measures of the day (coming from B file). If there are not available data, stop process.
  2. Get the last available configuration with date equal or earlier. If there is not a available configuration, stop process.
  3. Calulate the O3 and SO2 using the Brewer Python Module.
  4. Group the individual measures in groups of fives and calculate standard deviation.
  5. Store them into database.

Database

ozone_product_1_0 Table
Field Description Unit Reference
brewerid Brewer identification number db.brewer_t
gmt UT time of the measure in ISO 8601 format GMT ISO 8601
configid Configuration identification Numer db.configs_t
n_sum Index of the daily summary
date_index Continuous date index (0.0 = 0000-00-00T00:00:00Z) based in Matlab datenum days db.b_ds_t.date.toordinal + 366 + db.b_ds_t.mmmm_gmt / SECONDS_IN_A_DAY
sza Solar zenith angle using time function Degrees Brewer Python Module (Solar Zenith Angle)
airmass Calculated airmass using time function Brewer Python Module (Brewer Airmass)
temperature Instrument temperature Degrees db.b_ds_summary_t.temp
filt Applied neutral density filter db.b_ds_t.nd_filter_position / 64
o3 Calculated Ozone value with Standard algorithm + attenuation filter correction Ozone Level 1 Dobson Units Brewer Python Module (o3_so2)
std_o3 Standard deviation of the group of measures Dobson Units Brewer Operator Manual (Section 9.2.6)
so2 Calculated so2 value with Standard algorithm + attenuation filter correction Dobson Units Ozone Level 1
std_so2 Standard deviation of the group of measures Dobson Units Brewer Operator Manual (Section 9.2.6)
latitude Latitude of the Brewer Location Degrees db.b_t, db.configs_t
longitude Longitude of the Brewer Location Degrees db.b_t, db.configs_t
press Medium Pressure of the Brewer Location Milibars db.b_t, db.configs_t
ms8 First double ratio Brewer Python Module (o3_so2)
ms9 Second double ratio Brewer Python Module (o3_so2)
f1 Corrected Operational-Wavelength counts per second for slit 1 (includes only instrumental corrections, does not include the Rayleigh correction) Brewer Operator Manual (Appendix G)
f2 Corrected Operational-Wavelength counts per second for slit 2 (includes only instrumental corrections, does not include the Rayleigh correction) Brewer Operator Manual (Appendix G)
f3 Corrected Operational-Wavelength counts per second for slit 3 (includes only instrumental corrections, does not include the Rayleigh correction) Brewer Operator Manual (Appendix G)
f4 Corrected Operational-Wavelength counts per second for slit 4 (includes only instrumental corrections, does not include the Rayleigh correction) Brewer Operator Manual (Appendix G)
f5 Corrected Operational-Wavelength counts per second for slit 5 (includes only instrumental corrections, does not include the Rayleigh correction) Brewer Operator Manual (Appendix G)
configdate UT Date of the used configuration in ISO 8601 format GMT db.b_t, db.configs_t
configtype Type of the used configuration: B header(1), ICF (2), Config (3) db.b_t, db.icf_t db.configs_t
process_date Date and time when the product was processed GMT ISO 8601

Level 1.5 REALTIME OBSERVATION

They are level 1.0 observations with filter and corrections Level 1.0 O3 and SO2 observations

The list of config values and where to find them in the upload form can be found in Eubrewnet Configuration.

Filters

  1. Filtered by ozone standard deviation.
    • o3ds and o3ds_maxstd values have to be set in config
    • If not, a default value for std of 2.5 is used
  2. Filtered by ozone slant column / air mass
    • mu_ds and mu_ds_value values have to be set in config, and the model of the Brewer is Mk II or IV and the stray light correction has not been defined
    • If not, a default value of 3.5 is used
  3. The measures has valid hg (step change lees than 2) before and after (see Brewer Python Module (Check HG).
  4. Filtered by Minimun Values: lesser Ozone values than Maximun are discarded
    • Min value is set to 100
  5. Filtered by Maximun Values: greater Ozone values than Maximun are discarded
    • Max value is set to 500
  6. Filtered by the Exclusion List: measurements taken at a date and time included in the Exclusion List are discarded.

Corrections

  1. Standard lamp correction
    • sl_o3_cor has to be set in config for calculate Standard Lamp Correction (Default)
    • If not, Standard Lamp correction is not applied
  2. Filter correction: ETC filter dependent correction
    • etc_cor_fX value has to be provided in config (X filter)
    • If not or 0.0 value, ETC filter dependent correction is not applied
  3. Stray Light correction
    • stray_light_a and stry_light_b values has to be set in config
    • If not provided or both of them are 0.0, straylight correction is not applied

Two Flags have been added to the O3 and SO2 for filter a correction representation:

Flag Applied Value
Filter Flag Ozone standard deviation 1
Filter Flag Max airmass 2
Filter Flag Valid HG 4
Filter Flag Min Value 8
Filter Flag Max Value 16
Filter Flag Exclusion List 32
Correction Flag Standard Lamp 1
Correction Flag ETC filter dependent 2
Correction Flag Straylight 4

Flag Process

In case of Filter flag, the process of setting it works as related below:

  1. \begin{displaymath}Flag = 0 \end{displaymath}

  2. \begin{displaymath} \text{If } \sigma(Measures) > \text{Standard Deviation Value} \Rightarrow Flag = Flag + 1 \end{displaymath}

  3. \begin{displaymath} \text{If } \max(\text{Measure Airmasses}) > \text{Max Airmass} \Rightarrow Flag = Flag + 2 \end{displaymath}

  4. \begin{displaymath} \text{If } \text{Measure HG Test Failed} \Rightarrow Flag = Flag + 4 \end{displaymath}

  5. \begin{displaymath} \text{If } Ozone < \text{Ozone Min Value} \Rightarrow Flag = Flag + 8 \end{displaymath}

  6. \begin{displaymath} \text{If } Ozone > \text{Ozone Max Value} \Rightarrow Flag = Flag + 16 \end{displaymath}

  7. \begin{displaymath} \text{If } measurement's\; date\; and\; time \in \text{Exclusion List} \Rightarrow Flag = Flag + 32 \end{displaymath}

  8. The Group of measures is discarded if at least one of the flags is not 0

In case of Correction flag, the process of setting it works as related below:

  1. \begin{displaymath}Flag = 0 \end{displaymath}

  2. \begin{displaymath} \text{If SL Correction enabled } Flag = Flag + 1 \end{displaymath}

  3. \begin{displaymath} (\text{If ETC filter correction (config)} \ne 0) \And (\text{ETC filter Correction} \ne 0.0) \Rightarrow Flag = Flag + 2 \end{displaymath}

  4. \begin{displaymath} (\text{If Straylight Parameters} \ne 0) \And (\text{Straylight Correction} \ne 0.0) \Rightarrow Flag = Flag + 4 \end{displaymath}

The Process

This function computes O3 and SO2 amount from Brewer measurements:

\begin{displaymath} O_3 = O_3_0 + \Delta_{SL} -\Delta_{Filter}-\Delta_{StrayLight} \end{displaymath}

\begin{displaymath} O_3_0 = \frac{MS9 - ETC_0}{\mu \alpha } \end{displaymath}

\begin{displaymath} \Delta_{SL}=\frac{R6_{ref}- R6}{\mu \alpha } \end{displaymath}

\begin{displaymath} \Delta_{Filter}=\frac{ ECT_{N}}{\mu \alpha }, \end{displaymath}

\begin{displaymath} \Delta_{StrayLight_{N}}=\frac{ A*(\mu O_3_{N-1})^B}{\mu \alpha }, \end{displaymath}

  1. The $O_3_0$ is the Level 1 ozone
  2. The SL correction use the R6 reference value and the measure SL is smoothing using a Triangular moving average filter (see Brewer Python Module (Triangular Moving Average))
  3. The ETC filter correction is applied using the ETC correction function, the correction $ETC_N$ only affects to the measurements performed with filter #N
  4. The Stray Light ETC filter correction as depend of he ozone is an iterative process, in most cases only one iteration is needed (OSC<1600 DU) and is enough with two iterations for all the cases (up to OSC 2000DU). On the database two iterations are defined by default.
    • The configuration parameters A y B assume that osc ($\mu *O_3$) are in DU/1000.
    • We assume that the A parameter is negative in the configuration.

\begin{displaymath} \Delta_{StrayLight}=\frac{A*(\mu *〖(O_3_0〗+A*\frac{(\mu *O_3_0 )^B}{\alpha \mu}))^B}{\alpha \mu} $ \end{displaymath}

Database

Ozone Level 1.5

ozone_product_1_5 Table
Field Description Unit Reference
brewerid Brewer identification number db.brewer_t
gmt UT time of the measure in ISO 8601 format GMT ISO 8601
configid Configuration identification Numer db.configs_t
n_sum Index of the daily summary
date_index Continuous date index (1.0 = 0001-01-01T00:00:00Z) based in python date.toordinal days db.b_ds_t.date.toordinal + db.b_ds_t.mmmm_gmt / SECONDS_IN_A_DAY
sza Solar zenith angle using time function Degrees Brewer Python Module (Solar Zenith Angle)
airmass Calculated airmass using time function Brewer Python Module (Brewer Airmass)
temperature Instrument temperature Degrees db.b_ds_summary_t.temp
filt Applied neutral density filter db.b_ds_t.nd_filter_position / 64
o3 Calculated Ozone value with Standard algorithm + attenuation filter correction + config corrections Ozone Level 1.5 Dobson Units Brewer Python Module (o3_so2_sl)
std_o3 Standard deviation of the group of measures Dobson Units Brewer Operator Manual (Section 9.2.6)
so2 Calculated so2 value with Standard algorithm + attenuation filter correction Ozone Level 1.5 Dobson Units Brewer Python Module (o3_so2_sl)
std_so2 Standard deviation of the group of measures Dobson Units Brewer Operator Manual (Section 9.2.6)
latitude Latitude of the Brewer Location Degrees db.b_t, db.configs_t
longitude Longitude of the Brewer Location Degrees db.b_t, db.configs_t
press Medium Pressure of the Brewer Location Milibars db.b_t, db.configs_t
o3_0 Calculated Ozone value with Standard algorithm + attenuation filter correction Ozone Level 1 Dobson Units Brewer Python Module (o3_so2)
so2_0 Calculated so2 value with Standard algorithm + attenuation filter correction Ozone Level 1 Dobson Units Brewer Python Module (o3_so2)
r6 Standard Lamp Double Ratio 2 Brewer Python Module (Triangular Moving Average)
r6_ref Ozone value for Standard Lamp of Reference (Double Ratio 2) db.configs_t.r6_ref
r5 Standard Lamp Double Ratio 1 Brewer Python Module (Triangular Moving Average)
r5_ref Ozone value for Standard Lamp of Reference (Double Ratio 1) db.configs_t.r5_ref
filter_flag APPLIED FLAGS: HG Filter, Max Airmass, O3 std (1 means True) Level 1.5 Realtime Observation
correction_flag APPLIED CORRECTIONS: Straylight, ETC Correction, Standard Lamp correction(1 means True) Level 1.5 Realtime Observation
configdate UT Date of the used configuration in ISO 8601 format GMT db.b_t, db.configs_t
configtype Type of the used configuration: B header(1), ICF (2), Config (3) db.b_t, db.icf_t db.configs_t
process_date Date and time when the product was processed GMT

ISO 8601|

Level 2.0

Level 2.0 are 1.5 observations validated with a posterior calibration. The brewer instrument use the measurements of the internal halogen lamp to track the Ozone calibration, but is usually difficult to know if the changes recorded by the SL test are due to changes on the lamp or real changes on the instrumental response. Only with a subsequent calibration/comparison we can validate the SL correction.

The Level 2 data are produced when the operator activates the verification flag and validates (assign to level 2) the observations data between the current calibration and the previous verified one.

WOUDC extCSV files

EUBREWNET can generate files with monthly data in the extCSV format requested by the WOUDC (see https://guide.woudc.org/en/)

To generate the WOUDC files, use the URL http://www.eubrewnet.org/eubrewnet/woudc/csv with the following arguments:

* brewerid: Brewer serial number.

* date: month to process, in the format YYYY-MM

* level: EUBREWNET product level to be included in the file, either 1.5 or 2.0

For example, to get Brewer #157's L1.5 data for January 2019, one would use the URL

http://www.eubrewnet.org/eubrewnet/woudc/csv?brewerid=157&date=2019-01&level=1.5

The following arguments are optional and probably not useful for most users:

* Instead of brewerid, you can use one of gaw_id, woudc_id, or stationid to request the data for a station instead of a Brewer. stationid corresponds to the internal EUBREWNET ID of the station. Note that the Brewer and station products might be different if more than one Brewer operates at a station.

* version: this is EUBREWNET's processing algorithm version. If not specified, version 1.0 is used (note this is the only version available to the public as of April 2020)

* file_version: this is used in the Data Generation Version field of the extCSV file (see https://guide.woudc.org/en/#32-the-woudc-extcsv-data-file-metadata). If not specified, it defaults to the EUBREWNET's processing algorithm version, which itself defaults to 1.0 (see above)

codes/ozoneproduct.1586345620.txt.gz · Last modified: 2020/04/08 11:33 by jlsolano

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
CC Attribution-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki