Eubrewnet Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Calibration Functions =====

This page provides a brief overview of the functions avaliable in Eubrewnet to determine Brewer configuration parameters.

----
==== Langley ====
  * Function: ''langley''
  * Description: Determine the ETC and the ETC filter corrections applying the Langley-plot method to each half day.
  * Access function example: https://rbcce.aemet.es/eubrewnet/data/cal/langley?brewerid=185&date=2022-04-01&enddate=2022-04-10&format=text
  * Input: a complete list of input arguments is provided next, but we strongly recommed to use the web interface at https://rbcce.aemet.es/eubrewnet/caldata/langley to access this function
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
      - ''data'': input data used, available options are ''o3l15'' (default if the option is not passed in the url), ''o3l15v2'', ''o3l1'', ''sco3'', ''UNCL15'' for the O3L1.5, O3L1.5V2, O3L1, sun scan, and UNCL1.5 measurements, respectively. For the ''sco3'' option, the ''step'' option with the value of the micrometer step must be also passed in the url; if checking a mic step which does not correspond to the cal step, it's also recommended to pass the ozone absorption coefficient specific to that mic step using the ''urlconfig'' option, see below. Individual observations (not summaries) are used in all cases unless ''use_summaries'' is selected, see below. For the ''uncl15'' option, the fit is done using York's method (see [[https://github.com/duncombe/matlab/blob/master/york_curve_fit_0_01/york_fit.m|this MATLAB]] and [[https://gist.github.com/mikkopitkanen/ce9cd22645a9e93b6ca48ba32a3c85d0|this Python]] implementations)
      - ''so2'': calculate the SO2 ETC instead of the (default) O3 one
      - ''calc'': calculation to perform, available options are ''etc'' (Langley-plot with all the data, default option), ''filters'' (the Langley-plot fit is done using dummy variables for filters 3, 4, 5, and thus allows to determine the ETC filter correction of those filters with respect to the ETC obtained for the set measurements with filters 0, 1, and 2), and ''filters_etc'' (this option performs two fits: first, a ''filters'' run to determine the ETC filter corrections, and then an ''etc'' calculation using these ETC filter corrections). For ''etc'' runs, two different fits are performed: ''ms9'' vs ''airmass'' (brewer method) and ''ms9/airmass'' vs ''1/airmass'' (dobson method)
      - ''minairmass'': airmass minimum in ''etc'' calculations (defaults to 1.70 if the option is not passed in the url)
      - ''maxairmass'': airmass maximum in ''etc'' calculations (defaults to 3.75 if the option is not passed in the url)
      - ''minobs'': minimum number of observations (either individual observations or summaries) in the half-day to perform the Langley in ''etc'' calculations (default value: 25). Note by default all the ''data'' functions are configured to return individual observations, but see ''use_summaries'' below
      - ''filters_minairmass'': airmass minimum in ''filters'' calculations (defaults to 1 if the option is not passed in the url)
      - ''filters_maxairmass'': airmass maximum in ''filters'' calculations (defaults to 5 if the option is not passed in the url)
      - ''filters_minobs'': minimum number of observations in the half-day to perform the Langley in ''filters'' calculations (default value: 15)
      - ''maxstdo3'': don't perform the langley if the maximum of the standard deviation of the half-day ozone is above this value (default: 2.5)
      - ''minrsquared'': don't accept the langley results if the r<sup>2</sup> of the fits are below this value (default: 0.999)
      - ''residuals_filtering'': do each fit twice, using the residuals of the first to remove observations whose residual is larger than 1.5*std(residuals)
      - ''use_summaries'': if passed as an option, use summaries instead of individual observations for o3l15 and o3l15v2 data. Note the default values for ''minobs'' and ''filter_minobs'' (25 and 15, respectively) are provided as a guide for calculations with individual observations, the user must set suitable values (eg, 5 for ''minobs'') if ''use_summaries'' is passed.
      - ''urlconfig'': allows setting the configuration's parameters, overriding the values in Eubrewnet's database. For example, to set a value of 0.4 for the ozone absorption coefficient, add ''&urlconfig&o3o3rate=0.4'' to the url, and to set specific values for the filter corrections add ''&urlconfig&fc=1,2,3,4,5''
      - ''debug'': include debug output. As of 2023-06-29, this returns four additional vectorial fields, ''airmass'', ''ms9'', ''filt'', and ''solartime'', with the raw data used in the half-day langley fit.
  * Output:
      - ''date'': date and time of each half-day langley, morning langleys correspond to 00 time, and afternoon ones, to 12
      - ''etc'': calculated ETC
      - ''rmse'' or ''u_etc'': root mean squared error or uncertainty in the ETC as calculated in the York method (only for the ''uncl15'' input)
      - ''tau'': in the case of the determination of the O3 ETC, this corresponds to the ozone optical depth -- ie, the ozone multiplied by the ozone absorption coefficient. For the SO2 ETC, this is the sum of the SO2 and O3 optical depths, in the space of the SO2 ratios
      - ''rsquared'': r<sup>2</sup> of the langley fit
      - ''nobs'': number of observations used in the fit
      - ''stdo3'': standard deviation of the half-day ozone
      - ''meano3'': mean ozone of the half-day
      - ''filter_corr_3'',''filter_corr_4'',''filter_corr_5'': ETC filter corrections, either calculated in a ''filters'' run or supplied passing ''&urlconfig&fc=1,2,3,4,5'' in the url
      - ''info'': a brief description of the calculation performed

----
==== ETC filter corrections from ozone differences ====
    * Function: ''fcor_o3dif''
    * Description: this function returns the ETC filter corrections (with respect to the mean of filters 1 and 2) as calculated from the ozone differences of near-simultaneous measurements taken with different filters. Mathematically, for two measurements taken with different filters at almost the same airmass μ and with a ΔO3L1 difference in the (uncorrected) ozone, Δfiltercorr=ΔO3L1∗μ∗abscoeff∗10
    * Access function example: https://rbcce.aemet.es/eubrewnet/data/cal/fcor_o3dif?brewerid=185&format=text&minutes=15&airmass=0.03&date=2022-01-01&enddate=2022-02-01
    * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]. If not specified, the calculation will be carried out for all the date ranges between the configurations available in Eubrewnet
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]. If not specified, the current date is used.
      - ''format'': see [[codes:dbaccess#format|Format]]
      - ''maxminutes'': maximum difference in minutes between consecutive measurements, default value: 15
      - ''airmass'': maximum airmass between consecutive measurements, default value: 0.1
    * Output:
      - ''config_f1_cor'', ''config_f2_cor'', ''config_f3_cor'', ''config_f4_cor'' , ''config_f5_cor'': ETC filter corrections set in the configuration at Eubrewnet
      - ''calc_f1_cor'', ''calc_f2_cor'', ''calc_f3_cor'', ''calc_f4_cor'', ''calc_f5_cor'': median of all the ETC filter corrections calculated in the period, referred to the mean of filters 1 and 2
      - ''calc_f1_sd'', ''calc_f2_sd'', ''calc_f3_sd'', ''calc_f4_sd'', ''calc_f5_sd'': standard deviation of the ETC filter corrections in the period
      - ''calc_f01_nobs'', ''calc_f12_nobs'', ''calc_f23_nobs'', ''calc_f34_nobs'', ''calc_f45_nobs'': number of near simultaneous observations with the indicated filters


----

==== ETC filter corrections from ozone differences -- raw data ====
    * Function: ''fcor_o3dif_raw''
    * Description: this function returns the raw data used in the fcor_o3dif function. That is, it returns Δfiltercorr=ΔO3L1∗μ∗abscoeff∗10 for each pair of measurements, without referring the result to filters 1 and 2 nor calculating any median over a date range.
    * Access function example: https://rbcce.aemet.es/eubrewnet/data/cal/fcor_o3dif_raw?brewerid=185&format=text&minutes=15&airmass=0.03&date=2022-01-01&enddate=2022-02-01
    * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]. If not specified, the calculation will be carried out for all the date ranges between the configurations available in Eubrewnet
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]. If not specified, the current date is used.
      - ''format'': see [[codes:dbaccess#format|Format]]
      - ''maxminutes'': maximum difference in minutes between consecutive measurements, default value: 15
      - ''airmass'': maximum airmass between consecutive measurements, default value: 0.1
    * Output:
      - ''previous_gmt'': date and time of the first measurement in the pair
      - ''gmt'': date and time of the second measurement
      - ''previous_airmass'': airmass of the first measurement
      - ''airmass'': airmass of the second measurement
      - ''previous_filt'': filter position for the first measurement
      - ''filt'': filter position for the second measurement
      - ''previous_o3_l1'': level 1 ozone for the first measurement
      - ''o3_l1'': level 1 ozone for the second measurement
      - ''configdate'': date of the operative configuration
      - ''o3o3rate'': ozone absorption coefficient
      - ''fcor'': filter correction
      - ''fchange'': a single string with both filter positions 

----
==== ETC filter correction from FIOAVG data ====
    * Function: ''fcor_fioavg''
    * Description: this function returns the ETC filter corrections (with respect to the mean of filters 1 and 2) as calculated from the results of the Brewer filter tests with the standard lamp, as found in the FIOAVG file.
    * Access function example: https://rbcce.aemet.es/eubrewnet/data/cal/fcor_fioavg?brewerid=185&format=text&date=2022-01-01&enddate=today
    * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]. If not specified, the calculation will be carried out for all the date ranges between the configurations available in Eubrewnet
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]. If not specified, the current date is used.
      - ''format'': see [[codes:dbaccess#format|Format]]
    * Output:
      - ''config_f1_cor'', ''config_f2_cor'', ''config_f3_cor'', ''config_f4_cor'' , ''config_f5_cor'': ETC filter corrections set in the configuration at Eubrewnet
      - ''calc_f1_cor'', ''calc_f2_cor'', ''calc_f3_cor'', ''calc_f4_cor'', ''calc_f5_cor'': median of all the ETC filter corrections calculated in the period, referred to the mean of filters 1 and 2
      - ''calc_f1_sd'', ''calc_f2_sd'', ''calc_f3_sd'', ''calc_f4_sd'', ''calc_f5_sd'': standard deviation of the ETC filter corrections in the period
      - ''calc_nobs'': number of measurements in the FIOAVG file

----

==== Temperature Coefficients ====


  * Function: ''TempCoef''
  * Description: This function analyses the temperature sensitivity of SL measurements.
  * Example: https://eubrewnet.aemet.es/eubrewnet/data/process/TempCoef?brewerid=40&tc&tmin=5&tmax=40&alpha=3&date=2021-07-06&enddate=2021-07-16&format=text
  * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
      - ''tc'': if we pass the tc parameter with no value, the fit is done with the SL measurements corrected using the temperature coefficients from the operational setup on Eubrewnet. If we pass six values separated by commas, the fit is done with the corrected SL measurements using these values as temperature coefficients. If the tc parameter is omitted, no temperature correction is applied to the SL measurements.
      - ''tmin'': all data below tmin are removed.
      - ''tmax'': all data above tmax are removed.
      - ''alpha'': Outliers can be removed by giving the alpha parameter a value greater than zero. In this case, after performing the linear regression, all measures whose residuals are greater than the standard deviation of all residuals multiplied by the alpha factor will be removed, and the linear regression will be repeated. This process is repeated iteratively until all values are below this threshold.
      - ''step'': optional, if passed uses the data from the JL measurements at the requested mic step. JL measurements are corrected using the linear temperature coefficient and the filter attenuations corresponding to each wavelength, as available in the AOD configuration operative at the beginning of the requested date range (''date'')
      - ''time'': optional, if passed, the time evolution of the lamp intensity is considered in the analysis, including in the linear regression a degree 3 dependence on time.
      - ''urlconfig'': allows setting the configuration's parameters, overriding the values in Eubrewnet's database.
  * Output: the following fields are included in the output:
      - ''channel'': Brewer measurement channels in the ozone position plus R5 and R6 (the latter are not included in the JL output)
      - ''a'': intercept value for the linear regression.
      - ''sa'': standard error of the intercept of the linear regression.
      - ''b'': slope value for the linear regression.
      - ''sb'': standard error of the slope of the linear regression.
      - ''r2'': R-squared of the linear regression.
      - ''tmin'': Minimum temperature used in the linear regression.
      - ''tmax'': Maximum temperature used in the linear regression.
      - If the parameter ''time'' is passed, three more coefficients are returned (the time dependence considered is of degree 3) together with their standard errors: c,sc,d,sd,e,se. 

----

==== Sun scan ====
  * Function: ''SC''
  * Description: Returns SC measurements from B file. It adds the temperature of the last HG test made before measurement
  * Example: [[http://eubrewnet.aemet.es/eubrewnet/data/get/SC?brewerid=153&date=2016-11-16&format=text]]
  * Inputs:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
  * Output: SC measurements with extra field:
      - ''temp'': Temperature from the last made HG before each SC measurement

----

==== Sun scan with extra information ====
  * Function: ''SCO3''
  * Description: Same as SC above with additional information (O3, ms9, longitude, ...). A specific micrometer step can be also selected using ''&step=XXX''. Note by default the ozone will be calculated with the ozone absorption coefficient stored in the operative configuration, which corresponds to the cal step. An ozone absorption coefficient specific for the requested mic step can be passed using the  ''urlconfig'' method: add e.g. ''&urlconfig&o3o3rate=0.4'' to the url
  * Example: https://eubrewnet.aemet.es/eubrewnet/data/get/SCO3?brewerid=185&date=2022-03-07&enddate=2022-09-06&format=text
  * Inputs:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
  * Output: SC measurements with additional fields, including:
      - ''temp'': Temperature from the last made HG before each SC measurement
      - ''O3'': ozone
      - ''ms9'': ms9 double ratio

----

==== CI and CJ lamp tests ====
  * Function: ''CI'' or ''CJ'' 
  * Description: Return the data from the CI (forward scan) or CJ (forward and backward scans) lamp tests. For CJ, the two scans are shown as two measurements with close but different times.
  * Example: [[https://rbcce.aemet.es/eubrewnet/data/get/CJ?brewerid=258&date=2023-04-01&enddate=2023-05-01&format=text&fields=gmt,lamp,raw_counts*]]
  * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
      - ''lamp'': optional, allowed values are ''b1'' and ''b2'', used to select only data with the ''b1'' (Hg) or ''b2'' (Standard) lamp
  * Output: standard output plus other fields, including
      - ''gmt'': time of the first wavelength measured -- the shortest for the CI and CJ forward scan, and the longest for the CJ backward.
      - ''mmmm_l'': time in minutes of the measurement of wavelength ''l'', where ''l = 2865, 2870, ... 3630''
      - ''raw_counts_l'': raw counts of wavelength ''l''
      - ''counts_per_second_l'': counts/second for wavelength ''l''
      - ''lamp'': either ''b1'' (Hg) or ''b2'' (Standard) 
  * Debug output: if ''&debug'' is passed in the url, two extra fields are included in the output:
      - ''lamp_time'': time at which the lamp was switched on
      - ''lamp_co'': comment from the B file stating which lamp was switched on. This comment corresponds to the most recent in a search on all the comments containing the keywords ''b1'' or ''b2'' in the B file in the 20 minutes before the first measurement in the CI/CJ

----

==== CZ lamp tests ====
  * Function: ''CZ'' 
  * Description: Return the data from the CZ lamp tests. Because in this test the wavelengths are defined by the Brewer operator, Eubrewnet's output is different than in the case of the CI/CJ tests. In particular,  the ''l'' in fields such as ''mmmm_l'' is now just a numerical index from 0 to 149, and the wavelengths are shown in the new ''wavelengths_l'' fields
  * Example: [[https://rbcce.aemet.es/eubrewnet/data/get/CZ?brewerid=183&date=2023-02-11&format=text]]
  * Input: same as for the CI and CJ functions, see above
  * Output: the output is quite similar to that of the CI and CJ functions, the main difference being that the ''_l'' suffix is now a numerical index instead of a wavelength. There are also some new fields:
      - ''wavelength_l'': wavelength in nm corresponding to the l-th measurement
      - ''fwpos1'' and ''fwpos2'': filter wheel positions
      - ''startwv'' and ''finalwv'': initial and final wavelengths

----

==== NI responsivity tests ====
  * Function: ''NI'' 
  * Description: Return the data from the NI responsivity tests. 
  * Example: [[https://rbcce.aemet.es/eubrewnet/data/get/NI?brewerid=185&date=2022-08-31&format=text]]
  * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
  * Output: besides the usual ''brewerid'', ''gmt'',''lat'', ... output, some other fields returned include
      - ''mmmm'': minutes since 00:00
      - ''wavelength'': wavelength measured
      - ''fwpos1'' and ''fwpos2'': filter wheel positions
      - ''rawcountssml'': raw counts for the slit mask ''l'' position, where ''l=0,1,...,7''

----

==== HS and HL Mercury lamp tests ====
  * Functions: ''HS'' and ''HL''
  * Description: Return the data from the short (2950 – 2984 Å) and long (3331 – 3351 Å) wavelength-range Mercury lamp tests
  * Example: [[https://rbcce.aemet.es/eubrewnet/data/get/HL?brewerid=185&date=2022-08-31&format=text]]
  * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
  * Output: besides the usual ''brewerid'', ''gmt'',''lat'', ... output, some other fields returned include
      - ''mmmm'': minutes since 00:00
      - ''wavelength'': wavelength measured
      - ''fwpos1'' and ''fwpos2'': filter wheel positions
      - ''rawcounts'': raw counts
      - ''countspsecond'': counts per second

----

==== ERA5 comparison ====
  * Functions: ''compareERA5''
  * Description: Compares the hourly average 1.5 level ozone product for a given brewer with the [[https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels|Total ozone column product from the ERA5 reanalysis provided by ECMWF]].
  * Example: [[http://eubrewnet.aemet.es/eubrewnet/data/get/compareERA5?brewerid=157&date=2022-01-01&enddate=2022-01-31&format=text]]
  * Input:
      - ''brewerid'': see [[codes:dbaccess#common_inputs|Brewerid in common inputs]]
      - ''date'': see [[codes:dbaccess#common_inputs|Date in common inputs]]
      - ''enddate'': see [[codes:dbaccess#common_inputs|Enddate in common inputs]]
      - ''format'': see [[codes:dbaccess#format|Format]]
      - ''debug'': In the output, individual measurements are returned instead of hourly averages.
  * Output:
      - ''gmt'': UTC time.
      - ''o3_avg'': Mean hourly ozone L1.5 for brewerid.
      - ''o3_era5'' Total ozone column from ERA5 for the station at which the brewer was at ''date''.
      - ''diff'': difference between o3_avg and o3_era5 in Dobson units.
      - ''reldiff'': relative difference between o3_avg and o3_era5 in %
.
----
Eubrewnet Wiki

User Tools

Site Tools

Page Tools
