Log In
Home
Topic RSS
Offline
Dear fellow arithmophiles,
we are in a search for the best possible way to include standard lamp correction to our Brewer data. Please feel free to share your way of using the standard lamp measurements to correct your total ozone measurements. If you do it by e-mail please use "reply all" not to cover the same method multiple times. I will set up this conversation to the forums at eubrewnet.org on Monday. I will also move all the discussion from e-mail thread there.
Please share your method in clear human language (preferably english) but any additional information in forms of code and/or mathematical entities is very welcome.
Feel free to share this message to anyone who might be interested and for some reason don't seem to be on this mailing list and/or in Working Group 2.
best regards and a nice weekend
Tomi
P.S. When you think of the best possible solution from those suggested:
I think the timespan of any solution using average should be such that it is a realistic time for the response of the instrument to change. I am not sure if the change should be allowed to be step or smooth. But for example let us not use only the last value because there is some noise in SL-test and that noise would be just moved to the measurements of Ozone in my mind. Also the average of too long (but how long is that) might miss some characteristics of response change in the instrument.
Offline
Dear Tomi,
my few and messy ideas:
1. I take for granted that we already have all necessary information to
discriminate between changes in the Brewer vs changes in the lamp only
(e.g., lamp replacements). I suspect that only the user of each Brewer
knows that (and maybe logged it in his logbook). Any algorithm should
easily let the user to introduce discontinuities.
The Brewer Processing Software (BPS) from EC, for example, expects a
discontinuity whenever a new configuration file comes up and, if I'm
right, scales the sl results to assure continuity between the series;
2. points which are too far from the bulk of the series should not be
used (or at least investigated) in my opinion, since they could arise
from instrumental issues that don't necessarily affect the ozone
retrieval in the same way as the sl. Can we get to an agreement on this,
too?
3. misalignments in filter wheels (e.g., FW#3) or other problems could
originate strange behaviours in sl series, such as broad or double
strips of points. How to cope with them? We just not use these series?
4. an extensive study (STSM ?) of several sl series from many Brewers
could be very useful to identify common issues and typical time scales
of variation (can high-frequency noise and the "real" signal be
completely separated?). If we haven't enough time to go into detail, we
should rely on the experience of our experts.. It would be good to have
an overview of the most common issues affecting sl measurements
(Volodya?). Also, I think that the RBCC-E campaigns can provide an
indication on how many Brewers are actually performing well and why
others aren't (Alberto?).
I personally use the BPS by Vitali. It's open source, so everybody can
look at its code for further information.
Cheers,
Henri
Offline
Hello
This is what we currently use:
1- SL calculation, we use the daily median to avoid outliers without any smooth. So the correction is done in daily intervals.
2- If during the day no SL measurement are done we use the day before or last recorded value
3- SL reference value are from config file (O3brewer), it comes from calibration report.
(as suggested by Volodya, if this not available a mean of 10 days after ICF data can be used as BDMS)
4- Sl changes of less than +/- 5 units are not used for the correction, is our estimation of the noise but can be calculated as the 1.5 std of the daily mean.
5- We record and store the correction applied (SL_ref - SL_obs)
Regards
Alberto
Offline
Hi Tomi,
We use the last SL test results for the corrections as well as the calibration references between 2 intercomparison dates.
The formulae we use are:
etco3 = b1cal(ical1) + (r6(jtest) - r6cal(ical1)) + (r6cal(ical1) - r6cal(ical2) - b1cal(ical1) + b1cal(ical2)) * dt / tt
and then:
o3 = (ms9 - etco3) / (a1 * mu)
were:
b1cal(ical1): etco3 reference at calibration date ical1
b1cal(ical2): etco3 reference at calibration date ical2
r6cal(ical1): SL R6 ratio reference at calibration date ical1
r6cal(ical2): SL R6 ratio reference at calibration date ical2
r6(jtest): Last SL R6 ratio at date jtest
dt: Time diference (days) between de last SL test and the calibration date ical1
tt: Time diference (days) between the calibration dates ical1 and ical2 (ical2 > ical1).
o3, ms9, a1 and mu have the usual meanings.
Regards,
Tino.
Offline
Hi Tomi,
Our method is close to Alberto's and Diamantino's. We correct using a daily mean SL ratio where the reference is either the last calibration SL ratio (from Volodya's report), or for reprocessing after a calibration, we use an interpolated reference between two calibrations. We take account of any known steps. The reference values and steps are stored in a text file that is read by a lower level routine so that whenever a new value is requested the most recent references are used.
Yours,
Andy
Offline
Hi Tomi
I asked Hugo to send me some information on how we apply the standard lamp correction at RMIB.
This is the answer I got:
" RMI has developed the following procedure to take into account the results of the SL tests:
- First a visualisation tool is used to see the evolution of the R6 (R5) readings.
It shows (monthly) means with the standard deviation. If there is a gradual change
then at a certain point in time a monthly mean is used to interpolate the ETC linearly
from the previous point according to the change in SL reading. This is done with an
off line program that reprocesses all the data. The distances in time between these
points are to be chosen in such a way that the linear aproximation is adequate.
If a sudden jump is detected then the corresponding new ETC is also applied as a jump.
- Before applying such changes the ensemble of information has to be checked (i.e. comparison
with co-located instrument, instrument in the vicinity, satellite data) in order to be sure that it is a
real change in the instrument. This has as a consequence that the corrections with respect to the
SL tests can only be done a posteriori."
Best regards
Veerle
Offline
Hello all,
we have already seen several replies with good practices for SL
corrections. Here is my 2 cents.
1. Fitting a quadratic polynomial (vs. time) to a set of SL ratios may
help coping with trends and curvatures in the changes.
2. Regardless of what type of fit is used, the difference between the
fitted and un-fitted data is a good metric for how well the fit
describes the changes, especially if those are fast and/or non-linear.
This can also reveal steps in the record.
3. Routinely calculating correlation between R6 and temperature may help
preventing running with non-optimal temperature coefficients. This can
be done together with fitting against time to separate the two effects.
4. While not strictly in the theme of this topic, we need to make sure
enough SL tests are done every day to be representative of the state of
the instrument at different temperatures throughout the day.
5. Whatever function is used, comparing the extrapolated data into today
with the actual data from today is an important measure of quality of
the model.
Cheers,
Volodya
Offline
Dear Tomi,
standard lamp correction on the Brewer MKIV 097 in Poprad Ganovce has been
performed by following procedures:
1. SL tests have been run at least 3 times per day
2. Interdiurnal SL corrections of total ozone have been done using software
of Martin Stanek (O3Brewer v. 5.0 - SL test, O3 correction and
recalculation). There is apparent correlation between the instrument
temperature (an also ambient temperature) and the SL correction (R5,R6
ratios).
3. The SL corrections after the instrument calibration have been done in
accordance with recommendations set in the instrument calibration report. If
the ozone data recalculation is recommended, the correction is done using
linear interpolation in accordance with the SL history.
4. Correction of the ozone has not been applied on data measured before
sudden changes in the instrument like the SL exchange or by other serious
technical problems (e.g.exchange of some instruments part).
Anna and Oliver
Offline
Hello all,
thank you very much on the input for my survey and sorry for so late response.
All the replies I got will be on the Eubrewnet.org forum with this message. (please inform me if you don't want to have your response in the forum)
Here is a short overview on the replies and my thoughts on those.
We have to decide how to manage two different things the measurements and the reference value. First thing about the measurements is that they are kind of noisy. I look at the standard lamp history of Brewer 037 and the variability seems to be around 10 to 15 units, so some kind of smoothing/mean value could be good as representative of the measurements.
The used/suggested methods to manage the measured R6 values were:
-daily median
-daily mean
-10 days running mean with weighting (O3Brewer)
-21 days running mean with weighting (found it on some WOUDC related website by Googling :D)
-fitting a polynomial to the measurements
The time window of any of these methods should be investigated?
Also there is a problem of outliers every now and then. When I calculated the running mean for a figure that will be attached I took out all the values that were more than 15 units off the median value for the time window (10 or 21 days).
On the other hand we should also manage the reference value. If all the changes are changes in the instrument and not in the lamp the reference value can (and should) be kept the same as also the ETC. This way the SL-corrected ETC will be ok. On the other hand if we see in the next calibration that the changes in the R6 value are not the same than the change in the ETC, we should figure out what to use as the reference. Of course if there is a step or a jump a new reference value should be introduced. How about slow change?
Many of the methods in your answers had an interpolation of the reference value. Should this be used for each instrument?
Attached are some figures with sl time series of Brewer 037. The firs figure shows the whole time series. There is some strange behaviour in the beginning and the stability is reached in the early 2000's 
There is also an example of these different smoothing methods for the year 2012.
Feel free to comment on the figures or my thoughts.
best regards and Merry Christmas
Tomi
figures: https://www.dropbox.com/sh/at4etrl3efk2crm/AABI9UJUuBXaE1P9Txs3EphVa?dl=0
Offline
Some helpful views from Andew:
Hi Tomi,
Looking at your five options for managing the R6 values, the one that makes the most sense to me is a daily median. I think it is very hard to manage the steps and jumps that you see in the time series with any other option with a longer time scale. For example, how would you decide on a criterion for a step and when not to apply the time averaging? Besides do you not want the R6 that is most representative of the instrument behaviour on the day in question -- that you only get by using a daily mean of some sort. Also I think that the median has the benefit of rejecting outliers nicely, whereas the mean is influenced by these.
Yours,
Andy
Offline
Hello Andrew,
only thing I am afraid of is that the R6 does not really follow the
changes in the ETC (meaning the changes in the response) but if you for
example have five sl-test during a day they are on average quite close
to the truth. If the +-10 units is not due to changes in the instruments
response but due to lamp spectrum fluctuation the we should most
definately use something that has smoother features.
How to detect steps is then another story and how to determine if its a
change in the response or in the lamp is then maybe even harder. I think
at least it calls for a human eye to decide when the reference value
should be changed. And this should be done after the next calibration
because if the lamp reference has changed but not the ETC then we
should apply new calibration constants from the point of change onwards.
best regards Tomi
Offline
Hi Tomi,
Fair point. I suspect that this comes down to separating the variability of the lamp and the instrument. The most simple way of treating this is to assume that all the variation is due to an instrument, and that the reference lamp is stable -- as one would hope in the first instance. To go beyond this I guess you need some characterisation of lamp stability via an independent instrument (either in the lab or by a co-located instrument). Perhaps this would give some idea of typical responses of the lamp that you could then use to make objective decisions about what are likely instrument response changes and which are due to the lamp.
As to the time interval, I'm still of the opinion that you'd want something sort that represents the behaviour of the instrument close to the measurement you're interested in (having separated out the lamp, as above)
Yours,
Andy
Most Users Ever Online: 22
Currently Online:
2 Guest(s)
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Member Stats:
Guest Posters: 0
Members: 87
Moderators: 0
Admins: 2
Forum Stats:
Groups: 1
Forums: 5
Topics: 1
Posts: 1
Newest Members:
ttech, sidalamine, test3, test2, test, LucasI, TaylorP, PinheiroO, ThompsL, AgostoyAdministrators: brewermaster: 1, Webmaster: 0
