Each Brewer spectrophotometer communicates with a local PC where the raw data are recorded. For uploading the raw data from the client PC to the Eubrewnet database a set of routines written in python has to be installed on the local PC (or any other computer which has access to the Brewer raw data). When these routines are executed the local (client) PC is connected to the Eubrewnet server. Then, the local brewer files and all relative information (file size, modification date etc.) are listed and compared to the corresponding information for the files that have been already stored in the server. Only the files which are either new or have changed, are transferred from the former to the later. This process is repeated in regular time intervals (prefferably ~15 mins). In sections 1.1 - 1.4, the software installation and application processes are analytically described. A fast installation guide is available in section 1.5.

Before the installation of the software the operator has to communicate with the administrator(eubrewnet@aemet.es) and ensure that his Brewer has been included in theEubrewnet Brewer list. Then, there are two possible methods to install the client software:

1^st  method(recommended): Install the python interpreter and download the python client

For Windows and Mac, Python 2.7 is recommended while for Linux, the Python Version should come bundled with the Linux distribution. Below is a list of the appropriate interpreters for Windows and Mac.

●       Python 2.7 Windows x86 MSI Installer (Windows binary)

●       Python 2.7 Windows X86-64 MSI Installer(Windows AMD64 / Intel 64 /X86-64 binary)

●       Python 2.7 Mac OS X 64-bit/32-bit x86-64/i386 Installer (for Mac OS X 10.5 and later, if needed)

●       Python 2.7 Mac OS X 32-bit i386/PPC Installer(for Mac OS X 10.3 and later, if needed)

●       Linux Version should come bundled with your Linux distribution.

You have to be sure that your python interpreter is callable from every path in your system (Open a console and try python, exit() for exit).

Then, client_python.zipshould be downloaded and uncompressed in the system.

2^nd  method(not fully tested): Download the exe client

In this case, the client_exe.zip should be downloaded and uncompressed in the client PC system. After unzipping the compressed folder, try: $>refresh.exe (in dist folder) in a console to test if client is working properly. No error message should appear.

The client configuration file (client.ini) is used by the refresh.py program. The later is the routine used to transfer the files from the client PC to the database. Depending on the installation method, these two files can be found either in client_python\source (1st installation method) or in client_exe\dist (2nd installation method). Client.ini is a plain ASCII INI file which consists of a global section and several server sections (one section for each server to which the files are sent).

In the following example, the content of a client.ini file is presented. The lines in bold should not be changed by the operator since they provide information relative to the database server. The information in the remaining (regular font) lines should be defined by the operator.

In the global section of the client.ini file the following parameters should be declared:

●       brewerid : brewer identifier (i.e. the brewer serial number).

●       input: the directory which contains the files produced by the Brewer.

●       servers: comma separated list of server aliases. The files are sent to each server in this list.

●       working_dir: the directory where the local refresh database is stored.

●       proxy: proxy url, if required. If not required, this property should be commented (add # at the beginning of the line).

A server section is identified by the server alias (i.e. the alias is the name between the brackets - iberonesia3 in the case of Example 1), and has the following properties:

●       url : server url.

●       db : a local database (a simple data file) is created the first time that client software is executed. This line defines the name of this file.

●       noproxy : if a proxy is specified in the global section, noproxy may be set to yes to use a direct connection (avoiding the proxy) to access the server.

●       serverport: the used port in the server (8081)

The lines that are not used should be commented (i.e. the proxy property if no proxy is used).  The easiest way to configure the client is to properly specify all options in the client.ini file and then execute refresh.py without arguments. However, if desired, some of the above properties can be overridden using command line parameters when executing refresh.py or alive.py. If for example, the client.ini file of Example 1 is used, a different Brewer id and a different input path can be defined in the command line:

  $>python refresh.py -b 086 -i c:\brewer086\data

Though, it is preferable to specify the Brewer id and the input path (as well as any other parameters) in the client.ini file instead of using the -b and -i commands.

As already described, most of the refresh.py options may be configured though command line arguments or client.ini configuration file. Any option specified in command line overrides the option in client.ini. All options are shown when the the command  “$ ./refresh.py -h” is used (see Example 2):

The mandatory options are:

●       working_dir :This option must be specified in client.ini. It defines the directory where the local refresh database is stored.

●       input: Defines the file which will be processed. If a directory is defined, then all files in the particular directory will be processed.

●       brewerid : Brewer identifier.

●       servers: Comma separated list of server aliases. The files are sent to each server in this list.

Open a browser and try access eubrewnet. If you cannot, then contact your system administrator and grant access to the internet. For debugging, it is recommended to execute the client software using the -v parameter (the more v, the more debugging information shown on the screen, with -vvv the maximum):

  $>python refresh.py -b XXX -vvv
      - XXX: Id of a brewer

If you want to save the errors and log, define log and error files:

  $>python refresh.py 1>client.log 2>client_error.log

The two files (client.log and client_error.log) should not contain error messages after the execution of the client software. In case of errors in the communication, the Eubrewnet administrator should be informed.

The Brewer data can be sent to Eubrewnet either directly from the operational Brewer PC or by using another computer which has access to the data.

From the Brewer computer:

Sending the Brewer data to Eubrewnet in regular time intervals can be achieved by adding a routine in your Brewer schedule. The routine is executed in desired time intervals and calls the client software.

In W2k W7 the brewer software waits until the execution of each command is finished so if the communication is slow or fails it takes too much time to send the data, and may block or delay the brewer software. This does not happen if DOSBOX is used. The old MSDOS/GWBASIC does not allow the sharing of the files. This is because the B file is always open, which blocks the access to the file and stops the recording of the brewer observations by the client software.

Thus, it is recommended to send the data in two steps:

1.      Inside the brewer software or a task scheduler run a routine to copy the bdata directory content in another place/computer like for example bdata1:

  set DEST_DIR=e:\brewer
  set ORIG_DIR=e:\bdata1

  net use x: /DELETE
  net use x: \\10.2.20.20\brewer   /PERSISTENT:NO

  robocopy  %ORIG_DIR%   %DEST_DIR%  /S /XO /V 

2.      Run the data sending software from this directory as scheduled task

$>python refresh.py   -b %%X  -s iberonesia -i %DEST_DIR%\brw#%%X\bdata%%X >client%%X.out 2»client_error%%X.log

1- Installthe python interpreter and then download and uncompress client script (client_python.zip) or client executable (client_exe.zip).

Copy the scripts to the brewer program directory

2- Check the configuration file (client.ini)

Are you bellow a proxy ?

no: then change noproxy in client.ini to YES !!!

yes: then edit the client.ini and indicate your proxy server and port, set the noproxy flag to NO !!

Then edit the necessary fields in the client.ini file, without changing the following fields:

CLIENT.INI

[global] servers=iberonesia3

proxy=proxy.xxx.xxx:port

[iberonesia3] url=http://rbcce.aemet.es/refresh

db=iberonesia3.db

serverport=8081

noproxy=no

3- execute the script

python refresh.py – b XXX -i c:\brewer_data -vvv 1>client.log 2>client_error.log

XXX is the name of your brewer and brewer_data is a directory where the brewer files are.

If it takes long, it is probably is working fine. Check the log files to see if all the data of your test directory have been sent.

4- Schedule the data sent: We use robocopy to make a copy of the bdata directory. Edit the following two files (EUBREWNET.BAT and CK.RTN) properly and add them to the brewer program directory. Then put CK on your brewer schedule to send data to eubrewnet

EUBREWNET.BAT

set DEST_DIR=e:\eubrewnet\

setORIG_DIR=e:\bdata157\bdata

set X=157

robocopy %ORIG_DIR% %DEST_DIR%\Brw#%%X\bdata%%X\ *.%%X /XO /V /MAXAGE:3

python refresh.py -b %%X -s iberonesia -i %DEST_DIR%\brw#%%X\bdata%%X >client%%X.out 2»client_error%%X.log

CK.RTN

10000 rem Call eubrewnet.bat

10010 data ck

20000 shell”eubrewnet

30000 return

65529 rem dummy

The raw data (Brewer spectral measurements) sent to the Eubrewnet database have to be processed in order to get the final products (currently Total Ozone Column (O₃) and sulfur dioxide (SO₂) and in the future, Aerosol Optical Depth (AOD), spectral UV irradiance, Nitrogen dioxide (NO₂), and ozone profiles). For this purpose, a number of instrumental and calibration constants have to be inserted in the data processing algorithm. The operator of each instrument is responsible for uploading and saving the configuration (i.e. the needed constants) for each product in the database. In addition to the configuration of each product, information regarding the station and the instruments should be also provided. A detailed list of all necessary configuration parameters and other useful information can be found in the following link:

http://rbcce.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration

Before uploading the configuration for the station and the products, the operators have to contact the system administrator (eubrewnet@aemet.es) and provide contact information and the list of Brewers for which they are responsible. Then, they have to register in the system. When the registration is complete they are able to login the website of Eubrewnet and upload the configuration. In the following there is a more detailed description of the configuration uploading process.

Registration and login

As shown in Figure 1, at the right of the Eubrewnet webpage the user can find the sign in button. There, he/she has two options: Register or Login. If Register is chosen, a form appears which should be filled by the database user. If Log in is chosen the user is asked for his username and password and then he can have full access to the database.

Figure 1: Sign in options.

When a user is logged in he/she can edit/fill the station information. The option Stations should be chosen from the User list (see Figure 2). Then, by clicking on the name of the station, a form containing the station information appears. There, the user can add information or edit the form and submit the changes. The information can be added either by hand or by uploading a document which contains the station configuration information.

Figure 2: Accessing the station configuration form

Figure 3: Example of the station configuration form.

As shown in Figure 4, the user can have access to the ozone configuration form by clicking on the Ozone Configuration option from the User list. In the ozone configuration form the user has the option to update the configuration for more products such as AOD and NO₂, which however are not operational yet.

Figure 4: Access the ozone configuration form

The process begins by selecting the Brewer for which the configuration constants will change. Then, as shown in Figure 5, the user has three options:

1. Add new configuration
2. Modify configuration
3. Download configuration

If the Download configuration option is chosen, the entire record of the configurations which have been either used in the past or are currently in use is downloaded in a single csv file.

Figure 5: Example of the ozone configuration initial page.

If the user wants to modify an existing configuration then he\she has to choose the configuration version he/she wants to modify and press the Modify Configuration button. The date since which an existing configuration is valid cannot be changed. In this case the configuration has to be resubmitted as new. For adding a new configuration the user has to define the date in the Date box (next to the Brewerid box) since which the new configuration is valid, and then press the Add new configuration button.

By pressing the Add new Configuration or Modify Configuration buttons, a new form appears. In this form, the user has the option to either upload files which contain the new configuration constants (Figure 5), or modify the currently used configuration parameters and save them as new (or modified) configuration. In the Configuration and Actions section (Figure 6), in addition to the choice of uploading files with the new configuration parameters, the user has the following options: (i)  to decide if the specific configuration is operative or inactive and (ii) to define if the generated product from the current configuration is level 2 or not. Level 2 is the level 1.5 product after it has been validated with a posterior calibration. In this section the user has also the options to either go back or submit the performed changes by clicking the corresponding buttons.   

Figure 6: Example of the “Configuration and Actions” section.

In the beginning of the Values to be submitted section, there are two lines of buttons. The buttons of the first line can be used to choose where the submitted configuration is applicable. The possible choices are:

●       O₃ and SO₂

●       AOD

●       Umkehr

●       Data and Metadata

●       NO₂

The AOD and Umkehr products are under development (at the time of writing this manual) and it is not possible to provide or edit any configuration constants. The configuration constants in the NO₂ product and most variables in the Data and Metadata form are inactive, although they can be updated. Thus, analytical description is provided only for the O₃ and SO₂ configuration form. For the contents of the Data and Metadata and NO₂ forms there is a brief description at the end of the section.

In the O₃ and SO₂ configuration form, the user can edit the Setup, Algorithm and Instrument constants by clicking on the corresponding button.

●       Setup

In this form there are three tables which should be filled. The first is for the setup of the DS measurements (Figure 7). In the first raw the operator should provide the needed constants for the retrieval of the total ozone column (O₃). These constants can be found in the used icf file (instrument configuration file). These are the extraterrestrial constant used for the determination of O₃ (ETC O3), the ozone to ozone ratio (O3 Ratio on O3) and the R6 reference value (R6 Reference). If the “SL test O3 correction and recalculation” box is checked then the change of the R6 ratio relative to its reference value is taken into account for the calculation of O₃. More specifically, the median of the R6 for each day is calculated and a 7 day (day of the calculations ± 3 days) triangular moving average is applied to calculate the R6 that will be used for the correction. If the difference between the reference and the calculated R6 exceeds 5 the difference is taken into account in the calculation of the O₃.

Figure 7: Example of the Table for the DS setup.

In the second line the operator should provide the needed constants for the retrieval of the total column of sulfur dioxide (SO₂). The constants that have to be determined are the same as those described for ozone: the extraterrestrial constant used for the determination of SO₂ (ETC SO2), the SO₂ to SO₂ ratio (SO2 Ratio on SO2) and the R5 reference value (R5 Reference). The ozone to SO₂ ratio (SO2 Ratio on O3) should be also provided. The constant regarding the calculation of the SO₂ can be also found in the icf file used for the period for which the specified configuration is valid.

In the third line the applied filters for the calculation of the level 1.5 and level 2 products are specified. Each single O₃ and SO₂ value is the mean of five consecutive measurements performed. If the operator chooses to use these filters, the O₃ and  SO₂ (direct and zenith sky) measurements are considered to be valid only when the mean air mass (MU DS,ZS) and the standard deviation of the set of the five measurements (Max STD) are below specific limits, which can be determined by the operator, and by default are 3.5 and 2.5 DU respectively.

In the fourth line the parameters used for stray light correction are provided (A and B). These parameters are available only if the instrument has been properly characterized. In the fifth line the dead time of the photomultiplier tube (Deadtime), the zero position of the micrometer (WL cal step number) and the constant used for ozone measurements through the diffuser (GI calibration factor) are required. The former two are available in the icf file while the latter is available in the O3Brewer file (http://www.o3soft.eu/o3brewer.html).

Figure 8: Example of the Table for the slit and the filters.

In the Slit and Filters table, there is information regarding the applied temperature and neutral density (ND) filter corrections, the individual ETC correction for each slit and the Rayleigh coefficients used in the ozone retrieval algorithm. The Temperature coefficients and the attenuation of the Neutral Density Filters can be found in the icf file. The ETC correction for each slit is determined during the ozone calibration of the Brewer and is provided in the report of the calibration. The Rayleigh coefficients are standard and can be found in the O3Brewer file.

The setup form contains one more table wherein the constants for the zenith sky measurements are determined. However, the specific product is still under development, thus the corresponding table is inactive.

●       Algorithm

This form contains information regarding the Ozone Weights and the SO2 Weights for each slit, used in the algorithm for the retrieval of O₃ and SO₂ respectively. The values presented in Figure 9 are standard for all Brewer spectrophotometers.

Figure 9: Example of the form for the algorithm constants.

●       Instrumental constants

In this form the operational constants of the instrument should be provided. An example is provided in Figure 10. In the first line, the interval scaling factor is asked. This quantity is equal to the half of the integration time and is usually standard (equal to 0.1147). The constants provided in the next three lines can be all found in the operational icf file and are described in detail in the Brewer operator manual (http://www.kippzonen.com/Download/207/Brewer-MkIII-Operator-s-Manual?ShowInfo=true). The information provided in the last line (EXTRA ICF) is available in the icf files of the more recent versions of the Brewer operational software. The constants that have to be filled in this last line are:

●       Auto HP/HG/FR/RE: Defines whether or not HP/HG/FR/RE are done automatically when the instrument recovers and begin operating again after an unexpected error (e.g. communication failure) and if HP is automatically performed before HG for MKIII Brewers. The possible values are 1 (yes) and 0 (no).

●       LOW / OSC / HIGH neutral density: Defines how the ND filter disk moves when testing intensity. The possible values are 0 and -1 for going to lower or higher attenuation filter respectively, or 1 when the disk oscillates and either a higher or a lower attenuation ND filter can be chosen.

●       FW2 position for HG: Defines the position of the filter wheel #2 (used ND filter) when the Hg test is performed.

●       Upper limit (count rate 1 cycle): Defines the upper limit for count rate at 1 cycle that is acceptable before switching to lower FW2 position. Standard is 80000.

●       Lower limit (count rate 1 cycle): Defines the lower limit for count rate at 1 cycle that is acceptable before switching to higher FW2 position. Standard is 25000.

●       Cubic dispersion: If the dispersion coefficients have been calculated using a cubic polynomial then the specific value is set to 1, else it is set to 0.

●       Skip DS in 1 cycle at ND = 0: If 1 then DS is skipped if less than 500 counts are measured in 1 cycle using ND = 0.

Figure 10: Example of the form for the operative constants.

In this Section the user has the choice to fill the Data processing form and the Metadata and Submission form. The upper table in the Data processing form contains the rejection criteria for the final products. At the moment only the values in the first line of the table are operative. The SZA limit defines a maximum SZA above which the TOC is rejected. The O3 low limit and the O3 high limit define the limits between which the measured TOC is acceptable. These limits are applied on the product of levels 1.5 and 2. The O3brewer table contains a number of options existing in the O3brewer file, which however are currently inactive. Similarly, most options in the Metadata and Submission form are currently inactive. This form contains information which is planned to be used for direct submission of the data in the WOUDC database.

Figure 11: Example of the Data Processing form.

Figure 12: Example of the Metadata and Submission form.

In this form, information regarding the production of NO₂ can be provided, although the specific product is still under development.

2.2.4.      Comments

A table wherein the operator can add his comments regarding the product and station configuration is available at the end of the page.

Registered users have the ability to download either the final Eubrewnet products (currently TOC) or the Brewer files which have been stored in the database. Downloading is possible either directly from the Eubrewnet interface or by using the proper access functions. In the following the process that has to be followed by the registered users so that they can download the data using the Eubrewnet interface is described, while the use of access functions is described in section 4.

The register user should first choose the Download files option shown in Figure 13. Then, a form providing the possible download options is appeared (Figure 14).

Figure 13: Brewer files downloading option

Figure 14: File downloading options

In the first line of the form the user can select the instruments for which he/she wants to download files. The “Show” button can be used to define which instruments will be displayed on the screen, and subsequently can be chosen for data downloading. The possible options are:

●       all: all Brewers participating in Eubrewnet

●       active: Brewers which send data regularly

●       preferred: the Brewers for which the user is responsible. If the database user is not an operator no Brewer appears.

After choosing the instruments for which the files will be downloaded, the user should choose the type of files he/she wants from a list of several possible file-types, and the period for which the files will be downloaded. Note that not all the displayed file types are available for each instrument. The files are directly downloaded in a zip folder when the user presses the submit button.

In order to download TOC for a specific station, the user has to choose the station from the interactive map in the first page of the Eubrewnet interface. Then, a new page with information for the chosen station appears (Figure 15). By choosing product (currently only ozone is active) and product level for a specific date, the relative information is displayed visually. It is also possible to download the chosen product (for the chosen day, the chosen month or the entire chosen year) by following one of the links at the bottom left of the page. The downloaded product is in this case for the instrument with the lowest serial number (for Brewer with serial number 157 in the case of the presented example). If the user wants to download the product from another instrument then he/she has to follow the link for the instrument he/she wants.

Figure 15: Information for the site of Izaña.

If the user chooses for example the Brewer with serial number 185, then a new page opens with information for the specific instrument. There, information regarding the instrument and the station is displayed, including the current TOC and UV index (if available) values and the status of the instrument. The user can scroll down and, in the Products section, choose the level and the date he/she wants and display visually the corresponding information.

Figure 16: Products for the Brewer with serial number 185.

In the last section of the specific page (Downloads section) the products which are available for downloading are displayed. If the level 0 has been chosen (as in the example of Figure 16), the user has the option to download the files containing the raw data.

Before downloading the data the user has to accept the data usage conditions of Eubrewnet (see Figure 17). If the user does not agree with the terms in the “DATA-Usage and guidelines” form then he/she is redirected to the main page.

Figure 17: Usage and guidelines of the Eubrewnet data.

In addition to the desired data, ancillary information can be found in the beginning of the ozone files as shown in the example of Figure 18. More specifically:

●       Information regarding the level of the product.

●       Information regarding the Eubrewnet data usage rules.

●       Configuration information.

●       Analytical information regarding the content of each of the 27 columns composing the provided data.

●       A line with the data format of each line (which can be directly used in matlab).

●       Short information regarding the content of each column.

Figure 18: Example of the information contained in an ozone file downloaded from Eubrewnet.

As already discussed, more information is provided in the files, in addition to the date and the ozone and sulfur dioxide values. More specifically, each file contains the following 27 columns:

●       Column 01: (brewerid) Brewer identification number (Brewerid)

●       Column 02: (gmt) UT time of the measure in ISO 8601 format (GMT)

●       Column 03: (configid) Configuration identification number (Configid)

●       Column 04: (n_sum) Index of daily summary (Index)

●       Column 05: (date_index) Continuous date index (0.0 = 0000-00-00T00:00:00Z) based in Matlab datenum (Days)

●        Column 06: (sza) Solar zenith angle using time function (Degrees)

●        Column 07: (airmass) Calculated airmass using time function (Airmass)

●        Column 08: (temperature) Instrument temperature (C Degrees)

●        Column 09: (filt) Applied neutral density filter (Filt)

●        Column 10: (o3) Calculated Ozone value with algorithm version 2 + SL correction (DU) (DU)

●        Column 11: (std_o3) Standard deviation of the group of measurements (DU)

●        Column 12: (so2) Calculated so2 value with Standard algorithm + attenuation filter correction (DU) (DU)

●        Column 13: (std_so2) Standard deviation of the group of measurements (DU)

●        Column 14: (latitude) Latitude of the Brewer Location (Degrees)

●        Column 15: (longitude) Longitude of the Brewer Location (Degrees)

●        Column 16: (press) Medium Pressure of the Brewer Location (hP) (Milibars)

●        Column 17: (r6) Standard Lamp R6 ratio (DU) (DU)

●        Column 18: (r6_ref) Standard Lamp R6 reference ratio (DU) (DU)

●        Column 19: (r5) Standard Lamp R5 ratio (DU) (DU)

●        Column 20: (r5_ref) Standard Lamp Reference R5 ratio (DU) (DU)

●        Column 21: (o3_l1) O3 Level 1 (DU) (DU)

●        Column 22: (so2_l1) SO2 Level 1 (DU) (DU)

●        Column 23: (filter_flag) APPLIED FLAGS: HG Filter, Max Airmass, O3 std (1 means True) (Applied flags)

●        Column 24: (correction_flag) APPLIED CORRECTIONS: Straylight, ETC filter dependent, Standard Lamp correction(1 means True) (Applied corrections)

●        Column 25: (configdate) UT Date of the used configuration in ISO 8601 format (GMT)

●        Column 26: (configtype) Type of the used configuration (B header(1), ICF (2), Config (3))

●        Column 27: (process_date) UT process time of the product in ISO 8601 format (process GMT)

The registered users have access to functions which can be used to directly get the information they need (without downloading and processing the files containing this information). The root for all access functions is:

 http://rbcce.aemet.es/eubrewnet/data/

If the user wants to use a specific function, then he/she should add the name of the function to the above link, followed by comments which specify how the function is going to be used. There are three different types of functions:

●       Get functions: Can be used to access stored data, without further processing.

●       Process functions: Can be used for processing on demand

●       Update functions: Can be used by the operators to update the products of their Brewers.

The three different types of available functions are listed and shortly described in Tables 1, 2 and 3 respectivelly. The link in the last column is an example of getting data for the Brewer with serial number 157. The extended documentation for each function can be directly accessed in the Eubrewnet website by clicking on the links in the column Long Description. Extended description for each function is also provided in the following link:

http://rbcce.aemet.es/dokuwiki/doku.php?id=codes:dbaccess

Table 1:Eubrewnet access functions

Table 2:process functions

Table 3: update functions

Functions’ input and output

Almost all functions can receive specific inputs and produce output files using four different formats. The common inputs and links to specific examples are provided in Table 4. The corresponding information for the output formats is provided in Table 5.

Table 4: Common functions’ inputs

Table 5:The four different ways of data access using the format input parameter.