namespace:page
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| namespace:page [2018/03/21 15:08] – ilias | namespace:page [Unknown date] (current) – removed - external edit (Unknown date) 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ======= Eubrewnet user’s manual ======= | ||
| - | ---- | ||
| - | ====== 1. Setup and data sending ====== | ||
| - | |||
| - | Each Brewer spectrophotometer communicates with a local PC where the raw data are recorded. For uploading the raw data from the client PC to the Eubrewnet database a set of routines written in python has to be installed on the local PC (or any other computer which has access to the Brewer raw data). When these routines are executed the local (client) PC is connected to the Eubrewnet server. Then, the local brewer files and all relative information (file size, modification date etc.) are listed and compared to the corresponding information for the files that have been already stored in the server. Only the files which are either new or have changed, are transferred from the former to the later. This process is repeated in regular time intervals (prefferably ~15 mins). In sections 1.1 - 1.4, the software installation and application processes are analytically described. A fast installation guide is available in section 1.5. | ||
| - | |||
| - | |||
| - | ===== 1.1. Software installation ===== | ||
| - | |||
| - | Before the installation of the software the operator has to communicate with the [[eubrewnet@aemet.es|administrator]](eubrewnet@aemet.es) and ensure that his Brewer has been included in the[[http:// | ||
| - | |||
| - | __1< | ||
| - | |||
| - | For Windows and Mac, Python 2.7 is recommended while for Linux, the Python Version should come bundled with the Linux distribution. Below is a list of the appropriate interpreters for Windows and Mac. | ||
| - | |||
| - | ● [[https:// | ||
| - | |||
| - | ● [[https:// | ||
| - | |||
| - | ● [[https:// | ||
| - | |||
| - | ● [[https:// | ||
| - | |||
| - | ● Linux Version should come bundled with your Linux distribution. | ||
| - | |||
| - | You have to be sure that your python interpreter is callable from every path in your system (Open a console and try **python**, **exit(****)** for exit). | ||
| - | |||
| - | Then, [[http:// | ||
| - | |||
| - | __2< | ||
| - | |||
| - | In this case, the [[http:// | ||
| - | (in dist folder) in a console to test if client is working properly. No error message should appear. | ||
| - | |||
| - | |||
| - | ===== 1.2. Client configuration file ===== | ||
| - | |||
| - | The client configuration file (client.ini) is used by the refresh.py program. The later is the routine used to transfer the files from the client PC to the database. Depending on the installation method, these two files can be found either in client_python\source (1st installation method) or in client_exe\dist (2nd installation method). Client.ini is a plain ASCII [[https:// | ||
| - | |||
| - | In the following example, the content of a client.ini file is presented. The lines in bold should not be changed by the operator since they provide information relative to the database server. The information in the remaining (regular font) lines should be defined by the operator. | ||
| - | |||
| - | | **Example 1:** The content of a client.ini file\\ **[global]**\\ brewerid=157\\ working_dir=.\\ input=c: | ||
| - | |||
| - | In the **global section** of the client.ini file the following parameters should be declared: | ||
| - | |||
| - | ● **brewerid** : brewer identifier (i.e. the brewer serial number). | ||
| - | |||
| - | ● **input**: the directory which contains the files produced by the Brewer. | ||
| - | |||
| - | ● **servers**: | ||
| - | |||
| - | ● **working_dir**: | ||
| - | |||
| - | ● **proxy**: proxy url, if required. If not required, this property should be commented (add # at the beginning of the line). | ||
| - | |||
| - | A **server section** is identified by the server alias (i.e. the alias is the name between the brackets - iberonesia3 in the case of Example 1), and has the following properties: | ||
| - | |||
| - | ● **url** : server url. | ||
| - | |||
| - | ● **db** : a local database (a simple data file) is created the first time that client software is executed. This line defines the name of this file. | ||
| - | |||
| - | ● **noproxy** : if a proxy is specified in the global section, noproxy may be set to yes to use a direct connection (avoiding the proxy) to access the server. | ||
| - | |||
| - | ● **serverport**: | ||
| - | |||
| - | The lines that are not used should be commented (i.e. the proxy property if no proxy is used). The easiest way to configure the client is to properly specify all options in the client.ini file and then execute refresh.py without arguments. However, if desired, some of the above properties can be overridden using command line parameters when executing refresh.py or alive.py. If for example, the client.ini file of Example 1 is used, a different Brewer id and a different input path can be defined in the command line: | ||
| - | |||
| - | ** $>python refresh.py -b 086 -i c: | ||
| - | |||
| - | Though, it is preferable to specify the Brewer id and the input path (as well as any other parameters) in the client.ini file instead of using the **-b** and **-i** commands. | ||
| - | |||
| - | As already described, most of the refresh.py options may be configured though command line arguments or client.ini configuration file. Any option specified in command line overrides the option in client.ini. All options are shown when the the command ** “****$ ./ | ||
| - | |||
| - | | **Example 2:** The refresh.py options\\ ** $ ./ | ||
| - | |||
| - | The mandatory options are: | ||
| - | |||
| - | ● **working_dir** ****:This option must be specified in client.ini. It defines the directory where the local refresh database is stored. | ||
| - | |||
| - | ● **input**: Defines the file which will be processed. If a directory is defined, then all files in the particular directory will be processed. | ||
| - | |||
| - | ● **brewerid** : Brewer identifier. | ||
| - | |||
| - | ● **servers**: | ||
| - | |||
| - | |||
| - | ===== 1.3. Communication with the server ===== | ||
| - | |||
| - | Open a browser and try access [[http:// | ||
| - | |||
| - | ** $>python refresh.py -b XXX -vvv\\ - XXX: Id of a brewer** | ||
| - | |||
| - | If you want to save the errors and log, define log and error files: | ||
| - | |||
| - | ** $>python refresh.py 1> | ||
| - | |||
| - | The two files (client.log and client_error.log) should not contain error messages after the execution of the client software. In case of errors in the communication, | ||
| - | |||
| - | |||
| - | ===== 1.4. Schedule the data sending ===== | ||
| - | |||
| - | The Brewer data can be sent to Eubrewnet either directly from the operational Brewer PC or by using another computer which has access to the data. | ||
| - | |||
| - | __From the Brewer computer:__ | ||
| - | |||
| - | Sending the Brewer data to Eubrewnet in regular time intervals can be achieved by adding a routine in your Brewer schedule. The routine is executed in desired time intervals and calls the client software. | ||
| - | |||
| - | In W2k W7 the brewer software waits until the execution of each command is finished so if the communication is slow or fails it takes too much time to send the data, and may block or delay the brewer software. This does not happen if DOSBOX is used. The old MSDOS/ | ||
| - | |||
| - | Thus, it is recommended to send the data in two steps: | ||
| - | |||
| - | 1. Inside the brewer software or a task scheduler run a routine to copy the bdata directory content in another place/ | ||
| - | |||
| - | ** set DEST_DIR=e: | ||
| - | |||
| - | 2. Run the data sending software from this directory as scheduled task | ||
| - | |||
| - | **$> | ||
| - | |||
| - | |||
| - | ===== 1.5. Fast installation guide ===== | ||
| - | |||
| - | **1- Install**the python interpreter and then download and uncompress client script ([[http:// | ||
| - | |||
| - | Copy the scripts to the brewer program directory | ||
| - | |||
| - | **2- Check the configuration file (client.ini)** | ||
| - | |||
| - | Are you bellow a proxy ? | ||
| - | |||
| - | **no**: then change noproxy in client.ini to YES !!! | ||
| - | |||
| - | **yes**: then edit the client.ini and indicate your proxy server and port, set the noproxy flag to NO !! | ||
| - | |||
| - | Then edit the necessary fields in the client.ini file, without changing the following fields: | ||
| - | |||
| - | **CLIENT.INI** | ||
| - | |||
| - | **[global] servers=iberonesia3** | ||
| - | |||
| - | **proxy=****proxy.xxx.xxx: | ||
| - | |||
| - | **[iberonesia3] url=****[[http:// | ||
| - | |||
| - | **db****=****iberonesia3.db** | ||
| - | |||
| - | **serverport****=****8081** | ||
| - | |||
| - | **noproxy****=****no** | ||
| - | |||
| - | **3- execute the script** | ||
| - | |||
| - | **python** **refresh.py -- b XXX -i c: | ||
| - | |||
| - | XXX is the name of your brewer and brewer_data is a directory where the brewer files are. | ||
| - | |||
| - | If it takes long, it is probably is working fine. Check the log files to see if all the data of your test directory have been sent. | ||
| - | |||
| - | **4- Schedule the data sent**: We use robocopy to make a copy of the bdata directory. Edit the following two files (EUBREWNET.BAT and CK.RTN) properly and add them to the brewer program directory. Then put CK on your brewer schedule to send data to eubrewnet | ||
| - | |||
| - | **EUBREWNET.BAT** | ||
| - | |||
| - | **set** **DEST_DIR=e: | ||
| - | |||
| - | **set****ORIG_DIR=e: | ||
| - | |||
| - | **set** **X=157** | ||
| - | |||
| - | **robocopy** **%ORIG_DIR% %DEST_DIR%\Brw#< | ||
| - | |||
| - | **python** **refresh.py -b < | ||
| - | |||
| - | **CK.RTN** | ||
| - | |||
| - | **10000 rem Call eubrewnet.bat** | ||
| - | |||
| - | **10010 data ck** | ||
| - | |||
| - | **20000 shell”eubrewnet** | ||
| - | |||
| - | **30000 return** | ||
| - | |||
| - | **65529 rem dummy****** | ||
| - | |||
| - | \\ ** ** | ||
| - | |||
| - | ---- | ||
| - | ====== 2. Configuration ====== | ||
| - | |||
| - | The raw data (Brewer spectral measurements) sent to the Eubrewnet database have to be processed in order to get the final products (currently Total Ozone Column (O< | ||
| - | |||
| - | [[http:// | ||
| - | |||
| - | Before uploading the configuration for the station and the products, the operators have to contact the system administrator ([[eubrewnet@aemet.es|eubrewnet@aemet.es]]) and provide contact information and the list of Brewers for which they are responsible. Then, they have to register in the system. When the registration is complete they are able to login the website of [[http:// | ||
| - | |||
| - | __Registration and login__ | ||
| - | |||
| - | As shown in Figure 1, at the right of the Eubrewnet webpage the user can find the **sign in** button. There, he/she has two options: **Register** or **Login**. If **Register** is chosen, a form appears which should be filled by the database user. If **Log in** is chosen the user is asked for his username and password and then he can have full access to the database. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | |||
| - | ===== 2.1. Station configuration ===== | ||
| - | |||
| - | When a user is logged in he/she can edit/fill the station information. The option **Stations** should be chosen from the **User** list (see Figure 2). Then, by clicking on the name of the station, a form containing the station information appears. There, the user can add information or edit the form and submit the changes. The information can be added either by hand or by uploading a document which contains the station configuration information. | ||
| - | |||
| - | **{{ : | ||
| - | |||
| - | **{{ : | ||
| - | |||
| - | ** ** | ||
| - | |||
| - | ===== 2.2. Ozone configuration ===== | ||
| - | |||
| - | As shown in Figure 4, the user can have access to the ozone configuration form by clicking on the **Ozone Configuration** option from the **User** list. In the ozone configuration form the user has the option to update the configuration for more products such as AOD and NO< | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | The process begins by selecting the Brewer for which the configuration constants will change. Then, as shown in Figure 5, the user has three options: | ||
| - | |||
| - | - Add new configuration | ||
| - | - Modify configuration | ||
| - | - Download configuration | ||
| - | |||
| - | If the **Download configuration** option is chosen, the entire record of the configurations which have been either used in the past or are currently in use is downloaded in a single csv file. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | If the user wants to modify an existing configuration then he\she has to choose the configuration version he/she wants to modify and press the **Modify Configuration** button. The date since which an existing configuration is valid cannot be changed. In this case the configuration has to be resubmitted as new. For adding a new configuration the user has to define the date in the Date box (next to the Brewerid box) since which the new configuration is valid, and then press the **Add new configuration** button. | ||
| - | |||
| - | By pressing the **Add new Configuration** or **Modify Configuration** buttons, a new form appears. In this form, the user has the option to either upload files which contain the new configuration constants (Figure 5), or modify the currently used configuration parameters and save them as new (or modified) configuration. In the **Configuration and Actions** section (Figure 6), in addition to the choice of uploading files with the new configuration parameters, the user has the following options: (i) to decide if the specific configuration is operative or inactive and (ii) to define if the generated product from the current configuration is level 2 or not. Level 2 is the level 1.5 product after it has been validated with a posterior calibration. In this section the user has also the options to either go back or submit the performed changes by clicking the corresponding buttons. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | In the beginning of the **Values to be submitted** section, there are two lines of buttons. The buttons of the first line can be used to choose where the submitted configuration is applicable. The possible choices are: | ||
| - | |||
| - | ● O< | ||
| - | |||
| - | ● AOD | ||
| - | |||
| - | ● Umkehr | ||
| - | |||
| - | ● Data and Metadata | ||
| - | |||
| - | ● NO< | ||
| - | |||
| - | The AOD and Umkehr products are under development (at the time of writing this manual) and it is not possible to provide or edit any configuration constants. The configuration constants in the NO< | ||
| - | |||
| - | ==== 2.2.1. O3 and SO2 ==== | ||
| - | |||
| - | In the O< | ||
| - | |||
| - | ● **__Setup__****** | ||
| - | |||
| - | In this form there are three tables which should be filled. The first is for the setup of the DS measurements (Figure 7). In the first raw the operator should provide the needed constants for the retrieval of the total ozone column (O< | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | In the second line the operator should provide the needed constants for the retrieval of the total column of sulfur dioxide (SO< | ||
| - | |||
| - | In the third line the applied filters for the calculation of the level 1.5 and level 2 products are specified. Each single O< | ||
| - | |||
| - | In the fourth line the parameters used for stray light correction are provided (**A** and **B**). These parameters are available only if the instrument has been properly characterized. In the fifth line the dead time of the photomultiplier tube (**Deadtime**), | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | In the **Slit and Filters** table, there is information regarding the applied temperature and neutral density (ND) filter corrections, | ||
| - | |||
| - | The setup form contains one more table wherein the constants for the zenith sky measurements are determined. However, the specific product is still under development, | ||
| - | |||
| - | ● **__Algorithm__****** | ||
| - | |||
| - | This form contains information regarding the **Ozone Weights** and the **SO2 Weights** for each slit, used in the algorithm for the retrieval of O< | ||
| - | |||
| - | **__{{ : | ||
| - | |||
| - | ● **__Instrumental constants__****** | ||
| - | |||
| - | In this form the operational constants of the instrument should be provided. An example is provided in Figure 10. In the first line, the interval scaling factor is asked. This quantity is equal to the half of the integration time and is usually standard (equal to 0.1147). The constants provided in the next three lines can be all found in the operational icf file and are described in detail in the Brewer operator manual ([[http:// | ||
| - | |||
| - | ● **Auto HP/ | ||
| - | |||
| - | ● **LOW / OSC / HIGH neutral density:** Defines how the ND filter disk moves when testing intensity. The possible values are 0 and -1 for going to lower or higher attenuation filter respectively, | ||
| - | |||
| - | ● **FW2 position for HG:** Defines the position of the filter wheel #2 (used ND filter) when the Hg test is performed. **** | ||
| - | |||
| - | ● **Upper limit (count rate 1 cycle):** Defines the upper limit for count rate at 1 cycle that is acceptable before switching to lower FW2 position. Standard is 80000.**** | ||
| - | |||
| - | ● **Lower limit (count rate 1 cycle):** Defines the lower limit for count rate at 1 cycle that is acceptable before switching to higher FW2 position. Standard is 25000.**** | ||
| - | |||
| - | ● **Cubic dispersion: | ||
| - | |||
| - | ● **Skip DS in 1 cycle at ND = 0**: If 1 then DS is skipped if less than 500 counts are measured in 1 cycle using ND = 0. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | ==== 2.2.2. Data and Metadata ==== | ||
| - | |||
| - | In this Section the user has the choice to fill the **Data processing** form and the **Metadata and Submission** form. The upper table in the **Data processing** form contains the rejection criteria for the final products. At the moment only the values in the first line of the table are operative. The **SZA limit** defines a maximum SZA above which the TOC is rejected. The **O3 low limit** and the **O3 high limit** define the limits between which the measured TOC is acceptable. These limits are applied on the product of levels 1.5 and 2. The O3brewer table contains a number of options existing in the O3brewer file, which however are currently inactive. Similarly, most options in the **Metadata and Submission** form are currently inactive. This form contains information which is planned to be used for direct submission of the data in the WOUDC database. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | ==== 2.2.3. NO2 ==== | ||
| - | |||
| - | In this form, information regarding the production of NO< | ||
| - | |||
| - | **2.2.4. ** **Comments** | ||
| - | |||
| - | A table wherein the operator can add his comments regarding the product and station configuration is available at the end of the page. | ||
| - | |||
| - | \\ ** ** | ||
| - | |||
| - | ---- | ||
| - | ====== 3. Download data ====== | ||
| - | |||
| - | Registered users have the ability to download either the final Eubrewnet products (currently TOC) or the Brewer files which have been stored in the database. Downloading is possible either directly from the Eubrewnet interface or by using the proper access functions. In the following the process that has to be followed by the registered users so that they can download the data using the Eubrewnet interface is described, while the use of access functions is described in section 4. | ||
| - | |||
| - | ===== 3.1. Download Brewer files ===== | ||
| - | |||
| - | The register user should first choose the **Download files** option shown in Figure 13. Then, a form providing the possible download options is appeared (Figure 14). | ||
| - | |||
| - | **{{ : | ||
| - | |||
| - | **{{ : | ||
| - | |||
| - | In the first line of the form the user can select the instruments for which he/she wants to download files. The “Show” button can be used to define which instruments will be displayed on the screen, and subsequently can be chosen for data downloading. The possible options are: | ||
| - | |||
| - | ● **all**: all Brewers participating in Eubrewnet | ||
| - | |||
| - | ● **active**: Brewers which send data regularly | ||
| - | |||
| - | ● **preferred**: | ||
| - | |||
| - | After choosing the instruments for which the files will be downloaded, the user should choose the type of files he/she wants from a list of several possible file-types, and the period for which the files will be downloaded. Note that not all the displayed file types are available for each instrument. The files are directly downloaded in a zip folder when the user presses the **submit** button. | ||
| - | |||
| - | ===== 3.2. Download products ===== | ||
| - | |||
| - | In order to download TOC for a specific station, the user has to choose the station from the interactive map in the first page of the Eubrewnet interface. Then, a new page with information for the chosen station appears (Figure 15). By choosing product (currently only ozone is active) and product level for a specific date, the relative information is displayed visually. It is also possible to download the chosen product (for the chosen day, the chosen month or the entire chosen year) by following one of the links at the bottom left of the page. The downloaded product is in this case for the instrument with the lowest serial number (for Brewer with serial number 157 in the case of the presented example). If the user wants to download the product from another instrument then he/she has to follow the link for the instrument he/she wants. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | If the user chooses for example the Brewer with serial number 185, then a new page opens with information for the specific instrument. There, information regarding the instrument and the station is displayed, including the current TOC and UV index (if available) values and the status of the instrument. The user can scroll down and, in the **Products** section, choose the level and the date he/she wants and display visually the corresponding information. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | In the last section of the specific page (**Downloads** section) the products which are available for downloading are displayed. If the level 0 has been chosen (as in the example of Figure 16), the user has the option to download the files containing the raw data. | ||
| - | |||
| - | Before downloading the data the user has to accept the data usage conditions of Eubrewnet (see Figure 17). If the user does not agree with the terms in the “**DATA-Usage and guidelines**” form then he/she is redirected to the main page. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | ===== 3.3. Ozone files format ===== | ||
| - | |||
| - | In addition to the desired data, ancillary information can be found in the beginning of the ozone files as shown in the example of Figure 18. More specifically: | ||
| - | |||
| - | ● Information regarding the level of the product. | ||
| - | |||
| - | ● Information regarding the Eubrewnet data usage rules. | ||
| - | |||
| - | ● Configuration information. | ||
| - | |||
| - | ● Analytical information regarding the content of each of the 27 columns composing the provided data. | ||
| - | |||
| - | ● A line with the data format of each line (which can be directly used in matlab). | ||
| - | |||
| - | ● Short information regarding the content of each column. | ||
| - | |||
| - | {{ : | ||
| - | |||
| - | As already discussed, more information is provided in the files, in addition to the date and the ozone and sulfur dioxide values. More specifically, | ||
| - | |||
| - | ● Column 01: **(brewerid)** Brewer identification number (Brewerid) | ||
| - | |||
| - | ● Column 02: (**gmt**) UT time of the measure in ISO 8601 format (GMT) | ||
| - | |||
| - | ● Column 03: (**configid**) Configuration identification number (Configid) | ||
| - | |||
| - | ● Column 04: (**n_sum**) Index of daily summary (Index) | ||
| - | |||
| - | ● Column 05: (**date_index**) Continuous date index (0.0 = 0000-00-00T00: | ||
| - | |||
| - | ● Column 06: (**sza**) Solar zenith angle using time function (Degrees) | ||
| - | |||
| - | ● Column 07: (**airmass**) Calculated airmass using time function (Airmass) | ||
| - | |||
| - | ● Column 08: (**temperature**) Instrument temperature (C Degrees) | ||
| - | |||
| - | ● Column 09: (**filt**) Applied neutral density filter (Filt) | ||
| - | |||
| - | ● Column 10: (**o3**) Calculated Ozone value with algorithm version 2 + SL correction (DU) (DU) | ||
| - | |||
| - | ● Column 11: (**std_o3**) Standard deviation of the group of measurements (DU) | ||
| - | |||
| - | ● Column 12: (**so2**) Calculated so2 value with Standard algorithm + attenuation filter correction (DU) (DU) | ||
| - | |||
| - | ● Column 13: (**std_so2**) Standard deviation of the group of measurements (DU) | ||
| - | |||
| - | ● Column 14: (**latitude**) Latitude of the Brewer Location (Degrees) | ||
| - | |||
| - | ● Column 15: (**longitude**) Longitude of the Brewer Location (Degrees) | ||
| - | |||
| - | ● Column 16: (**press**) Medium Pressure of the Brewer Location (hP) (Milibars) | ||
| - | |||
| - | ● Column 17: (**r6**) Standard Lamp R6 ratio (DU) (DU) | ||
| - | |||
| - | ● Column 18: (**r6_ref**) Standard Lamp R6 reference ratio (DU) (DU) | ||
| - | |||
| - | ● Column 19: (**r5**) Standard Lamp R5 ratio (DU) (DU) | ||
| - | |||
| - | ● Column 20: (**r5_ref**) Standard Lamp Reference R5 ratio (DU) (DU) | ||
| - | |||
| - | ● Column 21: (**o3_l1**) O3 Level 1 (DU) (DU) | ||
| - | |||
| - | ● Column 22: (**so2_l1**) SO2 Level 1 (DU) (DU) | ||
| - | |||
| - | ● Column 23: (**filter_flag**) APPLIED FLAGS: HG Filter, Max Airmass, O3 std (1 means True) (Applied flags) | ||
| - | |||
| - | ● Column 24: (**correction_flag**) APPLIED CORRECTIONS: | ||
| - | |||
| - | ● Column 25: (**configdate**) UT Date of the used configuration in ISO 8601 format (GMT) | ||
| - | |||
| - | ● Column 26: (**configtype**) Type of the used configuration (B header(1), ICF (2), Config (3)) | ||
| - | |||
| - | ● Column 27: (**process_date**) UT process time of the product in ISO 8601 format (process GMT) | ||
| - | |||
| - | **\\ **====== ====== | ||
| - | |||
| - | ---- | ||
| - | |||
| - | |||
| - | |||
| - | |||
namespace/page.1521644889.txt.gz · Last modified: 2018/03/21 15:08 by ilias