codes:eubrewnet_report
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| codes:eubrewnet_report [2018/03/23 15:35] – ilias | codes:eubrewnet_report [2022/10/20 07:57] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 32: | Line 32: | ||
| * Near real time data. | * Near real time data. | ||
| - | Eubrewnet system is a Data Storage and Scientific Information Processing system for the Brewer spectrophotometers (Brewer et al., 1973) operating in different European stations. Currently it also includes Brewer spectrophotometers operating in Asia, Africa, Australia and South America ([[https:// | + | Eubrewnet system is a Data Storage and Scientific Information Processing system for the Brewer spectrophotometers (Brewer et al., 1973) operating in different European stations. Currently it also includes Brewer spectrophotometers operating in Asia, Africa, Australia and South America ([[https:// |
| The database system consists of a number of subsystems or components of varying complexity, which interact and perform tasks such as: | The database system consists of a number of subsystems or components of varying complexity, which interact and perform tasks such as: | ||
| Line 41: | Line 41: | ||
| * Product Generation: | * Product Generation: | ||
| - | Interaction between the users and the database is achieved through the Eubrewnet website ([[https:// | + | Interaction between the users and the database is achieved through the Eubrewnet website ([[https:// |
| All the above processes are performed by one or more components, which are implemented using variable technologies and open source free software. It is always possible to extend the systems by own development. | All the above processes are performed by one or more components, which are implemented using variable technologies and open source free software. It is always possible to extend the systems by own development. | ||
| Line 50: | Line 50: | ||
| ===== 2. General description ===== | ===== 2. General description ===== | ||
| - | In Figure 1 there is a simplified diagram of the Eubrewnet system. The system works using Hypertext Transfer Protocol (HTTP)([[https:// | + | In **Figure 1** there is a simplified diagram of the Eubrewnet system. The system works using Hypertext Transfer Protocol (HTTP)([[https:// |
| * The website of Eubrewnet: | * The website of Eubrewnet: | ||
| Line 70: | Line 70: | ||
| ==== Import module ==== | ==== Import module ==== | ||
| - | This module is the basis of the entire project. It is responsible for verifying the validity of each file obtained in the acquisition module and then store the desired information in the database. This module consists by four sub-modules described above. A schematic diagram of this process is shown in Figure 2. | + | This module is the basis of the entire project. It is responsible for verifying the validity of each file obtained in the acquisition module and then store the desired information in the database. This module consists by four sub-modules described above. A schematic diagram of this process is shown in **Figure 2**. |
| {{ : | {{ : | ||
| Line 97: | Line 97: | ||
| ===== 3. Interface ===== | ===== 3. Interface ===== | ||
| - | The interface of the Eubrewnet webpage is presented in the following figures. An interactive map shows the stations which provide data by one or more Brewers. The user can choose a station either from the map (Figure 3) or from a list (Figure 4). As already explained, registered users have more options. | + | The interface of the Eubrewnet webpage is presented in the following figures. An interactive map shows the stations which provide data by one or more Brewers. The user can choose a station either from the map (**Figure 3**) or from a list (**Figure 4**). As already explained, registered users have more options. |
| + | |||
| + | | {{ : | ||
| + | **Figure 3:** The interactive map of the Eubrewnet webpage for (a) non-registered and (b) registered users. | ||
| - | | {{ : | ||
| {{: | {{: | ||
| - | When a station has been selected, then additional options and information appear for the particular station and instruments (Figure 5). | + | When a station has been selected, then additional options and information appear for the particular station and instruments (**Figure 5**). |
| {{: | {{: | ||
| Line 110: | Line 112: | ||
| - | ===== 4. Products ===== | + | ===== 4. Products ===== |
| As already discussed, three products are available: TOC, UV and AOD. The generation of the TOC product is already operative while the generation of the UV and AOD products is still under development. There are three different levels of the products (1.0, 1.5 and 2.0). Levels 1.5 and 2.0 are available to the public, while level 1.0 is available only to the registered users. Raw data (which could be described as Level 0) are stored in the database, though graphical information is not available through the web page. Though, registered users can download raw data using proper functions (see Section 5.5). In general, a higher product level means a more reliable product. | As already discussed, three products are available: TOC, UV and AOD. The generation of the TOC product is already operative while the generation of the UV and AOD products is still under development. There are three different levels of the products (1.0, 1.5 and 2.0). Levels 1.5 and 2.0 are available to the public, while level 1.0 is available only to the registered users. Raw data (which could be described as Level 0) are stored in the database, though graphical information is not available through the web page. Though, registered users can download raw data using proper functions (see Section 5.5). In general, a higher product level means a more reliable product. | ||
| Line 120: | Line 122: | ||
| The TOC is calculated by the direct sun (DS) Brewer measurements which have been processed using the standard Brewer algorithm (Kipp & Zonnen 2008). In order to obtain a reliable product with uncertainties within the desirable uncertainty limits, proper calibration and correction constants have to be used for the processing of the DS measurements. These constants are provided by the operator and stored in a proper configuration table. A detailed list of the configuration parameters which have to be declared by the operator can be found at the following link: | The TOC is calculated by the direct sun (DS) Brewer measurements which have been processed using the standard Brewer algorithm (Kipp & Zonnen 2008). In order to obtain a reliable product with uncertainties within the desirable uncertainty limits, proper calibration and correction constants have to be used for the processing of the DS measurements. These constants are provided by the operator and stored in a proper configuration table. A detailed list of the configuration parameters which have to be declared by the operator can be found at the following link: | ||
| - | [[https:// | + | [[https:// |
| while an overview of the processes which can be used by the operators to fill the configuration table is provided here: | while an overview of the processes which can be used by the operators to fill the configuration table is provided here: | ||
| - | [[https:// | + | [[https:// |
| - | In the Eubrewenet wiki there is also analytical information regarding the processing ([[https:// | + | In the Eubrewenet wiki there is also analytical information regarding the processing ([[https:// |
| * level 0: The TOC coming directly from the Brewers.These values have been calculated from the Direct Sun measurements using the constants from the configuration file which is stored in the client (Brewer) pc. | * level 0: The TOC coming directly from the Brewers.These values have been calculated from the Direct Sun measurements using the constants from the configuration file which is stored in the client (Brewer) pc. | ||
| Line 158: | Line 160: | ||
| Since the UV and AOD products are under development, | Since the UV and AOD products are under development, | ||
| - | [[https:// | + | [[https:// |
| - | The TOC configuration constants have to be updated after a new calibration of the instrument (usually every ~2 years) or after important changes in its operational characteristics. Occasionally, | + | The TOC configuration constants have to be updated after a new calibration of the instrument (usually every ~2 years) or after important changes in its operational characteristics. Occasionally, |
| - | {{: | + | {{: |
| - | {{: | + | {{: |
| Different sets of configuration constants correspond to different periods (Figure 7). When a user is logged in, he/she should select whether he/she wants to modify an existing set of configuration constants or to add a new one (for a new period). Then the user can either upload a file with all the needed information or change the information he/she wants by hand. The corresponding tools and procedures for UV and AOD will be more complicated since a larger amount of information has to be provided by the user. More information regarding the configuration versioning can be found in: | Different sets of configuration constants correspond to different periods (Figure 7). When a user is logged in, he/she should select whether he/she wants to modify an existing set of configuration constants or to add a new one (for a new period). Then the user can either upload a file with all the needed information or change the information he/she wants by hand. The corresponding tools and procedures for UV and AOD will be more complicated since a larger amount of information has to be provided by the user. More information regarding the configuration versioning can be found in: | ||
| - | [[https:// | + | [[https:// |
| ==== 4.5. Dissemination: | ==== 4.5. Dissemination: | ||
| Line 174: | Line 176: | ||
| === Access functions === | === Access functions === | ||
| - | A number of functions have been developed, which allow the registered users to directly get the information stored in the database (i.e. products of different levels, raw data, results from diagnostic tests etc). These functions are written in python and are included in the module getdata.py (projects/ | + | A number of functions have been developed, which allow the registered users to directly get the information stored in the database (i.e. products of different levels, raw data, results from diagnostic tests etc). These functions are written in python and are included in the module getdata.py (projects/ |
| **Table 1:** Possible ways of accessing the Eubrewnet data using access functions and links for examples for the Get DS function. | **Table 1:** Possible ways of accessing the Eubrewnet data using access functions and links for examples for the Get DS function. | ||
| | Format input | Description | Example | | | Format input | Description | Example | | ||
| - | | jsonM | JSON matrix formed by lists of lists (default value). The first list is formed by the parameters’ names and the following lists are the query outputs | [[https:// | + | | jsonM | JSON matrix formed by lists of lists (default value). The first list is formed by the parameters’ names and the following lists are the query outputs | [[https:// |
| - | | jsonO | JSON object of lists, where each key is the parameter name and its value is the time sorted list of them | [[https:// | + | | jsonO | JSON object of lists, where each key is the parameter name and its value is the time sorted list of them | [[https:// |
| - | | text | Human readable, comma separated values where first row is the parameters’ names and the following are the query outputs | [[https:// | + | | text | Human readable, comma separated values where first row is the parameters’ names and the following are the query outputs | [[https:// |
| - | | csv | CSV direct download where in first row are the parameters’ names and the following are the query outputs | [[https:// | + | | csv | CSV direct download where in first row are the parameters’ names and the following are the query outputs | [[https:// |
| Almost all functions can receive the inputs listed in Table 2: | Almost all functions can receive the inputs listed in Table 2: | ||
| Line 189: | Line 191: | ||
| | Parameter | Description | Example | | | Parameter | Description | Example | | ||
| - | | brewerid | Brewer identification number | [[https:// | + | | brewerid | Brewer identification number | [[https:// |
| - | | date | Date in YYYY-MM-DD format | [[https:// | + | | date | Date in YYYY-MM-DD format | [[https:// |
| - | | enddate | Date in YYYY-MM-DD format. If provided, the function will return the query in a date range | [[https:// | + | | enddate | Date in YYYY-MM-DD format. If provided, the function will return the query in a date range | [[https:// |
| More analytical information regarding the access functions can be found in: | More analytical information regarding the access functions can be found in: | ||
| - | [[https:// | + | [[https:// |
| === Brewer Python Library === | === Brewer Python Library === | ||
| Line 201: | Line 203: | ||
| A library for Brewer data processing has been developed using Python and has been integrated in the Eubrewnet database. The standard algorithm for Brewer data processing is based on the version 4.1 of the brewer software ([[https:// | A library for Brewer data processing has been developed using Python and has been integrated in the Eubrewnet database. The standard algorithm for Brewer data processing is based on the version 4.1 of the brewer software ([[https:// | ||
| - | [[https:// | + | [[https:// |
| The last version of the brewer python module is available at the following link: | The last version of the brewer python module is available at the following link: | ||
| Line 226: | Line 228: | ||
| When the user clicks on a brewer, then he is redirected to a page with details regarding the state of the specific instrument, where a list of the original error messages from the D or B files are also included. More details regarding the Brewer diagnostic tool can be found to the following link: | When the user clicks on a brewer, then he is redirected to a page with details regarding the state of the specific instrument, where a list of the original error messages from the D or B files are also included. More details regarding the Brewer diagnostic tool can be found to the following link: | ||
| - | [[https:// | + | [[https:// |
| ---- | ---- | ||
| Line 249: | Line 251: | ||
| López-Solano J., Carlund T., Hernández-Cruz B., León-Luis S.F., Carreño V., Berjón A., Santana-Díaz D., Bais A.F., Moreta J.R., Shirotov V., Rimmer J., Vilaplana J.M., Wilson K.M., Gröbner J., Boulkelia L., Eriksen P., Karppinen T., Diémoz H., Rodríguez Valido M., Redondas A.: “Aerosol optical depth in the ultraviolet range: a new product in EUBREWNET”, | López-Solano J., Carlund T., Hernández-Cruz B., León-Luis S.F., Carreño V., Berjón A., Santana-Díaz D., Bais A.F., Moreta J.R., Shirotov V., Rimmer J., Vilaplana J.M., Wilson K.M., Gröbner J., Boulkelia L., Eriksen P., Karppinen T., Diémoz H., Rodríguez Valido M., Redondas A.: “Aerosol optical depth in the ultraviolet range: a new product in EUBREWNET”, | ||
| + | |||
codes/eubrewnet_report.1521819313.txt.gz · Last modified: 2018/03/23 15:35 by ilias