codes:eubrewnet_report
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
codes:eubrewnet_report [2018/03/23 15:21] ilias created |
codes:eubrewnet_report [2022/10/20 07:57] (current) |
||
|---|---|---|---|
| Line 32: | Line 32: | ||
| * Near real time data. | * Near real time data. | ||
| - | Eubrewnet system is a Data Storage and Scientific Information Processing system for the Brewer spectrophotometers (Brewer et al., 1973) operating in different European stations. Currently it also includes Brewer spectrophotometers operating in Asia, Africa, Australia and South America ([[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/brewer/index&sa=D&ust=1521821549664000&usg=AFQjCNH9HP71Z1YhNdY9Yc8w7fxyVzTPTA|http://rbcce.aemet.es/eubrewnet/brewer/index]]). Data from all the participating instruments are sent to the Eubrewnet database several times per day, processed in almost real time and products of different confidence levels are generated. | + | Eubrewnet system is a Data Storage and Scientific Information Processing system for the Brewer spectrophotometers (Brewer et al., 1973) operating in different European stations. Currently it also includes Brewer spectrophotometers operating in Asia, Africa, Australia and South America ([[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/brewer/index&sa=D&ust=1521821549664000&usg=AFQjCNH9HP71Z1YhNdY9Yc8w7fxyVzTPTA|http://eubrewnet.aemet.es/eubrewnet/brewer/index]]). Data from all the participating instruments are sent to the Eubrewnet database several times per day, processed in almost real time and products of different confidence levels are generated. |
| The database system consists of a number of subsystems or components of varying complexity, which interact and perform tasks such as: | The database system consists of a number of subsystems or components of varying complexity, which interact and perform tasks such as: | ||
| Line 41: | Line 41: | ||
| * Product Generation: Various tasks (some of which are automated while others are on demand) are in charge of generating products (elaborated information for analytical and graphical purpose) as system’s output. | * Product Generation: Various tasks (some of which are automated while others are on demand) are in charge of generating products (elaborated information for analytical and graphical purpose) as system’s output. | ||
| - | Interaction between the users and the database is achieved through the Eubrewnet website ([[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/default/index&sa=D&ust=1521821549665000&usg=AFQjCNEplsrPCpJPN0nvTIuh5cIfsc1wbA|http://rbcce.aemet.es/eubrewnet/default/index]]). The website of COST1207 can be used as an additional tool for the users. It provides information for events and activities relative to Eubrewnet network. Additionally it provides links to the Eubrewenet forum where all registered users can post their comments and questions and the owncloud, which can be used to upload documents and data and exchange information. | + | Interaction between the users and the database is achieved through the Eubrewnet website ([[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/default/index&sa=D&ust=1521821549665000&usg=AFQjCNEplsrPCpJPN0nvTIuh5cIfsc1wbA|http://eubrewnet.aemet.es/eubrewnet/default/index]]). The website of COST1207 can be used as an additional tool for the users. It provides information for events and activities relative to Eubrewnet network. Additionally it provides links to the Eubrewenet forum where all registered users can post their comments and questions and the owncloud, which can be used to upload documents and data and exchange information. |
| All the above processes are performed by one or more components, which are implemented using variable technologies and open source free software. It is always possible to extend the systems by own development. | All the above processes are performed by one or more components, which are implemented using variable technologies and open source free software. It is always possible to extend the systems by own development. | ||
| - | ==== General description ==== | + | ---- |
| - | In Figure 1 there is a simplified diagram of the Eubrewnet system. The system works using Hypertext Transfer Protocol (HTTP)([[https://www.google.com/url?q=https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol&sa=D&ust=1521821549665000&usg=AFQjCNHYmkYGoGrsH4S14LfX_7ZDGpCMaA|https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol]]), the outer layer of which is a HTTP proxy. This proxy attends external requests and redirects them to the appropriate subsystem which can be: | + | |
| + | ===== 2. General description ===== | ||
| + | |||
| + | In **Figure 1** there is a simplified diagram of the Eubrewnet system. The system works using Hypertext Transfer Protocol (HTTP)([[https://www.google.com/url?q=https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol&sa=D&ust=1521821549665000&usg=AFQjCNHYmkYGoGrsH4S14LfX_7ZDGpCMaA|https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol]]), the outer layer of which is a HTTP proxy. This proxy attends external requests and redirects them to the appropriate subsystem which can be: | ||
| * The website of Eubrewnet: Attends the user requests and shows visual information. It also provide links to access the information in the database and performs product generation tasks on demand. | * The website of Eubrewnet: Attends the user requests and shows visual information. It also provide links to access the information in the database and performs product generation tasks on demand. | ||
| Line 55: | Line 58: | ||
| * The website of COST1207: Attends the user requests and shows visual information regarding news, events, etc. of the COST1207 action and the Eubrewnet. It also provides links to the Forum and the owncloud. | * The website of COST1207: Attends the user requests and shows visual information regarding news, events, etc. of the COST1207 action and the Eubrewnet. It also provides links to the Forum and the owncloud. | ||
| - | {{ :codes:image1.png |}}| Figure 1: Simplified Diagram of the Eubrewnet system | + | {{ :codes:image1.png |}}**Figure 1:** Simplified Diagram of the Eubrewnet system |
| ==== Modules ==== | ==== Modules ==== | ||
| Line 65: | Line 68: | ||
| A set of routines (projects/client/src/) written in python are installed in each client PC. Every time they are executed the client pc is connected to the Eubrewnet server. The local brewer files and all relative information (file size, modification date etc.) are listed and compared to the corresponding information for the files that have been already stored in the server and only the files which are either new or have changed, are transferred from the former to the later, checking md5 checksums ([[https://www.google.com/url?q=https://en.wikipedia.org/wiki/MD5&sa=D&ust=1521821549668000&usg=AFQjCNFlov60DgO4G4ZFDaShmk2oFVtqRw|https://en.wikipedia.org/wiki/MD5]]). Each file transferred to the server is stored in the input directory corresponding to each brewer. Both the client and server programs are implemented in python. Communication between client and server is achieved via the xml-rpc protocol ([[https://www.google.com/url?q=https://en.wikipedia.org/wiki/XML-RPC&sa=D&ust=1521821549669000&usg=AFQjCNFscykefnoFsMnMw76Pxs9PBEzh1g|https://en.wikipedia.org/wiki/XML-RPC]]). | A set of routines (projects/client/src/) written in python are installed in each client PC. Every time they are executed the client pc is connected to the Eubrewnet server. The local brewer files and all relative information (file size, modification date etc.) are listed and compared to the corresponding information for the files that have been already stored in the server and only the files which are either new or have changed, are transferred from the former to the later, checking md5 checksums ([[https://www.google.com/url?q=https://en.wikipedia.org/wiki/MD5&sa=D&ust=1521821549668000&usg=AFQjCNFlov60DgO4G4ZFDaShmk2oFVtqRw|https://en.wikipedia.org/wiki/MD5]]). Each file transferred to the server is stored in the input directory corresponding to each brewer. Both the client and server programs are implemented in python. Communication between client and server is achieved via the xml-rpc protocol ([[https://www.google.com/url?q=https://en.wikipedia.org/wiki/XML-RPC&sa=D&ust=1521821549669000&usg=AFQjCNFscykefnoFsMnMw76Pxs9PBEzh1g|https://en.wikipedia.org/wiki/XML-RPC]]). | ||
| - | - Import module | + | ==== Import module ==== |
| - | This module is the basis of the entire project. It is responsible for verifying the validity of each file obtained in the acquisition module and then store the desired information in the database. This module consists by four sub-modules described above. A schematic diagram of this process is shown in Figure 2. | + | This module is the basis of the entire project. It is responsible for verifying the validity of each file obtained in the acquisition module and then store the desired information in the database. This module consists by four sub-modules described above. A schematic diagram of this process is shown in **Figure 2**. |
| - | {{images:image7.png|import_module.png}}Figure 2: Schematic diagram of the import Module | + | {{ :codes:image2.png |}}**Figure 2:** Schematic diagram of the import Module |
| * Import module: Importing files is a periodic task, which checks the input by all the Brewer directories. The appropriate parser is called for each Brewer file, the status of the file is stored and then it is moved to its final location. The master plan (implemented as periodic task) (projects/data/bin/import_all.py) is implemented in python. The program that imports the files for each individual Brewer is implemented in perl (projects/data/bin/import.pl). | * Import module: Importing files is a periodic task, which checks the input by all the Brewer directories. The appropriate parser is called for each Brewer file, the status of the file is stored and then it is moved to its final location. The master plan (implemented as periodic task) (projects/data/bin/import_all.py) is implemented in python. The program that imports the files for each individual Brewer is implemented in perl (projects/data/bin/import.pl). | ||
| Line 77: | Line 80: | ||
| * Data archiving: Different file types contain different types of measurements. Some files contain a unique type of measurements (e.g. UV files) while other (e.g. B files) contain multiple types of measurements. There is a file structure definition routine for every file type (in: projects/data/include/defs/) which defines how the different types of measurements are written in each specific file type. Then, After applying this routine, another routine (projects/iberonesia3/modules/iberonesia3/database.py) contains proper functions which store the measurements into tables, each of which contains a specific type of measurements. When a file is correct and its content is inserted into the database successfully, it moves to the directory tree of data files. Otherwise, it moves to a defined directory error, such that the operator can decide what to do with it. | * Data archiving: Different file types contain different types of measurements. Some files contain a unique type of measurements (e.g. UV files) while other (e.g. B files) contain multiple types of measurements. There is a file structure definition routine for every file type (in: projects/data/include/defs/) which defines how the different types of measurements are written in each specific file type. Then, After applying this routine, another routine (projects/iberonesia3/modules/iberonesia3/database.py) contains proper functions which store the measurements into tables, each of which contains a specific type of measurements. When a file is correct and its content is inserted into the database successfully, it moves to the directory tree of data files. Otherwise, it moves to a defined directory error, such that the operator can decide what to do with it. | ||
| - | - Product module | + | ==== Product module ==== |
| The product generation scripts (in: projects/iberonesia3/modules/iberonesia3/) process the data which are stored in the database and generate specific products. They are currently implemented in python. The products currently generated are: | The product generation scripts (in: projects/iberonesia3/modules/iberonesia3/) process the data which are stored in the database and generate specific products. They are currently implemented in python. The products currently generated are: | ||
| Line 85: | Line 88: | ||
| * The Aerosol Optical Depth (AOD) product | * The Aerosol Optical Depth (AOD) product | ||
| - | - Web module | + | ==== Web module ==== |
| The website can be used by the users to perform several tasks related to the database. There are two versions of the website. In the version which is available to the public it is allowed to check the status of the brewers (e.g. brewer activity and status of the B file of the current day) and get any products they want. In the other version which is for registered users, it is also possible to download files, get reports for possible problems of the instruments or consult/manage checklists and charts to obtain the status of the brewers. The web module is implemented in Python, using web2py as a development framework. | The website can be used by the users to perform several tasks related to the database. There are two versions of the website. In the version which is available to the public it is allowed to check the status of the brewers (e.g. brewer activity and status of the B file of the current day) and get any products they want. In the other version which is for registered users, it is also possible to download files, get reports for possible problems of the instruments or consult/manage checklists and charts to obtain the status of the brewers. The web module is implemented in Python, using web2py as a development framework. | ||
| - | 3. Interface | + | ---- |
| - | The interface of the Eubrewnet webpage is presented in the following figures. An interactive map shows the stations which provide data by one or more Brewers. The user can choose a station either from the map (Figure 3) or from a list (Figure 4). As already explained, registered users have more options. | ||
| - | | {{images:image5.png}} || {{images:image8.png}} |Figure 3: The interactive map of the Eubrewnet webpage for (a) non-registered and (b) registered users. | + | ===== 3. Interface ===== |
| - | {{images:image3.png}}Figure 4:The list of Brewers. | + | The interface of the Eubrewnet webpage is presented in the following figures. An interactive map shows the stations which provide data by one or more Brewers. The user can choose a station either from the map (**Figure 3**) or from a list (**Figure 4**). As already explained, registered users have more options. |
| - | When a station has been selected, then additional options and information appear for the particular station and instruments (Figure 5). | + | | {{ :codes:image5.png |}} || {{:codes:image8.png}} | |
| + | **Figure 3:** The interactive map of the Eubrewnet webpage for (a) non-registered and (b) registered users. | ||
| - | {{images:image6.png}}Figure 5:Information for the Brewer operating at Thessaloniki. | ||
| - | 4. Products | + | {{:codes:image3.png}}**Figure 4:** The list of Brewers. |
| + | |||
| + | When a station has been selected, then additional options and information appear for the particular station and instruments (**Figure 5**). | ||
| + | |||
| + | {{:codes:image6.png}}**Figure 5:** Information for the Brewer operating at Thessaloniki. | ||
| + | |||
| + | ---- | ||
| + | |||
| + | |||
| + | ===== 4. Products ===== | ||
| As already discussed, three products are available: TOC, UV and AOD. The generation of the TOC product is already operative while the generation of the UV and AOD products is still under development. There are three different levels of the products (1.0, 1.5 and 2.0). Levels 1.5 and 2.0 are available to the public, while level 1.0 is available only to the registered users. Raw data (which could be described as Level 0) are stored in the database, though graphical information is not available through the web page. Though, registered users can download raw data using proper functions (see Section 5.5). In general, a higher product level means a more reliable product. | As already discussed, three products are available: TOC, UV and AOD. The generation of the TOC product is already operative while the generation of the UV and AOD products is still under development. There are three different levels of the products (1.0, 1.5 and 2.0). Levels 1.5 and 2.0 are available to the public, while level 1.0 is available only to the registered users. Raw data (which could be described as Level 0) are stored in the database, though graphical information is not available through the web page. Though, registered users can download raw data using proper functions (see Section 5.5). In general, a higher product level means a more reliable product. | ||
| Line 107: | Line 118: | ||
| There is also an option to have different versions of the same products in the database. This way, a new version of the products can be added after e.g. re-evaluation or slightly different processing of the dataset, without erasing the previous one. | There is also an option to have different versions of the same products in the database. This way, a new version of the products can be added after e.g. re-evaluation or slightly different processing of the dataset, without erasing the previous one. | ||
| - | 4.1. The TOC product | + | ==== 4.1. The TOC product ==== |
| The TOC is calculated by the direct sun (DS) Brewer measurements which have been processed using the standard Brewer algorithm (Kipp & Zonnen 2008). In order to obtain a reliable product with uncertainties within the desirable uncertainty limits, proper calibration and correction constants have to be used for the processing of the DS measurements. These constants are provided by the operator and stored in a proper configuration table. A detailed list of the configuration parameters which have to be declared by the operator can be found at the following link: | The TOC is calculated by the direct sun (DS) Brewer measurements which have been processed using the standard Brewer algorithm (Kipp & Zonnen 2008). In order to obtain a reliable product with uncertainties within the desirable uncertainty limits, proper calibration and correction constants have to be used for the processing of the DS measurements. These constants are provided by the operator and stored in a proper configuration table. A detailed list of the configuration parameters which have to be declared by the operator can be found at the following link: | ||
| - | [[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Ddevel:eubrewnetconfiguration&sa=D&ust=1521821549674000&usg=AFQjCNHAHUzoG_Gt8sf3jr-2YLGlyXePWw|http://rbcce.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Ddevel:eubrewnetconfiguration&sa=D&ust=1521821549674000&usg=AFQjCNHAHUzoG_Gt8sf3jr-2YLGlyXePWw|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration]] |
| while an overview of the processes which can be used by the operators to fill the configuration table is provided here: | while an overview of the processes which can be used by the operators to fill the configuration table is provided here: | ||
| - | [[https://www.google.com/url?q=http://www.eubrewnet.org/cost1207/2015/03/04/configuration-upload-intructions/&sa=D&ust=1521821549675000&usg=AFQjCNEFyPTScHJ3-JKSlH5Z8tPS0Vj3Bw|http://www.eubrewnet.org/cost1207/2015/03/04/configuration-upload-intructions/]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/cost1207/2015/03/04/configuration-upload-intructions/&sa=D&ust=1521821549675000&usg=AFQjCNEFyPTScHJ3-JKSlH5Z8tPS0Vj3Bw|http://eubrewnet.aemet.es/cost1207/2015/03/04/configuration-upload-intructions/]] |
| - | In the Eubrewenet wiki there is also analytical information regarding the processing ([[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Dcodes:ozoneproduct&sa=D&ust=1521821549675000&usg=AFQjCNGDUWukRByqSYFlSayczHizMqs4pQ|http://rbcce.aemet.es/dokuwiki/doku.php?id=codes:ozoneproduct]]) and the storage ([[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Dcodes:productsdescription&sa=D&ust=1521821549676000&usg=AFQjCNE0xKYX11zmOFrvboo2mDyU83_57Q|http://rbcce.aemet.es/dokuwiki/doku.php?id=codes:productsdescription]]) of the different levels of the TOC. In the following there is a short description of the four different levels of the TOC product: | + | In the Eubrewenet wiki there is also analytical information regarding the processing ([[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Dcodes:ozoneproduct&sa=D&ust=1521821549675000&usg=AFQjCNGDUWukRByqSYFlSayczHizMqs4pQ|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=codes:ozoneproduct]]) and the storage ([[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Dcodes:productsdescription&sa=D&ust=1521821549676000&usg=AFQjCNE0xKYX11zmOFrvboo2mDyU83_57Q|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=codes:productsdescription]]) of the different levels of the TOC. In the following there is a short description of the four different levels of the TOC product: |
| * level 0: The TOC coming directly from the Brewers.These values have been calculated from the Direct Sun measurements using the constants from the configuration file which is stored in the client (Brewer) pc. | * level 0: The TOC coming directly from the Brewers.These values have been calculated from the Direct Sun measurements using the constants from the configuration file which is stored in the client (Brewer) pc. | ||
| Line 124: | Line 135: | ||
| * level 2.0: The level 1.5 TOC validated with a posterior calibration. | * level 2.0: The level 1.5 TOC validated with a posterior calibration. | ||
| - | 4.2. The UV product | + | ==== 4.2. The UV product ==== |
| The generation of the UV product is still under development. Though, a general description of the products of different levels that will be available is the following: | The generation of the UV product is still under development. Though, a general description of the products of different levels that will be available is the following: | ||
| Line 141: | Line 152: | ||
| More information regarding the UV product can be found in Lakkala et al. (2016) | More information regarding the UV product can be found in Lakkala et al. (2016) | ||
| - | 4.3. The AOD product | + | ==== 4.3. The AOD product ==== |
| The AOD product is also under development. In this case the processing procedures for the products of different levels are still under discussion. The calculation of the AOD is based on the DS measurements and is performed for the same wavelengths used for the calculation of the AOD (~306, 310, 313, 317 and 320 nm ). The operator has to provide the file of the AOD calibration. More information for the retrieval and the state of development of the AOD product can be found in Solano et al. (2016). | The AOD product is also under development. In this case the processing procedures for the products of different levels are still under discussion. The calculation of the AOD is based on the DS measurements and is performed for the same wavelengths used for the calculation of the AOD (~306, 310, 313, 317 and 320 nm ). The operator has to provide the file of the AOD calibration. More information for the retrieval and the state of development of the AOD product can be found in Solano et al. (2016). | ||
| - | 4.4. Configuration versioning | + | ==== 4.4. Configuration versioning ==== |
| Since the UV and AOD products are under development, it is not yet possible to update the configuration constants for them. Currently the operators are able to provide the configuration constants for the station (location, name of the operator etc.) and the TOC product (calibration constants, temperature coefficients etc.). More detailed information regarding the configuration constants can be found at: | Since the UV and AOD products are under development, it is not yet possible to update the configuration constants for them. Currently the operators are able to provide the configuration constants for the station (location, name of the operator etc.) and the TOC product (calibration constants, temperature coefficients etc.). More detailed information regarding the configuration constants can be found at: | ||
| - | [[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Ddevel:eubrewnetconfiguration%23brewer_configuration_constants&sa=D&ust=1521821549678000&usg=AFQjCNHmymYyBfT1KVobElXc3sFglIRJkw|http://rbcce.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration#brewer_configuration_constants]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Ddevel:eubrewnetconfiguration%23brewer_configuration_constants&sa=D&ust=1521821549678000&usg=AFQjCNHmymYyBfT1KVobElXc3sFglIRJkw|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration#brewer_configuration_constants]] |
| - | The TOC configuration constants have to be updated after a new calibration of the instrument (usually every ~2 years) or after important changes in its operational characteristics. Occasionally, update of the station configuration constants may be also necessary. Proper tools have been developed and are available to the users so that they can update the configuration constants in an easy way ([[https://www.google.com/url?q=http://www.eubrewnet.org/cost1207/2015/03/04/configuration-upload-intructions/&sa=D&ust=1521821549679000&usg=AFQjCNFwHvVISTkrx1wv97UvCKoAaMBtNw|http://www.eubrewnet.org/cost1207/2015/03/04/configuration-upload-intructions/]]). A flowchart of the TOC configuration change procedure is shown in Figure 6, while in Figure 7 the interface through which the operator can update the TOC configuration constants is presented. | + | The TOC configuration constants have to be updated after a new calibration of the instrument (usually every ~2 years) or after important changes in its operational characteristics. Occasionally, update of the station configuration constants may be also necessary. Proper tools have been developed and are available to the users so that they can update the configuration constants in an easy way ([[https://www.google.com/url?q=http://eubrewnet.aemet.es/cost1207/2015/03/04/configuration-upload-intructions/&sa=D&ust=1521821549679000&usg=AFQjCNFwHvVISTkrx1wv97UvCKoAaMBtNw|http://eubrewnet.aemet.es/cost1207/2015/03/04/configuration-upload-intructions/]]). A flowchart of the TOC configuration change procedure is shown in **Figure 6**, while in **Figure 7** the interface through which the operator can update the TOC configuration constants is presented. |
| - | {{images:image2.png}}Figure 6: Generalized flowchart of the TOC configuration change procedure | + | {{:codes:image2.png}}**Figure 6:** Generalized flowchart of the TOC configuration change procedure |
| - | {{images:image1.png}}Figure 7: Interface of the TOC configuration tool | + | {{:codes:image1.png}}**Figure 7:** Interface of the TOC configuration tool |
| Different sets of configuration constants correspond to different periods (Figure 7). When a user is logged in, he/she should select whether he/she wants to modify an existing set of configuration constants or to add a new one (for a new period). Then the user can either upload a file with all the needed information or change the information he/she wants by hand. The corresponding tools and procedures for UV and AOD will be more complicated since a larger amount of information has to be provided by the user. More information regarding the configuration versioning can be found in: | Different sets of configuration constants correspond to different periods (Figure 7). When a user is logged in, he/she should select whether he/she wants to modify an existing set of configuration constants or to add a new one (for a new period). Then the user can either upload a file with all the needed information or change the information he/she wants by hand. The corresponding tools and procedures for UV and AOD will be more complicated since a larger amount of information has to be provided by the user. More information regarding the configuration versioning can be found in: | ||
| - | [[https://www.google.com/url?q=http://www.eubrewnet.org/cost1207/2015/03/04/configuration-upload-intructions/&sa=D&ust=1521821549681000&usg=AFQjCNEsK_pvY8Z2tv_OMvzx4iZWlucbdg|http://www.eubrewnet.org/cost1207/2015/03/04/configuration-upload-intructions/]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/cost1207/2015/03/04/configuration-upload-intructions/&sa=D&ust=1521821549681000&usg=AFQjCNEsK_pvY8Z2tv_OMvzx4iZWlucbdg|http://eubrewnet.aemet.es/cost1207/2015/03/04/configuration-upload-intructions/]] |
| - | 4.5. Dissemination: Files, Access functions | + | ==== 4.5. Dissemination: Files, Access functions ==== |
| - | Access functions | + | === Access functions === |
| - | A number of functions have been developed, which allow the registered users to directly get the information stored in the database (i.e. products of different levels, raw data, results from diagnostic tests etc). These functions are written in python and are included in the module getdata.py (projects/iberonesia3/modules/iberonesia3/getdata.py). The root to all access functions is the same: [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/&sa=D&ust=1521821549682000&usg=AFQjCNHcU_R2oU7xzhltPL3ZqSdNBYas4w|http://rbcce.aemet.es/eubrewnet/data/]]. From this point, users can select the proper function and get the data they need . The functions provide four different ways of accessing data using one of the format input parameters, described in Table 1: | + | A number of functions have been developed, which allow the registered users to directly get the information stored in the database (i.e. products of different levels, raw data, results from diagnostic tests etc). These functions are written in python and are included in the module getdata.py (projects/iberonesia3/modules/iberonesia3/getdata.py). The root to all access functions is the same: [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/&sa=D&ust=1521821549682000&usg=AFQjCNHcU_R2oU7xzhltPL3ZqSdNBYas4w|http://eubrewnet.aemet.es/eubrewnet/data/]]. From this point, users can select the proper function and get the data they need . The functions provide four different ways of accessing data using one of the format input parameters, described in Table 1: |
| - | Table 1: Possible ways of accessing the Eubrewnet data using access functions and links for examples for the Get DS function. | + | **Table 1:** Possible ways of accessing the Eubrewnet data using access functions and links for examples for the Get DS function. |
| | Format input | Description | Example | | | Format input | Description | Example | | ||
| - | | jsonM | JSON matrix formed by lists of lists (default value). The first list is formed by the parameters’ names and the following lists are the query outputs | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01&sa=D&ust=1521821549685000&usg=AFQjCNHVEw324GC1wV0ZNn8MF4-mzRfMww|Get DS by default]] | | + | | jsonM | JSON matrix formed by lists of lists (default value). The first list is formed by the parameters’ names and the following lists are the query outputs | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01&sa=D&ust=1521821549685000&usg=AFQjCNHVEw324GC1wV0ZNn8MF4-mzRfMww|Get DS by default]] | |
| - | | jsonO | JSON object of lists, where each key is the parameter name and its value is the time sorted list of them | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3DjsonO&sa=D&ust=1521821549686000&usg=AFQjCNEIhq8t58zVoHyfOr3SdbHS9hJhyA|Get DS with jsonO format]] | | + | | jsonO | JSON object of lists, where each key is the parameter name and its value is the time sorted list of them | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3DjsonO&sa=D&ust=1521821549686000&usg=AFQjCNEIhq8t58zVoHyfOr3SdbHS9hJhyA|Get DS with jsonO format]] | |
| - | | text | Human readable, comma separated values where first row is the parameters’ names and the following are the query outputs | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dtext&sa=D&ust=1521821549687000&usg=AFQjCNFPE93Th3fBum6fQ_1Owy-zlJEERQ|Get DS with text format]] | | + | | text | Human readable, comma separated values where first row is the parameters’ names and the following are the query outputs | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dtext&sa=D&ust=1521821549687000&usg=AFQjCNFPE93Th3fBum6fQ_1Owy-zlJEERQ|Get DS with text format]] | |
| - | | csv | CSV direct download where in first row are the parameters’ names and the following are the query outputs | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dcsv&sa=D&ust=1521821549688000&usg=AFQjCNHCIVCaIsb4aY5nT3IreMtZ1w6yaQ|Get DS with text format]] | | + | | csv | CSV direct download where in first row are the parameters’ names and the following are the query outputs | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dcsv&sa=D&ust=1521821549688000&usg=AFQjCNHCIVCaIsb4aY5nT3IreMtZ1w6yaQ|Get DS with text format]] | |
| Almost all functions can receive the inputs listed in Table 2: | Almost all functions can receive the inputs listed in Table 2: | ||
| - | Table 2: Inputs of the access functions and links for examples for the Get DS function. | + | **Table 2:** Inputs of the access functions and links for examples for the Get DS function. |
| | Parameter | Description | Example | | | Parameter | Description | Example | | ||
| - | | brewerid | Brewer identification number | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dtext&sa=D&ust=1521821549691000&usg=AFQjCNF6EXr9Y4Mi0ARBEa3sh8Hm1msv1Q|Get function with brewerid input parameter]] | | + | | brewerid | Brewer identification number | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dtext&sa=D&ust=1521821549691000&usg=AFQjCNF6EXr9Y4Mi0ARBEa3sh8Hm1msv1Q|Get function with brewerid input parameter]] | |
| - | | date | Date in YYYY-MM-DD format | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dtext&sa=D&ust=1521821549692000&usg=AFQjCNH_V6efcLmrtClXaNr-mknPyeuWPw|Get function with date input parameter]] | | + | | date | Date in YYYY-MM-DD format | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26format%3Dtext&sa=D&ust=1521821549692000&usg=AFQjCNH_V6efcLmrtClXaNr-mknPyeuWPw|Get function with date input parameter]] | |
| - | | enddate | Date in YYYY-MM-DD format. If provided, the function will return the query in a date range | [[https://www.google.com/url?q=http://rbcce.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26enddate%3D2015-01-05%26format%3Dtext&sa=D&ust=1521821549694000&usg=AFQjCNFzWAYRqpk8U67IsKhbwdDmWB3FIg|Get function with enddate input parameter]] | | + | | enddate | Date in YYYY-MM-DD format. If provided, the function will return the query in a date range | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/eubrewnet/data/get/DS?brewerid%3D157%26date%3D2015-01-01%26enddate%3D2015-01-05%26format%3Dtext&sa=D&ust=1521821549694000&usg=AFQjCNFzWAYRqpk8U67IsKhbwdDmWB3FIg|Get function with enddate input parameter]] | |
| More analytical information regarding the access functions can be found in: | More analytical information regarding the access functions can be found in: | ||
| - | [[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Dcodes:dbaccess&sa=D&ust=1521821549694000&usg=AFQjCNHHwy0JY5Q7h9wGi5jJc2xQvXzdwA|http://rbcce.aemet.es/dokuwiki/doku.php?id=codes:dbaccess]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Dcodes:dbaccess&sa=D&ust=1521821549694000&usg=AFQjCNHHwy0JY5Q7h9wGi5jJc2xQvXzdwA|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=codes:dbaccess]] |
| - | Brewer Python Library | + | === Brewer Python Library === |
| A library for Brewer data processing has been developed using Python and has been integrated in the Eubrewnet database. The standard algorithm for Brewer data processing is based on the version 4.1 of the brewer software ([[https://www.google.com/url?q=http://www.o3soft.eu/&sa=D&ust=1521821549695000&usg=AFQjCNGLPSLCsiKIHKS7y6mFce0-TnLtpQ|http://www.o3soft.eu/]]), written in GWBASIC, and the Brewer MkIII - Operator's Manual (Kipp & Zonen, 2008), available through the Kipp & Zonen web page ([[https://www.google.com/url?q=http://www.kippzonen.com&sa=D&ust=1521821549695000&usg=AFQjCNFKMYfuANyrCZkAzwvqsz2aDyWRNg|http://www.kippzonen.com]]). Analytical documentation for the library, including instructions for its implementation and usage is published on the EUBREWNET documentation page: | A library for Brewer data processing has been developed using Python and has been integrated in the Eubrewnet database. The standard algorithm for Brewer data processing is based on the version 4.1 of the brewer software ([[https://www.google.com/url?q=http://www.o3soft.eu/&sa=D&ust=1521821549695000&usg=AFQjCNGLPSLCsiKIHKS7y6mFce0-TnLtpQ|http://www.o3soft.eu/]]), written in GWBASIC, and the Brewer MkIII - Operator's Manual (Kipp & Zonen, 2008), available through the Kipp & Zonen web page ([[https://www.google.com/url?q=http://www.kippzonen.com&sa=D&ust=1521821549695000&usg=AFQjCNFKMYfuANyrCZkAzwvqsz2aDyWRNg|http://www.kippzonen.com]]). Analytical documentation for the library, including instructions for its implementation and usage is published on the EUBREWNET documentation page: | ||
| - | [[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Ddevel:brewerpythonmodule&sa=D&ust=1521821549696000&usg=AFQjCNG19IjvoLsCglqwNdzhZianNxt25Q|http://rbcce.aemet.es/dokuwiki/doku.php?id=devel:brewerpythonmodule]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Ddevel:brewerpythonmodule&sa=D&ust=1521821549696000&usg=AFQjCNG19IjvoLsCglqwNdzhZianNxt25Q|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=devel:brewerpythonmodule]] |
| The last version of the brewer python module is available at the following link: | The last version of the brewer python module is available at the following link: | ||
| Line 202: | Line 213: | ||
| [[https://www.google.com/url?q=http://rbcce.aemet.es/svn/python/libbrewerjson/&sa=D&ust=1521821549697000&usg=AFQjCNEj0ttz9BDPg3hnaNZBohUosYkXRw|http://rbcce.aemet.es/svn/python/libbrewerjson/]] | [[https://www.google.com/url?q=http://rbcce.aemet.es/svn/python/libbrewerjson/&sa=D&ust=1521821549697000&usg=AFQjCNEj0ttz9BDPg3hnaNZBohUosYkXRw|http://rbcce.aemet.es/svn/python/libbrewerjson/]] | ||
| - | 4.6. QA/QC | + | ==== 4.6. QA/QC ==== |
| The Brewer diagnostic tool has been created to assist the Quality Control and Quality Assurance (QA/QC) of the measurements stored in the Eubrewnet database. The main component of this tool is a parser which extracts information regarding possible errors from the B and D files received from the system and stores it in status files and the database. Up to today, the error detection is based on the error messages contained in the B and D files. Most types of error messages written in these files (48 different error messages) can be detected. However, there are features of the diagnostic tool that are still under development. In the future the detection of errors will be also based on a range of acceptable values for each type of Brewer measurements. Depending on the type of measurements, proper thresholds will be specified, above or below which the measurement will be considered problematic. It is also planned to include more error messages that can be detected in the B and D files. | The Brewer diagnostic tool has been created to assist the Quality Control and Quality Assurance (QA/QC) of the measurements stored in the Eubrewnet database. The main component of this tool is a parser which extracts information regarding possible errors from the B and D files received from the system and stores it in status files and the database. Up to today, the error detection is based on the error messages contained in the B and D files. Most types of error messages written in these files (48 different error messages) can be detected. However, there are features of the diagnostic tool that are still under development. In the future the detection of errors will be also based on a range of acceptable values for each type of Brewer measurements. Depending on the type of measurements, proper thresholds will be specified, above or below which the measurement will be considered problematic. It is also planned to include more error messages that can be detected in the B and D files. | ||
| Line 208: | Line 219: | ||
| The B and D files are processed three minutes after their receival. Given that files are received every 15 minutes, the detection of errors is performed in almost real time. As soon as an error is detected, the color of the corresponding instrument shown in the Eubrewnet Interface (for registered users) changes depending on the type of error. More specifically, the different colors used to describe the state of each Brewer are the following: | The B and D files are processed three minutes after their receival. Given that files are received every 15 minutes, the detection of errors is performed in almost real time. As soon as an error is detected, the color of the corresponding instrument shown in the Eubrewnet Interface (for registered users) changes depending on the type of error. More specifically, the different colors used to describe the state of each Brewer are the following: | ||
| - | * RED: CRITICAL | + | * **RED:** CRITICAL |
| - | * ORANGE: ERROR | + | * **ORANGE:** ERROR |
| - | * YELLOW: WARNING | + | * **YELLOW:** WARNING |
| - | * GREEN: LATEST INFO RECEIVED TODAY | + | * **GREEN:** LATEST INFO RECEIVED TODAY |
| - | * GREY: LATEST INFO RECEIVED YESTERDAY | + | * **GREY:** LATEST INFO RECEIVED YESTERDAY |
| - | * BLACK: LATEST INFO RECEIVED FIFTEEN DAYS AGO | + | * **BLACK:** LATEST INFO RECEIVED FIFTEEN DAYS AGO |
| When the user clicks on a brewer, then he is redirected to a page with details regarding the state of the specific instrument, where a list of the original error messages from the D or B files are also included. More details regarding the Brewer diagnostic tool can be found to the following link: | When the user clicks on a brewer, then he is redirected to a page with details regarding the state of the specific instrument, where a list of the original error messages from the D or B files are also included. More details regarding the Brewer diagnostic tool can be found to the following link: | ||
| - | [[https://www.google.com/url?q=http://rbcce.aemet.es/dokuwiki/doku.php?id%3Dcodes:brewerdiagnostictool&sa=D&ust=1521821549698000&usg=AFQjCNEX7e1do1VLs4HMWfCZ-Lx6I44s3Q|http://rbcce.aemet.es/dokuwiki/doku.php?id=codes:brewerdiagnostictool]] | + | [[https://www.google.com/url?q=http://eubrewnet.aemet.es/dokuwiki/doku.php?id%3Dcodes:brewerdiagnostictool&sa=D&ust=1521821549698000&usg=AFQjCNEX7e1do1VLs4HMWfCZ-Lx6I44s3Q|http://eubrewnet.aemet.es/dokuwiki/doku.php?id=codes:brewerdiagnostictool]] |
| - | References | + | ---- |
| + | |||
| + | ===== References ===== | ||
| Antón, M., Serrano, A., Cancillo, M. L., Vilaplana, J., Cachorro, V. E., and Gröbner, J.: Correction of Angular Response Error in Brewer UV Irradiance Measurements, Journal of Atmospheric and Oceanic Technology, 25, 2018-2027, 2008. | Antón, M., Serrano, A., Cancillo, M. L., Vilaplana, J., Cachorro, V. E., and Gröbner, J.: Correction of Angular Response Error in Brewer UV Irradiance Measurements, Journal of Atmospheric and Oceanic Technology, 25, 2018-2027, 2008. | ||
| Line 238: | Line 251: | ||
| López-Solano J., Carlund T., Hernández-Cruz B., León-Luis S.F., Carreño V., Berjón A., Santana-Díaz D., Bais A.F., Moreta J.R., Shirotov V., Rimmer J., Vilaplana J.M., Wilson K.M., Gröbner J., Boulkelia L., Eriksen P., Karppinen T., Diémoz H., Rodríguez Valido M., Redondas A.: “Aerosol optical depth in the ultraviolet range: a new product in EUBREWNET”, Quadrennial Ozone Symposium 2016 (QOS 2016), Edinburg, UK, 4 - 9 September, 2016\\ \\ Slaper, H., Reinen, H. A. J. M., Blumthaler, M., Huber, M., and Kuik, F.: Comparing ground-level spectrally resolved solar UV measurements using various instruments: A technique resolving effects of wavelength shift and slit width, Geophysical Research Letters, 22, 2721-2724, 1995. | López-Solano J., Carlund T., Hernández-Cruz B., León-Luis S.F., Carreño V., Berjón A., Santana-Díaz D., Bais A.F., Moreta J.R., Shirotov V., Rimmer J., Vilaplana J.M., Wilson K.M., Gröbner J., Boulkelia L., Eriksen P., Karppinen T., Diémoz H., Rodríguez Valido M., Redondas A.: “Aerosol optical depth in the ultraviolet range: a new product in EUBREWNET”, Quadrennial Ozone Symposium 2016 (QOS 2016), Edinburg, UK, 4 - 9 September, 2016\\ \\ Slaper, H., Reinen, H. A. J. M., Blumthaler, M., Huber, M., and Kuik, F.: Comparing ground-level spectrally resolved solar UV measurements using various instruments: A technique resolving effects of wavelength shift and slit width, Geophysical Research Letters, 22, 2721-2724, 1995. | ||
| + | |||
codes/eubrewnet_report.1521818486.txt.gz · Last modified: 2018/03/23 15:21 by ilias
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
