Differences

This shows you the differences between two versions of the page.

--- namespace:page [2018/03/21 14:24] – ilias
+++ namespace:page [Unknown date] (current) – removed - external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
-======= Eubrewnetuser’s manual =======
-----
-====== 1.    Setup and data sending ======
-Each Brewer spectrophotometer communicates with a local PC where the raw data are recorded. For uploading the raw data from the client PC to the Eubrewnet database a set of routines written in python has to be installed on the local PC (or any other computer which has access to the Brewer raw data). When these routines are executed the local (client) PC is connected to the Eubrewnet server. Then, the local brewer files and all relative information (file size, modification date etc.) are listed and compared to the corresponding information for the files that have been already stored in the server. Only the files which are either new or have changed, are transferred from the former to the later. This process is repeated in regular time intervals (prefferably ~15 mins). In sections 1.1 - 1.4, the software installation and application processes are analytically described. A fast installation guide is available in section 1.5.
-===== 1.1.         Software installation =====
-Before the installation of the software the operator has to communicate with the [[eubrewnet@aemet.es|administrator]](eubrewnet@aemet.es) and ensure that his Brewer has been included in the[[http://rbcce.aemet.es/eubrewnet/brewer/list|Eubrewnet]][[http://rbcce.aemet.es/eubrewnet/brewer/list| Brewer list]]. Then, there are two possible methods to install the client software:
-__1<sup>st</sup>  method____(recommended): Install the python interpreter and download the python client__
-For Windows and Mac, Python 2.7 is recommended while for Linux, the Python Version should come bundled with the Linux distribution. Below is a list of the appropriate interpreters for Windows and Mac.
-●       [[https://www.python.org/ftp/python/2.7/python-2.7.msi|Python 2.7 Windows x86 MSI Installer]] (Windows binary)
-●       [[https://www.python.org/ftp/python/2.7/python-2.7.amd64.msi|Python 2.7 Windows X86-64 MSI Installer]](Windows AMD64 / Intel 64 /X86-64 binary)
-●       [[https://www.python.org/ftp/python/2.7/python-2.7-macosx10.5.dmg|Python 2.7 Mac OS X 64-bit/32-bit x86-64/i386 Installer]] (for Mac OS X 10.5 and later, if needed)
-●       [[https://www.python.org/ftp/python/2.7/python-2.7-macosx10.3.dmg|Python 2.7 Mac OS X 32-bit i386/PPC Installer]](for Mac OS X 10.3 and later, if needed)
-●       Linux Version should come bundled with your Linux distribution.
-You have to be sure that your python interpreter is callable from every path in your system (Open a console and try **python**, **exit(****)** for exit).
-Then, [[http://rbcce.aemet.es/eubrewnet/static/files/client_python.zip|client_python.zip]]should be downloaded and uncompressed in the system.
-__2<sup>nd</sup>  method____(not fully tested): Download the exe client__
-In this case, the [[http://rbcce.aemet.es/eubrewnet/static/files/client_exe.zip|client_exe.zip]] should be downloaded and uncompressed in the client PC system. After unzipping the compressed folder, try: **$>refresh.exe**
- (in dist folder) in a console to test if client is working properly. No error message should appear.
-===== 1.2.         Client configuration file =====
-The client configuration file (client.ini) is used by the refresh.py program. The later is the routine used to transfer the files from the client PC to the database. Depending on the installation method, these two files can be found either in client_python\source (1st installation method) or in client_exe\dist (2nd installation method). Client.ini is a plain ASCII [[https://en.wikipedia.org/wiki/INI_file|INI]] file which consists of a global section and several server sections (one section for each server to which the files are sent).
-In the following example, the content of a client.ini file is presented. The lines in bold should not be changed by the operator since they provide information relative to the database server. The information in the remaining (regular font) lines should be defined by the operator.
-| **Example 1:** The content of a client.ini file\\   **[global]**\\    brewerid=157\\    working_dir=.\\    input=c:\brewer\data\\ **   servers=iberonesia3**\\    proxy=proxy.xxx.xxx:port\\    **[iberonesia3]**\\ **   url=****[[http://rbcce.aemet.es/refresh|http://rbcce.aemet.es/refresh]]**\\    db=iberonesia3.db\\    noproxy=yes\\ **   serverport=8081**   |
-In the **global section** of the client.ini file the following parameters should be declared:
-●       **brewerid** : brewer identifier (i.e. the brewer serial number).
-●       **input**: the directory which contains the files produced by the Brewer.
-●       **servers**: comma separated list of server aliases. The files are sent to each server in this list.
-●       **working_dir**: the directory where the local refresh database is stored.
-●       **proxy**: proxy url, if required. If not required, this property should be commented (add # at the beginning of the line).
-A **server section** is identified by the server alias (i.e. the alias is the name between the brackets - iberonesia3 in the case of Example 1), and has the following properties:
-●       **url** : server url.
-●       **db** : a local database (a simple data file) is created the first time that client software is executed. This line defines the name of this file.
-●       **noproxy** : if a proxy is specified in the global section, noproxy may be set to yes to use a direct connection (avoiding the proxy) to access the server.
-●       **serverport**: the used port in the server (8081)
-The lines that are not used should be commented (i.e. the proxy property if no proxy is used).  The easiest way to configure the client is to properly specify all options in the client.ini file and then execute refresh.py without arguments. However, if desired, some of the above properties can be overridden using command line parameters when executing refresh.py or alive.py. If for example, the client.ini file of Example 1 is used, a different Brewer id and a different input path can be defined in the command line:
-**  $>python refresh.py -b 086 -i c:\brewer086\data******
-Though, it is preferable to specify the Brewer id and the input path (as well as any other parameters) in the client.ini file instead of using the **-b** and **-i** commands.
-As already described, most of the refresh.py options may be configured though command line arguments or client.ini configuration file. Any option specified in command line overrides the option in client.ini. All options are shown when the the command ** “****$ ./refresh.py -h”** is used (see Example 2):
-| **Example 2:** The refresh.py options\\ **  $ ./refresh.py -h**\\ \\    Usage: refresh.py [options]\\ \\    Options:\\      -h, --help            show this help message and exit\\      -i FILE, --input=FILE\\                            File to process. If the given file is a directory, all\\                            files within will be processed.\\      -s SERVERS, --servers=SERVERS\\                            Comma separated list of xmlrpc server aliases\\      -b BREWERID, --brewerid=BREWERID\\                            Brewerid of file/s to process\\      -c FILE, --config=FILE\\                            Configuration file path\\      -v, --verbose         Verbosity level. The more '-v' the more verbose is the\\                            program.\\ \\ |
-The mandatory options are:
-●       **working_dir** ****:This option must be specified in client.ini. It defines the directory where the local refresh database is stored.
-●       **input**: Defines the file which will be processed. If a directory is defined, then all files in the particular directory will be processed.
-●       **brewerid** : Brewer identifier.
-●       **servers**: Comma separated list of server aliases. The files are sent to each server in this list.
-===== 1.3.         Communication with the server =====
-Open a browser and try access [[http://rbcce.aemet.es/eubrewnet/default/index|eubrewnet]]. If you cannot, then contact your system administrator and grant access to the internet. For debugging, it is recommended to execute the client software using the -v parameter (the more v, the more debugging information shown on the screen, with -vvv the maximum):
-**  $>python refresh.py -b XXX -vvv\\       - XXX: Id of a brewer**
-If you want to save the errors and log, define log and error files:
-**  $>python refresh.py 1>client.log 2>client_error.log******
-The two files (client.log and client_error.log) should not contain error messages after the execution of the client software. In case of errors in the communication, the Eubrewnet [[eubrewnet@aemet.es|administrator]] should be informed.
-===== 1.4.         Schedule the data sending =====
-The Brewer data can be sent to Eubrewnet either directly from the operational Brewer PC or by using another computer which has access to the data.
-__From the Brewer computer:__
-Sending the Brewer data to Eubrewnet in regular time intervals can be achieved by adding a routine in your Brewer schedule. The routine is executed in desired time intervals and calls the client software.
-In W2k W7 the brewer software waits until the execution of each command is finished so if the communication is slow or fails it takes too much time to send the data, and may block or delay the brewer software. This does not happen if DOSBOX is used. The old MSDOS/GWBASIC does not allow the sharing of the files. This is because the B file is always open, which blocks the access to the file and stops the recording of the brewer observations by the client software.
-Thus, it is recommended to send the data in two steps:
-.      Inside the brewer software or a task scheduler run a routine to copy the bdata directory content in another place/computer like for example bdata1:
-**  set DEST_DIR=e:\brewer\\   set ORIG_DIR=e:\bdata1\\ \\   net use x: /DELETE\\   net use x: \\10.2.20.20\brewer   /PERSISTENT:NO\\ \\   robocopy  %ORIG_DIR%   %DEST_DIR%  /S /XO /V ** **\\ \\ ******
-.      Run the data sending software from this directory as scheduled task
-**$>python refresh.py   -b <nowiki>%%</nowiki>X  -s iberonesia -i %DEST_DIR%\brw#<nowiki>%%</nowiki>X\bdata<nowiki>%%</nowiki>X >client<nowiki>%%</nowiki>X.out 2>>client_error<nowiki>%%</nowiki>X.log**
-===== 1.5.          Fast installation guide =====
-**1- Install**the python interpreter and then download and uncompress client script ([[http://rbcce.aemet.es/eubrewnet/static/files/client_python.zip|client_python.zip]]) or client executable ([[http://rbcce.aemet.es/eubrewnet/static/files/client_exe.zip|client_exe.zip]]).
-Copy the scripts to the brewer program directory
-**2- Check the configuration file (client.ini)**
-Are you bellow a proxy ?
-**no**: then change noproxy in client.ini to YES !!!
-**yes**: then edit the client.ini and indicate your proxy server and port, set the noproxy flag to NO !!
-Then edit the necessary fields in the client.ini file, without changing the following fields:
-**CLIENT.INI**
-**[global] servers=iberonesia3**
-**proxy=****proxy.xxx.xxx:port******
-**[iberonesia3] url=****[[http://rbcce.aemet.es/refresh|http://rbcce.aemet.es/refresh]]**
-**db****=****iberonesia3.db**
-**serverport****=****8081**
-**noproxy****=****no**
-**3- execute the script**
-**python** **refresh.py -- b XXX -i c:\brewer_data -vvv 1>client.log 2>client_error.log**
-XXX is the name of your brewer and brewer_data is a directory where the brewer files are.
-If it takes long, it is probably is working fine. Check the log files to see if all the data of your test directory have been sent.
-**4- Schedule the data sent**: We use robocopy to make a copy of the bdata directory. Edit the following two files (EUBREWNET.BAT and CK.RTN) properly and add them to the brewer program directory. Then put CK on your brewer schedule to send data to eubrewnet
-**EUBREWNET.BAT**
-**set** **DEST_DIR=e:\eubrewnet\**
-**set****ORIG_DIR=e:\bdata157\bdata**
-**set** **X=157**
-**robocopy** **%ORIG_DIR% %DEST_DIR%\Brw#<nowiki>%%</nowiki>X\bdata<nowiki>%%</nowiki>X\ *.<nowiki>%%</nowiki>X /XO /V /MAXAGE:3**
-**python** **refresh.py -b <nowiki>%%</nowiki>X -s iberonesia -i %DEST_DIR%\brw#<nowiki>%%</nowiki>X\bdata<nowiki>%%</nowiki>X >client<nowiki>%%</nowiki>X.out 2>>client_error<nowiki>%%</nowiki>X.log**
-**CK.RTN**
-**10000 rem Call eubrewnet.bat**
-**10010 data ck**
-**20000 shell”eubrewnet**
-**30000 return**
-**65529 rem dummy******
-\\ ** **
-----
-====== 2.    Configuration ======
-The raw data (Brewer spectral measurements) sent to the Eubrewnet database have to be processed in order to get the final products (currently Total Ozone Column (O<sub>3</sub>) and sulfur dioxide (SO<sub>2</sub>) and in the future, Aerosol Optical Depth (AOD), spectral UV irradiance, Nitrogen dioxide (NO<sub>2</sub>), and ozone profiles). For this purpose, a number of instrumental and calibration constants have to be inserted in the data processing algorithm. The operator of each instrument is responsible for uploading and saving the configuration (i.e. the needed constants) for each product in the database. In addition to the configuration of each product, information regarding the station and the instruments should be also provided. A detailed list of all necessary configuration parameters and other useful information can be found in the following link:
-[[http://rbcce.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration|http://rbcce.aemet.es/dokuwiki/doku.php?id=devel:eubrewnetconfiguration]]____
-Before uploading the configuration for the station and the products, the operators have to contact the system administrator ([[eubrewnet@aemet.es|eubrewnet@aemet.es]]) and provide contact information and the list of Brewers for which they are responsible. Then, they have to register in the system. When the registration is complete they are able to login the website of [[http://rbcce.aemet.es/eubrewnet/|Eubrewnet]] and upload the configuration. In the following there is a more detailed description of the configuration uploading process.
-__Registration and login__
-As shown in Figure 1, at the right of the Eubrewnet webpage the user can find the **sign in** button. There, he/she has two options: **Register** or **Login**. If **Register** is chosen, a form appears which should be filled by the database user. If **Log in** is chosen the user is asked for his username and password and then he can have full access to the database.
-{{ :namespace:user_fig1.png?700 |}}**Figure 1:** Sign in options.