Data sending

Before any data sending be sure if your brewers have sent data to iberonesia before. Look for your brewers in Brewer List. If they are not there, contact the admin.

A FAQ section has been added to this post
A QuickGuide for the impatient is also available.

Install the Software

The client software can be installed in any computer with access to the brewer data files.

Setup instructions

There are two different ways to use the client:

Install the python interpreter and download the python client (preferred method).
Download the exe client (not fully tested).

Python Client

Python Interpreter

Download the needed interpreter and install it

Python 2.7 Windows x86 MSI Installer (Windows binary)
Python 2.7 Windows X86-64 MSI Installer (Windows AMD64 / Intel 64 /X86-64 binary)
Python 2.7 Mac OS X 64-bit/32-bit x86-64/i386 Installer (for Mac OS X 10.5 and later, if needed)
Python 2.7 Mac OS X 32-bit i386/PPC Installer (for Mac OS X 10.3 and later, if needed)
Linux Version should come bundled with your Linux distribution.

Be sure that your python interpreter is callable from every path in your system (Open a console and try python, exit() for exit).

Python client

Download and uncompress it in your system.

You can download client_python.zip.

   + client_python
      + source
         alive.py
         client.ini
         configure.py
         download.py
         log.py
         refresh.py
         refreshdb.py
         util.py
         xmlrpcproxy.py

Exe Client (Windows Only)

Download and uncompress it in your system.

You can download client_exe.zip .

   + client_exe
      + build
         - bdist.win32
            ...
      + dist
         _bbsddb.pyd
         _hashlib.pyd
         _socket.pyd
         client.ini
         library.zip
         pyexpat.pyd
         python27.dll
         refresh.exe
         select.pyd
         unicodedata.pyd
         w9xpopen.exe
      + source
         ...

Unzip the .zip and try $>refresh.exe (in dist folder) in a console to test if client is working properly. No error message should appear.

Configuration

Client configuration file

The client configuration file is used by the refresh.py program. The default configuration file is client.ini.

The file is a plain ASCII INI file with the INI file syntax. The file consists of a global section and several server sections, one for each server to send the files to.

The content of the present example includes lines (bold) that has not to be changed. This information is relative to the server or critical:

   [global]
   brewerid=157
   working_dir=.
   input=c:\brewer\data
   servers=iberonesia3
   proxy=proxy.xxx.xxx:port
   [iberonesia3]
   url=http://rbcce.aemet.es/refresh
   db=iberonesia3.db
   noproxy=yes
   serverport=8081

The easiest way to configure the client is to specify all options in client.ini file (like the one above) and execute refresh.py without arguments; and execute with arguments overriding the client.ini options if you want any unusual operation, such as sending only certain files. E.g. (using the client.ini above):

gt;python refresh.py -b 157 -i c:\brewer\data
You only have to edit the brewer (-b) and the brewer data directory (-i).

Parameters -b and -i are not necessary if specified in client.ini file (recommended).

The global section has the following properties:

brewerid : brewer identifier of the brewer owner of the files to send.
input : directory that contains the files to send.
servers : comma separated list of server aliases. The files are sent to each server in this list.
working_dir : directory where the local refresh database is stored.
proxy : proxy url, if required. If not required, this property must not be present.

A server section is identified by the server alias (i.e. the alias is the name between the brackets), and it has the following properties:

url : server url.
db : local database filename. A local database (a simple data file) is created the first time client is run. The database path is determined by the rundir property and this db property.
noproxy : if a proxy is specified in the global section, noproxy may be set to yes to use a direct connection (avoiding the proxy) to access this server.
serverport : port in the server (8081)

Some of these properties may be overridden using command line parameters when executing refresh.py or alive.py.

You only have to edit the brewerid to your brewer. Comment the lines (put # at the beginning of the line) if not needed (proxy is an example).

refresh.py options

Most of the refresh.py options may be configured though command line arguments or client.ini configuration file.

Any option specified in command line overrides the option in client.ini.

All options are shown executing the client with parameter -h

   $ ./refresh.py -h

   Usage: refresh.py [options]

   Options:
     -h, --help            show this help message and exit
     -i FILE, --input=FILE
                           File to process. If the given file is a directory, all
                           files within will be processed.
     -s SERVERS, --servers=SERVERS
                           Comma separated list of xmlrpc server aliases
     -b BREWERID, --brewerid=BREWERID
                           Brewerid of file/s to process
     -c FILE, --config=FILE
                           Configuration file path
     -v, --verbose         Verbosity level. The more '-v' the more verbose is the
                           program.

The mandatory options are:

working_dir : This option must be specified in client.ini. Directory where the local refresh database is stored.
input : File to process. If the given file is a directory, all files within will be processed.
brewerid : Brewer identifier of the brewer owner of the files to send.
servers : Comma separated list of server aliases. The files are sent to each server in this list.

Check the communications

Try to open a browser and access eubrewnet. If not, try to contact your system admins to grant access to the internet.

To debug any problem, it is recommended to execute the client with a command line and using the -v parameter (the more v, the more debugging information shown, with -vvv the maximum).

gt;python refresh.py -b XXX -vvv
- XXX: Id of a brewer
If you want to save the errors and log , try to define log and error file like this:

gt;python refresh.py 1>client.log 2>client_error.log
Look for the content in client.log and client_error.log (If there’s any error in the communication contact the Admin and send the content of these files).

Schedule the data sending

There are two usual methods:

Send the data from the Brewer computer.
Use another computer which has access to the data.

From the Brewer computer

As a brewer routine you can put a routine (an example below) in your schedule who call the sending program to send the data.

In W2k W7 the brewer software waits until the command is finished so if the communication fails or it’s slow it can take too much time for sending and can stop or delay the brewer software. This is not an issue if you use DOSBOX

As an scheduled task you can program the execution of the sending program.

The old MSDOS/GWBASIC do not allow the sharing use of the files, as the B file is always open could happen that the sending program block the access to the B file and stop the record of the brewer observations.

So we recommend the send of the data in two steps:

Inside the brewer software or an schedule task run a routine to copy the bdata directory content in another place/computer like for example bdata1.

   set DEST_DIR=e:\brewer
   set ORIG_DIR=e:\bdata1

   net use x: /DELETE
   net use x: \\10.2.20.20\brewer   /PERSISTENT:NO

   robocopy  %ORIG_DIR%   %DEST_DIR%  /S /XO /V

Run the sending software from this directory as scheduled task


gt;python refresh.py   -b %%X  -s iberonesia -i %DEST_DIR%\brw#%%X\bdata%%X >client%%X.out 2>>client_error%%X.log

Appendix

How it works

When all parameters are processed, it is known the files to send and the servers to send to. The program loops though the servers and tries to send all the files specified. The client maintains a local database storing md5sum and size of the files that has been transferred to the server, used to determine if a file needs to be sent.

Sending files to a server

The protocol to send the files is:

   Check if the refresh database for the server exists. If not exists, the database is (re)generated. 
   For each file: 
      Check if the file needs to be sent. The client knows if a file needs to be sent comparing the (md5sum, size) stored in the local database with the (md5sum, size) of the actual file. 
         If the file entry in the local database does not exist, the file is sent 
         If any of the md5sum, size stored in the database differs from the ones in the actual file, the file is sent. 
         In other case (md5sum and sizes are equal), the file is not sent.

The transfers cannot be done file to file (would be too many) nor all the files simultaneously (too many data to transfer could cause a time out). So, the files are transferred in blocks. If a threshold size of files is reached, the files are packed in a zip file and sent to the server. Once the file is actually sent, the (md5sum, size) in the local database is updated.

Regenerating local database

The protocol to regenerate the database is:

   The client sends to the server all the files that is willing to transfer (i.e., the files specified with the input parameter) 
   For each file in the previous list, the server returns the appropiate (md5sum,size) tuple of the file stored in the server. If the server knows nothing about that file (i.e., it has never been transferred), no entry is returned. 
   Each entry returned to the client is written to the local database.

This way, the local database stores the same info about the file that is stored in the server, so there is no need to query that info to the server for each file in the input, thus saving a lot of network time.

It is possible that the local database gets corrupted or out of sync with the server (e.g., a server is replaced and some files had been sent to the old server that are not in the new server). In that case, the database needs to be deleted, and so regenerated in the next execution.

Quick Guide

Quick guide for the impatient

1- Install the python interpreter and then client script (client_python.zip) or client executable (client_exe.zip).

Copy the scripts to the brewer program directory

2- Check the configuration

Are you bellow a proxy ?

no: change noproxy in client.ini to YES !!!
yes: Edit the client.ini and indicate your proxy server and port, set the noproxy flag to NO !!

CLIENT.INI
[global] servers=iberonesia3
proxy=proxy.xxx.xxx:port
[iberonesia3] url=http://rbcce.aemet.es/refresh
db=iberonesia3.db
serverport=8081
noproxy=no

3 execute the script
python refresh.py – b XXX -i c:\brewer_data -vvv 1>client.log 2>client_error.log

XXX is the name of your brewer and brewer_data is a directory where the brewer files are.

it takes long, probably is working fine, its sending all the data of your test directory, check the log files.

4- Schedule the data sent : We use robocopy to make a copy of the bdata directory, edit this two files to the brewer program directory . Now you can put CK on your brewer schedule to sent data to eubrewnet

EUBREWNET.BAT

set DEST_DIR=e:\eubrewnet\
set ORIG_DIR=e:\bdata157\bdata
set X=157
robocopy %ORIG_DIR% %DEST_DIR%\Brw#%%X\bdata%%X\ *.%%X /XO /V /MAXAGE:3
python refresh.py -b %%X -s iberonesia -i %DEST_DIR%\brw#%%X\bdata%%X >client%%X.out 2>>client_error%%X.log

CK.RTN

10000 rem Call eubrewnet.bat
10010 data ck
20000 shell”eubrewnet
30000 return
65529 rem dummy

FAQ

My first attempt to send Brewer data finished with this error:
xmlrpclib.ProtocolError: ProtocolError for rbcce.aemet.es/refresh/xmlrpc/: 403 Forbidden

This case of error is because you have not access to the service.
The first thing to do is try to access Refresh service address in a browser. If the failure is that the method is unsuported the problem is in the configuration of your system.
Brewer Operator has corrected it because he had to pass through a proxy and by default in the client.ini the noproxy option is set to yes that means that you are not behind one.
Other problem could be a firewall configuration that forbids access to the client program. In this case you may speak with your IT people to fix it.

During my trials I’ve got an error:
Syntax error: Invalid syntax

Be sure that the python version that you are using is 2.7, not 3.X. The correct links are in the instructions.

There’s an error while running the client:
xmlrpclib.Fault: (Fault1 : “(type ‘exceptions.AtributeError’): ‘NoneType’ object has no attribute ‘last_online’

System cannot receive data from brewers that have never sent data to iberonesia. If you don’t be sure about it go to our Brewer List and look for your brewers.
If they are not there, contact admins to add the new brewers to the database.

We are using a unix database ourselves, so one possibility is to transfer the data automatically from that to your database.

The eubrewnet use a client to send data based in the brewer files. Communication with other databases is not contemplated in this project.

What is the best way? Ftp is easy for us to implement via the built-in scheduler in Windows.

The eubrewnet use the client to send the data, we are not going to develop any solution based on ftp.

Do we have to send data in real time or are you planning to start with data files from past years?

All the files that are stored in the source forder are going to be sent. We can find to different type of files:

Files that are not going to be changed (old): They are going to be sent once.

Files that are being modified: Only the new information that doesn’t form part of our filesystem is going to be sent.

In case it would be real time data that you want, what would be the frequency at which we would have to send the data? (daily, weekly, monthly)?

Our client sends a group of files each time that it’s executed (old files too). An example of scheduled task is provided in the instructions. The brewers of the triad are sending data every fifteen minutes. If you don’t get a way to make this, contact your system admins to try to get a solution for it. If not, contact the eubrewnet admin.

We are very cautious with our Brewer computers and we (together with the informatics people from our institute) are not very keen to install something new on these computers.

The source code is provided (for security purposes), if your IT people have problems installing things in the computers that manage the brewers there’s no problem. You can run it from a different one that can access the brewer files.

I just sent you data from my computer: did you receive it?

You have some tools to see if your file has been sent:

The first of all is the graphic information in the Brewer List. Green or Yellow is ok!

Why not change the run putting there a log and an error file?
$>python refresh.py 1>client.log 2>client_error.log
In client_error.log you will see the errors.

File status tool in eubrewnet: http://rbcce.aemet.es/eubrewnet/files/status (You can access it in the web service from the Admin menu: Status)
Look for the B files that have been sent from your brewer and if the status is 0 everything is ok.

I’m already sent data files. Now, how to send calibration files that usually are in the \data\###\ folder ?

If you mean ICFs, DCFs… they should be recognized. Do not be afraid. Just send them and if they can be parsed, they are going to be stored in our database, if not, at least we are going to have a copy of the file s in our file schema to see what is not working.