COST1207 | Eubrewnet’s data submission guide

Each Brewer spectrophotometer communicates with a local PC where the raw data is stored. To upload the raw data from the client PC to the Eubrewnet database, a code written in Python 2.7 has to be set up in the local PC (or any other computer which has access to the Brewer raw data). When this code is executed the local (client) PC connects to Eubrewnet’s data server. Then, the local Brewer files and all related information (file size, modification date, etc) are compared to the data already stored in Eubrewnet. Only the files which are either new or have changed are then submitted. This process should be repeated at regular time intervals (~15 mins).

This post explains how to download and setup Eubrewnet’s client software in your local PC. We include here two Sections from Eubrewnet’s User Manual, which is available in full at http://rbcce.aemet.es/dokuwiki/doku.php?id=codes:usermanual&#eubrewnet_user_manual. A Quick setup guide is provided in Sec. 1.1, and Sec. 1.3 contains a FAQ.

1.1. Quick setup guide

1.1.1. Download Eubrewnet’s client

Either

a) Download the client executable for Windows from http://rbcce.aemet.es/eubrewnet/static/files/client_exe.zip or the experimental Linux static binary from http://rbcce.aemet.es/eubrewnet/static/files/client_linuxstatic.zip

b) Setup the Python 2.7 interpreter in your computer, and then download the client Python script from http://rbcce.aemet.es/eubrewnet/static/files/client_python.zip

1.1.2. Unzip the downloaded file to a pre-existing folder

Create a new folder anywhere in your system and extract all the contents of the zip file there. Note that a new folder will not be created when you unzip the downloaded file.

Depending on the version of Eubrewnet’s software client, the files extracted will be

a) for the client executable for Windows (Linux): refresh.exe (refresh) and example_client.ini

b) for client Python script: refresh.py, example_client.ini, refreshdb.py, util.py, xmlrpcproxy.py, log.py, alive.py (in total, 6 .py files and 1 .ini)

1.1.3. Configure the client.ini file

Open the example_client.ini file which was included in the zip. It contains the following lines

[global]
brewerid=157
working_dir=.
input=c:\brewer#157\bdata157
servers=iberonesia3
proxy=proxy.xxx.xxx:port

[iberonesia3]
url=http://rbcce.aemet.es/refresh
db=iberonesia3.db
noproxy=yes

Write your Brewer ID number in the brewerid= line.

In the input= line, write the full path to the folder containing your B files.

Is your PC behind a proxy? If the answer is
– No: then do not change the line noproxy=yes
– Yes: change the line to noproxy=no, and write the proxy’s address and port in the proxy= line

Once you have finished editing the file, save it as client.ini in the same folder where refresh.exe, refresh or refresh.py reside.

1.1.4. Execute the client

In the case of the client executable, in Windows just double click on refresh.exe, and in Linux run ./refresh from a terminal inside the refresh directory.

To execute the Python script, open a console/terminal, change to your work directory, and use the command python refresh.py.

In both cases, you should see some information on the screen, but note that the window might open and close quite fast if you double-clicked on refresh.exe in Windows.

To save a log file, if using the client executable, run it from the console with the command refresh.exe >client.log. If using the Python script, use the command python refresh.py > client.log

1.1.5. Create an automated task

We recommend running the client automatically every ~15 minutes. In the case of Windows, look for information on the Windows Task Scheduler for your Windows OS version. For *nix systems, check the crontab command.

1.3. Frequently Asked Questions

Q What are the “old” and “new” versions of Eubrewnet’s client?

A The first release of the “new” version was distributed to a small group of beta testers on July 2019, and was then published on Eubrewnet for general use at the end of September of the same year. Compared to the “old” version, it features many improvements, including

any subfolder of input= which includes the Brewer ID number at the end of its name is now also checked for new files

an option to check all subfolders hanging from input= has been added

for files with duplicate filenames in different folders, only the most recently modified one is submitted

added pauses between connections to the server

after a server error, the client now tries the operation again a number of times

if a server does not answer, the client now continues with the next one

the type of local database is now checked

Most important, Eubrewnet’s client will now write a lot of information to screen. This will help the user to understand what is going on and solve errors. This is also the most noticeable difference between the old and new version of the software: if you see a lot of text on the screen, you’re likely using the new client.

Despite all these changes, note that the old and new versions of Eubrewnet’s client software share the same code at their core, so you should be able to use either version to submit your data without making any change to your client.ini file.

Q Do you plan any further updates to Eubrewnet’s software client?

A Yes, in the near future we expect to release a Python 3-compatible version. Until this version is available, if your running a linux distro which does not include Python 2.7, please try to use the experimental linux static binary.

Down the road, we are also looking into other quality of life improvements, such as automatically writing a log file with all the on-screen information, as suggested by V. Savastiouk. If you have suggestions, remember that you can reach us at eubrewnet@aemet.es!

Q During my trials I’ve got an error: Syntax error: Invalid syntax

A Make sure you’re using Python 2.7 and not 3. A Python 3-compatible version of Eubrewnet’s client software will be provided in the near future. If you can’t install the Python 2.7 distributable in your Windows or Linux system, please try to use the Windows client executable and the experimental linux static binary, respectively.

Q We are using a unix database ourselves, so one possibility is to transfer the data automatically from that to your database.

A Eubrewnet requires the original Brewer files. Direct communication between databases is not considered at this moment.

Q What is the best way? Ftp is easy for us to implement via the built-in scheduler in Windows.

A There are no plans to allow submissions via FTP. Please check the Quick guide above – we’re sure that you will find that Eubrewnet’s client is very easy to use and automatize. And if you have any problem setting up the software, don’t hesitate to contact us at eubrewnet@aemet.es

Q Do we have to send data in real time or are you planning to start with data files from past years?

A On each execution, Eubrewnet’s client always checks all the files in the directory specified in the input= line of your client.ini

So, if you have all your files (new and old) in a single directory, they will all be checked and submitted if they are not already in Eubrewnet’s database (or if they have been modified since they were submitted).

If you have old files in a subdirectory, you can modify client.ini‘s input= line and run Eubrewnet’s software once.

Newer versions of Eubrewnet’s client also provide a -r flag to look for files in all the subdirectories hanging from the one specified in the input= line. Just run the software with refresh.exe -r or python refresh.py -r to enable this functionality, but note that this might take a lot of time!

Q In case it would be real time data that you want, what would be the frequency at which we would have to send the data? (daily, weekly, monthly)?

A Eubrewnet’s data server is capable of processing your data in near real time, so you can submit your files as often as you like! We suggest running Eubrewnet’s client every ~15 minutes using an automated task.

Q We are very cautious with our Brewer computers and we (together with the IT staff from our institute) are not very keen to install something new on these computers.

A You are welcome to check the source code of Eubrewnet’s software client – just download the Python script version and inspect the .py files, no other files besides those are used to generate the executable version of the client.

If you are worried about the possibility of getting your computer infected by a virus, please note that we run a virus scan on the executable version of Eubrewnet’s client before we publish it on the web. Nevertheless, you also have the option to use the Python script version of Eubrewnet’s client.

Finally, if for some reason you’re still not allowed to run the software directly on the local PC connected to the Brewer, we suggest that you try to share the data with another computer under your full control and run the software on it.

Q I just sent you data from my computer: did you receive it?

A You can check the data yourself. The easiest way is perhaps to check the color of your Brewer on the map at http://rbcce.aemet.es/eubrewnet/default/index – Green or yellow is OK.

Another possibility is to use the “File Status Check” tool available at http://rbcce.aemet.es/eubrewnet/files/status. A status value of 0 for your file means everything is OK.

Also, please note that communication errors usually produce some messages in the output of Eubrewnet’s client. If you want to debug these messages, we recommend running the client redirecting the output to a log file: you just have to use refresh.exe > client.log or python refresh.py > client.log, depending on the package you donwloaded.

Finally, please remember that you can always contact us at eubrewnet@aemet.es

Q I have already sent data files. Now, how to send the calibration files that usually are in the \data\###\ folder?

A Newer versions of Eubrewnet’s software client will automatically check all the folders ending in the Brewer ID number and located inside the one specified in the input= line of the client.ini file – just take a look at the information printed on the screen.

If you’re using an older version of the software, you can temporarily modify your input= line and run the client once.

Note that Eubrewnet’s data server knows how to parse most files generated or used by Brewer instruments (and, if the file can’t be automatically parsed, it will remain in the server waiting for us to check it manually!), so please don’t hesitate to submit all the files you have!

Q My first attempt to send Brewer data finished with this error: xmlrpclib.ProtocolError: ProtocolError for rbcce.aemet.es/refresh/xmlrpc/: 403 Forbidden

A You can’t connect to Eubrewnet’s submission service.

Please, try to contact the refresh service from a web browser using the URL http://rbcce.aemet.es/refresh/xmlrpc/ – you should get a message like this

Error response

Error code 501.

Message: Unsupported method ('GET').

Error code explanation: 501 = Server does not support this operation.

If you don’t get any answer from the refresh service, then your connection might be blocked by a proxy, please check with your sysadmin. Remember that if your computer is behind a proxy, your client.ini must include the line noproxy=no, and the proxy’s address and port must be written in the line proxy=

If you’re not behind a proxy, a firewall might be blocking the connection. Please contact your sysadmin.

Q There’s an error while running the client: xmlrpclib.Fault: (Fault1 : “(type ‘exceptions.AtributeError’): ‘NoneType’ object has no attribute ‘last_online’

A Most likely, the registration of your Brewer in Eubrewnet has not been completed yet. Please, contact us at eubrewnet@aemet.es

Q My Windows computer is very old and I can’t setup the refresh client following the steps in the Quick Setup – help!!!

A If you have a very old Windows computer which can’t run the compiled .exe of the new version of the client, and furthermore can’t run the official Python2.7 installer to use the python version, try the following options, sorted in order of complexity:

Easiest) Try to use the .exe of the old version of the client, available from http://rbcce.aemet.es/eubrewnet/static/files/client_exe_2014.zip

Normal) If you have an old setup of Python in your computer, try to use the python version of the old client, available from http://rbcce.aemet.es/eubrewnet/static/files/client_python_2014.zip

Most difficult) Download the file http://rbcce.aemet.es/eubrewnet/static/files/Python27_32bits.zip instructions in the running_refresh_client.txt file included in the zip. In short, the zip contains a folder with the Python2.7 setup of a WindowsXP 32-bit computer, and you can you use it to run the python version of the new client (available from http://rbcce.aemet.es/eubrewnet/static/files/client_python.zip)

Q My Linux distro does no longer include Python 2.7 – help!!!

A We are working on providing a Python 3-compatible version of the refresh client, but it’s not ready yet. Meanwhile, you can try to use the experimental linux static binary available from http://rbcce.aemet.es/eubrewnet/static/files/client_linuxstatic.zip