General
=======
Folding@Home Client Info, FCI in short, is a set of Perl scripts to
monitor Folding@Home clients over a network.
FCI consists of 2 parts, a client and a server.
The clients send their information to the server where this information is
processed into XML files and displayed on a web page.
The server runs on Linux, FreeBSD & OpenBSD, this because of the applications
which run on the serverside, and the client runs on Linux, FreeBSD, OpenBSD,
Mac OS X and Windows.
Below is a semi-graphic representation of the servers and clients involved,
and describes the traffic between them.
+-------------------------+ +-------------------------+
| Stanford Website | | qd Website |
| | | |
| Hosts: | | Hosts: |
| daily_team_summary.txt | | qdinfo.dat |
| daily_user_summary.txt | | qd |
| psummary.html | | qd.exe |
| team0.txt | | qd.openbsd |
| | | qd.freebsd |
| | | qd.darwin |
+-------------------------+ +-------------------------+
HTTP GET || HTTP GET ||
|| ||
The server downloads || || The server downloads
the files hosted on || || the latest qdinfo.dat
the Stanford website, || || and qd binaries from the
either via the script || || website of the qd author,
fci-update-stanford-files.pl || || either via the script
or via the web interface || || fci-update-qd-files.pl
|| || or via the web interface
\||/ \||/
\/ \/
+--------------------------------+
| fci-server | STEP 3:
| |
| Runs: | The server runs
| index.pl (Apache mod_perl) | fci-update-stanford-files.pl,
| fci-update-stanford-files.pl | fci-update-qd-files.pl,
| fci-update-qd-files.pl | fci-update-xml-files.pl,
| fci-update-project-images.pl | fci-update-project-images.pl,
| fci-update-xml-files.pl | and optionally
| fci-jmol-missing-projects.pl | fci-jmol-missing-projects.pl
+--------------------------------+
HTTP GET || HTTP POST /\
|| /||\
STEP 1: || || STEP 2:
|| ||
The client downloads || || The client uploads the
the latest qdinfo.dat || || information it gathered
and optionally the qd || || to the server
binary from the server || ||
if newer versions are || ||
available before it || ||
submits its information || ||
to the server || ||
\||/ ||
\/ ||
+--------------------------------+
| fci-client |
| |
| Runs: |
| fci-client.pl (which uses qd) |
| Folding@Home client |
+--------------------------------+
Client
======
The client is a Perl script, which invokes the qd binary to parse the queue.dat
file and to send its output and some other files to the server over HTTP.
Besides some files of the Folding@Home client it also tries to find the CPU
speed and amount of installed RAM. Depending on which Operating System FCI is
being run on, it determines this via different methods:
on Linux it uses the files /proc/cpuinfo & /proc/meminfo (if readable)
on Windows it uses the commandline tools systeminfo
on FreeBSD and OpenBSD variants it will try to get the information from the
file /var/run/dmesg.boot (if readable).
on MacOS X v10.2 - v10.x it uses /usr/sbin/system_profiler,
and on MacOS X v10.0 - v10.1 it uses AppleSystemProfiler
This is not a fullproof method, therefor it is not mandated by the server to
supply this information.
In previous versions of FCI, the unitinfo.txt file was used as the main source
of information about the current Work Unit of the Folding@Home-client, but it
turned out not to be a reliable source, some cores tend to generate incomplete
unitinfo.txt files or because of the maximum length of the project name only
the first part of the project name was shown.
Since v1.0 qd is used to parse the queue.dat file which is a far more accurate
source of information, but as with the project summary on the server, it needs
to be frequently updated to be able to recognise new projects.
When the client is started, it first checks the ~/.fci/ directory and its
content. ~/.fci/ is not used for the qd files on Windows, there the same
directory which contains the fci-client.pl script is used for the qd files.
If ~/.fci/ does not exists, it is created and the qd binary and qdinfo.dat
files are copied from the paths where they are found.
By default /usr/local/bin/ on Linux, FreeBSD and OpenBSD, but it will also try
to find the qd files in the same directory as the fci-client.pl script. This to
make sure upgrades from FCI version prior to v1.0 will still work.
If it does exists, the required files are checked and the permissions of the
residing files are verified.
After the ~/.fci/ directory has been verified, the clients checks for newer
versions of qdinfo.dat. If the qdinfo.dat file is newer it is downloaded and
saved in the ~/.fci/ directory.
For security reasons fci-client.pl v1.0 does not automatically download the qd
binary anymore. If a more recent functional revision is available, it will only
be downloaded if the --update-qd paramenter is given.
By automatically downloading the qdinfo.dat file, the most recent project
information is available to qd when it parses the Folding@Home-client data.
After the qd files are updated, the client gathers the files to submit to the
server and tries to discover the CPU speed and installed RAM. All this
information is than submitted to the server via an HTTP POST where it will be
processed further.
Uploaded data
-------------
The client can upload serveral files and other information to the server.
By default it only requires to upload:
- a client name
- the output of the qd command
- the fci-client version
The client will upload additional data if present on the system.
The additional data included by default:
- the unitinfo.txt file
- the current.xyz file
- the client.cfg file
- the FAHlog.txt file
- the FAHlog-Prev.txt file
- the MyFolding.html file
You can skip the uploading of these additional files with the respective
--skip-* commandline parameter. Use e.g. --skip-client-cfg to skip the upload
of the client.cfg file. Run fci-client.pl --help to see all available
commandline parameters.
Data sources
------------
The client uses several files and applications to gather information of the
Folding@Home-client, which it will submit to the server. On the server this
information will be processed and presented on a web page. Below is a list of
data sources which the client uses and a description of what information it
provides.
The directory in which the Folding@Home-client runs is: $folding_dir
- fci-client.conf (optional)
fci-client.pl configuration file
provides configuration for the client,
The main hash with configuration parameters in fci-client.pl is filled with
hardcoded defaults first, those are overridden by parameters from
fci-client.conf, which ultimately can be overridden by commandline
parameters.
- qd
Application to parse the queue.dat file
main source of information,
provides information of the work units in the queue and
Folding@Home-client configuration
- $folding_dir/client.cfg (optional)
Folding@Home client configuration
provides additional information of the Folding@Home-clients configuration
- $folding_dir/FAHlog.txt (optional)
Folding@Home client log file
provides information of the Folding@Home-clients environment and progress
- $folding_dir/FAHlog-Prev.txt (optional)
Folding@Home client log file of previous work units
optional file, might not exist
provides information of the Folding@Home-clients environment and progress of
previous work units
- $folding_dir/MyFolding.html (optional)
Folding@Home client web page
provides URLs of the users web pages at the Stanford website
- $folding_dir/unitinfo.txt (optional)
Folding@Home clients work unit summary
optional file, might not exist
(some cores don't generate (a proper) unitinfo.txt)
provides some information of the current work unit
- $folding_dir/work/current.xyz (optional)
Molecule display file of the current work unit
provides data to generate an image of the molecule
Install Paths
-------------
On Linux, FreeBSD and OpenBSD all the files requered by the client are stored
in ~/.fci/, the client itself can be located anywhere. If the client cannot find
the files it needs in ~/.fci/ it will try to find them in the directory in
which itself is stored. If on startup the ~/.fci/ directory is not found, it is
created and the required files will be copied from /usr/local/bin/ and
/usr/local/share/fci/ on Linux, FreeBSD and OpenBSD. On Windows all the files
are stored in the same folder, the folding in which the client was installed,
C:\Program Files\FCI\ by default.
- fci-client.pl
On Linux, FreeBSD and OpenBSD this file should be installed in a directory in
$PATH, /usr/local/bin/ by default on Linux, FreeBSD and OpenBSD.
On Windows this files is installed in C:\Program Files\FCI\ by default, the
user can specify the installetion directory when using the install.pl script.
Default:
Linux : /usr/local/bin/fci-client.pl
FreeBSD : /usr/local/bin/fci-client.pl
OpenBSD : /usr/local/bin/fci-client.pl
Windows : C:\Program Files\FCI\fci-client.pl
- qd
On Linux, FreeBSD and OpenBSD this file is located in the ".fci" subdirectory
of the home directory of the user executing fci-client.pl.
On Windows fci-client.pl assumes this file to be in the same place as itself.
The master binary which will be copied to the ".fci" directory on the first
run, it is located in /usr/local/bin/ on Linux, FreeBSD and OpenBSD.
Default:
Linux : ~/.fci/qd
FreeBSD : ~/.fci/qd
OpenBSD : ~/.fci/qd
Windows : C:\Program Files\FCI\qd.exe
- qdinfo.dat
On Linux, FreeBSD and OpenBSD this file is located in the ".fci" subdirectory
of the home directory of the user executing fci-client.pl.
On Windows fci-client.pl assumes this file to be in the same place as itself.
The master file which will be copied to the ".fci" directory on the first
run is located in /usr/local/share/fci/ on Linux, FreeBSD and OpenBSD.
Default:
Linux : ~/.fci/qdinfo.dat
FreeBSD : ~/.fci/qdinfo.dat
OpenBSD : ~/.fci/qdinfo.dat
Windows : C:\Program Files\fci\qdinfo.dat
- fci-client.conf
On Linux, FreeBSD and OpenBSD this file is located in the ".fci" subdirectory
of the home directory of the user executing fci-client.pl.
On Windows fci-client.pl assumes this file to be in the same place as itself.
Default:
Linux : ~/.fci/fci-client.conf
FreeBSD : ~/.fci/fci-client.conf
OpenBSD : ~/.fci/fci-client.conf
Windows : C:\Program Files\fci\fci-client.conf
Server
======
The server side of FCI consists of serveral parts. Most important is the Apache
mod_perl application which handles the client data uploads, and the display of
the processed data. The actual processing of the uploaded data is done by
separate scripts.
Older versions of FCI processed the uploaded data every time a webpage was
requested, this worked fine on >1 Ghz machines, but slower machines had quite
a rough time processing the data. Also with the pre-1.0 releases of fci, the
generation of the protein image from the uploaded current.xyz file was done
at the moment of upload. This made uploads quite slow, the client had to wait
for a long time to get its response on his upload.
Now the server only stores the data the client uploads, making the handling of
the uploads much faster. The server will also be able to handle serveral
concurrent uploads much better than before. As it was not uncommon to see some
10 clients uploading their data all at once, thanks to the nature of scheduling
uploads.
When clients now upload their data, everything is saved as-is. The only file
which is processed is the current.xyz file if it was uploaded. The project
number and work unit name are extracted from the file, and the project number
is checked against the list of known projects. If the project is listed in the
known projects file no further processing is done. If the project is not listed
the xyz file will be copied to the project-data directory and a new record will
be created for this project in the known-projects list.
The script fci-update-project-images.pl is run to generate an image for this
new project. By default the image type for uploads is a single still image,
because these take the least amount of time to generate. The idea is to
schedule fci-update-project-images.pl to run once a day or once an hour to
generate the images for the missing image types for all known projects.
This way every image is generated only once, instead of at very upload. The
more load intensive image types as rotate and xyz180 can be scheduled to be
generated at a time when the system has least load.
The uploaded data will only be used by the display mode of the webapplication
after it has been processed into the XML files used by that same display mode.
These XML files are generated by fci-update-xml-files.pl. This script will
process the data uploaded by all the clients, the most important client data
being the data output by qd as stored in qd-data.xml.new on upload. If other
data was also uploaded by a client, these files will also be parsed by
fci-update-xml-files.pl and used to update the XML files.
Besides client data, fci-update-xml-files.pl will also use some of the stats
files provides by Pandegroup on the Stanford webservers. These files are
downloaded and processed by fci-update-stanford-files.pl. The files in
question are the teamstats files (team0.txt), project summary (psummary.html),
the user stats file (daily_user_summary.txt) and the team stats file
(daily_team_summary.txt). The psummary is updated every hour, the teamstats
files are updated every 3 hours and the daily summaries are updated every 4
hours. The files will only be downloaded and processed if their Last-Modified
date has changed since it was last downloaded. It is therefore advised to
schedule fci-update-stanford-files.pl to run every, so your project summary
will be updated as soon as possible, whereas the other files are only also
downloaded if they have actually changed.
Because the Stanford files are used by fci-update-xml-files.pl,
fci-update-stanford-files.pl should have been run before
fci-update-xml-files.pl. I personally schedules my clients to upload every
hour on the 0th minute. On the server I schedule
fci-update-stanford-files.pl to run every hour on the 0th minute. Every hour
on the 30th minute, I then run fci-update-xml-files.pl. Scheduling the
different scripts in this way will make sure the fci-update-xml-files.pl is run
after the data sources it uses have been updated.
Independent of the data required by fci-update-xml-files.pl, which is mostly
focussed on client, user and team data, there is the data of the projects
which are being worked on by the clients. On client uploads an image is already
generated for each project, and so an xyz and pdb file are available for it
too. To generate the images for the remaining image types run
fci-update-project-images.pl, whose default behaviour is to generate all
missing images and their accompanying rasmol scripts and pdb files.
I have scheduled fci-update-project-images.pl to run every hour on the 45th
minute, after fci-update-xml-files has run (on the 30th minute).
Data sources
------------
The server connects all the data source together, the client supplied data and
that provided by Stanford and third parties. Below is a list of data sources
which the server uses and a description of what information it provides.
- fci-client.pl
The client application of fci
main source of information,
provides information from the clients in the form of several files
file upload is done via HTTP POST from the client machine
- http://fah-web.stanford.edu/daily_user_summary.txt
Folding@Home project user stats
provides information of the users in the project
previously at: http://vspx27.stanford.edu/daily_user_summary.txt
- http://fah-web.stanford.edu/daily_team_summary.txt
Folding@Home project user stats
provides information of the teams in the project
previously at: http://vspx27.stanford.edu/daily_team_summary.txt
- http://fah-web.stanford.edu/psummary.html
Folding@Home projects list
provides information of the different projects running on Folding@Home
previously at: http://vspx27.stanford.edu/psummary.html
Applications:
-------------
The server part of FCI uses and hosts some applications for use by either the
server or the clients. The qd application is used by the clients to gather
information of their Folding@Home client. Updated versions of qd and its
data file, qdinfo.dat, are hosted on the server, before each upload, the client
downloads the updated files from the server if a newer version are available.
Some other applications are only used by the server, for processing of the
uploaded information.
- http://www.boston.quik.com/rph/qd
Linux application to parse the queue.dat file of the Folding@Home client
Runs on the Client, updates are stored on the server for automatic update
- http://www.boston.quik.com/rph/qd.exe
Windows version of qd
- http://www.boston.quik.com/rph/qdinfo.dat
Data file containing the information of the running projects for use in qd
- http://www.boston.quik.com/rph/xyz2pdb
Linux application to convert an xyz molecule display file to a pdb file
Used in combination with rasmol to generate an image of the molecule
- http://www.bernstein-plus-sons.com/cgi-bin/yaya/decomp.cgi/software/ \
RasMol_2.7.2.1.1/RasMol.LINUX/RedHat_7/i386/rasmol_32BIT
Linux application to manipulate molecule display files
Used to generate images from display files
- http://www.bernstein-plus-sons.com/cgi-bin/yaya/decomp.cgi/software/ \
RasMol_2.7.2.1.1/doc/rasmol.hlp
Help document for rasmol binary
- fci-update-qd-files.pl
Application to download the latest qd files to update the
client application pool
Downloads the files:
http://www.boston.quik.com/rph/qd
http://www.boston.quik.com/rph/qd.exe
http://www.boston.quik.com/rph/qdinfo.dat
- fci-update-stanford-files.pl
Application to download the latest stanford files to update the
client application pool
Downloads the files:
http://vspx27.stanford.edu/daily_team_summary.txt
http://vspx27.stanford.edu/daily_user_summary.txt
http://vspx27.stanford.edu/psummary.html
|