This document describes the Grid Engine source code modules, and is
written for use by software developers. Our goal is to provide you with
useful information if you are considering participating in the Grid Engine
project or will be using Grid Engine source code. In addition to this source
documentation, the user product documentation is necessary for a full understanding
of the functional operation of the software. Consequently the man
pages are refered to herein where appropriate.
If you have problems with finding functions or browsing through the
souce code we suggest the use of grep or preferably ctags.
1. Job life cycle
The job life cycle is a good way to understand to overall system.
-
Execds report load information to qmaster
-
User submits job using qsub command
-
Qmaster notifies Schedd about new job
-
Schedd dispatches job to an Execd
-
Qmaster delivers job to Execd; Execd starts job using Shepherd
-
At job end Execd notfies Qmaster about job finish
-
Qmaster feeds jobs resource consumption in accounting database
2. Components
2.1 Daemon components
In the "source/daemons" subdirectory there is one directory for each daemon.
The Qmaster (Queue Master) controls the overall behaviour in a cluster.
For the purpose of scheduling jobs Qmaster cooperates with Schedd. The
Qmaster is the responsible for answering GDI requests from clients and
for delivering dispatched jobs to the assigned Execd's.
The Schedd (Scheduling Daemon) gets notified via events about all scheduling
relevant information. The resulting scheduling decisions are sent as orders
to Qmaster.
The Execd (Execution Daemon) provides Qmaster with information about utilization
and availability of resources. Each job sent to Execd by Qmaster is started
by writing all relevant information into a couple files describing the
job and forking a Sheperd. After Shepherds termination Execd reports details
about the job run back to Qmaster.
The Shepherd starts all kinds of jobs according to what he finds in the
per-job configuration files written by Execd. If necessary administrator
supplied procedures are started before the job is started (prolog, PE start-up),
during job run (checkpoint/migrate, terminate/suspend/resume) and after
job termination (PE shut-down, epilog).
Nearly all network communication in a cluster is handled via the Commd
(Communication Daemon). Daemons and clients use the commlib (communication
library) calls to send and receive messages (sync/async), but also a hostname
resolving service is provided by Commd.
The Shadowd (Shadow Daemon) detects failures of the Qmaster and starts
a new Qmaster if necessary.
2.2 Client components
In the "source/clients" subdirectory there is one directory for each client.
Qsub combines all information provided with the command line and in per-cluster
or per-user default settings files in a job object and sends the job via
GDI to Qmaster. As response Qsub receives a job ID and prints it.
Qrsh combines all job relevant information in a job object like Qsub does
and sends it as an interactive job via GDI to Qmaster. For stdin/stdout/stderr
redirection rsh command is forked by qrsh. The rsh command cooperates with
the rshd forked by shepherd for jobs of that type. Instead of rsh/rshd
also other commands can be used.
Qmon is a Motif GUI allowing to administer and monitor the system but also
to submit and monitor jobs. Qmon uses GDI to retrieve from Qmaster the
whole configuration and to change it.
Qdel uses GDI to initiate job termination.
Qacct produces accounting reports by scanning the accounting data file.
Qlogin invokes an interactive login session in the current terminal using
GDI.
Qresub creates a copy from an existing job using GDI.
Qselect prints a list of queues matching the specified criteria from the
command line. The necessary information about the clusters configuration
is fetched from Qmaster using GDI.
Qalter modifies an already submitted job using GDI.
Qhold puts a hold on a job using Qalter.
Qhold releases a hold from a job using Qalter.
Qsh submits an interactive X-windows session job using GDI.
Qconf allows to add, delete, modify and view the configuration of all kind
of configuration objects. Qconf uses GDI to do this.
Qhost shows the current status of all hosts in the cluster based on information
retrieved using GDI.
Qstat shows the current status of all queues in the cluster based on information
retrieved using GDI.
Qmod allows to modify the state of queues and jobs by using GDI.
2.3 Libraries
In the "source/libs" subdirectory there is one directory for each library.
This low level library provides functionality to run daemons in a monitoring
mode.
A low level library with several utility functions. This library bases
on functionality of librmon.
The generic list library which is used in nearly all components of the
system. The cull library bases on functionality of libuti and librmon.
The communication libraryis the interface providing communication via Commd.
This library bases on librmon.
The Gridengine Database Interface library provides methods to retrieve,
add, delete and modify all internal objects hosted by the Qmaster. The
GDI library bases on libcomm and libcull.
This library contains many important high-level functions mainly used by
Schedd. Some of them however have turned out to be very helpful also in
other clients like Qstat, Qhost or qmon and also in Qmaster.
The event client interface allows a client to register at qmaster and
receive events about changes in the
system more user friendly than with polling.
2.4 Third-party components
The "source/3rdparty" directory contains modules with different licenses
than SISSL.
This client is a GNU make enhanced with transparent remote execution. The
implementation of transparent remote execution bases on a forked 'qrsh
-inherit' command.
This client program is an extension to the popular csh(1) derivative tcsh.
The implementation of transparent remote execution bases on libgdi.
This directory is the home of rsh, rshd, rlogin.
2.5 Utilities
The "source/utilbin" directory contains binaries
that are needed during installation of Grid Engine, for diagnosis purposes
or for the job samples.
2.6 Dist
The "source/dist" directory contains files
and directories which are part of a Grid Engine distribution.
3. Security
The "source/security" directory contains modules handling security.
Contains security related modules.
4. Experimental
Each subdirectory of the "source/experimental" directory contains an experimental
module. Those modules cannot be expected to work - although some may, at
least with a bit of work. The experimental
modules are provided here for information purposes mainly and may be
helpful if you plan to enhance Grid Engine in a direction similar to one
of the modules. Some also can be a valid starting point for such an effort. |